By doing so, you escape the usual shop frontages and can drink in the beauty of art nouveau architecture, masterful brickwork and, if voyeuristically inclined, the open windows of unsuspecting residents.
Tilt your head back a couple more degrees and another of the city's wonders becomes apparent: castles in the air.
This is the phrase I use to describe those small, odd, seemingly unnecessary additions to the residential skyline: tiny rooms and miniature towers that seem to serve no real purpose other than to act as a playful architectural flourish. They are remarkably abundant in the art nouveau district but also crop up on the wooden buildings of Agenskalns and Kalnciema district, and even occasionally smuggle themselves onto the roofs of brick-built functionalist apartment blocks in the Soviet-era suburbs.
The thing they have in common, other than their innate kookiness, is the fact that they are too small to be of much practical use as living space. According to modern modes of how to build they shouldn't exist, or else they should be exploited and expanded into grimy hot-house atriums or a 'studio' for someone who toys with the idea of picking up a paintbrush.
Certainly some do have limited practical uses as disguised windows for attic space or to encase a rusted-out old water tank in an elegant way. It can be a mistake to find out too much about what's inside them. A few times I have gained access and been saddened to observe nothing more than storage space for a few stacked chairs or, worse still, to see that up close the windows are gone and the dream room in the clouds is just a dirty doss-house for pigeons.
But to me they seem to perform a role similar to the 'folly' on an aristocratic estate: intended more as a stimulus to imagination than to serve any mundane purpose.
They command the best views across the rooftops of Riga. From down on the street you imagine standing inside them, shoulders hunched and slowly rotating like a human lighthouse bulb to take in the panorama of the city.
Or again, you might press your nose, Gulliver-like, up against a doll's-house window, peering at the telephone wires, shingles and chimneys like the pilot of a bathysphere surveying the unseen sea bed.
From one of these little outposts you can usually see several others, of different shapes and styles but all possessing that strange fairy-tale quality that makes it easy to believe these are the residences of fairy-folk, rūķīši or similar diminutive beings who pass their mysterious messages about our conduct from one outpost to the next, far above our unsuspecting heads.
It's nice to think these elevated beings live in a good deal of luxury, having long ago (perhaps by arrangement with the original architects, and in return for inspiration) secured this chain of valuable interconnected real estate across the capital.
So if you happen to be a real estate developer with a project on the drawing board, you would be well advised to include an unobtrusive but intriguing castle in the sky for the unrestricted use of these secret guardians, even if it interferes a little with your clean steel lines.