There's a lot of ways to get at least a surface level understanding of the general affluence of an area.

How well upkept are the buildings? Are they clean and well maintained or are they dingy and broken with overgrowth? Are there a lot of open shops around? Do people seem to be buying things? What kind of clothes are the people wearing? Does it seem like many people are homeless? How is the state of the transit (both public and private?) Do people feel the need to have bars on their windows and security stationed around to prevent theft, or do storefronts feel safe enough to even have merchandise sitting out? Are people eating in restaurants? Are those expensive or cheap restaurants? Do people seem to be comfortable spending a night out on the town, going to bars and shows or are the streets empty because people can't afford outside entertainment?

You're describing a scene that I have seen in a fairly un-affluent country (namy, Thailand back in the day) that looked fairly affluent (central Bangkok, the place was thriving). There were a lot of beggers, I grant you. But homelessness beggars are famously common in some of the most affluent places in the world, like San Francisco. It is probably more linked to outdoor min & max temperatures than anything else.

You can find relative class status of an area by the number of beggars, I grant you that. But you aren't going to get a bead on the actual affluence of the high-class people. Are they thousandairs or billionairs? All we really know is they aren't beggars.