This is the price you pay for living downtown in a cat box. Personally I like to living in a house in the burbs away from all the scum.
While the scum in your community may not be homeless, that by no means they aren't scum... and most likely they are far worse, on a much larger scale.
...and personally, give me a modern and stylish cat box, where there is actually some culture and vibrancy, over more square footage I would have to clean everyday, grass to cut, carbon copy blue collar neighbors all in a race to one-up each other, living in cookie-cutter houses, all nicely grouped in the trendiest of subdivisions.
I've lived that life, and I think I'd probably take homeless people living on my balcony, before I ever went down that path again.