It's tough. My inner progressive is "fuck those NIMBYs, people need affordable housing!" But I also bike past two different sets of affordable housing twice a week for soccer.
One is near our soccer field and playground. The playground is mostly kid free these days as there are all sorts of needles and other paraphanalia strewn about. Meanwhile, there are usually people smoking hard drugs right by the pitches. Most of us don't leave our bikes at the parking rack as you can come back to a stripped bike or having had the spokes kicked, presumably as punishment for locking our bikes.
The other one is a little out of the way of most people. The street by it is still littered with refuse and drug paraphanalia and for some reason, they seem to occasionally move rocks into the street (as I painfully learned one day when not paying enough attention.)
If I had kids, I'm not sure I'd want that around my kids either. I know people who live in the same area as one, in what is otherwise a nice part of town. Some of the girls are afraid to go outside alone at night in their own neighbourhoods, and frankly I don't blame them. I'm a scaryish looking dude and keep my head on a swivel, headphones off whenever I visit.
I don't know what the answer is but I don't blame the NIMBYs for feeling like they do.