We've neglected our infrastructure for decades. Both the physical stuff like housing, transit, community centers, and sewers; but also the human side, like doctors, nurses, and skilled trades.
Once we get ourselves to the point where everyone has a doc, people can get around easily, and schools have enough space/teachers, we'll be ready to invite a significant number of people in.