The "Undesirable Neighbor" Theory of Gentrification
The key insight, I think, is that you are not indifferent to who your neighbors are. Neighbors could be good or bad neighbors for several reasons.
1. In the most extreme example, no one wants to live near people who are prone to violent crime. That is, people who victimize others are very bad neighbors. (I highlight violent crime because some kinds of crime, like tax evasion, don't have a particularly strong effect on the neighborhood—presumably all taxpayers, not just ones living nearby, are affected.) A neighborhood characterized by high rates of violent crime and property crime is almost certainly going to be regarded as an undesirable, "bad" neighborhood.
2. You want to live near people who interact with you in positive ways—people you want to socialize with, generally. Note that unlike example 1, in this case there may be different tastes, and so you can imagine a neighborhood that is neither "good" nor "bad" but rather good for some people and bad for others. A Chinese-speaking neighborhood might be great for some people but not so great for others. Likewise for a neighborhood with lots of people in their 20s, or lots of people with babies.
3. You also want to live near people who are "good consumers"—that is, they support the kind of businesses and other institutions that you would like to see in the neighborhood. (This is similar to example 2, except that you never need to meet these people—you experience their presence through the variety of businesses etc. in your neighborhood.) Picture a neighborhood with a lot of people who like bánh mì and know what good bánh mì is. In a neighborhood like that, there is a decent probability that there will be one or more good bánh mì places. If you like bánh mì, then having neighbors like this is probably a good thing. (There is some possibility that if you have too many bánh mì-loving neighbors, the local bánh mì restaurants will be over-crowded and over-priced. But unless there is some kind of barrier to entry, the long-term result should just be a lot of bánh mì places.) Note again that unlike example 1, it is possible to have different conceptions of what makes a good neighbor. If you despise bánh mì, then you probably won't be thrilled if all your neighbors love it. On the other hand, you may enjoy having beautiful churches in your neighborhood even if you aren't religious. So it's not strictly that you want to live near people who are like you. It's that you want to live near people who support the existence of things you want to see in the neighborhood.
Okay, so, you care who your neighbors are. But there is no true "market for neighbors." You generally can't pay people to live near you or to move away. The real estate market functions as a rough proxy for the "neighbor" market (which, again, doesn't exist). And in fact, I would say for many urban neighborhoods, the "neighbor" shadow-market overwhelms other considerations. Sure, you care about things like mass transit access, parks, that kind of thing. But you also care tremendously about the mix of businesses and the safety of the neighborhood. There are good neighborhoods in New York City with relatively poor access to public amenities, while until recently there were bad (or at least cheap) neighborhoods right next to Prospect Park.
We can see how two kinds of neighborhood can emerge. First, an easy case: if people want to live near other people who are like them, they will form distinct clusters. So there are ethnic neighborhoods, young neighborhoods, etc. These neighborhoods aren't necessarily good or bad, they are just different. (A related point is that ethnic neighborhoods may disappear as the ethnic group becomes affluent and accepted in the community. So the existence of an ethnic neighborhood may be an indicator of some problem with society. Similarly, gay neighborhoods may go into decline as gay equality advances. But that is beyond the scope of this post.)
You can even imagine a "good" sense in which rich neighborhoods and poor neighborhoods emerge. Rich people want to shop at fancy organic stores, yoga studios, that kind of thing. Poor people want cheaper stores, cheaper apartments, etc. and rarely shop at luxury stores. So we might not be overly concerned to see hugely unequal neighborhoods—people are just sorting according to their preferences.
But there is a darker possibility. We could observe segregation by wealth because people are essentially buying proximity to good neighbors (or distance from bad neighbors). Now, this is a delicate area, so I want to be very precise. Poor people can be excellent neighbors, and rich people can be terrible neighbors. But some of the worst neighbors are criminals, and relatively few affluent people are street criminals. (I'm not saying that rich people are less prone to crime than poor people. Recall that some kinds of criminality, which may be more prevalent among wealthy people, have much smaller effects on the criminal's immediate neighbors. I would much rather raise a family next to an inside trader than next to a violent criminal.)
So one way to make a neighborhood "good" is to make it expensive. This doesn't guarantee great neighbors, but it largely eliminates the worst neighbors, the violent criminals. Rich people are playing a coordination game in which they cluster together around the best public amenities (parks, museums, subway stops, good public schools, old trees, beautiful buildings). Poor people are left not just with smaller apartments (that is next to inevitable given their lower incomes), but with less public safety, worse access to public goods (including good schools), worse access to transportation (and therefore worse access to good jobs), etc.
So it's a very unfair process. It basically stamps the class system onto the physical geography of a city, so that which neighborhood you grow up in plays out outsized role in determining your opportunities. But I am getting ahead of myself, so I will stop here, and in my next post I will look at the process of gentrification.