Frequently, our society fails to achieve the progress it is technologically capable of providing for social, political and legal reasons. Often social issues drive political action or inaction, which in turn, creates legal barriers. Most often, the social issues are driven by fear.
The Case Of Affordable Housing
For example, the reason that affordable housing is scarce is not because we don't know how to build adequate housing for modest prices. In fact, a wide array of ways to do so are widely known. Instead, affordable housing is scarce is due to NIMBY (not in my back yard) political pressure at the local government level that officials at higher levels of government have chosen not to address. This is usually effectuated through land use regulations such as zoning and building codes, subdivision limitations, and similar municipal ordinances.
Affordable housing triggers a NIMBY response primarily because existing residents of neighborhoods don't want people who can only afford "affordable housing" living near them. In other words, people oppose affordable housing because they fear poor people. It is the poor people, and especially, concentrations of poor people, that they feel will make their neighborhoods less desirable places to live.
Also, property owners believe that if their neighborhood is less desirable, because it has more poor people in it, that the value of their property will decline, causing them economic harm. And, property owners who reside in a neighborhood at risk of become less desirable are disproportionately wealthy and disproportionately capable of mobilizing local government political action in local governments, compared to the hypothetical people who would move to the neighborhood if affordable housing were built there, for whom real estate developers are the main political proxy. But, real estate developers can often be encouraged to focus on new luxury housing that does not prompt a NIMBY reaction, in lieu of affordable housing with fairly modest tweaks to land use regulations and the hassles that can arise in both the short and long terms when there is community opposition to development.
The fears are often greatly exaggerated and are frequently fueled by socially shunned rationales, like racism and xenophobia. But they aren't entirely baseless either.
People who seek to live in affordable housing, almost by definition, have average or below average incomes.
When a neighborhood has more lower income people, especially when concentrated in an income segregated community, rather than blended in with more affluent neighbors, the neighborhood tends to have homes that are less well maintained, more unruly behavior, less respect for minor social norms like picking up dog poop and not littering, more rudeness in the community, more crime, more gang activity, more domestic violence, and more child abuse and neglect.
Most of these impacts of lower income residents also drive up the cost of taxpayer provided services in the neighborhood that must be paid for, in part, with local tax dollars that disproportionately paid by higher income residents (even when local taxes are regressive) who don't directly utilize the services that those tax dollars finance.
Lower income people are disproportionately black, Hispanic, low skilled immigrants, or "white trash" (i.e. culturally Appalachian and/or poor Southern and/or poor rural in their origins). They are disproportionately less educated. They are disproportionately single parents and unstable cohabiting couple who have some children who aren't shared by both parents.
Higher income people are disproportionately, Anglo whites with urban and "Yankee" origins, or are East Asian or South Asian, and/or are highly skilled immigrants. They are disproportionately more educated. They are disproportionately stable married couples who if they have children have only children shared by both members of the couple.
When a neighborhood has a mix of lower income people and higher income people, especially when there are stark demographic differences between them, members of these two communities often don't share the same social norms and priorities in ways that manifest in the community from parenting practices displayed in public to ideas about how it is appropriate to behave at home when one is home and how to maintain one's home and how to interact with neighbors.
Another factor that adds potency to opposition to affordable housing, where this factor is present, is that most U.S. jurisdictions determine which public schools that a school aged child may attend from of charge at taxpayer expense based upon where a child lives. Rightly or wrongly, parents and community members routinely evaluate school quality by the absolute academic performance level of the children at a school (rather than "value added"). The best and predominant predictor of the absolute academic performance level of children at a school is the socioeconomic status of the parents of the children at the school. So, when affordable housing allows less affluent people to move into a neighborhood, this is perceived as reducing the quality of the local schools, which also makes the neighborhood less desirable. Further, rightly or wrongly, many parents fear that exposure to children who are culturally different from their own children will be a "bad influence" on their children.
When these socio-economic and cultural divisions are absent, neighbors are often unconcerned about housing that they would normally oppose.
Colleges routinely house students in dorms that have very high residential density and don't provide residents with even their own bathrooms or kitchens. Military bases routinely house soldiers in barracks that have the same character. But neither kind of housing generates the same kind of opposition in neighboring communities.
Neighbors rarely mind "granny flats" (a.k.a. "accessory dwelling units) that are actually used to house the parents of the residents of a primary home, or are used to house older children of the residents of a primary home, or are used to house household servants of the residents of a primary home, likewise, rarely generate community opposition.
The main reason that there exists a class of "senior housing" restricted to older residents, which is almost always relatively high in residential density and is often relatively affordable, is the neighbors don't fear elderly neighbors who need someplace affordable to live as much as they fear younger people and families who need an affordable place to live. Elderly people are perceived (mostly accurately) to be unlikely to be significant sources of crime, unruly behavior, gang activity, troubling parenting practices in public display, incivility, and neglectfulness in maintaining their homes.
The Case of Transit
These issues discourage the level of residential density that create an environment that is favorable for transit, and for communities that are geographically compact enough to make getting around by walking and bicycling very viable for a large share of daily trips.
In addition to the indirect effects that socially driven barriers to high residential density and affordable housing create, similar factors also directly discourage well functioning transit systems.
In the United States, with just a handful of exceptions, such as resort communities, some college towns, and extremely dense large central cities (mostly in the Northeast and Pacific Northwest), municipal bus systems are predominantly used by people too poor to be able to afford to have a car of their own, by people whose illegal conduct has caused their driver's licenses to be revoked, and by low to middle income disabled people who can't drive.
One of the main barriers to increasing municipal bus ridership by people who have other options is the fear of incivility, unruliness, and crime from fellow riders. Prospective riders also disinclined to mingle with riders who they observe are often so ill kept that they are smelly because they haven't had the ability to shower or wash their clothes, beg of money, or otherwise simply make them uncomfortable to be around.
But unless one can overcome a tipping point caused by these barriers, the fact that only those people who absolutely must take the bus do so, means that the frequency of service is lower, especially in communities where fewer people have no choice but to take the bus, and that the ridership per bus that does arrive, especially in these more affluent communities, is often much lower. But the resulting low frequency of service makes taking the bus a slower and less viable alternative to driving, and the low utilization level makes the subsidy per passenger mile higher, increasing the tax burden created by the bus service and generating political opposition from the more affluent people who provide the taxes that pay for these subsidies.
This happens, to a lesser degree, in intercity buses, which people avoid in favor of trains and planes and driving when they can.
The very same people who are loathe to take city buses, have no problem at all traveling by a chartered tour bus, or a school bus to a sports event or field trip, or to sending their children on a school bus to school. The issues are not technological, they are social.
Similarly, Israelis continued to make heavy use of buses even when suicide bombing tactics were common. Europeans, Latin Americans and the Japanese, all of whom mostly have systems that have surpassed a tipping point that prevents their systems from falling into a vicious cycle, make much heavier use of municipal buses and fixed route transit systems.
I'll address solutions and ways to address social, political and legal barriers to progress in a follow up post to this one.