A recent article in The Economist illustrates the fact that opposition to immigration has very little to do with absolute levels of immigration.
In Japan, just 1.7% of the population is considered non-Japanese (about 2.2 million out of 127 million), of which 400,000 are people of Korean descent born in Japan and assimilated almost entirely into its culture (some even have parents born in Japan). Thus, only about 1.4% of the people in Japan are foreign born.
Another large share of immigrants in Japan, probably about 350,000 people, are of Japanese descent and came to Japan since the late 1980s from Brazil and Peru "to work in the industrial clusters around Tokyo and Nagoya in Aichi prefecture that serve the country’s giant carmakers and electronics firms." The immigrants may not be as assimilated, but look much more like typical Japanese natives and often have Japanese last names.
There are about 600,000 Chinese immigrants and about 200,000 first generation Korean immigrants.
Despite these very low absolute levels of immigration, and the fact that Japan has among the highest life expectencies and lowest fertility rates on the planet, creating a shortage of Japanese residents of working age, there is great political and public concern about immigration, with many of the Japanese wanting to restrict immigration further.
Also, notably, the Japanese see profound differences between East Asian ethnicities, like Koreans, the Chinese, and the Japanese, that would all be lumped into a single sub-ethnicity or race by American demographers, something that illustrates how socially constructed the boundaries of race are, even though the differences in apparence that come with ancestry are purely hereditary.
By comparison, the United States, with a population of a little more than 300 million people, has a foreign born population in excess of 33 million (about 12 percent of the total population), and the U.S. considers everyone born in the United States to be an American (as well as many naturalized U.S. citizens who are foreign born). About 19% of the residents of Canada and Australia are foreign born.
If I recall correctly, there have also been studies that show that the greatest political opposition to immigration comes from places like the American South, where immigration is least common in absolute terms, while places like Southern California, the Texas border and New York City with very large immigrant populations, tend to support more lenient immigration policies. Familiarity breeds comfort, not contempt.