Neanderthals and Haldane's Law

Autosomal DNA studies comparing ancient Neanderthal DNA to modern human DNA from the farthest reaches of the Out of Africa world, have concluded that modern humans all have 1% to 4% Neanderthal ancestry, relative to the Neanderthal ancestry, if any, observed in Africa.

This was a surprise. A comparison of Neanderthal mtDNA (which is inherited in the maternal line uniparentally) with that of modern humans (which has been studied extensively) shows that no modern humans alive today have a direct Neanderthal matriline ancestor. There is also strong evidence that all Y-DNA in men alive today (Y-DNA Adam), in African and outside it, have a common ancestor within a time frame younger than the common maternal ancester (mitochondrial Eve).

How can this be explained?

I think it can be done with four assumptions: (1) Neanderthals and early modern humans largely lived in separate tribes of hunter-gathers that were predominantly endogamous, (2) the effective population size of the Eurasian founder population which was interacting with Neanderthals was small, (3) hybrid Neanderthal-modern human children ended up with their mother's tribe, and (4) Haldane's law (a.k.a. Haldane's rule) caused most first generation hybrids of Neanderthals and modern humans to be females or infertile males.

The possibility that there were a small number of outliers that deviated from this assumption (out of an already very small number of instances of admixture) can be accounted for via some combination of random chance and selective advantages that early modern humans may have had over Neanderthal hybrids in the context of early modern human tribes.

The Level Of Neanderthal-Modern Human Admixture Was Low

It is reasonable to infer that the 1%-4% level of Neanderthal admixture in non-Africans implied a low level of admixture. The archaeological evidence strongly suggests that during the period when Neanderthals and early modern humans shared the same territory, in the Levant and in Europe, that there were Neanderthal tribes and early modern human tribes, rather than tribes of significantly mixed populations.

This is consistent with the from data in existing human populations that there are strong endogamous tendency now even between populations far more culturally, genetically and phenotypically similar populations than those of the Neanderthal and early modern humans in the period when they shared territory or adjacent territories.

The Effective Size of the Eurasian Ancestor Founding Population Was Small

The phylogeny of uniparental genetic markers (Y-DNA and mtDNA) provide ample evidence that all Out of Africa populations (with the possible exception of a small wave of men with Y-DNA D whose Y-DNA is least closely related to that of other Eurasian Y-DNA haplogroups), descend from a single common founder population whose ancestral matriline population was most similar to that of women with an African specific mtDNA haplogroup found most often in and around Ethiopia.

These Eurasians, in turn, all descend from a relatively modest sized population of ancestors of all humans (including Neanderthals) within the last 1.2 million years.

[Scientists] put the number at 18,500 people, but this refers only to breeding individuals, the “effective” population. The actual population would have been about three times as large, or 55,500. . . . Geneticists have long known that the ancestors of modern humans numbered as few as 10,000 at some time in the last 100,000 years.

According to a 2007 total genome analysis of modern humans, the effective population size of all non-African modern humans is much smaller, an effective population of about 3,100, which translates to about 10,000 in the community as a whole.

Thus, to have an average 2% Neanderthal contribution to this population suggests an effective Neanderthal population of about 61 individuals over the duration of the entire period when an early modern human Out of Africa founder population was in contact with a Neanderthal population. If this happened over a thousand years or more with twenty year generations, we are talking about admixture rate on the order of one or two hybrid Neanderthal children per generation in the entire community.

Explaining The Lack of Neanderthal mtDNA

So, Neanderthal/human hybrids are quite rare. Suppose that all of those children are the product of some combination of (1) brief liasons of less than nine months, perhaps during periodic meet ups of human and Neanderthal tribes for trade or coordinated hunting, rather than sustained relationships, (2) rapes, and (3) long term relationships that were matrilocal (i.e. the husband joined the wife's tribe), something that is is characteristic of the !Kung San of Southern Africa who are frequently considered archetypical to the greatest extent of any modern population of the kind of society that was present in the hunter-gatherer era of early modern humans.

The result, if any combination of these scenarios is true (or overwhelmingly dominant, an outlier or two doesn't necessarily produce a different result), is that the children ended up in the mother’s tribe and not the father’s as a result. Thus, children of Neanderthal mothers ended up in Neanderthal tribes, and children of modern human mothers ended up in modern human tribes.

There could have been as many or more hybrid Neanderthals with Neanderthal mothers as there were hybrid Neanderthals with modern human mothers. The hybrid Neanderthals with Neanderthal mothers would have had Neanderthal mtDNA, although for reasons discussed below, they would be predominantly girls who would lack the modern human Y-DNA of their fathers. Their affiliation with Neanderthal tribes would have sealed their fate. These hybrid children and their descendants would have died out with the rest of the Neanderthal population of which they were a part.

Meanwhile, descendants of hybrid Neanderthals with modern human mothers would have been part of a population that did survive, and their descendants survived to become about 2% of the future Eurasians (they were probably small in number but introduced when the founder population of Eurasians was quite small).

But, since Neanderthal populations did not survive and only hybrid children with modern human mothers who lacked Neanderthal mtDNA managed to have modern descendants, no surviving Neanderthal descendants in modern human populations would be expected to have any Neanderthal mtDNA.

Explaining The Lack of Neanderthal Y-DNA with Haldane's Law

According to Haldane’s law, the heterogametic offspring of interspecific hybrids are likely to be absent, rare, or sterile (Short, 1997). If Haldane’s Law applied to the offspring of H.neanderthalensis and H.sapiens, we would expect to find female hybrids quite commonly, but male hybrids much more rarely.

From here.

Thus, of the sixty-one or so first generation fertile hybrid children whose descendants have survived from the Eurasian founder population, Haldane's law implies that all or almost all would have been girls, who would have had neither Neanderthal mtDNA (since they had modern human mothers) or Neanderthal Y-DNA (since they were girls). There might have been some additional Neanderthal hybrid males, but they would have disproportionately or entirely been infertile or have reduced fertility, one of the classic traits that evolution selects against.

As a result, we would expect what we observe, that there are little or no traces of Neanderthal Y-DNA or mtDNA in modern humans, despite the fact that a significant minority of our autosomal DNA is attributable to Neanderthal ancestors, all of whom among our collective ancestors would have been female children of modern human women and Neanderthal men.

Why Do All Eurasians Have About The Same Amount Of Neanderthal Ancestry?

Upper Paleolithic early modern humans in Europe were in contact with Neanderthal populations much longer than those elsewhere. Why don't they have more of a Neanderthal genetic component?

The Initial Admixture Period

A source for the common Neanderthal ancestors of all Eurasians can be explained fairly easily. Neanderthals admixed with the founder populations of the Out of Africa early modern humans before it expanded and dispersed. One place where we have archaeological evidence that this could have happened in in the Levant from about 100,000 years ago to 75,000 years ago, give or take, after which modern humans were absent from Europe and the Near East for 25,000 years or so. We know that both Neanderthals and early modern humans were present there close in time, more or less adjacent to each other, although the details are fuzzy. If that first exit from Africa was followed by a retreat to Ethiopia and second exit from Africa via the Sea of Tears, rather than being an expanding node of an Eurasian founder population in Africa that went extinct, the admixture could have happened then.

Alternately, "Eurasian Eden" could have been in the Persian Gulf, Iran or South Asia (i.e. greater India) where there is also evidence that there was a Neanderthal presence, prior to expansion of the Out of Africa founder population into different new lands. The presence of evidence of Neanderthal descent in Australia puts a "Eruasian Eden" admixture event sometime prior to 45,000-50,000 years ago, but that isn't sufficient to distinguish admixture in the first wave Levant scenario, from admixture in a later Eurasian Eden scenario.

The Case The Highly Admixed Populations In Europe Were Greatly Diluted

The harder question is why Europeans don't have elevated levels of Neanderthal admixture. This can also be explained.

There is good reason to believe that Upper Paleolithic early modern humans who had much longer periods of interaction as a population with Neanderthals than any other modern human population probably did have greater levels of admixture of Neanderthals than other ealry modern humans. Indeed, there is some evidence from physical anthropology, i.e. old bones, that they were more heavily admixed with Neanderthals.

But, there is also good reason to believe that the population of Upper Paleolithic early modern humans with elevated levels of Neanderthal admixture make up a pretty small component of the ancestors of modern Europeans.

They were hunter-gatherers. They were forced to retreat to refugia in Southern Europe as the ice pack expanded, where they would have had to compete and mix with local early modern humans who had not had contact with Neanderthals, and their numbers would have been greatly reduced relative to early modern humans in the Near East by the ice age. As the glaciers retreated from the Last Glacial Maximum reached around 20,000 years ago (at which point Neanderthals had been extinct for 10,000 years or so), there was an influx of modern humans into the newly opened virgin territory not only from the European refugia, but also from adjacent areas like the Near East and Siberia where populations that had not had contact with Neanderthals lived. Thus, the Neanderthal component of the gene poool in the hunter-gatherer populations of Europe around 15,000 years ago would have been significantly smaller than the Neanderthal component of the gene pool in the hunter-gather populations of Europe around 30,000 years ago (right around the time that further admixture with Neanderthals would have ceased because Neanderthals went extinct).

Then, there was massive demographic shift as these European hunter-gatherers saw fishers, pre-farmers and farmers in and immediately before the European Neolithic Revolution from Anatolia, Mesopotamia, the Levant and perhaps the Caucuses, all with low levels of Neanderthal admixture, outnumber them in Europe and take control of the territory of the post-Last Glacial Maximum hunter-gatherers, starting around 6,000 BCE (8,000 years ago). Some of the older hunter-gatherer population members may have been assimiliated into farming populations, but they would have been a small minority component. The evidence seems to indicate that hunter-gatherer populations and farmer populations had pretty limited admixture with each other for the first thousand years or so of their interactions with each other.

Subsequent waves of population influx into Europe from areas where modern human populations had not admixed with Neanderthals after the "Eurasian Eden" period, most notably the influx of population associated with the switch to Indo-European languages in Europe from earlier non-Indo-European languages, and various populations migrations documented in history, would have diluted European exceptionalism in Neanderthal admixture even more.

Estimating The Genetic Contribution Of "Interacting" Hunter-Gatherers In Europe

Nobody disputes that substantial dilution of this kind has happened in Europe's population history. But, there is intense dispute over how diluted the indigenous hunter-gather population was by later populations.

Estimates of the genetic contribution of people who arrived in Europe as farmers in the last 8,000 years range from about 80%+ at the high end, to 20% at the low end. I personal favor the high end estimate based on ancient DNA evidence, but respect the fact that the data can be interpreted in different ways, particularly in Atlantic/Mediterranean Europe and North Africa where the genetic break between hunter-gatherer ancient DNA and early farmer ancient DNA isn't nearly so stark. But, most of those who favor a low end figure for a farmer contribution tend to see a major expansion in European hunter-gatherer populations from ca. 18,000 years ago and a fairly fuzzy notion of the regional origins of the population in the pre-LGM period. Even if Europeans are 80% descended from post-LGM hunter-gatherer populations, that doesn't tell you how strongly they are descended from hunter-gatherer populations that interacted with Neanderthals.

There is likewise no serious dispute that the hunter-gather population of Europe immediately before farmers arrived had a different population genetic mix than the hunter-gatherer population of Europe as of 30,000 years ago. Almost the entire continent was evacuated and then repopulated and it would be remarkable indeed if the population genetic mixes were the same, given different mixes in different refugia and different rates of population expansion from each.

It is also uncontroversial that the hunter-gatherer population of Europe 8,000 years ago was probably enriched with populations that had not had significant contact with Neanderthals after the Eurasian founder population era relative to the hunter-gatherer population of 30,000 years ago who interacted with Neanderthals. But, to my knowledge no one has ever seriously quantified the percentage of 8,000 years ago European hunter-gatherers who were descended from 30,000 years ago European hunter-gatherers.

I personally believe that a 10% contribution from hunter-gatherer populations that had contact with Neanderthal populations is a pretty high estimate, but the numbers are admittedly fuzzy. Even if the enriched Neanderthal descent population made a 20% contribution to the population genetics of modern Europe, the conclusions wouldn't be all that different. But, if I am wildly wrong, and the gene pool of modern Europe is really 80% from populations that admixed significantly with Neanderthals, then one needs a different explanation (probably natural selection against Neanderthal genes which is the opposite of what you would expect in an environment where Neanderthals had a several hundred thousand year head start in adapting to local conditions or a major methodological flaw in determining the Neanderthal percentages) to explain the data.

The notion that the hunter-gatherer populations of Northern Europe had an enriched Neanderthal admixture around 30,000 years ago is itself controversial, and even among those who accept that idea, no one has to my knowledge made credible estimates of the population wide levels of Neanderthal admixture in that population. It would have been higher than for other Eurasians, and well under fifty percent. The notion that each episode of Neanderthal-modern human contact would have produced similar levels of admixture, seems like a reasonable assumption, and this would suggest it might have been two or three times that of other Eurasians (perhaps 5%-12%). But, the percentage might depend quite a bit on how narrowly the admixing population was defined (e.g. interacting tribes of modern human hunter-gatherers v. all modern humans in Europe). Lower percentages of admixture in the Neanderthal enriched group can compensate for higher percentages of descent in modern humans from the enriched group to fit the reality that we find.

If the gene pool of pre-Last Glacial Maximum European hunters and gatherers had an 8-12% level of Neanderthal admixture, while the average incoming population members had an average 2% level of Neanderthal admixture, and modern Europeans trace 10% of their descent to that most admixed Northern pre-Last Glacial Maximum hunter-gatherer population, then the average European would have 2.6% to 3% Neanderthal admixture, which would be indistinguishable from the data so far.

One would see less of a Neanderthal component in European populations with a smaller pre-LGM hunter-gatherer component in their descent, and more in European populations most strongly descended from pre-LGM hunter-gatherer populations. The Basque, Estonians, the Saami and the people of the Northern Caucuses would be a natural place to start looking for high levels of Neanderthal admixture since they seem to be the most indigeneous of European populations, although it isn't entirely clear how large a share of genetic ancestry any of these populations has going back 30,000 years in Europe. No one can say anything very definitive about the origins of these populations going back more than 12,000 to 18,000 years. There is no doubt that nobody lived anywhere in Northern Europe until that time period, but the majority view is that the populations most similar to the ancient populations of Europe derived from populations who were pushed to the fringes of Europe by the expansion of farming and then either adopted herding and farming while remaining ethnically isolated, or survived in isolated refugia as hunter-gatherers until modern times.

The "uniform level of Neanderthal admixture" fact is based on examination of only a handful of people, only a couple of whom are Europeans, so if Neanderthal admixture percentages are high only in relict populations of Europe or Central Asia, the data so far wouldn't reveal that fact. I wouldn't be surprised at all to find that somewhere in the world, probably in an ethnic subculture with historically established old roots in Europe, there is someone with a 10%-15% level of autosomal Neanderthal admixture.

Since initial Neanderthal admixture would have had to have happened at least 50,000 years ago, and secondary Neanderthal admixture would have had to have happened at least 30,000 years ago, the Neanderthal percentage should be pretty much fixed in any given genetically distinct population cluster by now. If the secondary admixture followed by dilution theory I've proposed is correct, there should be some "old European" populations in Europe that are hot spots of high levels of Neanderthal admixture, but none anywhere else in the world (with the possible exceptions of Iran and Pakistan where the Neanderthal range extended at least at some point).

But, if the vast majority of Europeans trace only a modest percentge of their descent to this ancient population that had sustained contract with Neanderthals and may have had greater levels of admixture, then the vast majority of those populations may have a quite typically Eurasian level of Neanderthal autosomal DNA. For example, it is plausible that any enhancement in the Neanderthal component of European autosomal DNA from the old hunter-gatherer descent element may be overwhelmed by founder effects in the founding populations of the various continental groups of early modern humans in Eurasia.

Consider the following example. If the Eurasian group that broke from the Eurasian founding population happened to have slightly less Neanderthal admixture than the Eurasian group that settled East Asia and Australia and the Americas, for reasons of random chance in a still small community where Neanderthal genes had not yet quite reached a fixed proportion across the entire community, that deficit of Neanderthal ancestry in the founding population might balance out an excess of Neanderthal ancestry attributable to the portion of modern populations traceable of the more Neanderthal admixed pre-Last Glacial Maximum hunter-gatherers of the Northern European fringe.

Suppose that the overall Eurasian population was 2.5% Neanderthal and split into two equal groups, one that went East and the other than went West. Suppose further, that the average person in the East Eurasian founding population was 3% Neanderthal, and the average person in the European founding population was 2% Neanderthal. If the break were along tribal lines that had extended family elements to them, and Neanderthal genes had not quite reached fixation in that population yet, and the founder subpopulations that expanded West and East were reasonably small, a 0.5% random difference in Neanderthal descent of each of the two groups from the average would hardly be exceptional. Suppose further that 10% of the average Europeans genetic ancestors were from a hunter-gatherer population that was 12% Neanderthal, Europeans and East Eurasians overall would have statistically identical proportions of Neandethal descent.

In the case, the only way to distinguish secondary enrichment in Europe would be to look at what specific Neanderthal contributions were present in different populations and to note that there was a large subset of private Neanderthal contribution found only in Europe, and a second subset of public Neanderthal contributions shared by all Eurasians, but present at lower frequencies in Europe than elsewhere. And, even this test would not be a statistically definitive one.