🏠
Gene trees and the multispecies coalescent
Lecturer: Alexei Drummond
13 August 2026

One species, many trees

Sequence two different genes from the same set of species and you will often get two different trees. Neither of them has to be wrong.
  • Last lecture: within a single population, the coalescent makes gene trees a random product of the population's history.
  • That randomness does not stop at the species boundary. A gene's ancestry runs back through ancestral species that were themselves populations with variation in them.
  • The multispecies coalescent is the model that puts gene trees inside a species tree and treats the difference between them as something to be expected rather than explained away.

The boundary between population genetics and phylogenetics

Closely related species are exactly where the two fields collide:

  • Different genes give different tree topologies, through incomplete lineage sorting.
  • Divergence times between species are systematically overestimated if you read them straight off a gene tree.
  • Sometimes the species identity of an individual is not known, and sometimes species are delimited from a split in a single gene tree, which is a fragile basis.

Where the disagreement comes from

Inside a single population, lineages drift. Some leave many descendants, some leave none, and which lineages survive is largely luck:

When a species splits, the variation that was present in the ancestor is divided between the daughter species. Two gene copies that end up in different species may have separated long before the species did.

Ancestral polymorphism

If two species split faster than the ancestral population could sort its own variation, the gene lineages carry no memory of the order in which the species separated.

Try it yourself: incomplete lineage sorting

measured in multiples of the ancestral population size: how many generations the branch lasted, divided by how many individuals were in it
Try this. Press 500 genes. Then drag the internal branch down to 0.1 and press it again. When the branch is short compared with the population, all three tree shapes become almost equally likely.

What that simulation shows

  • Two gene lineages can only be sorted by a species split if they coalesce inside the ancestral branch. If they do not, they enter the older ancestral population still separate, and there they meet at random.
  • Then each of the three possible groupings is equally likely, so two thirds of unsorted genes disagree with the species tree.
  • What decides this is not the length of the branch in years but its length relative to the population size. A long branch in a small population sorts cleanly; a short branch in a large population does not.
  • Disagreement between gene trees is therefore not noise or error. It is a prediction of the model, and its frequency is informative about ancestral population sizes and divergence times.

This is not a hypothetical

Human, chimpanzee and gorilla

Humans and chimpanzees are each other's closest living relatives. Even so, Scally et al. (2012) estimated that for roughly 30% of the gorilla genome, the gorilla sequence is closer to the human or the chimpanzee sequence than those two are to each other.

  • The human and chimpanzee lineages separated only a few million years after the gorilla lineage did, and the ancestral ape population was large. That is exactly the regime the simulation shows.
  • A single gene, or a single genomic region, is therefore a poor witness to a species tree. Consistency across many independent loci is what carries the signal.
Scally et al., Nature, 2012

Monophyly is not guaranteed

Sample several individuals per species and a species may come out monophyletic, paraphyletic or polyphyletic in a given gene tree, purely through incomplete sorting. A species that fails to be monophyletic in one gene tree has not thereby been refuted.

Divergence times come out too old

  • Two gene copies in different species must have separated at or before the species split, never after.
  • So the depth of a gene tree overshoots the depth of the species tree, by an amount that depends on the ancestral population size.
  • The error is worst exactly where it matters most: recent, rapid radiations.

How large is the error?

Edwards and Beerli, Evolution, 2000

The multispecies coalescent

Trees inside trees

  • The species tree is a set of populations joined by speciation events. Each branch has a duration and a population size.
  • Each gene tree is generated by running the coalescent inside those populations. Lineages can only meet once they are in the same branch.
  • Every gene sees the same species tree but gets its own independent draw, which is why they disagree.
In the language of the Bayesian lecture: the species tree is the prior on gene trees, and each gene's alignment is the data for its own gene tree.

What a multispecies coalescent analysis estimates

The species tree topology and divergence times
Corrected for ancestral polymorphism, so no longer systematically too old.
Ancestral population sizes
Estimated from how much the gene trees disagree. A branch with many discordant genes had a large population, or was short, or both.
A separate tree for every gene
Each with its own uncertainty, all constrained to fit inside the same species tree.
All of it sampled jointly by MCMC, exactly as in Lecture 8.

Four gene trees inside a three-species tree

The species tree is the shaded outline. The four gene trees inside it are each a valid history, and they do not have to agree with one another.

Why concatenating genes is a bad idea

the true species tree
one tree from all the genes glued together
Gluing the alignments together forces one tree on genes that genuinely have different histories. The result can be confidently wrong: strong support for a tree that is not the species tree.

A rapid radiation of seven species

species tree
species tree with one gene tree inside it

Short internal branches leave almost no opportunity for sorting, so the gene tree wanders freely inside the species tree.

Pocket gophers

  • 27 individuals, 7 loci (Belfiore, 2008).
  • Analysed jointly under the multispecies coalescent in BEAST (Heled and Drummond, 2010).
  • The result is a species tree with credible intervals on its divergence times, and branch widths showing estimated population sizes.

What this changes in practice

  • Do not concatenate. Analyse loci as separate genes sharing one species tree.
  • Do not read a species tree off one gene. A single gene tree, however well supported, is one draw from a random process.
  • Disagreement is data. The proportion of discordant loci is what identifies ancestral population sizes.
  • Not all discordance is lineage sorting. Hybridisation, introgression and horizontal transfer also produce conflicting gene trees, and the multispecies coalescent alone cannot distinguish them. Network models extend it to try.
  • It is expensive. Every gene tree is an unknown, so the state space grows quickly with the number of loci.

Summary

  • Gene trees and species trees are different objects. A gene tree is generated by the coalescent running inside the branches of a species tree.
  • When a species branch is short relative to its population size, lineages fail to sort within it, and the gene tree disagrees with the species tree. Two thirds of unsorted genes will disagree.
  • This is common in real data: roughly 30% of the gorilla genome does not follow the human and chimpanzee grouping.
  • Reading divergence times off gene trees makes species look older than they are, by an amount set by the ancestral population size.
  • The multispecies coalescent estimates the species tree, its divergence times and its ancestral population sizes from many loci at once, using the disagreement rather than being misled by it.

Recommended Reading

Decoding Genomes (Stadler et al., 2024)
  • Chapter 9: Gene trees, species trees and the coalescent

Also useful:

  • Degnan & Rosenberg (2009), "Gene tree discordance, phylogenetic inference and the multispecies coalescent", Trends in Ecology & Evolution 24:332–340.
  • Edwards (2009), "Is a new and general theory of molecular systematics emerging?", Evolution 63:1–19.