Orthogonal Decomposition of the Genetic Variance for Epistatic Traits Under Linkage Disequilibrium—Applications to the Analysis of Bateson-Dobzhansky-Müller Incompatibilities and Sign Epistasis

The one-century-old theory of orthogonal genetic variance decomposition originated the field of quantitative genetics and has kept on being improved ever since. Recently, serious concerns about the possibility of attaining a satisfactory implementation of genetic variance decomposition with linkage disequilibrium (LD) and epistasis have been raised. In this paper we dissipate such doubts by completing the classical theory of variance decomposition into additive, dominance and epistasis components with LD. We apply that theory to the analysis of the genotype-to-phenotype maps of two cases of particular evolutionary interest—Bateson-Dobzhansky-Müller incompatibilities and sign epistasis. For the first case we show how negative LD and reduction of heterozygotes may contribute to maintain genetic variability after secondary contact. For the second case we show that LD transforms the set of frequencies leading to an evolutionary plateau into a ridge. Our theoretical developments reassuringly reflect the complexity LD conveys to genetic systems throughout novel properties—as compared with systems under linkage equilibrium. We argue that such particularities might have actually contributed to cause confusion about the feasibility of developing this methodology. In any case, the theory we provide in this paper enables new perspectives in both evolutionary and quantitative genetics studies.