We assume that m SNPs are measured on n individuals. The genotype measurements are denoted by x_ij for i=1,…,m and j=1,…,n. For each SNP, one of the alleles is counted as a 0 and the other as a 1, implying that the SNP genotypes are xij∈{0,1,2} where x_ij = 0 is homozygous for the 0 allele, x_ij = 1 is a heterozygote, and x_ij = 2 is homozygous for the 1 allele. We assume that 𝔼[xij|πij]=2πij for IAF πij. This IAF parameterization allows each individual-SNP pair to possibly have a distinct allele frequency. The classical scenario where there is one allele frequency per SNP is a special case where πi1=πi2=⋯=πin. The conditional expected value 𝔼[xij|πij]=2πij also allows for the IAFs πij to be random parameters, which we assume here.

We utilize an existing coancestry model where the IAFs are random parameters with respect to some ancestral population T that is common to all n individuals [4,5]. (Note that other definitions of “coancestry” and “kinship” exist, but we use the models defined below and in refs. [4,5].) This is a neutral model where

for i=1,…,m and j,k=1,…,n. The parameter a_i is the ancestral allele frequency in T for SNP i and 0≤θjk≤1 is the coancestry for individuals j and k with respect to T. (Note that the a_i and θjk parameters depend on T and could be different if conditioning on a different common ancestral population.) The coancestry model we utilize also makes the assumption used in previous work [4,5,7–12] that xij|πij∼Binomial(2,πij), where the xij|πij are jointly independent. Under this model, it follows that

A one-to-one mapping exists with the identity-by-descent kinship model (often used in GWAS methods), denoted by ϕjk, by matching variances and covariances [4,5]. The parameters map so that

When minjkθjk=0, then T is the most recent common ancestral population [4]. The full set of parameters is denoted by the n×n symmetric matrix Θ with (j,k) entry θjk.

Here, we propose a method to estimate the antecedent population-level coancestry Λ under the super admixture model, with the standard admixture model estimate as a special case. The rationale is to leverage the relationship, Θ=Q′ΛQ. Given values for Θ and Q, we identify values of Λ that make Q′ΛQ close to Θ, while obeying the geometic constraints of Λ (i.e., 0≤λuv≤1 and λuv=λvu).

Given values for Θ and Q, we formulate the problem of estimating the antecedent population-level coancestry Λ under the super admixture model as follows.

Problem 1.

where ‖·‖F represents the Frobenius norm (Appendix A, S1 Text). We utilize the proximal forward-backward (PFB) method [26] to solve this optimization problem, resulting in Algorithm 1 for solving Problem 1. Every sequence of (Λt)t∈ℕ generated from this algorithm is guaranteed to converge to a solution of the corresponding problem. The PFB method and how to employ it in our setting are detailed in Appendix D in S1 Text. The performance of Algorithm 1 is demonstrated in Appendix J in S1 Text.

Algorithm 1. Estimating Λ for the super admixture model given Θ and Q.

Note that since the optimization criterion for fitting Λ is driven by relatedness averaged over the genome as quantified by Θ, there may be localized signals that show interesting admixture events. To this end, one could pursue an optimization on a more localized basis. Also, note that rotations of Q and Λ can yield the same product Q′ΛQ, so it is important to not over-interpret Q and Λ, which is true of admixture models in general [10].

To estimate all components of the super admixture model, one needs estimates of the n×n individual-level coancestry matrix Θ, the K×K antecedent population-level coancestry matrix Λ, the m×K matrix of antecedent population allele frequencies P, and the K×n matrix of admixture proportions Q. There exists a wide range of methods for estimating P and Q [8–10,15–17]. Here, we utilize the ALStructure method [10], which is formulated under the general admixture model with no assumptions about the structure of Λ. Therefore, ALStructure applies equally to both the super admixture and the standard admixture settings. This is also the case for some other methods, such as ADMIXTURE [9]. The method proceeds by deriving a linear basis for Q from X that has theoretical guarantees to span the true basis as the number of SNPs m grows large. A projection-based estimate Π^ of the IAFs is also formed. The quantity ‖Π^−QP‖F is then algorithmically minimized through geometrically constrained alternating least squares to yield estimates Q^ and P^.

We utilize the structural Hardy-Weinberg (sHWE) test [12] for determining the number of antecedent populations K, as outlined in that work. The approach is to consider a range of K values to test the assumption that xij|πij∼Binomial(2,πij) based on the estimates π^ij and a goodness-of-fit statistic with a parametric bootstrap null distribution; K is then parsimoniously chosen to satisfy this modeling assumption from a genome-wide perspective. A method of moments estimator of Θ was derived in [4], where it was shown to have favorable properties and is consistent for the true values under certain assumptions. We denote this Ochoa-Storey (OS) estimate by Θ^OS and review its details in Appendix C in S1 Text. If one has alternative ways to estimate Θ and Q, and to determine K, then those can be used within our framework as well.

Note that one can further calculate a corresponding estimate for individual-level coancestry by

which can be compared to Θ^OS in order to aid in model fit assessment.

We can estimate Λ under the standard admixture model by modifying the constraints in Problem 1. This leads to the Algorithm B described in Appendix D in S1 Text. Algorithm 2 can then be used to form the estimate Λ^std under the standard admixture model with Algorithm 1 replaced by Algorithm B in Line 4. The corresponding estimate for individual-level coancestry can be calculated as Θ^std=Q^′Λ^stdQ^. The performance of Algorithm B is also demonstrated in Appendix J in S1 Text.

Algorithm 2. Estimating Λ for the super admixture model given X.

  input: Genotype matrix X

1 calculate the OS estimate of individual-level coancestry Θ^OS;

2 choose K from the structural Hardy-Weinberg (sHWE) goodness of fit procedure;

3 calculate the estimate Q^ for K via the ALStructure method;

4 calculate the estimate Λ^sup by applying Algorithm 1 with inputs Θ^OS and Q^ ;

5 return Λ^sup

We now introduce a method to generate antecedent population allele frequencies with given coancestry Λ. We note above in Eq (5) that for the standard admixture model, one way to generate allele frequencies pi1,pi2,…,piK is via independent realizations from the Balding-Nichols (BN) distribution: piu∼BN(ai,λuu) for u=1,2,…,K. As there is no default approach to extending this to the super admixture case, we propose a method here called “double-admixture”. The main idea of the method is that we form two layers of allele frequencies: the first layer is composed of independent draws from the BN distribution, and the second layer mixes these to create pi1,pi2,…,piK with coancestry Λ.

Let S be the number of components that will be mixed, W be the S×K matrix of mixture proportions, and Γ an S×S diagonal matrix. The entries of W are w_su where 0≤wsu≤1 and ∑s=1Swsu=1 for u=1,2,…,K. The diagonal values of Γ are represented by γs where 0≤γs≤1, and all other values are 0. Suppose that for i=1,…,m we generate

independently for s=1,…,S, and we then set

for u=1,…,K. It can be verified that

for u,v=1,2,…,K. By matching these equations with Eq (6), one can see that if

then pi1,pi2,…,piK has coancestry Λ as desired. In matrix terms, Eq (10) is equivalent to

Therefore, the double-admixture method is based on the following optimization problem.

Problem 2.

We adopt the proximal alternating linearized minimization (PALM) method [27] to solve Problem 2, resulting in Algorithm 3 for calculating the parameters in the double-admixture method. Every sequence (Wt,Γt)t∈ℕ generated from Algorithm 3 is guaranteed to converge to a critical point. Integrating Algorithm 3 with the generative steps for p_iu described above, Algorithm 4 simulates antecedent population allele frequencies with the desired coancestry. We note that the parameter S should be chosen to ensure that Λ≈W′ΓW while avoiding unnecessarily large values, which may increase computational time without improving accuracy. A practical guideline is to plot ‖Λ−W′ΓW‖F against S and identify a value of S where the curve plateaus; empirically, setting S = 2K works well in the examples considered here. In Appendix E in S1 Text, the PALM method is briefly introduced and the convergence of Algorithm 3 is proved.

Algorithm 3. Calculating W and Γ in the double-admixture method.

Algorithm 4. The double-admixture algorithm for simulating P.

One possible drawback of the double-admixture method is that the approach relies on the existence of W and Γ so that Λ=W′ΓW. We do not currently have a theoretical guarantee for such W and Γ (although one may exist since S can be made large). Therefore, we provide a complementary method in Appendix F in S1 Text, the NORmal To Anything (NORTA) approach [28], serving as a tool for simulating P when the double-admixture method is not applicable. It should be noted that the double-admixture method solves the optimization one time for the entire process so that its running time is independent of the number of loci m. In contrast, the NORTA method has to solve K×(K−1)/2 root-finding problems per locus and therefore has a complexity of 𝒪(K2m), rendering it significantly more time consuming. The performances of the double-admixture and NORTA methods are demonstrated in Appendix K in S1 Text.

Note that if we set Γ=Λ for a diagonal standard admixture Λ and W=IK (where IK is the K×K identity matrix), then the double-admixture method reduces to the BN sampling from Eq (5), which produces valid antecedent population frequencies for the standard admixture model. From this observation, the double-admixture method can be viewed as a generalization of BN sampling.

By utilizing the double-admixture method, we introduce Algorithm 5 to generate genotypes under the super admixture model. When the true parameters (a,Λ,Q) are available, the algorithm can be used for direct simulation. We verify in Appendix L of S1 Text that the generated genotypes satisfy the moment constraints imposed by the super admixture model. When these parameters are not available, the same procedure can be applied in a semi-parametric bootstrap framework by first estimating them from observed genotype data. Specifically, Q^ can be obtained with ALStructure, Λ^ with the proposed super admixture estimator, and a^ with the vector of observed allele frequencies from each SNP. In this setting, the algorithm yields bootstrap replicates X* to recapitulate the structure and relatedness of the observed data X. Importantly, the bootstrap process does not merely resample genotypes from a fixed matrix of estimated individual allele frequencies (IAFs). Rather, the antecedent population allele frequencies are resampled, also leading to resampled IAFs, so both evolutionary and statistical resampling occur.

Algorithm 5. Generating genotypes from the super admixture model.

Here, we develop a hypothesis test of the standard admixture model (null) versus the super admixture model (alternative). We show below that on real data sets the test results are highly significant against the null in favor of the alternative. In terms of model parameters, the test is defined as follows:

A straightforward test-statistic is U=‖Λ^sup−Λ^std‖F. The larger U is, the more evidence there is against the null hypothesis in favor of the alternative hypothesis. In order to calculate a p-value for this test-statistic, we need to know the distribution of U when the null hypothesis is true. To this end, we adapt the bootstrap method of Algorithm 5, leading to Algorithm 6.

Algorithm 6. Hypothesis test of no coancestry among antecedent populations.

To evaluate the validity of the proposed test, we performed this hypothesis testing on various simulation designs (Appendices H and I, S1 Text). Our simulations show that the test produces valid p-values, which are conservative (Fig D in Appendix M, S1 Text), meaning the test has a maximum type I error rate less than or equal to the nominal level of the test. On real data sets analyzed below, these p-values are small, so the conservative behavior that we observe in simulations does not appear to be relevant for populations with nontrivial levels of structure.

We processed each data set and performed quality control (Appendix N, S1 Text). Algorithm 2 was then applied to estimate antecedent population coancestry under the super admixture model (Λ^sup) and Algorithm B under the standard admixture model (Λ^std), along with their corresponding individual-level coancestry matrices (Θ^sup and Θ^std). As a part of these algorithms, the number of antecedent populations K was determined using the structural Hardy–Weinberg method [12] (Appendix S, S1 Text), yielding values from K = 11 for HO to K = 3 for AMR, consistent with earlier studies [10,12,29]. Admixture proportion matrices Q^ were estimated with ALStructure [10].

To assess accuracy, we compared Θ^sup and Θ^std to the Ochoa–Storey (OS) estimate Θ^OS [4], a consistent estimator under general population structures that makes no assumptions about IAF or coancestry distributions. As such, Θ^OS provides a natural benchmark (Appendix C, S1 Text), allowing us to observe if the super admixture or standard admixture models lose information about individual-level coancestry relative to OS. Table A in S1 Text shows that Frobenius distances between Θ^sup and Θ^OS are about 10–40 times smaller than those between Θ^std and Θ^OS. The difference between Θ^sup and Θ^OS is arguably practically irrelevant, meaning that Θ^sup achieves the resolution of Θ^OS for practical purposes.

We further tested the super admixture model against the standard admixture model using Algorithm 6 with B = 1000 bootstrap iterations. In all five data sets, none of the bootstrap null statistics exceeded the observed statistic, giving p < 0.001 (Fig I, S1 Text). To generate bootstrap replicates X*, we applied Algorithm 5 to each observed X, using both the double-admixture method (Algorithm 4) and the NORTA method (Algorithm E, S1 Text). For each replicate, we computed the OS estimate Θ^OS* of individual-level coancestry.

We visualized the results in two complementary ways. First, we constructed heatmaps of individual-level coancestry estimates Θ^OS, Θ^sup, and Θ^std, along with bootstrap-based estimates Θ^OS* obtained using both the double-admixture and NORTA methods. Across all data sets, Θ^OS and Θ^sup were nearly indistinguishable, consistent with their close agreement (Table A, S1 Text), while Θ^std differed substantially, indicating the standard admixture model is not sufficient for these data sets.

Second, we extended the standard stacked bar plots of estimated admixture proportions Q^ by incorporating the covariance structure among the antecedent populations, as captured by the super admixture coancestry matrix Λ^sup. This matrix encodes additional information beyond Q^, revealing how populations themselves are related. To visualize it, we displayed both a heatmap of Λ^sup and a dendrogram from a distance matrix derived from Λ^sup (Appendix G, S1 Text; see also [30]). The dendrogram, built from the standard agglomerative clustering algorithm, provides an intuitive summary of relationships among populations, but it would only reflect true phylogeny if the assumptions of the clustering algorithm are satisfied, which should be evaluated on a case-by-case basis. Taken together, these plots connect the relationships among populations with the admixture proportions of individuals in a way that the standard model cannot.

The Human Origins data sets (HO) consists of 2124 individuals from 170 sub-subpopulations grouped into 11 subpopulations. We observed the estimated individual-level coancestry agrees with current knowledge of early human migrations [31–34]. In Fig 2, we observed the first major split between Sub-Saharan Africa and North Africa. This split reflects the divergence between Sub-Saharan Africans and the rest of human populations resulting from an out-of-Africa migration around 50-60 kya. Another split occurred between South Asia and East Asia, revealing the separation between West Eurasians and East Asians around 40-45 kya. Among the East Asia clade, we identified that the Oceanians have highest coancestry within and lowest coancestry between other subpopulations, consistent with the theory that Oceanians split earliest from the rest of East Asians.

The coancestry among antecedent populations is also compatible with early human dispersals (Fig 3). Specifically, in the dendrogram plot of the antecedent population coancestry (Fig 3B), we noted that the first branch split individuals from Sub-Saharan Africa represented by the antecedent populations S₁ and S₂ from individuals outside of Sub-Saharan Africa represented by the other antecedent populations. Individuals outside of Sub-Saharan Africa further branched off into two lineages: the West Eurasians represented by antecedent populations S₃, S₄ and S₅, and the East Asians represented by antecedent populations S₆ - S₁₁. Then the Oceanians represented by the antecedent population S₉ split off from the majority of East Asian ancestry, while the latter further diverged into present-day Asians (antecedent populations S₆, S₇, S₈) and present-day Americans (antecedent populations S₁₀ and S₁₁).

The AMR subset of TGP has 353 individuals from four regions (Mexican-American (MXL): 65, Puerto Rican (PUR): 104, Colombian (CLM): 97, Peruvian (PEL): 87). The individual-level coancestry plot (Fig 4) revealed that this data set does not have a discrete population structure. Instead, the coancestry changes smoothly over individuals, indicating wide-ranging historical admixture events. This is consistent with the AMR population descending from European, Native American, and Sub-Saharan African ancestries during the post-Columbian era [35,36].

In the analysis of the coancestry among antecedent populations (Fig 5), we identified three major sources of ancestry: Sub-Saharan African ancestry represented by the antecedent population S₁, West Eurasian ancestry represented by the antecedent population S₂, and Native American ancestry represented by the antecedent population S₃. The first split occurred between Sub-Saharan Africans (S₁) and individuals outside of Sub-Saharan Africa (S₂ and S₃), and the second split between the West Eurasians (S₂) and the Native Americans (S₃). We also noted that the Puerto Ricans contain the highest amount of Sub-Saharan African ancestry; the Peruvians have the highest proportion of Native American ancestry; the Colombians and the Mexican-Americans display extensive variation in in their admixture proportions of European and Native American ancestry. Our observations were confirmed by previous analyses of AMR populations [29,35,36].

We combined the mainland Indians from the IND study with the Central/South Asia and the East Asia populations from the HGDP to study the relationship between present-day Indians and other populations in Asia. Our merged data set consists of 298 mainland Indians from four linguistic groups (Indo-European (IE): 92, Dravidian: 53, Austro-Asiatic (AA): 79, Tibeto-Burman (TB): 74), together with 190 Central/South Asians and 210 East Asians from HGDP. Previous analyses of South Asian populations have shown that the Indo-European speakers show a considerable amount of the Western Eurasian relatedness and are ancestrally close to Central Asians. The Austro-Asiatic speakers and the Tibeto-Burman speakers were mixed from East Asian ancestry. The Tibeto-Burman speakers generally have significant genomic proportions derived from East Asian ancestry so that some Tibeto-Burman speakers can be difficult to distinguish from East Asian populations based on genome-wide measures of relatedness. Consistent with these findings [24,37,38], we observe a split between Indo-European speakers and the rest of mainland Indians in the heatmap of individual-level coancestry (Fig 6). The Indo-European speakers and the Central/South Asians of HGDP have relatively similar levels of coancestry. The second split occurred between the Austro-Asiatic speakers and the Tibeto-Burman speakers. The Tibeto-Burman speakers and East Asians of HGDP have relatively similar levels of coancestry.

Our analysis reveals that there are three major branches of antecedent populations for this data set (Fig 7). The branch of antecedent populations S₁ and S₂ is most prevalent in Central/South Asians of HGDP and Indo-European speakers, suggesting this branch was at least partially derived from a West Eurasian source. The branch of the antecedent populations S₃, S₄ and S₅ is widespread in Dravidian speakers and Austro-Asiatic speakers, indicating it is relevant to South Indian ancestry and Austro-Asiatic speaker ancestry. The third branch of the antecedent populations S₆ and S₇ likely represents East Asian ancestry due to its high prevalence in the Tibeto-Burman speakers and East Asians of HGDP.