A comparison of three methods for estimating average levels of gene flow

Three methods for estimating the average level of gene flow in natural population are discussed and compared. The three methods are FST, rare alleles, and maximum likelihood. All three methods yield estimates of the combination of parameters (the number of migrants [Nm] in a demic model or the neighborhood size [4πDσ2] in a continuum model) that determines the relative importance of gene flow and genetic drift. We review the theory underlying these methods and derive new analytic results for the expectation of FST in stepping-stone and continuum models when small sets of samples are taken. We also compare the effectiveness of the different methods using a variety of simulated data. We found that the FST and rare-alleles methods yield comparable estimates under a wide variety of conditions when the population being sampled is demographically stable. They are roughly equally sensitive to selection and to variation in population structure, and they approach their equilibrium values at approximately the same rate. We found that two different maximum-likelihood methods tend to yield biased estimates when relatively small numbers of locations are sampled but more accurate estimates when larger numbers are sampled. Our conclusion is that, although FST and rare-alleles methods are expected to be equally effective in analyzing ideal data, practical problems in estimating the frequencies of rare alleles in electrophoretic studies suggest that FST is likely to be more useful under realistic conditions.