Skip to main content


Higher-order asymptotics for random effects meta-analysis: An empirical evaluation

Date and Location




Sunday 22 September 2013 - 13:30 - 15:00


Presenting author and contact person

Presenting author

Joseph Beyene

Contact person

Joseph Beyene
Abstract text
Background: A random-effects model is often used to dealing with unexplained heterogeneity in meta-analysis. However, typical applications use a small number of studies suggesting that statistical inference based on first-order asymptotic theory may lead to inaccurate results. Objectives: To empirically evaluate performance of higher-order asymptotic methods in random effects meta-analysis. Methods: We applied higher-order asymptotic methods to a series of meta-analyses using data extracted from the Cochrane Library. We focused on continuous outcomes and compared traditional methods with a second-order likelihood method based on Skovgaard’s statistic. We have also used a signed-likelihood ratio test approach. Three effect measures (mean difference (MD), standardized mean difference (SMD), Ratio of Means (RoM)), and three methods of estimation for the heterogeneity parameter (DerSimonian-Laird (DL), Maximum Likelihood (ML), and Restricted Maximum Likelihood (REML)) were investigated. We included meta-analyses for which the heterogeneity parameter is significantly different from zero. Results: 66 meta-analyses were used in which the effect measure was MD. The largest average discrepancy in p-values was between Skovgaard’s method and a p-value based Z test based on ML estimator (mean(SD) of difference = 0.05 (0.09)). For SMD and ROM, 106 meta-analyses were available for analysis. Once again the largest discrepancy for SMD occurred between Skovgaard’s and Z test with ML (mean (SD) of difference = 0.03 (0.04)). The results for ROM were similar to that of the SMD. In one example with 14 studies, the p-values for effect of intervention using a signed log likelihood ratio test and the 2nd order Skovgaard’s statistic were 0.2526, and 0.635, respectively, while Z test with ML and REML resulted in p-values of 0.0547 and 0.0653, respectively. Conclusions: Traditional first-order asymptotic methods in random-effects model might not be reliable when the number of studies is small. Higher-order asymptotics methods are available and need to be assessed in parallel.