The number of medical studies being published is increasing exponentially, and clinicians must routinely process large amounts of new information. Moreover, the results of individual studies are often insufficient to provide confident answers, as their results are not consistently reproducible. A meta-analysis is a statistical method for combining the results of different studies on the same topic and it may resolve conflicts among studies. Meta-analysis is being used increasingly and plays an important role in medical research. This review introduces the basic concepts, steps, advantages, and caveats of meta-analysis, to help clinicians understand it in clinical practice and research. A major advantage of a meta-analysis is that it produces a precise estimate of the effect size, with considerably increased statistical power, which is important when the power of the primary study is limited because of a small sample size. A meta-analysis may yield conclusive results when individual studies are inconclusive. Furthermore, meta-analyses investigate the source of variation and different effects among subgroups. In summary, a meta-analysis is an objective, quantitative method that provides less biased estimates on a specific topic. Understanding how to conduct a meta-analysis aids clinicians in the process of making clinical decisions.
New medical studies are being published continuously and clinicians are faced with increasingly large amounts of new information, to the point where it has become nearly impossible for clinicians to read and evaluate all available data in a medical field. In addition, the study results are not consistently reproducible and the results of individual studies are often insufficient to provide confident answers [1]. Many studies have false-positive results (type I errors) or are unable to detect small effects (false-negative, type II errors) [2]. Consequently, clinical decision making is particularly difficult when the published data are conflicting or the sample size is too small to be reliable [3].
Evidence-based medicine is the best available evidence in the medical literature [4]. Moreover, the best evidence in evidence-based medicine is from meta-analyses, which provide a less biased, more precise estimate on a clinical issue [5]. “Meta” comes from the Greek for “after” or “beyond”; a meta-analysis is an “analysis of analyses” [6]. In other words, it is a statistical technique for combining the results from different studies on the same topic [7], and is becoming popular for resolving discrepancies in clinical research. As such, a meta-analysis is an objective, quantitative synthesis of research findings [7] that increases the statistical strength and precision for estimating effects by combining the results of previous studies and, thus, overcoming the problem of small sample sizes and inadequate statistical strength [8]. A meta-analysis can explore the sources of heterogeneity, and identify subgroups associated with the factor of interest, potentially providing new insights for future studies [9]. When conducted properly, a meta-analysis of medical studies is considered decisive evidence, as it occupies a top level in the hierarchy of evidence [5]. Therefore, meta-analysis is a more efficient and effective standard method for summarizing the results of many studies than is subjective judgment; therefore, it has become an important research strategy, and is progressively expanding. For example, a PubMed search of the title word “meta-analysis” produced 120,537 possible articles.
It is necessary to understand the statistical principles of meta-analysis in evidence-based medical practice, and it is important that clinicians understand its methods, advantages, and limitations. Therefore, this review introduces the basic concepts, steps, advantages, and caveats of meta-analysis, to help clinicians understand meta-analysis in clinical practice and research.
HOW TO PERFORM A META-ANALYSIS
Table 1 summarizes the general process of meta-analysis in medical studies. General or specialized statistical programs, such as STATA, SAS, R, Review Manager, or Comprehensive Meta-Analysis, are used to perform the statistical analysis of a meta-analysis.
Identification of relevant studies
The identification of all relevant papers is a critical component in a meta-analysis because its outcome depends on the studies included [10]. The search strategy has to be comprehensive, and more than one database should be searched. The three electronic databases that are considered to be the most important sources of medical studies are commonly searched, i.e., PubMed, EMBASE, and The Cochrane Central Register of Controlled Trials (CENTRAL) [11].
Heterogeneity
A meta-analysis examines the existence of heterogeneity among primary studies and analyzes the variance in the results of different studies; meta-analysis heterogeneity is the degree of dissimilarity in the individual study results [7]. The heterogeneity test examines the null hypothesis, i.e., there are no differences in the findings of the primary studies. Statistical tests, such as Cochran’s Q test and the I2 value, have been developed to detect and quantify heterogeneity in meta-analysis. Specifically, Cochran’s Q test is used to determine whether there are differences between primary studies or if the variation seen is due to chance [12]. Cochran’s Q-value is calculated by summing the squared deviations of the estimate of each study from the overall estimate and subsequently comparing it with the chi-square distribution with κ–1 degrees of freedom (df), where κ is the number of studies [12]. However, the Q test may be unreliable when the meta-analysis involves a small number of studies. Therefore, a heterogeneity p < 0.10 (not 0.05) indicates the presence of heterogeneity, since Cochran’s Q test has low statistical strength and is insensitive [13]. Another commonly used method for testing heterogeneity is the I2 value, which quantifies the effect of heterogeneity, and does not depend on the number of studies or the type of outcome data. I2 values range between 0% and 100%, and represent the proportion of inter-study variability that can be attributed to heterogeneity rather than chance [I2 = 100% × (Q – df)/Q] [14]. I2 values of 25%, 50%, and 75% are considered low, moderate, and high estimates, respectively [14,15].
Fixed versus random effects models
A meta-analysis combines the effect sizes of the included studies by weighting the data according to the different amounts of information in each study. The weights are calculated using the sample size or the variance of each study [8]. There are two statistical models for a meta-analysis: the fixed effect and random effect models. The fixed effect model assumes that all of the studies in the meta-analysis have one true effect size, and the observed variation among studies is caused by sampling errors or chance [16]. The random effect model assumes that different studies exhibit substantial diversity, and the true effect size may vary from study to study [17]. Consequently, the fixed effect model assesses only intra-study sampling errors (intra-study variation), while the random effect model assesses both intra-study sampling errors and inter-study variance (between-study variation) [18]. As such, the choice of meta-analysis model depends on the presence or absence of heterogeneity. In the absence of heterogeneity (heterogeneity p ≥ 0.10), a fixed effect model is used. However, when the Q-value is significant (p < 0.10), indicating the existence of heterogeneity in the studies, a random effect model should be used for the meta-analysis [19]. When the study groups are homogeneous, both models offer similar results; nonetheless, in the case of heterogeneity, the random effect model typically provides wider confidence intervals (CIs) than the fixed effect one [20].
Evaluation of the causes of heterogeneity
It is important to assess the presence of heterogeneity among the studies included in a meta-analysis, and determine the possible causes of heterogeneity, since it can lead to bias, referred to as “mixing apples and oranges,” in the meta-analysis results [21]. As a result, subgroup analysis is used to assess the impact of heterogeneity, using factors such as ethnicity, number of studies, or clinical features to assess the impact of a potential source of heterogeneity. Sensitivity testing may also be performed to assess the influence of each individual study on the pooled effect size by omitting each individual study [22]. Meta-regression is used to explore the reasons for the heterogeneity and adjust for confounding effects, and is feasible if sufficient data are reported in the individual studies [23].
Publication bias
Studies showing positive effects tend to be published more frequently than those that do not, and studies showing no significant results tend to remain unpublished [24]. As a meta-analysis includes only published studies, it might overestimate the actual effect degree [24]. This outcome is called “publication bias.” In other words, a meta-analysis may be subject to publication bias. The funnel plot is a commonly used graphic test to assess publication bias in a meta-analysis [25]. This test is a scatterplot of the effect estimate from each study in the meta-analysis against the measure of its precision (1/standard error) or sample size [26]. The effect estimates of small studies will scatter at the bottom of the graph, while the spread of larger studies will be narrower. In the absence of publication bias, the funnel plot produces a symmetrical inverted funnel, asymmetry being suggestive of a publication bias [27]. The funnel plot is a simple method, but it is difficult to interpret when the number of studies is small and can be misleading. Due to the limitations of funnel plots, which require a range of studies of varying sizes, involving subjective judgments, publication bias can be evaluated using other methods, such as Egger’s linear regression test [25], which measures funnel plot asymmetry using a natural logarithm scale of odds ratios. Egger’s regression test examines whether the intercept deviates significantly from zero in a regression of the standardized effect estimates against their precision. When asymmetry is present, the “trim and fill” method adjusts the summary estimates for observed bias [28]. This method removes or adds small studies until funnel plot symmetry is achieved by recalculating the center of the funnel before removing studies and replacing them with their missing mirrorimage counterparts. A revised summary estimate is subsequently calculated using all of the original studies and the hypothetical “filled” studies [28].
Meta-analysis publication
Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) is an evidence-based set of items developed for reporting systematic reviews and meta-analyses [29]. Following the PRISMA recommendations helps authors to improve meta-analysis reporting.
STRENGTHS AND WEAKNESSES OF META-ANALYSIS
A major advantage of a meta-analysis is that it produces a precise estimate of the effect size with considerably increased statistical power, which is especially important when the power of the primary study is limited because of the small sample size. A meta-analysis also analyzes the variation in the results of different studies and quantifies result inconsistency (heterogeneity) across studies. It is also an objective, quantitative method that provides a less biased estimate on a specific topic. The main criticism of meta-analysis is that it combines different types of studies (“mixing apples and oranges”) [7]. Nonetheless, this problem can be overcome by assessing the heterogeneity in the studies and performing subgroup analysis [30]. However, if studies are too heterogeneous to be comparable, a meta-analysis should be avoided, as the meta-analysis result may be meaningless and any true effect may be obscured. The other limitation of meta-analysis is “garbage in, garbage out,” which means that if a meta-analysis includes low-quality studies with bias, the results of the meta-analysis will be biased and incorrect [31]. As such, the results of the meta-analysis depend on the quality of the primary research. Therefore, any meta-analysis should include studies selected based on strict inclusion criteria.
EXAMPLE OF A META-ANALYTIC STUDY
Tofacitinib is a novel oral Janus kinase inhibitor [32]. Several clinical trials have attempted to evaluate the efficacy and safety of tofacitinib in active rheumatoid arthritis (RA). Therefore, a meta-analysis approach to randomized clinical trial (RCT) data was used in an attempt to increase the precision and accuracy of estimates of the efficacy and safety of tofacitinib 5 and 10 mg twice daily in active RA. We performed a systematic review and meta-analysis of RCTs, examining the efficacy and safety of tofacitinib in active RA patients using PubMed, EMBASE, CENTRAL, and manual searches [33]. Five RCTs, including three phase 2 and two phase 3 trials and 1,590 patients, met the inclusion criteria [33]. The three phase 2 RCTs included 452 RA patients in the meta-analysis. The American College of Rheumatology 20% response rate (ACR20) response rate was significantly higher in the tofacitinib 5 mg group than in the controls (relative risk [RR], 2.445; 95% CI, 1.229 to 4.861; p = 0.011) (Table 2, Fig. 1). Similarly, the ACR20 response rate was significantly higher in the tofacitinib 10 mg group than in the controls (RR, 2.597; 95% CI, 1.514 to 4.455; p = 0.001) (Table 2, Fig. 1). Significant improvements were observed in the tofacitinib 5 and 10 mg groups compared with the controls for all efficacy outcomes, such as the number of tender and swollen joints, pain, patient and physician global assessments of disease activity, the Health Assessment Questionnaire, and C-reactive protein levels (Table 2). The safety outcomes did not differ between the tofacitinib 5 and 10 mg groups and placebo groups, except for infection in the tofacitinib 10 mg group (RR, 2.133; 95% CI, 1.268 to 3.590; p = 0.004) (Table 2, Fig. 2). The two phase 3 trials confirmed the findings of the meta-analysis of the phase 2 studies. The meta-analysis found that tofacitinib, at dosages 5 or 10 mg twice daily, was effective in active RA and had a manageable safety profile.
CONCLUSIONS
Meta-analyses in medical research cover a wide range of topics, from risk factors to prognosis [34,35]. Therefore, meta-analysis is applicable to a broad spectrum of topics, including biomarkers, genetic factors, diagnosis, and treatment [33,36-38]. It is also a powerful method for combining the results of different studies, and for summarizing current evidence on a specific issue objectively. Consequently, a meta-analysis provides a more precise estimate of effect sizes and investigates sources of variation and difference effects among subgroups. It can also resolve conflicts between studies, and yield conclusive results when individual studies are inconclusive. Meta-analysis is an invaluable bridge between past and future studies. Therefore, understanding meta-analysis is valuable for clinicians in the process of making clinical decisions.
No potential conflict of interest relevant to this article was reported.
REFERENCESMuradMHMontoriVMIoannidisJPHow to read a systematic review and meta-analysis and apply the results to patient care: users’ guides to the medical literatureJAMA2014312171179ButtonKSIoannidisJPMokryszCPower failure: why small sample size undermines the reliability of neuroscienceNat Rev Neurosci201314365376HaidichABMeta-analysis in medical researchHippokratia201014Suppl 12937SackettDLRosenbergWMGrayJAHaynesRBRichardsonWSEvidence based medicine: what it is and what it isn’t. 1996Clin Orthop Relat Res200745535GuyattGHSackettDLSinclairJCHaywardRCookDJCookRJUsers’ guides to the medical literature. IX. A method for grading health care recommendations. Evidence-Based Medicine Working GroupJAMA199527418001804GlassGVPrimary, secondary, and meta-analysis of researchEduc Res1976538WalkerEHernandezAVKattanMWMeta-analysis: Its strengths and limitationsCleve Clin J Med200875431439EggerMSmithGDPhillipsANMeta-analysis: principles and proceduresBMJ199731515331537GotzschePCWhy we need a broad perspective on meta-analysis. It may be crucially important for patientsBMJ2000321585586LeenaarsMHooijmansCRvan VeggelNA step-by-step guide to systematically identify all relevant animal studiesLab Anim2012462431QiXYangMRenWFind duplicates among the PubMed, EMBASE, and Cochrane Library Databases in systematic reviewPLoS One20138e71838WhiteheadAWhiteheadJA general parametric approach to the meta-analysis of randomized clinical trialsStat Med19911016651677MunafoMRFlintJMeta-analysis of genetic association studiesTrends Genet200420439444HigginsJPThompsonSGQuantifying heterogeneity in a meta-analysisStat Med20022115391558ThakkinstianAMcElduffPD’EsteCDuffyDAttiaJA method for meta-analysis of molecular association studiesStat Med20052412911306Davey SmithGEggerMMeta-analyses of randomised controlled trialsLancet19973501182DerSimonianRKackerRRandom-effects model for meta-analysis of clinical trials: an updateContemp Clin Trials200728105114HedgesLVVeveaJLFixed- and random-effects models in meta-analysisPsychol Methods19983486504RiedKInterpreting and understanding meta-analysis graphs: a practical guideAust Fam Physician200635635638ZintzarasELauJSynthesis of genetic association studies for pertinent gene-disease associations requires appropriate methodological and statistical approachesJ Clin Epidemiol200861634645SalantiGSandersonSHigginsJPObstacles and opportunities in meta-analysis of genetic association studiesGenet Med200571320CopasJShiJQMeta-analysis, funnel plots and sensitivity analysisBiostatistics20001247262ThompsonSGHigginsJPHow should meta-regression analyses be undertaken and interpreted?Stat Med20022115591573DickersinKMinYIPublication bias: the problem that won’t go awayAnn N Y Acad Sci1993703135146EggerMDavey SmithGSchneiderMMinderCBias in meta-analysis detected by a simple, graphical testBMJ1997315629634SterneJAEggerMFunnel plots for detecting bias in meta-analysis: guidelines on choice of axisJ Clin Epidemiol20015410461055LauJIoannidisJPTerrinNSchmidCHOlkinIThe case of the misleading funnel plotBMJ2006333597600DuvalSTweedieRTrim and fill: a simple funnel-plot-based method of testing and adjusting for publication bias in meta-analysisBiometrics200056455463MoherDLiberatiATetzlaffJAltman DG; PRISMAGroupPreferred reporting items for systematic reviews and meta-analyses: the PRISMA statementAnn Intern Med2009151264269Bailar JC3rdThe promise and problems of meta-analysisN Engl J Med1997337559561SharpeDOf apples and oranges, file drawers and garbage: why validity issues in meta-analysis will not go awayClin Psychol Rev199717881901ChangelianPSFlanaganMEBallDJPrevention of organ allograft rejection by a specific Janus kinase 3 inhibitorScience2003302875878SongGGBaeSCLeeYHEfficacy and safety of tofacitinib for active rheumatoid arthritis with an inadequate response to methotrexate or disease-modifying antirheumatic drugs: a meta-analysis of randomized controlled trialsKorean J Intern Med201429656663LeeYHMeta-analysis of genetic association studiesAnn Lab Med201535283287LeeYHBaeSCSongGGGene-environmental interaction between smoking and shared epitope on the development of anti-cyclic citrullinated peptide antibodies in rheumatoid arthritis: a meta-analysisInt J Rheum Dis201417528535SongGBaeSCLeeYHDiagnostic accuracy of lung ultrasound for interstitial lung disease in patients with connective tissue diseases: a meta-analysisClin Exp Rheumatol2016341116SongGGSeoYHKimJHChoiSJJiJDLeeYHAssociation between TNF-α (-308 A/G, -238 A/G, -857 C/T) polymorphisms and responsiveness to TNF-α blockers in spondyloarthropathy, psoriasis and Crohn’s disease: a meta-analysisPharmacogenomics20151614271437LeeYHNathSKSystemic lupus erythematosus susceptibility loci defined by genome scan meta-analysisHum Genet2005118434443Figures and Tables
Meta-analysis of the efficacy of tofacitinib (A) 5 mg and (B) 10 mg twice a day on American College of Rheumatology 20% response rate in rheumatoid arthritis. CI, confidence interval.
Meta-analysis of the efficacy of tofacitinib (A) 5 mg and (B) 10 mg twice a day on number of patients withdrawn due to adverse events in rheumatoid arthritis. CI, confidence interval.
General method of medical studies meta-analysis
1. Identify relevant research: PubMed, EMBASE, CENTRALa
4. Evaluate sources of the heterogeneity: subgroup analysis, sensitivity test, meta-regression
5. Check publication bias: funnel plot, Egger’s regression test, trim and fill method
6. Present meta-analysis result based on PRISMAb
CENTRAL: The Cochrane Central Register of Controlled Trials.
PRISMA: preferred reporting items for systematic reviews and meta-analyses.
Meta-analysis of randomized controlled trials on the efficacy of tofacitinib in active RA
Tofacitinib dosedose
Outcome
No. of Studies
Test of association
Test of heterogeneity
WMD
95% CI
p value
Model
p value
I2
Tofacitinib 5 mg, bid
ACR20
3
2.445a
1.229 to 4.861
0.011
R
0.014
76.7
Tender joint count
3
–5.731
–8.054 to –3.048
1.3 × 10-7
F
0.279
21.6
Swollen joint count
3
–5.422
–9.593 to –1.252
0.011
R
0.008
79.4
Pain (VAS)
3
–12.72
–18.06 to –7.376
3.0 × 10-7
F
0.464
0
Patient global assessment
3
–17.82
–28.20 to –7.444
< 1.0 × 10-8
R
0.017
75.6
Physician global assessment
3
–17.88
–26.48 to –9.286
< 1.0 × 10-8
R
0.067
63.0
HAQ
3
–0.341
–0.455 to –0.226
< 1.0 × 10-8
F
0.526
0
CRP
3
–16.43
–28.09 to –4.778
0.006
R
0.000
87.8
Tofacitinib 10 mg, bid
ACR20
3
2.597a
1.514 to 4.455
0.001
R
0.054
65.8
Tender joint count
3
–6.295
–8.517 to –4.073
2.0 × 10-9
F
0.639
0
Swollen joint count
3
–5.970
–9.630 to –2.311
0.001
R
0.010
78.5
Pain (VAS)
3
–18.20
–29.50 to –8.230
0.002
R
0.020
74.5
Patient global assessment
3
–17.70
–27.17 to –8.230
< 1.0 × 10-8
R
0.044
68.0
Physician global assessment
3
–17.45
–28.82 to –6.082
0.003
R
0.010
78.2
HAQ
3
–0.344
–0.461 to –0.227
< 1.0 × 10-8
F
0.143
48.5
CRP
3
–17.07
–32.15 to –1.999
0.026
R
0.000
92.2
WMD, weighted mean difference; CI, confidence interval; bid, twice daily; ACR20, American College of Rheumatology 20% response rate; R, random effects model; F, fixed effects model; VAS, visual analog scale; HAQ, Health Assessment Questionnaire; CRP, C-reactive protein.