XI Conference – European Congress of Methodology

Methodological Advances in Meta-analysis

Not scheduled

1h 30m

EAM2025

Symposium Statistical analyses Symposium : "Methodological Advances in Meta-analysis"

Speaker

Julio Sánchez-Meca (University of Murcia (Spain))

Abstract

In meta-analysis, the Q statistic is traditionally used for testing the hypothesis of homogeneity of the parametric effect sizes of the set of studies. Several critiques have been posed to that test, especially when applied to the standardized mean difference (g). Among them, that the weights are based on estimated, not true, variances, that the variances of the estimates correlate with the own g values, and that it is assumed a wrong distribution of g (normal) although it is actually a linear transformation of a Student’s t. We present an improved test of homogeneity of g values based in the Mixture Model of Suero et al (in press) in which most of the problems highlighted are solved or greatly reduced. Specifically, the variances of g in the studies are independent of the own g values, and the true distribution of g is acknowledged and transformed to a normal distribution. Although the variances are still estimated, we show that their impact in the performance of the test is negligible under the Mixture Model. We present the results of an extensive Monte Carlo simulation to assess the performance of the classical Q test, an alternative that use weights based only on the samples sizes (effective sample size, Qñ) and two normalizing transformations (those of Johnson-Welch and Laubscher). The results show that whereas the classical Q test yields a too low rate of type I errors and Qñ an unacceptable large rate, the two normalizing transformations yield rates within a comfortable 4% – 6 % range. Furthermore, the rate of correct rejections (estimated power) is always higher than that of the Q test. Taking all the results together, we conclude recommending the new test for meta-analysis of g values, using the estimates provided by the Mixture Model and the normalizing transformation of Johnson-Welch.

Communication 6

Reliability generalization of the Emotional Quotient Inventory Youth Version (EQ-i: YV): A meta-analytic structural equation modelling approach

Abstract

The findings of a collection of studies addressing a common research question can be visualized in terms of a forest plot, showing the effect sizes of the individual studies together with a corresponding confidence interval. A four-sided polygon (sometimes called a summary ‘diamond’) is often added to such a plot to depict the results from a meta-analysis pooling together the effect sizes, where the center of the polygon corresponds to the pooled estimate and the ends of the polygon represent the bounds of the confidence interval for the pooled estimate. However, this only communicates the size of the average effect and how precisely it is estimated. In addition, it is equally important to indicate the degree of heterogeneity among the findings, that is, the variability in the underlying true effects. Such information (e.g., the results from the Q-test, the I2 statistic, and the estimate of τ2 from a random-effects model) is often only added textually underneath the plot. In this talk, I will describe several alternative visualizations of the amount of heterogeneity in terms of the prediction interval and by showing the entire prediction distribution. This also raises interesting issues when applying a back-transformation of the results (such as exponentiation when meta-analyzing log-transformed estimates or the hyperbolic tangent function when meta-analyzing Fisher r-to-z transformed correlation coefficients), since this impacts not only the shape of the prediction distribution, but also what the back-transformed estimates represent. The various types of visualizations discussed are already implemented in the metafor package for R and can be readily used by practitioners.

Abstract

Recent research has identified several limitations in traditional methods for conducting meta-analyses of reliability generalization, such as the lack of equivalence between total and subscale reliability indices and the violation of error independence assumptions. In response, multivariate statistical techniques have been developed to offer more accurate estimations of measurement instruments, one of which is meta-analysis of structural equation modelling (MASEM). MASEM offers significant advantages, including the ability to combine correlation matrices from primary studies and to estimate factor models more efficiently. This communication demonstrates the application of MASEM to the Emotional Quotient Inventory Youth Version (EQ-i:YV), a widely used tool for assessing emotional intelligence in children and adolescents worldwide. By employing a MASEM approach, we will derive more robust reliability estimates using Omega values, which are more suitable for multidimensional measures compared to traditional Cronbach’s alpha. Ultimately, we aim to enhance our understanding of the psychometric properties of the EQ-i:YV and contribute to advancing theoretical development in the field of emotional intelligence assessment.
Funding: MICIU/AEI /10.13039/501100011033/ and FEDER funds, European Union, grant no. PID2022-137328NB-I00

Abstract

Meta-analysis is a research methodology designed to statistically integrate effect sizes obtained from a set of empirical studies on a common topic. Although meta-analysis is now a well-established methodology, there is still significant room for improvement. This symposium features six presentations that address various methodological and statistical challenges arising from the application of meta-analysis. In the first presentation, Wolfgang Viechtbauer (Maastricht University) discusses strategies for visualizing heterogeneity among effect sizes using enhanced forest plots. In the second presentation, José A. López-López (University of Murcia) explores different standardization approaches for calculating standardized mean changes in pretest-posttest one-group designs and presents the results of an empirical study on the assumptions underlying these effect size estimators. In the third presentation, Manuel J. Albaladejo shares the results of a Monte Carlo simulation assessing the statistical properties of various standardized mean change estimators when certain assumptions are violated. The fourth presentation, delivered by Ignacio Durán (Complutense University), reports on the results of a Monte Carlo simulation evaluating the performance of an improved heterogeneity test based on a mixture model within a random-effects framework. In the fifth presentation, Rubén López-Nicolás (University of Castilla-La Mancha) provides an empirical evaluation of the statistical power of random-effects meta-analyses, as typically applied in clinical psychological interventions. Finally, in the sixth presentation, Alejandro Veas (University of Murcia) conducts a reliability generalization meta-analysis of the Emotional Quotient Inventory Youth Version, employing Meta-Analytic Structural Equation Modeling (MASEM).
Funding: MICIU/AEI /10.13039/501100011033/ and FEDER funds, European Union, grant no. PID2022-137328NB-I00

Abstract

Effect sizes are commonly used in meta-analysis, as they provide a tool to summarize the results from each primary study in a common metric. In psychology and related fields, meta-analyses often involve integrating continuous variables measured with different scales across studies, which leads to using standardized mean differences as the effect size index. One of these indices is the standardized mean change (SMC), which quantifies within-group treatment effects when the primary studies have examined the effectiveness of a treatment program using a repeated measures design, and the dependent variable has been measured on a quantitative scale. However, different procedures have been proposed to calculate this effect size, and some of them make assumptions which are hardly verifiable in practice, namely homoscedasticity and a specific value for the correlation between measurement points. This presentation will explore the potential impact on the results if some of these assumptions are violated, using a range of simulated scenarios.
Funding: MICIU/AEI /10.13039/501100011033/ and FEDER funds, European Union, grant no. PID2022-137328NB-I00

Abstract

The standardized mean change is widely recognized as a key effect size index in pretest-posttest one-group designs with quantitative dependent variables. Different parametric versions of this index are available, depending on the standardizer used to scale the mean difference into standardized units. In addition, various estimators can be applied to each parameter. This study used a Monte Carlo simulation to assess the bias and mean squared error of several estimators for the standardized mean change. Key factors, such as population effect size, population correlation, sample size, and heterogeneity of pretest and posttest population variances, were systematically manipulated. The results offer valuable insights for selecting the most efficient estimator, taking into account the chosen parameter, study characteristics, and the potential of integrating effect sizes into meta-analyses.
Funding: MICIU/AEI /10.13039/501100011033/ and FEDER funds, European Union, grant no. PID2022-137328NB-I00

Communication 5

Statistical power of random-effects meta-analyses of clinical psychological interventions