XI Conference – European Congress of Methodology

Intervention programs evaluation: effect size, moderator variables and methodological quality

Not scheduled

1h 30m

Symposium Design/Research methods Symposium : "Intervention programs evaluation: effect size, moderator variables and methodological quality"

Abstract

Intervention programs evaluation across various fields (sports, organizations, health, etc.) often suffer from methodological shortcomings. These programs may not be implemented with sufficient rigor, and the chosen solutions may not always be the most adequate. This lack of methodological quality hinders the accumulation of reliable knowledge, thereby impeding scientific progress and transference to intervention contexts. This symposium introduces part of our research group studies ('Methodological Innovations in Program Evaluation' - HUM-649, Junta de Andalucía) alongside other European and American research groups. We present six oral presentations to highlight our advances in addressing these challenges. The first work shows validity evidence of a scale to measure methodological quality in observational studies (with low level of intervention). The second presentation is focused in Likert scales data, applied in the evaluation of intervention programs extensively. It compares Robust Weighted Least Squares (RWLS) and Unweighted Least Squares (ULS) estimations in terms of Type I error, power, and accurate fit index values in Confirmatory Factor Analysis. The third study is a meta-analysis that presents the effectiveness of training programs in mental health professionals and the influence of significant moderator variables. The fourth work is an overview about the risk of bias of the primary studies included in meta-analyses in clinical psychology. The fifth presentation is the evaluation of a psychoeducational intervention to improve the quality of life and well-being of patients with myositis. Finally, the last presentation is the validation of a scale to measure hospital anxiety and depression in patients with chronic kidney disease in Chile.

Funding: This work was supported by the research project PID2020-115486GB-I00 funded by the Ministerio de Ciencia, Innovación y Universidades, MICIU/AEI/10.13039/501100011033, Government of Spain; and the Chilean government project FONDECYT Regular 1250316 funded by the National Fund for Scientific and Technological Development, ANID.

Communication 6

Validity evidence of the Hospital Anxiety and Depression Scale (HADS) in Chilean patients with chronic kidney disease

Communication 5

Effectiveness of psychoeducation on myositis: Quality of life and well-being

Abstract

Introduction: designs based on observational methodology allow the systematic recording and subsequent quantification of the spontaneous behavior displayed by participants in natural contexts. These research methods are frequently used in psychology, as well as in the social, educational and health fields due to their multiple advantages, such as a low level of intervention, independence with respect to standardized measurement instruments or their flexibility when applied in non-standardized intervention contexts. A Methodological Quality Scale for Studies Based on Observational Methodology (MQSOM), a tool to measure the methodological quality of these studies, has recently been validated with adequate psychometric properties (RMSEA = 0.000, NNFI = 1, GFI = .98, AGFI = .97). The MQSOM comprises a second-order factor of Methodological quality (ω = .87; D = .55) containing two first-order factors: Quality of design (6 items; ω = .90; D = .46; ICC = .933 - .967) and Quality of measurement and analysis (5 items; ω = .68; D = .67; ICC = .797 - .988). Objective: the aim of this study is to present the evidence of convergent and discriminant validity of MQSOM. Methods: a multitrait-multimethod analysis (MTMM) with Spearman correlations was carried out to examine the relationship between the dimensions of MQSOM and those of the methodological quality instruments Rigorous Mixed-Methods (RMM), Guidelines for Publishing Evaluations Based on Observational Methodology (GREOM) and Mixed Methods Appraisal Tool (MMAT), circumscribed to the field of Mixed-Methods studies. Ninety-six articles based on observational methodology were coded with MQSOM and each of the contrast instruments. Results: adequate levels of inter- and intra-coder reliability were obtained (ICC between .73 and 1). MQSOM dimension of Design showed empirical evidence of convergence with MRMM (ρ between .22 and .47), GREOM (ρ between .22 and .34) and MMAT (ρ = .21). It also showed empirical evidence of discriminant validity with the contrast instruments (ρ between -.05 and .03 regarding MRMM; ρ between -.03 and .03 regarding GREOM; ρ = -.04 regarding MMAT). MQSOM dimension of Measurement and Analysis showed empirical evidence of convergence with MRMM (ρ between .21 and .61), GREOM (ρ between .22 and .61), and MMAT (ρ between .21 and .64). Conclusions: these results support the use of MQSOM, a brief instrument that addresses methodological quality in observational methodology in a diagnostic way, measuring the quality of design, measurement and analysis of results in studies based on observational methodology, but also in a prescriptive way, serving as a reference for applied researchers, editorial boards and other decision-making committees.

Funding: This work was supported by the research project PID2020-115486GB-I00 funded by the Ministerio de Ciencia, Innovación y Universidades, MICIU/
AEI/10.13039/501100011033, Government of Spain; and the Chilean government project FONDECYT Regular 1250316 funded by the National Fund for Scientific and Technological Development, ANID.

Abstract

The use of Likert scales in the field of social research is becoming more and more common every day, it is necessary to investigate which is the most appropriate methodology to carry out the analysis of the data obtained. If they are ordinal, they should be treated as such, however, they are frequently analyzed considering them as continuous variables. One of the most widely used techniques to obtain construct validity evidence through internal structure of the nomological models, is Confirmatory Factor Analysis. Using simulation studies in which four factors have been manipulated (number of factors, number of items response categories, skewness and sample size) our objective is twofold: firstly, when ordinal variables are used, analyze the type I error and power of the most common fit indices, such as RMSEA and SRMR obtained using ULS and RULS estimation methods; and secondly, using Receiver Operating Characteristic Curve (ROC) review the cut-off points of RMSEA and SRMR. It is found that, depending on the estimation method chosen, the type I error and power differ, as well as the values reported by RMSEA and SRMR. RULS seems to obtain better results regardless of experimental factors manipulated. Finally, it is found that it would be convenient to review the cut-off points for these global fit indices recommended by the literature.

Abstract

Background: Chronic kidney disease (CKD) is a global health issue that significantly impacts patients' quality of life due to physical and emotional symptoms. Anxiety and depression are common in these patients, negatively affecting their prognosis and treatment adherence. The Hospital Anxiety and Depression Scale (HADS) is a popular tool for assessing these disorders, but it has not been validated in Chilean renal patients. Methods: In a sample of 442 CKD patients from hospital centers in Chile, the factor structure, internal consistency, and concurrent validity of the HADS were evaluated using confirmatory factor analysis, Cronbach's alpha, McDonald's omega, and correlations with the Depression Anxiety Stress Scale (DASS-21), respectively. Results: Analyses showed a good fit for the two correlated factors model, with anxiety and depression subscales demonstrating high internal consistency. Significant correlations between HADS and DASS-21 confirmed concurrent validity. Conclusions: These findings suggest that the HADS is a valid and reliable tool for assessing anxiety and depression in Chilean CKD patients, facilitating timely psychological interventions and improving patients' quality of life. Future studies should include more diverse samples and assess the temporal stability of the scales to confirm these findings.

Funding: This work was supported by the research project PID2020-115486GB-I00 funded by the Ministerio de Ciencia, Innovación y Universidades, MICIU/AEI/10.13039/501100011033, Government of Spain; the Chilean government project FONDECYT Regular 1250316 funded by the National Fund for Scientific and Technological Development, ANID; and Fondecyt Iniciación 11200099, Government of Chile.

Abstract

This study aims to analyze the effectiveness of training programs designed for mental health professionals. The analysis focuses on randomized controlled trials (RCTs) and cluster-randomized studies, examining the impact of these interventions across three levels of outcomes (based on Kirkpatrick & Kirkpatricks’ model): knowledge acquisition, attitude changes, and behavioral modifications. The study includes 18 eligible studies, each meeting rigorous inclusion criteria, and evaluates the moderating effects of methodological quality, study type, and intervention duration. Methodological quality was assessed using the 10-item Methodological Quality Scale, providing a standardized measure to gauge the robustness of the included studies. The analysis further investigates the differential effects of research design studies (RCTs versus clusters) and intervention and measurement times. Three distinct meta-analyses were conducted to integrate the outcomes across the selected levels. Preliminary findings suggest a positive overall effect size, with decreasing magnitude observed as the analysis progresses from knowledge to attitudes and, ultimately, to behaviors. These results align with the hypothesis of diminishing returns through the hierarchical pyramid of training impact. This work underscores the critical importance of methodological rigor and contextual factors in determining the efficacy of training programs in mental health services. Insights from this analysis provide actionable evidence to enhance future program design, implementation, and evaluation.

Funding: This work was supported by the research project PID2020-115486GB-I00 funded by the Ministerio de Ciencia, Innovación y Universidades, MICIU/AEI/10.13039/501100011033, Government of Spain; and the Chilean government project FONDECYT Regular 1250316 funded by the National Fund for Scientific and Technological Development, ANID.

Abstract

One of the biggest limitations of meta-analyses is that the information they provide can be affected by the biases of the included primary studies. To address this, evaluations of primary study risk of bias (RoB) can be performed and incorporated into the meta-analysis. However, research on this topic in clinical psychology is scarce. In this study, we examined this issue using a sample of clinical psychology meta-analyses that included RoB assessments. First, we evaluated meta-analysts’ assessment practices. Second, we summarized the RoB ratings of the primary studies included in the meta-analyses. Lastly, we examined the relationship between RoB ratings and effect sizes. We found some suboptimal practices in the assessment procedures, such as only half of the studies reporting that the assessment was conducted in duplicate. Regarding RoB ratings, the domains with the highest ratings were random sequence generation, blinding of outcome assessment, and incomplete outcome data, with about half of the primary studies rated as low RoB. The lowest ratings were found for allocation concealment and, especially, blinding of participants and personnel. Importantly, we found a positive association between the publication year of the primary studies and a lower RoB in most domains. Lastly, performing our own re-analysis, we found an association between RoB and effect sizes, which contrasts with the results of the analyses reported in the meta-analyses that combined those studies. We recommend caution when interpreting a lack of modulation of effect sizes in meta-analyses, as they may not have sufficient statistical power for moderator analyses.

Funding: MICIU/AEI /10.13039/501100011033/ and FEDER funds, European Union, grant no. PID2022-137328NB-I00