Description
Introduction: designs based on observational methodology allow the systematic recording and subsequent quantification of the spontaneous behavior displayed by participants in natural contexts. These research methods are frequently used in psychology, as well as in the social, educational and health fields due to their multiple advantages, such as a low level of intervention, independence with respect to standardized measurement instruments or their flexibility when applied in non-standardized intervention contexts. A Methodological Quality Scale for Studies Based on Observational Methodology (MQSOM), a tool to measure the methodological quality of these studies, has recently been validated with adequate psychometric properties (RMSEA = 0.000, NNFI = 1, GFI = .98, AGFI = .97). The MQSOM comprises a second-order factor of Methodological quality (ω = .87; D = .55) containing two first-order factors: Quality of design (6 items; ω = .90; D = .46; ICC = .933 - .967) and Quality of measurement and analysis (5 items; ω = .68; D = .67; ICC = .797 - .988).
Objective: the aim of this study is to present the evidence of convergent and discriminant validity of MQSOM.
Methods: a multitrait-multimethod analysis (MTMM) with Spearman correlations was carried out to examine the relationship between the dimensions of MQSOM and those of the methodological quality instruments Rigorous Mixed-Methods (RMM), Guidelines for Publishing Evaluations Based on Observational Methodology (GREOM) and Mixed Methods Appraisal Tool (MMAT), circumscribed to the field of Mixed-Methods studies. Ninety-six articles based on observational methodology were coded with MQSOM and each of the contrast instruments. Results: adequate levels of inter- and intra-coder reliability were obtained (ICC between .73 and 1). MQSOM dimension of Design showed empirical evidence of convergence with MRMM (ρ between .22 and .47), GREOM (ρ between .22 and .34) and MMAT (ρ = .21). It also showed empirical evidence of discriminant validity with the contrast instruments (ρ between -.05 and .03 regarding MRMM; ρ between -.03 and .03 regarding GREOM; ρ = -.04 regarding MMAT). MQSOM dimension of Measurement and Analysis showed empirical evidence of convergence with MRMM (ρ between .21 and .61), GREOM (ρ between .22 and .61), and MMAT (ρ between .21 and .64). Conclusions: these results support the use of MQSOM, a brief instrument that addresses methodological quality in observational methodology in a diagnostic way, measuring the quality of design, measurement and analysis of results in studies based on observational methodology, but also in a prescriptive way, serving as a reference for applied researchers, editorial boards and other decision-making committees.
Funding: This work was supported by the research project PID2020-115486GB-I00 funded by the Ministerio de Ciencia, Innovación y Universidades, MICIU/ AEI/10.13039/501100011033, Government of Spain; and the Chilean government project FONDECYT Regular 1250316 funded by the National Fund for Scientific and Technological Development, ANID.
