Abstract
Method variance due to wording effects represents a critical challenge in psychometric research and measurement validity, often distorting factor structures and inflating dimensionality estimates. The wording effect stems from the inclusion of negatively oriented items and occurs when individuals provide inconsistent responses to positively worded (PW) and negatively worded (RW) items that are intended to reflect the same substantive dimension. Traditional dimensionality reduction techniques, such as Exploratory Graph Analysis (EGA) and Parallel Analysis (PA), tend to overestimate the number of factors when applied to mixed-worded scales, potentially compromising the interpretability of psychological measures. This study evaluates the effectiveness of the random intercept item factor analysis (RIIFA) in addressing this issue by isolating method variance at the exploratory stage. RIIFA introduces an additional latent variable—the random intercept factor—which can be conceptualized as a method factor capturing individual differences in the use of the response scale. By doing so, it helps separate substantive variance from systematic variance introduced by item wording, ultimately leading to a more accurate operationalization of the intended construct. Using a large UK sample (N = 977) who responded to the Short Grit Scale (Grit-S), we first performed a redundancy analysis to detect locally dependent items and then compared standard EGA and PA solutions with their RIIFA-based counterparts (i.e., riEGA and riPA), where a random intercept factor was specified with unit loadings for both PW and RW items and its variance freely estimated. To compare the robustness of the factor solutions, stability was further assessed through bootstrap analyses. Based on the redundancy analysis, one item was removed, resulting in the refined 7-item version of the scale (Grit-S7). The results confirmed that EGA and PA overestimated the number of factors, identifying one factor loaded by PW items and another by RW items, whereas both riEGA and riPA provided a unidimensional solution. Moreover, RIIFA-based techniques demonstrated greater stability across 5000 bootstrap resamples compared to their traditional counterparts. These findings highlight the potential of RIIFA as a valuable framework for mitigating method variance in exploratory analyses, offering a more accurate estimation of the substantive dimensionality. By effectively controlling for an approximate portion of variance caused by item wording, RIIFA reduces artificial factor proliferation and enhances structural validity, making it a promising approach for researchers dealing with mixed-worded scales.
| Poster | Addressing Overfactoring in Mixed-Worded Scales: An Exploratory Application of the Random Intercept Item Factor Analysis (RIIFA) |
|---|---|
| Author | Giuliana Nasonte & Palmira Faraci |
| Affiliation | Psychometrics Laboratory, Department of Human and Social Sciences, University Kore of Enna (UKE), Italy |
| Keywords | wording_effects, dimensionality, exploratory_graph_analysis, parallel_analysis, mixed-worded_scales |
