Speakers
Description
Background. Careless and insufficient effort responding (C/IER) on self-report measures produces responses that fail to accurately reflect the trait being measured, posing a major threat to the quality and validity of survey data. While detecting C/IER is vital to ensure validity of conclusions drawn from self-report data, it is a non-trivial endeavor, with each detection method involving distinct assumptions and limitations.
Objectives. This study compares two prominent approaches for C/IER identification and adjustment based on respondent behavior: (1) attention check items, which offer clear interpretability but require careful and parsimonious administration, limiting their ability to monitor C/IER comprehensively, and (2) a model-based mixture IRT approach, which avoids the need for additional items but relies on strong assumptions about respondent behavior.
Methods. Using data from five scales of a job quality survey completed by 707 respondents, we fitted an explanatory mixture IRT approach by means of R.
Results and Conclusions. We observed strong alignment between the two approaches: respondents identified as less attentive by one method were similarly flagged by the other. Overall, both approaches suggested that C/IER remained relatively stable throughout the course of the questionnaire. However, single attention check items consistently indicated substantially lower levels of C/IER at multiple points throughout the questionnaire compared to the scale-level C/IER rates implied by the model-based approach. Both methods had comparable impacts on adjusted psychometric properties. While correlations between latent constructs did not differ markedly from their unadjusted counterparts, adjusted trait estimates were less reliable, especially when obtained using the model-based approach, reflecting greater uncertainty in respondents’ trait levels. Implications for C/IER identification and adjustment are discussed, arguing for a triangulation of different approaches.
This work was partially supported by the Research Council of Norway through its Centres of Excellence scheme, project number 33160 and by the Spanish project PID2022-141339NB-I00, funded by MCIU /AEI
/10.13039/501100011033 / and FEDER A way to make Europe, EU
