XI Conference – European Congress of Methodology

Innovative Approaches to Address Validity

Not scheduled

1h 30m

EAM2025

Symposium Measurement Symposium : "Innovative Approaches to Address Validity"

Speaker

Prof. José Luis Padilla (University of Granada)

Abstract

Item functioning is typically evaluated through pilot studies to identify problematic items and assess their performance. However, such analyses often fail to provide insights into the underlying causes of these problems. To address this gap, alternative strategies such as psychophysiological measures, including eye movements, may offer valuable insights into participants' response processes. This study investigates the potential of eye-tracking data to inform item functioning and provide validity evidence. Two studies were conducted. The first study had two phases: Phase 1 involved creating sentences with varying levels of legibility and examining corresponding eye movement patterns. In Phase 2, eye movements were analyzed in relation to three sentence features: syntactic complexity, lexical frequency, and sentence length. Results from both phases established criteria linking specific eye movement patterns to potentially problematic items. The second study compared two versions of an instrument administered to different groups. The first group responded to the original version, while the second group received a modified version with adjustments to reduce problematic elements. Differences in responses between the groups demonstrated the extent to which eye-tracking data can guide the development of improved items. Conclusions include practical recommendations for researchers designing educational and psychological assessments. The study also highlights the utility of eye movements in providing validity evidence.

Abstract

This oral communication examines evidence of content validity in psychometrics, focusing on established procedures and potential complementary approaches. We review traditional validation methods, acknowledging their foundational importance. Our discussion emphasizes the need for isomorphic relationships between construct definitions and operational representations, a cornerstone of measurement theory. We explore how classical methods aim to achieve this isomorphism in capturing construct domains, considering both item content and latent structure. As a complementary approach, we introduce psychometric network analysis, suggesting its potential to offer additional insights into construct representation and item interrelationships. This method may enhance understanding of content coverage and construct isomorphism, working alongside established techniques. We propose an integrative approach that combines expert judgments, quantitative indices, and latent structure techniques to improve evidence of content validity and construct isomorphism. By presenting a balanced examination of current practices and emerging perspectives, we aim to contribute to the ongoing methodological dialogue in measurement science, encouraging collaborative exploration of content validity assessment strategies. We discuss the implications of these integrative approaches for improving the accuracy and comprehensiveness of psychological measurements, highlighting their potential to refine our understanding of complex psychological constructs and their underlying structures.

Abstract

Practitioners and researchers use open-ended questions when designing survey questions to obtain validity evidence of response processes to survey items and questions. Data cleaning and response coding pose significant challenges, particularly for “web probes,” given the self-administered nature of “web probes” and the large number of participants compared to the smaller number of people interviewed in the cognitive interview method. The integration of generative AI, especially models based on GPT architectures, offers new opportunities to automate these processes efficiently and accurately. This proposal focuses on the development of a data post-processing solution for automated debugging and coding of responses to web probes. This solution could be implemented by using advanced prompting techniques, in an application of the GPTs store, a standardised data processing procedure with these AI-tools; or by an application that uses the OpenAI API to offer advanced features, depending on the results or performance of each option.
The objective of the paper is twofold: a) to present the development and validation of a generative AI-based data post-processing application and procedure that allows; and b) to illustrate how such a procedure, depending on how and in which model it is applied, deductively codes themes and sub-themes in the substantive responses, and automatically detects indicators of low involvement in the response process, such as mismatches and motivational losses.
Textual data from the CAS questionnaire on climate change anxiety and other questionnaires on “quality-of-life” will be categorized into substantive (1) and non-substantive (0) by coders. Subsequently, this coding will be compared with the coding generated by four different AI models (4th, 4th custom, O1, and Deepseek) and by the state-of-the-art OpenAI model using the API. A one-shot approach will be applied, calculating the correlation between manual and automated ratings.
The application is expected to demonstrate high accuracy in response debugging and coding, significantly reducing the time and effort required in manual data processing.
This project aims to set a new standard in the automated processing of open-ended responses in psychometrics and contribute to fostering “web probing” to obtain validity evidence of response processes. Future developments of AI generative for improving validation of response processes “in vivo” will be also discussed.

Abstract

Validity remains a foundational yet complex issue in psychological and educational testing. While research frequently adheres to conventional, repetitive validation strategies, validation studies must be tailored to the unique characteristics of the construct, the instrument being developed or used, and the assessment context (the intended decision, administration mode, etc.). This symposium will showcase innovative methodologies for designing validation studies, introducing novel procedures to generate relevant and quality data, and improving assessment quality.
We will examine validity challenges in responsive assessment and explore strategies to enhance validation studies across different sources of validity evidence. Specifically, we will discuss the contributions of various procedures and techniques for gathering validity evidence based on test content, response processes, and internal structure. By addressing these advancements, the symposium aims to promote more effective, construct-specific validation practices in behavioral science research.

Abstract

An exciting development in educational and psychological testing is culturally responsive assessment, which is assessment that is “mindful of student differences and employs assessment methods appropriate for different student groups” (Montenegro & Jankowski, 2017, p. 9). Although the call for culturally responsive assessment is strong, there are few examples of good practices in this area and even fewer examples of studies validating that assessments are truly culturally responsive. In this presentation, we briefly describe the goals of culturally responsive assessments and its design principles. Next, we describe studies that could be done to evaluate the degree to which assessments that strive to be culturally responsive are achieving that goal. We end with suggestions for future research and practice in this area, including suggestions for test development and validation.