XI Conference – European Congress of Methodology

Reliability and validity of experimental measures of value-driven attention: a meta-analysis on individual differences studies

23 Jul 2025, 10:30

15m

Faculty of Social Sciences and Communication. (The Pyramid)/13 - Room (Faculty of Social Sciences and Communication. (The Pyramid))

Oral Presentation Applications/Substantive areas Session 12 : "Validity and Reliability in Psychological measurement"

Speakers

Francisco Garre Frutos (Universidad de Granada)Mr Pablo López-Mártinez (Universidad de Málaga)Mr Pablo Solana (Universidad de Granada)Prof. Juan Lupiáñez (Universidad de Granada)Prof. Miguel A. Vadillo (Universidad Autónoma de Madrid)

Abstract

Value-driven attention refers to the automatic allocation of attentional resources to reward-associated stimuli, even when such allocation conflicts with task goals. This phenomenon has been proposed as a manifestation of individual differences in "attentional sign-tracking" and has been linked to psychopathological traits. However, the experimental measures commonly employed in this field are difference scores, which often fail to meet psychometric standards. The low reliability in these measures casts doubt on the validity of many findings. To address these concerns, we conducted two meta-analyses. The first one examined the reliability of measures typically employed in studies of value-driven attention. Using 18 publicly available raw datasets, we found that eye-tracking measures consistently demonstrated superior reliability compared to response time measures. This advantage is explained by lower correlations among component measures in eye-tracking data, which reduces measurement error and enhances reliability in differences scores often employed in experimental psychology. Additionally, our analyses indicate that response time measures can attenuate observed effects by up to 31% relative to eye-tracking, even assuming perfect reliability in the other measure. The second meta-analysis assessed the presence of publication bias and the expected replicability rate (statistical power) of the existing literature using z-curve analysis. Analyzing 25 studies (27 contrasts), we estimated a quite low replication rate of only 19%. In conclusion, the present results challenge the validity and reliability of the existing literature on value-driven attention and its individual differences. We advocate for the use of highly reliable measures (such as eye-tracking), alongside pre-registration of hypotheses, adequately powered study designs, and transparent reporting practices.