XI Conference – European Congress of Methodology

Where Psychometrics Meets Experimental Psychology: Bayesian Hierarchical Factor Models for Response Times

25 Jul 2025, 08:30

15m

Oral Presentation Measurement Session 5 : "Bayesian methods in Psychology and Statistics"

Abstract

Traditionally, experimental psychology has focused on examining differences between groups or experimental conditions. In recent decades, however, interest in individual differences has intensified. Yet, moving from an experimental approach to a psychometric framework is not straightforward because well-replicated experimental effects (e.g., the Stroop effect) often exhibit low reliability. As reliability decreases, the rank order of participants on a measured attribute becomes less consistent upon repeated measurement. This directly affects the correlation between two measures, as correlation reflects whether they similarly rank participants. Consequently, the probability of incorrectly concluding that two experimental measures are independent increases. This spurious independence can encourage substantive interpretations, even when the tasks were originally designed to measure the same underlying cognitive process. Thus, low reliability makes it difficult to determine (1) whether multiple processes are related and (2) whether multiple tasks measure the same cognitive process.
To address this issue, several experimental researchers have developed hierarchical models tailored to the unique features of experimental designs. Among them, the proposal by Mehrvarz and Rouder (2024) stands out as particularly innovative, as it directly aligns with classical psychometric models. These authors introduced a hierarchical model—implemented in JAGS—that recovers common factors across multiple response-time experimental tasks. Conceptually akin to exploratory factor analysis, their model assumes a Gaussian distribution for the dependent variable. Their simulation study demonstrates that this model provides a more accurate estimate of the true correlation among experimental measures than previous approaches.
Although the original model assumes a Gaussian distribution for response times, in practice these often follow markedly skewed distributions. The consequences of this specification error remain unclear, highlighting the need for alternative models that better reflect the empirical shape of response-time distributions. Building on Mehrvarz and Rouder’s (2024) work, this study develops exploratory, confirmatory, and 'semi-exploratory' hierarchical psychometric models incorporating skewed response time distributions, such as the ex-Gaussian and shifted-lognormal. These models are efficiently implemented in Stan, minimizing estimation time. A simulation study was conducted to examine the impact of fitting Gaussian models to data generated from skewed distributions—more closely reflecting the empirical nature of response times—and to assess the performance of the appropriate skewed-distribution models. Results indicate that (1) using Gaussian models on skewed data systematically underestimates correlations and increases their uncertainty, and (2) skewed models yield unbiased and efficient parameter estimates. Critically, the combined effects of underestimation and uncertainty may lead researchers to conclude that a correlation is not significantly different from zero even when a true correlation exists in the population. In sum, skewed-distribution models provide a robust alternative for studying individual differences in response-time-based experimental measures, offering significant advantages over Gaussian models in accurately capturing correlations between experimental measures.