XI Conference – European Congress of Methodology

A quantitative approach to the evaluation of response processes.

23 Jul 2025, 11:30

30m

Poster Measurement Poster Session 1

Abstract

Data are the basis of every psychometrical inference that psychologists make. The quality of the procedure followed by the researchers is not relevant if data are deficient; every kind of inference based on them will be erroneous.
Of the five sources of validity evidence proposed by the Standards, those related to the response process (along with the consequences) have received the least attention. The Standards provide few sources of validity evidence that allow us to assess whether the cognitive process involved in responding to an item (or set of items) is appropriate for the intended use and the definition of the evaluated construct. Among the different aspects that can affect how a respondent answers a question is the item format. The number of response alternatives or the verbal labels linked to the numerical alternatives, among others, can produce the cognitive process implied in the answer that does not match the expected process. Several answer biases, such as central tendency and acquiescence (among others), can produce erroneous ability estimation of the construct score.
One of the current strategies to approach the study of response styles from the theoretical framework of IRT are the IRTree models. These models are becoming increasingly popular due to their flexibility, ease of interpretation, and "easy" implementation. As the name suggests, these models adopt a tree structure in which multiple pseudo-items are generated representing different cognitive processes. Although there are some multidimensional models nowadays, traditionally, it is assumed that these processes are unidimensional.
This approach is particularly beneficial because it allows the study of various response behaviors involved in each process. In this work, we analyze the response styles of a 5-point Likert-type scale that evaluates perfectionism in adolescents.
Since the number of alternatives is odd, the model called the midpoint primary process (MPP) will be used, in which the variance is decomposed into three nodes: a) central response tendency; b) agreement with the item; c) extreme response. Analyzing the results, it can be seen how this new approach, through IRTree models, allows us to have a 'purer' estimate of the ability level of the evaluated individuals after controlling for both the central and extreme response tendency effects. This approximation, as argued by different studies, allows for better psychometric inference, instilling confidence in the validity of the assessments.
IRTree models are shown as a promising approach to evaluating the underlying response processes when answering questionnaires. Obtaining ability levels from which the possible effects of response tendencies, such as central and extreme response biases, have been subtracted will provide a better approximation to other sources of validity evidence. Of course, this approximation to the evaluation of the cognitive processes implied in the answer to questionnaires can be complemented by the classical approaches such as, for example, cognitive interviews or thinking aloud.