Abstract
Ipsativity is an important concern in psychological assessment, particularly with forced-choice response formats. It refers to the lack of comparability of test scores between persons, providing information only about the predominance of traits within a person and little or no information about the person’s absolute standing in each trait. In other words, while it is possible to correctly estimate a person's difference across traits being measured (i.e., their relative predominance), the person’s average or sum across the traits is unidentified. This problem often originates from multicollinearity between trait scores. For instance, in forced-choice formats, endorsing statements associated with a given trait implies not endorsing statements from remaining dimensions, resulting in negative interdependence between scores. As a result, it can largely affect score validity, as the sum of covariances with external variables and the sum of the trait variance-covariance matrix are biased toward zero. In practice, when scores are fully ipsative, the composite/average scores across traits tend to be constant, so the absolute position of examinees is unidentified. However, the amount of ipsativity in forced-choice scores greatly depends on the psychometric characteristics of the blocks of stimuli.
This study proposes two methods to quantify ipsativity/normativity as the proportion of variance of the respondent’s true absolute trait standing (composite score) within observed scores. In essence, for a given respondent, the average (or sum) of the posterior covariance matrix of the scores is equivalent to the variance of the person’s average (or sum) score across the different traits. Two indices are derived, the first being a theoretical approximation based on the inverse of the test posterior information, and the second being an empirical index, based on posterior trait covariances conditional on the observed response patterns. In essence, low values for these indices suggest ipsativity, whereas high values denote normativity. Furthermore, through this formulation, one can quantify the measurement error due to ipsativity conditional to the trait score. That is, depending on a test's psychometric properties, it may be easier to properly identify the absolute standing of people in different regions of the trait continuum.
A simulation study was conducted to illustrate the two proposed indices. Ipsative data were generated for 5-factor datasets through a binary forced-choice design following the Thurstonian IRT model. Questionnaire lengths ranging from 10 to 60 forced-choice pairs were considered. The accuracy of the indices is quantified as the ability to recover the true ipsativity/normativity (squared correlation between true and estimated person-wise trait averages), as well as its impact on simulated continuous external variables (i.e., on external validity). Two types of external variables were simulated: one with correlations of 0.3 with all the measured traits (i.e., higher correlation with the common variance across traits), and the other with a correlation of 0.3 solely with one of the traits (i.e., higher correlation with the relative score of the corresponding trait). The simulation results support good recovery of both theoretical and empirical ipsativity/normativity indices, as well as a positive relationship between the indices and the estimated correlations with external variables.
| Oral presentation | Ipsative indices for forced-choice assessments |
|---|---|
| Author | Rodrigo S. Kreitchmann |
| Affiliation | National University of Distance Education |
| Keywords | Ipsativity, Forced-choice, Thurstonian IRT |
