Speaker
Abstract
Differential item functioning (DIF) is a critical psychometric feature in educational testing, evaluating whether distinct subgroups respond differently to test items despite possessing the same latent ability. DIF occurs when examinees with equivalent underlying traits have varying probabilities of correctly responding to an item, influenced by subgroup membership. This study investigated whether the position of response options in the reading comprehension section of a high-stakes test contributes to DIF across subgroups defined by booklet and gender. Unlike the conventional DIF methods, the Rasch tree does not require a pre-specification of groups for detecting DIF. To investigate DIF of the test, the study analyzed item responses from 10,000 examinees on 20 multiple-choice items presented across four booklets using the ‘psychotree’ package in R. The test content and item positioning were consistent across booklets, with only the positions of the answer options varying. Following a Rasch model fit assessment, the Rasch tree analysis identified three non-predefined nodes, highlighting varying patterns of item difficulties. Four items were flagged as exhibiting DIF, with findings revealing that a combination of booklet and gender influenced test performance. Notably, booklet emerged as a primary source of DIF, particularly in interaction with gender, as DIF patterns varied between females and males, with some items displaying more pronounced deviations. A content analysis explored potential sources of DIF. One hypothesis considered was the positional effects of answer choices, such as the primacy and recency effects, where examinees might favor the first or last options. If correct answers were positioned accordingly in certain booklets, this could alter accuracy rates. It turned out that the performance of examinees in Node 2 was affected by the position of response options, specifically due to the close proximity between the correct answers and the distractors. Another hypothesis examined examinees’ strategic guessing based on perceived patterns in correct answer positions. However, this was refuted, as no discernible pattern emerged across booklets. The content of flagged items was also scrutinized to identify inherent features requiring varying levels of cognitive load. Items demanding deeper comprehension or subtle interpretation, such as identifying the main idea or understanding specific details, were more likely to show DIF. Variations in cognitive styles, educational backgrounds, and test-taking strategies among subgroups likely amplified these effects. For instance, methodical examinees considering all options may be less susceptible to positional biases, while those relying on heuristics or elimination strategies might be more affected by answer placement. This study underscores the potential for DIF arising from subtle interactions between booklet design, answer positioning, and subgroup characteristics, particularly gender. The Rasch tree effectively captured these interactions, providing a detailed understanding of how covariates influence test performance. The findings emphasize the importance of investigating non-obvious sources of DIF, such as response option positioning and its interaction with cognitive processes. This research contributes to the field by demonstrating the utility of the Rasch tree model in detecting and understanding complex DIF patterns, ultimately enhancing the fairness and validity of high-stakes assessments.
| Oral presentation | Investigating Differential Item Functioning of the Reading Comprehension Section of a High-Stakes Test across Booklet and Gender: An Analysis with Recursive Partitioning Rasch Tree |
|---|---|
| Author | Farshad Effatpanah Hesari |
| Affiliation | TU Dortmund University |
| Keywords | DIF, Reading, Rasch Tree, Booklet/Gender |
