Speakers
Abstract
Background and objectives:
Multiple-choice items are widely used in assessment contexts as they enable efficient sampling of a broad range of content. However, not all students employ the same strategies when answering tests. Besides their level of knowledge on the subject being assessed, their response behavior can influence performance, making it essential to categorize these behaviors to better understand the response processes. In this regard, the study of eye movements has proven useful in related fields such as reading comprehension. This project aims to analyze information processing in multiple-choice items using eye-tracking techniques while students solve these tasks. Three objectives are proposed: 1) To examine eye movement patterns in assessment contexts using multiple-choice items, establishing categories based on information processing strategies; 2) To explore the relationship between eye movement measures and psychometric properties of items; and 3) To analyze the effects of violations of item-writing guidelines, which may impact response processing, and determine whether general processing strategies change depending on these violations.
Methods:
Eye movements during the reading of multiple-choice items will be recorded using an EyeLink 1000 Plus eye-tracker. Considering the characteristics of eye-tracking research, in which each participant is measured multiple times in a very precise way, the sample size calculated for the required statistical modeling and a minimum power of 0.80 was 43 participants. Data collected will be analyzed using multilevel models, in which we will consider two random factors (item and participant), and different fixed factors according to the objectives of each study. Separate models will be developed for each dependent variable analyzed, considering accuracy rates and different eye movement measures. Random intercepts and slopes for participants and items will also be included as random effects.
Proposed studies and expected contributions:
Study 1. Establishing a baseline framework for processing under standard assessment conditions: This study will develop a general classification, grouping participants' processing styles through eye-movements when facing multiple-choice items.
Study 2. Relationship between psychometric properties and eye movement patterns: This study will examine the relationships between eye movement measures and item characteristics, providing insights into how items are processed based on their difficulty, the selection rates for distractors, and the discrimination of options, contributing validity evidence for response processes when answering multiple-choice items.
Study 3. Effect of including item-writing guidelines’ violations: This study will assess the impact of some guidelines’ violations on the psychometric properties of items.
Study 4. Relationship between processing level and item-writing guidelines’ violations: This study will further investigate the relationship between participants' processing strategies and some item-writing guidelines’ violations, offering conclusions on differences in item difficulty and discrimination. This will inform improved multiple-choice item design, supported by validity evidence against non-recommended item-writing practices.
Conclusions:
This is an emerging field of study for which no empirical evidence currently exists in scientific literature. The goal of this project is to establish the foundations of the relationships between eye movements and the resolution of multiple-choice items, enabling the design of more reliable and valid knowledge assessment tools.
| Poster | Visual Strategies in Educational Assessment: An Analysis of Eye Movements in Multiple-Choice Item Resolution |
|---|---|
| Author | José David Moreno (1), Susana Sanz (2), María Pilar Aivar (1) & Miguel A. Sorrel (1) |
| Affiliation | (1) Universidad Autónoma de Madrid (UAM); (2) San Pablo CEU University |
| Keywords | Educational-assessment, multiple-choice-Items, eye-movements, visual-strategies, research-design |
