Speakers
Abstract
Introduction: When writing multiple-choice items, the "None of the Above" (NOTA) option has been widely defended and criticized. Some authors argue that it artificially increases item difficulty and may even be detrimental because it negatively affects item discrimination, making it harder to distinguish between students with more knowledge and those who have not acquired it. However, other authors suggest that, apart from being an easy-to-create option, forcing students to confront this option, especially in items with non-numerical solutions, positively impacts their reasoning, particularly when NOTA is the correct option. This could lead to improved performance on similar or future tests. Therefore, the objective of this study is to assess the impact of this response option. Method: A sample of 443 students from various degree programs, such as Economics, Business Administration and Management, and Marketing, completed an inferential statistics test consisting of 12 items with three response options. Different versions of the same test were created to ensure that all items were answered in three formats: (1) a version with three distinct response options, (2) a version where NOTA was the correct answer, and (3) a version where NOTA was one of the distractors. Additional data on student performance, such as results from other tests in the same subject, were also collected. Comprehensive analyses will be conducted in R, first considering the Classical Test Theory (CTT) indicators for the scores of the test and the items, and later, performing some crossed random-effects multilevel models, to consider the items and students as two random factors or the model, to account the variability of both factors. Results: As preliminary results, the reliability of the obtained scores was calculated using Cronbach’s alpha coefficient for the three models: Model A had a value of 0.638, Model B had a value of 0.406, and Model C had a value of 0.685. Given that the test consisted of only 12 items, the reliability coefficient was also calculated, and the Spearman-Brown formula was applied to determine the optimal test length for each version: 38 items for Model A, 68 for Model B, and 23 for Model C. Additionally, difficulty and discrimination indicators were calculated for each item. Regarding difficulty, the findings confirm that the inclusion of NOTA increases item difficulty, particularly when NOTA is the correct answer. In terms of discrimination, a general decrease was observed when NOTA was presented as a response option, with the lowest discrimination occurring when NOTA was the correct answer. However, this indicator depends on the specific model used, requiring further analysis to draw precise conclusions. Conclusion: The inclusion of NOTA appears to negatively impact item properties by reducing item discrimination. Regarding difficulty, it would be interesting to explore whether the increase in difficulty is artificial and therefore negative, or whether it enhances students' reasoning skills. This relationship will be further investigated with additional assessment tests on the subject.
| Poster | Response Options in Multiple-Choice Items: Increased Difficulty or Improved Reasoning? |
|---|---|
| Author | Susana Sanz Velasco, Pablo Ares-Gastesi, Ángel Manuel García-Carmona, Fermín González Pereiro, María del Valle Jiménez Jaraba, Joaquín Juliani Aguado, Manuel López Pavón, Dulcinea Raboso Paniagua, Samuel Ranz Castañeda, Clara Patricia de Lacy Pérez de los Cobos |
| Affiliation | Universidad San Pablo CEU |
| Keywords | Multiple-choice-items, NOTA, difficulty, discrimination, reasoning |
