Speakers
Description
Cognitive diagnostic models (CDMs) serve as an effective approach to diagnostic assessment in education. This study explores how CDMs can be applied to criterion-referenced standard setting. The relevance of the study lies in the increasing demand for more detailed assessment in education. Modern education systems require not only the sorting of students on the basis of their performance, but also a detailed analysis of their specific knowledge gaps in relation to their level of proficiency.
The study employed the Legal Literacy Test, developed for 9th-grade students. Based on their test results, students are assigned to one of three proficiency levels of legal literacy: developing, basic, or advanced. According to the theoretical framework, the test encompasses three broad subject areas, which are further divided into seven specific topics. The test consists of 30 items, including 11 dichotomous and 19 polytomous items (from 2 to 7 items per topic). The study sample included 767 9th grade students.
As in the Bookmark method of standard setting, experts review the descriptors of achievement levels and determine which items students at different levels are likely to answer correctly. For some dichotomous items, the experts assume that students at basic level or above would be able to answer correctly and receive 1 point; for other dichotomous items, advanced students would be more likely to answer correctly, or a developing level would be sufficient. Polytomous items provide more detailed diagnostic information. For some items, experts suggest that advanced students are more likely to score 2 points and basic students are more likely to score 1 point. In other cases, developing students might get 1 point while basic or advanced students might get 2 points.
The Q-matrix for CDM analysis combines information about specific topics for each item and achievement level for each item category according to expert opinion. We assume that the probability of answering the item correctly is a combination of specific topic and achievement level. Two different types of CDM models were tested. The sequential attribute CDM model (Ma, W., & de la Torre, J., 2016) included dichotomous attributes for levels, while the polytomous CDM model (de la Torre, J., Qiu, X.-L., & Santos, K. C., 2021) used a polytomous attribute for levels. Different cognitive diagnostic models (DINA, RRUM, LLM, ACDM, GDINA) were calibrated and compared using the GDINA package in R (Ma, W., & de la Torre, J., 2020).
CDMs allow the integration of information about item content and proficiency levels, providing a more nuanced analysis of student performance. The applied approach to criterion-referenced standard setting using CDM models provides precise insights into which specific components of literacy students have mastered and which require further development, and also allows students to be placed into proficiency levels for the overall test.
Abstract
Cognitive diagnostic models (CDMs) serve as an effective approach to diagnostic assessment in education. This study explores how CDMs can be applied to criterion-referenced standard setting. The relevance of the study lies in the increasing demand for more detailed assessment in education. Modern education systems require not only the sorting of students on the basis of their performance, but also a detailed analysis of their specific knowledge gaps in relation to their level of proficiency.
The study employed the Legal Literacy Test, developed for 9th-grade students. Based on their test results, students are assigned to one of three proficiency levels of legal literacy: developing, basic, or advanced. According to the theoretical framework, the test encompasses three broad subject areas, which are further divided into seven specific topics. The test consists of 30 items, including 11 dichotomous and 19 polytomous items (from 2 to 7 items per topic). The study sample included 767 9th grade students.
As in the Bookmark method of standard setting, experts review the descriptors of achievement levels and determine which items students at different levels are likely to answer correctly. For some dichotomous items, the experts assume that students at basic level or above would be able to answer correctly and receive 1 point; for other dichotomous items, advanced students would be more likely to answer correctly, or a developing level would be sufficient. Polytomous items provide more detailed diagnostic information. For some items, experts suggest that advanced students are more likely to score 2 points and basic students are more likely to score 1 point. In other cases, developing students might get 1 point while basic or advanced students might get 2 points.
The Q-matrix for CDM analysis combines information about specific topics for each item and achievement level for each item category according to expert opinion. We assume that the probability of answering the item correctly is a combination of specific topic and achievement level. Two different types of CDM models were tested. The sequential attribute CDM model (Ma, W., & de la Torre, J., 2016) included dichotomous attributes for levels, while the polytomous CDM model (de la Torre, J., Qiu, X.-L., & Santos, K. C., 2021) used a polytomous attribute for levels. Different cognitive diagnostic models (DINA, RRUM, LLM, ACDM, GDINA) were calibrated and compared using the GDINA package in R (Ma, W., & de la Torre, J., 2020).
CDMs allow the integration of information about item content and proficiency levels, providing a more nuanced analysis of student performance. The applied approach to criterion-referenced standard setting using CDM models provides precise insights into which specific components of literacy students have mastered and which require further development, and also allows students to be placed into proficiency levels for the overall test.
| Oral presentation | Using Cognitive Diagnostic Models for Criterion-Referenced Standard Setting in Legal Literacy Assessment |
|---|---|
| Author | Sergei Tarasov, Daria Gracheva |
| Affiliation | HSE University |
| Keywords | cognitive diagnostic models, standard setting |
