Date
Publisher
arXiv
Evaluating the abilities of learners is a fundamental objective in the field
of education. In particular, there is an increasing need to assess higher-order
abilities such as expressive skills and logical thinking. Constructed-response
tests such as short-answer and essay-based questions have become widely used as
a method to meet this demand. Although these tests are effective, they require
substantial manual grading, making them both labor-intensive and costly. Item
response theory (IRT) provides a promising solution by enabling the estimation
of ability from incomplete score data, where human raters grade only a subset
of answers provided by learners across multiple test items. However, the
accuracy of ability estimation declines as the proportion of missing scores
increases. Although data augmentation techniques for imputing missing scores
have been explored in order to address this limitation, they often struggle
with inaccuracy for sparse or heterogeneous data. To overcome these challenges,
this study proposes a novel method for imputing missing scores by leveraging
automated scoring technologies for accurate IRT-based ability estimation. The
proposed method achieves high accuracy in ability estimation while markedly
reducing manual grading workload.
Study at a glance
- Question: Can using AI models to grade missing student responses improve the accuracy of overall ability estimates?
- Setting and data: Japanese university students across reading comprehension (511 students) and natural science essays (327 students), plus 33 large language models across 100 writing tasks.
- Method: Model evaluation comparing ability estimation using automated scoring imputation against traditional imputation techniques and non-imputation across varying missing-score ratios.
Why it matters
Constructed-response tests capture higher-order skills but require costly, labor-intensive manual grading. Using statistical models to estimate learner ability from partially graded tests often loses accuracy as the proportion of missing scores increases.
Key findings
- Imputing missing scores with AI graders produced significantly more accurate ability estimates across all missing-data conditions than traditional imputation methods or ignoring missing scores.
- The AI-based approach maintained high estimation accuracy even under extreme data sparsity, successfully estimating ability at up to 80% to 100% missing ratios.
- Conventional imputation methods such as k-nearest neighbors and random forests performed worse than simply leaving scores missing, showing that inaccurate score imputation introduces substantial bias.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
What is the application?
Who age?
Why use AI?
Study design
