Date
Publisher
arXiv
BERT and its variants are extensively explored for automated scoring. However, a limit of 512 tokens for these encoder-based models showed the deficiency in automated scoring of long essays. Thus, this research explores generative language models for automated scoring of long essays via summarization and prompting. The results revealed great improvement of scoring accuracy with QWK increased from 0.822 to 0.8878 for the Learning Agency Lab Automated Essay Scoring 2.0 dataset.
Study at a glance
- Question: Can generative language model summarization improve automated essay scoring accuracy for long essays?
- Setting and data: 17,307 argumentative student essays from the Learning Agency Lab Automated Essay Scoring 2.0 Kaggle competition dataset.
- Method: Model evaluation comparing transformer language models, neural networks, and ensembles trained on LLM-summarized essays.
Why it matters
Encoder-based models like BERT face a 512-token limit, creating deficiencies in automated scoring of long essays. Truncating long essays to fit this limit can omit essential semantic information.
Key findings
- A voting ensemble combining RoBERTa embeddings from GPT-5 summaries with handcrafted features achieved the highest scoring agreement of 0.8878 Quadratic Weighted Kappa.
- Using GPT-5 summarization substantially outperformed truncation-based models from earlier research, which scored below 0.80 for individual language models and 0.822 for ensembles.
- Across all summarization methods, larger GPT-5 summaries led to higher scoring accuracy than GPT-5 mini or nano, with RoBERTa consistently outperforming BERT, DeBERTa, and ELECTRA.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
What is the application?
Who is the user?
Who age?
Why use AI?
Study design
