Date
Publisher
arXiv
Automated essay scoring (AES) enables scalable assessment and timely feedback but remains challenged by transformer input-length limitations, which can cause information loss when processing long essays. This study proposes a generative AI-assisted summarization framework to improve long-form essay representation while maintaining scoring reliability. Using the ASAP 2.0 dataset, we generate controlled-length summaries with three GPT-5 variants (GPT-5, GPT-5 mini, and GPT-5 nano) and use them as inputs for downstream AES models. To preserve original writing signals, handcrafted linguistic features extracted from full essays are integrated with summary representations to form a hybrid framework. The approach is evaluated in terms of scoring performance, summarization quality, and computational cost. Scoring reliability is measured using quadratic weighted kappa (QWK), while summary quality is assessed through lexical overlap, semantic similarity, information retention, and redundancy metrics. Results show that GPT-5 mini achieves the highest agreement with human ratings, whereas GPT-5 produces the strongest summarization quality. Summary quality decreases for higher-scoring essays, indicating that more complex writing is more difficult to compress without information loss. These findings reveal trade-offs among model capacity, summary fidelity, cost efficiency, and preservation of educational constructs. This study provides an initial controlled evaluation of GPT-based summarization for AES and identifies important baselines and ablation studies required for future generalization. Overall, generative AI summarization offers a promising approach for scalable writing assessment while requiring careful validation of information preservation and fairness.
Study at a glance
- Question: Can generative AI summarization reduce essay length without hurting automated scoring accuracy or increasing costs?
- Setting and data: 13,845 student argumentative essays from the Learning Agency Lab Automated Essay Scoring 2.0 dataset on Kaggle.
- Method: Model evaluation comparing three summarizers combined with handcrafted features and gradient-boosted decision trees (XGBoost and LightGBM).
Why it matters
Transformer models in automated essay scoring face strict input-length limits, causing truncation and loss of educationally relevant information in long student essays.
Key findings
- GPT-5 mini achieved the highest human-machine scoring agreement (QWK of 0.8435), slightly outperforming full GPT-5 (0.8350) and GPT-5 nano (0.8332).
- GPT-5 demonstrated the strongest summarization quality across lexical overlap and semantic preservation metrics, but its small advantage over GPT-5 mini did not produce better final scoring.
- Summarization quality dropped systematically for higher-scoring essays, suggesting that longer, more complex student writing is harder to compress without losing key information.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
What is the application?
Who is the user?
Who age?
Why use AI?
Study design
