Date
Publisher
arXiv
The emergence of large language models (LLMs) has significantly accelerated recent research on LLM-based automatic grading (AG) systems. Benefiting from the strong instruction-following capabilities and broad prior knowledge of LLMs, educators can deploy AG systems across diverse tasks using only natural language rubrics while achieving satisfactory grading performance. Despite these advantages, new security concerns may also arise. In particular, prompt injection (PI) attacks have recently become a major threat to LLM-based applications. In the context of AG, attackers can potentially exploit PI vulnerabilities to manipulate grading systems into assigning artificially high scores regardless of the actual answer quality. Such behavior poses serious risks to the fairness, reliability, and integrity of educational assessment. In this work, we study PI attacks in AG systems, and systematically investigate the effectiveness of such attacks in educational scenarios. We further evaluate the effectiveness of existing defensive strategies against these attacks. Through comprehensive experiments under rubric-based grading settings, we demonstrate that current LLM-based AG systems remain highly vulnerable to PI attacks. We hope that our findings raise awareness of this emerging threat and motivate future research toward secure, robust, and trustworthy LLM-based educational systems.
Study at a glance
- Question: Can prompt injection attacks manipulate large language model automatic grading systems into assigning undeserved higher scores?
- Setting and data: 30 questions across four subjects, 100 student responses per question, spanning college classrooms and grades 7-10 plus elementary and middle school classes.
- Method: Model and benchmark evaluation testing heuristic, agentic, and long-context prompt injection attacks alongside preventive instruction and guard model defenses.
Why it matters
Prompt injection attacks allow students to manipulate large language model grading systems into giving higher scores regardless of answer quality, threatening the fairness, reliability, and integrity of educational assessments.
Key findings
- Prompt injection attacks succeeded across five grading models and four datasets with an average attack success rate of 56.9%.
- Many-shot jailbreaking achieved the strongest attack performance, successfully manipulating even Claude-Sonnet-4.6, which had resisted under 10% of other attacks.
- While preventive instructions and guard models reduced attack success, the attack success rate for most models remained above 30% under preventive instructions and above 15% under guard defenses against many-shot attacks.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
What is the application?
Who is the user?
Why use AI?
Study design
