Date
Publisher
arXiv
The integration of Large Language Models (LLMs) in K--12 education offers
both transformative opportunities and emerging risks. This study explores how
students may Trojanize prompts to elicit unsafe or unintended outputs from
LLMs, bypassing established content moderation systems with safety guardrils.
Through a systematic experiment involving simulated K--12 queries and
multi-turn dialogues, we expose key vulnerabilities in GPT-3.5 and GPT-4. This
paper presents our experimental design, detailed findings, and a prototype
tool, TrojanPromptGuard (TPG), to automatically detect and mitigate Trojanized
educational prompts. These insights aim to inform both AI safety researchers
and educational technologists on the safe deployment of LLMs for educators.
Study at a glance
- Question: Can multi-turn educational prompts bypass large language model safety guardrails to elicit unsafe responses?
- Setting and data: 500 simulated multi-turn prompt trials using educational scenarios across GPT-3.5 and GPT-4.
- Method: Adversarial red-teaming experiment with chi-square testing and prototype system design.
Why it matters
Students may craft multi-turn prompts using educational or moral framing to bypass moderation safeguards, inadvertently eliciting unsafe model outputs when unmonitored in classrooms.
Key findings
- High-risk prompt chains bypassed safety guardrails in 50% of trials, with neither model flagging transitions into semantically risky territory at the third conversational turn.
- Bypass rates did not show a statistically significant difference between model types or prompt exploit categories across the 500 trials.
- GPT-4 exhibited higher vulnerability to subtle misdirection than GPT-3.5, reaching bypass rates of up to 62.86% in literary escalation scenarios.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
What is the application?
Who is the user?
Why use AI?
