Date
Publisher
arXiv
The integration of Large Language Models (LLMs) in K--12 education offers
both transformative opportunities and emerging risks. This study explores how
students may Trojanize prompts to elicit unsafe or unintended outputs from
LLMs, bypassing established content moderation systems with safety guardrils.
Through a systematic experiment involving simulated K--12 queries and
multi-turn dialogues, we expose key vulnerabilities in GPT-3.5 and GPT-4. This
paper presents our experimental design, detailed findings, and a prototype
tool, TrojanPromptGuard (TPG), to automatically detect and mitigate Trojanized
educational prompts. These insights aim to inform both AI safety researchers
and educational technologists on the safe deployment of LLMs for educators.
Study at a glance
- Question: How can schools prevent students from tricking educational AI tools into producing unsafe or unintended outputs?
- Setting and data: Simulated K–12 queries and multi-turn dialogues with GPT-3.5 and GPT-4.
- Method: Systematic experiment testing AI models and development of a prototype detection tool (TrojanPromptGuard).
Why it matters
Integrating Large Language Models in K-12 education brings emerging risks. Students may Trojanize prompts to bypass established content moderation systems and safety guardrails, eliciting unsafe or unintended outputs.
Key findings
- In simulated K-12 queries and multi-turn dialogues, GPT-3.5 and GPT-4 revealed key vulnerabilities to manipulated prompt chains.
- Students may elicit unsafe or unintended outputs by using chained prompts that bypass established AI content moderation systems.
- A prototype tool can automatically detect and mitigate manipulated educational prompts that bypass safety guardrails.
Summary drafted with AI from the paper’s abstract and may contain errors. See the full paper for details.
What is the application?
Who is the user?
Why use AI?
