Date
Publisher
arXiv
Large Language Models (LLMs) have recently demonstrated strong emergent
abilities in complex reasoning and zero-shot generalization, showing
unprecedented potential for LLM-as-a-judge applications in education, peer
review, and data quality evaluation. However, their robustness under prompt
injection attacks, where malicious instructions are embedded into the content
to manipulate outputs, remains a significant concern. In this work, we explore
a frustratingly simple yet effective attack setting to test whether LLMs can be
easily misled. Specifically, we evaluate LLMs on basic arithmetic questions
(e.g., "What is 3 + 2?") presented as either multiple-choice or true-false
judgment problems within PDF files, where hidden prompts are injected into the
file. Our results reveal that LLMs are indeed vulnerable to such hidden prompt
injection attacks, even in these trivial scenarios, highlighting serious
robustness risks for LLM-as-a-judge applications.
Study at a glance
- Question: Can hidden instructions easily trick AI models when answering simple multiple-choice questions?
- Setting and data: Basic arithmetic multiple-choice and true-false questions within PDF files containing injected hidden prompts.
- Method: Evaluates AI models on basic multiple-choice and true-false questions containing hidden prompt injection attacks.
Why it matters
Large language models show unprecedented potential for LLM-as-a-judge applications in education. However, their robustness under prompt injection attacks, where malicious instructions are embedded into content to manipulate outputs, remains a significant concern.
Key findings
- Language models evaluating basic arithmetic questions inside PDF files were easily misled by hidden instructions embedded in the documents.
- Hidden text successfully manipulated model outputs on simple multiple-choice and true-false judgment questions.
- Language models showed serious vulnerabilities to manipulation despite their strong reasoning abilities for educational evaluation tasks.
Summary drafted with AI from the paper’s abstract and may contain errors. See the full paper for details.
What is the application?
Who is the user?
Who age?
Why use AI?
Study design
