Date
Publisher
arXiv
As AI becomes part of everyday learning, many courses teach students to use it mainly as a productivity tool: how to prompt, search, summarize, write, code, and use tools more efficiently. We argue that AI education also needs a setting in which students learn to test AI and understand their own role in judging machine-produced knowledge. To this end, we introduce a course-based practice that teaches AI through benchmark construction, using deep research systems as a concrete example of AI-era knowledge work. Students turn disciplinary knowledge into verifiable expert-level questions, review one another's designs for ambiguity and shortcuts, and evaluate AI systems on the resulting tasks. This activity gives students direct exposure to a powerful tool while asking them to specify what a trustworthy answer would require. The produced benchmark, QuestBench, consists of 256 questions across 14 humanities and social-science domains. Evaluation on QuestBench shows that student-designed tasks reveal hidden failures in current deep research systems: across thirteen evaluated systems, the mean question-level pass rate is only 16.85%, and the best-performing system, GPT-5.5, reaches a 57.58% pass rate. The failures are educationally useful because they show how fluent, source-backed answers can still miss the right query, source, term, or evidence standard. Reflections from five student contributors suggest that benchmark construction can help students see professional knowledge not only as content AI may retrieve, but as the basis for judging AI outputs. We present QuestBench as a benchmark artifact and as a reusable classroom setting for a larger educational question: how students can remain responsible knowledge actors as AI enters learning and professional work. The dataset is available at https://huggingface.co/datasets/PKUAIWeb/QuestBench/tree/main.
Study at a glance
- Question: Can student benchmark construction help teach accountable evaluation and reveal hidden limitations in deep research AI systems?
- Setting and data: Peking University undergraduate students across 14 humanities and social-science disciplines, evaluating 13 AI systems on 256 student-designed questions.
- Method: Benchmark development and model evaluation, supplemented by qualitative error analysis and student reflections.
Why it matters
AI instruction often focuses only on tool use and productivity. If students merely receive outputs, they may not learn to inspect processes, check evidence, or remain accountable when AI participates in knowledge work.
Key findings
- Thirteen evaluated deep research systems achieved a mean question pass rate of only 16.85%, with the best-performing system, GPT-5.5, reaching 57.58%.
- Across 1,457 non-passing baseline responses, primary failure points were retrieval failure (32.8%), unsupported inference (24.8%), and incomplete or off-target answering (17.2%).
- Higher search tool usage did not consistently yield higher accuracy across evaluated models, indicating that effective search relies on query formulation and source judgment.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
What is the application?
Who is the user?
Who age?
Why use AI?
