Search and Filter

Beagle: Behavior-Enforced Agent For Grounded Learner Emulation

Date

AI Overview

Why it matters

Collecting real student data raises privacy, cost, and ethical issues, while large language models struggle to simulate realistic novice learners because they optimize for correctness rather than iterative trial-and-error.

Study at a glance
  • Question: Can an AI agent realistically emulate the iterative struggles, errors, and behaviors of novice learners?
  • Setting and data: Student programming logs across physics and math tasks, plus 71 human evaluators (undergraduates, graduate students, professionals, educators).
  • Method: System development, simulation benchmark evaluation against student logs, and a human Turing test (Two One-Sided Tests).
  • AI tools: BEAGLE, Gemini 2.0 Flash, Gemini 2.5 Flash, Gemini 3 Flash, GPT-4o-mini, Claude Haiku 4.5
Key findings
  • In a human Turing test with 71 participants, classification accuracy between BEAGLE traces and real student data was statistically equivalent to chance at 52.8 percent.
  • BEAGLE substantially reduced behavioral divergence from real student data compared to baselines and achieved an 86.2 percent error recurrence rate, capturing persistent novice struggle.
  • Ablations showed that removing the semi-Markov controller caused behavioral divergence to spike, while merging the strategist and executor modules reduced error recurrence by 21 percent.

Summary drafted with AI from the full paper and may contain errors. See the full paper for details.

Simulating student learning behaviors in open-ended problem-solving environments holds potential for education research, from training adaptive tutoring systems to stress-testing pedagogical interventions. However, collecting authentic data is challenging due to privacy concerns and the high cost of longitudinal studies. While Large Language Models (LLMs) offer a promising path to student simulation, they suffer from competency bias, optimizing for efficient correctness rather than the erratic, iterative struggle characteristic of novice learners. We present BEAGLE, a neuro-symbolic framework that addresses this bias by incorporating Self-Regulated Learning (SRL) theory into a novel architecture. BEAGLE integrates three key technical innovations: (1) a semi-Markov model that governs the timing and transitions of cognitive behaviors and metacognitive behaviors; (2) Bayesian Knowledge Tracing with explicit flaw injection to enforce realistic knowledge gaps and "unknown unknowns"; and (3) a decoupled agent design that separates high-level strategy use from code generation actions to prevent the model from silently correcting its own intentional errors. In evaluations on Python programming tasks, BEAGLE significantly outperforms state-of-the-art baselines in reproducing authentic trajectories. In a human Turing test, users were unable to distinguish synthetic traces from real student data, achieving an accuracy indistinguishable from random guessing (52.8%).