Search and Filter

Edueval-Db: A Role-Based Dataset For Pedagogical Risk Evaluation In Educational Explanations

Date

AI Overview

Why it matters

Few educational datasets evaluate instructional explanations using a holistic, risk-based approach across diverse teaching styles. Automated evaluators are needed to detect pedagogical and epistemic risks in AI-generated and human instructional explanations.

Study at a glance
  • Question: Can language models reliably evaluate pedagogical risks in K–12 instructional explanations across different simulated teaching styles?
  • Setting and data: 139 ScienceQA questions across science, language arts, and social science in grades K–12, with 854 human and simulated explanations.
  • Method: Model or benchmark evaluation with supervised fine-tuning using Low-Rank Adaptation (LoRA) under 5-fold cross-validation.
  • AI tools: GPT-5, Gemini 2.5 Pro, Llama 3.1 8B Instruct, LearnLM
Key findings
  • Fine-tuning a lightweight Llama 3.1 8B model reduced evaluation error across all rubric dimensions, lowering error on student-level appropriateness by 98.6%.
  • Gemini 2.5 Pro achieved the lowest error on factual accuracy (0.012 MAE), but the fine-tuned Llama model outperformed it on all non-factual pedagogical dimensions.
  • Supervised fine-tuning substantially improved the model's sensitivity to detecting pedagogical risks, reducing the zero-shot baseline's strong bias toward predicting risk-free explanations.

Summary drafted with AI from the full paper and may contain errors. See the full paper for details.

This work introduces EduEVAL-DB, a dataset based on teacher roles designed to support the evaluation and training of automatic pedagogical evaluators and AI tutors for instructional explanations. The dataset comprises 854 explanations corresponding to 139 questions from a curated subset of the ScienceQA benchmark, spanning science, language, and social science across K-12 grade levels. For each question, one human-teacher explanation is provided and six are generated by LLM-simulated teacher roles. These roles are inspired by instructional styles and shortcomings observed in real educational practice and are instantiated via prompt engineering. We further propose a pedagogical risk rubric aligned with established educational standards, operationalizing five complementary risk dimensions: factual correctness, explanatory depth and completeness, focus and relevance, student-level appropriateness, and ideological bias. All explanations are annotated with binary risk labels through a semi-automatic process with expert teacher review. Finally, we present preliminary validation experiments to assess the suitability of EduEVAL-DB for evaluation. We benchmark a state-of-the-art education-oriented model (Gemini 2.5 Pro) against a lightweight local Llama 3.1 8B model and examine whether supervised fine-tuning on EduEVAL-DB supports pedagogical risk detection using models deployable on consumer hardware.