Edumod-Llm: A Modular Approach For Designing Flexible And Transparent Educational Assistants

Meenakshi Mittal, University of California, Berkeley

Rishi Khare, University of California, Berkeley

Mihran Miroyan, University of California, Berkeley

Chancharik Mitra, Carnegie Mellon University

Narges Norouzi, University of California, Berkeley

Date

11/2025

AI Overview

Why it matters

Educational question-answering systems must align with instructional goals, yet standard benchmarks fail to capture classroom constraints. Evaluating modular AI pipeline components separately helps pinpoint bottlenecks and design transparent, pedagogically aligned systems.

Study at a glance

Question: How do different tool-selection, document-retrieval, and response-generation components perform in an educational AI assistant?
Setting and data: Undergraduate data science course, UC Berkeley, Spring 2024, 1,000 student discussion board questions with teaching assistant answers.
Method: System development and benchmark evaluation using teaching assistant rubrics and an AI-as-a-judge model.
AI tools: EduMod-LLM, Edison, GPT-4o, GPT-4.1, DeepSeek-V3, DeepSeek-R1

Key findings

Hierarchical, structure-aware retrieval achieved a Recall@1 of 0.784 and Recall@5 of 0.946, outperforming general-purpose embedding methods by nearly 30 percent.
A multi-step function-calling strategy matched or exceeded human-designed rules without requiring pre-labeled question categories.
GPT-4.1 generated the highest-rated responses across factuality, relevance, and style, surpassing several specialized reasoning models on course questions.

Summary drafted with AI from the full paper and may contain errors. See the full paper for details.

With the growing use of Large Language Model (LLM)-based Question-Answering (QA) systems in education, it is critical to evaluate their performance across individual pipeline components. In this work, we introduce {\model}, a modular function-calling LLM pipeline, and present a comprehensive evaluation along three key axes: function calling strategies, retrieval methods, and generative language models. Our framework enables fine-grained analysis by isolating and assessing each component. We benchmark function-calling performance across LLMs, compare our novel structure-aware retrieval method to vector-based and LLM-scoring baselines, and evaluate various LLMs for response synthesis. This modular approach reveals specific failure modes and performance patterns, supporting the development of interpretable and effective educational QA systems. Our findings demonstrate the value of modular function calling in improving system transparency and pedagogical alignment. Website and Supplementary Material: https://chancharikmitra.github.io/EduMod-LLM-website/