Date
AI Overview
Why it matters
General AI models are optimized for helpfulness rather than pedagogy, making them prone to giving direct answers. Translating learning science into AI prompts and standard evaluations remains a major challenge.
Study at a glance
- Question: Can fine-tuning and specialized evaluation benchmarks improve the pedagogical capabilities and safety of a generative AI tutor?
- Setting and data: UK educators and students; online adult learners; and 74 college students using a Chrome extension at Arizona State University.
- Method: Mixed methods combining participatory co-design workshops, user interviews, model fine-tuning, automated LLM-based evaluations, and human rating experiments.
- AI tools: LearnLM-Tutor, Gemini 1.0, PaLM 2.0, HallMate, ShiffBot
Key findings
- Educator raters significantly preferred LearnLM-Tutor over prompt-tuned Gemini 1.0 across multiple pedagogical dimensions, including asking questions, providing openings to engage, and promoting active learning.
- In side-by-side comparisons of full tutoring conversations, expert raters preferred LearnLM-Tutor over Gemini 1.0 on four out of five high-level pedagogical criteria.
- Targeted safety fine-tuning reduced inappropriate model behaviors, lowering the failure rate of praising harm-inducing queries from 0.73 in the initial model to 0.02 in LearnLM-Tutor.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
A major challenge facing the world is the provision of equitable and universal access to quality education. Recent advances in generative AI (gen AI) have created excitement about the potential of new technologies to offer a personal tutor for every learner and a teaching assistant for every teacher. The full extent of this dream, however, has not yet materialised. We argue that this is primarily due to the difficulties with verbalising pedagogical intuitions into gen AI prompts and the lack of good evaluation practices, reinforced by the challenges in defining excellent pedagogy. Here we present our work collaborating with learners and educators to translate high level principles from learning science into a pragmatic set of seven diverse educational benchmarks, spanning quantitative, qualitative, automatic and human evaluations; and to develop a new set of fine-tuning datasets to improve the pedagogical capabilities of Gemini, introducing LearnLM-Tutor. Our evaluations show that LearnLM-Tutor is consistently preferred over a prompt tuned Gemini by educators and learners on a number of pedagogical dimensions. We hope that this work can serve as a first step towards developing a comprehensive educational evaluation framework, and that this can enable rapid progress within the AI and EdTech communities towards maximising the positive impact of gen AI in education.
Research Categories
What is the application?
Who age?
Why use AI?
Study design
Publication Source
Publisher
arXiv
