Search and Filter

Llm Prompt Evaluation For Educational Applications

Date

AI Overview

Why it matters

Educational technology often relies on ad-hoc prompt engineering rather than systematic evaluation. Without evidence-based methods to evaluate prompts, it is difficult to determine which prompting strategies produce personalized, pedagogically aligned learning interactions.

Study at a glance
  • Question: How do different prompt engineering designs compare in generating high-quality educational follow-up questions?
  • Setting and data: 120 user interactions from three textbook reading deployments: 93 Prolific crowd workers, 17 psychology university students, and 9 college programming students.
  • Method: Tournament-style comparative judgment evaluation using the Glicko2 rating system with eight judges.
  • AI tools: Llama 3, STAIRS, Longformer, all-MiniLM-L6-v2
Key findings
  • A Strategic Reading Coach prompt outperformed all other templates, achieving win probabilities between 81% and 100% in pairwise comparisons.
  • The Scaffolding Expert prompt finished second overall, securing a 100% win probability over the baseline template.
  • A baseline prompt not designed with modern patterns placed third overall, outperforming several templates grounded in learning theory.

Summary drafted with AI from the full paper and may contain errors. See the full paper for details.

As large language models (LLMs) become increasingly common in educational applications, there is a growing need for evidence-based methods to design and evaluate LLM prompts that produce personalized and pedagogically aligned out-puts. This study presents a generalizable, systematic approach for evaluating prompts, demonstrated through an analysis of LLM-generated follow-up questions in a structured dialogue activity. Six prompt templates were designed and tested. The templates incorporated established prompt engineering patterns, with each prompt emphasizing distinct pedagogical strategies. The prompt templates were compared through a tournament-style evaluation framework that can be adapted for other educational applications. The tournament employed the Glicko2 rating system with eight judges evaluating question pairs across three dimensions: format, dialogue support, and appropriateness for learners. Data was sourced from 120 authentic user interactions across three distinct educational deployments. Results showed that a single prompt related to strategic reading out-performed other templates with win probabilities ranging from 81% to 100% in pairwise comparisons. This prompt combined persona and context manager pat-terns and was designed to support metacognitive learning strategies such as self-directed learning. The methodology showcases how educational technology re- searchers can systematically evaluate and improve prompt designs, moving beyond ad-hoc prompt engineering toward evidence-based prompt development for educational applications.