Date
Publisher
arXiv
As a publicly funded body in the UK, Oak National Academy is in a unique position to innovate within this field as we have a comprehensive curriculum of approximately 13,000 open education resources (OER) for all National Curriculum subjects, designed and quality-assured by expert, human teachers. This has provided the corpus of content needed for building a high-quality AI-powered lesson planning tool, Aila, that is free to use and, therefore, accessible to all teachers across the country. Furthermore, using our evidence-informed curriculum principles, we have codified and exemplified each component of lesson design. To assess the quality of lessons produced by Aila at scale, we have developed an AI-powered auto-evaluation agent,facilitating informed improvements to enhance output quality. Through comparisons between human and auto-evaluations, we have begun to refine this agent further to increase its accuracy, measured by its alignment with an expert human evaluator. In this paper we present this iterative evaluation process through an illustrative case study focused on one quality benchmark - the level of challenge within multiple-choice quizzes. We also explore the contribution that this may make to similar projects and the wider sector.
Study at a glance
- Question: How accurately can an AI tool evaluate AI-generated lesson materials compared to human expert teachers?
- Setting and data: United Kingdom; about 13,000 open educational resources across national curriculum subjects, and AI-generated multiple-choice quizzes.
- Method: Case study comparing automated AI evaluations of lesson quizzes with ratings from an expert human evaluator.
Why it matters
Improving the quality and safety of AI-generated lesson resources requires assessing materials at scale. Auto-evaluation serves as a critical measure to guide these improvements and align outputs with expert human standards.
Key findings
- In the UK, an AI evaluation agent assessed lesson plans at scale, facilitating informed improvements to AI-generated instructional materials across subjects.
- Comparing automated ratings with expert human evaluations helped increase the AI evaluator's accuracy and alignment with human judgments.
- A collection of approximately 13,000 teacher-designed open resources provided the content needed to build a free AI lesson-planning tool.
Summary drafted with AI from the paper’s abstract and may contain errors. See the full paper for details.
What is the application?
Who is the user?
Why use AI?
Study design
