Date
Publisher
arXiv
The rapid development of Large Language Models (LLMs) opens up the
possibility of using them as personal tutors. This has led to the development
of several intelligent tutoring systems and learning assistants that use LLMs
as back-ends with various degrees of engineering. In this study, we seek to
compare human tutors with LLM tutors in terms of engagement, empathy,
scaffolding, and conciseness. We ask human tutors to annotate and compare the
performance of an LLM tutor with that of a human tutor in teaching grade-school
math word problems on these qualities. We find that annotators with teaching
experience perceive LLMs as showing higher performance than human tutors in all
4 metrics. The biggest advantage is in empathy, where 80% of our annotators
prefer the LLM tutor more often than the human tutors. Our study paints a
positive picture of LLMs as tutors and indicates that these models can be used
to reduce the load on human teachers in the future.
Study at a glance
- Question: How do educators rate AI tutors compared to human tutors when teaching grade-school math word problems?
- Setting and data: Human tutors with teaching experience evaluating tutoring on grade-school math word problems.
- Method: Blind comparison where human tutors evaluate and annotate AI and human tutoring performance.
Why it matters
Rapid development of large language models has led to new intelligent tutoring systems and personal learning assistants. Evaluating their promise requires comparing how LLM tutors perform alongside human tutors in engagement, empathy, scaffolding, and conciseness.
Key findings
- In grade-school math tutoring, evaluators with teaching experience rated AI tutors higher than human tutors across all four evaluated teaching qualities.
- Eighty percent of evaluators preferred the AI tutor over human tutors more often for showing empathy, representing its biggest measured advantage.
- Evaluators perceived AI tutors as delivering higher performance than human tutors in engagement, conciseness, and step-by-step learning support.
Summary drafted with AI from the paper’s abstract and may contain errors. See the full paper for details.
What is the application?
Who is the user?
Who age?
Why use AI?
Study design
