Date
Publisher
PLOS ONE
Authoring of help content within educational technologies is labor intensive, requiring many iterations of content creation, refining, and proofreading. In this paper, we conduct an efficacy evaluation of ChatGPT-generated help using a 3 x 4 study design (N = 274) to compare the learning gains of ChatGPT to human tutor-authored help across four mathematics problem subject areas. Participants are randomly assigned to one of three hint conditions (control, human tutor, or ChatGPT) paired with one of four randomly assigned subject areas (Elementary Algebra, Intermediate Algebra, College Algebra, or Statistics). We find that only the ChatGPT condition produces statistically significant learning gains compared to a no-help control, with no statistically significant differences in gains or time-on-task observed between learners receiving ChatGPT vs human tutor help. Notably, ChatGPT-generated help failed quality checks on 32% of problems. This was, however, reducible to nearly 0% for algebra problems and 13% for statistics problems after applying self-consistency, a hallucination mitigation technique for Large Language Models.
Study at a glance
- Question: Does math help written by ChatGPT improve student learning as effectively as help written by human tutors?
- Setting and data: 274 participants working on problems in Elementary Algebra, Intermediate Algebra, College Algebra, or Statistics.
- Method: Randomized controlled trial (3 x 4 study design) comparing ChatGPT help, human tutor help, and no help.
Why it matters
Authoring help content within educational technologies is labor intensive, requiring many iterations of content creation, refining, and proofreading.
Key findings
- Across 274 algebra and statistics learners, ChatGPT-generated hints significantly improved learning over no help, producing gains equivalent to human tutor-authored help.
- Learners spent similar amounts of time on tasks whether receiving ChatGPT help or human tutor help.
- ChatGPT help failed quality checks on 32% of problems, but a mitigation technique reduced failures to nearly 0% in algebra and 13% in statistics.
Summary drafted with AI from the paper’s abstract and may contain errors. See the full paper for details.
What is the application?
Who is the user?
Who age?
Why use AI?
Study design
