Date
Publisher
arXiv
This study examines the impact of Socratic Mind, a Generative Artificial
Intelligence (GenAI) powered formative assessment tool that employs Socratic
questioning to support student learning in a large, fully online
undergraduate-level computing course. Employing a quasi-experimental,
mixed-methods design, we investigated participants' engagement patterns, the
influence of user experience on engagement, and impacts on both perceived and
actual learning outcomes. Data were collected from the system logs, surveys on
user experience and perceived engagement and learning gains, student
reflections, and course performance data. Results indicated that participants
consistently reported high levels of affective, behavioral, and cognitive
engagement, and these were strongly linked to positive user experiences and
perceived learning outcomes. Quantitative analysis further revealed that
students who engaged with the GenAI tool experienced significant gains in their
quiz scores compared to those who did not, particularly benefiting students
with lower baseline achievement. Additionally, thematic analysis of qualitative
feedback revealed substantial perceived improvements in higher-order thinking
skills, including problem solving, critical thinking, and self-reflection. Our
findings highlight the promise of AI-mediated dialogue in fostering deeper
engagement and higher-order cognitive skills. As higher education institutions
expand GenAI integration in curriculum, this dialogic, GenAI powered assessment
tool can offer a scalable strategy to promote students' meaningful learning
outcomes.
Study at a glance
- Question: How does an AI-powered Socratic assessment tool affect student engagement, learning outcomes, and higher-order thinking?
- Setting and data: Undergraduate online introductory computing course at a U.S. university, Spring 2025, 171 students analyzed.
- Method: Quasi-experimental mixed methods with difference-in-differences analysis, regression, and qualitative thematic analysis.
Why it matters
AI assessment tools often emphasize lower-level recall rather than higher-order thinking, and studies report mixed effects on learning outcomes. Socratic dialogue can deepen reasoning, but scaling it requires testing adaptive, automated tools in actual courses.
Key findings
- Engagement with the AI tool buffered against declining quiz performance, reducing the average drop in quiz scores by an estimated 3.30 points.
- The quiz score benefits were greater for students with lower baseline scores, increasing by about 0.21 points for each 1-point lower pre-intervention score.
- Most survey respondents reported improvements in higher-order thinking skills, with 69% reporting positive changes in problem-solving and 62% in critical thinking.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
What is the application?
Who is the user?
Who age?
Why use AI?
Study design
