Date
Publisher
arXiv
Motivation: Undergraduate computing students increasingly turn to generative AI (GenAI) tools to understand abstract concepts through analogies. Analogies compare an unfamiliar concept to something familiar, but judging whether the comparison holds requires knowledge of both. GenAI may also embed assumptions about who the learner is. GenAI education research centers on output correctness, leaving students' critical reception of analogies largely unexamined.
Method: We investigate how students evaluate the accuracy, appropriateness, and assumptions in GenAI-generated analogies, and their perceptions of interest-personalized versus generic technical explanations. Ten students with CS2 experience participated in a pre-survey, a think-aloud task with linked-list and recursion explanations, and a semi-structured interview grounded in the Paul-Elder framework. They judged accuracy, clarity, engagement, and trust separately.
Results: Most participants described interest-personalized analogies as more engaging or memorable than generic technical explanations, while trust was mixed. Some trusted the tailored analogies more; others scrutinized them more closely or distrusted the tailoring. Participants with deep source-domain knowledge identified structural flaws requiring that knowledge to recognize. Because personalization and explanation format differed together, these findings do not isolate an effect of personalization alone.
Implications: A familiar source flips the student's role. On the concept they are still learners, but on the familiar source they are the expert, and that is the position from which an analogy can be judged. We call this two-sided analogy auditing. GenAI systems should ask what students know, not just what interests them, and treat a flawed analogy as something to inspect and fix rather than accept.
Study at a glance
- Question: How do undergraduate computing students evaluate the accuracy, appropriateness, and assumptions of interest-personalized AI analogies?
- Setting and data: University of Toronto, 10 undergraduate students who completed CS2.
- Method: Think-aloud annotation task and semi-structured interview (reflexive thematic analysis).
Why it matters
Computing students frequently use generative AI tools to explain abstract concepts using analogies. However, analogies can misrepresent core concepts or embed alienating cultural assumptions, leaving students' critical evaluation of these explanations largely unexamined.
Key findings
- Most participants found interest-personalized analogies more engaging and memorable than generic technical explanations, though reports of trust were mixed.
- Participants evaluated analogies using low-level structural correspondence and precise language, identifying mechanical flaws when they possessed deep knowledge of the source domain.
- Students interpreted unfamiliar cultural references and excessive explanation as negative social signals regarding their assumed knowledge and competence.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
What is the application?
Who is the user?
Who age?
Study design
