Date
Publisher
arXiv
Artificial intelligence (AI) teaching assistants powered by large language models (LLMs) offer scalable educational support but often provide limited personalization. This study presents a prompt-engineering-based framework for personalizing general-purpose LLM/RAG-based AI teaching assistants such as Jill Watson across academic disciplines and courses. The framework adapts responses using six learner-specific dimensions: self-assessment, abstraction preference, verbosity preference, perceptual orientation, information processing style, and level of understanding, yielding 96 distinct learner profiles. Student queries are additionally analyzed using Bloom's Taxonomy to estimate cognitive complexity at the interaction level. Learner attributes and cognitive assessments are encoded in structured prompts that condition the LLM without requiring model retraining. The framework is evaluated through experiments using NLP metrics and a human study with five participants. Results show perceived differences in response style and structure across personalization conditions, with statistical analyses identifying learner attributes associated with measurable response changes. These findings provide preliminary evidence that prompt-based personalization can support adaptive behavior in LLM-powered educational agents.
Study at a glance
- Question: Can prompt engineering tailor an AI teaching assistant's responses to individual student preferences and question complexity?
- Setting and data: 30 student questions from a Spring 2023 college knowledge-based AI course; 2,910 generated responses; 5 human evaluators.
- Method: Automated text analysis, regression analysis (OLS, mixed-effects models), and a human evaluation study.
Why it matters
AI teaching assistants provide scalable instructional support, but they often offer limited personalization. Questions remain about which learner traits should guide personalization, how to implement it, and whether adaptations yield meaningfully different interactions.
Key findings
- Prompt-based configurations produced high semantic similarity but lower lexical similarity across 2,910 responses, showing that answers remained course-grounded while varying in phrasing and structure.
- Higher learner self-assessment, verbosity preferences, reflective processing, and technical abstraction preferences were systematically associated with greater linguistic complexity in generated responses.
- Five human evaluators perceived clear differences in response quality, complexity, abstraction, and processing style that generally aligned with the intended learner preferences.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
What is the application?
Who is the user?
Who age?
Why use AI?
