Date
Publisher
arXiv
Academic advising is critical to student success in higher education, yet high student-to-advisor ratios limit advisors' capacity to provide timely support, particularly during peak periods. Recent advances in Large Language Models (LLMs) present opportunities to enhance the advising process. We present AdvisingWise, a multi-agent system that automates time-consuming tasks, such as information retrieval and response drafting, while preserving human oversight. AdvisingWise leverages authoritative institutional resources and adaptively prompts students about their academic backgrounds to generate reliable, personalized responses. All system responses undergo human advisor validation before delivery to students. We evaluate AdvisingWise through a mixed-methods approach: (1) expert evaluation on responses of 20 sample queries, (2) LLM-as-a-judge evaluation of the information retrieval strategy, and (3) a user study with 8 academic advisors to assess the system's practical utility. Our evaluation shows that AdvisingWise produces accurate, personalized responses. Advisors reported increasingly positive perceptions after using AdvisingWise, as their initial concerns about reliability and personalization diminished. We conclude by discussing the implications of human-AI synergy on the practice of academic advising.
Study at a glance
- Question: Can a human-in-the-loop multi-agent system deliver accurate, personalized academic advising while reducing advisor workload?
- Setting and data: Computer science graduate program at a private U.S. university; 100 synthesized advising queries and 8 academic advisors.
- Method: Mixed methods combining expert evaluation, LLM-as-a-judge comparison against a RAG baseline, and a semi-structured user study.
Why it matters
High student-to-advisor ratios limit advisors' capacity to provide timely guidance. While generative language models could automate routine advising tasks, they risk factual hallucinations and lack human oversight, personalization, and emotional connection.
Key findings
- In expert evaluation of 19 advising queries, 84.2 percent achieved high accuracy scores, with lower scores resulting primarily from conservative refusals when uncertain.
- The ReAct-style multi-agent retrieval system outperformed a standard retrieval-augmented baseline with a 3:1 preference ratio in LLM-as-a-judge pairwise comparisons.
- Advisors welcomed delegating straightforward factual questions, but remained reluctant to delegate emotionally sensitive situations like failing a course despite accurate AI responses.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
What is the application?
Who age?
Why use AI?
