Date
Publisher
arXiv
This study explores ChatGPT's capabilities, stability, and risks in simulating psychological counseling sessions in a school counseling context. Using scripted role-plays between a human counselor and an AI client, we examine how a large language model performs core counseling skills such as empathy, reflection, summarizing, and asking open-ended questions, as well as its ability to maintain therapeutic communication over time. We focus on how consistently ChatGPT can behave like a "virtual client" for school counselors in training, and how its responses might support or disrupt counselor skill development, supervision, and practice. At the same time, we analyze potential risks, including inaccurate or unsafe suggestions, over-compliance with counselor prompts, and the illusion of a competent therapist where no real professional judgment exists. The findings suggest that ChatGPT can serve as a low-cost, always-available training tool for practicing counseling techniques and interviewing skills in education and mental health settings, but it should not be viewed as a replacement for a human therapist or school counselor. We propose practical guidelines for educators, supervisors, and researchers who wish to use ChatGPT or similar LLM-based conversational agents in counseling training, highlighting how to leverage its potential while managing ethical, pedagogical, and psychological risks.
Study at a glance
- Question: How capable and stable is ChatGPT-4 in providing supportive psychological counseling responses to student concerns?
- Setting and data: Simulated school counseling setting; 80 adolescent counseling queries from the ChatCounselor dataset translated into English, accessed July 2024.
- Method: Text-based simulation evaluating repeated outputs using natural language processing tools, intraclass correlation coefficients, Fleiss' kappa, chi-square tests, and ANOVA.
Why it matters
Gaining access to mental healthcare remains a global challenge due to provider shortages, costs, and stigma. While conversational AI could expand support, key implementation details and risk factors like response instability are not yet fully understood.
Key findings
- ChatGPT-4 demonstrated high warmth in 97.5% of responses, empathy in 94.2%, and an overwhelmingly positive tone with an average compound sentiment score of 0.93.
- Repeated runs showed moderate consistency across responses for empathy (Fleiss' kappa = 0.59) and overall sentiment (intraclass correlation coefficient = 0.62).
- A small fraction of responses (2.5%) exhibited uncertainty or misapprehension, and longer student questions moderately correlated with longer AI replies (r = 0.60).
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
What is the application?
Who is the user?
Who age?
Why use AI?
