Date
Publisher
arXiv
The current study describes a cost-effective method for adapting large language models (LLMs) for academic advising with study-abroad contexts in mind and for application in low-resource methods for acculturation. With the Mistral-7B-Instruct model applied with a Low-Rank Adaptation (LoRA) method and a 4-bit quantization method, the model underwent training in two distinct stages related to this study's purpose to enhance domain specificity while maintaining computational efficiency. In Phase 1, the model was conditioned with a synthetic dataset via the Gemini Pro API, and in Phase 2, it was trained with manually curated datasets from the StudyAbroadGPT project to achieve enhanced, contextualized responses. Technical innovations entailed memory-efficient quantization, parameter-efficient adaptation, and continuous training analytics via Weights & Biases. After training, this study demonstrated a reduction in training loss by 52.7%, 92% accuracy in domain-specific recommendations, achieved 95% markdown-based formatting support, and a median run-rate of 100 samples per second on off-the-shelf GPU equipment. These findings support the effective application of instruction-tuned LLMs within educational advisers, especially in low-resource institutional scenarios. Limitations included decreased generalizability and the application of a synthetically generated dataset, but this framework is scalable for adding new multilingual-augmented and real-time academic advising processes. Future directions may include plans for the integration of retrieval-augmented generation, applying dynamic quantization routines, and connecting to real-time academic databases to increase adaptability and accuracy.
Study at a glance
- Question: Can an open-weight language model be fine-tuned across free-tier GPUs to provide reliable academic advising?
- Setting and data: Synthetic dataset of 2,676 multi-turn student-advisor conversations generated via the Gemini 1.0 Pro API across eight advising topics.
- Method: Model fine-tuning and evaluation using automated metrics, blind LLM-as-judge scoring, and a source-verified factuality audit.
Why it matters
Scaling accessible study-abroad guidance with large language models is attractive for resource-constrained advising settings, but standard models lack domain grounding and can produce fluent but incorrect advice on policy-sensitive topics.
Key findings
- Fine-tuning raised reference-based similarity to the synthetic training distribution, but an LLM judge preferred the original base model on 46% of prompts versus 18% for the fine-tuned model.
- A source-verified factuality audit identified four confident factual errors produced by the fine-tuned model on policy-sensitive topics, compared to zero errors from the base model.
- Auditing the synthetic training data revealed that 28% to 40% of sampled answers contained verifiable factual errors, directly matching the failure modes inherited by the fine-tuned model.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
What is the application?
Who is the user?
Who age?
Why use AI?
Study design
