Date
Publisher
arXiv
The growing integration of Artificial Intelligence (AI) into education has
intensified the need for transparency and interpretability. While hackathons
have long served as agile environments for rapid AI prototyping, few have
directly addressed eXplainable AI (XAI) in real-world educational contexts.
This paper presents a comprehensive analysis of the XAI Challenge 2025, a
hackathon-style competition jointly organized by Ho Chi Minh City University of
Technology (HCMUT) and the International Workshop on Trustworthiness and
Reliability in Neurosymbolic AI (TRNS-AI), held as part of the International
Joint Conference on Neural Networks (IJCNN 2025). The challenge tasked
participants with building Question-Answering (QA) systems capable of answering
student queries about university policies while generating clear, logic-based
natural language explanations. To promote transparency and trustworthiness,
solutions were required to use lightweight Large Language Models (LLMs) or
hybrid LLM-symbolic systems. A high-quality dataset was provided, constructed
via logic-based templates with Z3 validation and refined through expert student
review to ensure alignment with real-world academic scenarios. We describe the
challenge's motivation, structure, dataset construction, and evaluation
protocol. Situating the competition within the broader evolution of AI
hackathons, we argue that it represents a novel effort to bridge LLMs and
symbolic reasoning in service of explainability. Our findings offer actionable
insights for future XAI-centered educational systems and competitive research
initiatives.
Study at a glance
- Question: How effectively can hybrid language models and symbolic reasoning answer university policy questions with verifiable explanations?
- Setting and data: International hackathon competition with 107 participants across 28 teams, using a newly generated dataset of 531 university policy records.
- Method: Benchmark competition evaluating models on answer accuracy, premise relevance, and human-scored explanation clarity.
Why it matters
Conventional language models in education often provide concise answers with limited explanatory depth, making it difficult to verify reasoning. Combining language models with symbolic logic can improve transparency and trustworthiness for high-stakes university policies.
Key findings
- Finalist selection scores remained modest and barely surpassed 20, representing under 50% of the maximum possible score across 50 test cases.
- A multi-agent architecture combining lightweight open-source models with the Z3 theorem prover for symbolic reasoning achieved first place.
- Two-phase evaluation enabled rapid iterative improvements, with one team increasing its score by nearly 86% between phases.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
What is the application?
Who is the user?
Who age?
Why use AI?
Study design
