Date
Publisher
arXiv
This paper addresses the critical need for scalable and high-quality
educational assessment tools within the Malaysian education system. It
highlights the potential of Generative AI (GenAI) while acknowledging the
significant challenges of ensuring factual accuracy and curriculum alignment,
especially for low-resource languages like Bahasa Melayu. This research
introduces and compares four incremental pipelines for generating Form 1
Mathematics multiple-choice questions (MCQs) in Bahasa Melayu using OpenAI's
GPT-4o. The methods range from non-grounded prompting (structured and basic) to
Retrieval-Augmented Generation (RAG) approaches (one using the LangChain
framework, one implemented manually). The system is grounded in official
curriculum documents, including teacher-prepared notes and the yearly teaching
plan (RPT). A dual-pronged automated evaluation framework is employed to assess
the generated questions. Curriculum alignment is measured using Semantic
Textual Similarity (STS) against the RPT, while contextual validity is verified
through a novel RAG-based Question-Answering (RAG-QA) method. The results
demonstrate that RAG-based pipelines significantly outperform non-grounded
prompting methods, producing questions with higher curriculum alignment and
factual validity. The study further analyzes the trade-offs between the ease of
implementation of framework-based RAG and the fine-grained control offered by a
manual pipeline. This work presents a validated methodology for generating
curriculum-specific educational content in a low-resource language, introduces
a symbiotic RAG-QA evaluation technique, and provides actionable insights for
the development and deployment of practical EdTech solutions in Malaysia and
similar regions.
Study at a glance
- Question: Can retrieval-augmented generation produce more accurate, curriculum-aligned secondary mathematics questions in Bahasa Melayu than ungrounded prompting?
- Setting and data: Malaysia; Malaysian secondary Form 1 Mathematics Chapter 1 curriculum documents and 400 generated multiple-choice questions.
- Method: Automated pipeline comparison evaluating generated questions using semantic textual similarity and retrieval-based question-answering.
Why it matters
Malaysian educators face a heavy burden creating curriculum-aligned assessments at scale. Unconstrained AI models risk generating factually inaccurate questions, especially in lower-resource languages like Bahasa Melayu where training data is limited.
Key findings
- Retrieval-augmented generation pipelines produced substantially higher curriculum alignment scores (0.86 and 0.89) than non-grounded prompting methods (0.58 and 0.55).
- Questions created using retrieval grounding achieved 92% and 96% validity rates when tested against curriculum objectives, compared to 15% and 12% for ungrounded prompts.
- All four generative approaches demonstrated high linguistic fluency in Bahasa Melayu, but grounded methods consistently ensured content was specifically tailored to syllabus examples.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
What is the application?
Who is the user?
Who age?
Why use AI?
Study design
