Date
Publisher
arXiv
Many students struggle with math word problems (MWPs), often finding it
difficult to identify key information and select the appropriate mathematical
operations. Schema-based instruction (SBI) is an evidence-based strategy that
helps students categorize problems based on their structure, improving
problem-solving accuracy. Building on this, we propose a Schema-Based
Instruction Retrieval-Augmented Generation (SBI-RAG) framework that
incorporates a large language model (LLM). Our approach emphasizes step-by-step
reasoning by leveraging schemas to guide solution generation. We evaluate its
performance on the GSM8K dataset, comparing it with GPT-4 and GPT-3.5 Turbo,
and introduce a "reasoning score" metric to assess solution quality. Our
findings suggest that SBI-RAG enhances reasoning clarity and facilitates a more
structured problem-solving process potentially providing educational benefits
for students.
Study at a glance
- Question: Can schema-based instruction and retrieval-augmented generation improve large language models' step-by-step reasoning in math word problems?
- Setting and data: 360 schema-based instruction math word problems and the 8,500 grade-school math word problems from the GSM8K dataset.
- Method: System development and model evaluation using a DistilBERT schema classifier, retrieval-augmented Llama 3.1 generation, and LLM-as-a-judge comparisons.
Why it matters
Many students struggle with math word problems, failing to identify key information or select appropriate operations despite understanding underlying mathematical concepts. Poor problem-solving skills in math word problems can lead to academic challenges and failure in school.
Key findings
- The schema-based system achieved a higher reasoning score (0.588) on GSM8K problems compared to GPT-4 (0.491) and GPT-3.5 Turbo (0.290).
- The DistilBERT schema classifier identified problem types across six additive and multiplicative sub-categories with 97 percent overall accuracy.
- Using an LLM judge, schema-guided responses consistently outperformed GPT-4 and GPT-3.5 Turbo in clarity, logical progression, completeness, and educational effectiveness.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
What is the application?
Who is the user?
Who age?
Why use AI?
Study design
