Date
AI Overview
Why it matters
General-purpose benchmarks often fail to predict reasoning performance in high-stakes educational environments. Most existing benchmarks overlook the integrated bilingual reasoning and procedural rigor required in multilingual school systems.
Study at a glance
- Question: How accurately do large language models solve authentic Indian high-stakes exam questions across subjects, languages, and prompting strategies?
- Setting and data: India; 1,304 multiple-choice exam questions across STEM and humanities from UPSC, JEE, and NEET in English and Hindi.
- Method: Model or benchmark evaluation using Zero-Shot, Few-Shot, and Chain-of-Thought prompting across three independent runs.
- AI tools: Gemini 2.0 Flash, GPT-4, Claude, LLaMA 3-70B
Key findings
- Chain-of-thought prompting increased accuracy by an average of 11.59 percentage points across subjects and languages compared to zero-shot prompting.
- Models showed marked multilingual performance drops in Hindi relative to English, with average cross-language accuracy gaps reaching 34.77 percentage points in lower-performing contexts.
- Gemini 2.0 Flash consistently outperformed LLaMA 3-70B across subjects, with cross-model gaps reaching up to 45.35 percentage points on zero-shot Hindi exams.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
The rapid advancement of large language models (LLMs) necessitates evaluation frameworks that reflect real-world academic rigor and multilingual complexity. This paper introduces IndicEval, a scalable benchmarking platform designed to assess LLM performance using authentic high-stakes examination questions from UPSC, JEE, and NEET across STEM and humanities domains in both English and Hindi. Unlike synthetic benchmarks, IndicEval grounds evaluation in real examination standards, enabling realistic measurement of reasoning, domain knowledge, and bilingual adaptability. The framework automates assessment using Zero-Shot, Few-Shot, and Chain-of-Thought (CoT) prompting strategies and supports modular integration of new models and languages. Experiments conducted on Gemini 2.0 Flash, GPT-4, Claude, and LLaMA 3-70B reveal three major findings. First, CoT prompting consistently improves reasoning accuracy, with substantial gains across subjects and languages. Second, significant cross-model performance disparities persist, particularly in high-complexity examinations. Third, multilingual degradation remains a critical challenge, with marked accuracy drops in Hindi compared to English, especially under Zero-Shot conditions. These results highlight persistent gaps in bilingual reasoning and domain transfer. Overall, IndicEval provides a practice-oriented, extensible foundation for rigorous, equitable evaluation of LLMs in multilingual educational settings and offers actionable insights for improving reasoning robustness and language adaptability.
Research Categories
Who age?
Study design
Methods
AI tools
Publication Source
Publisher
arXiv
