Pustak Ai: Curriculum-Aligned And Interactive Textbooks Using Large Language Models

Shivam Sharma, Riya Naik, Tejas Gawas, Heramb Patil, Kunal Korgaonkar

Date

11/2025

AI Overview

Why it matters

Adapting large language models to non-Western curricula presents challenges in accuracy and pedagogical relevance. Existing educational datasets largely focus on general content rather than specific regional syllabi.

Study at a glance

Question: How accurately and efficiently do different language models answer curriculum-specific questions from Indian textbooks under various prompting methods?
Setting and data: India, NCERT English and Science textbooks for grades 6 to 8, 83 chapter documents and 739 curated question-answer pairs.
Method: Benchmark evaluation comparing multiple language models and prompting strategies against ground truth answers with and without textbook context.
AI tools: PustakAI, Gemma3:1b, Llama3.2:3b, Nemotron-mini:4b, Llama-4-Scout-17B-16E-Instruct, deepseek-r1-distill-llama-70b

Key findings

Providing textbook context substantially improved model accuracy, as Llama-4-Scout-17B's overall F1 score dropped to 0.395 without context compared to 0.894 on the general SQuAD benchmark.
Meta one-shot prompting delivered the most consistent performance, yielding the highest F1 scores across models in both English and Science subsets.
Chain-of-Thought prompting reduced answer faithfulness, dropping to 0.532 for Llama-4-Scout-17B, which suggests detailed step-by-step reasoning caused models to hallucinate details beyond the textbook context.

Summary drafted with AI from the full paper and may contain errors. See the full paper for details.

Large Language Models (LLMs) have demonstrated remarkable capabilities in understanding and generating human-like content. This has revolutionized various sectors such as healthcare, software development, and education. In education, LLMs offer potential for personalized and interactive learning experiences, especially in regions with limited teaching resources. However, adapting these models effectively to curriculum-specific content, such as the National Council of Educational Research and Training (NCERT) syllabus in India, presents unique challenges in terms of accuracy, alignment, and pedagogical relevance. In this paper, we present the framework "PustakAI"\footnote{Pustak means `book' in many Indian languages.} for the design and evaluation of a novel question-answering dataset "NCERT-QA" aligned with the NCERT curriculum for English and Science subjects of grades 6 to 8. We classify the curated QA pairs as Factoid, Inferential, and Others (evaluative and reasoning). We evaluate the dataset with various prompting techniques, such as meta-prompt, few-shot, and CoT-style prompting, using diverse evaluation metrics to understand which approach aligns more efficiently with the structure and demands of the curriculum. Along with the usability of the dataset, we analyze the strengths and limitations of current open-source LLMs (Gemma3:1b, Llama3.2:3b, and Nemotron-mini:4b) and high-end LLMs (Llama-4-Scout-17B and Deepseek-r1-70B) as AI-based learning tools in formal education systems.