Search and Filter

Farsimcqgen: A Persian Multiple-Choice Question Generation Framework

Date

AI Overview

Why it matters

Manually creating multiple-choice questions is time-consuming and requires significant expertise. Generating high-quality questions and plausible wrong choices remains a major challenge in low-resource languages such as Persian.

Study at a glance
  • Question: Can an automated system effectively generate high-quality Persian multiple-choice questions and plausible wrong answer choices from text?
  • Setting and data: Wikipedia passages and 10,289 generated Persian multiple-choice questions; 200 questions evaluated by two native Persian-speaking undergraduate students.
  • Method: Pipeline system development, language model evaluation across six large language models, and human rating.
  • AI tools: FarsiMCQGen, Qwen2.5-14B-Instruct, Qwen2.5-7B-Instruct, Meta-Llama-3.1-8B-Instruct, Dorna2-Llama3.1-8B-Instruct, Mistral-7B-Instruct-v0.3
Key findings
  • In human evaluation of 200 questions, 97.5% of questions and options were judged logically valid, and 94.5% of wrong choices were effectively distractive.
  • Among six tested models, Qwen2.5-14B-Instruct achieved the highest accuracy, scoring 49.2% hard accuracy and 48.8% soft accuracy.
  • Evaluating models under 16-bit versus 8-bit quantization had minimal impact on both accuracy and confidence across all tests.

Summary drafted with AI from the full paper and may contain errors. See the full paper for details.

Multiple-choice questions (MCQs) are commonly used in educational testing, as they offer an efficient means of evaluating learners' knowledge. However, generating high-quality MCQs, particularly in low-resource languages such as Persian, remains a significant challenge. This paper introduces FarsiMCQGen, an innovative approach for generating Persian-language MCQs. Our methodology combines candidate generation, filtering, and ranking techniques to build a model that generates answer choices resembling those in real MCQs. We leverage advanced methods, including Transformers and knowledge graphs, integrated with rule-based approaches to craft credible distractors that challenge test-takers. Our work is based on data from Wikipedia, which includes general knowledge questions. Furthermore, this study introduces a novel Persian MCQ dataset comprising 10,289 questions. This dataset is evaluated by different state-of-the-art large language models (LLMs). Our results demonstrate the effectiveness of our model and the quality of the generated dataset, which has the potential to inspire further research on MCQs.

Publication Source

Publisher
arXiv