Date
AI Overview
Why it matters
Automated subjective question generation and answer evaluation can help teachers assess student work and help students self-assess their higher-order understanding, but existing systems predominantly focus on objective question generation.
Study at a glance
- Question: Can fine-tuned open-source language models effectively generate subjective questions and evaluate student answers?
- Setting and data: Synthetic datasets of 6,978 question generation and 4,523 answer evaluation samples created with GPT-4 across various subjects.
- Method: Fine-tuning language models with QLoRA and instruct-tuning, followed by model benchmark evaluation using GPT-4 rankings.
- AI tools: GPT-4, GPT-3.5, Mistral 7B, LLaMA 7B, Falcon 7B
Key findings
- Mistral 7B performed best in subjective question generation, ranking first across 65.78% of test prompts.
- GPT-3.5 ranked first in answer evaluation for 58.22% of test cases, followed by Mistral 7B which ranked first 40.44% of the time.
- Llama-2 7B and Falcon 7B performed poorly across both tasks, predominantly ranking in the third and fourth positions respectively.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
Natural Language Processing (NLP) is one of the most revolutionary technologies today. It uses artificial intelligence to understand human text and spoken words. It is used for text summarization, grammar checking, sentiment analysis, and advanced chatbots and has many more potential use cases. Furthermore, it has also made its mark on the education sector. Much research and advancements have already been conducted on objective question generation; however, automated subjective question generation and answer evaluation are still in progress. An automated system to generate subjective questions and evaluate the answers can help teachers assess student work and enhance the student's learning experience by allowing them to self-assess their understanding after reading an article or a chapter of a book. This research aims to improve current NLP models or make a novel one for automated subjective question generation and answer evaluation from text input.
Research Categories
What is the application?
Who age?
Why use AI?
Study design
AI tools
Publication Source
Publisher
arXiv
