Date
Publisher
arXiv
This study compared the classification performance of Gemini Pro and GPT-4V
in educational settings. Employing visual question answering (VQA) techniques,
the study examined both models' abilities to read text-based rubrics and then
automatically score student-drawn models in science education. We employed both
quantitative and qualitative analyses using a dataset derived from
student-drawn scientific models and employing NERIF (Notation-Enhanced Rubrics
for Image Feedback) prompting methods. The findings reveal that GPT-4V
significantly outperforms Gemini Pro in terms of scoring accuracy and Quadratic
Weighted Kappa. The qualitative analysis reveals that the differences may be
due to the models' ability to process fine-grained texts in images and overall
image classification performance. Even adapting the NERIF approach by further
de-sizing the input images, Gemini Pro seems not able to perform as well as
GPT-4V. The findings suggest GPT-4V's superior capability in handling complex
multimodal educational tasks. The study concludes that while both models
represent advancements in AI, GPT-4V's higher performance makes it a more
suitable tool for educational applications involving multimodal data
interpretation.
Study at a glance
- Question: Which AI model is better at automatically scoring student science drawings using written rubrics?
- Setting and data: A dataset of student-drawn scientific models in science education.
- Method: Quantitative and qualitative comparison testing two AI models' accuracy in scoring student drawings using written rubrics.
Why it matters
Using AI to automatically score student-drawn models in science education requires systems that can read text-based rubrics and interpret multimodal data. Evaluating model performance helps determine which tools are suitable for these educational tasks.
Key findings
- In science education, GPT-4V scored student-drawn models using text rubrics significantly more accurately than Gemini Pro.
- Performance differences may be due to variations in each tool's ability to read detailed text within images and accurately classify visual content.
- Even when modifying image sizes to assist evaluation, Gemini Pro seemed unable to perform as well as GPT-4V.
Summary drafted with AI from the paper’s abstract and may contain errors. See the full paper for details.
What is the application?
Who is the user?
Why use AI?
Study design
