Search and Filter

Zero-Shot Speech Llms For Multi-Aspect Evaluation Of L2 Speech: Challenges And Opportunities

Date

AI Overview

Why it matters

Accurate L2 English pronunciation assessment is crucial for personalized feedback and fair evaluation of progress, yet automated scoring remains difficult for sentence-level fluency, prosody, and completeness.

Study at a glance
  • Question: Can an instruction-tuned speech language model assess multiple aspects of non-native speech pronunciation in a zero-shot setting?
  • Setting and data: Speechocean762 dataset, 5,000 English utterances from 250 Mandarin-speaking children and adults.
  • Method: Zero-shot model benchmark evaluation comparing AI-generated rubric scores with expert human ratings.
  • AI tools: Qwen2-Audio-7B-Instruct
Key findings
  • Within a relaxed margin of plus or minus 2 points, model ratings matched human scores in over 85 percent of cases for accuracy, fluency, and prosody.
  • Exact match rates between model predictions and human ratings were relatively low, reaching 23.3 percent for accuracy, 26.1 percent for fluency, and 25.4 percent for prosody.
  • The model systematically overestimated lower-quality speech, failing to assign any score of 6 or below to utterances where human accuracy scores were 6 or lower.

Summary drafted with AI from the full paper and may contain errors. See the full paper for details.

An accurate assessment of L2 English pronunciation is crucial for language learning, as it provides personalized feedback and ensures a fair evaluation of individual progress. However, automated scoring remains challenging due to the complexity of sentence-level fluency, prosody, and completeness. This paper evaluates the zero-shot performance of Qwen2-Audio-7B-Instruct, an instruction-tuned speech-LLM, on 5,000 Speechocean762 utterances. The model generates rubric-aligned scores for accuracy, fluency, prosody, and completeness, showing strong agreement with human ratings within +-2 tolerance, especially for high-quality speech. However, it tends to overpredict low-quality speech scores and lacks precision in error detection. These findings demonstrate the strong potential of speech LLMs in scalable pronunciation assessment and suggest future improvements through enhanced prompting, calibration, and phonetic integration to advance Computer-Assisted Pronunciation Training.

Research Categories

What is the application?
Who is the user?
Who age?
AI tools
Datasets

Publication Source

Publisher
arXiv