ETAK 2026 / INTERNATIONAL LANGUAGE WEEK
Probing Internal Layers
of AI Speech Encoders
for L2 Korean Pronunciation Assessment
Tae-Jin Yoonᵃ
SungSook Kimᵇ, Hee-Kyung Leeᵇ, and Seok-Chae Rheeᵇ
ᵃ Department of English Language and Literature,
Sungshin Women’s University, Seoul, Republic of Korea
ᵇ Korean Language Institute, Yonsei University, Seoul, Republic of Korea
Friday, 4 December 2026 · Sungshin Women’s University
Open slides
01 / 30
Why look inside a speech encoder?
Recognizing words and assessing pronunciation ask different questions.
Assessment
need
Recognizing the intended words does not establish how
well they were pronounced.
Internal layers
Each layer produces a different numerical representation
of the same speech.
Research
question
Which layer provides the most useful information for
predicting human ratings?
02 / 30
Three questions guide the study
Prediction performance · layer choice · assessment interpretation
Prediction
How closely can speech-model features predict human
pronunciation ratings?
Layer choice
Does the most useful layer differ across models and rating
targets?
Assessment
use
How should teachers interpret the scores when discussing
feedback?
03 / 30
What do accuracy and fluency mean here?
Two human rating targets · each on a 0–4 scale
Pronunciation accuracy
How accurately are the target
sounds realized?
Prosodic fluency
How naturally are phrasing,
intonation and pace managed?
04 / 30
What do the predicted scores describe?
One Korean paragraph · one recording · two overall scores
Task
The learner reads a displayed Korean passage aloud.
Prediction
The model estimates accuracy and fluency for the
complete recording.
Feedback
A teacher listens before offering advice about a specific
word or phrase.
05 / 30
How were the recordings collected?
Original assessment records · before research filtering
Stage
Collection procedure
Assessment
First 2026 online Korean speaking test
Institutions
Yonsei University: 700 learners
Hankuk University of Foreign Studies: 100 learners
Test tasks
One reading item (PR1) + six speaking items
Reading task
Read a Korean paragraph displayed on screen
Recording
Browser recording → server-side WAV conversion
Audio content
Learner responses only; no system prompts
800 learners from two institutions.
06 / 30
Corpus audit and a fixed evaluation split
755 recordings · 41 passages · 8.33 hours
800 responses − 43 without consent − 2 invalid audio = 755
07 / 30
Human scores define the prediction targets
Two raters per response · 0–4 scale · development distribution
Mid-scale ratings dominate; extreme scores are less frequent.
08 / 30
Deep learning: what is a layer?
From inputs to internal representations and outputs
Vision example; speech encoders use Transformer blocks.
Deep learning
Learns patterns through
multiple processing layers.
A layer
One processing stage; its
output is a representation.
This study
Which layer best predicts
human ratings?
Image source: https://pythongeeks.org/wp-content/uploads/2022/02/use-case-of-deep-learning.webp
09 / 30
Seven encoders, different training objectives
Speech representations from 12–32 encoder blocks
Encoder family
Comparison
Blocks
HuBERT
English / Korean
12 / 12
XLS-R
Base / Korean ASR*
24 / 24
Whisper
Small / Medium
12 / 24
Qwen2-Audio
Audio encoder only
32
* Korean ASR checkpoint: w11wo/wav2vec2-xls-r-300m-korean
HuBERT / XLS-R: SSL; Whisper / Qwen: other training objectives.
10 / 30
Extracting fixed-length speech representations
Frozen encoder · 16 kHz mono audio · 10-second chunks
Waveform
16 kHz mono
→
Frozen encoder
Weights fixed
→
Valid frame vectors
h₁, …, hₜ (d features)
z = [mean(h); standard deviation(h)]
One response → one 2d-dimensional vector per layer
Pool across valid frames; exclude padding from the statistics.
11 / 30
Probing a layer with Ridge regression
Measure linearly accessible rating information; keep the encoder frozen.
Predicted score = b + wᵀz
Fit squared error + α × squared coefficient norm
Standardize features using training data only.
Tune α from 0.1 to 10,000 in inner validation.
Encoder weights fixed · Ridge coefficients fitted
Fit separate regressions for accuracy and prosodic fluency.
12 / 30
Evaluation Metric
MAE = mean absolute error. Lower is better.
Human rating
Model estimate
Absolute error
2.0
2.5
0.5
3.0
2.5
0.5
1.0
1.0
0.0
Average error = (0.5 + 0.5 + 0.0) ÷ 3 = 0.33 points
13 / 30
B0: predict the training median
A reference score without listening to the recording
Training ratings
1 · 2 · 2 · 3 · 4
Median = 2
Every new response
Predicted score = 2
No audio features
Illustrative ratings · calculate a separate median for each target.
B0 asks: does using the audio improve on a constant score?
14 / 30
B1: duration and low-energy features
Five measurements from the waveform → Ridge regression
01
Recording duration
02
Low-energy proportion
03
Count of low-energy runs
04
Mean run length
05
Maximum run length
Low-energy rule
RMS at least 35 dB
below the recording
maximum
Count runs ≥ 0.20 s
Low-energy intervals are proxies—not verified speech pauses.
15 / 30
B2: conventional acoustic features
Hand-defined signal measurements → 93 values per recording
5
B1 features
Duration +
low-energy features
80
MFCCs + deltas
20 of each
Mean + SD of each
8
Other acoustics
RMS, ZCR,
centroid, rolloff
Mean + SD of each
25 ms windows · 10 ms steps · 512-point FFT
Training-only scaling → Ridge → two rating predictions
5 + 80 + 8 = 93 features per recording
16 / 30
Locked test: all encoders outperform B2
151 responses · un-clipped MAE on the 0–4 scale
B2 = conventional acoustic features; lower MAE is better.
17 / 30
Fluency: a stronger result in this task
Held-out test · 151 responses · MAE on the 0–4 scale
Acoustic baseline
0.673
HuBERT English
0.459
XLS-R
0.432
Qwen2-Audio
0.412
Whisper Medium
0.393
XLS-R Korean ASR
0.391
0.0
0.2
0.4
0.6
0.8
18 / 30
Relevant training matters
Korean ASR adaptation improved XLS-R in this comparison.
Accuracy
0.527
→
0.478
Fluency
0.432
→
0.391
XLS-R
Korean ASR version
Test MAE fell by about 0.05 and 0.04 points.
19 / 30
Accuracy information varies across layers
Fixed-layer development OOF MAE · alpha selected within folds
SSL models often favor middle-to-upper layers.
20 / 30
Fluency shows a similar family contrast
Fixed-layer development OOF MAE · stars mark the lowest MAE
Whisper and Qwen tend to favor later encoder layers.
21 / 30
Why do useful layers differ across models?
Training objectives shape representations · a plausible interpretation
HuBERT / XLS-R
ORIGINAL TRAINING
Masked-unit prediction or contrastive
learning
POSSIBLE EXPLANATION
Sound detail and context may be
easier to read in middle-to-upper
layers.
Whisper / Qwen2-Audio
ORIGINAL TRAINING
Audio representations support text
prediction
POSSIBLE EXPLANATION
Later layers may expose contextual
speech cues that help rating
prediction.
Choose layers by development performance for each task.
22 / 30
One sentence,
four layers
XLS-R Korean ASR
Layer 16
Accuracy-selected
Layer 19
Fluency-selected
Attention patterns
do not establish
causal importance.
23 / 30
Human agreement after re-rating
755 responses · individual final ratings · rater pairs vary
Exact score agreement: 44% accuracy · 45% fluency
24 / 30
Best model vs. human ratings
XLS-R Korean ASR · held-out test (n = 151) · final human ratings
Human targets include re-rating and adjudication.
25 / 30
Average error hides uneven performance
Test predictions tended toward the middle of the scale.
Lower ratings
+0.77
Overpredicted · band 1
Higher ratings
−0.83
Underpredicted · band 4
Accuracy mean error · averaged across seven encoders
Rounded score bands · n = 15 and n = 22
26 / 30
Discuss scores, then choose a learning goal
Proposed teacher–learner workflow · not tested in this study
Learner
Record a
reading
›
AI
Estimate two
ratings
›
Teacher
Listen and
verify
›
Together
Choose a
practice goal
Let learners question the score, practise, and reflect.
27 / 30
From evidence to feedback
Teacher-guided example · listen before correcting
First sentence: 1.37–7.31 s
“Last week, I wore hanbok and walked around an old palace.”
Accuracy: check the word
Fluency: natural phrasing
Reference → separate ASR transcript
Teacher listening: appropriate phrasing
“Let’s listen again. If unclear,
practise the word, then the phrase.”
“This pause sounds natural. Keep
this phrasing; there is no need to
remove the pause.”
Full recording · XLS-R base · development cross-validation
Predicted accuracy: 2.22 / 4 Fluency: 2.20 / 4
28 / 30
What probing internal layers tells us
Select representations for the task; interpret scores with teacher judgment.
Prediction
Internal speech representations improved rating prediction
over the acoustic baseline.
Layer choice
The most useful layer depends on the model and target;
the final layer is not always best.
Assessment use
Whole-recording scores can inform teacher review;
specific feedback requires listening.
29 / 30
Acknowledgments
Supported by the Seoul National University R&DB Foundation
Consortium through the Ministry of Culture, Sports and Tourism’s 2025
R&D Designated Call.
PROJECT
AI-assisted Korean speaking diagnosis for foreign learners
June 2025–June 2028
30 / 30
←
→
Notes (N)
Overview (O)
Full screen (F)
Arrow keys to navigate