You will be provided with a reference and some statements. Please determine whether each statement is 'supported', 'unsupported', or 'unknown' with respect to the reference. Please note:
First, assess whether the reference contains any valid content. If the reference contains no valid information, such as a 'page not found' message, then all statements should be considered 'unknown'.
If the reference is valid, for a given statement: if the facts or data it contains can be found entirely or partially within the reference, it is considered 'supported' (data accepts rounding); if all facts and data in the statement cannot be found in the reference, it is considered 'unsupported'.

You should return the result in a JSON list format, where each item in the list contains the statement's index and the judgment result, for example:
[
    {
        "idx": 1,
        "result": "supported"
    },
    {
        "idx": 2,
        "result": "unsupported"
    }
]

Below are the reference and statements:
<reference>
This CVPR Findings paper is the Open Access version, provided by the Computer Vision
Foundation. Except for this watermark, it is identical to the accepted version;
the final published version of the proceedings is available on IEEE Xplore.

FineGrade: A Rule-Consistent Scoring Framework for Fine-Grained Action
Quality Assessment
Yicong Li Howard Leung
City University of Hong Kong
{yicongli4-c@my.cityu.edu.hk, howard@cityu.edu.hk}

%- ##$$ "

%$

"
#

 


  
 




&
"






$
$

!
%!
!




 "
("#

!$%!$ "
! #$ '$/,/$%"

$%$)"$ 
%$)* .+0

$#


  



!
%!


%"#

$#

%"#
)
 



%"
#

 

%$

-"  "  $$ 
$"$ ! "%!
'$/,/$%"

%$)* .+1

" $
$
%$)* .+0


$


  

!
%! !



&$#

  
  



  

 
   





Predicted Total
Score









Difficulty
Score



Execution
Score



Figure 1. An overview of the FineGym-AQA dataset and FineGrade. FineGym-AQA is a ﬁne-grained sports video dataset with detailed
annotations on action procedures. It supports a rule-consistent action quality assessment approach. The FineGrade is an action quality
assessment method with enhanced interpretability by constructing a new Segment Difﬁculty Aggregation module and considering both the
difﬁculty score and the execution score.

Abstract

prises a Temporal Boundary Parser that decodes meaningful sub-action boundaries, an Event-Conditioned Segment
Attention module that aggregates segment-level difﬁculty
in an event-speciﬁc manner, and a Rule-Consistent Totalizer that enforces the ofﬁcial scoring rule for interpretable
prediction. FineGrade outperforms state-of-the-art methods while supporting more interpretable ﬁne-grained action
understanding. Data and code are available at https:
//github.com/yicong1/FineGym-AQA.

Action Quality Assessment (AQA) evaluates how well an
action is performed. However, existing AQA datasets
and methods often suffer from coarse supervision, limited
diversity, and weak consistency with real-world judging
standards, restricting both interpretability and generalization. To address these challenges, we introduce FineGymAQA, a large-scale, score-augmented benchmark that extends FineGym with ofﬁcial competition results and provides veriﬁed annotations of difﬁculty, execution, deduction, and total scores aligned with hierarchical action structures. This dataset bridges ﬁne-grained action understanding and judge-aligned quantitative evaluation. Based on
this foundation, we propose FineGrade, a rule-consistent,
interpretable AQA framework that integrates ﬁne-grained
temporal parsing with judging priors. FineGrade com-

1. Introduction
Action Quality Assessment (AQA) evaluates how well an
action is performed and is typically modeled as a score
regression problem. Unlike conventional action understanding tasks that focus solely on recognition or localization, AQA requires perceiving subtle execution differences

8258

within the same action category. For instance, in gymnastics, two athletes may complete a similar motion, yet one
slightly over-rotates on landing and steps forward: an imperfection that hardly changes the action category but results in a visible deduction in score. Such ﬁne-grained distinctions embody the essence of AQA, which is capturing
the perceptual and physical nuances that deﬁne performance
quality. This capability is crucial for real-world applications
such as sports judging [4, 12, 26, 27, 43, 47], rehabilitation assessment [16, 20, 24], and surgical skill training and
evaluation [8, 15, 44], where objective, consistent, and interpretable evaluation of skill execution is indispensable.

level signals—Difﬁculty (D), Execution (E), Neutral Deductions (ND), and Total (T)—all consolidated through a
systematic semi-automated pipeline. This turns FineGym’s
“what is performed” into “what and how well,” providing
ﬁne-grained, judge-aligned supervision for AQA.
We further propose FineGrade, a rule-consistent
AQA framework that couples temporal parsing with
judge-aligned scoring. Given a video with an event label,
a Temporal Boundary Parser yields a few high-conﬁdence
segment tokens. An Event-Conditioned Segment Attention
then aggregates per-segment difﬁculty evidence using an
event-speciﬁc query, thereby making the source of the difﬁculty score transparent. To adhere to Rule-Consistency, our
Totalizer incorporates two constraints derived from ofﬁcial
judging protocols: (i) the prior T = D + E, which enforces global coherence between difﬁculty and execution;
and (ii) the FIG-prescribed top-K selection, which aggregates difﬁculty values only from the most signiﬁcant subactions. Consequently, this explicit modeling ensures granular, element-level interpretability throughout the inference
process.
Overall, our work contributes to reﬁning ﬁne-grained action quality assessment from both data and modeling perspectives. (1) We construct FineGym-AQA, a large-scale,
score-augmented benchmark that uniﬁes hierarchical semantic annotations with authentic, judge-aligned scoring
labels, offering standardized supervision for interpretable
AQA research. (2) We propose a rule-consistent AQA
model FineGrade, which collects action segments and uses
event-conditioned query–segment attention to isolate top-K
action segments, aggregating difﬁculty and estimating execution from context. (3) Together, the dataset and framework provide a scalable foundation for studying structured
motion understanding and real-world AQA, enabling the interpretable, standardized, and generalizable evaluation of
human performance.

Despite the increasing research interest, existing datasets
and methods for AQA face several limitations. First, most
benchmarks provide only coarse-grained supervision, i.e.,
a single total score per video, lacking ﬁne-grained correspondence between performance segments and judging outcomes. This coarse supervision makes it difﬁcult to trace
which speciﬁc motion components contribute to the ﬁnal
score, limiting interpretability and transparency. Second,
most methods remain domain-speciﬁc with limited diversity: they are typically conﬁned to a single environment
and require training separate models for different scenarios,
which restricts the generalization of AQA models across
domains. Finally, there is weak alignment with real-world
judging standards: most methods treat score prediction as
a black-box regression problem, without explicitly modeling components such as difﬁculty, execution, or the ofﬁcial
protocol of identifying key sub-actions that underlie scoring systems. These limitations collectively restrict progress
toward interpretable and generalizable AQA.
To address these challenges, we aim to build a benchmark that uniﬁes the structured understanding of complex
actions with hierarchical ﬁne-grained supervision of performance quality. An important step toward this goal
was taken by FineGym [33], which established a hierarchical framework for ﬁne-grained action understanding in
gymnastics. FineGym covers multiple gymnastics events,
such as vault, uneven bars, balance beam, and ﬂoor exercise, providing a diverse and representative benchmark
for ﬁne-grained analysis. It organizes videos into a threelevel semantic hierarchy (event–set–element) and a twolevel temporal decomposition (action–sub-action), enabling
ﬁne-grained modeling of routines across multiple semantic
and temporal scales. Such annotations facilitate analyzing
how temporal and semantic granularity inﬂuence recognition performance. However, the original FineGym dataset
lacks quantitative evaluation signals for AQA. To bridge
this gap, we leverage its hierarchical semantics to construct FineGym-AQA, a score-augmented benchmark that
aligns FineGym’s hierarchical instances with comprehensive judging metadata such as athlete identities and subaction difﬁculty values, it also incorporates ofﬁcial event-

2. Related Work
Datasets for AQA. Early studies on Action Quality Assessment primarily focused on small-scale datasets collected
from competitive sports such as diving and ﬁgure skating [26, 27, 31, 41]. These benchmarks typically consist
of short, single-view video clips accompanied by an overall
score assigned by human judges. Subsequent works further
reﬁned these datasets by introducing ﬁner temporal and semantic annotations or multi-task objectives [21, 22, 37, 43],
but they largely remain constrained to the same two sports.
To broaden the scope of AQA, later efforts explored additional competitive events such as rhythmic gymnastics [46]
and synchronized swimming [47], enriching the understanding of motion quality across different athletic domains.
Beyond sports, several studies extended AQA to structured
human activities, including ﬁtness training [18, 30], in-

8259

 
 

     
  

 

  



explicitly model temporal structure. From the feature perspective, AQA research has evolved from early pose-based
representations [25] and deep video-based encoders such
as C3D [36] and I3D [1] to Transformer based [40] and
more recently, multimodal fusion frameworks that integrate RGB, optical ﬂow, and audio cues [45]. While these
advances have improved performance, most existing approaches lack hierarchical modeling that captures both local
sub-action quality and global event composition. In contrast, our rule-consistent FineGrade models temporal semantics, aligning sub-action and event representations to
predict ﬁne-grained, interpretable scores consistent with
real-world judging logic, while ensuring independent scoring without relying on pairwise references.

"number": ”22",
"name": ”XXX",
"score": {
”D": 5.0,
”E": 7.75,
”ND": 0.0,
”T": 12.75
}








  


 
   

Figure 2. Overview of the FineGym-AQA dataset annotation pipeline. We detect and verify athlete bib numbers using
YOLOv5 [13] and EasyOCR [11], then parse ofﬁcial result books
(PDF) with rule-based scripts to extract athlete names, Difﬁculty
(D), Execution (E), Neutral Deductions (ND), and Total (T) scores.
Expert veriﬁcation ensures correct bib–score matching and consistent event-level annotations.

3. The FineGym-AQA Dataset
FineGym-AQA is a large-scale and ﬁne-grained gymnastics
dataset designed to facilitate interpretable action quality assessment. In this section, we introduce the construction of
FineGym-AQA in detail.

3.1. Dataset Construction

dustrial assembly [32], and daily-skill actions [6, 7, 29],
paving the way for broader applications of quality assessment in general human motion analysis. However, compared with large-scale action recognition benchmarks such
as Human3.6M [10] and Kinetics [14], and other recent
datasets designed for video understanding [3, 9, 17, 35], existing AQA datasets still face critical challenges in scale,
action diversity, data modalities, and annotation richness.
Most current benchmarks remain limited in the number of
participants, motion categories, and viewpoints, and provide only coarse video-level scores without multimodal signals such as ﬁne-grained textual descriptions for each subaction. We introduce FineGym-AQA, a score-augmented
benchmark that uniﬁes FineGym’s hierarchical structure
with detailed and interpretable judging scores across multiple gymnastics events, bridging the gap between ﬁnegrained action understanding and quantitative performance
evaluation.

Bib-based Score Alignment. As illustrated in Figure 2,
we pair each annotated event with its judging scores via
a semi-automatic pipeline. We parse mixed-gender result
books with rule-based extractors, map FineGym event IDs
to apparatus columns, and detect athletes in video frames
using YOLOv5 [13]; a torso crop is fed to digit-only EasyOCR [11], and frame-wise voting yields the bib ID. We then
retrieve the corresponding row in the result book to extract
D/E/ND/T and athlete name, with layout-aware parsers handling common format variants. This Bib-based parsing accelerates annotation while maintaining cross-event consistency.
Quality Control. FineGym-AQA extracts all labels directly from ofﬁcial result books and FIG Code of Points,
ensuring traceability. Veriﬁcation followed a structured independent review process involving two collegiate gymnasts with over 10 years of experience. Discrepancies were
resolved through factual backtracking until consensus was
achieved.

Action Quality Assessment. Existing AQA methods can
be broadly categorized into three main formulations: (1)
Direct regression, which predicts a continuous score from
video features under the supervision of the overall label, is
the most widely used paradigm [26, 27, 31]. These methods do not rely on additional references, they align more
closely with the judging process and offer better versatility
and fairness. (2) Pairwise ranking methods [6, 7, 19] learn
relative orderings between samples when absolute scores
are unavailable, but are limited to coarse comparisons without interpretable quantitative outputs. (3) Pairwise regression and procedure-aware approaches [37, 43] extend the
ranking paradigm by introducing step-level alignment or
cross-attention between corresponding action segments to

3.2. Dataset Statistics
The original FineGym dataset [33] organizes gymnastics
movements into a three-level semantic hierarchy of event,
set, and element. It temporally localizes and semantically
annotates elements within each clip, with common evaluation settings featuring 99, 288, or 530 element classes.
Based on this setup, FineGym-AQA augments event-level
instances with ofﬁcial judging information: including total score, execution score, difﬁculty score, and neutral deductions to support the quality-scoring task. At the element level, FineGym-AQA provides the element code and

8260

!

  
!

$

Mask

Execution
Predict



#

Total = D + E

 
Execution
Score

Difficulty
Score

Masked
Mean

Event clip

 $ "

VST

Event ID
Embedding

Segment
Token

Dot Product



Softmax

Query



 

Dagg
Attention Score





DVi

Predict

Weighted Sum

 

"


Conv

Transformer
Encoder

 σ

Dec



Figure 3. Overview of FineGrade. Given clip features X, an encoder produces token embeddings Z and a masked context G. Temporal
Boundary Parser decodes actionness into up to Kevt event-aware segments Sevt and pools them into segment tokens S={si }. EventConditioned Segment Difﬁculty Aggregation uses an event-conditioned query to attend over S, aggregating per-segment difﬁculty values
vi into D̂agg . Rule-Consistent Totalizer sets the total to T̂ = D̂ + Ê, thereby improving interpretability and ﬁdelity to judging rules.
Boundary heads are trained with masked BCE, while Huber losses and a within-event ranking loss supervise D̂/Ê/T̂ .
Table 1. Statistics of the FineGym-AQA dataset at the event, set,
and element levels, reporting the number of set classes (Set Cls),
element classes (Element Cls), annotated instances (Inst), and the
subset of instances augmented with ofﬁcial judging scores for each
event (Score-Aug Inst).
Event

#Set Cls

#Element Cls

#Inst

#Score-Aug Inst

VT
FX
BB
UB

1
5
5
4

67
137
193
133

2042
916
1255
1271

1202
661
795
858

Total

15

530

5484

3516

Table 2. Comparison of representative datasets used for action
quality assessment. Annotation Granularity indicates the ﬁnest
level of temporal or semantic labeling provided. Coarse: only
overall action class or total score; Fine: ﬁne-grained sub-action
boundaries and scores.
Dataset
MIT Dive/Skate [31]
AQA-7 [26]
MTL-AQA [27]
Rhythmic Gymnastics [46]
Fis-V [41]
FS1000 [39]
FineDiving [43]
RFSJ [22]
FineFS [12]
LucidAction [5]
FineGym-AQA (Ours)

its Difﬁculty Value (DV). It also incorporates athlete metadata such as name and number. Additionally, we correct a
small number of annotation inconsistencies. Table 1 summarizes the statistics for the FineGym-AQA.
Table 2 summarizes representative datasets for action
quality assessment and highlights the distinctions of our
FineGym-AQA benchmark. Early datasets such as MIT
Dive/Skate [31] and AQA-7 [26] provide only coarsegrained supervision, where each video is annotated with
a single total score or action class. MTL-AQA [27] and
Rhythmic Gymnastics [46] introduce action-level temporal
boundaries or multiple event types, yet remain limited in
semantic detail. FineDiving [43] advances to ﬁne-grained

#Event #Inst #Sub-act Granularity
2
7
1
4
1
1
1
1
2
8
4

309
1189
1412
1000
500
1604
3000
1304
1167
6702
3516

/
/
/
/
/
/
29
/
232
259
530

Coarse
Coarse
Coarse
Coarse
Coarse
Coarse
Fine
Coarse
Fine
Fine
Fine

temporal segmentation with sub-action labels but focuses
on a single event, while more recent datasets such as LucidAction [5] expand to multi-event coverage under controlled MoCap environments, they rely on subjective expert annotations rather than ofﬁcial judging scores, and
thus lack standardized scoring components. In contrast,
FineGym-AQA stands out by combining large-scale, ﬁnegrained semantic and temporal annotations with standardized multi-component judging scores across four gymnas-

8261

We then group consecutive 1’s into runs rj = [τjst , τjed ]
and drop runs shorter than Lmin
evt . Each remaining
 run is
scored by its mean actionness Ā(rj ) = |r1j | t∈rj At ,
sorted by Ā(rj ), and the top Kevt segments are kept.
If no run survives, we create a single fallback segment
at t = arg maxt At to ensure at least one segment.
The survivors are Sevt = { si = [τist , τied ] }L
i=1 with
. Each segment is pooled into a segment token
L ≤ Kevt
si = |s1i | t∈si zt ∈ Rd , with its mean actionness ai =

(seg)
1
∈ {0, 1}
t∈si At and a binary validity ﬂag mi
|si |
(mi =1 for real segments, 0 for padding/fallback). Stacking
yields S = [s1 , . . . , sL ] ∈ RL×d , a = [a1 , . . . , aL ] ∈ RL ,
(seg)
(seg)
and m(seg) = [m1 , . . . , mL ] ∈ {0, 1}L .
Event-Conditioned Segment Difﬁculty Aggregation. Our
goal is to aggregate per-segment difﬁculty values in an
(seg)
event-aware manner. Let {si , ai , mi }L
i=1 be the set of
decoded segments, where si is the segment embedding,
ai ∈ R is an actionness prior, higher means more likely to
contain valid content. We ﬁrst build an event-conditioned
query and project segment features to a common space:


s̃i = Ws si .
(4)
qevt = Wq LN G + eevt ,

tics events, providing both interpretability and scalability
for future AQA research.

4. Approach
In this section, we present FineGrade. The key idea is
to decode temporal segments and aggregate them with an
event-conditioned attention mechanism while enforcing the
rule-consistent prior, leading to reliable and interpretable
action quality assessment. The overall architecture of FineGrade is illustrated in Figure 3.

4.1. Problem Formulation
Given a video of a single routine, we form a length-N feature sequence X = {xt ∈ RC }N
t=1 with a validity mask
m ∈ {0, 1}N and an event index evt ∈ {1, 2, 3, 4} (VT,
FX, BB, UB). Each sample provides scalar ground-truth
scores (D∗ , E ∗ , T ∗ ) for difﬁculty, execution, and total, with
the judging prior T ∗ = D∗ + E ∗ ; neutral deductions
(ND) are included in E ∗ . Our network fθ (X, evt) predicts
difﬁculty and execution (D̂, Ê). In parallel, a boundary
head outputs token-wise start/end/actionness logits to support segment decoding and interpretability.

4.2. The FineGrade Network

where G is the global representation, LN is Layer Norm,
eevt is the learnable event embedding, and Wq , Ws are
learnable projections, so qevt , s̃i ∈ Rd .
Segment scoring and normalization. We score each segment
by a dot product between the event query and the projected
segment, with an additive actionness bias, and normalize
with a masked softmax:

Encoding and Masked Global Context. We encode X
with a Transformer encoder with positional encodings to
obtain token embeddings Z = {zt }N
t=1 , where each zt ∈
Rd and Z ∈ RN ×d . A masked global context G ∈ Rd is
computed as:
1
G = MaskedMean(Z, m) = N

N


t=1 mt t=1

mt zt . (1)

qevt s̃i
e u i mi
+ ai , w i =
.
ui = √
L
(seg)
d
ε + j=1 euj mj
(seg)

Here m ∈ {0, 1}N marks valid token-steps (1) vs. padding
(0), and d is the model width.
Temporal Boundary Parser. A boundary head predicts
start/end/actionness logits (s , e , a ) ∈ RN ×1 from the token embeddings Z ∈ RN ×d using a lightweight 1D convolution. All computations and losses are masked by the
token-level mask m so that padding time-steps do not contribute. We then decode actionness into segments by thresholding and run-length grouping, and keep at most Kevt
high-conﬁdence segments with a per-event top-K budget;
very short runs are removed via a per-event minimal-length
constraint. Formally:


(a)
Sevt = DecodeSeg σ(a ), m; Kevt , θevt , Lmin
evt . (2)

(5)

where d = dmodel and ai adds a prior bias on top of
the semantic similarity, encouraging the weights to focus
on segments that are more likely to contain valid actions.
(seg)
∈ {0, 1} is a binary mask that retains at most Kevt
mi
segments for event. We add a small ε > 0 in the denominator to avoid division-by-zero.
Difﬁculty Value Aggregation. A scalar Difﬁculty Value
(DV) is predicted per segment via a lightweight head vi =
φDV (si ). The event-aware aggregated difﬁculty is the
attention-weighted sum:
D̂agg =

L


wi vi .

(6)

i=1

where σ(·) is the sigmoid and m ∈ {0, 1}N masks padding.
Concretely, let A = σ(a )  m with  the element-wise
(a)
product. We ﬁrst threshold At at θevt to obtain a binary
sequence:


(a)
(3)
bt = 1 At ≥ θevt .

Eqs. (4)–(6) implement a single-query, masked dot-product
attention over at most Kevt segments: the event embedding
eevt steers selection toward event-speciﬁc evidence; the actionness bias ai promotes segments that likely contain valid
content. The resulting D̂agg is used in the fusion stage (see

8262

Table 3. Main results on FineGym-AQA. We report per-apparatus
SRCC (VT/FX/BB/UB) and the Fisher average, plus MSE. Bold
indicates the best in column; higher is better.

Eq. (7)) to respect the structural prior T =D+E while remaining robust to noisy segments.
Rule-Consistent Totalizer. Given a global representation G and event index evt, we employ two lightweight,
(D)
(E)
event-speciﬁc heads φevt and φevt and a learnable
per-event scalar gate αevt ∈ [0, 1].
Difﬁculty. We interpolate a segment-aggregated difﬁculty
D̂agg with a context head on G:
(D)

D̂ = αevt D̂agg + (1 − αevt ) φevt (G).

Per-apparatus SRCC↑

(7)

Features

VT

C3D-SVR [28]
C3D-AVG-STL [27]
MS-LSTM [41]
USDL [34]
ACTION-NET [46]
GDLT [40]
FineGrade (Ours)

C3D
C3D
C3D
I3D
VST
VST
VST

0.502 0.426 0.581 0.558
0.651 0.614 0.583 0.557
0.716 0.571 0.623 0.571
0.742 0.608 0.634 0.629
0.664 0.679 0.730 0.662
0.728 0.654 0.755 0.719
0.746 0.716 0.794 0.750

UB

Average
SRCC↑ MSE↓
0.520
0.602
0.624
0.657
0.685
0.716
0.753

–
0.906
0.849
0.896
0.782
0.813
0.752

(E)
φevt (G).

Methods

Per-apparatus SRCC↑

(8)

T̂ = D̂ + Ê.

(9)

Average

Ball Clubs Hoop Ribbon SRCC↑ MSE↓

Features

C3D-SVR [28]
C3D
0.357 0.551 0.495
MS-LSTM [41]
I3D
0.515 0.621 0.540
ACTION-NET [46]
I3D
0.528 0.652 0.708
ACTION-NET [46]
VST
0.684 0.737 0.733
GDLT [40]
VST
0.746 0.802 0.765
CoFInAl [48]
VST
0.809 0.806 0.804
QGVT [42]
VST+AST
0.828 0.827 0.830
LMAC-Net [38]
VST+AST+I3D 0.803 0.806 0.856
FineGrade (Ours)
VST
0.779 0.832 0.893

This totalization enforces the additive scoring rule
by construction, improves interpretability, and retains
(D)
(E)
apparatus-speciﬁc ﬂexibility through {φevt , φevt , αevt }.

0.516
0.522
0.578
0.754
0.741
0.810
0.836
0.881
0.886

0.483
0.551
0.623
0.728
0.765
0.807
0.830
0.840
0.847

–
8.97
7.91
7.41
5.53
5.73
4.78
–
4.54

Table 5. Total score ablation in FineGrade. Top: switch summary
indicating which components feed the rule-consistent total (T =
D̂ + Ê): context difﬁculty (Dctx ), aggregated difﬁculty (Dagg ), and
execution (E). Bottom: per-apparatus SRCC and Fisher’s average;
Baseline uses all three sources. Bold = best; higher is better.

4.3. Optimization
Training. For a training clip X of event evt with boundary labels (ys , ye , ya ) and judging targets (D∗ , E ∗ , T ∗ ), we
optimize a rule-consistent objective where the total is constrained by T̂ =D̂+Ê:

Switch summary
Setting
E only
Dctx only
Dagg only
NO Dctx
NO Dagg
NO E
Baseline

J (X) = λT LHuber (D̂+Ê, T ∗ ) + λD LHuber (D̂, D∗ )
+ λE LHuber (Ê, E ∗ ) + λr Lrank (T̂ , T ∗ | evt)

(10)
λh LBCE (logith , yh ).
+

Dctx → T

Dagg → T

E→ T

























Per-apparatus SRCC↑

h∈{s,e,a}

Here LHuber is applied to scores, Lrank enforces pairwise
ordering among clips of the same event, and boundary heads
use masked BCE.
Inference. Given a test clip Xtest with event index evt, we
perform a single forward pass: predict actionness to decode
segments and compute T̂ :

Average
SRCC↑ AIoU@0.5↑

Setting

VT

T Contribution
E only
Dctx only
Dagg only
NO Dctx
NO Dagg
NO E

0.300 0.422 0.572 0.553
0.660 0.634 0.690 0.698
0.745 0.569 0.759 0.703
0.712 0.655 0.759 0.711
0.645 0.625 0.748 0.722
0.698 0.601 0.749 0.739

0.469
0.671
0.701
0.711
0.688
0.701

0.134
0.493
0.402
0.444
0.443
0.473

Baseline

0.746 0.716 0.794 0.750

0.753

0.510

FX

BB

UB

5. Experiment

(D)

D̂ = αevt D̂agg + (1 − αevt )φevt (G),
(E)

BB

Table 4. Comparisons of performance with state-of-the-art on
Rhythmic Gymnastics Dataset [46]. Bold indicates the best in column; higher is better.

Total. The total strictly follows the judging prior:

Ê = φevt (G),

FX

(D)

This event-speciﬁc context φevt (G) supplies a global prior
for different events.
Execution. Execution is predicted from context via an
event-speciﬁc head:
Ê =

Methods

5.1. Implementation Details

(11)

We adopt VST [23] pretrained on Kinetics [2] as the visual
backbone and train the model on pre-extracted features. For
FineGym-AQA dataset, Neutral deductions are folded into
E at the dataset layer. Training uses AdamW (lr 5×10−5 ,
wd 5×10−3 ), cosine schedule, batch size 32, AMP, and

T̂ = D̂ + Ê.
This inference preserves the additive judging rule, yielding an interpretable prediction where each point in T̂ is attributed to either difﬁculty or execution.

8263

Table 6. Ablation of segment formation in FineGrade. Setting: whole clip (no segmentation), uniform-k (k equal splits), and
Baseline (learned tokenizer predicting up to Kevt segments).
Per-apparatus SRCC↑

Average
SRCC↑ MSE↓

Setting

VT

Segment Strategy
Whole clip
Uniform4
Uniform8
Uniform12

0.655 0.679 0.700 0.722
0.665 0.724 0.725 0.731
0.587 0.719 0.694 0.696
0.658 0.697 0.707 0.713

0.690
0.705
0.711
0.700

1.036
0.904
0.817
0.840

Baseline

0.746 0.716 0.794 0.750

0.753

0.752

FX

BB

UB

FineGym-AQA dataset. Our FineGrade signiﬁcantly improved the performance of Spearman’s rank correlation
compared to all methods.
Results on Rhythmic Gymnastics. As shown in Table 4,
with the same VST features, our method attains an average
SRCC of 0.847 surpassing LMAC-Net [38] by 0.007 and
QGVT [42] by 0.017, while reducing MSE to 4.54. At the
apparatus level, we set new best SRCC on Clubs, Hoop and
Ribbon (0.832, 0.893, 0.886), improving by 0.026, 0.037
and 0.005, respectively. On Ball we are below QGVT [42],
yet the overall average remains clearly superior. These results indicate that FineGrade’s ﬁne-grained modeling consistently boosts rank correlation and reduces error across
apparatus.

gradient clipping at 0.5. We enable per-event z-norm for
(D, E, T ) and event-balanced sampling; segment thresh(a)
olds (θevt , Lmin
evt ) and budgets Kevt are ﬁxed across training.
For batching, we pad the number of segments to the perevent upper bound Kevt . Following the experiment settings
in [41, 46], we selected 80 percent of samples for training
and 20 percent for testing in all the experiments. For Rhythmic Gymnastics dataset, we generate uniform pseudo segments to form (ys , ye , ya ). Optimization uses AdamW (lr
2×10−5 ) with batch size 24. More details are in the Appendix.

5.4. Ablation studies
We conducted an ablation study on the FineGym-AQA
dataset to demonstrate the effectiveness of individual components in FineGrade, including contributions to the ruleconsistent total score, segment formation strategies, and
boundary prediction heads.
Contributions to Total Score. We summarize the experimental results in Table 5, which explores various conﬁgurations for incorporating the components: context difﬁculty Dctx , aggregated difﬁculty Dagg , and execution E into
T = D̂ + Ê. The execution-only approach (E_only)
produces a low average SRCC of 0.469 and AIoU@0.5 of
0.134. In contrast, relying solely on aggregated difﬁculty
(Dagg_only) achieves a strong SRCC of 0.701, surpassing the context-only method (Dctx_only) at 0.671. This
underscores the effectiveness of Dagg as a core component
in capturing essential difﬁculty elements. Omitting individual components reduces performance: NO_Dctx yields
0.711 SRCC, NO_Dagg yields 0.688, and NO_E yields
0.701. The experimental results suggest that Dctx serves as
a supplement to Dagg , particularly in capturing the connection and transition components within the difﬁculty score.
The baseline, which integrates all three components, delivers the highest per-apparatus SRCC and AIoU@0.5. Therefore, full integration is crucial for rule-consistent scoring
and optimal accuracy, as the ablation studies highlight the
vital synergy among these elements.
Segment Formation Strategies. We examined various segmentation methods in Table 6. Treating the input as a whole
clip yields an average SRCC of 0.690, while uniform splitting into equal segments raises this to 0.700–0.711 by enabling ﬁner-grained analysis. Among uniforms, uniform8
performs best at 0.711, suggesting moderate granularity
balances detail and efﬁciency without over-fragmentation.
The baseline using a learned tokenizer to predict up to Kevt
segments, outperforms all with an SRCC of 0.753. This
shows the beneﬁts of adaptive segmentation in capturing
routine boundaries for precise scoring.

5.2. Evaluation Metrics
Score Prediction. Our primary metric is Spearman’s rank
correlation (SRCC) on totals:
n
(xri − x̄r )(yir − ȳ r )
n
ρ = n i=1
.
(12)
r
r
r 2
r 2
i=1 (xi − x̄ )
i=1 (yi − ȳ )
where xri and yir denote the ranks of the i-th sample, and
x̄r , ȳ r represent the mean ranks across all samples. The
coefﬁcient ρ ∈ [−1, 1], with higher values indicating better
ranking consistency. Following previous works [12, 40], we
also report the average ρ for per apparatus using Fisher’s ztransformation for averaging. MSE is used to measure the
numerical difference between the predicted scores ŝ and the
ground-truth scores s:
MSE =

1
ŝ − s22 .
N

(13)

Average Intersection-over-Union (AIoU). Given a threshold d ∈ [0, 1], a segment is correct if IoUi,k ≥ d. Across N
N
videos, with M = i=1 (Li − 1) total segments, inducing
Li −1 segments, we report:
AIoU@d =

N Li −1


1 
I IoUi,k ≥ d .
M i=1

(14)

k=1

where I(·) is the indicator. Higher is better.

5.3. Comparison with the State-of-the-Arts
Results on FineGym-AQA. Table 3 summarizes the experimental results of state-of-the-art AQA methods on the

8264

 

   
S

 

 

Top-6 Segments
Score:

















 

   
S

 

 

Top-6 Segments
Score:

Figure 4. Qualitative case study of temporal segmentation and Tok-6 segments score. In the above example cases, we visualize sampled
frames, the predicted segment probability pseg (t) with ground-truth (GT) segments aligned on the same timeline, and the segment scores.
Color bars on the right indicate normalized values in [0,1]. High attention often coincides with high pseg (t) inside GT segments, peaking
near critical skills and landings.
Table 7. Boundary-head ablation. Bnd s: start-boundary head;
Bnd e: end-boundary head; Bnd a: actionness head; Bnd up: uniﬁed setting with all three heads enabled (s+e+a). Metrics are
AIoU@0.3/0.5/0.7 (higher is better). Bold is the best in column.

the ground-truth segment labels. Meanwhile, the high-value
regions of the temporal attention a(t) primarily concentrate
on key actions and transition phases, forming distinct peaks
at moments such as takeoff and landing. This observation
indicates that: the attention mechanism automatically focuses on important segments containing scoring information and the synchronization of attention peaks with high
pseg (t) values within segments reﬂects the synergy between
segment modeling and global aggregation, thereby improving temporal alignment and scoring correlation.

Boundary Heads
Switches
Setting
Bnd s only
Bnd e only
Bnd a only
Bnd up

Metrics

Bnd s Bnd e Bnd a AIoU@0.3 AIoU@0.5 AIoU@0.7















0.397
0.400
0.496
0.634

0.342
0.343
0.432
0.510

0.341
0.322
0.338
0.393

6. Conclusion
Boundary Prediction Heads. We ablated the boundary
heads in Table 7, toggling start (Bnd s), end (Bnd e), and
actionness (Bnd a) heads. Single-head settings show that
Bnd a only achieves the strongest standalone performance
(i.e., AIoU@0.5 of 0.432), indicating actionness is crucial for detecting meaningful segments. Bnd s only and
Bnd e only are comparable but weaker, as they focus on
edges without holistic activity cues. The uniﬁed setting
(Bnd up), enabling all heads, yields the best results across
thresholds, demonstrating that joint optimization of start,
end, and actionness improves boundary precision and overall temporal parsing in FineGrade.

In this paper, we extend FineGym to FineGym-AQA using a semi-automatic pipeline, incorporating ﬁne-grained
score annotations at the sub-action level for assessing action quality. On FineGym-AQA, we propose a ruleconsistent action quality assessment approach by constructing a new Segment Difﬁculty Aggregation module, which
learns event-conditioned, segment-level difﬁculty attributions and, via a rule-consistent totalizer enforcing the realworld judging rule, makes inference more interpretable and
delivers substantial gains over prior AQA methods.
Limitations. The proposed method requires ﬁne-grained
score labels: difﬁculty, execution, and total for effective
model training.
Acknowledgements. The work described in this paper was
fully supported by a grant from the Research Grants Council
of the Hong Kong Special Administrative Region, China
[Project No. CityU11208324]

5.5. Visualization
As illustrated in Figure 4, we conduct a qualitative analysis
on videos from two different apparatuses. The model’s output segment probability trajectory pseg (t) aligns well with

8265

References

models for identifying surgical actions and measuring performance. 3(3):e201664, 2020. 2
[16] Ikram Kourbane, Panagiotis Papadakis, and Mihai Andries.
Optimized assessment of physical rehabilitation exercises
using spatiotemporal, sequential graph-convolutional networks. 186:109578, 2025. 2
[17] Jianwei Li, Jun Xue, Rui Cao, Xiaoxia Du, Siyu Mo, Kehao Ran, and Zeyan Zhang. Finerehab: A multi-modality
and multi-task dataset for rehabilitation analysis. In CVPRW,
pages 3184–3193, 2024. 3
[18] Yuan-Ming Li, Wei-Jin Huang, An-Lan Wang, Ling-An
Zeng, Jing-Ke Meng, and Wei-Shi Zheng. Egoexo-ﬁtness:
Towards egocentric and exocentric full-body action understanding. In ECCV, pages 363–382. Springer, 2024. 2
[19] Zhenqiang Li, Yifei Huang, Minjie Cai, and Yoichi Sato.
Manipulation-skill assessment from videos with spatial attention network. pages 0–0, 2019. 3
[20] Yalin Liao, Aleksandar Vakanski, and Min Xian. A deep
learning framework for assessing physical rehabilitation exercises. 28(2):468–477, 2020. 2
[21] Shenlan Liu, Xiang Liu, Gao Huang, Lin Feng, Lianyu Hu,
Dong Jiang, Aibin Zhang, Yang Liu, and Hong Qiao. FSD10: a dataset for competitive sports content analysis. arXiv
preprint arXiv:2002.03312, 2020. 2
[22] Yanchao Liu, Xina Cheng, and Takeshi Ikenaga. A ﬁgure
skating jumping dataset for replay-guided action quality assessment. In ACM MM, pages 2437–2445, 2023. 2, 4
[23] Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang,
Stephen Lin, and Han Hu. Video swin transformer. In CVPR,
pages 3202–3211, 2022. 6
[24] Ciro Mennella, Umberto Maniscalco, Giuseppe De Pietro,
and Massimo Esposito. A deep learning system to monitor
and assess rehabilitation exercises in home-based remote and
unsupervised conditions. 166:107485, 2023. 2
[25] Jia-Hui Pan, Jibin Gao, and Wei-Shi Zheng. Action assessment by joint relation graphs. In ICCV, pages 6331–6340,
2019. 3
[26] Paritosh Parmar and Brendan Morris. Action quality assessment across multiple actions. pages 1468–1476. IEEE, 2019.
2, 3, 4
[27] Paritosh Parmar and Brendan Morris. What and how well
you performed? a multitask learning approach to action quality assessment. In CVPR, pages 304–313, 2019. 2, 3, 4, 6
[28] Paritosh Parmar and Brendan Tran Morris. Learning to score
olympic events. In CVPRW, pages 20–28, 2017. 6
[29] Paritosh Parmar, Jaiden Reddy, and Brendan Morris. Piano
skills assessment. pages 1–5. IEEE, 2021. 3
[30] Paritosh Parmar, Amol Gharat, and Helge Rhodin. Domain knowledge-informed self-supervised representations
for workout form assessment. In ECCV, pages 105–123.
Springer, 2022. 2
[31] Hamed Pirsiavash, Carl Vondrick, and Antonio Torralba. Assessing the quality of actions. In ECCV, pages 556–571.
Springer, 2014. 2, 3, 4
[32] Fadime Sener, Dibyadip Chatterjee, Daniel Shelepov, Kun
He, Dipika Singhania, Robert Wang, and Angela Yao. Assembly101: A large-scale multi-view video dataset for un-

[1] Joao Carreira and Andrew Zisserman. Quo vadis, action
recognition? a new model and the kinetics dataset. In CVPR,
pages 6299–6308, 2017. 3
[2] Joao Carreira, Eric Noland, Andras Banki-Horvath, Chloe
Hillier, and Andrew Zisserman. A short note about kinetics600. arXiv preprint arXiv:1808.01340, 2018. 6
[3] Zijian Chen, Wei Sun, Yuan Tian, Jun Jia, Zicheng Zhang,
Wang Jiarui, Ru Huang, Xiongkuo Min, Guangtao Zhai, and
Wenjun Zhang. GAIA: Rethinking action quality assessment
for ai-generated videos. NeurIPS, 37:40111–40144, 2024. 3
[4] Yuning Ding, Sifan Zhang, Liu Shenglan, Jinrong Zhang,
Wenyue Chen, Duan Haifei, Bingcheng Dong, and Tao Sun.
2m-af: A strong multi-modality framework for human action
quality assessment with self-supervised representation learning. In ACM MM, pages 1564–1572, 2024. 2
[5] Linfeng Dong, Wei Wang, Yu Qiao, and Xiao Sun. Lucidaction: A hierarchical and multi-model dataset for comprehensive action quality assessment. NeurIPS, 37:96468–96482,
2024. 4
[6] Hazel Doughty, Dima Damen, and Walterio Mayol-Cuevas.
Who’s better? who’s best? pairwise deep ranking for skill
determination. In CVPR, pages 6057–6066, 2018. 3
[7] Hazel Doughty, Walterio Mayol-Cuevas, and Dima Damen.
The pros and cons: Rank-aware temporal attention for skill
determination in long videos. In CVPR, pages 7862–7871,
2019. 3
[8] Isabel Funke, Sören T. Mees, Jürgen Weitz, and Stefanie
Speidel. Video-based surgical skill assessment using 3D convolutional neural networks. 14(7):1217–1225, 2019. 2
[9] Ziliang Gan, Lei Jin, Yi Cheng, Yu Cheng, Yinglei Teng,
Zun Li, Yawen Li, Wenhan Yang, Zheng Zhu, Junliang Xing,
et al. Skatingverse: A large-scale benchmark for comprehensive evaluation on human action understanding. 18(7):
888–906, 2024. 3
[10] Catalin Ionescu, Dragos Papava, Vlad Olaru, and Cristian
Sminchisescu. Human3.6M: Large scale datasets and predictive methods for 3D human sensing in natural environments.
IEEE TPAMI, 36(7):1325–1339, 2013. 3
[11] Jaided AI Research. EasyOCR: Ready-to-use OCR with 80+
Languages Supported, 2020. Accessed: 2025-10-10. 3
[12] Yanli Ji, Lingfeng Ye, Huili Huang, Lijing Mao, Yang Zhou,
and Lingling Gao. Localization-assisted uncertainty score
disentanglement network for action quality assessment. In
ACM MM, pages 8590–8597, 2023. 2, 4, 7
[13] Glenn Jocher, Ayush Chaurasia, Alex Stoken, Jirka Borovec,
Yonghye Kwon, Kalen Michael, Jiacong Fang, Zeng Yifu,
Colin Wong, Diego Montes, et al. ultralytics/YOLOv5: v7.0YOLOv5 sota realtime instance segmentation. 2022. 3
[14] Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang,
Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola,
Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint arXiv:1705.06950,
2017. 3
[15] Shuja Khalid, Mitchell Goldenberg, Teodor Grantcharov,
Babak Taati, and Frank Rudzicz. Evaluation of deep learning

8266

derstanding procedural activities. In CVPR, pages 21096–
21106, 2022. 3
[33] Dian Shao, Yue Zhao, Bo Dai, and Dahua Lin. Finegym: A
hierarchical video dataset for ﬁne-grained action understanding. In CVPR, pages 2616–2625, 2020. 2, 3
[34] Yansong Tang, Zanlin Ni, Jiahuan Zhou, Danyang Zhang,
Jiwen Lu, Ying Wu, and Jie Zhou. Uncertainty-aware score
distribution learning for action quality assessment. In CVPR,
pages 9839–9848, 2020. 6
[35] Yansong Tang, Jinpeng Liu, Aoyang Liu, Bin Yang, Wenxun
Dai, Yongming Rao, Jiwen Lu, Jie Zhou, and Xiu Li.
FLAG3D: A 3D ﬁtness activity dataset with language instruction. In CVPR, pages 22106–22117, 2023. 3
[36] Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani,
and Manohar Paluri. Learning spatiotemporal features with
3D convolutional networks. In ICCV, pages 4489–4497,
2015. 3
[37] Shunli Wang, Dingkang Yang, Peng Zhai, Chixiao Chen, and
Lihua Zhang. TSA-Net: Tube self-attention network for action quality assessment. In ACM MM, pages 4902–4910,
2021. 2, 3
[38] Xin Wang, Peng-Jie Li, and Yuan-Yuan Shen. Attentiondriven multimodal alignment for long-term action quality assessment. page 113649, 2025. 6, 7
[39] Jingfei Xia, Mingchen Zhuge, Tiantian Geng, Shun Fan,
Yuantai Wei, Zhenyu He, and Feng Zheng. Skating-mixer:
Long-term sport audio-visual modeling with mlps. In AAAI,
pages 2901–2909, 2023. 4
[40] Angchi Xu, Ling-An Zeng, and Wei-Shi Zheng. Likert scoring with grade decoupling for long-term action assessment.
In CVPR, pages 3232–3241, 2022. 3, 6, 7
[41] Chengming Xu, Yanwei Fu, Bing Zhang, Zitian Chen, YuGang Jiang, and Xiangyang Xue. Learning to score ﬁgure skating sport videos. IEEE TCSVT, 30(12):4578–4590,
2019. 2, 4, 6, 7
[42] Huangbiao Xu, Huanqi Wu, Xiao Ke, Yuezhou Li, Rui Xu,
and Wenzhong Guo. Quality-guided vision-language learning for long-term action quality assessment. IEEE TMM,
2025. 6, 7
[43] Jinglin Xu, Yongming Rao, Xumin Yu, Guangyi Chen, Jie
Zhou, and Jiwen Lu. FineDiving: A ﬁne-grained dataset for
procedure-aware action quality assessment. In CVPR, pages
2949–2958, 2022. 2, 3, 4
[44] Erim Yanik, Uwe Kruger, Xavier Intes, Rahul Rahul, and Suvranu De. Video-based formative and summative assessment
of surgical tasks using deep learning. 13:1038, 2023. 2
[45] Ling-An Zeng and Wei-Shi Zheng. Multimodal action quality assessment. IEEE TIP, 33:1600–1613, 2024. 3
[46] Ling-An Zeng, Fa-Ting Hong, Wei-Shi Zheng, Qi-Zhi Yu,
Wei Zeng, Yao-Wei Wang, and Jian-Huang Lai. Hybrid
dynamic-static context-aware attention network for action
assessment in long videos. In ACM MM, pages 2526–2534,
2020. 2, 4, 6, 7
[47] Shiyi Zhang, Wenxun Dai, Sujia Wang, Xiangwei Shen, Jiwen Lu, Jie Zhou, and Yansong Tang. LOGO: A long-form
video dataset for group action quality assessment. In CVPR,
pages 2405–2414, 2023. 2

[48] Kanglei Zhou, Junlin Li, Ruizhi Cai, Liyuan Wang, Xingxing Zhang, and Xiaohui Liang. Coﬁnal: Enhancing action
quality assessment with coarse-to-ﬁne instruction alignment.
arXiv preprint arXiv:2404.13999, 2024. 6

8267
</reference>

<statements>
1. Modern architectures construct domain models by compiling statistical shape and motion models from elite performers, establishing continuous kinematic manifolds against which novice executions can be measured
2. The system processes synchronized, multimodal streams through four operational analytic modules: temporal action parsing, kinematic tracking, biomechanical fault detection, and automated Action Quality Assessment.
3. Temporal action parsing divides continuous movement into biomechanical phases.
4. Automated Action Quality Assessment algorithms quantify overall execution quality by assigning scores aligned with competitive judging standards or pedagogical rubrics.
5. Recent models, such as FineGrade on the FineGym-AQA benchmark, use hierarchical, rule-based assessment structures.
6. These models divide scoring into a Difficulty Value that evaluates technical complexity and an Execution Score that applies deductions for detected kinematic errors, mirroring official sports judging frameworks.
7. In the benchmark dataset comparison, FineGym / FineGym-AQA is listed as a benchmark dataset for action quality assessment.
8. FineGym / FineGym-AQA evaluates artistic gymnastics disciplines including vault, floor, uneven bars, and balance beam.
9. FineGym / FineGym-AQA uses broadcast RGB video streams as input data modalities.
10. FineGym / FineGym-AQA has a 3-level semantic hierarchy with element codes, official D-scores, E-scores, and deductions.
11. The primary performance baseline for FineGym / FineGym-AQA is 0.89–0.93 for FineGrade and temporal transformers.
12. These benchmark datasets demonstrate high Spearman's Rank Correlation Coefficients between automated model predictions and calibrated human judging panels.
13. In aesthetic and acrobatic sports, including competitive diving, figure skating, and artistic gymnastics, procedure-aware AQA models on datasets like FineGym and FineDiving have matched expert judging evaluations (SRCC \(> 0.90\)), confirming the viability of automated scoring and corrective feedback systems [10].
14. System designers must also address demographic representation biases within training datasets
15. A persistent challenge in automated action quality assessment is the interpretability of deep learning models
16. Pure score-regression models that output an isolated numerical rating without clarifying the underlying evaluation criteria offer limited instructional value to athletes and coaches
17. To make feedback actionable, next-generation architectures incorporate Explainable AI (XAI) and causal modeling techniques
18. Frameworks such as FineCausal, TechCoach, and CoT-AFA parse complex movements into distinct technical components
</statements>

Begin the assessment now. Output only the JSON list, without any conversational text or explanations.