You will be provided with a reference and some statements. Please determine whether each statement is 'supported', 'unsupported', or 'unknown' with respect to the reference. Please note:
First, assess whether the reference contains any valid content. If the reference contains no valid information, such as a 'page not found' message, then all statements should be considered 'unknown'.
If the reference is valid, for a given statement: if the facts or data it contains can be found entirely or partially within the reference, it is considered 'supported' (data accepts rounding); if all facts and data in the statement cannot be found in the reference, it is considered 'unsupported'.

You should return the result in a JSON list format, where each item in the list contains the statement's index and the judgment result, for example:
[
    {
        "idx": 1,
        "result": "supported"
    },
    {
        "idx": 2,
        "result": "unsupported"
    }
]

Below are the reference and statements:
<reference>
1

Husformer: A Multi-Modal Transformer for
Multi-Modal Human State Recognition

arXiv:2209.15182v2 [cs.HC] 11 Apr 2023

Ruiqi Wang1 †, Wonse Jo1 †, Dezhong Zhao2 , Weizheng Wang1
Baijian Yang1 , Guohua Chen2 , and Byung-Cheol Min1

Abstract—Human state recognition is a critical topic with
pervasive and important applications in human-machine systems.
Multi-modal fusion, which involves combining metrics from
multiple data sources, has been shown to be an effective method
for improving the recognition performance. While promising
results have been reported by recent multi-modal-based models,
they generally fail to leverage the sophisticated fusion strategies
that would model sufficient cross-modal dependencies when
producing the fusion representation; instead, they rely on costly
and inconsistent feature crafting and alignment. To address this
limitation, we propose an end-to-end multi-modal transformer
framework for multi-modal human state recognition called Husformer. Specifically, we propose using cross-modal transformers,
which inspire one modality to reinforce itself through directly
attending to latent relevance revealed in other modalities, to
fuse different modalities while ensuring sufficient awareness
of the cross-modal interactions introduced. Subsequently, we
utilize a self-attention transformer to further prioritize contextual
information in the fusion representation. Extensive experiments
on two human emotion corpora (DEAP and WESAD) and two
cognitive load datasets (MOCAS and CogLoad) demonstrate that
in the recognition of human state, our Husformer outperforms
both state-of-the-art multi-modal baselines and the use of a single
modality by a large margin, especially when dealing with raw
multi-modal features. We also conducted an ablation study to
show the benefits of each component in Husformer1 .
Index Terms—Cognitive Load Recognition, Emotion Prediction, Multi-modal Deep Learning, Cross-modal Attention, Transformer

I. I NTRODUCTION

R

ECOGNITION of human state, including human affective state, known as emotion, and cognitive load, known
as mental stress, plays an enormous role in any humanmachine interaction systems, as enabling machines to perceive,
understand, and adapt to different human emotional and cognitive states, and improves the performance of the whole systems
[1], [2], [3]. Human state assessment methods can generally be
divided into two main categories based on the types of signals
used: physiological and behavioral assessments. Physiological
assessments involve measuring human physiological metrics,
such as galvanic skin response (GSR), electroencephalography (EEG), electrooculography (EOG), electrocardiography
1 Department of Computer and Information Technology, Purdue University,
West Lafayette, IN 47907, USA [wang5357, jow, wang5716,

byang, minb] @purdue.edu
2 College of Mechanical and Electrical Engineering, Beijing University
of Chemical Technology, Beijing, China. DZ_Zhao@buct.edu.cn,

chengh@mail.buct.edu.cn
† Equal contribution
1 Source code for our Husformer and experiments is available at: https:
//github.com/SMARTlab-Purdue/Husformer.

(ECG), electromyogram (EMG), and heart rate (HR), which
change in response to involuntary reactions of the human
nervous system under specific states. On the other hand,
behavioral assessments analyze subconscious human behavioral responses, including facial expressions, body and eye
movements, and mouse movements, and associate them with
different human states [4].
Unfortunately, owing to the complexity of human state
reasoning, it is improbable that signals from a single modality
are sufficient to achieve optimal recognition performance in
terms of accuracy and robustness [5], [6], [7]. For instance,
different signals possess different levels of sensitivity to different task environments and even different human subjects,
and determining one modality that is efficient for every task
scenario and subject is impractical and impossible. Furthermore, reliance on a unimodal data source means that noise
or interruption of the signals can result in extensive errors or
even a failure of the recognition system [8].
Recently, multi-modal fusion-based human state recognition
methods that combine data from multiple modalities have been
proposed and demonstrated promise as a solution for the aforementioned challenges faced by single-modal-based approaches
[9], [10], [11], [12]. The adoption of multi-modal signals can
reduce the noise-to-signal ratio and enhance tolerance against
sensor failures. More importantly, fusing different metrics collected from the same subject under one particular human state
through multiple modalities can reveal important and comprehensive indexes of human emotion and cognitive load that are
inaccessible via a single modality [5], [7]. Nevertheless, the
inherent heterogeneity of multiple modalities poses challenges
in generating an efficient fusion index of the human state.
These challenges include: 1) different modalities are typically
unaligned, resulting in unbalanced feature length and temporal
resolutions; and 2) combining multi-modal features may lead
to the inclusion of biased or irrelevant information due to
feature noncommensurability across multiple modalities; and
3) inference of long-term and complicated dependencies across
modalities is required for accurate fusion [5].
Current multi-modal fusion approaches for human state
recognition remain in their early stages and have not yet
fully addressed the challenges arising from the heterogeneity
across multiple modalities. Most methods rely on extensive
feature engineering and alignment to concatenate features from
different modalities and produce the fusion representation[5],
[13], [14]. However, such direct concatenation fusion schemes
ignore the latent correlations across modalities and may
be limited by non-instantaneous coupling, even if manually
aligned. While several methods have been proposed to learn

2

���������� �

�� ∈ ℝ � � , �

Self-Attention
Transformer

Self-attention Module

Concatenation

�� ∈ ℝ ��, �

Concatenation

Positional
Encoding

���� ∈ ℝ ����, �

���� ∈ ℝ ����, �

���� ∈ ℝ ����, �

Cross-Modal
Transformer

Cross-Modal
Transformer

Cross-Modal
Transformer

Cross-Modal
Transformer

Fusion → EEG

Fusion → EMG

Fusion → GSR

Fusion → EB

���� ∈ ℝ ����, �

���� ∈ ℝ ����, �

���� ∈ ℝ ����, �

��� ∈ ℝ ���, �

Conv1DEEG

Conv1DEMG

Conv1DGSR

Conv1DEB

���� ∈ ℝ ���� , ����

���� ∈ ℝ ���� , ����

���� ∈ ℝ ���� , ����

��� ∈ ℝ ��� , ���

EEG Modality

EMG Modality

GSR Modality

Cross-modal
Attention Module

….
Other Modalities

��� ∈ ℝ ���, �

Eye Blinking Modality

Fig. 1. Framework of the proposed Husformer, taking multi-modal data input from four example modalities: EEG, EMG, GSR, and eye blinking (EB). The
multi-modal data inputs, X(.) ∈ RL(.) ,D(.) , where L(.) and D(.) separately present the length and dimension of the input sequence of one modality, are
passed through multiple one-dimension temporal convolution layers, Conv1D(.) (Section III-B), and then encoded with positional information to produce
the low-level unimodal features Y(.) ∈ RL(.) ,D which all have the same dimension D (Section III-C); these are concatenated to generate the low-level fusion
representation YF ∈ RLF ,D . This representation is then fed alongside the unimodal low-level features of each modality into each respective cross-modal
attention transformer, wherein the target modality is adapted and reinforced according to the other resource modalities through learning the attention between its
unimodal features and the low-level representation (Section III-D). Then all reinforced unimodal features Z(.) ∈ RL(.) ,D are concatenated into the mid-level
fusion representation, which is passed through a self-attention transformer to generate the high-level fusion representation ZF ∈ RLF ,D with important
contextual information prioritized. Finally, the high-level fusion representation is transported to fully-connected layers to make predictions (Section III-E and
III-F).

cross-modal interactions by training shared representations
[12], [15], [16], [17], their network structures are too shallow
to capture complicated cross-modal dependencies and introduce sufficient complementarity across different modalities.
Additionally, the feature crafting methods required before
fusion necessitated by these approaches are costly and may
differ for different modalities, and their parameters require
prior expert knowledge and extensive cross-validation to be
optimal [18], [19]. Such expensive and inconsistent feature
engineering undermines the simplicity of the model and its
applicability to new task scenarios. Furthermore, to our best
knowledge, none of the existing multi-modal methods have
been shown to effectively predict both human affective state
and cognitive load. Most methods are tailored for a specific
combination of modalities under certain task scenarios, leading
to a potential deficiency of generality.
To address the aforementioned gaps, we present Husformer:
an end-to-end multi-modal transformer for human state recognition that efficiently learns representations of human state
from heterogeneous multi-modal streams. Figure 1 illustrates
its structure with four example input modalities: EEG, EMG,
GSR, and eye blinking (EB). The core components of Husformer are the cross-modal attention module and self-attention
module, which consist of multiple cross-modal attention transformers and one self-attention transformer, respectively. The

cross-modal attention transformers model the latent interactions across modalities by continuously adapting and reinforcing features from one modality with those of other modalities
(e.g., EEG ← EMG, GSR, and EB). Unlike direct concatenation of multi-modal modalities or learning cross-modal shared
representations through shallow neural networks, our crossmodal attention mechanism encourages the target modality to
directly attend to low-level features in other modalities where
strongly relevant and complementary information is revealed.
This leads to more adaptive and efficient complementarity
and cooperation across multiple modalities. The self-attention
transformer prioritizes important contextual information in the
fusion representation concatenated from reinforced unimodal
features of all modalities supplied by the cross-modal attention
transformers. This finally generates a weighted high-level
fusion representation based on which predictions are made.
To evaluate the performance of Husformer, we conducted
extensive experiments on four multi-modal datasets: DEAP
[20] and WESAD [21] for emotion recognition, and MOCAS
[22] and CogLoad [23] for cognitive load estimation. Additionally, We performed a comprehensive ablation study to
investigate the benefits of each module in Husformer.
The main contributions of this work can be summarized as
follows:
• Husformer is an end-to-end model that learns directly and

3

efficiently from heterogeneous multi-modal physiological
and behavioral signals without the massive feature crafting and alignment required in previous works.
• We introduce cross-modal attention transformers to fuse
features from different modalities and sufficiently model
long-term cross-modal interactions, and then a selfattention transformer to prioritize effectual contextual
information in the fusion representation.
• To the best of our knowledge, this is the first time a
generic model for human state recognition is presented
and proven to be effective for both human emotion and
cognitive load.
• Our extensive experiments on four publicly available
datasets demonstrate the benefits of the Husformer and
each of its constituent modules.
II. BACKGROUND
This section reviews existing research on human state
recognition using multi-modal fusion approaches and the preliminary transformer networks that serve as the basis of our
model.
A. Multi-Modal Fusion for Human State Recognition
Given the complex nature of human state, a single modality
is insufficient to achieve recognition with satisfactory performance in terms of accuracy and robustness, especially in
real-world task scenarios where signals are more subject to
interruption, noise, and delay [7]. To solve this issue, multimodal fusion that integrates human signals from more than one
source modality into a synchronized compact representation
has been adopted. Nevertheless, multi-modal fusion methods
for human state recognition are still at the initiatory stages
and suffer from several defects in fusion strategies. Firstly, the
direct concatenation of different modalities at sensor, feature,
or decision level has been adopted for most existing works
to produce the fusion representation [5], [13]. This fusion
strategy may result in meaningless added information or even
bias and noise owing to the feature noncommensurability
across different modalities. For instance, simply fusing HR
with other modalities can introduce additional valuable information in some task scenarios but bring disruption in others,
as HR-related signals are not only influenced by cognitive
load and emotion but also by irrelevant physical activities
[5]. Also, directly fusing different modalities whose features
have unbalanced length and temporal dimensions necessitates
considerable feature alignment preprocessing, which could
reduce the richness and diversity of the input data, potentially
limiting the model’s ability to capture complex patterns and
relationships within or across modalities. In general, simple
concatenation operations fail to model the cross-modal interactions and thus are unlikely to capture important representations
that would otherwise be revealed by multi-modal fusion.
Lately, some advanced fusion schemes have been proposed
to solve the above issues; these aim to model cross-modal
interactions by training shared representations across different
modalities. For example, [12] adopts restricted Boltzmann
machines (RBM) [24] to train a hidden layer that is expected

to learn the shared representations of sub-layers from different
modalities; [25] introduces cross-modal temporal correlations
obtained via learning shared weights across sub-layers of different modalities; and [16] utilizes deep canonical correlation
analysis (DCCA) [26] to maximize the correlations among
features of two modalities. However, due to their utilization of
shallow network structures, these existing methods are unlikely
to comprehensively capture tangled correlations across different modalities, especially from raw multi-modal signals where
features are more interrupted. Moreover, the DCCA utilized
in [16], [17] can only analyze the correlation between two
modalities, limiting the application of the model to general
task scenarios that may require the simultaneous fusion of
more than two modalities.
Additionally, existing works usually require extensive feature crafting procedures to reach sound recognition performance due to the abundant collinearity or heterogeneity in
the raw multi-modal features. In addition, different modalities
are usually preprocessed by different methods whose optimal
parameters are initially unknown. For example, in [27], infinite impulse response (IIR) high-pass and Hanning window
filters were used for EEG modality feature engineering while
continuous decomposition analysis and other physiological
libraries were utilized for GSR and HR modalities. Afterward,
independent component analysis (ICA) and manual feature
selection procedures depending on prior knowledge were
adopted to preprocess candidate features a second time. Such
involved non-uniform feature engineering procedures harm the
simplicity and universality of the model, making it difficult
to apply to general task scenarios and especially to realistic
applications, which runs counter to the original intention of
multi-modal fusion. In addition, as far as we know, existing
works only focus on the recognition of either affective state or
cognitive state, and most are tailored for specific task scenarios
and not validated on publicly available datasets; thus, the
models lack generality and replicability.
Distinct from these prior studies, our proposed Husformer
1) focuses on the general recognition of human state, including
emotion and cognitive load; 2) does not require any feature
alignment and extensive feature crafting procedures in existing
works, but rather learns from raw multi-modal feature streams;
and 3) utilizes cross-modal attention transformers as the fusion
strategy, thereby introducing efficient cooperation and complementary adaptions across modalities regardless of the number
of modalities.
B. Transformer Network
The transformer network was originally proposed by [28]
to solve sequence-to-sequence machine translation tasks in
the natural language processing area. Unlike the traditional
encoder-to-decoder structure, the transformer network adopts
a multi-head self-attention mechanism to substitute for the
attention-based convolution and recurrence layers. The selfattention mechanism aims to calculate a global representation of a sequence input that reveals meaningful contextual
information by relating different components within the sequence. Basically, a self-attention block adapts each entity in

4

a sequence by considering the global contextual information
of the whole sequence. Multi-head self-attention splits the
attention into multiple latent sub-spaces (heads), which enables
the modeling of multiple complex contextual relations across
elements in the sequence, leading to a more comprehensive
global representation.
III. A PPROACH
In this section, we present Husformer, an end-to-end multimodal transformer for multi-modal human affective and cognitive state recognition that learns the fusion representation
directly and efficiently from raw multi-modal data streams.
A. Overview
As depicted in Fig. 1, from a high-level perspective, the
Husformer uses a position-wise feed-forward process to fuse
multi-modal signal series supplied by multiple cross-modal
transformers (Section III-D). In each cross-modal transformer,
the target modality is repeatedly enhanced with low-level
features from other modalities by calculating the latent crossmodal attention between the target low-level unimodal feature and the low-level fusion representation. A sequence-tosequence model, i.e., a self-attention transformer, is then utilized to process the mid-level fusion representation sequence
consisting of all enhanced unimodal features and generate the
adaptively weighted high-level fusion representation (Section
III-E). Specifically, by computing multi-head self-attention
(Section II-B), the self-attention transformer analyzes the
pairwise relationships across elements in the mid-level fusion
representation, namely, the reinforced unimodal features, to
calculate adaptive weights at different positions and so highlight critical contextual information. Finally, the high-level
fusion representation is passed through fully-connected layers
to make predictions (Section III-F).
B. Temporal Convolutions
Let M1 , M2 , ... Mn denote n modalities. And let
X{M1 ,...,Mn } ∈ RLM1 ,...,Mn ,DM1 ,...,Mn present the raw multimodal data sequences input from these n modalities. L(.)
and D(.) represent the sequence length (e.g., channel number)
and dimension (e.g., sampling rate) of each unimodal input
respectively in this paper. The multi-modal input sequences are
passed through multiple one-dimension temporal convolution
layers with different kernels to generate multiple convoluted
sequences Ẋ{M1 ,...,Mn } ∈ RLM1 ,...,Mn ,D with the same dimension D:


Ẋ{M1 ,...,Mn } = Conv1D X{M1 ,...,Mn } , k{M1 ,...,Mn }
(1)
where, k{M1 ,...,Mn } denotes the temporal convolution kernel
sizes for n modalities M1 , M2 , ... Mn .
Each convoluted sequence aims to contain low-level temporal features of each modality. Furthermore, it is important
that after temporal convolutions, different unimodal input
sequences are projected to the same dimension, making the
dot-product calculation in the following cross-modal attention
module mathematically feasible.

C. Positional Encoding
As mentioned in the introduction of the transformer network (Section II-B), the transformer model has no inherent
awareness of the positional information of each sequence
component, such as the relative or absolute position of features
within a modality sequence. To introduce sufficient awareness
of relations across neighboring elements, i.e., features of
adjacent channels within one modality sequence, and thus
spatial information, we follow the method proposed in [28] and
apply positional encoding (PE) to the convoluted sequences
Ẋ{M1 ,...,Mn } using sin and cos functions with different frequencies. The PE of one convoluted sequence Ẋ ∈ RL,D can
be defined as a matrix:


pos
PEẊ [pos, 2k] = sin
2k
D
10000 
(2)
pos
PEẊ [pos, 2k + 1] = cos
2k
10000 D
where pos ∈ [1, ..., L] and k ∈ [0, ..., D
2 ).
Each characteristic dimension (i.e., column) of P EẊ is
a position index displayed in the sinusoidal pattern. The
calculated PEs P EẊ{M ,...,M } ∈ RLM1 ,...,Mn ,D , are then aug1

n

mented with convoluted sequences Ẋ{M1 ,...,Mn } to obtain lowlevel unimodal feature sequences with both initial temporal
and spatial information encoded YM1 ,...,Mn ∈ RLM1 ,...,Mn ,D :
Y{M1 ,...,Mn } = Ẋ{M1 ,...,Mn } + P EẊ{M ,...,M }
1

n

(3)

Additionally, the extracted low-level unimodal feature sequences of all modalities are then concatenated to produce
the low-level fusion representation YF ∈ RLF ,D :
YF = Concat (YM1 , . . . , YMn )

(4)

D. Cross-modal Attention Module
To provide sufficient complementary interactions and adaptions across different modalities, we respectively feed the lowlevel unimodal feature sequence of each modality YMi ∈
RLMi ,D with the low-level multi-modal fusion representation YF ∈ RLF ,D to a cross-modal attention module that
is comprised of multiple cross-modal transformer networks.
Each cross-modal transformer is expected to continuously
reinforce low-level unimodal features of the target modality
with those of other source modalities by learning cross-modal
attention between the input unimodal sequence and the lowlevel fusion representation. Namely, the learned cross-modal
attention inspires each target modality to directly engage
in low-level unimodal features of other source modalities,
which are encoded in the low-level fusion representation,
to adaptively find relevant and useful information that can
serve as complementary reinforcements for itself. In addition
to fostering adaptive and accurate awareness of correlations
across multiple modalities, this module can also help address
potential artifacts and feature incompatibilities across within
raw multi-modal features by disregarding irrelevant aspects,
such as interrupted or insensitive features in the low-level
unimodal feature sequence, when calculating cross-modal attention.

5

�����

���
Cross-Modal
Attention Score
�������(

�
���� ∈ ℝ��� , ��

�������(

�

�

���� ����
��

⊺

�
� ⊺
���� ����

��

∈ ℝ��� , ��

Cross-modal Attention
��(�� ⟶ ��� )

�

)����

headj

�

�

Target Unimodal Feature

�����

�� ∈ ℝ��, �

Fusion Representation

(a) Illustration of the cross-modal attention between each low-level unimodal
feature and the low-level fusion representation.
[�]

���

Cross-modal
Transformer
���(�� ⟶ ��� )

Encoder
Layer u

[�]

���

Add

×U
Multi-head
Cross-modal Attention
[�]
[�]
[�−�]
����� (�� ⟶ ��� )

�����

Norm

[�−�]

���

[�]

���

[�]

��

Encoder
Layer 0

[0]

ZF = YF

(b) Illustration of the cross-modal Transformer network stacked by
multiple identical cross-modal attention encoder layers.

[0]

ZMi = YMi

Fig. 2. Architectural description of cross-modal attention and cross-modal
transformer network.

1) Cross-modal Attention: The purpose of our cross-modal
attention is to learn an attention score between a target lowlevel unimodal feature sequence YMi ∈ RLMi ,D and the lowlevel multi-modal fusion representation YF ∈ RLF ,D , which
guides the adaption and reinforcement for the target unimodal
features using other source unimodal features embedded in the
fusion representation. We formulate unimodal Query QU ni ,
fusion Key KF us and Value VF us as:
QU ni = YMi · WQU ni
KF us = YF · WK F us

(7)

M ul
∈ RLMi ,mDV and m is the head number.
where ẎM
i
2) Cross-modal Transformer: Based on the structure of the
self-attention transformer network in [28], we build up the
cross-modal transformer using the above-defined multi-head
cross-modal attention. As shown in Figure 2b, the cross-modal
transformer with the input of the ith target unimodal feature
sequence and the fusion representation, CM TYF →YMi , is composed of multiple identical cross-modal attention encoder layers, each of which contains a multi-head cross-modal attention
block and a position-wise feed-forward network with residual
connection and layer normalization. Generally, a cross-modal
transformer network with U cross-modal attention encoder
layers calculates the reinforced unimodal feature sequence
ZMi ∈ RLMi ,D feed-forwardly as:

Add&Norm

Norm

(6)

where the softmax(.) ∈ RLMi ,LF presents the scaled crossmodal attention score matrix between the fusion representation
and the target unimodal feature.
head
We define the ẎMi j in Eq. (6) as the single-head crossmodal attention. Accordingly, the multi-head cross-modal attention between the ith target modality and the fusion representation can be formulated as:
M ul
ẎM
= CMM ul (YF → YMi )
i


head1
headm
= Concat ẎM
,
.
.
.
,
Ẏ
Mi
i

Feed-Forward

�����

= CM (YF → YMi )
= Attention(QjU ni , KFj us , VFj us )


>
j
j
Q
·
K
= softmax  U ni√ F us  · VFj us
DK

�

�����

��� ∈ ℝ��� , �

ẎMi

�
���� ∈ ℝ��, ��

�
���� ∈ ℝ��, ��

)

�����

�����

RD,DV are learnable weights.
As depicted in Figure 2a, similar to the self-attention process described in [28], the latent adaption and reinforcement
from the fusion representation to the target unimodal feature,
head
i.e., the learned cross-modal attention ẎMi j ∈ RLMi ,DV , in
the j th head cross-modal attention can be defined as:

(5)

VF us = YF · WV F us
where WQU ni ∈ RD,DQ , WK F us ∈ RD,DK and WV F us ∈

[u]

[u]

[0]

[u−1]

ẐMi = CMM ul (L(ZF ) → L(ZMi

))

(8)

[u]
[u]
[u−1]
ŻMi = ẐMi + L(ZMi )
[u]
[u]
[u]
ZMi = Fθ (L(ŻMi )) + L(ŻMi )

where u ∈ [1, U ] presents the uth cross-modal attention
encoder layer, L(.) and Fθ (.) denote the layer normalization
operation and position-wise feed-forward network with a parameter set θ respectively.
In each cross-modal transformer, the target unimodal features are continuously encoded and enhanced with external
information from other source unimodal features embedded
in the fusion representation. Specifically, the low-level unimodal features of source modalities from the low-level fusion
representation are converted to different pairs of the fusion
Keys and Values in Eq. (5) to compute the multi-head crossmodal attention of the target modality in Eq. (7). Following the

6

process described in Eq. (8), each target modality is merged
with other source modalities by a position-wise feed-forward
process from each cross-modal transformer.
E. Self-attention Module
The outputs of the cross-modal attention module, to wit the
[U ]
reinforced unimodal features ZMi ∈ RLMi ,D of all modalities,
are concatenated together as a sequence, i.e., the mid-level
fusion representation Z{M1 ,...,Mn } ∈ RLM1 ,...,Mn ,D , which is
transported to the self-attention transformer [28] described in
Section II-B to produce a weighted high-level fusion representation ZF ∈ RLF ,D with meaningful contextual information
highlighted. Specifically, the self-attention transformer assigns
adaptive self-attention scores to different positions in the input
sequence, i.e., each minimum unit of the reinforced unimodal
features of different modalities, by integrating the global
contextual information of the whole sequence. As a result,
reinforced unimodal features in the sequence are given adaptive weights to generate the high-level global representation,
where important contextual information for the recognition of
the human state, e.g., sensitive and efficient features under a
certain task scenario, is highlighted, while unimportant one,
e.g., disrupted and insensitive features, is neglected. Such an
adaptive self-attention process further reduces the potential
feature noncommensurability across modalities and improves
the efficiency of the model.

Algorithm 1 Procedures of Husformer Training
1: Given multi-modal data series X{M1 ,...,Mn } of n modalities and true classification labels y
2: Given training steps T
3: Initialize convolution kernel sizes k{M1 ,...,Mn } , crossmodal attention weights WQU ni , WKF us , WVF us , selfattention weights WQ , WK , WV and other model parameters
4: t ← 0
5: // Convolutions and Positional Encoding
6: Compute
low-level unimodal feature sequences
Y{M1 ,...,Mn } by (1)-(3)
7: Compute low-level fusion representation YF by (4)
8: while t < T do
9:
// Cross-modal Attention Module
10:
Compute reinforced unimodal feature sequences
Z{M1 ,...,Mn } by (8)
11:
// Self-attention Module
12:
Compute high-level fusion representation ZF
13:
// Linear Layers
14:
Compute prediction probability P by (9)
15:
// Model Optimization
16:
Compute loss F L(y, p) by (10)
17:
Update model parameters using back propagation
18:
t←t+1
19: end while

IV. E XPERIMENT AND R ESULTS
F. Model Training
At the final step, the output of the self-attention module,
namely, the high-level global feature ZF ∈ RLF ,D , is passed
through two linear layers with a residual connection operation
and a softmax nonlinear activation function that calculates
prediction probabilities P as:
Ẑ = ZF + ζρ (ZF )
P = softmax(ζβ (Ẑ))

(9)

where ζρ and ζβ present two linear layers with parameter sets
ρ and β, and P contains the prediction probability for each
class.
To reduce the class imbalance resulting from biased data
distribution and varied recognition difficulty, we adopt the
multi-class focal loss function [29] for training, which is
formulated as:
F L(y, p) = −

C
X

αc yc (1 − pc )γ log(pc )

(10)

c=1

where C is the total number of classes, yc and pc correspond to
the true label and predicted probability for class c respectively,
αc and γ present the balancing parameter that controls the
trade-off between the positive and negative samples within
class c and a focusing parameter that down-weights the
contribution of well-classified samples respectively.
The overall procedures of Husformer model training are
summarized in Algorithm 1.

In this section, we describe the experiments we conducted
on four public multi-modal datasets for human affective and
cognitive state recognition. These datasets are widely used
in the field, and we compared our Husformer with four
state-of-the-art baselines of multi-modal fusion-based human
state recognition. For each dataset, the performance of using
a single modality was also reported to investigate if our
proposed multi-modal fusion-based Husformer can outperform
recognition methods using a single modality. Additionally, we
conducted an ablation study to evaluate the benefits of each
module in Husformer.
A. Datasets
We selected two multi-modal affective datasets: DEAP
[20] and WESAD [21], and two multi-modal cognitive load
datasets: MOCAS [30] and CogLoad [23] for experiments.
The DEAP dataset contains multiple physiological signals,
including EEG, EMG, EOG, and GSR, collected from 32
participants watching 40 different music video clips that
elicited different emotional states. After each video clip,
participants were requested to report their affective state levels
in terms of arousal, valence, liking, and dominance from
1 to 9 using the Self-Assessment Manikin (SAM). In our
experiment, we utilized two versions of the DEAP dataset:
the downloaded Data-original.zip, which contained collected
raw multi-modal features, was regarded as the raw DEAP
dataset, while the downloaded Data-preprocessed.zip, which

7

TABLE I
D ESCRIPTION OF UTILIZED MODALITIES IN THE DEAP DATASETS .
F REQUENCY: TIME SAMPLING RATE ; C HANNELS : NUMBER OF CHANNELS ;
AND A RRAY S HAPE : NUMBER OF DATA ROWS × CHANNELS× FREQUENCY.

Modality
EEG
EMG
EOG
GSR
Modality
EEG
EMG
EOG
GSR

Raw DEAP Dataset
Frequency
Channels
Array Shape
512
32
69535×32×512
512
4
69535×4×512
512
4
69535×4×512
512
1
69535×1×512
Preprocessed DEAP Dataset
Frequency
Channels
Array Shape
128
32
80640×32×128
128
2
80640×2×128
128
2
80640×2×128
128
1
80640×1×128

TABLE II
D ESCRIPTION OF UTILIZED MODALITIES IN THE WESAD DATASET
Modality
GSR (chest)
BVP (wrist)
EMG (chest)
ECG (chest)
RESP (chest)
GSR (wrist)

WESAD Dataset
Frequency
Channels
700
1
64
1
700
1
700
1
700
1
4
1

Array Shape
27287×1×700
27287×1×64
27287×1×700
27287×1×700
27287×1×700
27287×1×4

crafted features with several procedures2 , was regarded as the
preprocessed DEAP dataset. Valence and arousal were selected
as the evaluation criteria of human emotion, where we mapped
the scales (1-9) into three levels: “negative” or “passive”(1-3);
“neutral” (4-6); and “positive” or “active” (7-9).
The WESAD dataset contains physiological data, consisting
of GSR, BVP, EMG, ECG and respiration (RESP), collected
by one chest-worn and one wrist-worn wearable sensor from
15 participants who conducted different tasks that aimed to
elicit different emotional states. Specifically, participants were
asked to close their eyes for seven minutes to stimulate the
neutral state. The stress state was elicited by the Trier Social
Stress Test (TSST) [31], where participants delivered a fiveminute speech on their personal traits to three-person panels.
Moreover, for the amusement state, participants were required
to watch funny videos for 392 seconds. After each task,
participants reported subjective emotional states using SAM
and other self-report questionnaires, and three kinds of labels,
i.e., neutral vs. stress vs. amusement, were provided.
The MOCAS dataset contains physiological data, including
5-channel EEGs, EEG band powers (or EEG POW, including
theta, low and high beta, alpha, and gamma bands of 5channel EEGs), BVP, GSR and HR, and behavioral data,
including Eye Aspect Ratio (EAR) and Action units (AUs),
from 21 participants conducting Closed-Circuit Television
(CCTV) monitoring tasks that aimed to elicit different levels
of cognitive load. After each task, participants were required to
report subjective cognitive load via NASA-TLX. Based on the
weighted NASA-TLX scores, three categories of annotations,
i.e., low vs. medium vs. high, of cognitive load were given.
Apart from the raw multi-modal physiological and behavioral
2 https://www.eecs.qmul.ac.uk/mmv/datasets/deap/readme.html

TABLE III
D ESCRIPTION OF UTILIZED MODALITIES IN THE MOCAS DATASETS
Raw MOCAS Dataset
Frequency
Channels
Array Shape
128
6
215341×5×128
8
25
215341×25×8
128
1
215341×1×128
6
1
215341×1×6
1
1
215341×1×1
Preprocessed MOCAS Dataset
Modality
Frequency
Channels
Array Shape
EEG
128
6
215341×5×128
EEG POW
8
25
215341×25×8
BVP
128
1
215341×1×128
GSR
6
1
215341×1×6
EAR
1
1
215341×1×1
Modality
EEG
EEG POW
BVP
GSR
EAR

TABLE IV
D ESCRIPTION OF UTILIZED MODALITIES IN THE C OG L OAD DATASET
Modality
HR
IBI
GSR
SKT
ACC

CogLoad Dataset
Frequency
Channels
1
1
1
1
1
1
1
1
1
2

Array Shape
89225×1×1
89225×1×1
89225×1×1
89225×1×1
89225×2×1

features collected from two off-the-shelf wearable sensors:
Empatica E4 and Emotiv Insight, and a webcam, the MOCAS
also contains the data prepossessed by NeuroKit2 [32] and
other methods [30]. In this experiment, we utilized both raw
and preprocess MOCAS dataset.
The CogLoad dataset contains physiological signals, including HR, IBI, GSR, SKT, and motion data (ACC) collected from 23 participants through a Microsoft band. The
participants conducted six dual tasks including primary and
secondary cognitive-load tasks that were expected to stimulate
target levels of cognitive load. The primary task was randomly
selected from six psycho-physiological tests proposed in [33].
The secondary task was to click on the appearing target on
screen while conducting the primary task. After each task, participants were asked to report subjective cognitive load based
on the TLX mental demand of NASA-TLX questionnaire, and
the data collected in the baseline (rest) section was labelled as
-1. In our experiment, we mapped the subjective scales into
three classes of labels as low (-1-3), medium (4-6) and high
(7-9).
In our experiments, to simulate realistic application scenarios, the input sample of each modality is a 2-D feature
matrix extracted from a 1-second segment with the dimension
of LMi × DMi , i.e., the channel number plus the sampling
frequency of the modality. The details of the modalities used
in the aforementioned datasets are described in Table I-IV.
B. Baselines
We selected four state-of-the-art baselines of multi-modal
fusion-based human state recognition as the comparisons with
our proposed Husformer:
• EF-SVM: Support Vector Machines (SVMs) with early
fusion [34], [35], [19]. This is a popular benchmark

8

model for human state recognition, which concatenates
different modalities together at the sensor or feature
level and builds a SVM as the classifier for the fused
representation.
• LF-SVM: SVMs with late fusion [36], [35], [19]. This
model is also a strong benchmark to predict human state,
which fuses different modalities at the decision level.
Each modality is processed with a SVM classifier to
make individual predictions, which are combined together
through a voting scheme. In our experiment, we select
Dempster-Shafer Theory (DST) voting [37], [38], which
is reported as the best performing voting scheme by [19],
as the late fusion process.
• EmotionMeter [15]. This is a multi-modal deep learning
model for human emotion prediction. Multiple individual
RBMs are built to process features of each modality,
where the hidden layers of those individual RBMs are
concatenated together to learn the shared representations
across different modalities. Then a linear SVM is adopted
for classification using the learned shared representations.
• MMResLSTM: Multimodal Residual Long Short-Term
Memory Neural Network [39]. This is another stateof-the-art multi-modal deep learning model for human
emotion recognition. Multiple individual LSTM blocks
are constructed for each modality, where each LSTM
layer of these individual LSTM blocks shares the same
weights to learn the temporal correlations across different
modalities. Finally, the outputs of all LSTM blocks are
concatenated and passed through a dense layer to make
predictions. Layer normalization and residual connection
are also applied to accelerate training process.
Moreover, to comprehensively evaluate if our Husformer
could improve the performance compared with using single
modality, we also implemented Long Short-Term Memory
Nerual Network (LSTM) [40], Graph Neural Network (GNN)
[41] and Transformer [28], which were broadly utilized for
single-modal-based human state recognition [42], [43], [44],
to test each single modality in each dataset, and the best
performing results among these three models were reported.
C. Ablation Study
To evaluate the benefits of each module in our Husformer,
we also built three ablation models for the ablation study:
• HusFuse: Deleting the cross-modal attention module in
the Husformer, and fusing the low-level features of all
modalities directly to the self-attention module.
• HusLSTM: Replacing the self-attention transformer in the
Husformer with an LSTM layer.
• HusPair: Replacing the cross-modal attention module
in the Husformer with the directional pairwise crossmodal attention widely adopted in multi-modal natural
language processing and computer vision areas [45],
[25]. However, Husformer differs from HusPair in that
it focuses on the cross-modal attention between each
individual modality and the multi-modal fusion signal,
rather than between one single modality and another. This
allows the model to consider the coordination of more

than a pair of modalities at the same time, reducingthe
potential information redundancy caused by parallel pairwise fusion.
D. Evaluation and Metrics
For each dataset, we randomly shuffled all data and conducted the K-folder cross validation (K = 10) [46], [47]. We
reported the average multi-class accuracy (Acc) and multiclass average F1-score (F 1) [48] with standard deviations for
each model in each experiment. Furthermore, samples from
the same trials, such as a film clip in the DEAP dataset or
a monitoring task in the MOCAS dataset, were exclusively
included in either the training or test sets.
Assuming the number of classes in a multi-class classification task is n, we can define Acc as:
Pn
accuracyi
(11)
Acc = i=1
n
where accuracyi denotes the binary accuracy of the ith class
as:
T Pi + T Ni
accuracyi =
(12)
lenthi
where T Pi , T Ni and lenthi present true positive samples, true
negative samples, and the number of total samples in the ith
class, respectively.
The F 1 can be formulated as:
Pn
f 1i
(13)
F 1 = i=1
n
where f 1i denotes the binary F1 score of the ith class as:
f 1i =

2 × precisioni × recalli
precisioni + recalli

precisioni =
recalli =

T Pi
T Pi + F Pi

T Pi
T Pi + F Ni

(14)
(15)
(16)

where precisioni and recalli present the binary precision and
recall separately, F Pi and F Ni denote false positive and false
negative samples in the ith class respectively.
We also conducted two-sample independent t-tests to compare the classification results of different models, and the
significance is asserted when p < 0.01.
E. Implementation Details
All training and experiments were conducted on a NVIDIA
Tesla V100 GPU. We trained all baseline networks by following the implementation procedures described in their respective original papers. Also, for the SVM classifier, we
followed the approach descrbied in [19] to select the Radial
Basis Function (RBF) kernel and optimize the values of C and
γ. Note that EF-SVM cannot be directly applied to unaligned
datasets, which means datasets that contain multiple modalities
with different time sampling rates, since the concatenation
operation is mathematically impossible due to different feature
dimensions. Therefore, we added multiple one-dimensional

9

convolution sub-networks with the same structures and parameters as those in the Husformer before the EF-SVM to extract
low-level unimodal features with the same dimension for the
concatenation operation on unaligned datasets, i.e., WESAD
and MOCAS datasets. Furthermore, to ensure a fair comparison, we kept the hyper-parameters of ablation models the same
as those in the Husformer during the experiments. The hyperparameters of the Husformer used in each experiment can be
found in Appendix A.
F. Quantitative Measurements
1) Comparative Results with Baselines: Table V and VI
summarize the performance of our Husformer when compared
with the four multi-modal baselines of human state recognition in terms of average Acc and F 1 with stand deviations
during the experiments. From the comparative results, we can
observe that regarding the overall performance, our proposed
Husformer outperforms the other four state-of-the-art baselines
in the recognition of both human affective state (DEAP and
WESAD datasets) and cognitive load (MOCAS and CogLoad
datasets), which demonstrates that the Husformer can serve
as a more effective backbone network for general human
state prediction tasks. Also, such performance enhancements
are more evident on datasets containing raw multi-modal
features (raw DEAP and MOCAS, WESAD and CogLoad
datasets), which shows that the Husformer can learn from
raw multi-modal features series more efficiently, and thus is
more applicable to real-world task scenarios where extensive
feature crafting and alignment is quite impractical and expensive. These performance improvements mainly result from
following reasons:
• Experimental results show that our Husformer significantly outperforms EF-SVM and LF-SVM regarding
Acc and F 1 on each dataset. This is reasonable since
these two methods generate fusion representations by a
simple concatenation of multiple modalities at the feature or decision level, which ignores correlations across
different modalities, and may be limited by the curse
of dimensionality [49], while the Husformer fuses all
unimodal features through a feed-forward process from
multiple cross-modal attention transformers, where the
complementary interactions among different modalities
are considered sufficiently. Such neglect of cross-modal
interactions even causes the performance of EF-SVM and
LF-SVM to underperform compared to EmotionMeter
and MMResLSTM by approximately 10% in terms of
accuracy (Acc) and F1 score (F 1). These comparative
results highlight the significance of awareness of correlations across multiple modalities for multi-modal human
state recognition. Moreover, the SVM, which serves as
the backbone model in both EF-SVM and LF-SVM, has
been shown to have low efficiency when dealing with
a large amount of input data [50]. Additionally, it is
quite sensitive to missing data, outliers, and noise [51],
making it heavily reliant on careful data cleaning and
feature selection procedures. Therefore, while some studies demonstrate that EF-SVM and LF-SVM are sufficient

to estimate human state with a high volume of data
cleaning and careful feature engineering [39], [19], it is
reasonable to conduct that EF-SVM and LF-SVM may
not perform well when dealing with relatively raw multimodal features.
• The experimental results show that our proposed Husformer significantly outperforms EmotionMeter in terms
of Acc and F 1 on all datasets. We argue that this is
due to the fact that although EmotionMeter considers
cross-modal interactions by building shared hidden layers
to learn shared representations, the constructed RBMs
neglect time-related cross-modal interactions, which are
critical as the temporal characteristic is a vital attribute
of signals reflecting human state [5], [39]. On the other
hand, MMResLSTM achieves better performance than
EmotionMeter and even performs competitively with our
Husformer on the preprocessed DEAP and preprocessed
MOCAS datasets. This is because MMResLSTM constructs LSTM layers that share the same weights for
different modalities to learn shared representations, which
can model the temporal cross-modal correlations effectively. However, on the other four datasets that involve
minimal or no feature engineering, Husformer consistently and significantly outperforms the MMResLSTM in
terms of both Acc and F 1. We argue that the reason
lies in the fact that our proposed cross-modal attention
mechanism considers cross-modal interactions by encouraging one modality to directly engage in the unimodal
features of other modalities where strongly complementary representation information is presented to enhance
itself. Such an attention-based fusion strategy can capture
complicated and long-term complementary interactions
across multiple modalities more efficiently, especially
when dealing with raw multi-modal features involving
less feature engineering.
• The performance improvements of our proposed Husformer compared to the other four baselines become more
pronounced when transitioning from datasets that involve
feature engineering preprocessing to raw datasets. This
can be attributed to the cross-modal attention and selfattention modules utilized in the Husformer. Specifically,
in addition to modeling the complicated complementary
multi-modal interactions from raw multi-modal signals
efficiently, the cross-modal transformers can neglect irrelevance, such as interrupted or insensitive low-level
unimodal features, when calculating adaptive cross-modal
attention. Moreover, the self-attention process further prioritizes the important information and diminishes the irrelevance and interruptions by assigning adaptive weights
to each element in the sequence consisting of reinforced
unimodal features of different modalities. Compared to
the baselines, these two adaptive attention mechanisms,
at the fusion process and high-level feature level, respectively, can help the model to distill effective representations of human state without being disturbed by potential
feature noncommensurability and interruptions in the raw
multi-modal features to a greater extent.

10

TABLE V
P ERFORMANCE OF DIFFERENT MODELS ON RAW DEAP AND PREPOSSESSED DEAP DATASETS IN TERMS OF AVERAGE MULTI - CLASS AVERAGE
ACCURACY (Acc) AND MULTI - CLASS AVERAGE F1- SCORE (F 1) WITH STAND DEVIATIONS . R ESULTS OF OTHER MODELS THAT ARE WITHIN 5% OF
Husformer’ S PERFORMANCE ON Acc OR F 1 ARE HIGHLIGHTED . h : HIGHER VALUES INDICATE BETTER PERFORMANCE .
Dataset
Criteria
Metric
EF-SVM
LF-SVM
EmotionMeter
MMResLSTM
HusFuse
HusLSTM
HusPair
HusFormer

Raw DEAP
Valence
Arousal
Acc(%)h
F 1(%)h
Acc(%)h
F 1(%)h
43.95±2.17
47.36±2.53
46.02±2.10
48.69±2.16
45.09±4.82
49.90±5.18
48.18±4.01
51.40±3.96
61.71±3.45
62.00±3.39
62.08±3.16
62.18±3.08
65.68±2.13
66.39±2.05
66.31±1.78
66.39±1.86
67.45±3.23
68.64±3.16
67.85±2.67
68.08±2.53
72.66±2.34
73.09±2.37
71.03±1.90
71.40±1.93
77.14±2.40
76.71±2.18
77.55±2.22
77.05±2.09
79.64±1.52
79.87±1.54
79.94±2.18
80.44±2.25

Preprocessed DEAP
Valence
Arousal
Acc(%)h
F 1(%)h
Acc(%)h
F 1(%)h
70.68±6.30
72.40±6.52
71.04±5.97
71.18±6.14
67.59±5.60
69.37±5.77
70.24±4.95
71.11±5.09
85.26±2.52
79.59±2.70
80.02±3.32
80.18±3.25
86.78±2.56
87.03±2.55
86.55±1.78
87.13±2.25
80.48±1.58
80.77±1.71
81.26±1.38
81.42±1.48
83.41±1.90
84.15±2.09
84.61±1.58
84.73±1.47
89.42±3.33
89.26±2.99
90.31±2.99
90.15±3.06
90.67±2.20
90.74±2.29
91.33±1.59
91.35±1.67

TABLE VI
P ERFORMANCE OF DIFFERENT MODELS ON WESAD, RAW MOCAS, PREPROCESSED MOCAS, AND C OG L OAD DATASETS IN TERMS OF AVERAGE
MULTI - CLASS AVERAGE ACCURACY (Acc) AND MULTI - CLASS AVERAGE F1- SCORE (F 1) WITH STAND DEVIATIONS . R ESULTS OF OTHER MODELS THAT
ARE WITHIN 5% OF THE Husformer’ S PERFORMANCE ON Acc OR F 1 ARE HIGHLIGHTED . h : HIGHER VALUES INDICATE BETTER PERFORMANCE .
Dataset
Metric
EF-SVM
LF-SVM
EmotionMeter
MMResLSTM
HusFuse
HusLSTM
HusPair
HusFormer

WESAD
Acc(%)h
42.46±4.34
44.98±2.48
63.01±1.41
65.76±1.12
68.77±1.56
70.64±1.21
73.57±1.72
78.68±2.05

Raw MOCAS

F 1(%)h
44.39±4.08
47.51±3.00
63.21±1.34
66.32±1.24
68.48±1.31
71.00±1.28
73.77±2.13
79.51±2.28

Acc(%)h
51.48±4.39
48.74±3.40
71.15±3.48
75.33±2.41
70.65±2.36
78.98±2.72
82.12±1.83
87.37±2.40

Multi-class Average Accuracy (Acc) (%)

100

90

80

70

60

50
0

Husformer
Raw
DEAP
(Valance)

Raw
DEAP
(Arousal)

MMResLSTM

Single Modality

Preprocessed Preprocessed
DEAP
DEAP
(Valance)
(Arousal)

WESAD

Raw
MOCAS

Preprocessed
MOCAS

Preprocessed MOCAS

F 1(%)h
51.63±5.00
48.85±3.42
70.98±3.39
75.44±2.21
71.22±2.39
79.28±2.61
82.46±1.63
87.47±2.55

CogLoad

Dataset

Fig. 3. Performance comparison of our proposed Husformer and the bestperforming multi-modal fusion baseline, MMResLSTM, with the highest
recognition result achieved using a single modality on each dataset in terms
of accuracy (Acc).

2) Comparative Results with Single Modality: Figure 3
presents a comparison of the performance of our proposed Husformer and the best-performing multi-modal fusion
baseline, MMResLSTM, with the highest recognition result
achieved using a single modality on each dataset, in terms
of accuracy (Acc). Details of the classification outcomes for
each individual modality in every dataset, with respect to
both accuracy (Acc) and F1 score (F 1), can be found in
Appendix B. As shown in Figure 3, our proposed Husformer

Acc(%)h
62.73±4.91
59.80±5.16
78.80±2.54
82.81±1.34
78.00±2.10
82.40±1.80
88.83±3.97
90.09±2.25

F 1(%)h
61.87±4.19
60.68±5.07
79.94±2.61
83.25±1.40
78.81±1.86
82.54±1.78
88.75±3.99
90.17±2.17

CogLoad
Acc(%)h
41.67±3.80
38.98±2.71
59.57±1.42
61.44±1.67
58.49±0.68
67.09±1.06
65.11±3.07
74.06±2.48

F 1(%)h
47.52±3.14
45.87±2.12
62.99±1.30
63.39±1.71
57.65±0.83
66.60±1.02
66.55±3.25
74.93±2.77

significantly outperforms the best-performing single-modalbased recognition results on all four datasets in terms of Acc.
We assume that these improvements result from the fact that
our Husformer can effectively leverage the aforementioned
advantages of multi-modal fusion for human state recognition,
such as combing metrics from different sources to unveil
essential representation information that cannot be obtained
from one single source [52] and reducing the noise-to-signal
ratio [5]. Additionally, we observe that the MMResLSTM,
another multi-modal fusion-based model that achieves the
best performance among all baselines, does not outperform
the single EEG-related modality-based recognition with transformer networks on the raw DEAP and preprocessed MOCAS
datasets. This demonstrates that the cross-modal attention and
self-attention processes in our Husformer enable it to take
greater advantage of multi-modal fusion by modeling sufficient
and long-term complementary cross-modal correlations and
adaptively highlighting meaningful contextual representations.
3) Results of the Ablation Study: Table V and VI present
the performance of the Husformer in terms of Acc and F 1 on
each dataset, compared to three ablation models, the HusFuse,
HusPair, and HusLSTM. From the results, we can observe
the effectiveness of the cross-modal and self-attention module
inside the Husformer as follows:
Effectiveness of the cross-modal attention module
• Compared to the HusFuse, which removes the crossmodal attention module from the Husformer, the Husformer achieves an absolute improvement in terms of
Acc and F 1. This improvement can be attributed to

11

TABLE VII
T HE NUMBER OF PARAMETERS AND GPU MEMORY USAGE DURING TRAINING OF THE Husformer AND Huspair ON EACH DATASET. N OTE THAT BOTH
MODELS HAD THE SAME BATCH SIZE DURING TRAINING . PARA : THE NUMBER OF PARAMETERS ; M EM : GPU MEMORY USAGE .
Raw DEAP
Para
Mem
2.90M
3253.24MB
0.63M
1999.56MB

Preprocessed DEAP
Para
Mem
2.92M
2908.66MB
0.66M
1773.59MB

Para
3.90M
0.71M

WESAD
Mem
5416.90MB
3084.35MB

the fact that the HusFuse fuses low-level features from
different modalities together using a simple concatenation
operation, instead of utilizing a feed-forward process
from cross-modal transformers. While the self-attention
processes in the HusFuse can be viewed as a way to consider cross-model ineractions, by relating entities of the
sequence concatenated from low-level features of different modalities to contextual information to calculate the
high-level fusion representation, it fails to provide direct
complementary adaptions for features of one modality
with those of other modalities during the fusion process.
This result highlights the effectiveness of the proposed
cross-modal attention-based fusion strategy compared to
simple concatenation with self-attention.
• Compared with the HusPair, which replaces the crossmodal attention module in the Husformer with the directional pair-wise cross-modal attention in [25], the
Husformer achieves an improvement in Acc and F 1.
We argue that this is because the pairwise cross-modal
attention in the HusPair can only consider the complementary interactions between a pair of modalities at once
and thus ignores the coordination among more than two
modalities. In contrast, our proposed cross-modal attention computes the complementary interactions between
the low-level unimodal features of one target modality
and the low-level fusion representation embedded with
unimodal features of the other source modalities. This
allows the model to consider the coordination across
all modalities at the same time, hence considering more
long-term and comprehensive cross-modal interactions.
Moreover, it has been shown that the pairwise fusion
approach can produce redundant fusion information that
may serve as additional noise rather than effectual multimodal features [53], [54].
• We can also observe that HusPair is quite comparable to
the Husformer on the DEAP datasets and preprocessed
MOCAS dataset. However, as presented in Table VII, the
parameter number of the HusPair is about 4∼8 times
that of the Husformer. This is because the number of
the pairwise cross-modal attention transformers in the
HusPair increases exponentially with the increase in
the number of modalities. Specifically, when applied to
multi-modal fusion of n modalities, the pairwise crossmodal attention requires n2 −n cross-modal transformers,
while ours only requires n of them. For instance, on
the WESAD dataset that contains six modalities, the
HusPair requires 30 cross-modal transformers, while our
Husformer only requires 6 of them. Such a high volume
of parameters can result in slow convergence and high

Raw MOCAS
Para
Mem
6.21M 519.67MB
0.74M 210.97MB

Mean Average Error (MAE)

Dataset
Metric
HusPair
Husformer

1.1
1.0
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1

Preprocessed MOCAS
Para
Mem
6.22M
531.80MB
0.75M
220.53MB

Cogload
Para
Mem
3.12M
202.47MB
0.72M
94.28MB

Preprocessed DEAP & MOCAS Validation Performance

MOCAS-Husformer
MOCAS-HusPair
DEAP-Husformer
DEAP-HusPair

0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40

Epochs

Fig. 4. Learning curves of Husformer when compared to HusPair on
the preprocessed DEAP and MOCAS datasets in terms of validation set
convergence.

training difficulty. For example, on the preprocessed
DEAP and the preprocessed MOCAS, where the HusPair
gets its most competitive results, we empirically observe
that the Husformer can converge faster to a lower loss of
mean absolute error compared to the HusPair during the
training process (see Figure 4). Furthermore, we assume
that the pairwise cross-modal attention in HusPair with
potential over-parameterization issues may also produce
attention redundancy when the feature complexity cannot
meet that of the attention, e.g., regarding the noise or
artifacts as a part of the learned attention. For example, on
the CogLoad dataset, where the feature length (channel
number) and dimension (sampling frequency) are very
shallow compared to those in other datasets, the performance of the HusPair drops significantly, exceeded by the
Husformer by 10% (see Table V and VI). Therefore, the
high storage and computation costs (see Table VII), and
the potential ineffectiveness on shallow modalities due to
the large number of parameters limit the applicability of
the HusPair to general and practical scenarios of multimodal human state recognition. On the other hand, our
Husformer achieves comparable and even significantly
superior performance with much fewer parameters than
the HusPair. These results highlight the effectiveness
of our proposed cross-modal attention compared to the
directional pairwise attention approach.
Effectiveness of the self-attention module
•

Compared with the HusLSTM that replaces the selfattention module in the Husformer with an LSTM layer,
the Husformer achieves an absolute improvement on Acc
and F 1. We believe that such an improvement results
from the fact that the self-attention mechanism applied in

12

8 - 32

BVP
High

GSR
EAR

Target Modality

EEG

EEG
POW

Low
Length

Batch_1

Each segment divided by pink dot lines is a crossmodal attention score matrix between a target
modality and the fusion representation on one batch.

Batch_2

Fusion Representation

1-7

Batch_3

Each block is a cross-modal attention score between the
minimum units (summarized by temporal convolutions)
of a target modality and the fusion representation.

Length

Fig. 5. Visualization of an example cross-modal attention weight group consisting of learned cross-modal attention matrices at the final layer of each crossmodal transformer within 3 batches during the training on the raw MOCAS dataset. Note that the cross-modal attention score matrix between a target low-level
unimodal feature and the low-level fusion representation on one batch has the dimension of LMi × LF , i.e., the length of the target unimodal feature plus
that of the fusion representation (BVP, GSR and EAR: 1 × 33; EEG: 5 × 33; EEG POW: 25 × 33).

the transformer network [28] enables the model to capture
long-term temporal dependencies by considering the sequence consisting of all reinforced unimodal features as a
whole. In contrast, the LSTM processes the sequence element by element, which can suffer from long-dependency
issues [55]. Moreover, the self-attention mechanism can
adaptively highlight meaningful contextual information
while reducing useless one by computing adaptive attention scores at a different position in the sequence, leading
to a more effectual global representation of the human
state. This result confirms the effectiveness of the selfattention module in the Husformer.
G. Qualitative Analysis
To demonstrate how the cross-modal attention and selfattention in the Husformer work when learning from multimodal signals of the human state, we visualize the attention
activation for qualitative analysis. Figure 5 shows an example cross-modal attention weight group consisting of learned
cross-modal attention matrices at the final layer of each crossmodal transformer within 3 batches during the training on
the raw MOCAS dataset. Note that the original cross-modal
attention score matrix between a target low-level unimodal
feature and the low-level fusion representation on one batch
has the dimension of LMi × LF (see Figure 2a). We can observe that the cross-modal attention has learned how to attend
to positions revealing relevant and meaningful information
across the target modality and source modalities embedded in
the fusion representation without the requirement of feature
alignment. For instance, higher cross-modal attention scores
are assigned to some intersections of the BVP and GSR unimodal features and part of the EEG POW unimodal features
embedded in the later part (8-32) of the fusion representation.
This shows that our cross-modal attention can reveal cross-

modal contingencies that are inaccessible with manual feature
alignment.
Furthermore, we can observe that the learned cross-modal
attention is adaptive; i.e., different cross-modal attention patterns are learned between different target modalities and the
fusion representation. Moreover, these patterns may differ for
the same target modality across different patches. For instance,
while the EEG modality is always encouraged to attend to
the later section (8-32) of the fusion representation, i.e., the
EEG POW modality, the intersections assigned with higher
cross-modal attention scores vary across different batches.
However, despite the above adaptive differences, we can notice
that some stable and consistent cross-modal attention patterns
exist in different batches of the same target modality. For
example, higher cross-modal attention scores are always assigned to the intersections of the EEG POW unimodal features
and the BVP, GSR and EEG unimodal features embedded in
the front positions (1-7) in the fusion representation. These
observations over the visualized cross-modal attention score
matrices demonstrate that our proposed cross-modal attention
module can capture and model an adaptive but relatively
consistent and long-term pattern of cross-modal interactions.
Figure 6a shows an example of the output of the selfattention transformer in the Husformer, namely the highlevel fusion representation ZF ∈ RLF ,D , within 3 batches
during training on the raw MOCAS dataset. We can observe
that the self-attention has learned how to prioritize important contextual information for human state recognition and
reduce insignificant one by assigning high (blue) and low
(red) self-attention scores to different positions in the fusion
representation. Also, similar to the cross-modal attention, the
learned self-attention is adaptive; i.e., different minimum units
in the fusion representation are assigned with self-attention
scores in different patterns, especially from the ‘Dimension’

13

BVP
GSR
EEG

EEG POW

EAR

Dimention

High

Batch_2

Batch_1

Each segment divided by black dot lines is a high-level
fusion representation generated by the self-attention
process on one batch.

Length

Low

Each block is a minimum unit in the high-level fusion
representation assigned with a self-attention score.

(a)

BVP
GSR
EEG

Batch_3

EEG POW

EAR

Dimention

High

Batch_1

Each segment divided by black dot lines is a high-level
fusion representation generated by the self-attention
process on one batch.

Batch_2

Length

Batch_3

Low

Each block is a minimum unit in the high-level fusion
representation assigned with a self-attention score.

(b)
Fig. 6. Visualization of an example high-level fusion representation group generated by the final layer of the self-attention transformer in the (a) Husformer
and (b) HusFuse within 3 batches during the training on the raw MOCAS dataset. Note that the high-level fusion representation produced on one batch has
the dimension of LF × D, i.e., 33 × 30.

axis. Meanwhile, we can notice that the learned self-attention
patterns of the features of the same modality share many
similarities across different batches, especially for EEG and
EEG POW modalities. These observations demonstrate that
the self-attention in the Husformer can highlight effectual
contextual features of human state and diminishes ineffectual
ones in the fusion representation with an adaptive while
relatively steady pattern.
Meanwhile, Figure 6b depicts an example of the high-level
fusion representation output of the self-attention transformer
in the HusFuse. Note that the inputs of the self-attention transformers in the Husformer and HusFuse are concatenated by
unimodal features of each modality within the same fragments.
The only difference is that the unimodal features in the input of
the Husformer are reinforced by cross-modal attention transformers while those in the HusFuse are not. Comparing the
different self-attention patterns assigned to the same fragments
of unimodal features in the fusion representation as illustrated
in Figure 6a and 6b, we can notice that without the crossmodal attention modelling the cross-modal interactions, the
self-attention learned in the HusFuse is less efficient. That is,

only few of high (blue) and low (red) self-attention scores
are assigned to features in the fusion representation, leading
to insufficient prominence of critical contextual information
and diminishing of unimportant information respectively. We
can also notice that the self-attention is less consistent. That
is, no stable self-attention patterns are shown on features of
the same modality across batches, especially between Batch 1
and Batch 3. Such differences demonstrate that the reinforcements for unimodal features from the cross-modal attention
in the Husformer can help the self-attention highlight critical
contextual information in the fusion representation with a more
efficient and consistent pattern.
H. Real-world Experiment
In addition, we applied the Husformer to a real-world CCTV
monitoring task scenario [56]. The Husformer effectively
predicted the cognitive workload (low vs. medium vs. high) of
multiple humans when performing the CCTV monitoring task
with a multi-robot system at 100 Hz in real-time. Additionally,
by utilizing the predicted objective cognitive load from the
Husformer model, the overall performance of the human-robot

14

team can be enhanced by adjusting the workload of each
human operator accordingly. More details can be found in [56].
V. C ONCLUSION AND F UTURE W ORK
In this paper, we proposed the Husformer, an end-to-end
multi-modal transformer framework for recognizing multimodal human states, including affective states and cognitive
load. The Husformer fuses modalities with adaptive and
sufficient cross-modal interactions, enabling one modality to
attend to features of other modalities where strong crossmodal relevance exists. It also adaptively highlights important
contextual information in the fusion representation. These two
attention mechanisms, operating at the inter-modal and fusion
representation levels, enable our model to efficiently learn
from multi-modal features, eliminating the need for extensive
feature engineering and alignment required in previous works.
Our experimental results on four public benchmark multimodal datasets of human emotion and cognitive load demonstrated the effectiveness of the proposed Husformer for general
human state recognition, outperforming four other state-of-theart multi-modal-based baselines and demonstrating enhanced
performance over using a single modality. Additionally, our
ablation study highlighted the effectiveness of two key components in the Husformer: the cross-modal attention and
self-attention modules. Through the visualization of attention
activation, we demonstrated that the cross-modal attention
and self-attention transformers introduced in Husformer can
respectively model adaptive but relatively consistent and longterm cross-modal interactions of multiple modalities and contextual interactions in the high-level fusion representation.
In the future, we plan to integrate user sensitivity considerations into the framework to improve its performance
and generalization ability. Additionally, we will explore the
application of the Husformer to more real-world scenarios and
investigate its robustness and adaptability in various practical
scenarios.
ACKNOWLEDGEMENTS
This material is based upon work supported by the National
Science Foundation under Grant No. IIS-1846221. Any opinions, findings, and conclusions or recommendations expressed
in this material are those of the author(s) and do not necessarily
reflect the views of the National Science Foundation.
R EFERENCES
[1] J. Zhang, Z. Yin, and R. Wang, “Recognition of mental workload levels
under complex human–machine collaboration by using physiological
features and adaptive support vector machines,” IEEE Transactions on
Human-Machine Systems, vol. 45, no. 2, pp. 200–214, 2014.
[2] C. D. Wickens, “Multiple resources and performance prediction,” Theoretical issues in ergonomics science, vol. 3, no. 2, pp. 159–177, 2002.
[3] S. Yang, Z. Yin, Y. Wang, W. Zhang, Y. Wang, and J. Zhang, “Assessing
cognitive mental workload via eeg signals and an ensemble deep learning
classifier based on denoising autoencoders,” Computers in biology and
medicine, vol. 109, pp. 159–170, 2019.
[4] S. P. Marshall, “The index of cognitive activity: Measuring cognitive
workload,” in Proceedings of the IEEE 7th conference on Human Factors
and Power Plants. IEEE, 2002, pp. 7–7.

[5] E. Debie, R. F. Rojas, J. Fidock, M. Barlow, K. Kasmarik, S. Anavatti,
M. Garratt, and H. A. Abbass, “Multimodal fusion for objective assessment of cognitive workload: a review,” IEEE transactions on cybernetics,
vol. 51, no. 3, pp. 1542–1555, 2019.
[6] N. Sebe, I. Cohen, and T. S. Huang, “Multimodal emotion recognition,”
in Handbook of pattern recognition and computer vision.
World
Scientific, 2005, pp. 387–409.
[7] Z. He, Z. Li, F. Yang, L. Wang, J. Li, C. Zhou, and J. Pan, “Advances
in multimodal emotion recognition based on brain–computer interfaces,”
Brain sciences, vol. 10, no. 10, p. 687, 2020.
[8] R. R. Brooks and S. S. Iyengar, Multi-sensor fusion: fundamentals and
applications with software. Prentice-Hall, Inc., 1998.
[9] F. Chen, J. Zhou, Y. Wang, K. Yu, S. Z. Arshad, A. Khawaji, and
D. Conway, Robust multimodal cognitive load measurement. Springer,
2016.
[10] J.-L. Kruger, S. Doherty, W. Fox, and P. De Lissa, “Multimodal
measurement of cognitive load during subtitle processing,” Innovation
and expansion in translation process research, vol. 267, 2018.
[11] Y. Huang, J. Yang, P. Liao, and J. Pan, “Fusion of facial expressions and
eeg for multimodal emotion recognition,” Computational intelligence
and neuroscience, vol. 2017, 2017.
[12] H. Tang, W. Liu, W.-L. Zheng, and B.-L. Lu, “Multimodal emotion
recognition using deep neural networks,” in International Conference
on Neural Information Processing. Springer, 2017, pp. 811–819.
[13] J. Singh and R. Gill, “Multimodal emotion recognition system using
machine learning and psychological signals: A review,” Soft Computing:
Theories and Applications: Proceedings of SoCTA 2020, Volume 1, pp.
657–666, 2022.
[14] T. Horii, Y. Nagai, and M. Asada, “Modeling development of multimodal
emotion perception guided by tactile dominance and perceptual improvement,” IEEE Transactions on Cognitive and Developmental Systems,
vol. 10, no. 3, pp. 762–775, 2018.
[15] W.-L. Zheng, W. Liu, Y. Lu, B.-L. Lu, and A. Cichocki, “Emotionmeter:
A multimodal framework for recognizing human emotions,” IEEE
transactions on cybernetics, vol. 49, no. 3, pp. 1110–1122, 2018.
[16] J.-L. Qiu, W. Liu, and B.-L. Lu, “Multi-view emotion recognition using
deep canonical correlation analysis,” in International Conference on
Neural Information Processing. Springer, 2018, pp. 221–231.
[17] W. Liu, J.-L. Qiu, W.-L. Zheng, and B.-L. Lu, “Comparing recognition
performance and robustness of multimodal deep learning models for
multimodal emotion recognition,” IEEE Transactions on Cognitive and
Developmental Systems, 2021.
[18] Y. Zhou, S. Huang, Z. Xu, P. Wang, X. Wu, and D. Zhang, “Cognitive
workload recognition using eeg signals and machine learning: a review,”
IEEE Transactions on Cognitive and Developmental Systems, 2021.
[19] T. Zhou, J. S. Cha, G. Gonzalez, J. P. Wachs, C. P. Sundaram, and
D. Yu, “Multimodal physiological signals for workload prediction in
robot-assisted surgery,” ACM Transactions on Human-Robot Interaction
(THRI), vol. 9, no. 2, pp. 1–26, 2020.
[20] S. Koelstra, C. Muhl, M. Soleymani, J.-S. Lee, A. Yazdani, T. Ebrahimi,
T. Pun, A. Nijholt, and I. Patras, “Deap: A database for emotion analysis;
using physiological signals,” IEEE transactions on affective computing,
vol. 3, no. 1, pp. 18–31, 2011.
[21] P. Schmidt, A. Reiss, R. Duerichen, C. Marberger, and K. Van Laerhoven, “Introducing wesad, a multimodal dataset for wearable stress
and affect detection,” in Proceedings of the 20th ACM international
conference on multimodal interaction, 2018, pp. 400–408.
[22] W. Jo, R. Wang, S. Sun, R. K. Senthilkumaran, D. Foti, and B.-C. Min,
“MOCAS: A Multimodal Dataset for Objective Cognitive Workload
Assessment on Simultaneous Tasks,” Aug. 2022. [Online]. Available:
https://doi.org/10.5281/zenodo.7023242
[23] M. Gjoreski, T. Kolenik, T. Knez, M. Luštrek, M. Gams, H. Gjoreski,
and V. Pejović, “Datasets for cognitive load inference using wearable
sensors and psychological traits,” Applied Sciences, vol. 10, no. 11, p.
3843, 2020.
[24] J. Ngiam, A. Khosla, M. Kim, J. Nam, H. Lee, and A. Y. Ng,
“Multimodal deep learning,” in ICML, 2011.
[25] Y.-H. H. Tsai, S. Bai, P. P. Liang, J. Z. Kolter, L.-P. Morency, and
R. Salakhutdinov, “Multimodal transformer for unaligned multimodal
language sequences,” in Proceedings of the conference. Association for
Computational Linguistics. Meeting, vol. 2019. NIH Public Access,
2019, p. 6558.
[26] G. Andrew, R. Arora, J. Bilmes, and K. Livescu, “Deep canonical
correlation analysis,” in International conference on machine learning.
PMLR, 2013, pp. 1247–1255.
[27] J. Zhou, K. Yu, F. Chen, Y. Wang, and S. Z. Arshad, “Multimodal
behavioral and physiological signals as indicators of cognitive load,” in

15

The Handbook of Multimodal-Multisensor Interfaces: Signal Processing,
Architectures, and Detection of Emotion and Cognition-Volume 2, 2018,
pp. 287–329.
[28] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez,
Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in
neural information processing systems, vol. 30, 2017.
[29] T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár, “Focal loss
for dense object detection,” in Proceedings of the IEEE international
conference on computer vision, 2017, pp. 2980–2988.
[30] W. Jo, R. Wang, S. Sun, R. K. Senthilkumaran, D. Foti, and B.-C.
Min, “Mocas: A multimodal dataset for objective cognitive workload
assessment on simultaneous tasks,” arXiv preprint arXiv:2210.03065,
2022.
[31] C. Kirschbaum, K.-M. Pirke, and D. H. Hellhammer, “The ‘trier social
stress test’–a tool for investigating psychobiological stress responses in
a laboratory setting,” Neuropsychobiology, vol. 28, no. 1-2, pp. 76–81,
1993.
[32] D. Makowski, T. Pham, Z. J. Lau, J. C. Brammer, F. Lespinasse,
H. Pham, C. Schölzel, and S. Chen, “Neurokit2: A python toolbox
for neurophysiological signal processing,” Behavior research methods,
vol. 53, no. 4, pp. 1689–1696, 2021.
[33] E. Haapalainen, S. Kim, J. F. Forlizzi, and A. K. Dey, “Psychophysiological measures for assessing cognitive load,” in Proceedings of
the 12th ACM international conference on Ubiquitous computing, 2010,
pp. 301–310.
[34] M. A. Hogervorst, A.-M. Brouwer, and J. B. Van Erp, “Combining and
comparing eeg, peripheral physiology and eye-related measures for the
assessment of mental workload,” Frontiers in neuroscience, vol. 8, p.
322, 2014.
[35] P. Zhang, X. Wang, J. Chen, and W. You, “Feature weight driven
interactive mutual information modeling for heterogeneous bio-signal
fusion to estimate mental workload,” Sensors, vol. 17, no. 10, p. 2315,
2017.
[36] F. Putze, J.-P. Jarvis, and T. Schultz, “Multimodal recognition of cognitive workload for multitasking in the car,” in 2010 20th International
Conference on Pattern Recognition. IEEE, 2010, pp. 3748–3751.
[37] T. Zhou and J. P. Wachs, “Early prediction for physical human robot
collaboration in the operating room,” Autonomous Robots, vol. 42, no. 5,
pp. 977–995, 2018.
[38] G. Shafer, “Dempster-shafer theory,” Encyclopedia of artificial intelligence, vol. 1, pp. 330–331, 1992.
[39] J. Ma, H. Tang, W.-L. Zheng, and B.-L. Lu, “Emotion recognition using
multimodal residual lstm network,” in Proceedings of the 27th ACM
international conference on multimedia, 2019, pp. 176–183.
[40] A. Graves, “Long short-term memory,” in Supervised sequence labelling
with recurrent neural networks. Springer, 2012, pp. 37–45.
[41] F. Scarselli, M. Gori, A. C. Tsoi, M. Hagenbuchner, and G. Monfardini, “The graph neural network model,” IEEE transactions on neural
networks, vol. 20, no. 1, pp. 61–80, 2008.
[42] T. Song, W. Zheng, P. Song, and Z. Cui, “Eeg emotion recognition using
dynamical graph convolutional neural networks,” IEEE Transactions on
Affective Computing, vol. 11, no. 3, pp. 532–541, 2018.
[43] P. Zhong, D. Wang, and C. Miao, “Eeg-based emotion recognition using
regularized graph neural networks,” IEEE Transactions on Affective
Computing, 2020.
[44] J. Sun, J. Xie, and H. Zhou, “Eeg classification with transformer-based
models,” in 2021 IEEE 3rd Global Conference on Life Sciences and
Technologies (LifeTech). IEEE, 2021, pp. 92–93.
[45] R. Hu and A. Singh, “Unit: Multimodal multitask learning with a unified
transformer,” in Proceedings of the IEEE/CVF International Conference
on Computer Vision, 2021, pp. 1439–1449.
[46] S. J. Russell, Artificial intelligence a modern approach.
Pearson
Education, Inc., 2010.
[47] J. Gareth, W. Daniela, H. Trevor, and T. Robert, An introduction to
statistical learning: with applications in R. Spinger, 2013.
[48] C. M. Bishop and N. M. Nasrabadi, Pattern recognition and machine
learning. Springer, 2006, vol. 4, no. 4.
[49] C.-H. Wu, J.-C. Lin, and W.-L. Wei, “Survey on audiovisual emotion
recognition: databases, features, and data fusion strategies,” APSIPA
transactions on signal and information processing, vol. 3, 2014.
[50] K. Hou, G. Shao, H. Wang, L. Zheng, Q. Zhang, S. Wu, and W. Hu,
“Research on practical power system stability analysis algorithm based
on modified svm,” Protection and Control of Modern Power Systems,
vol. 3, no. 1, pp. 1–7, 2018.
[51] Y. Wu and Y. Liu, “Robust truncated hinge loss support vector machines,” Journal of the American Statistical Association, vol. 102, no.
479, pp. 974–983, 2007.

[52] D. Lahat, T. Adali, and C. Jutten, “Multimodal data fusion: an overview
of methods, challenges, and prospects,” Proceedings of the IEEE, vol.
103, no. 9, pp. 1449–1477, 2015.
[53] Y. Bian, J. Huang, X. Cai, J. Yuan, and K. Church, “On attention redundancy: A comprehensive study,” in Proceedings of the 2021 Conference
of the North American Chapter of the Association for Computational
Linguistics: Human Language Technologies, 2021, pp. 930–945.
[54] S. Bhojanapalli, A. Chakrabarti, A. Veit, M. Lukasik, H. Jain, F. Liu,
Y.-W. Chang, and S. Kumar, “Leveraging redundancy in attention with
reuse transformers,” arXiv preprint arXiv:2110.06821, 2021.
[55] A. Zeyer, P. Bahar, K. Irie, R. Schlüter, and H. Ney, “A comparison of
transformer and lstm encoder decoder models for asr,” in 2019 IEEE
Automatic Speech Recognition and Understanding Workshop (ASRU).
IEEE, 2019, pp. 8–15.
[56] W. Jo, R. Wang, B. Yang, D. Foti, M. Rastgaar, and B.-C. Min, “Affective
workload allocation for multi-human multi-robot teams,” arXiv preprint
arXiv:2303.10465, 2023.

16

A PPENDIX A
H YPER - PARAMETERS OF THE Husformer
TABLE VIII
H YPER - PARAMETERS OF THE Husformer UTILIZED FOR EACH EXPERIMENT
Parameter name
Batch Size
Initial Learning Rate
Optimizer
Transformer Hidden Unit Size
Crossmodal Attention Heads
Crossmodal Attention Block Dropout
Output Dropout
Focal Loss αc
Focal Loss γ
Epochs

Raw DEAP
1024
2e-3
Adam
40
3
0.1
0.1
[0.1,0.1,0.8]
3
20

Preprocessed DEAP
1024
2e-3
Adam
40
3
0.1
0.1
[0.15,0.05,0.8]
3
20

WESAD
512
1e-3
Adam
40
3
0.05
0.1
[0.4,0.3,0.3]
2
60

Raw MOCAS
64
1e-3
Adam
40
5
0.05
0.1
[0.2,0.1,0.7]
3
40

Preprocessed MOCAS
128
1e-3
Adam
40
5
0.05
0.1
[0.15,0.15,0.7]
3
40

Cogload
1024
1e-3
Adam
40
3
0.1
0.1
[0.1,0.1,0.8]
2
80

A PPENDIX B
C LASSIFICATION RESULTS USING EACH SINGLE MODALITY
TABLE IX
B EST PERFORMING CLASSIFICATION RESULTS OF USING SINGLE MODALITY ON THE RAW DEAP AND PREPROCESSED DEAP DATASET IN TERMS OF
MULTI - CLASS AVERAGE ACCURACY (Acc) AND MULTI - CLASS AVERAGE F1- SCORE (F 1) WITH STAND DEVIATIONS . A LL BEST PERFORMING RESULTS
ARE OBTAINED WITH THE TRANSFORMER NETWORK .
Dataset
Criteria
Metric
EEG
EMG
EOG
GSR

Raw DEAP
Valence
Arousal
Acc(%)h
F 1(%)h
Acc(%)h
F 1(%)h
72.80±2.03
72.93±2.20
73.27±1.72
73.71±1.93
64.66±2.38
64.71±2.33
65.32±2.24
65.66±1.83
46.99±1.93
45.98±1.83
48.27±1.57
50.97±1.44
45.18±2.56
45.63±2.38
46.70±2.33
48.07±2.17

Preprocessed DEAP
Valence
Arousal
Acc(%)h
F 1(%)h
Acc(%)h
F 1(%)h
84.98±1.40
85.09±1.42
84.86±0.30
84.86±0.80
76.69±1.04
76.71±4.02
74.09±0.89
74.07±0.86
62.76±1.85
65.57±1.45
64.42±0.44
65.43±0.39
61.65±1.16
59.74±0.97
62.75±0.91
57.28±0.88

TABLE X
B EST PERFORMING CLASSIFICATION RESULTS OF USING SINGLE MODALITY ON THE RAW MOCAS AND PREPROCESSED MOCAS DATASET IN TERMS OF
MULTI - CLASS AVERAGE ACCURACY (Acc) AND MULTI - CLASS AVERAGE F1- SCORE (F 1) WITH STAND DEVIATIONS ; ♣ : CLASSIFICATION WITH
T RANSFORMER , AND ♠ : CLASSIFICATION WITH GCN.
Dataset

Raw MOCAS

Preprocessed MOCAS

Metric

Acc(%)h

F 1(%)h

Acc(%)h

F 1(%)h

EEG
EEG POW
GSR
BVP
EAR

34.17±0.57♠
68.25±1.31♣
33.70±0.61♠
42.55±1.63♠
47.34±3.12♣

34.01±0.53♠
67.87±1.45♣
35.59±0.64♠
43.49±1.71♣
49.29±2.29♣

44.80±0.37♣
84.98±1.74♣
43.44±0.57♣
71.18±2.03♣
51.46±0.42♣

46.21±0.37♠
84.90±1.72♣
46.98±0.53♣
71.14±2.06♣
48.37±0.45♣

TABLE XI
B EST PERFORMING CLASSIFICATION RESULTS OF USING SINGLE MODALITY ON THE ( A ) WESAD DATASET AND ( B ) C OG L OAD DATASET IN TERMS OF
MULTI - CLASS AVERAGE ACCURACY (Acc) AND MULTI - CLASS AVERAGE F1- SCORE (F 1) WITH STAND DEVIATIONS . ♣ : CLASSIFICATION WITH
T RANSFORMER ; ♠ : CLASSIFICATION WITH GCN.

(b)

(a)
Dataset

WESAD

Metric

Acc(%)h

F 1(%)h

EMG
EDA
BVP
RESP

52.71±0.46♠

55.62±0.54♠

53.85±0.77♣

56.45±0.52♠

60.75±0.95♣
64.09±1.20♣

61.27±0.99♣
65.85±1.04♣

Dataset

Cogload

Metric

Acc(%)h

F 1(%)h

GSR
HR
RR
ACC

56.45±0.73♣

57.52±0.80♣
30.03±1.21♠
42.58±2.47♣
33.35±1.53♠

30.54±1.13♠
39.88±2.32♠
34.59±1.81♠
</reference>

<statements>
1. Deep architectures dominate: per-modality CNNs, 3D-CNNs for spatiotemporal video, and Transformers with cross-modal attention that learn cross-modal contingencies "without the requirement of feature alignment."
</statements>

Begin the assessment now. Output only the JSON list, without any conversational text or explanations.