You will be provided with a reference and some statements. Please determine whether each statement is 'supported', 'unsupported', or 'unknown' with respect to the reference. Please note:
First, assess whether the reference contains any valid content. If the reference contains no valid information, such as a 'page not found' message, then all statements should be considered 'unknown'.
If the reference is valid, for a given statement: if the facts or data it contains can be found entirely or partially within the reference, it is considered 'supported' (data accepts rounding); if all facts and data in the statement cannot be found in the reference, it is considered 'unsupported'.

You should return the result in a JSON list format, where each item in the list contains the statement's index and the judgment result, for example:
[
    {
        "idx": 1,
        "result": "supported"
    },
    {
        "idx": 2,
        "result": "unsupported"
    }
]

Below are the reference and statements:
<reference>
Deep Knowledge Tracing for Personalized
Adaptive Learning at Historically Black Colleges
and Universities

arXiv:2410.13876v1 [cs.CY] 2 Oct 2024

Ming-Mu Kuo† , Xiangfang Li† , Lijun Qian† , Pamela Obiomon† ,
Xishuang Dong†
CREDIT Center and ECE Department, Prairie View A&M University,
700 University Drive, Prairie View, 77446, TX, USA.

*Corresponding author(s). E-mail(s): xidong@pvamu.edu;
Contributing authors: mmkuo@pvamu.edu; xili@pvamu.edu;
liqian@pvamu.edu; phobiomon@pvamu.edu;
† These authors contributed equally to this work.
Abstract
Personalized adaptive learning (PAL) stands out by closely monitoring individual students’ progress and tailoring their learning paths to their unique
knowledge and needs. A crucial technique for effective PAL implementation is
knowledge tracing, which models students’ evolving knowledge to predict their
future performance. Recent advancements in deep learning have significantly
enhanced knowledge tracing through Deep Knowledge Tracing (DKT). However, there is limited research on DKT for Science, Technology, Engineering,
and Math (STEM) education at Historically Black Colleges and Universities
(HBCUs). This study builds a comprehensive dataset to investigate DKT for
implementing PAL in STEM education at HBCUs, utilizing multiple state-of-theart (SOTA) DKT models to examine knowledge tracing performance. The dataset
includes 352, 148 learning records for 17, 181 undergraduate students across
eight colleges at Prairie View A&M University (PVAMU). The SOTA DKT models employed include DKT, DKT+, DKVMN, SAKT, and KQN. Experimental
results demonstrate the effectiveness of DKT models in accurately predicting students’ academic outcomes. Specifically, the SAKT and KQN models outperform
others in terms of accuracy and AUC. These findings have significant implications for faculty members and academic advisors, providing valuable insights for
identifying students at risk of academic underperformance before the end of the

1

semester. Furthermore, this allows for proactive interventions to support students’
academic progress, potentially enhancing student retention and graduation rates.
Keywords: Historically Black Colleges and Universities, STEM Education,
Personalized Adaptive Learning, Knowledge Tracing

Introduction
In contemporary educational settings, personalized adaptive learning (PAL) has
emerged as a dynamic approach to cater to the diverse needs and learning styles of
students. This pedagogical framework tailors instructional content, pace, and learning experiences to the individual learner, leveraging technology and data analytics to
facilitate targeted interventions (Bajaj and Sharma, 2018). In particular, PAL systems utilize artificial intelligence techniques to analyze student performance data,
enabling educators to gain insights into each learner’s strengths, weaknesses, and
areas for improvement (Zine et al., 2019; Yağcı, 2022; Waheed et al., 2020). By
providing customized learning pathways and real-time feedback, these systems aim
to optimize student engagement, comprehension, and overall academic achievement
(Essa et al., 2023). Deep Knowledge Tracing (DKT) emerges as a crucial component for implementing PAL systems. It represents a paradigm shift from traditional
assessment methods, offering a sophisticated approach to modeling and predicting
students’ mastery of specific concepts or skills over time (Piech et al., 2015). Unlike
conventional methods that rely solely on grades or assessments, DKT leverages deep
learning algorithms to analyze sequential student interactions with learning materials,
including exercises, quizzes, and assignments. By capturing the dynamic and temporal aspects of learning, DKT predicts future learning outcomes, aiding educators in
understanding and addressing students’ individual progress and needs. This provides a
more intricate comprehension of students’ knowledge acquisition processes, facilitating
timely interventions and personalized support (Liu et al., 2023). This nuanced understanding enables tailored support, fostering a more effective and responsive learning
environment.
Despite the advancements in DKT, there remains a gap in its investigation and
application within specific educational contexts, particularly at Historically Black Colleges and Universities (HBCUs) (Pokrajac et al., 2016). These institutions play a vital
role in fostering academic excellence and opportunity for underrepresented minority students, yet there is a lack of comprehensive research on the effectiveness and
implementation of DKT within HBCU settings. Addressing this gap is crucial for
advancing the understanding of knowledge tracing in diverse educational environments
and ensuring equitable access to effective learning technologies.
This study builds a comprehensive dataset to investigate DKT for implementing
PAL in STEM education at HBCUs. It utilizes multiple state-of-the-art (SOTA) DKT
models, including DKT, DKT+, DKVMN, SAKT, and KQN, to examine knowledge
tracing performance. The dataset comprises 352, 148 learning records for 17, 181 undergraduate students across eight colleges at Prairie View A&M University (PVAMU),

2

with a specialized focus on College of Engineering majors. The study centers on
the unique challenges and opportunities within engineering education, detailing the
tailored data collection processes, model development strategies, and evaluation procedures employed. Experimental results demonstrate the effectiveness of these DKT
models in accurately predicting students’ academic outcomes. Specifically, the SAKT
and KQN models outperform others in terms of accuracy and AUC. Additionally,
the implications of these findings for personalized adaptive learning initiatives and
educational practices in HBCU settings are discussed.
The contributions of this study are summarized as follows:
• To the best of our knowledge, this is the first dataset to investigate DKT at HBCUs,
exploring the potentials of PAL. It enables the capture, analysis, and interpretation
of data relevant to STEM education at HBCUs using advanced knowledge tracing
techniques to implement PAL.
• This study evaluates the effectiveness and limitations of current SOTA DKT models in predicting student performance and identifying at-risk individuals early in
the semester for HBCUs. Moreover, it addresses the gap in the investigation and
application of DKT within the specific educational contexts of HBCUs. It not only
broadens the scope of DKT applications but also provides valuable insights into
effective implementation strategies and implications for future research and practice
in diverse educational settings.

Data collection and preparation
Data Introduction
Educational institutions routinely maintain comprehensive electronic records of student information, encompassing diverse data types and volumes, ranging from
demographic details to academic achievements. This study collected a real dataset
of undergraduate students from the Student Information System (SIS) at PVAMU,
one HBCU in Texas. The data spans four years, from fall 2020 to summer 2023. The
collected data includes five essential elements: Academic Year, Universal ID, Course
Subject, Course Level, and Pass/Fail grades. These features are strictly academic and
relate to students’ grades in their courses over the academic years. Table 1 provides a
few examples from the dataset.
Table 1 Data samples
Academic Year
2020
2020
2021
2022
2023
2023
...

Universal ID
5517806
5578432
5626380
5966264
4929554
4929554
...

Course Subject
BIOL
ENGL
MATH
ELEG
HLTH
KINE
...

3

Course Level
1000
1000
2000
2000
4000
4000
...

Pass/Fail
1
1
0
1
0
1
...

where “Academic Year” denotes the specific academic year during which the course
was undertaken, providing temporal context to the data. The original student ID
was substituted with a randomly generated universal ID to ensure the anonymity of
individual students. The course subject consists of the four characters associated with
the course. The course level is determined by the thousands digit in the course number.
Freshman courses commence at the one thousand level, sophomore courses at the two
thousand level, junior courses at the three thousand level, senior courses at the four
thousand level, graduate-level courses at five thousand and above, and development
courses initiate at the zero level. The Pass/Fail data is binary, denoting a pass with
the value 1 or a failure with the value 0. This study considers grades of A, B, C, and
’Credit/Pass’ as indicative of passing the course, while grades of D, F, withdraw and
’No credit/did not pass’ signify failure.

Data Preparation
To prepare data for DKT, a few steps of data preprocessing are completed: 1) Data
records lacking grades, incomplete courses, or non-gradable courses were excluded
from the dataset; 2) The features comprise categorical data, which is incompatible
with machine learning algorithms. Consequently, it is necessary to transform these
categorical course data into a numerical format before inputting them into the machine
learning model. In this research, we employed a technique known as label encoding
to convert our categorical data into numerical values (Patil and Thorat, 2016). This
method assigns an integer to each distinct nominal variable. Course subject and level
features were transformed into integer numbers, representing the student’s knowledge
skill; 3) Courses sharing the same subject and level were treated as possessing the
same knowledge skill. For example, courses like MATH 3201 and MATH 3800, both
falling under the 3000-level math category, were considered part of the same skill set.
Table 2 shows examples of the mapping between course subject and level encoded to
integer number as skill id.
Table 2 Examples of mapping between course
subject and level encoded to integer number.
Course Subject
ACCT
ACCT
ACCT
ACCT
ADMN
AFAM
...
...
SPED
SPED
SPMT

Course Level
2000
3000
4000
5000
5000
1000

4000
5000
1000

4

Skill id
1
2
3
4
5
6
...
...
231
232
233

Statistical Analysis
Over the four-year period, the total university (UNIV) dataset comprises 352, 148
records for undergraduate students, where Table 3 show the statistics of the dataset
and sub datasets.
Table 3 Dataset statistics
Items
Total Records
Records after data cleaning
Students
Types of Courses
Types of Knowledge Components

UNIV
352,148
326,269
17,181
2,124
233

COE+COAS
143,982
131,857
7,210
1,797
214

COE
52,206
46,477
2,387
1,027
172

Following the exclusion of incomplete, non-gradable, and no-grade information,
326, 269 records remain. A total of 17, 181 students participated in 2, 124 courses,
with courses sharing the same subject and level treated as possessing the same knowledge skill. The research identifies a total of 233 knowledge components (KCs) among
all undergraduate students. In the subset of records from the College of Engineering
(COE), the dataset comprises 52, 206 records. Following the exclusion of incomplete,
non-gradable, and no-grade information, 46, 477 records remain. A total of 2, 387 students participated in 1, 027 courses. In the entirety of the university college system,
the College of Arts and Sciences (COAS) stands as the largest academic entity. Consequently, we amalgamated student data from both COAS and COE disciplines to form
another subset for training purposes. Within the subset of records originating from
COE and COAS, the dataset encompasses 143, 982 entries. After removing incomplete, non-gradable, and no-grade data, 131, 857 records persist. Across 1, 797 courses,
a collective of 7, 210 students took part in the study. This nuanced analysis not only
provides insights into the overall predictive power of ML models but also delves into
the domain-specific efficacy, showcasing the potential impact on targeted academic
success initiatives. Table 4 represents the final dataset utilized for training and testing
DKT models. The DKT models aim to discern correlations among courses, enabling
the prediction of a student’s likelihood of passing or failing a course based on historical
records.
Table 4 Examples of samples for DKT at HBCUs.
Index
1
2
3
4
...
....
326267
326268
326269

Universal ID
304883
304883
308438
308438
...
...
6796570
6796693
6796693

5

Skill id
167
200
82
1
...
...
66
2
149

Pass/Fail
1
1
1
0
...
...
1
1
0

Methodology
Deep Knowledge Tracing (DKT) has emerged as a powerful approach for modeling
and predicting the knowledge mastery of learners in educational settings. Over time,
several variants of DKT have been proposed, each introducing unique enhancements
to address specific challenges. In this research, we delve into the methodologies of
five prominent DKT models: DKT, DKT+, DKVMN, SAKT, and KQN to predict
whether a student will pass or fail a course for HBUC education.
• Deep Knowledge Tracing (DKT) (Piech et al., 2015) is the first work to apply neural
networks for knowledge tracing tasks. It uses recurrent neural network (RNNs) to
effectively capture the temporal dependencies inherent within a sequence of interactions comprising a student’s questions and corresponding answers. This enables the
model to predict a student’s response to a new question based on their historical
interactions.
A fundamental representation of a simple RNN network for DKT is defined as
follows:
ht = tanh(Whx xt + Whh ht−1 + bh )
(1)
yt = σ(Wyh ht + by )

(2)

where tanh is the activation function, Whs is the input weights, Whh is the recurrent
weights, Wyt is the readout weights, and bh and by are the bias terms.
Within the DKT framework, the hidden state ht of the RNN is interpreted as the
latent representation of the student’s knowledge state. Moreover, ht is subjected
to prediction through a Sigmoid-activated linear layer, denoted as yt . This layer is
of the same length as the number of exercises, with each element representing the
model’s predicted probability of the student correctly answering the corresponding
exercise. The empirical findings indicated that DKT surpassed traditional KT models across multiple benchmark datasets. This underscores the promise of employing
deep learning models to tackle the KT challenge. Since then, deep learning-based
methods reached state of the art on knowledge tracing (Wang et al., 2023).
• DKT model with regularization (DKT+) (Yeung and Yeung, 2018) presents a
notable extension of the DKT framework. It introduces regularization terms that
correspond to reconstruction and waviness to the loss function of the original DKT
model to enhance the consistency in prediction. The initial loss function is expanded
by integrating three regularization terms, resulting in the subsequent regularized
loss function:
L′ = L + λr r + λw1 w1 + λw2 w22
(3)
where λr , λw1 , and λw2 are regularization parameters.
This refinement was devised to mitigate inherent constraints observed in DKT,
particularly in its proficiency to accurately reconstruct input responses and diminish
incongruities in predicting answers for questions associated with similar KCs.

6

• Dynamic Key-Value Memory Network (DKVMN) (Zhang et al., 2017) incorporates
a key-value memory matrix, where each row represents an item or skill, and the
columns correspond to key-value pairs associated with that item. The memory allows
the model to store information about students’ mastery of different skills and update
this information as they interact with educational items. When a student responds
to an item, the memory is updated to reflect their mastery of that item. The update
process involves modifying the values associated with relevant keys in the memory,
where the memory update for item i and key k is computed as:
Memory[i, k] = Memory[i, k] + updatek

(4)

where i is the index of the item, k represents the key associated with the item, and
updatek is the update value for key k, computed based on the student’s response
to the item.
• Self-Attentive Knowledge Tracing (SAKT) (Pandey and Karypis, 2019) tried to
handles with sparse data which students interact with few KCs. It adopts a
transformer-based architecture, replacing Long Short-Term Memory (LSTMs) with
self-attention mechanisms to capture the relevance between the KCs and the students’ historical interactions. SAKT employs a novel architecture that allows for
the dynamic weighting of input sequences, enabling the model to focus on the most
relevant interactions within a student’s learning history. The self-attention mechanism computes attention scores for each pair of elements in a sequence and then
calculates a weighted sum of the values based on these scores. Here’s the formula for
computing the attention score for element i with respect to element j is computed
as:


Qi · Kj
√
Attention(Q, K, V )i,j = softmax
· Vj
(5)
dk
where Q is the query matrix, K is the key matrix, V is the value matrix, and dk is
the dimensionality of the key vectors.
• Knowledge Query Network (KQN) (Lee and Yeung, 2019) combines memoryaugmented structures with attention mechanisms, encoding each student’s current
knowledge state as a query vector. The attention mechanisms enables KQN to
retrieve pertinent information from its memory, prioritizing the most relevant items
for each student. The attention weights are computed based on the similarity
between each node and the current knowledge query, ensuring that the model prioritizes nodes most relevant to the student’s current knowledge state. The attention
weights αij for each node j in the graph based on its similarity to the knowledge
query qi are computed as:
αij = softmax(qi · hj )
(6)
where hj represents the node embedding for node j, qi is the knowledge query
for student i. Furthermore, KQN introduces a novel concept called probabilistic
skill similarity, which relates pairwise cosine and Euclidean distances between skill

7

vectors to the odds ratios of corresponding skills, making KQN interpretable and
intuitive (Lee and Yeung, 2019).

Experiments
It focuses on conducting a comparative evaluation of DKT models across different
scales within the academic institution, specifically within the College of Engineering
(COE) and at the university-wide level.

Dataset
We adhere to conventional practices by allocating dataset resources primarily for
training purposes, typically dedicating 80% to 90% for this phase, while reserving the
remaining portion for testing. Leveraging a four-year span of authentic data, we aim
to predict student achievement in contemporary courses by analyzing historical academic performance. Specifically, the initial three years of data serve as the training
dataset, while the fourth year is designated for testing. Figure 1 offers an overview of
the training data extracted from various colleges, with the College of Arts & Sciences
(COAS) emerging as the largest college within PVAMU. The training dataset encompasses three distinct subsets: data from the College of Engineering (COE), data from
both the College of Engineering and the College of Arts and Sciences (COE + COAS),
and data covering all undergraduate students across the university (UNIV).

Fig. 1 Training data distribution.

For the testing phase, we exclusively utilize data from the five departments within
the College of Engineering: Civil and Environment Engineering (CEE), Chemical
Engineering (CHE), Computer Science (CSC), Electrical and Computer Engineering
(ECE), and Mechanical Engineering (MCE). Detailed insights into these Engineering
departments are depicted in Figure 2.
8

Fig. 2 Testing data distribution.

Furthermore, Table 5 presents details of data distributions for training and testing
of DKT.
Table 5 Dataset statistics
Training Dataset
COE
COE+COAS
UNIV
Testing Dataset
CEE
CHE
CSC
ECE
MCE
COE
UNIV

# of records
36,026
101,529
246,964
# of records
1,043
1,437
2,461
2,173
3,337
10,451
79,305

# of students
2,036
6,179
14,549
# of students
130
147
284
244
387
1,182
9,102

# of KCs
165
206
224
# of KCs
57
57
67
75
83
125
200

Experiment Setup
In this study, we employed the default settings and configurations of DKT models sourced from the GitHub repository. The original PyTorch implementation of
these DKT models can be accessed at https://github.com/hcnoh/knowledge-tracingcollection-pytorch.
• Batch Size: The batch size denotes the number of instances processed in a single
iteration during the training process. The default value is set to 256.
• Number of Epochs: Epochs represent the number of complete passes through the
entire training dataset. The default number of epochs utilized in this research is 100.

9

• Learning Rate: This parameter governs the step size during the optimization process
of the training algorithm. The default learning rate employed in this study is 0.001.
• Optimizer: The optimizer determines the specific algorithm employed for optimizing
the model parameters during the training process. The available optimizers include
Stochastic Gradient Descent (SGD) and Adaptive Moment Estimation (Adam). For
this research, the default optimizer chosen is Adam.
• Sequence Length: The sequence length represents the number of time steps or elements considered in the dataset for each training instance. The default sequence
length utilized in this research is 100.
These default settings serve as the foundational configuration for the DKT models
applied throughout the experimental investigations conducted in this study. These
DKT models were developed using PyTorch and trained on a workstation equipped
with NVIDIA V-100 GPUs.

Evaluation Metrics
In the realm of knowledge tracing, numerous methodologies employ binary classification techniques to forecast students’ academic performance, such as determining the
accuracy of exercise completion (Song et al., 2022). This study aims to evaluate the
effectiveness of various DKT models in predicting students’ likelihood of success in a
course, leveraging their historical subject grades as predictive features. All DKT models are subjected to a binary classification task that is to predict whether a student
will pass or fail the course. The metrics used are accuracy, recall, precision, F1-score,
and the Area Under the Receiver Operating Characteristic Curve (AUC).
Accuracy =

TP + TN
TP + FP + TN + FN

(7)

Accuracy, a commonly employed metric in classification tasks, measures the ratio
of correctly classified instances, encompassing both true positives and true negatives, relative to the total instances evaluated, thereby providing a straightforward
assessment of prediction accuracy.
TP
TP + FP

(8)

TP
TP + FN

(9)

P recision =
Recall =

P recision ∗ Recall
(10)
P recision + Recall
where True Positive (TP), False Positive (FP), True Negative (TN), and False
Negative (FN) come from confusion matrix terminology.
F1 = 2 ∗

• TP: correct predictions of passing courses.
• TN: correct predictions of failing courses.
• FP: incorrect predictions of passing courses.
• FN: incorrect predictions of failing courses.
10

P recision measures the proportion of true positive predictions out of all positive
predictions. Recall measures the proportion of true positive predictions out of all
actual positive instances. The F 1 score is a metric used to evaluate the performance of
a classification model, particularly when dealing with imbalanced classes. It combines
both precision and recall into a single measure.
Conversely, AUC serves as a valuable metric for assessing the performance of
binary classifiers, particularly in scenarios characterized by class imbalance or variable
importance of false positives and false negatives. A higher AUC value signifies superior discrimination capabilities of the model in distinguishing between positive and
negative classes (Gervet et al., 2020).

Results and Discussion
The findings and discussions present the comprehensive prediction performance analysis of DKT models across various testing and training datasets, as delineated in
Tables 6 through 13. These tables encapsulate the AUC and accuracy scores, providing
insights into the model’s efficacy. Recall, Precision, and F1 scores, further enriching our understanding of the model’s capabilities under different training and testing
conditions.

Knowledge tracing on departments within the COE when training
on COE
Table 6 and Table 7 illustrate the performance of these models when tested on five
engineering departments with training on the COE dataset. DKT and KQN models
demonstrate superior performance compared to other models. Specifically, the DKT
model achieves an average AUC of 0.6267, accuracy of 0.7604, and F1 score of 0.8567.
The KQN model performs similarly well, with an AUC of 0.6255, accuracy of 0.7817,
and F1 score of 0.8692. In contrast, the DKVMN and SAKT models exhibit moderate
performance. The DKVMN model records an AUC of 0.5921, accuracy of 0.7892, and
F1 score of 0.8779, while the SAKT model shows an AUC of 0.6099, accuracy of
0.7668, and F1 score of 0.8590. Among these models, the DKT+ model performs the
least effectively, with an AUC of 0.5461, accuracy of 0.6277, and F1 score of 0.7469.

Knowledge tracing on departments within the COE when training
on COE and COAS
Similarly, Table 8 and Table 9, testing the same engineering departments but with
training on the COE + COAS dataset, indicates that the KQN models continue to
deliver strong performance, with average metrics of AUC 0.6820, accuracy 0.8092,
and F1 score 0.8884. The SAKT and DKVMN models show noticeable improvements
compared to previous experiments and have surpassed the DKT model. Specifically,
the SAKT model achieves an AUC of 0.6692, accuracy of 0.7903, and F1 score of
0.8750, while the DKVMN model reports an AUC of 0.6618, accuracy of 0.7949, and
F1 score of 0.8805. In comparison, the DKT model has an average AUC of 0.6474,
accuracy of 0.7897, and F1 score of 0.8768.

11

12

0.6030
0.6605
0.5963
0.6169
0.6568
0.6267

CEE
CHE
CSC
ECE
MCE
Average

0.7604

0.6840
0.8029
0.8011
0.7655
0.7487

DKT
Accuracy

0.5464

0.6277

0.5828
0.6419
0.6658
0.6481
0.5998

DKT+
Accuracy

0.5302
0.5583
0.5370
0.5484
0.5582

AUC

0.5921

0.5690
0.6069
0.5262
0.6202
0.6384
0.7892

0.7288
0.8509
0.8378
0.7875
0.7412

DKVMN
AUC
Accuracy

0.6099

0.7668

0.7131
0.8270
0.8132
0.7872
0.6934

SAKT
Accuracy

0.5787
0.6851
0.6327
0.5758
0.5772

AUC

0.6255

0.5523
0.6725
0.5816
0.6531
0.6678

AUC

0.7817

0.6805
0.8486
0.8179
0.7922
0.7695

KQN
Accuracy

Recall
0.8496
0.8923
0.9304
0.9194
0.9236
0.9031

EM

CEE
CHE
CSC
ECE
MCE

Average

Model

0.8159

0.7536
0.8794
0.8472
0.8123
0.7869

Precision

DKT

0.8567

0.7987
0.8858
0.8869
0.8625
0.8498

F1

0.6902

0.6427
0.6891
0.7285
0.7267
0.6638

Recall

0.8144

0.7548
0.8658
0.8551
0.8132
0.7830

Precision

DKT+

0.7469

0.6943
0.7674
0.7867
0.7675
0.7185

F1

0.9484

0.9101
0.9631
0.9752
0.9620
0.9314

Recall

0.8175

0.7719
0.8769
0.8546
0.8091
0.7752

Precision

DKVMN

0.8779

0.8353
0.9180
0.9109
0.8790
0.8462

F1

0.8979

0.8959
0.9038
0.9311
0.9477
0.8111

Recall

0.8249

0.7588
0.8958
0.8591
0.8161
0.7948

Precision

SAKT

0.8590

0.8217
0.8998
0.8937
0.8770
0.8029

F1

0.9205

0.8322
0.9726
0.9421
0.9300
0.9256

Recall

0.8235

0.7583
0.8672
0.8568
0.8307
0.8045

Precision

KQN

0.8692

0.7935
0.9169
0.8974
0.8775
0.8608

F1

Table 7 Performance comparison between engineering departments via Recall, Precision and F1 scores between engineering departments when
DKT models are trained on College of Engineering (COE) data.

AUC

Model
EM

Table 6 Performance comparison between engineering departments via Accuracy and AUC when DKT models are trained on on College of
Engineering (COE) data. EM denotes evaluation metrics.

13

0.6474

Average

0.7897

0.5927

0.4841
0.6627
0.6107
0.6314
0.5745
0.6804

0.5753
0.7171
0.7261
0.7126
0.6707

DKT+
AUC
Accuracy

0.6618

0.6106
0.7342
0.6935
0.6455
0.6254
0.7949

0.7251
0.8431
0.8533
0.7954
0.7574

DKVMN
AUC
Accuracy

0.6692

0.6015
0.6992
0.6653
0.6978
0.6823
0.7903

0.7183
0.8715
0.8021
0.7931
0.7666

SAKT
AUC
Accuracy

0.6820

0.6187
0.7162
0.7017
0.6713
0.7022

0.8092

0.7337
0.8819
0.8433
0.8065
0.7804

KQN
AUC
Accuracy

Recall
0.9188
0.9550
0.9419
0.9418
0.9478
0.9411

CEE
CHE
CSC
ECE
MCE

Average

0.8221

0.7419
0.8891
0.8758
0.8244
0.7794

Precision

DKT

0.8768

0.8209
0.9209
0.9076
0.8792
0.8554

F1

0.7659

0.6722
0.7644
0.8017
0.8103
0.7811

Recall

0.8198

0.7302
0.8903
0.8630
0.8269
0.7885

Precision

DKT+

0.7914

0.7000
0.8226
0.8312
0.8185
0.7848

F1

0.9431

0.9098
0.9249
0.9701
0.9632
0.9474

Recall

0.8270

0.7691
0.8977
0.8713
0.8152
0.7818

Precision

DKVMN

0.8805

0.8336
0.9111
0.9180
0.8830
0.8567

F1

0.9225

0.9167
0.9637
0.8774
0.9244
0.9304

Recall

0.8341

0.7543
0.8948
0.8873
0.8350
0.7993

Precision

SAKT

0.8750

0.8276
0.9280
0.8823
0.8774
0.8599

F1

0.9550

0.9455
0.9758
0.9365
0.9566
0.9604

Recall

0.8321

0.7552
0.8959
0.8844
0.8284
0.7964

Precision

KQN

0.8884

0.8397
0.9341
0.9097
0.8879
0.8707

F1

Comparing Recall, Precision and F1 scores for each engineering department when DKT models are trained on COE and COAS data.

EM

Model

Table 9

0.5578
0.6974
0.7012
0.6324
0.6480

CEE
CHE
CSC
ECE
MCE

0.7046
0.8593
0.8382
0.7930
0.7536

DKT
AUC
Accuracy

Model
EM

Table 8 Comparing AUC and Accuracy for each engineering department when DKT models are trained on COE and COAS data.

Knowledge tracing on departments within the COE when training
on UNIV data
In Table 10 and Table 11, where testing is conducted on the same engineering departments but with training on the UNIV dataset, all models exhibit moderate to strong
performance. The analysis reveals that the KQN model consistently delivers superior
performance, with an average AUC of 0.6965, accuracy of 0.7996, and F1 score of
0.8834. The SAKT and DKVMN models also show competitive performance, reflecting their adaptability across various domains. Specifically, the SAKT model averages
an AUC of 0.6764, accuracy of 0.8028, and F1 score of 0.8831, while the DKVMN
model achieves an AUC of 0.6676, accuracy of 0.8066, and F1 score of 0.8889. Overall, the KQN model stands out for its robust performance across different testing and
training datasets, establishing it as a reliable choice for educational prediction tasks.
The SAKT and DKVMN models also demonstrate strong performance, suggesting
their suitability for real-world educational prediction applications alongside KQN. In
contrast, the DKT and DKT+ models show moderate performance and are generally surpassed by the KQN and SAKT models, underscoring the advantages of newer
model architectures.

Knowledge tracing on COE and UNIV when training on UNIV data
Table 12 and Table 13 illustrate the performance of these models when tested on
COE and UNIV with training on the UNIV dataset. For COE testing dataset, the
analysis shows that the KQN model exhibits strong performance across datasets, with
an average AUC of 0.6847, accuracy of 0.7963, and F1 score of 0.8805. The SAKT
model also performs competitively, with an average AUC of 0.6845, accuracy of 0.7848,
and F1 score of 0.8712. In contrast, the DKVMN and DKT models display moderate
performance. The DKVMN model has an AUC of 0.6525, accuracy of 0.7987, and F1
score of 0.8837, while the DKT model shows an AUC of 0.6579, accuracy of 0.7779,
and F1 score of 0.8672.
For the UNIV testing dataset, both the SAKT and KQN models maintain strong
performance. The SAKT model averages an AUC of 0.6644, accuracy of 0.8094, and F1
score of 0.8892, while the KQN model achieves an AUC of 0.6639, accuracy of 0.8197,
and F1 score of 0.8974. The DKVMN and DKT models continue to show moderate
performance, with the DKVMN model having an AUC of 0.6326, accuracy of 0.8153,
and F1 score of 0.8961, and the DKT model reporting an AUC of 0.6500, accuracy of
0.7917, and F1 score of 0.8791.
When evaluating the models trained on the UNIV dataset, it is observed that testing on the smaller COE dataset yields better performance metrics than testing on
the entire UNIV dataset. This discrepancy is largely due to the differences in testing
and training ratios. The smaller COE testing dataset might have fewer examples but
could present a more homogeneous and controlled environment, allowing the model
to perform more effectively. In contrast, the larger UNIV dataset introduces greater
variability and complexity, which can challenge the model’s ability to generalize,
potentially leading to lower performance metrics.

14

15

0.6259
0.7109
0.6930
0.6115
0.6881
0.6659

CEE
CHE
CSC
ECE
MCE
Average

0.7793

0.7213
0.8508
0.8336
0.7309
0.7598

DKT
Accuracy

0.6024

0.6756

0.5843
0.6953
0.7298
0.6737
0.6950

DKT+
Accuracy

0.5031
0.6505
0.6658
0.5945
0.5981

AUC

0.6676

0.6278
0.7097
0.6963
0.6463
0.6578
0.8066

0.7535
0.8450
0.8637
0.8095
0.7614

DKVMN
AUC
Accuracy

0.6764

0.8028

0.7271
0.8653
0.8584
0.7935
0.7697

SAKT
Accuracy

0.6584
0.7129
0.6875
0.6332
0.6899

AUC

0.6965

0.6468
0.7221
0.7475
0.6720
0.6943

AUC

0.7996

0.7308
0.8536
0.8492
0.7915
0.7731

KQN
Accuracy

Recall
0.9192
0.9604
0.9403
0.8495
0.9090
0.9157

Average

Recall

0.8260

Precision
0.7557
0.8773
0.8723
0.8205
0.8042

Precision

DKT

0.8679

F1
0.8295
0.9170
0.9050
0.8347
0.8534

F1

0.7649

Recall
0.6961
0.7484
0.8059
0.7579
0.8163

Recall

0.8169

Precision
0.7275
0.8785
0.8641
0.8207
0.7936

Precision

DKT+

0.7893

F1
0.7115
0.8082
0.8340
0.7881
0.8048

F1

0.9644

Recall
0.9617
0.9417
0.9797
0.9806
0.9585

Recall

0.8258

Precision
0.7697
0.8867
0.8747
0.8181
0.7799

Precision

DKVMN

0.8889

F1
0.8551
0.9134
0.9242
0.8920
0.8600

F1

0.9386

Recall
0.9156
0.9596
0.9542
0.9482
0.9154

Recall

0.8345

Precision
0.7622
0.8918
0.8871
0.8214
0.8100

Precision

SAKT

0.8831

F1
0.8319
0.9245
0.9194
0.8803
0.8595

F1

0.9566

Recall
0.9362
0.9645
0.9859
0.9447
0.9516

Recall

0.8212

Precision
0.7565
0.8772
0.8566
0.8216
0.7941

Precision

KQN

0.8834

F1
0.8368
0.9188
0.9167
0.8789
0.8657

F1

Comparing Recall, Precision and F1 scores for each engineering department when DKT models are trained on university (UNIV) data.

AUC

Model
EM

Comparing AUC and Accuracy for each engineering department when DKT models are trained on university (UNIV) data.

EM
CEE
CHE
CSC
ECE
MCE

EM

Model

Table 11

Table 10

16

0.7779
0.7917

DKT
Accuracy
0.6551
0.6700

DKT+
Accuracy

0.5805
0.5773

AUC
0.6525
0.6326

0.7987
0.8153

DKVMN
AUC
Accuracy
0.7848
0.8094

SAKT
Accuracy

0.6845
0.6644

AUC
0.6847
0.6639

AUC

0.7963
0.8197

KQN
Accuracy

Recall
0.9040
0.9132

COE
UNIV

0.8333
0.8474

Precision

DKT

0.8672
0.8791

F1
0.7275
0.7360

Recall
0.8220
0.8460

Precision

DKT+

0.7719
0.7872

F1
0.9489
0.9642

Recall
0.8269
0.8369

Precision

DKVMN

0.8837
0.8961

F1
0.9079
0.9224

Recall

0.8374
0.8583

Precision

SAKT

0.8712
0.8892

F1

0.9351
0.9516

Recall

0.8319
0.8491

Precision

KQN

Comparing Recall, Precision and F1 scores for COE and UNIV when DKT models are trained on university (UNIV) data.

0.6579
0.6500

COE
UNIV

EM

Model

Table 13

AUC

Model
EM

Table 12 Comparing AUC and Accuracy for COE and UNIV when DKT models are trained on university (UNIV) data.

F1
0.8805
0.8974

Both SAKT and KQN exhibit superior performance compared to traditional other
DKT models due to their innovative architectural choices. Unlike DKT, which relies
on RNNs or LSTMs networks to capture temporal dependencies, SAKT and KQN
leverage more advanced techniques. SAKT employs transformer-based architectures,
which can capture long-range dependencies in sequential data using self-attention
mechanisms, while KQN utilizes query mechanisms to represent interactions between
students and items, capturing complex dependencies more effectively than traditional
sequential models. SAKT utilizes attention mechanisms, while KQN employs the dot
product process to dynamically focus on relevant information when making predictions. Both models adaptively select the most informative items for each student,
enhancing their ability to tailor learning experiences and improve educational outcomes. These approaches allow SAKT and KQN to better adapt to the nuances of
student learning patterns, resulting in enhanced accuracy and efficiency in knowledge
tracing tasks.
Moreover, as evident from the provided dataset statistics Table 5 and the information gleaned from Tables 6-13 , there is a discernible trend indicating that the
prediction performance of DKT models tends to improve as the training dataset
increases in size and diversity. This trend is particularly notable when comparing performance across different training datasets. For instance, transitioning from smaller
training datasets like COE, with 36,026 records, to larger and more comprehensive
ones like UNIV, with 246,964 records, results in a notable improvement in prediction
performance metrics (AUC, accuracy, and F1) across different testing datasets.
Additionally, combining training datasets, as observed in the COE+COAS scenario, leads to a more diverse and extensive dataset (101,529 records), further
enhancing model performance. The increase in training data size enables the models
to learn more effectively from a broader range of student interactions and knowledge
components, thereby leading to improved prediction accuracy. Overall, the observed
trend suggests that scaling up the training dataset size and diversity positively impacts
the prediction performance of DKT models. This underscores the importance of utilizing large and diverse datasets in educational data mining tasks, as it enables more
accurate predictions of student knowledge mastery and learning outcomes.

Related Work
Knowledge tracing involves modeling students’ learning progress based on their activity sequences, making it a challenging task due to the need for accurate performance
prediction and understanding of students’ mastery levels. Researchers have tackled
this challenge through various approaches. There are two most representative models: Bayesian Knowledge Tracking (BKT) and Deep Knowledge Tracing (DKT). Early
efforts focused on BKT models (Corbett and Anderson, 1994), wherein each student’s
knowledge state is represented as a binary variable. These models employ probabilistic
techniques like the Hidden Markov Model (HMM) to assess students’ grasp of concepts
(Awad and Khanna, 2015). It characterizes a student’s progress in tackling problems
associated with a specific concept by utilizing a binary indicator (either correct or

17

incorrect) and consistently refines its assessment of the student’s understanding of
that concept (Mao et al., 2018).
DKT (Piech et al., 2015), a pioneering algorithm designed to model students’ learning states through recurrent neural networks (RNN), has demonstrated significant
enhancements in predictive accuracy. It grasps the temporal nuances within a sequence
of student-question interactions. Empirical findings demonstrated DKT’s superiority over traditional knowledge tracing models across various benchmark datasets
(Abdelrahman et al., 2023). The DKT models can be categorized into the following
groups (Song et al., 2022):
• DKT and its Variants. Early deep learning KT models often used RNNs architectures like Long Short-Term Memory (LSTM) or Gated Recurrent Unit (GRU).
RNNs are well-suited for sequential data, making them appropriate for modeling
students’ learning trajectories over time. e.g., DKT (Piech et al., 2015) and DKT+
(Yeung and Yeung, 2018).
• Memory Network based models. Memory networks introduce an external memory
component that the model can read from and write to at each time step. This
memory serves as a repository for storing past interactions, knowledge concepts, or
context information. By accessing this memory, the model can maintain a richer
representation of the student’s learning history and better infer their current knowledge state. e.g., DKVMN (Zhang et al., 2017), and SKVMN (Abdelrahman and
Wang, 2019).
• Attention Mechanism based models. To improve the modeling of long-term dependencies and focus on relevant parts of the input sequence, attention mechanisms were
introduced in KT models. Attention mechanisms allow the model to dynamically
weigh the importance of different time steps or features, enhancing its predictive
power. e.g., SAKT (Pandey and Karypis, 2019), DKVMN (Zhang et al., 2017) and
SAINT (Choi and et al., 2020).
• Graph Structure based models. Another direction of development involves using
graph neural networks to represent the knowledge structure and student interactions
as graphs. Graph Neural Networks (GNNs) allow models to capture complex relationships between knowledge concepts and how they influence each other’s mastery.
e.g., GKT (Nakagawa et al., 2021).
The pioneering use of DKT in online exercises has significantly advanced the prediction of students’ performance. By leveraging past student records, the original
DKT model successfully forecasts whether a student will pass or fail their current academic exercises. This research extends beyond mere prediction, utilizing past grades
to anticipate students’ future course outcomes. The research findings underscore the
effectiveness of DKT models in accurately predicting students’ academic trajectories,
thereby highlighting their potential for enhancing educational outcomes.

18

Conclusion and Future Work
PAL allows for monitoring individual students’ progress and tailoring their learning
paths to their unique knowledge and needs. DKT for PAL enhanced modeling students’ evolving knowledge to predict their future performance. This study mitigates
the gap in its investigation and application within specific educational contexts, particularly at HBCUs through building a comprehensive dataset from PVAMU and DKT
model evaluation on this dataset. This study advocates for the utilization of DKT
models to forecast the academic performance of undergraduate students, particularly
in determining whether they will pass or fail a course based on their historical grade
records. Through the utilization of diverse training datasets and various DKT models, the findings affirm the efficacy of DKT models in predicting students’ academic
outcomes for HBCUs. Furthermore, the adoption of DKT models in academic settings
can serve as a valuable tool for enhancing student support mechanisms and promoting
academic success for HBCUs.
In the future, it plans to explore the potential of adopting different DKT architectures not considered in this study such as large language models (LLMs) to develop
more sophisticated models capable of capturing complex patterns in student learning
data. Furthermore, there is a need to expand the way of knowledge skills are determined within the DKT models. While current research rely on course subject and
course level, future research could explore incorporating additional course features to
provide a more comprehensive understanding of student knowledge and learning trajectories. By incorporating these advancements, future DKT models have the potential
to offer more accurate and personalized insights into student learning processes,
ultimately leading to improved educational outcomes.
Acknowledgements. This research work is supported by the U.S. NSF awards
2205891 and 2235731. The U.S. Government is authorized to reproduce and distribute
reprints for governmental purposes notwithstanding any copyright notation thereon.
The views and conclusions contained herein are those of the authors and should not
be interpreted as necessarily representing the official policies or endorsements, either
expressed or implied, of the U.S. NSF or the U.S. Government.

References
Awad, M., Khanna, R.: Hidden markov model. In: Efficient Learning Machines. Apress,
Berkeley, CA (2015)
Abdelrahman, G., Wang, Q.: Knowledge tracing with sequential key-value memory
networks. Proceedings of the ACM on Human-Computer Interaction (2019) https:
//doi.org/10.1145/3331184.3331195
Abdelrahman, G., Wang, Q., Nunes, B.: Knowledge tracing: A survey. ACM Computing Surveys 55(11), 1–37 (2023) https://doi.org/10.1145/3569576
Bajaj, R., Sharma, V.: Smart education with artificial intelligence based determination
of learning styles. Procedia Computer Science 132, 834–842 (2018) https://doi.org/
19

10.1016/j.procs.2018.05.095
Corbett, A.T., Anderson, J.R.: Knowledge tracing: Modeling the acquisition of procedural knowledge. User Modeling and User-Adapted Interaction 4(4), 253–278
(1994)
Choi, Y., al.: Towards an appropriate query, key, and value computation for knowledge
tracing. In: Proceedings of the Seventh ACM Conference on Learning @ Scale (2020).
https://doi.org/10.1145/3386527.3405945
Essa, S.G., Celik, T., Human-Hendricks, N.E.: Personalized adaptive learning technologies based on machine learning techniques to identify learning styles: A
systematic literature review. IEEE Access 11, 48392–48409 (2023) https://doi.org/
10.1109/access.2023.3276439
Gervet, T., Koedinger, K., Schneider, J., Mitchell, T.: When is deep learning the best
approach to knowledge tracing? Journal of Educational Data Mining 12(3), 31–54
(2020) https://doi.org/10.5281/zenodo.4143614
Liu, Q., Shen, S., Huang, Z., Chen, E., Zheng, Y.: A Survey of Knowledge Tracing.
Available: https://arxiv.org/abs/2105.15106 (2023)
Lee, J., Yeung, D.-Y.: Knowledge Query Network for Knowledge Tracing. arXiv
(Cornell University) (2019). https://doi.org/10.1145/3303772.3303786
Mao, Y., Lin, C., Chi, M.: Deep learning vs. bayesian knowledge tracing: Student
models for interventions. Journal of Educational Data Mining 10(2), 28–54 (2018)
https://doi.org/10.5281/zenodo.3554691
Nakagawa, H., Iwasawa, Y., Matsuo, Y.: Graph-based knowledge tracing: Modeling
student proficiency using graph neural networks. Web Intelligence 19(1–2), 87–102
(2021) https://doi.org/10.3233/web-210458
Piech, C., Bassen, J., Huang, J., Ganguli, S., Sahami, M., Guibas, L.J., Sohl-Dickstein,
J.: Deep knowledge tracing. In: Advances in Neural Information Processing Systems,
pp. 505–513 (2015)
Pandey, S., Karypis, G.: A Self-Attentive Model for Knowledge Tracing. Available:
https://arxiv.org/abs/1907.06837 (2019)
Pokrajac, D.D., Sudler, K.R., Edamatsu, P.Y., Hardee, T.: Prediction of retention at
historically black college/university using artificial neural networks. In: 2016 IEEE
International Conference on Neural Networks (IJCNN) (2016). https://doi.org/10.
1109/neurel.2016.7800124 . https://doi.org/10.1109/neurel.2016.7800124
Patil, S.S., Thorat, R.R.: Categorical data encoding: Challenges and opportunities.
International Journal of Computer Applications 156(8), 23–27 (2016)

20

Song, X., Li, J., Cai, T., Yang, S., Yang, T., Liu, C.: A survey on deep learning based
knowledge tracing. Knowledge-Based Systems 258, 110036 (2022) https://doi.org/
10.1016/j.knosys.2022.110036
Wang, F., et al.: Dynamic cognitive diagnosis: An educational priors-enhanced deep
knowledge tracing perspective. IEEE Transactions on Learning Technologies, 1–17
(2023) https://doi.org/10.1109/tlt.2023.3254544
Waheed, H., Hassan, S.-U., Aljohani, N.R., Hardman, J., Alelyani, S., Nawaz, R.:
Predicting academic performance of students from vle big data using deep learning models. Computers in Human Behavior 104, 106189 (2020) https://doi.org/10.
1016/j.chb.2019.106189
Yağcı, M.: Educational data mining: prediction of students’ academic performance
using machine learning algorithms. Smart Learning Environments 9(1) (2022) https:
//doi.org/10.1186/s40561-022-00192-z
Yeung, C.-K., Yeung, D.-Y.: Addressing two problems in deep knowledge tracing via
prediction-consistent regularization. arXiv (2018) arXiv:1806.02180. Available at
https://arxiv.org/abs/1806.02180
Zine, O., Derouich, A., Talbi, A.: A comparative study of the most influential learning
styles used in adaptive educational environments. International Journal of Advanced
Computer Science and Applications 10(11) (2019) https://doi.org/10.14569/ijacsa.
2019.0101171
Zhang, J., Shi, X., King, I., Yeung, D.-Y.: Dynamic key-value memory networks for
knowledge tracing. arXiv (2017) arXiv:1611.08108. Available at https://arxiv.org/
abs/1611.08108

21
</reference>

<statements>
1. Student modeling spans overlay models, Bayesian Knowledge Tracing (BKT; Corbett & Anderson, 1994)—a hidden-Markov model with guess, slip, and learning parameters—and Deep Knowledge Tracing (DKT; Piech et al., 2015), which applies RNN/LSTM to response sequences, later extended by attention-based Self-Attentive Knowledge Tracing (SAKT).
</statements>

Begin the assessment now. Output only the JSON list, without any conversational text or explanations.