You will be provided with a reference and some statements. Please determine whether each statement is 'supported', 'unsupported', or 'unknown' with respect to the reference. Please note:
First, assess whether the reference contains any valid content. If the reference contains no valid information, such as a 'page not found' message, then all statements should be considered 'unknown'.
If the reference is valid, for a given statement: if the facts or data it contains can be found entirely or partially within the reference, it is considered 'supported' (data accepts rounding); if all facts and data in the statement cannot be found in the reference, it is considered 'unsupported'.

You should return the result in a JSON list format, where each item in the list contains the statement's index and the judgment result, for example:
[
    {
        "idx": 1,
        "result": "supported"
    },
    {
        "idx": 2,
        "result": "unsupported"
    }
]

Below are the reference and statements:
<reference>
International Journal of Artificial Intelligence in Education
https://doi.org/10.1007/s40593-022-00308-z
ARTICLE

Moving-Target Intelligent Tutoring System for
Marksmanship Training
Vladimir Zotov1

· Eric Kramkowski2

Accepted: 16 August 2022
© Crown 2022, corrected publication 2022

Abstract
Intelligent tutoring systems (ITSs) may augment military training systems and mitigate existing limitations in training personnel and resources. A study was conducted
to investigate the effectiveness of an embedded rifle marksmanship ITS for Moving
Targets (MT-ITS). MT-ITS has two main components: (1) a Smart Sight System
that provides a perceptual cue to help trainees adjust their point of aim to account
for a target’s speed, direction of movement, and distance, and (2) a performancebased algorithm that delivers shooting performance feedback to trainees.
The MT-ITS was tested in an experiment where participants engaged moving targets in a virtual shooting range. Moving targets were presented at different
speeds, direction of movement, and distances. Two types of marksmanship training
were compared: with ITS and without ITS (a standard training). The ITS training
group produced better hit rate and aiming accuracy scores than the standard training
group, requiring less practice to achieve asymptotic results. Implications for the design of embedded trainers with ITS for marksmanship specifically and for training
motor skills in general are discussed in the context of future research directions.

“The original online version of this article was revised: The article “Moving-Target Intelligent Tutoring
System for Marksmanship Training”, written by Vladimir Zotov and Eric Kramkowski, was originally
published electronically on the publisher’s internet portal on 12 September 2022 without open access.
With the author(s)’ decision to opt for Open Choice on 21 September 2022 the copyright © Crown
2022 was retained and the article is forthwith distributed under a Creative Commons Attribution 4.0
International License, which permits use, sharing, adaptation, distribution and reproduction in any
medium or format, as long as you give appropriate credit to the original author(s) and the source, provide
a link to the Creative Commons licence, and indicate if changes were made. The images or other third
party material in this article are included in the article’s Creative Commons licence, unless indicated
otherwise in a credit line to the material. If material is not included in the article’s Creative Commons
licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you
will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit

13

International Journal of Artificial Intelligence in Education

Introduction
Defence Research and Development Canada (DRDC) provides science and technology advice to the Canadian Armed Forces (CAF) including the evaluation of embedded computer-based technologies to improve training effectiveness. This study aims
to develop a proof-of-concept embedded Intelligent Tutoring System (ITS) that assists
marksmanship skills training. An ITS is commonly defined as a computer-based system that provides customized and immediate feedback to trainees (e.g., Nwana,1990;
Psotka et al., 1988; Sleeman & Brown, 1982). ITS incorporates some form of artificial intelligence (AI) and it requires minimal to no support from a human instructor.
It usually includes four major components (Sottilare et al., 2014): (a) a domain model
that contains expert knowledge and some typical errors and deficiencies that students
typically experience; (b) a learner model that represents the current state of knowledge of the user, letting ITS know who it is teaching; (c) a tutor model that enables
ITS to know how to teach by taking inputs from domain and learner models; and (d)
a user interface that registers and converts student’s inputs from one form (texting,
clicking, voice) to another (text, graphs, videos).
A training system is embedded when its components constitute an integral part of
a device or piece of equipment. Witmer & Knerr (1996) defined Embedded Training
(ET) as “a training concept … built into or added to a weapon system.” The two main
benefits of embedded trainers are that they are (1) embedded into operational hardware so the training they provide is an accurate representation of their equipment, and
(2) they are more accessible than formal, instructor-lead training, which increases the
amount of training time students can realistically receive.
Depending on the structure and methods of AI, an ITS can be classified into four
types: simple reflex, goal-based, model-based, and learning systems (Russell & Norvig, 2016). The simple reflex type includes the basic response systems that can react
directly to the current environment. Such systems can be built using simple if-then
rules and do not usually require the use of elaborate task representations or cognitive architectures. The other three types of models usually require cognitive architectures or task representations. Goal-based systems contain formal descriptions of their
intended final states and selection mechanisms that reduce the discrepancy between
their current states and their goal states. Learning system type ITS includes mechanisms that allow them to autonomously gather information and update their internal
states. The ITS presented in this paper is a training system that combines elements
of a simple reflex system with some more advanced mechanisms of a goal-based AI
system. The following section provides a background in the field of motor-cognitive
training systems, thus helping to place our ITS in the context of other similar systems, show its advantages and limitations, and discuss how our system can updated
to become a learning system.

13

International Journal of Artificial Intelligence in Education

Intelligent Tutoring Systems for Motor-Cognitive Skills
Intelligent Tutoring Systems have been successfully used in multiple academic,
industrial, and military settings (for reviews, see Kulik & Fletcher 2016; Nkambou et
al., 2010; Nwana, 1990; VanLehn, 2011). Nevertheless, the majority were designed
to train knowledge-based subjects—such as mathematics, economics, or computer
science (e.g., Steenbergen-Hu & Cooper 2014). The use of an ITS to train cognitivemotor skills has remained limited up until last few years.
Cognitive-motor skills are hard to train and evaluate outside of a controlled environment, requiring portable training tools to record and track body movements and
assess trainees’ motor performance in real time (Goldberg et al., 2018; Hodaie et
al., 2018). However, recent advances in portable technologies, large data processing, and sensor development lead to the emergence of new ITSs in such diverse
fields as medicine (Almiyad et al., 2017; Alvarez et al., 2015; Skinner et al., 2018),
sport (Lee & Kim, 2010), driving (Ropelato et al., 2018), and industry (Hodaie et
al., 2018; Marinescu-Muster et al., 2021; Westerfield et al., 2015). Despite the wide
proliferation of these applications, limited amount of them was tested and reported
in the research field of AI in education (Santos, 2016), the observation shared by
another recent review of psychomotor intelligent tutoring systems for training motorcognitive skills (Neagu et al., 2020).
In military, Mulgund and colleagues (1995) developed an Intelligent Flight
Trainer (IFT) aimed at training pilots for helicopter flight proficiency. The tutor
in this instance provided coaching and performance monitoring functions through
a synthetic voice generator. The testing of the IFT in a training simulator showed
that the system was successful in helping to improve skills related to maneuvering a
helicopter. Rickel & Johnson (1999) investigated a virtual agent-based system that
can be used as both a synthetic instructor and a virtual team member. Their training
system, “STEVE” (Soar Training Expert for Virtual Environments), was designed for
training procedural tasks related to mechanical maintenance and repair of naval ship
equipment. The synthetic agents guide the trainees in a shared virtual environment,
interacting through a voice processing module. STEVE proved to be a useful addition to naval operating procedures training despite its limited speech production and
a somewhat narrow selection of tasks.
More directly related to the current study is work examining deployment and
testing of ITSs to improve motor-cognitive marksmanship skills. Goldberg and colleagues (2018) developed an embedded, portable ITS using the Generalized Intelligent Framework for Tutoring (GIFT) architecture. The framework allows a quick
authoring and developing of training applications (Sottilare et al., 2012). Goldberg
and colleagues analyzed marksmanship performance in a shooting simulator, collecting data from multiple sensors embedded into an instrumented weapon (e.g., butt
pressure, respiration rate, barrel movement, trigger squeeze pressure). While the system is still being validated experimentally (including user-interface testing), it is one
of a few training systems that can assist with teaching a cognitive-motor skill.
Another ITS-based marksmanship tool was developed by Yeh & Ritter (2012).
Their Declarative to Procedural Moving Target Tutor (D2P/MTT) helps trainees to
learn to engage moving targets. D2P/MTT uses a step-by-step introduction to the

13

International Journal of Artificial Intelligence in Education

basics of shooting moving targets, such as estimating the distance to the target, the
speed of the target, and the angle of approach. Each of these elements is introduced
individually to trainees, allowing to transform factual knowledge that later become
part of their procedural knowledge. Later in training, these elements are combined,
and the process repeats until the trainees achieve a target level of performance in
adjusting a point of aim for distance and speed of the target. In experiments, the
D2P/MTT training tool was tested with both college students and active-duty Marine
instructors. The D2P/MTT-based training was compared with typical in-class instructions and showed an advantage in boosting participants’ procedural knowledge of
adjusting a point of aim (Yeh & Ritter, 2012).
Even though the development of ITSs for training motor-cognitive skills is
increasing, most of training systems described above were not tested experimentally.
The tutor presented in this paper attempts to fill this gap by evaluating and comparing an ITS-based training with a standard training technique modelled after currently
used by CAF marksmanship training.

The Moving Target ITS
The tutoring system presented in this paper helps shooters to improve marksmanship
skills required to engage moving targets. The task includes several motor-cognitive
skills and drills that are part of standard marksmanship training: estimating distance
to the target, estimating the speed of the moving target, adjusting point of aim (POA)
for a moving target, analyzing a missed shot, and learning tracking techniques. (for
review, see Schendel & Johnston 1983). Below, we describe the MT-ITS, including the functional components and mechanisms, and how they interact in a training
context.
Moving-Target ITS (MT-ITS) is a software-based tutor embedded into the digital sight of an instrumented rifle. The components of the tutor are shown in Fig.1.
The system is positioned into Virtual Immersive Soldier Simulator, a test bed that
provides the virtual setting for the tutor (VISS; Fig.1, top panel). The tutor includes
an instrumented rifle with a digital sight and a top-mounted tracking system (Fig.1,
bottom-left panel), and the Smart Sight System that assists trainee to properly aim
moving targets (Fig.1, bottom-right panel). The VISS environment allows a tracking system to pinpoint the orientation of the rifle and the shot location. When a shot
is fired at a moving target, the target continues to move during the time of flight of
the bullet. Thus, it is necessary to aim in front of the target, otherwise shots will fall
behind it. This aiming in front, to anticipate the movement of the target, is known as
lead. The amount of lead depends on the speed and distance to the moving target; the
Smart Sight System is responsible for processing and delivering this information to
trainees.
MT-ITS’s two feedback mechanisms—a real-time correction of aiming (delivered by the Smart Sight system) and an after-action feedback—comprise the main
software components of MT-ITS, representing the “intelligence” of the system. The
former guides trainees to adjust their target-tracking behaviour in real time and the
latter provides a customized feedback after each target engagement. Both mecha-

13

International Journal of Artificial Intelligence in Education

Fig. 1 Components of MT-ITS: VISS (top image), instrumented rifle (bottom left), and Smart Sight
(bottom right). (Note: The bottom-right panel shows Smart Sight System aiming reticle with full target
telemetry (range, speed, Military Grid Reference System location), shown as a vertical arrow and a
calculated point-of-aim indicator to hit the target’s center of mass shown as a red box.)

nisms were developed using Application Scripting Interface of Virtual Battle Space
2 (VBS2) simulation package (bisimulations.com). The VBS2 scripting language is
capable of a wide range of functions available through hundreds of script commands,
which satisfy the requirements of the feedback that MT-ITS provides.
For the real-time correction, the Smart Sight System calculates the optimal point
of aim and presents it to a trainee in a form of red rectangle superimposed on the
target, as shown on the bottom-right panel of Fig.1. The small red square shows the
qued aiming location adjusted for speed and distance to the target. MT-ITS presents
the same type of moving target (generated as a combination of distances, directions
of movement, and speed) with and without the aiming point sequentially, repeating
this combination of qued and unqued presentation for all types of targets. The process assumes that multiple instances of targets and corresponding aiming points are
transferred into memory, serving to improve and automate trainee’s performance.

13

International Journal of Artificial Intelligence in Education

Fig. 2 A training session feedback screens in MT-ITS training condition. The left panel shows a feedback screen generated after each shot and the right panel shows a summary of all shots fired at the
same target

This training mechanism is based on both theoretical grounds and practical considerations. The method of repeated presentation of multiple cued instances is based on
the Instance Theory of Automatization (Logan, 1988), which assumes that practice
leads to storing of multiple instances of trained material in memory, thus increasing the speed of retrieval and contributing to effortless performance. On a practical side, MT-ITS incorporated features that were added on advice and suggestion
of experienced combat arms soldiers of the CAF. These soldiers were recruited to
provide feedback and advice regarding the design of MT-ITS and its consistency with
CAF marksmanship training requirements. The feedback prompted a comprehensive
design modification addressing nearly all aspects the MT-ITS (e.g., target parameters,
settings of virtual environment, feedback form, and many others).
For the after-action feedback, MT-ITS records a trainee’s performance for each
target engagement, including hits, misses, position of shots, shot spread (grouping),
and the locations of the points of impact. This after-action feedback is delivered in
two ways: (a) a sequential presentation of each shots fired (Fig.2, left panel) and
(b) a statistical summary of all shots fired in the same trial (Fig.2, right panel). The
literature shows that blocked feedback (summarized by 5, 10, or 15 trials) improves
the retention of skills in transfer studies (Schmidt et al., 1989). Moreover, presenting
feedback on only the immediately preceding trial might be detrimental to training
effectiveness (Lavery & Sudden, 1962); one possible explanation is that immediate
feedback might overwhelm trainees with information and foster heavy reliance and
dependence on this form of feedback. For the same reason, the cued trials were combined with non-cued ones.
The feedback that MT-ITS provides are of two types: Knowledge of Results (KR),
which provides information about the outcomes of the action or the movement, and
Knowledge of Performance (KP), which provides information about the nature of the
action or movement (for further details see Schmidt et al., 2018). Both types of augmented feedback are important for learning a motor skill effectively, but most studies
up until last twenty years employed KR due to difficulties of capturing and analyzing

13

International Journal of Artificial Intelligence in Education

body motions (Santos, 2016). While MT-ITS was envisioned as a training tool incorporating both types of feedback, its current implementation principally presents KR
feedback in the form of shot statistics. KP feedback is partially supplied through the
Smart Sight, where trainees adjust the aiming point. The possibility of enhancing KP
feedback in MT-ITS is reviewed in the Future Works section.
In terms of standard ITS component described in Introduction, MT-ITS incorporates aspects of the domain and tutor models of ITS, as well as a user interface. The
learner model is largely undeveloped in this iteration of the system.
The MT-ITS’s domain module includes some key metrics of optimal marksmanship performance, as considered by expert marksmanship instructors: group size1
(the distance between the two most distant shot locations), distance between the
mean point of impact and the target’s centre of mass, and hit rate. These metrics
are compared with observed shooting performance and then incorporated into afteraction feedback.
The learner model requires details of each trainees’ knowledge states and, as such,
it is not well-elaborated in MT-ITS. We limited the applicability of our training tool
to a novice shooter. Consequently, the selection of the training settings was tuned for
this specific group. With a future plan to include a detailed evaluation of trainees’
marksmanship level, MT-ITS can be modified to customize its training difficulty,
depending on the trainee’s skill and knowledge level.
The tutor model takes input from the domain and the learner’s models in order
to customize its feedback to trainees. MT-ITS incorporates this information in both
forms of feedback it gives to trainees: as a guide to adjust point-of-aim in real time
and as a detailed performance feedback in after-action review. In its previous version, MT-ITS provided some tips and corrections as an after-action feedback (Zotov
et al., 2017). The content of tips was based on the discrepancy between observed
performance and expected optimal performance. For example, if most of the shots
were lagging the target, the tip would be “Your aim is generally lagging behind the
ideal aimpoint – increase lead significantly”. The tips were not included in the present version due to their limited training value (the issue is addressed in the General
Discussion section).
The goal of the user interface is to convert the formats of trainee’s inputs from one
form to another. MT-ITS records trainee performance as data in VBS2 script format
and converts it to texts, images, and diagrams presented to trainees in the two forms
of feedback outlined above.
Overall, MT-ITS aims to improve a standard training Small Arms Training (SAT)
simulation ranges used by Canadian Armed Forces for marksmanship training. The
differences between two training settings are in using Smart Sight System and the
amount of feedback that participants receive. What follows is our empirical efforts to
compare training that includes MT-ITS with training that does not. The objective was

1

It should be noted that the metric of group size should only be used as a measure of good or poor performance when the shooting task requires repeated, precise fire, as when asking to fire repeatedly at a
stationary target. Many tasks such as engaging moving targets, snap shooting or Close Quarters Battle
do not require small groupings to be effective but instead quick, accurate shots at various points of aim.

13

International Journal of Artificial Intelligence in Education

to test these two training methods using the same virtual environment and the same
amount of training.

Experiment
Two training methods were tested in the experiment: an ITS-enabled training that
used MT-ITS tool and a Standard training that replicated a typical training that soldiers of Canadian Armed Forces would receive in a SAT setting. The differences
in these two training conditions were the types of feedback. For Standard training,
trainees received no feedback during target engagement and limited feedback after
engagement, which consisted of showing target sheet with points of impact marked.
For the ITS-enabled training, trainees received aiming feedback in the form of the
Smart Sight and a detailed after-action feedback, described above. The experimental
design includes two control sessions (pre-training and post-training) and a training
session between them, thus the difference between the methods can be tested in the
analysis of pre- and post-training sessions. The analysis of the training session adds
some details of the learning dynamics.
Method
The experiment included three sessions: (1) a pre-experimental session (reading and
signing information forms, engaging in a warm-up exercise), (2) an experimental session consisting of pre-training (shooting different types of moving targets), training,
and post-training blocks (similar to pre-training) and (3) a post-experimental session, in which participants filled out training evaluation questionnaires. Table1 shows
the duration of each part; more detail will be provided below. Experienced combat
arms soldiers of the Canadian Armed Forces were recruited to provide feedback and
advice regarding the design and features of ITS-enabled training and Standard training conditions.
Participants Twenty-eight participants were recruited for this experiment. All
participants were from Reserve Canadian Army units. The participants were split
equally into two groups corresponding to the two training conditions (Standard and
MT-ITS) with fourteen participants in each group. The participants had little to no
experience with firearms and had served less than two years in the Canadian Armed
Forces2. Defence Research and Development Canada (DRDC) Human Research Ethics Board reviewed and approved the study protocol.
Apparatus and Stimuli The study was conducted in the VISS – a DRDC-built
research testbed used to examine marksmanship performance and small arms-related
technologies. The simulation is based on Bohemia Interactive Simulations’ Virtual
2

By absolute numbers most CAF soldiers are not considered “Combat Arms”. Combat Arms trades in the
CAF only include Infantry, Combat Engineer, Armoured and Artillery. The majority of people that wear an
army uniform are support trades. Units rotate through “High Readiness” or “Pre-Deployment Training” as
needed and weapons training is increased during these periods. In the case of the Reservist soldiers used
for this experiment it is not uncommon for soldiers with less than 2 years of service to have little experience with firearms.

13

International Journal of Artificial Intelligence in Education
Table 1 Sequence and duration
of experimental blocks

Sessions

Blocks

Pre-experimental

Pre-training

Duration
(min)
15

Experimental

Warm-up
Pre-Training

5
25

Break
Training, Part 1

5
20

Break
Training, Part 2

5
20

Break
Post-Training

10
25

Questionnaires

5

Post-experimental

Notes
Reading Information forms,
signing consent,
watching Pretraining videos
Shooting targets
Calibration
followed by 4
blocks of 24
targets
Calibration
followed by 64
training trials
Calibration
followed by 64
training trials
Calibration
followed by 4
blocks of 24
targets
Paper and
pencil forms

BattleSpace 2 (VBS2) software, with custom modifications that allow a fully motiontracked mock weapon to aim at and engage virtual targets in a CAVE-like background environment (Fig.1, top panel) spanning roughly 150 degrees of the shooter’s
field of view, projected across three 3m screens. An OLED micro-display fitted into
a Trijicon Advanced Combat Optical Gunsight housing acted as a simulated weapon
sight and was mounted on a modified airsoft rifle made by WETech, patterned after
a short-barrelled 10.3” M4 variant. Though the mock rifle did not physically recoil
when rounds were fired, virtual software recoil was added to the simulation to prevent participants from rapidly firing multiple rounds without having to re-acquire the
target. Reflective markers on the weapon allowed it to be tracked by a NaturalPoint
OptiTrack infrared camera system. The shooter was centred in the simulator, roughly
three meters away from the background screens, in a seated-supported position to
reduce discomfort and fatigue-induced aiming instability over the course of long trial
sessions.
The virtual environment consisted of an outdoor field scene with a long wall perpendicular to the shooter located either 100 or 250m away, with two gaps cut into it
(Fig.3). A human target appeared in one of the two gaps and attempted to run across
it at one of two speeds. The participant’s task was to hit the target as quickly and as
accurately as possible before it reached the end of the gap. To equalize exposure time
between slower-moving and faster-moving targets, gaps were 22 and 40m wide. The

13

International Journal of Artificial Intelligence in Education

Fig. 3 A three-walls, two gaps layout for the pre- and post-training sessions

Fig. 4 A training session hit feedback screen in the Standard condition, with a NATO E-Type silhouette
target

arrangement of the gaps and the distance to the walls was randomized and counterbalanced across trials.
Depending on the session and condition, two types of moving targets were used:
an animated target (Fig.2) and a simulated NATO E-Type silhouette target, 1m tall,
50cm wide (Fig.4). The surface areas of the silhouette of the E-type target and the
animated character were approximately equal.
Procedure During the pre-experimental session, participants read information
materials, signed consent forms, watched training videos, and engaged in a shooting
warming-up exercise (Table1). The printed information materials explained the procedure and provided some basic principles for engaging moving targets. The training
videos demonstrated the principles of shooting moving targets and presented information of how to adjust POA depending on the speed and the distance to target. For
the MT-ITS condition, the video contained additional information on how to use
the Smart Sight feature of the MT-ITS. The videos were designed and developed
specifically for this study using selected military marksmanship training materials
and advice from an Advanced Small Arms and Urban Operations qualified instructor.

13

International Journal of Artificial Intelligence in Education

The Advanced Small Arms course is the Canadian Armed Forces infantry qualification prerequisite for the rank of Sergeant. Its focus is to standardize and evaluate the
instructional and coaching methods used by candidates when teaching the use of
small arms to new and experienced soldiers. The Urban Operations Instructor course
is an advanced course that focuses on training soldiers of all experience levels to successfully engage the enemy at close ranges.
After completing the pre-experimental session, participants continued with the
experimental session, consisting of warm-up, pre-training, training, and post-training
blocks (Table1). Warm-up trials served to familiarize participants with the VISS, the
airsoft rifle, and the task by engaging 20 moving targets of varying speeds and distances. It was followed by the pre-training block, which was conducted to establish
baseline shooting performance for each participant. After a short break, the pre-training block was followed by the training block, in which participants were engaged
in a condition-specific coaching. After a break, participants started the post-training
block, which had a similar content to the pre-training block and served to test the
effectiveness of training. Prior to each block, the instrumented rifle was digitally
zeroed (calibrated), ensuring that the weapon sights aligned with the tracking system.
At the end of the experimental session, participants answered a questionnaire
about the quality of the training tools. The questions solicited information related
to the utility and training benefits of the training tools, asking participants’ opinions
about the structure of the training program, the number of targets presented, and their
opinions about the usefulness of the information presented in the training. Participants were also asked whether they felt they learned something new, and whether
the training methods could become a useful addition to marksmanship training. The
responses were rated on a five-point scale starting from “Strongly Disagree” (corresponding to 1 point) to “Strongly Agree” (5 points). The scale of the “Number of
Targets” question varied on a three-point scale (“Too few targets” - “About right
amount” - “Too many targets”, corresponding to -1, 0, 1 rates).
Sessions and Conditions The pre- and post-training sessions consisted of 96 trials each. On each trial, one of eight possible combinations of three target parameters
was presented: (1) two distances (100 and 250m), (2) two speeds (8km/h and 15km/h,
about the speed of a jog, and a fast run, respectively), and (3) two directions of movement (left-to-right and right-to-left) in all possible triplets. Pre- and post-training sessions were partitioned into four 24 trials chunks, with a short, participant-controlled
break between them (Table1). For each trial, participants were given up to eight shots
to successfully engage the target. In pre- and post-training sessions, after a target was
hit, it dropped, and a new trial was initiated. The pre- and post-training sessions were
identical and did not differ between two conditions.
There was a rest break halfway through the session. For each part of the training session, participant engaged 64 targets. These targets were obtained in the same
way as in the pre- and post-training sessions by combining all possible combinations
of distance, speed, and direction of movement. The training session used a pre-set
sequence of target presentations: participants in both conditions started with easyto-hit targets (100m, slow, either direction), continued with more challenging targets
(100m fast, 250m slow, either direction), concluding with the hardest to hit targets
(250m, fast, either direction). Considering that participants’ performance for easy-to-

13

International Journal of Artificial Intelligence in Education
Table 2 Number of training
trials as a function of speed and
distance

Speed
Distance
Trials

Table 3 Training sessions
details of MT-ITS and Standard
conditions

Conditions
Standard
MT-ITS
Pre-Training
A video introducing principles A video introducing principles of
of shooting moving targets
shooting moving targets, including
instruction on using Smart Sight
Training, Immediate feedback
No immediate feedback
Smart Sight
Training, After-Action feedback
• A target diagram that shows • Engagement diagrams that
all points of impact in the
shows an aiming point and a point
same trial
of impact for each shot
• A summary slide that shows all
points of impact, aiming points,
the target’s centre of mass, and a
number of hits
• In the second training session, a
target drops after three hits

Slow
100
20

250
20

Fast
100
40

250
48

hit targets was close to ceiling in the previous experiment (Zotov et al., 2017), the
proportion of these targets was lower: Table2 shows how many trials for each target
type were given to participants during the training session. The moving target’s exposure time was approximately 12s. As in pre- and post-training session, participant
could use up to eight shots.
The two training conditions differed on the type of instructions that participants
received: a standard training (Standard) and an ITS-based training (MT-ITS) with
partial KP feedback. The conditions differed in terms of the shape of target, the type
of feedback, and whether a point-of-aim cue was presented. In the Standard training
condition, the participants engaged an E-type silhouette target, moving either leftto-right or right-to-left between two barriers. Table3 summarizes the differences in
Standard and MT-ITS conditions. On each trial, participants could use up to eight
shots to hit the target. Even when a target was hit, it would not drop to allow participants to engage as many shots per targets as they could. After a target completed its
movement, participants received visual feedback in the form of a target diagram with
the location of all hits marked on it (Fig.2). The visual feedback was presented on an
auxiliary monitor (Fig.1, top-left corner) and in the digital sight; participants could
use either one depending on their preference.
For the MT-ITS training condition, participants were shown an animated target
moving across the range. Presentation of each target type was repeated, alternating
between showing and withholding a target cue. On the cued trial, participants were
shown the correct POA in the form of a small red square joined to the centre of mass
of the target by a red line (Fig.2). As in the Standard condition, participants could use
up to eight shots to hit the target. Whenever a hit was registered, the heads-up dis-

13

International Journal of Artificial Intelligence in Education

play flashed a red bracket around the reticle to provide additional real-time feedback.
After a target completed its movement (each target’s exposure was approximately
12s), participants received visual feedback in the form of sequential presentation of
a diagram for each shot taken in a trial with the point of impact and the optimal POA
of the target marked on it.
Like in the Standard condition, the visual feedback was presented on both an auxiliary computer monitor and on the digital aiming sight. Participants were able to
advance the feedback slides at their own pace using a button on the mock rifle. After
reviewing each shot, participants were shown a summary slide that showed the location of all hits and near misses for that trial. The MT-ITS training session consisted
of two parts that differed on the focus of the feedback: the first block emphasized
accuracy by indicating hit percentages, while the second block emphasized engagement speed by indicating “time-to-kill”, which was the timing lapsed between the
onset of the target’s movement and the moment when the target was hit. To encourage
participants to be more efficient and engage the target faster in this second block, the
target would drop after it was hit three times on the same trial.

Results
The data analyzed in this section are from the pre-training, training, and post-training sessions. The dependent measures were accuracy of shooting and lead distance.
Accuracy of shooting is the hit rate for each target type calculated as the ratio of hits
for each target, multiplied by 100. The number of shots to hit a target could vary
(up to eight shots per target), but this measure was not reported, considering that its
value depended on the different shooting styles of participants: some preferred to
use as few shots as possible and some would use multiple shots. The lead distance
(referred to hereafter as “lead”) is the horizontal difference between the observed and
the optimal POA in cm; with lead lagging behind optimal POA coded negative and
lead ahead of the optimal POA coded positive. The lead analysis tested if there were
any systematic biases in tracking moving targets. Considering that the recorded lead
is an instantaneous value at the moment of shot release, it does not provide any tracking behaviour over the course of the entire engagement.
The independent variables were training condition, distance to target, speed of
target movement, and training (Pre- versus Post-training). The results were analyzed
separately for testing sessions (comparing performance before and after training) and
for a training session. The final part of this section explores the results of the postexperimental questionnaires.
Pre- and Post-Training Sessions
The results of pre- and post-training sessions were analyzed in a mixed factor
ANOVA with three within-subject factors (distance × 2, speed × 2, training × 2) and
one between-subject factor (training condition × 2). We used an alpha level of 0.05
for all statistical tests. The direction of target movement has no effect on participants’
performance, therefore the left-to-right and the right-to-left data was collapsed.

13

International Journal of Artificial Intelligence in Education
Table 4 Effects of distance and speed of targets as function of training and condition
Measures
Accuracy
Lead
F
p
η2
F
p
η2
Distance
218.83
0.001
0.89
13.64
0.001
0.34
Distance x Condition
1.16
0.29
0.04
1.32
0.26
0.05
Distance x Speed
18.01
0.001
0.41
6.361 0.018
0.20
Distance x Speed x Condition
0.61
0.44
0.02
0.01
0.96
0.00
Distance x Training
11.96
0.001
0.31
4.66
0.05
0.15
Distance x Training x Condition
1.74
0.19
0.06
1.69
0.21
0.06
Distance x Training x Speed
7.76
0.05
0.23
0.25
0.62
0.01
Distance x Training x Speed x Condition
0.12
0.73
0.00
0.82
0.37
0.03
Speed
104.92
0.001
0.80
0.82
0.37
0.31
Speed x Condition
0.05
0.82
0.00
0.26
0.61
0.01
Speed x Training
0.15
0.70
0.00
0.25
0.62
0.01
Speed x Training x Condition
0.08
0.78
0.00
4.07
0.06
0.13
Training
48.41
0.001
0.65
7.17
0.01
0.22
Training x Condition
4.99
0.034
0.16
4.89
0.05
0.16
Condition
0.67
0.42
0.03
1.66
0.21
0.06
Notes. Bolded are significant effects. For p-values, we rounded significant results to 0.001 if p < .001;
0.01 if p < .01, and to 0.05 if p < .05. η2 is a partial eta squared. The corresponding effect sizes are small
for η2 = 0.1, medium η2 = 0.25, and large η2 = 0.40 (Cohen, 1988)

Table4 presents the results of the ANOVA for both accuracy of shooting and lead
measures. Considering a small sample size, the ANOVA was tested for homogeneity
using Levene’s test for equality of variances, which returned no significant value for
all tested pairs.
There was a strong effect of training (for accuracy, F(1, 26) = 48.41, p < .001), indicating that in the post-training session participants’ performance had improved. There
was also a significant interaction of training with condition, F(1, 26) = 4.99, p < .05,
indicating that the performance gain in MT-ITS training condition—for both shooting accuracy boost and a reduction in aiming lead—was significantly stronger than in
the Standard condition. That is, the MT-ITS training had produced significantly better
shooting accuracy and shorter discrepancy between optimal and observed lead distances than the standard training. The significant interaction of Condition x Distance
x Training shows that MT-ITS was especially effective for distant targets (250m). The
effect of Condition was not significant, confirming that performance in two training
groups differed only in post-training sessions. The effect of training was significant
on its own and in combination with a distance (Training x Distance and Training).
Table5 shows a summary of performance for average hit rates and lead as a function of distance, speed, training, and condition. Shooting accuracy was inversely
related to the distance to the target and the speed of the target; that is closer and
slower targets were hit with higher accuracy. Shooting accuracy for the easiest targets
(100m, slow) was at the ceiling, approaching 100%. As expected, the hardest targets
to hit were fast-moving distant targets (250m, fast): even with eight rounds at participants’ disposal, the targets were hit only around 60%. The effect of distance and
speed on performance were significant, confirming that it was harder to hit distant,
faster moving targets. What also worth mentioning is that the training was especially

13

International Journal of Artificial Intelligence in Education
Table 5 Mean values of performance measures as a function of speed, distance, and training
Independent Variables Speed
SLOW
FAST
Distance
100
250
100
250
Sessions
PRE POST PRE POST PRE POST PRE POST
TOTAL
Hits 96.1 98.9
71.7 87.6 86.9 94.7
60.6 70.3
Lead 13.9 13.3 -22.7 -12.
14.5 11.0 -27.0 -16.6
Conditions
STANDARD Hits 97.3 98.1
75.5 86.6 87.8 94.0
64.8 70.8
Lead 14.5 15.1 -24.0 -17.1 12.6 13.2 -23.3 -24.6
MT-ITS
Hits 94.9 99.1
67.8 88.6 86.0 95.5
56.4 69.9
Lead 13.3 11.5 -21.4 -7.2 16.4 8.8 -30.7 -8.5

effective for distant targets; in both training conditions the average gain in hit rates
was around 15% (unlike 3–4% for easy-to-hit 100m targets).
The lead measure fell into a bimodal pattern: for 100m targets participants were
leading too far ahead (leading too much), but for 250m targets participants were not
leading enough (lagging behind). One possible explanation to this finding is that
simultaneous change in distance and target speed result in a non-linear change in the
lead distance, leading to the underestimation of the lead—a well-documented human
bias of estimation of non-linear trends (e.g., Kwantes & Neal 2006).
Training Trials
The objective of the analysis was to test whether performance increments during
training trials follow a growth function and to analyze the shape of the function.
While the evidence of a training effect can be revealed through an increase in a linear,
power or exponential function (Anderson, 1982; Heathcote et al., 2000), the linear
function is the simplest function to test the observed data fit increasing, decreasing or
a flat trend (Pedhazur & Kerlinger, 1982). If there is a positive practice effect, then
the training data would fit an increasing function.
The linear function takes two parameters: b, which is a slope of the function, and a,
which is an intercept. The slope indicates whether the best fitting line follows increasing (positive value) or decreasing (negative value) function. The value of intercept
parameter is not important for analysis of training trials as it simply a translation
parameter that places the curve on the Y-axis without changing the shape of the linear
function.
The trial-to-trial data from the training trials were pulled for each type of target
and then fit to a linear function. Recall that participants in the MT-ITS condition
were trained with cued trials (presenting the location of the optimal POA for each
target). These cued targets were easier to hit; as a result, the hit rate for these trials
was higher. Thus, these trials were excluded from the sequence analysis to avoid contamination of the training performance by these boosted trials. The analysis included
128 training trials per participant for the Standard condition and 64 training trials per
participant for the MT-ITS condition. The number of the same type target types is
presented in Table2.
To avoid obscuring individual training trends by averaging, each participant’s
training data for each type of target (100m, 250m, separately for fast and slow tar-

13

International Journal of Artificial Intelligence in Education

gets) was fitted to a linear function3. using SPSS’ CURVEFIT method, which reports
significance of the fit and a and b parameters of the fit. The slope values of each participants’ fit to training data (b parameter) are shown in Table6. The positive values
indicate an increasing function (or a positive training trend) and the negative values
indicate decreasing function (or a negative training trend). As the table shows, individual results are quite diverse: there are cases of increasing, flat, and decreasing
trends. Nevertheless, most of the training curves in the MT-ITS condition show significant increasing function while most of the curves in the Standard condition show
flat curves. The training trials in the MT-ITS condition showed a consistent increasing function for all types of targets; participant performance was improving from
trial to trial. The training trials in the Standard condition showed a less consistent
pattern; the growing function was evident for some participants. Figure5 shows the
collapsed curves (averaged for each condition) for training trials in both conditions.
To show the training curves on the same scale, Fig.5 shows average gains in hit rates,
thus the scales’ range (-10 to 10) reflects a relative gain in performance expressed as
percentage. The figure demonstrates the overall trend in practice: the training trials in
the MT-ITS condition (top row) show a gradual improvement of shooting accuracy
with practice, whereas the training trials in the Standard condition (bottom row) do
not show a similar trend.
Post-Experimental Questionnaires
The post-experimental questionnaires asked participants about the quality of the
training tools: the structure of the training program, the number of targets presented,
the usefulness of the information presented in training, the novelty of the training
method, and the potential of the training methods become a useful addition to marksmanship training. Table7 presents the average values of the participants’ ratings for
each group. Overall, the participants in the MT-ITS condition gave more favourable
ratings than those in the Standard condition, but the differences were not significant.
Summary
The results of the experiment demonstrated that the MT-ITS training surpassed Standard training, helping to improve shooting accuracy and increased the precision of
aiming at the moving targets. The analysis of training trials revealed that that training rates with MT-ITS were more consistent in terms of producing steady gains in
performance with training; the same trend was not observed in the Standard condition. While the effect size corresponding to the advantage of MT-ITS training over
the Standard CAF training was small, it was impressive considering that the training
lasted less than an hour, our sample size was relatively small, and the participants rate
of improving was very diverse.
3

The lead measures were not included in the analysis due to technical issues involved in calculating
amount of lead for the Standard condition: the use of E-type1 targets instead of animated targets restricted
calculations of point of impact to only those that were on target; for off-target shots it was impossible to
calculate amount of deviation outside of target’s borders.

13

Table 6 Slope values of linear function fits to participants’ training data
Cond Tar- Participant
get
1
2
3
4
5
6
7
8
9
10
11
12
13
14
M
type
ITS 100S
6.91
1.00
4.88
6.73
3.25
3.78
7.32
3.09
3.02
-0.54
5.56
2.94
3.81
1.25
3.79
250S
2.60
-0.87
2.02
-1.64
-1.08
-0.96
1.67
4.29
5.71
1.62
0.62
-0.47
-2.15
0.66
0.86
100F
0.88
-0.25
1.65
2.36
-1.40
0.15
0.24
0.47
1.52
-0.43
1.42
1.66
0.28
0.78
0.67
250F
2.09
0.08
1.21
-0.78
1.58
0.89
-0.84
1.05
-0.93
0.08
-0.51
0.18
-0.08
0.00
0.29
ST
100S
-0.23
-0.11
-3.62
0.47
-0.47
1.26
0.96
0.43
0.00
1.76
0.35
1.41
1.25
-0.51
0.21
250S
-0.11
1.46
-0.20
0.20
-0.77
-0.58
0.73
1.29
-0.06
0.95
0.48
0.46
-0.50
0.83
0.30
100F
-0.32
0.27
0.25
-0.20
-0.08
-0.67
-0.12
-0.65
-0.05
-0.17
-0.10
-0.16
-0.08
0.02
-0.15
250F
0.03
0.52
0.01
0.18
0.50
-0.28
-0.22
0.13
-0.37
0.05
0.11
0.17
0.06
-0.25
0.05
Notes. Bolded numbers represent significant trends. Target type values consist of distance to target in meters (first number) and speed of target (S slow, F fast). ITS stands
for MT-ITS condition, ST stands for Standard condition

International Journal of Artificial Intelligence in Education

13

International Journal of Artificial Intelligence in Education

The advantages achieved by the MT-ITS marksmanship training against the “holes
in paper” approach used in the Standard condition might reflect the ability of the
MT-ITS to overcome the limitations inherent in the existing live-range procedures.
Unlike the conventional training, MT-ITS supplies trainees with a detailed information on their performance. For example, it supplies trainees with the locations of the
missed shots, which give an opportunity for participants to adjust their aiming points
for speed and distance. It shows each engaged target sequentially, allowing trainees
to link certain techniques or strategy to good or poor performance and make the
necessary adjustments. Finally, it helps to adjust point of aim with the help of cue
delivered by the Smart Sight system, thus helping trainees to perceiving intuitively
the correct point of aim. The advantage of Smart Sight system was especially obvious during training: the majority of our participants in MT-ITS condition steadily
progressed, unlike CAF participants, whose progress was not as steady. Possibly, in
the absence of cued trials in the Standard condition, participants used a trial-and-error
approach in estimating the proper POA.
Other results were consistent with a typical performance at marksmanship training: all participants performed well shooting slow targets at close range (100m).
Performance was noticeably worse for shooting fast targets at longer ranges where
accuracy dropped significantly. The study was consistent with a trend often observed
by marksmanship instructors: the accuracy of shooting was inversely related to the
distance and speed of the moving target. The direction of the target movement had no
effect on performance.

General Discussion
MT-ITS Modifications
The version of MT-ITS presented in this paper includes modifications that helped it to
overcome some of the shortcomings of the previous version (Zotov et al., 2017). That
version included text-based tips that informed participants of any systematic errors
they were making while engaging moving targets. The feedback in the previous version also included detailed statistics, such as the standard deviation of all shots and
the coordinates of each point of impact. While it sounds like a useful feature, participants in the previous study stopped paying attention to this feedback after only a few
trials. In the absence of behavioral and physiological responses collected (e.g., trigger squeeze pressure, barrel movement, respiration), the content of these tips became
repetitive and redundant; some participants reported that the detail of the provided
statistics was overwhelming.
Tips enhanced by rich sensory information can be re-introduced in the next iteration of MT-ITS (discussed in the Future Work section), but the most beneficial form
might not include verbal tips. Considering that at the cognitive/associative stage,
(which presumably trainees reach during the training sessions), the learners benefit
the most from brief, non-verbal corrections, preferably presented in graphical forms
(Singer et al., 1994). The sequential presentation of each engagement including

13

International Journal of Artificial Intelligence in Education

Fig. 5 Hit rates for training trials for both conditions. (Notes: For MT-ITS Condition, only non-que
trials are presented to avoid contamination by the higher accuracy cued trials)

Table 7 Mean values of participants ratings of the training
tools

Condition Structure

Number
Usefulness
of Targets

MT-ITS
Standard

0.20
0.01

5.00
4.67

4.73
4.52

Learned TrainNew
ing
Merit
4.53
4.73
4.44
4.52

observed and optimal POA for each shot, was added to help trainees pinpoint each
shot they fired in terms of aiming accuracy.
The experimental design underwent several changes too. In the previous study,
only one training session was used, whereas this version included two blocks. The
first part was focused on accuracy, while the second added a speed component, thus
mirroring cognitive and associative stages of skill acquisition theory (Fitts & Posner,
1967). The bulk of recruited participants were novices identified by their short duration of service (up to two years) and little to no experience shooting moving targets.
This was done to maximize the benefits of training. Participants were provided with
the post-experimental questionnaires to find out how useful the training tools were.
MT-ITS Limitations
Despite its benefits, MT-ITS does have several limitations related to poverty of sensor inputs, restrictions of the simulation environment, and deficiencies of instrumentation, placing the system in a category of an advanced feedback training system with
some AI elements rather than a fully-fledged ITS. A coaching system, such as MTITS, would benefit from adding a number of sensors to enhance quality of feedback,
as the work of Goldberg et al., (2018) showed (e.g., sensors for barrel movement,
trigger squeeze, recoil, and respiration).
The experimental sessions were conducted in the VISS, a synthetic environment
lacking certain features of the real world. One obvious way to improve on the “real-

13

International Journal of Artificial Intelligence in Education

ism” of the VISS is to update both the instruments used in the simulation and the
software. The experimental weapon used in the study was an airsoft rifle; although it
shares many similarities with the real weapon upon which it is based (an M4 CQBR),
some characteristics, such as recoil and especially trigger weight, pull, break and
reset are notably different. A more realistic weapon would bring the simulated training closer to live training. A better ballistic model4 than the one used in the simulation
would improve fidelity and accuracy of firing. The resolution of the simulation can be
further enhanced to allow higher, more accurate representation of the shooting environment. Nevertheless, it is worth mentioning that even the most “immersive “setting
may not yield any increase in training effectiveness and may create a less effective
training environment, as Smallman and colleagues (2007) noted.
Unlike a true embedded training system, which is installed in the actual operation hardware (in this case, a soldier’s issued weapon and sight system), the VISS
is a large external simulator which cannot be deployed with the trainee or provide a
100% accurate representation of the hardware’s features and handling characteristics.
While this study examined the content that might be featured in an embedded training
system, the benefits of physically embedded systems were not tested. In particular,
the greatly increased access and training time afforded by being installed in their own
weapon, ready to use at any time, was a factor that was actively avoided to maintain
an even comparison against the standard training content.
The MT-ITS condition used the same animated target types for all sessions, whereas
the Standard condition mixed animated targets in pre- and post-training sessions, and
“E-type” targets in the training session. The main reason for using “E-type” targets in
the Standard condition was to replicate the typical marksmanship training as closely
as possible, but this change added an artifact of switching types of targets. Possibly,
this change has no effect on performance: in our 2017 study, a standard condition
with “E-type” used as training targets (and animated targets for pre- and post-training
sessions) still showed significant advantage against a control, no-training condition
(Zotov et al., 2017), but in order to confirm that the process of switching targets
had not affected the performance in the Standard condition, the same target type
should be used in both conditions. Further, it is not clear what feature of MT-ITS was
responsible for training effectiveness. Was it the Smart Sight, the enhanced feedback,
or some other factors (or their combination) that contributed the most to the training
gains? This study can only confirm that MT-ITS has been a more effective training
system than a typical training that soldiers are trained with, but more research would
be required to reveal individual contributions of each of the MT-ITS training features.
By adding two new experimental conditions, the effect of individual contributions of
the Smart Sight system and the enhanced feedback (and their possible interaction)
can be identified and measured.
Considering that the participants were tested right after the training, it is important
to investigate the retention of the skill learned by ITS-based training. Re-testing participants in several intervals (e.g., after 1 month and 6 months) is necessary to find out
whether their initial training gain persists over time and by how much. In the previous
4

VISS used VBS2 as its underlying platform; a version with more advanced ballistics was added with
VBS3 module, as documented in Fügenschuh et al., 2016.

13

International Journal of Artificial Intelligence in Education

study (Zotov et al., 2017), six participants were re-tested six months after they completed their first experiment. The performance level of the re-tested participants was
on par with performance in their post-training session six months earlier and the difference between pre-training and re-test sessions was approaching significance. Nevertheless, with only six returning participants, the sample size was very small, and the
analysis lacked enough statistical power to detect significant differences on the order
of those observed; a larger sample size to obtain a definite answer is required. To better evaluate which of the training systems participants valued more, the two training
tools need to be provided to the same participants so they can compare them directly,
so that the post-experimental questionnaires can reflect which of two training conditions participants prefer and why. Finally, the effectiveness of MT-ITS must be tested
in a transfer of training trial, where two groups of participants would be trained with
either MT-ITS or live ranges and then tested in a live fire exercise. We hope that the
MT-ITS training would be as effective as the live range training, considering that
other marksmanship simulators showed their effectiveness in the transfer of training
studies (e.g., Grant 2013; Jensen & Woodson, 2012; Yates, 2004).
Future Work
As we pointed out, the main weakness of MT-ITS lays in its “low intelligence”. This
deficiency imposes considerable limits on its ability to provide a feedback customized for individual trainee’s needs. To improve training effectiveness of MT-ITS, we
propose several enhancements related to the type of feedback it provides and to the
system interface mechanism it uses. These enhancements are consistent with those
that Santos (2016) outlined and suggested for any future motor-cognitive ITS, such
as adding sensing movement and designing and delivering new forms of feedback.
To enhance the quality of real-time feedback we propose addition of a number of
sensors to monitor: a butt pressure, a barrel movement, a trigger squeeze, and a heart
rate and respiration monitors. A butt pressure sensor measures the force with which
a rifle’s butt is pressed against a trainee’s shoulder, optimally constant and within
a specific range. Too much or not enough pressure can negatively affect tracking
behaviour. A barrel movements sensor detects excessive barrel movements, which
can correlate with excessive butt pressure or ineffective breathing. A trigger squeeze
sensor detects amount of finger pressure on a trigger, possible revealing unproductive
trigger pressure during a target engagement. A respiration monitor can detect some
inefficient breathing prior to a target engagement (e.g., not holding a breath). Taken
together, information from these sensors can be incorporated into a real-time feedback and guide trainees to adjust their aiming behaviour. The feedback format can be
visual (as a diagram showing optimal and observed values of the sensor readings),
auditory (as a pitch that changes depending on the readings along with some verbal
feedback), and haptic (as vibrations triggered by too much butt pressure or excessive
barrel movements). This form of KP feedback might greatly enhance the quality of
information that a trainee gets in a real time. Finally, a hear rate monitor can supply
trainees with a way to estimate their stress level (detected by to elevated heart rate
and decreased heart rate variability, for more information see Thayer, 2012). Inte-

13

International Journal of Artificial Intelligence in Education

grated feedback could improve these techniques with such techniques as Tactical
Breathing (Bouchard et al., 2012).
The after-action feedback can be enhanced by applying a tracking algorithm to
analyze performance on a number of previous trials. This memory system would
receive input from the sensors that we described above, allowing some ineffective
patterns of engagement to be identified, shared with the trainees, and corrected.
These ineffective patters can be detected by comparing observed and optimal shooting behaviours, but the implementation of such mechanism requires advanced expert
model. Goldberg et al., (2018) presented a case of such a model for marksmanship
training, which was obtained by collecting performance and behaviour metrics of
shooting experts and then inputting these data into a regression analysis to obtain the
weight values for each metric to evaluate shooting behaviour. The form of feedback
could be a combination of text, visualization, and synthetic speech. Considering the
amount of customized information such a feedback would include, it might overcome
the problem of being too repetitive and redundant, as we observed in the verbal feedback that we included in our early versions of MT-ITS.
The system interface could be customized depending on skills and knowledge
level of a given trainee assessed during baseline evaluation and training sessions. As
we noticed, the baseline shooting performance of some of our participants was high
and the gains in hit rates were relatively low (and even were reduced by the end of
training session). With ability to customize training session, MT-ITS would adjust the
difficulty levels, including detecting plateaus in performance for some target types.
The system would sample performance data from a number of previous trials and
calculate slopes for different types of targets; as the slope values flatten, the training
algorithm will stop presenting this type of target. Another customization enhancement can be achieved by a performance tracking method that would allow to quickly
evaluate a trainees’ progress and to identify the most effective training intervention.
One of the ways to implement such method was demonstrated in Fenza, Orciuoli, and
Sampson study (2017), which presented a concept of ITS that uses a reinforcement
learning with artificial neural network to observe a trainee’s actions at different states
of the learning process. The selection mechanism of the tutor relies on a Next Task
Selection algorithm that evaluates a trainee’s performance and selects a new task to
keep a trainee in her zone of proximal development, which is an optimal level of task
difficulties most conducive to training improvements.
The effects of psychological stressors have been extensively investigated in recent
years (for review, see Vartanian, Boscarino, Jarmasz, & Zotov,2022). With added
ability to measure the level of stress, the future testing environment of MT-ITS might
include a number of stressors to bring the training condition closer to a live training
or even to a combat setting. The stress level during marksmanship training can be
induced by presenting multiple targets simultaneously, adding weapon malfunctions,
including hostile fire, and placing trainees in a competitive environment. Monitoring stress level will allow to place a trainee in the optimal task difficulty zone to
achieve the highest training benefits (Guadagnoli & Lee, 2002). Overall, the proposed enhancements that we listed above will substantially increase the autonomy
and the level of customization of MT-ITS, placing it in a category of learning AI
systems

13

International Journal of Artificial Intelligence in Education

Conclusion
Despite its limitations, MT-ITS is one of the first AI-based systems for coaching
motor-cognitive skills that has undergone extended testing against other training
methods. The GIFT-based marksmanship training system (Goldberg et al., 2018) is
another example of AI-based system that has undergone extensive testing against
both simple instruction and human instructor and might provide a good comparative
system to evaluate training effectiveness (personal interaction with the first author).
Marksmanship training with MTT/DP2 was extensively tested, but the MT/DP2 tutor
aims to teach only the cognitive aspect of the marksmanship skill (specifically, how
to adjust for target characteristics). Unlike MTT/DP2, MT-ITS helps master actual
skill rather than part of it.
MT-ITS may eventually be able to overcome known limitation of human instruction in the teaching of discrete cognitive-motor skills such as marksmanship. Rauter
and colleagues (Rauter et al., 2011) characterized these as inability to hold attention
and concentration over a longer period of time, no access to important physiological
and biomechanical variables that characterize movements, and considerable limitations in providing augmented concurrent feedback. As any other motor-cognitive
ITS, our tutoring system would be vigilant through the entire training session, have
a great potential to access numerous biomechanical and physiological variables, and
could give a real-time feedback that includes multiple streams of collected information. The results of this study show that an embedded training system can provide
substantial training augmentation —if not a partial substitution for live training—that
might be efficient, economical, and flexible in its ability to create a wide range of
scenarios. There are considerable potential benefits of delivering and practicing the
skill through an ITS that is embedded in the training tools. An autonomous ET system that requires little to no human support might become a practical alternative to
teach motor-cognitive skills, providing useful automated coaching in the absence of
a qualified instructor.
Declarations
Conflict of Interest The authors have no conflicts of interest to declare. The co-author has seen and agrees
with the contents of the manuscript and there is no financial interest to report.
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License,
which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long
as you give appropriate credit to the original author(s) and the source, provide a link to the Creative
Commons licence, and indicate if changes were made. The images or other third party material in this
article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to
the material. If material is not included in the article’s Creative Commons licence and your intended use
is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission
directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/
licenses/by/4.0/.

13

International Journal of Artificial Intelligence in Education

References
Almiyad, M. A., Oakden-Rayner, L., Weerasinghe, A., & Billinghurst, M. (2017). Intelligent augmented
reality tutoring for physical tasks with medical professionals. In International Conference on Artificial Intelligence in Education (pp.450–454). Springer, Cham
Alvarez, N., Sanchez-Ruiz, A., Cavazza, M., Shigematsu, M., & Prendinger, H. (2015). Narrative balance
management in an intelligent biosafety training application for improving user performance. International Journal of Artificial Intelligence in Education, 25, 35–59
Anderson, J. R. (1982). Acquisition of cognitive skill. Psychological Review, 89, 369–406
Bouchard, S., Bernier, F., Boivin, E., Morin, B., & Robillard, G. (2012). Using biofeedback while
immersed in a stressful videogame increases the effectiveness of stress management skills in soldiers.
PloS ONE, 7, e36169
Cohen, J. (1988). Statistical power analysis for the behavioral sciences (p. 54). New York. NY: Academic
Fenza, G., Orciuoli, F., & Sampson, D. G. (2017). Building adaptive tutoring model using artificial neural
networks and reinforcement learning. In 2017 IEEE 17th international conference on advanced learning technologies (ICALT) (pp.460–462). IEEE
Fitts, P. M., & Posner, M. I. (1967). Human performance. Brooks/Cole
Fügenschuh, A., Vierhaus, I., Fleischmann, S., Löffler, T., Diefenbach, T., Lechner, U., & Marahrens,
S. (2016). VBS3 as an Analytical Tool-Potentialities, Feasibilities and Limitations. Universität der
Bundeswehr Hamburg: Helmut Schmidt Universität
Goldberg, B., Amburn, C., Ragusa, C., & Chen, D. W. (2018). Modeling expert behavior in support of
an adaptive psychomotor training environment: A marksmanship use case. International Journal of
Artificial Intelligence in Education, 28, 194–224
Grant, S. C. (2013). Small Arms Trainer Validation and Transfer of Training: C7 Rifle. Defence Research
and Development Canada
Heathcote, A., Brown, S., & Mewhort, D. J. K. (2000). The power law repealed: The case for an exponential law of practice. Psychonomic Bulletin & Review, 7, 185–207
Hodaie, Z., Haladjian, J., & Bruegge, B. (2018). TUMA: Towards an Intelligent Tutoring System for
Manual-Procedural Activities. In International Conference on Intelligent Tutoring Systems, (pp.326–
331). Springer, Cham
Jensen, T., & Woodson, J. (2012). A naval marksmanship training transfer study: The use of indoor simulated marksmanship trainers to train for live fire. Naval Postgraduate School Monterey CA
Kulik, J. A., & Fletcher, J. D. (2016). Effectiveness of intelligent tutoring systems: a meta-analytic review.
Review of Educational Research, 86, 42–78
Kwantes, P. J., & Neal, A. (2006). Why people underestimate y when extrapolating in linear functions.
Journal of Experimental Psychology: Learning Memory and Cognition, 32, 1019
Lavery, J. J., & Suddon, F. H. (1962). Retention of simple motor skills as a function of the number of trials
by which KR is delayed. Perceptual and Motor Skills, 15, 231–237
Lee, D. H., & Kim, J. H. (2010). A framework for an interactive robot-based tutoring system and its application to ball-passing training. In 2010 IEEE International Conference on Robotics and Biomimetics
(pp.573–578). IEEE
Logan, G. D. (1988). Toward an instance theory of automatization. Psychological review, 95, 492
Marinescu-Muster, R., de Vries, S., & Vollenbroek, W. (2021). A Deep Learning Approach. In Ludic (Ed.),
Co-design and Tools Supporting Smart Learning Ecosystems and Smart Education (pp. 197–209).
Singapore: Springer. Data-Driven Intelligent Tutoring System for Accelerating Practical Skills
Development
Mulgund, S., Asdigha, M., Zacharias, G., Krishnakumar, K., & Dohme, J. (1995). An intelligent tutoring
system for simulator-based helicopter flight training. In Flight Simulation Technologies Conference
(p.3404)
Neagu, L. M., Rigaud, E., Travadel, S., Dascalu, M., & Rughinis, R. V. (2020, June). Intelligent Tutoring
Systems for Psychomotor Training–A Systematic Literature Review. In International Conference on
Intelligent Tutoring Systems (pp.335–341). Springer, Cham
Nkambou, R., Mizoguchi, R., & Bourdeau, J. (2010). Advances in intelligent tutoring systems (308 vol.).
Springer Science & Business Media
Nwana, H. S. (1990). Intelligent tutoring systems: an overview. Artificial Intelligence Review, 4, 251–277
Pedhazur, E. J., & Kerlinger, F. N. (1982). Multiple regression in behavioral research. Holt, Rinehart, and
Winston

13

International Journal of Artificial Intelligence in Education
Psotka, J., Massey, L. D., & Mutter, S. A. (1988). Intelligent tutoring systems: Lessons learned. Psychology Press
Rauter, G., Sigrist, R., Baur, K., Baumgartner, L., Riener, R., & Wolf, P. (2011). A virtual trainer concept
for robot-assisted human motor learning in rowing. In BIO Web of Conferences (Vol.1, p.00072).
EDP Sciences
Rickel, J., & Johnson, W. L. (1999). Animated agents for procedural training in virtual reality: Perception,
cognition, and motor control. Applied artificial intelligence, 13, 343–382
Ropelato, S., Zünd, F., Magnenat, S., Menozzi, M., & Sumner, R. (2018). Adaptive tutoring on a virtual
reality driving simulator.International SERIES on Information Systems and Management in Creative
EMedia(CreMedia).
Santos, O. C. (2016). Training the body: The potential of AIED to support personalized motor skills learning. International Journal of Artificial Intelligence in Education, 26, 730–755
Schendel, J. D., & Johnston, S. D. (1983). A study of methods for engaging moving targets. Human factors, 25, 693–700
Schmidt, R. A., Lee, T. D., Winstein, C., Wulf, G., & Zelaznik, H. N. (2018). Motor control and learning:
A behavioral emphasis. Human kinetics
Schmidt, R. A., Young, D. E., Swinnen, S., & Shapiro, D. C. (1989). Summary knowledge of results for
skill acquisition: Support for the guidance hypothesis. Journal of Experimental Psychology: Learning Memory and Cognition, 15, 352
Singer, R. N., Cauraugh, J. H., Chen, D., Steinberg, G. M., Frehlich, S. G., & Wang, L. (1994). Training
mental quickness in beginning/intermediate tennis players. The sport psychologist, 8, 305–318
Skinner, A., Diller, D., Kumar, R., Cannon-Bowers, J., Smith, R., Tanaka, A., & Perez, R. (2018). Development and application of a multi-modal task analysis to support intelligent tutoring of complex skills.
International journal of STEM education, 5, 1–17
Sleeman, D., & Brown, J. S. (1982). Intelligent tutoring systems. London: Academic Press
Smallman, H., Cook, M. B., Manes, I., & Cowen, M. (2007). Naive Realism in Terrain Appreciation. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 51, 1317–1321
Sottilare, R. A., Graesser, A., Hu, X., & Goldberg, B. (Eds.). (2014). Design Recommendations for Intelligent Tutoring Systems: Volume 2-Instructional Management (Vol. 2). US Army Research Laboratory
Sottilare, R. A., Brawner, K. W., Goldberg, B. S., & Holden, H. K. (2012). The generalized intelligent
framework for tutoring (GIFT). Orlando, FL: US Army Research Laboratory–Human Research &
Engineering Directorate (ARL-HRED)
Steenbergen-Hu, S., & Cooper, H. (2014). A meta-analysis of the effectiveness of intelligent tutoring systems on college students’ academic learning. Journal of Educational Psychology, 106, 331–347
Thayer, J. F., Ahs, F., Fredrikson, M., Sollers, J. J., & Wager, T. D. (2012). A meta-analysis of heart rate
variability and neuroimaging studies: implications for heart rate variability as a marker of stress and
health. Neuroscience & Biobehavioral Reviews, 36, 747–756
VanLehn, K. (2011). The relative effectiveness of human tutoring, intelligent tutoring systems, and other
tutoring systems. Educational Psychologist, 46, 197–221
Vartanian, O., Boscarino, C., Jarmasz, J., Zotov, V. (2022). Training-Related Stress and Performance in the
Military. In: Sookermany, A.M. (eds) Handbook of Military Sciences. Springer, Cham. https://doi.
org/10.1007/978-3-030-02866-4_60-1
Westerfield, G., Mitrovic, A., & Billinghurst, M. (2015). Intelligent augmented reality training for motherboard assembly. International Journal of Artificial Intelligence in Education, 25, 157–172
Witmer, B. G., & Knerr, B. W. (1996). A guide for early embedded training decisions (ARI Research Product 96 – 06). Alexandria, VA: US Army Research Institute for the Behavioral and Social Sciences
Yates, W. W. (2004). A training transfer study of the indoor simulated marksmanship trainer. Naval Postgraduate School Monterey CA
Yeh, K. C., & Ritter, F. E. (2012). An initial evaluation of the D2P/MTT, a computer-based, Declarative
to Procedural (D2P) theory driven moving target tutor. Pennsylvania State University State College,
Department of Computer Science and Engineering
Zotov, V., Jarmasz, J., Ueno, K., & Kramkowski, E. (2017). Assessing the Training Effectiveness of an
Intelligent Tutoring System for Marksmanship Skills. In Proceedings of the Human Factors and
Ergonomics Society Annual Meeting (Vol. 61, No. 1, pp.1492–1496). Sage CA: Los Angeles, CA:
SAGE Publications
Publisher’s Note Springer Nature remains neutral with regard to jurisdictional claims in published maps
and institutional affiliations.

13

International Journal of Artificial Intelligence in Education

Authors and Affiliations
Vladimir Zotov1 · Eric Kramkowski2
Vladimir Zotov
Vladimir.Zotov@drdc-rddc.gc.ca
1

Defence R&D Canada, 1133 Sheppard Ave West, M3M 3B9 Toronto, ONP.O. Box 2000,
Canada

2

QTAC Inc, Toronto, Canada

13
</reference>

<statements>
1. Sports ITS frameworks driven by multimodal fusion have been evaluated across general physical education classes, high-performance athletic training, and clinical injury prevention programs [7].
2. In competitive athletic environments, intelligent tutoring systems identify micro-kinematic deviations that often go unnoticed during standard coaching observations [7].
3. In marksmanship training programs utilizing rifle-mounted IMUs and optical sighting trackers, the MT-ITS platform provided real-time feedback on weapon stability, breathing cycles, and aim point tracking [7].
4. Trainees using this system demonstrated faster target acquisition times and higher hit accuracy on moving targets than control groups receiving standard human coaching [7].
</statements>

Begin the assessment now. Output only the JSON list, without any conversational text or explanations.