You will be provided with a reference and some statements. Please determine whether each statement is 'supported', 'unsupported', or 'unknown' with respect to the reference. Please note:
First, assess whether the reference contains any valid content. If the reference contains no valid information, such as a 'page not found' message, then all statements should be considered 'unknown'.
If the reference is valid, for a given statement: if the facts or data it contains can be found entirely or partially within the reference, it is considered 'supported' (data accepts rounding); if all facts and data in the statement cannot be found in the reference, it is considered 'unsupported'.

You should return the result in a JSON list format, where each item in the list contains the statement's index and the judgment result, for example:
[
    {
        "idx": 1,
        "result": "supported"
    },
    {
        "idx": 2,
        "result": "unsupported"
    }
]

Below are the reference and statements:
<reference>
Design Recommendations
for
Intelligent Tutoring Systems
Volume 1
Learner Modeling

Edited by:
Robert Sottilare
Arthur Graesser
Xiangen Hu
Heather Holden

A Book in the Adaptive Tutoring Series

Copyright © 2013 by the U.S. Army Research Laboratory.
Copyright not claimed on material written by an employee of the U.S. Government.
All rights reserved.
No part of this book may be reproduced in any manner, print or electronic, without written
permission of the copyright holder.
The views expressed herein are those of the authors and do not necessarily reflect the views of the U.S. Army
Research Laboratory.
Use of trade names or names of commercial sources is for information only and does not imply endorsement
by the U.S. Army Research Laboratory.
This publication is intended to provide accurate information regarding the subject matter addressed herein. The
information in this publication is subject to change at any time without notice. The U.S. Army Research
Laboratory, nor the authors of the publication, makes any guarantees or warranties concerning the information
contained herein.
Printed in the United States of America
First Printing, July 2013
First Printing (errata addressed), August 2013
U.S. Army Research Laboratory
Human Research & Engineering Directorate
SFC Paul Ray Smith Simulation & Training Technology Center
Orlando, Florida

International Standard Book Number: 978-0-9893923-0-3
We wish to acknowledge the editing and formatting contributions of Carol Johnson, ARL

Dedicated to current and future scientists and developers of adaptive learning technologies

CONTENTS

Preface

i

Section I: Fundamentals of Learner Modeling

1

Chapter 1 ‒ A Guide to Understanding Learner Models .......... 3
Arthur Graesser (University of Memphis)

Chapter 2 ‒ Lowering the Barrier to Adoption of Intelligent
Tutoring Systems through Standardization............................ 7
Robby Robson (Eduworks Corporation)
Avron Barr (Aldo Ventures)

Chapter 3 ‒ Important Considerations for Learner Models:
Transfer Potential and Pedagogical Content Knowledge.... 15
Alan Lesgold (University of Pittsburgh)
Arthur Graesser (University of Memphis)

Chapter 4 ‒ Matching Learner Models to Instructional
Strategies ................................................................................... 23
Andrew M. Olney (University of Memphis)
Whitney L. Cade (University of Memphis)

Chapter 5 ‒A Review of Student Models Used in
Intelligent Tutoring Systems................................................... 39
Philip I. Pavlik Jr. (University of Memphis)
Keith Brawner (U.S. Army Research Laboratory)
Andrew Olney (University of Memphis)
Antonija Mitrovic (University of Canterbury)

Section II: Current Learner Modeling Tools and
Methods

69

Chapter 6 ‒Understanding Current Learner Modeling
Approaches ............................................................................... 71
Heather K. Holden (U.S. Army Research Laboratory)

Chapter 7 ‒Affective-Behavioral-Cognitive (ABC)
Learner Modeling .................................................................... 75
Abhiraj Tomar (University of North Texas)
Rodney D. Nielsen (University of North Texas)

Chapter 8 ‒The Need for Empirical Evaluation of
Learner Model Elements ......................................................... 87
Heather K. Holden (U.S. Army Research Laboratory)
Anne M. Sinatra (U.S. Army Research Laboratory)

Chapter 9 ‒On the Use of Learner Micromodels as Partial
Solutions to Complex Problems in a Multiagent,
Conversation-based Intelligent Tutoring System ................. 97
Xiangen Hu (University of Memphis)
Donald M. Morrison (University of Memphis)
Zhiqiang Cai (University of Memphis)

Chapter 10 ‒Learner Models in the Large-Scale
Cognitive Modeling (LSCM) Initiative................................ 111
Scott A. Douglass (U.S. Air Force Research Laboratory)

Section III: Emerging Learner Modeling
Approaches

127

Chapter 11 ‒Emerging Learner Modeling Concepts ............. 129
Xiangen Hu (University of Memphis)
Donald M. Morrison (University of Memphis)

Chapter 12 ‒The Need for a Mathematical Model of
Intelligent Tutoring ............................................................... 133
Robby Robson (Eduworks Corporation)
Xiangen Hu (University of Memphis)
Donald M. Morrison (University of Memphis)
Zhiqiang Cai (University of Memphis)

Chapter 13 ‒Modeling Student Competencies in Video
Games Using Stealth Assessment ......................................... 141
Valerie Shute (Florida State University)
Matthew Ventura (Florida State University)
Matthew Small (Florida State University)
Benjamin Goldberg (U.S. Army Research Laboratory)

Chapter 14 ‒Assessing the Disengaged Behaviors of
Learners .................................................................................. 153
Ryan S.J.d. Baker (Teachers College Columbia University)
Lisa M. Rossi (Worcester Polytechnic Institute)

Chapter 15 ‒Knowledge Component (KC) Approaches to
Learner Modeling .................................................................. 165
Vincent Aleven (Carnegie Mellon University)
Kenneth R. Koedinger (Carnegie Mellon University)

Chapter 16 ‒Towards Learner Models based on Learning
Progressions (LPs) in DeepTutor ......................................... 183
Vasile Rus (University of Memphis)
William Baggett (University of Memphis)
Elizabeth Gire (University of Memphis)
Don Franceschetti (University of Memphis)
Mark Conley (University of Memphis)
Arthur Graesser (University of Memphis)

Section IV: Future Learner Modeling Concepts

193

Chapter 17 ‒ Pushing and Pulling Toward Future ITS
Learner Modeling Concepts ................................................. 195
Robert A. Sottilare (U.S. Army Research Laboratory)

Chapter 18 ‒ Learner Modeling to Predict Real-Time
Affect in Serious Games ....................................................... 199
James Lester (North Carolina State University)
Bradford Mott (North Carolina State University)
Jonathan Rowe (North Carolina State University)
Jennifer Sabourin (North Carolina State University)

Chapter 19 ‒Intelligent Creativity Support ............................ 209
Winslow Burleson (Arizona State University)
Kasia Muldner (Arizona State University)

Chapter 20 ‒Learner Modeling Considerations for a
Personalized Assistant for Learning (PAL) ........................ 217
Damon Regan (The Tolliver Group, Inc.)
Elaine M. Raybourn (Sandia National Laboratories)
Paula J. Durlach (Advanced Distributed Learning Initiative)

Chapter 21 ‒Eye-Tracking for Student Modelling in
Intelligent Tutoring Systems................................................. 227
Cristina Conati (University of British Columbia)
Vincent Aleven (Carnegie Mellon University)
Antonija Mitrovic (University of Canterbury)

Chapter 22 ‒Shared Mental Models of Cognition for
Intelligent Tutoring of Teams ............................................... 237
J.D. Fletcher (Institute for Defense Analyses)
Robert A. Sottilare (U.S. Army Research Laboratory)

Biographies

253

Acronyms List

267

Index

273

Design Recommendations for Intelligent Tutoring Systems - Volume 1: Learner Modeling

Preface

Robert A. Sottilare1, Arthur C. Graesser2, Xiangen Hu2, and
Heather Holden1, Eds.
U.S. Army Research Laboratory - Human Research and Engineering Directorate1
University of Memphis Institute for Intelligent Systems2

i

Design Recommendations for Intelligent Tutoring Systems - Volume 1: Learner Modeling
This book is the first in a planned series of books that examine key topics (e.g., learner modeling,
instructional strategies, authoring, domain modeling, learning effect, and team tutoring) in intelligent
tutoring system (ITS) design through the lens of the Generalized Intelligent Framework for Tutoring
(GIFT; Sottilare, Brawner, Goldberg, and Holden, 2012), a modular, service-oriented architecture created
to develop standards for authoring, managing instruction, and analyzing the effect of ITS technologies.
This preface introduces tutoring functions, provides learner modeling examples, and examines the
motivation for standards for the design, authoring, instruction, and analysis functions within ITSs. Next,
we introduce GIFT design principles, and finally, we discuss how readers might use this book as a design
tool. We begin by examining the concept of learner modeling.
Learner modeling (also known as student modeling or user modeling) is one of the major components of
ITS. Learner modeling is a key to the computer-based tutor’s understanding of the learner.
Comprehensive, real-time modeling of the learner is a critical element in the design and development of
truly adaptive tutoring systems that can tailor tutoring experiences to the needs of the individual learner
and teams of learners.
It is generally accepted that an ITS has four major components (Elson-Cook, 1993; Nkambou, Mizoguchi
& Bourdeau, 2010; Graesser, Conley & Olney, 2012; Psotka & Mutter, 2008; Sleeman & Brown, 1982;
VanLehn, 2006; Woolf, 2009): The domain model, the student model, the tutoring model, and the userinterface model. GIFT similarly adopts this four-part distinction, but with slightly different corresponding
labels (domain module, learner module, pedagogical module, and tutor-user interface) and the addition of
the sensor module, which can be viewed as an expansion of the user interface.
(1) The domain model contains the set of skills, knowledge, and strategies of the topic being tutored.
It normally contains the ideal expert knowledge and also the bugs, mal-rules, and misconceptions
that students periodically exhibit.
(2) The learner model consists of the cognitive, affective, motivational, and other psychological
states that evolve during the course of learning. It is often viewed as an overlay (subset) of the
domain model, which changes over the course of tutoring. For example, “knowledge tracing”
tracks the learner’s progress from problem to problem and builds a profile of strengths and
weaknesses relative to the domain model (Anderson, Corbett, Koedinger & Pelletier, 1995). An
ITS may also consider psychological states outside of the domain model that need to be
considered as parameters to guide tutoring.
(3) The tutor model (also known as the pedagogical model or the instructional model) takes the
domain and learner models as input and selects tutoring strategies, steps, and actions on what the
tutor should do next in the exchange. In mixed-initiative systems, the learners may also take
actions, ask questions, or request help (Aleven, McClaren, Roll & Koedinger, 2006; Rus &
Graesser, 2009), but the ITS always needs to be ready to decide “what to do next” at any point
and this is determined by a tutoring model that captures the researchers’ pedagogical theories.
(4) The user interface interprets the learner’s contributions through various input media (speech,
typing, clicking) and produces output in different media (text, diagrams, animations, agents). In
addition to the conventional human-computer interface features, some recent systems have
incorporated natural language interaction (Graesser et al., 2012; Johnson & Valente, 2008),
speech recognition (D’Mello, Graesser & King, 2010; Litman, 2013), and the sensing of learner
emotions (Baker, D’Mello, Rodrigo & Graesser, 2010; D’Mello & Graesser, 2010; Goldberg,
Sottilare, Brawner, Holden, 2011).

ii

Design Recommendations for Intelligent Tutoring Systems - Volume 1: Learner Modeling
The designers of the learner model need to decide what content, fields, variables, and parameters need to
be included in the representation. The representation needs to be complete with respect to handling the
distinctions made in the domain model, tutoring model, and user interface. Such representations vary in
grain size, reflecting the complexity of the ITS. There is a comparatively small number of distinctions
made in conventional computer-based training (O’Neil & Perez, 2003). For example, a simple system
would just keep track of whether the learner has mastered (yes versus no) a set of N learning objects in
the curriculum and the objects in a curriculum would be ordered theoretically, perhaps from simple to
complex or along a prerequisite ladder (Gagne, 1985). The tutoring module would select the next learning
object that the learner has not mastered and places that as lowest/earliest in the ordering. This kind of
simple system may go a long way. However, ITSs presume to go a large step further in grain size and
adaptability. Some of these ITSs are listed below, but there are many others that can be discussed at the
workshop. One foundational question is whether the increased grain size and adaptability has an
incremental return on investment with respect to learning gains.

Learner Modeling Examples
The following sections briefly describe four examples of learner modeling that have been developed and
tested in contemporary ITSs.

Knowledge Tracing in the Cognitive Tutor
This approach to learner modeling tracks the learner’s progress from problem to problem and builds a
profile of strengths and weaknesses relative to the production rules (Anderson et al., 1995). A production
rule is an “IF<state>THEN<action>” expression that specifies that a particular action, step, or cognitive
event occurs in a particular state of the task or cognition. Information from knowledge tracing can be
presented as a skillometer, a visual graph of the learner’s success in each of the monitored skills related to
solving problems in a step by step fashion. The skillometer is updated as the learner performs correct
actions, commits errors, and requests a hint. Step-by-step knowledge tracing is incorporated in a number
of tutors in the Pittsburgh Science of Learning Center (Aleven et al., 2006; Anderson et al., 1995;
Heffernan, Koedinger & Razzaq, 2008; Ritter, Anderson, Koedinger & Corbett, 2007; VanLehn, 2006).

Constraint-based Modeling
In constraint-based tutors, a good solution is represented as a declarative structure and the learner’s
actions are compared with these constraints (Mitrovic, Martin & Suraweera, 2007; Ohlson, 1992). Each
constraint is a declarative statement composed of a relevance condition (R) and a satisfaction condition
(S). The relevance condition specifies when the constraint is relevant and only in these conditions is the
state constraint meaningful. The satisfaction condition specifies whether the state constraint has been
violated. A relevant, satisfied state constraint corresponds to an aspect of the correct solution. A relevant,
unsatisfied state constraint indicates a flaw in the solution. Learner modeling is tracked by considering
what constraints are followed as learners solve problems. Successful constraint-based tutors include
Structured Query Language (SQL) tutor, Knowledge-based Entity Relationship Modeling Intelligent
Tutor (KERMIT), and Addison-Wesley’s Database Place (Mitrovic, Martin & Suraweera, 2007).

Knowledge Space Models
Knowledge space modeling underlies the Assessment and Learning in Knowledge Spaces (ALEKS)
mathematics tutor (Doignon & Falmagne, 1999; Hu et al., 2012). The domain model of knowledge space

iii

Design Recommendations for Intelligent Tutoring Systems - Volume 1: Learner Modeling
theory is a large number of possible knowledge states on a topic, whereas the learner model is a record of
which of the knowledge states are mastered, essentially a fine-grained overlay model. A learner’s
competence is reflected in the types of problems that the learner is capable of solving (among 250‒350
problems), given the profile of knowledge states mastered among millions of possible states. Bayesian
statistics are used to select the next problem to work on that is sensitive to the learner’s competence by
filling in deficits and correcting misconceptions. If a learner solves the next problem correctly, then each
knowledge state containing that problem incrementally increases in probability; if the learner answers
incorrectly, then the knowledge states are decreased in probability. Categories of skills are represented in
a pie chart that reflects the competence of the learner.

Expectation and Misconception Tailored Dialogue
This type of learner modeling is typical for ITSs that help learners learn by holding a conversation in
natural language, such as AutoTutor or Why-Atlas (Graesser et al., 2012; VanLehn et al., 2007). An
answer to a question is a set sentence-like expectation (good answer), but the tutor also anticipates the
learner articulating misconceptions (errors). An expectation or misconception is scored as being
expressed by a learner if the learner articulates it in natural language with a high enough semantic match.
Semantic matches can be assessed by a number of methods in computational linguistics, such as content
word overlap, latent semantic analysis, regular expressions, semantic entailment, or Bayesian statistics
(Cai et al., 2011; Graesser et al., 2007; Rus et al., 2009; VanLehn et al., 2007). When the total set of
problems is considered, there is a universal set of expectations (called principles or facets) and
misconceptions that are relevant to the various problems. These principles can be tracked over problems
and guide the selection of the next problem to work on.

Motivations for Intelligent Tutoring System Standards
An emphasis on self-regulated learning has highlighted a requirement for point-of-need training in
environments where human tutors are either unavailable or impractical. ITSs have been shown to be as
effective as expert human tutors (VanLehn, 2011) in one-to-one tutoring in well-defined domains
(e.g., mathematics or physics) and significantly better than traditional classroom training environments.
ITSs have demonstrated significant promise, but fifty years of research have been unsuccessful in making
ITSs ubiquitous in military training or the tool of choice in our educational system. Why?
The availability and use of ITSs have been constrained by their high development costs, their limited
reuse, a lack of standards, and their inadequate adaptability to the needs of learners (Picard, 2006). Their
application to military domains is further hampered by the complex and often ill-defined environments in
which our military operates today. ITSs are often built as domain-specific, unique, one-of-a-kind, largely
domain-dependent solutions focused on a single pedagogical strategy (e.g., model tracing or constraintbased approaches) when complex learning domains may require novel or hybrid approaches. Therefore, a
modular ITS framework and standards are needed to enhance reuse, support authoring, optimize
instructional strategies, and lower the cost and skillset needed for users to adopt ITS solutions for training
and education. It was out of this need that the idea for GIFT arose.
GIFT has three primary functions: authoring, instructional management, and analysis. First, it is a
framework for authoring new ITS components, methods, strategies, and whole tutoring systems. Second,
GIFT is an instructional manager that integrates selected tutoring principals and strategies for use in ITSs.
Finally, GIFT is an experimental testbed to analyze the effectiveness and impact of ITS components,
tools, and methods. GIFT is based on a learner-centric approach with the goal of improving linkages in
the adaptive tutoring learning effect chain (Figure P-1).

iv

Design Recommendations for Intelligent Tutoring Systems - Volume 1: Learner Modeling

Figure P-1. Adaptive Tutoring Learning Effect Chain (Sottilare, 2012)

A deeper understanding of the learner’s behaviors, traits, and preferences (learner data) collected through
performance, physiological and behavioral sensors, and surveys will allow for more accurate evaluation
of the learner’s states (e.g., engagement level, confusion, frustration), which will result in a better and
more persistent model of the learner. To enhance the adaptability of the ITS, methods are needed to
accurately classify learner states (e.g., cognitive, affective, psychomotor, social) and select optimal
instructional strategies given the learner’s existing states. A more comprehensive learner model will allow
the ITS to adapt more appropriately to address the learner’s needs by changing the instructional strategy
(e.g., content, flow, or feedback). An instructional strategy that is better aligned to the learner’s needs is
more likely to positively influence their learning gains. It is with the goal of optimized learning gains in
mind that the design principles for GIFT were formulated.

GIFT Design Principles
The methodology for developing a modular, computer-based tutoring framework for training and
education considered major design goals, anticipated uses, and applications. The design process also
looked at enhancing one-to-one (individual) and one-to-many (collective or team) tutoring experiences
beyond the state of practice for ITSs today. A significant focus of the GIFT design was on domaindependent elements in the domain module. This was done to allow large-scale reuse of the remaining
GIFT modules across different training domains and thereby reduce the development costs for ITSs.
One design principle adopted in GIFT is that each module should be capable of gathering information
from other modules according to the design specification. Designing to this principle resulted in standard
message sets and message transmission rules (i.e., request-driven, event-driven, or periodic
transmissions). For instance, the pedagogical module is capable of receiving information from the learner
module to develop courses of action for future instructional content to be displayed, manage flow and
challenge level, and select appropriate feedback. Changes to the learner’s state (e.g., engagement,
motivation, or affect) trigger messages to the pedagogical module, which then recommends general
courses of action (e.g., ask a question or prompt the learner for more information) to the domain module,
which provides a domain-specific intervention (e.g., what is the next step?).
Another design principle adopted within GIFT is the separation of content from the executable code (Patil
& Abraham, 2010). Data and data structures are placed within models and libraries, while software
processes are programmed into interoperable modules. Efficiency and effectiveness goals (e.g.,
accelerated learning and enhanced retention) were considered to address the time available for military
training and the renewed emphasis on self-regulated learning. An outgrowth of this emphasis on
efficiency and effectiveness led Dr. Sottilare to seek external collaboration and guidance. In 2012, U.S.
Army Research Laboratory (ARL) with the University of Memphis developed advisory boards of senior
tutoring system scientists from academia and government to influence the GIFT design goals moving
forward. An advisory board for learner modeling was completed in September 2012, and future boards
are planned for instructional strategy design, authoring and expert modeling, learning effect evaluations,
and domain modeling.
v

Design Recommendations for Intelligent Tutoring Systems - Volume 1: Learner Modeling

Design Goals and Anticipated Uses
GIFT may be used as any of the following:
1. An architectural framework with modular, interchangeable elements and defined relationships
2. A set of specifications to guide ITS development
3. A set of exemplars instantiating GIFT to support authoring and ease-of-use
4. A technical platform or testbed for guiding the development of concrete systems
These use cases have been distilled down into the three primary functional areas, or constructs:
authoring, instructional management, and analysis. Discussed below are the purposes, associated design
goals, and anticipated uses for each of the GIFT constructs.
GIFT Authoring Construct
The purpose of the GIFT authoring construct is to provide technology (tools and methods) to make it
affordable and easier to build ITSs and ITS components. Toward this end, a set of extensible markup
language (XML) configuration tools continues to be developed to allow for data-driven changes to the
design and implementation of GIFT-generated ITSs. The design goals for the GIFT authoring construct
have been adapted from Murray (1999, 2003) and Sottilare & Gilbert (2011). The GIFT authoring design
goals are as follow:


Decrease the effort (time, cost, and/or other resources) for authoring and analyzing ITSs by
automating authoring processes, developing authoring tools and methods, and developing
standards to promote reuse.



Decrease the skill threshold by tailoring tools for specific disciplines (e.g., instructional designers,
training developers, and trainers) to author, analyze, and employ ITS technologies.



Provide tools to aid designers/authors/trainers/researchers in organizing their knowledge.



Support (structure, recommend, or enforce) good design principles in pedagogy through user
interfaces, and other interactions.



Enable rapid prototyping of ITSs to allow for rapid design/evaluation cycles of prototype
capabilities.



Employ standards to support rapid integration of external training/tutoring environments (e.g.,
simulators, serious games, slide presentations, transmedia narratives, and other interactive
multimedia).



Develop/exploit common tools and user interfaces to adapt ITS design through data-driven
means.



Promote reuse through domain-independent modules and data structures.



Leverage open-source solutions to reduce ITS development and sustainment costs.

vi

Design Recommendations for Intelligent Tutoring Systems - Volume 1: Learner Modeling


Develop interfaces/gateways to widely used commercial and academics tools (e.g., games,
sensors, toolkits, virtual humans).

As a user-centric architecture, anticipated uses for GIFT authoring tools are driven largely by the
anticipated users, which include learners, domain experts, instructional system designers, training and
tutoring system developers, trainers and teachers, and researchers. In addition to user models and
graphical user interfaces, GIFT authoring tools include domain-specific knowledge configuration tools,
instructional strategy development tools, and a compiler to generate executable ITSs from GIFT
components in a variety of formats (e.g., PC, Android, and IPad).
Within GIFT, domain-specific knowledge configuration tools permit authoring of new knowledge
elements or reusing existing (stored) knowledge elements. Domain knowledge elements include learning
objectives, media, task descriptions, task conditions, standards and measures of success, common
misconceptions, feedback library, and a question library, which are informed by instructional system
design principles that, in turn inform concept maps for lessons and whole courses. The task descriptions,
task conditions, standards and measures of success, and common misconceptions may be informed by an
expert or ideal learner model derived through a task analysis of the behaviors of a highly skilled user.
ARL is investigating techniques to automate this expert model development process to reduce the time
and cost of developing ITSs. In addition to feedback and questions, supplementary tools are anticipated to
author explanations, summaries, examples, analogies, hints, and prompts in support of GIFT’s
instructional management construct.
GIFT Instructional Management Construct
The purpose of the GIFT instructional management construct is to integrate pedagogical best practices in
GIFT-generated ITSs. The modularity of GIFT will also allow GIFT users to extract pedagogical models
for use in tutoring/training systems that are not GIFT-generated. GIFT users may also integrate
pedagogical models, instructional strategies, or instructional tactics from other tutoring systems into
GIFT. The design goals for the GIFT instructional management construct are the following:


Support ITS instruction for individuals and small teams in local and geographically distributed
training environments (e.g., mobile training), and in both well-defined and ill-defined learning
domains.



Provide for comprehensive learner models that incorporate learner states, traits, demographics,
and historical data (e.g., performance) to inform ITS decisions to adapt training/tutoring.



Support low-cost, unobtrusive (passive) methods to sense learner behaviors and physiological
measures and use these data along with instructional context to inform models to classify (in near
real time) the learner’s states (e.g., cognitive and affective).



Support both macro-adaptive strategies (adaptation based on pre-training learner traits) and
micro-adaptive instructional strategies and tactics (adaptation based learner states and state
changes during training).



Support the consideration of individual differences where they have empirically been documented
to be significant influencers of learning outcomes (e.g., knowledge or skill acquisition, retention,
and performance).

vii

Design Recommendations for Intelligent Tutoring Systems - Volume 1: Learner Modeling


Support adaptation (e.g., pace, flow, and challenge level) of the instruction based the domain and
learning class (e.g., cognitive learning, affective learning, psychomotor learning, social learning).



Model appropriate instructional strategies and tactics of expert human tutors to develop a
comprehensive pedagogical model.

To support the development of optimized instructional strategies and tactics, GIFT is heavily grounded in
learning theory, tutoring theory, and motivational theory. Learning theory applied in GIFT includes
cognitive learning (Anderson & Krathwohl, 2001), affective learning (Krathwohl, Bloom, and Masia,
1964; Goleman, 1995), psychomotor learning (Simpson, 1972), and social learning (Sottilare, Holden,
Brawner, and Goldberg, 2011; Soller, 2001). Aligning with our goal to model expert human tutors, GIFT
considers the INSPIRE model of tutoring success (Lepper, Drake, and O’Donnell-Johnson, 1997) and the
tutoring process defined by Person, Kreuz, Zwaan, and Graesser (1995) in the development of GIFT
instructional strategies and tactics.
INSPIRE is an acronym that highlights the seven critical characteristics of successful tutors: Intelligent,
Nurturant, Socratic, Progressive, Indirect, Reflective, and Encouraging. Graesser & Person’s (1994)
tutoring process includes a tutor-learner interchange where the tutor asks a question, the learner answers
the question, the tutor gives feedback on the answer, then the tutor and learner collaboratively improve
the quality of (or embellish) the answer. Finally, the tutor evaluates learner’s understanding of the answer.
As a learner-centric architecture, anticipated uses for GIFT instructional management capabilities include
both automated instruction and blended instruction, where human tutors/teachers/trainers use GIFT to
support their curriculum objectives. If its design goals are realized, it is anticipated that GIFT will be
widely used beyond military training contexts as GIFT users expand the number and type of learning
domains and resulting ITS generated using GIFT.
GIFT Analysis Construct
The purpose of the GIFT analysis construct is to allow ITS researchers to experimentally assess and
evaluate ITS technologies (ITS components, tools, and methods). The design goals for the GIFT analysis
construct are the following:


Support the conduct of formative assessments to improve learning



Support summative evaluations to gauge the effect of technologies on learning



Support assessment of ITS processes to understand how learning is progressing throughout the
tutoring process



Support evaluation of resulting learning versus stated learning objectives



Provide diagnostics to identify areas for improvement within ITS processes



Support the ability to comparatively evaluate ITS technologies against traditional tutoring or
classroom teaching methods



Develop a testbed methodology to support assessments and evaluations (Figure P-2)

viii

Design Recommendations for Intelligent Tutoring Systems - Volume 1: Learner Modeling

Figure P-2. GIFT Analysis Testbed Methodology

Figure P-2 illustrates an analysis testbed methodology being implemented in GIFT. This methodology
was derived from Hanks, Pollack, and Cohen (1993) to allow manipulation of the learner model,
instructional strategies, and domain-specific knowledge within GIFT, and support analysis of artificiallyintelligent agents that influence the adaptive tutoring learning effect chain. In developing their testbed
methodology, Hanks et al. reviewed four testbed implementations (Tileworld, the Michigan Intelligent
Coordination Experiment [MICE], the Phoenix testbed, and Truckworld) for evaluating the performance
of artificially intelligent agents. Although agents have changed substantially in complexity during the past
20‒25 years, the methods to evaluate their performance have remained markedly similar.
The authors designed the GIFT analysis testbed based upon Cohen’s assertion (Hanks et al., 1993) that
testbeds have three critical roles related to the three phases of research. During the exploratory phase,
agent behaviors need to be observed and classified in broad categories. This can be performed in an
experimental environment. During the confirmatory phase, the testbed is needed to allow more strict
characterizations of agent behavior to test specific hypotheses and compare methodologies. Finally, in
order to generalize results, measurement and replication of conditions must be possible. Similarly, the
GIFT analysis methodology (Figure P-2) enables the comparison/contrast of ITS elements and assessment
of their effect on learning outcomes (e.g., knowledge acquisition, skill acquisition, and retention).

How to Use This Book
This book is organized into four sections:
I.
II.
III.
IV.

Fundamentals of Learner Modeling
Current Learner Modeling Tools and Methods
Emerging Learner Modeling Concepts
Future Learner Modeling Concepts

The Fundamentals of Learner Modeling section provides an overview of learner modeling terms and
concepts along with discussion topics, and a review of the learner modeling literature. The Current
Learner Modeling Tools and Methods section reviews current learner modeling tools and methods and
ix

Design Recommendations for Intelligent Tutoring Systems - Volume 1: Learner Modeling
provides design recommendations for GIFT and adaptive ITSs. The Emerging Learner Modeling
Concepts section analyzes emerging learner modeling concepts and discusses their potential impact on
design recommendations for GIFT and adaptive ITS. Finally, the Future Learner Modeling Concepts
section projects how ITSs might be applied in the future and provides design recommendations to realize
innovative capabilities in GIFT and adaptive ITSs.
Chapter authors in each section were carefully selected for participation in this project based on their
expertise in the field as ITS scientists, developers, and practitioners. Design Recommendations for
Intelligent Tutoring Systems: Learner Modeling (Volume I) is intended to be a design resource as well as
community research resource that can be of significant benefit as the following:


An educational resource for developing ITS scientists: Section I provides a wealth of
information about ITS concepts and design, and presents an in-depth review of the learner
modeling literature.



A roadmap for ITS research opportunities: Sections II, III, and IV present current, emerging,
and future concepts about learner modeling. This sampling of authors’ perspectives is based on
hundreds of cumulative years of experience in the ITS research domain and identifies significant
gaps in current and emerging ITS technology (tools and methods). Each of these gaps points to
yet unanswered research questions.



A roadmap to the development and application of GIFT: As noted previously, GIFT is an opensource, publically available ITS architecture that is intended to make it easy to author ITSs;
reduce the cost of ITS development by promoting reuse; automatically manage instruction based
on best pedagogical practices; and allow scientists to compare and contrast evolving ITS
capabilities to determine future best practices. As this book outlines issues and challenges
associated with learner modeling, it also provides guidelines on how GIFT might be designed to
address identified capability gaps. Future volumes of the “Design Recommendations for
Intelligent Tutoring Systems” book series will provide insight to other ITS design domains
including instructional strategy and tactics design, authoring and expert modeling, domain
modeling, learning effect assessment, and team tutoring design. We encourage readers to become
members of the GIFT community to build on its existing capabilities and support its future
capabilities with us. More information on GIFT can be found by registering at
www.GIFTtutoring.org. Registration provides access to GIFT source code, documentation, and
related publications.

References
Aleven, V., McLaren, B., Roll, I. & Koedinger, K. (2006). Toward meta-cognitive tutoring: A model of help seeking
with a cognitive tutor. International Journal of Artificial Intelligence in Education, 16, 101-128.
Anderson, J. R., Corbett, A. T., Koedinger, K. R. & Pelletier, R. (1995). Cognitive tutors: Lessons learned. Journal
of the Learning Sciences, 4, 167-207.
Anderson, L. W. & Krathwohl, D. R. (Eds.). (2001). A taxonomy for learning, teaching and assessing: A revision of
Bloom’s Taxonomy of Educational Objectives: Complete edition. New York : Longman.
Baker, R.S., D’Mello, S.K., Rodrigo, M.T. & Graesser, A.C. (2010). Better to be frustrated than bored: The
incidence, persistence, and impact of learners’ cognitive-affective states during interactions with three
different computer-based learning environments. International Journal of Human-Computer Studies, 68,
223-241.

x

Design Recommendations for Intelligent Tutoring Systems - Volume 1: Learner Modeling
Cai, Z., Graesser, A.C., Forsyth, C., Burkett, C., Millis, K., Wallace, P., Halpern, D. & Butler, H. (2011). Trialog in
ARIES: User input assessment in an intelligent tutoring system. In W. Chen & S. Li (Eds.), Proceedings of
the 3rd IEEE International Conference on Intelligent Computing and Intelligent Systems (pp.429-433).
Guangzhou: IEEE Press.
D’Mello, S. & Graesser, A.C. (2010). Multimodal semi-automated affect detection from conversational cues, gross
body language, and facial features. User Modeling and User-adapted Interaction, 20, 147-187.
D’Mello, S. K., Graesser, A. C. & King, B. (2010). Toward spoken human-computer tutorial dialogues. Human
Computer Interaction, 25, 289-323.
Doignon, J.P. & Falmagne, J. C. (1999). Knowledge Spaces. Berlin, Germany: Springer.
Elson-Cook, M. (1993). Student modeling in intelligent tutoring systems. Artificial Intelligence Review, 7, 227-240.
Gagne, R. M. (1985). The conditions of learning and theory of instruction (4th ed.). New York: Holt, Rinehart &
Winston.
Goldberg, B.S., Sottilare, R.A., Brawner, K.W. & Holden, H.K. (2011). Predicting Learner Engagement during
Well-Defined and Ill-Defined Computer-Based Intercultural Interactions. In S. D’Mello, A. Graesser, , B.
Schuller & J.-C. Martin (Eds.), Proceedings of the 4th International Conference on Affective Computing
and Intelligent Interaction (ACII 2011) (Part 1: LNCS 6974) (pp. 538-547). Berlin Heidelberg: Springer.
Graesser, A.C., Conley, M. & Olney, A. (2012). Intelligent tutoring systems. In K.R. Harris, S. Graham & T. Urdan
(Eds.), APA Educational Psychology Handbook: Vol. 3. Applications to Learning and Teaching (pp. 451473). Washington, DC: American Psychological Association.
Graesser, A. C., D’Mello, S. K., Hu. X., Cai, Z., Olney, A. & Morgan, B. (2012). AutoTutor. In P. McCarthy & C.
Boonthum-Denecke (Eds.), Applied natural language processing: Identification, investigation, and
resolution (pp. 169-187). Hershey, PA: IGI Global.
Graesser, A. C., Penumatsa, P., Ventura, M., Cai, Z. & Hu, X. (2007). Using LSA in AutoTutor: Learning through
mixed initiative dialogue in natural language. In T. Landauer, D. McNamara, S. Dennis & W. Kintsch
(Eds.), Handbook of latent semantic analysis (pp. 243–262). Mahwah, NJ: Erlbaum.
Graesser, A. C. & Person, N. K. (1994). Question asking during tutoring. American Educational Research Journal,
31, 104–137.
Hanks, S., Pollack, M.E. & Cohen, P.R. (1993). Benchmarks, Test Beds, Controlled Experimentation, and the
Design of Agent Architectures. AI Magazine, 14 (4), 17-42.
Heffernan,N.T., Koedinger, K.R. & Razzaq, L.(2008) Expanding the model-tracing architecture: A 3rd generation
intelligent tutor for Algebra symbolization. The International Journal of Artificial Intelligence in
Education, 18(2). 153-178.
Hu, X., Craig, S. D., Bargagliotti A. E., Graesser, A. C., Okwumabua, T., Anderson, C., Cheney, K. R. &
Sterbinsky, A. (2012). The effects of a traditional and technology-based after-school program on 6th grade
students’ mathematics skills. Journal of Computers in Mathematics and Science Teaching, 31, 17-38.
Johnson, L. W. & Valente, A. (2008). Tactical language and culture training systems: Using artificial intelligence to
teach foreign languages and cultures. In M. Goker & K. Haigh (Eds.), Proceedings of the Twentieth
Conference on Innovative Applications of Artificial Intelligence (pp. 1632-1639). Menlo Park, CA: AAAI
Press.
Krathwohl, D.R., Bloom, B.S. & Masia, B.B. (1964). Taxonomy of Educational Objectives: Handbook II: Affective
Domain. New York: David McKay Co.
Lepper, M. R., Drake, M. & O’Donnell-Johnson, T. M. (1997). Scaffolding techniques of expert human tutors. In K.
Hogan & M. Pressley (Eds), Scaffolding learner learning: Instructional approaches and issues (pp. 108144). New York: Brookline Books.
Litman, D. (2013). Speech and language processing for adaptive training. In P. Durlach & A. Lesgold (Eds.),
Adaptive technologies for training and education. Cambridge, MA: Cambridge University Press.

xi

Design Recommendations for Intelligent Tutoring Systems - Volume 1: Learner Modeling
Mitrovic, A., Martin, B. & Suraweera, P. (2007). Intelligent tutors for all: The constraint-based approach. IEEE
Intelligent Systems, 22(4), 38-45.
Murray, T. (1999). Authoring intelligent tutoring systems: An analysis of the state of the art. International Journal
of Artificial Intelligence in Education, 10(1), 98–129.
Murray, T. (2003). An Overview of Intelligent Tutoring System Authoring Tools: Updated analysis of the state of
the art. In Murray, T.; Blessing, S.; Ainsworth, S. (Eds.), Authoring tools for advanced technology learning
environments (pp. 491-545). Berlin: Springer..
Nkambou, R., Mizoguchi, R. & Bourdeau, J. (2010). Advances in intelligent tutoring systems. Heidelberg: Springer.
Ohlsson, S. (1992). Constraint-based student modeling. Journal of Artificial Intelligence in Education 3(4), 429-447.
O’Neil, H. F. & Perez, R. (Eds.). (2003). Technology applications in education: A learning view. Hillsdale, NJ.:
Erlbaum.
Patil, A. S. & Abraham, A. (2010). Intelligent and Interactive Web-Based Tutoring System in Engineering
Education: Reviews, Perspectives and Development. In F. Xhafa, S. Caballe, A. Abraham, T. Daradoumis
& A. Juan Perez (Eds.), Computational Intelligence for Technology Enhanced Learning. Studies in
Computational Intelligence (Vol 273, pp. 79-97). Berlin: Springer-Verlag.
Person, N. K., Kreuz, R. J., Zwaan, R. A. & Graesser, A. C. (1995). Pragmatics and pedagogy: Conversational rules
and politeness strategies may inhibit effective tutoring. Cognition and Instruction, 13(2), 161–188.
Picard, R. (2006). Building an Affective Learning Companion. Keynote address at the 8th International Conference
on Intelligent Tutoring Systems, Jhongli, Taiwan. Retrieved from
http://www.its2006.org/ITS_keynote/ITS2006_01.pdf
Psotka, J. & Mutter, S.A. (1988). Intelligent Tutoring Systems: Lessons Learned. Hillsdale, NJ: Lawrence Erlbaum
Associates.
Ritter, S., Anderson, J. R., Koedinger, K. R. & Corbett, A. (2007) Cognitive Tutor: Applied research in mathematics
education. Psychonomic Bulletin & Review, 14, 249-255.
Rus, V. & Graesser, A.C. (Eds.) (2009). The Question Generation Shared Task and Evaluation Challenge. Retrieved
from http://www.questiongeneration.org/.
Rus, V., McCarthy, P.M., McNamara, D.S. & Graesser, A.C. (2009). Identification of sentence-to-sentence relations
using a text entailer. Research on Language and Computation, 7, 209-229.
Simpson, E. (1972). The classification of educational objectives in the psychomotor domain: The psychomotor
domain. Vol. 3. Washington, DC: Gryphon House.
Sleeman D. & J. S. Brown (Eds.) (1982). Intelligent Tutoring Systems. Orlando, Florida: Academic Press, Inc.
Soller, A. (2001). Supporting social interaction in an intelligent collaborative learning system. International Journal
of Artificial Intelligence in Education, 12(1), 40-62.
Sottilare, R. & Gilbert, S. (2011). Considerations for tutoring, cognitive modeling, authoring and interaction design
in serious games. Authoring Simulation and Game-based Intelligent Tutoring workshop at the Artificial
Intelligence in Education Conference (AIED) 2011, Auckland, New Zealand, June 2011.
Sottilare, R., Holden, H., Brawner, K. & Goldberg, B. (2011). Challenges and Emerging Concepts in the
Development of Adaptive, Computer-based Tutoring Systems for Team Training. Interservice/Industry
Training Systems & Education Conference, Orlando, Florida, December 2011.
Sottilare, R.A., Brawner, K.W., Goldberg, B.S. & Holden, H.K. (2012). The Generalized Intelligent Framework for
Tutoring (GIFT). Orlando, FL: U.S. Army Research Laboratory – Human Research & Engineering
Directorate (ARL-HRED).
Sottilare, R. (2012). Considerations in the development of an ontology for a Generalized Intelligent Framework for
Tutoring. International Defense & Homeland Security Simulation Workshop in Proceedings of the I3M
Conference. Vienna, Austria, September 2012.

xii

Design Recommendations for Intelligent Tutoring Systems - Volume 1: Learner Modeling
VanLehn, K. (2006) The behavior of tutoring systems. International Journal of Artificial Intelligence in Education.
16(3), 227-265.
VanLehn, K., Graesser, A. C., Jackson, G. T., Jordan, P., Olney, A. & Rose, C. P. (2007). When are tutorial
dialogues more effective than reading? Cognitive Science, 31, 3-62.
VanLehn, K. (2011). The relative effectiveness of human tutoring, intelligent tutoring systems and other tutoring
systems. Educational Psychologist, 46(4), 197-221.
Woolf, B.P. (2009). Building intelligent interactive tutors. Burlington, MA: Morgan Kaufmann Publishers.

xiii

Design Recommendations for Intelligent Tutoring Systems - Volume 1: Learner Modeling

xiv

Design Recommendations for Intelligent Tutoring Systems - Volume 1: Learner Modeling

SECTION I

FUNDAMENTALS OF
LEARNER
MODELING

A. Graesser, Ed.

1

Design Recommendations for Intelligent Tutoring Systems - Volume 1: Learner Modeling

2

Design Recommendations for Intelligent Tutoring Systems - Volume 1: Learner Modeling

CHAPTER 1 ‒ A Guide to Understanding Learner Models
Arthur Graesser
University of Memphis

Introduction
The core notion of a learner model is not complex intuitively. We simply need to record, represent, and
track characteristics of the learner before, during, and after learning. The mission is accomplished when
the learner model accommodates all of the variables that are ever considered in the history and future of
intelligent tutoring systems, with all variables being adequately represented for all systems. The
theoretical problem is that the set of variables and their representation is not a closed system, but rather
grows over time as a moving target. The practical problem is that it is expensive to identify, track, store,
update, and later retrieve the ever-growing universal set of variables. The mapping problem is that the
alignment between the theoretical variables and computer code is often vague, incomplete, or
incompatible. The computational problem is that complex interactions among learner variables create a
combinatorial explosion that is temporally insurmountable. In light of these multiple problems, there is
no alternative than to pursue sensible compromises.
The chapters in this section have identified the major challenges in developing a learner model for GIFT.
Simply put, there is no consensus in how the community of researchers is to handle the theoretical,
practical, mapping, and computational problems. They all offer hints toward a solution, which was their
charge. It is difficult to decide whether any of them have answered the challenge. Perhaps yes. Perhaps
no.
The chapter by Robson and Barr expresses the need to establish standards. They argue that the standards
should be pitched at the macro level rather than the micro level. That is, the community of researchers
should agree on the ingredients of major content objects (i.e., knowledge, skills, procedures) but leave it
to the individual learning environments to realize the dynamics of mastering these learning objects. This
would require an agreement among curriculum experts and system developers in converging on an ideal
grain size that differentiates macro and micro.
The chapter by Olney and Cade recommends that researchers take stock of the pedagogical strategies
offered by researchers and to identify the learning model variables that support such strategies. The
Army Research Lab conducted a systematic study to identify the strategies from hundreds of studies.
Olney and Cade identified the classes of learner model variables that would support these strategies. This
is a sensible approach to identifying a complete set of learner variables, including cognitive, social,
emotion, and motivation dimensions.
The chapter by Lesgold and Graesser raised the persistent problem of transfer. It is comparatively easy to
develop a system that efficiently trains learners on the knowledge and skills of a specific learning object,
but it is difficult to do so in a way that transfers to a new learning object with related knowledge and
skills. Specific is easy, but general is difficult. The authors emphasize that it is absolutely critical to
acquire the materials in a general way during training that transfers to a broad range of situations later on.
If not, the learning episode is destined to reside in a very narrow corner of the space to be mastered.
The chapter by Pavlik, Brawner, Olney, and Mitrovic provides a serious comprehensive review of the
learning models in ITS applications over the decades. They take stock of the learner models in a variety
of ITS frameworks, including step-based cognitive tutors, constraint based tutors, knowledge space
models, dialogue systems, and trait-based assessments. They point out the value of systems with
branching architectures and identify the alternative grain sizes of the branching, as well as the
3

Design Recommendations for Intelligent Tutoring Systems - Volume 1: Learner Modeling
consequences. They also urge the GIFT to build on the major progress that has been made on previous
ITS with general architectures and empirical validation.
The four chapters in this section together have suggested a number of factors that need to be considered as
GIFT takes on the challenge of learner modeling. The goal of this section introduction is to provide a
landscape of relevant dimensions rather than to offer concrete solutions.

Landscape of Variables for Learner Models
The number of learner variables is potentially large and unlimited but a small number of variables is
typically tracked in systems that scale up. For example, the variables tracked in educational systems
throughout the country rarely go beyond attendance and 1-3 high stakes tests per year. The collection of
fine grained measures and learning portfolios are boutique enterprises that interest researchers and
teachers of the future. Consequently, any convergence on an intermediate, manageable set of learner
variables will need to satisfy numerous political and practical constraints.
The number of learner variables being considered in ITS research has substantially increased over the
decades. There are many measures of learning gains that assess changes between pretests and posttests.
The efficiency of learning is measured in economic models that consider how much learning occurs per
unit time. The mastery of specific knowledge, skills, and strategies is assessed at ever increasing grain
sizes. The learners’ engagement and persistence can be tracked by recording time on task, reactions to
computer requests, non-invasive measures (e.g., eye movements, body posture), and physiological
measures. The emotions and motivation of the learner are tracked by algorithms that mine the log files of
the person-system interactions. These behavioral measures are arguably more valid than self-report data,
such as rating scales that are contaminated by the learners’ metacognitive folklore. Measures of
personality, leadership, and social responsiveness are also tracked by algorithms that have evolved from
the educational data mining community. Contemporary ITS applications routinely collect thousands of
measures during a time span of 2 to 20 hours. The grain size of ITS measurement is currently 3 orders of
magnitude beyond the data collected in school systems throughout the country.

Farming and Mining the Landscape of Variables
Researchers need to be selective when analyzing the rich log file data that tracks the learner models. The
most straightforward approach is to focus on those raw or composite measures that are anticipated
theoretically. This top-down approach is the perfect place to start and impresses most reviewers of
academic journals and funding agencies. Large data sets can be “farmed” by researchers in a manner that
systematically tests and revises theories in the face of empirical data. Data are selected and organized to
test major learning theories of the day, well established ITS applications (see Pavlik et al. chapter),
educational standards (such as the Common Core or certification on specialty topics), the learning
strategies documented by the ARL (see Olney and Cade chapter), an existing repository of learning
objects that are shared by the community (see Robson and Barr chapter), and transfer between learning
objects, tasks, and subject matters (see Lesgold and Graesser chapter).
Theories are unfortunately limited and frequently not confirmed. Consequently, there is a need for
bottom-up methods to discover new learner measures and patterns from the log files. During the last
decade, the field has experienced the evolution of the data mining revolution. New categories of learners
are revealed by clustering analyses on learners and on tasks, as well as the tracking of individual learner
data over time. Longitudinal research designs (which track individuals over a long period of time) are
preferred over cross sectional research designs. Sequences of events in the log files are diagnostic of
specific psychological attributes. Once these patterns are discovered, they can be tracked automatically
4

Design Recommendations for Intelligent Tutoring Systems - Volume 1: Learner Modeling
and tested further. This approach is expected to lead to the development of more sophisticated learning
theories that have a better chance of scaling up.

Representing Learner Models
ITS developers are known to disagree over the representation of knowledge, skills, and strategies in the
learner model. This is because the computational models are very different in the cognitive tutors,
constraint-based tutors, knowledge space tutors, dialogue-based tutors, scenario-based tutors, and other
classes of ITS architectures. The representation of production rules also may differ in ITS applications
that adopt production rules. The primitive elements, symbolic expressions, and quantitative parameters
differ among production rules systems because the subject matter applications have very different
constraints.
It is too early in the history of ITS development to force researchers to adopt a particular representation of
knowledge, skills, and strategies. The SCORM initiative never was able to achieve such a lofty objective,
even though there were some discussions to try. Researchers are deeply wedded to their pet
computational architectures and algorithms as they pursue cycle after cycle of model testing. With this
context in mind, GIFT developers may consider focusing on specific exemplar ITS applications that have
proven the test of time and empirical validation (see Pavlik et al. chapter). There are a limited set of
successful ITS applications so this approach would be a practical first step that is within reach. New ITS
applications can build upon these prototypical exemplar ITS applications. The learner models of these
exemplars could be expressed in a general formal notation in addition to the actual software residing in a
repository of concrete applications. The specific ITS developers would of course need to agree to
annotate and release the software. Once the collection of ITS exemplars is available, the time would be
ripe to identify a first-cut landscape of variables for learner modeling.
The complexity and variations in representations have left us with some significant barriers in scaling up
ITS. There are not enough trained personnel to build new applications on new subject matters because of
the idiosyncratic features of the ITS representations. An ideal author would have expertise in the subject
matter, cognitive science, information sciences, education, ITS pedagogy, human computer interaction,
and sometimes computational linguistics. Authoring tools are often created to minimize this barrier, but
there have never been sufficient efforts to build high quality authoring tools. There needs to be
systematic R&D on authoring tool development that is tested on personnel outside of the camp of the
original ITS developers. We need an applied empirical science of authoring tool development that has
analogues to research on writing or to design. To what extent are the learner model representations
developed with sufficient fidelity, scope, grain-size, and level of abstraction? How much training is
needed for new personnel to develop learner models for new applications? What is the time course and
costs of developing new learner models?

GIFT in the Short-Term Horizon
As mentioned earlier, the contributors to this section of the book offered different recommendations for
developing the learner model component of GIFT. The recommendations addressed challenges and
pressure points that need attention in the roadmap ahead. This final section enumerates some actions that
might be considered in the short-term horizon.
(1) A prototype has been developed that implements characteristics of GIFT, including the learner model.
A systematic analysis could be conducted on the learner model variables in order to assess the extent to
which they cover the variables present in the learner models of different classes of mainstream ITS
applications (e.g., cognitive tutors, constraint-based tutors, knowledge space tutors, dialogue-based tutors)
5

Design Recommendations for Intelligent Tutoring Systems - Volume 1: Learner Modeling
and the pedagogical strategies identified by the ADL. A repository of variables could be assembled with
definitions and links to alternative ITS applications.
(2) It is widely acknowledged that it is difficult to achieve successful transfer from one domain or subject
matter to that of another. The field would benefit from an analysis of training strategies, representation
specification, degree of abstraction, grain size, and other characteristics of learner models of ITS
applications that have achieved good transfer.
(3) Shareable learning objects (such as SCORM) have been a persistent dream of many communities that
want to scale up advanced learning environments. The field would benefit from a review of the successes
and failures of these attempts. This includes an analysis of the level of abstraction and types of
representations that are likely to be shareable and serve as standards.
(4) Authoring tools have nearly always been difficult to use for ITS as well as other learning
architectures. The field could benefit from a review of empirical research that has systematically
analyzed how new personnel use such tools as well as the quantity and quality of their products from the
standpoint of learner modeling in particular. New empirical studies are needed that are more systematic
than anecdotal.
(5) The time and costs of developing an ITS on a new subject matter has frequently been a focus of
questions with respect to scaling up the ITS enterprise. The field would benefit from an economic
analysis that helps answer these questions. It is important to segregate the initial up-front costs in
developing initial ITS applications, incremental costs in developing new ITS applications that piggyback
on existing systems, and scale-up costs after an existing system is ready to be used by thousands or
millions of students.
(6) Classes of ITS are ideally tailored to different types of learning, such as strategically guided
perception, memory for facts, execution of procedures, explanations of events within complex systems,
principle-based prediction/forecasting, and removal of chronic misconceptions in mental models. The
field would benefit from a typology and possibly a consensus on what learning environments are
appropriate for each type of learning.

6

Design Recommendations for Intelligent Tutoring Systems - Volume 1: Learner Modeling

CHAPTER 2 ‒ Lowering the Barrier to Adoption of Intelligent

Tutoring Systems through Standardization
Robby Robson1 and Avron Barr2
Eduworks Corporation; 2Aldo Ventures

1

Introduction
To have an impact, a learning system must be effective and must be used. In the case of ITSs, studies
have repeatedly and consistently shown significant learning gains (Dodds & Fletcher, 2004; Durlach &
Ray, 2011; Kulik & Kulik, 1991; VanLehn, 2011). Despite their demonstrated value and over thirty-year
history (Barr, Beard & Atkinson, 1975; Sleeman & Brown, 1982), the use of ITSs remains restricted to
research projects and a few commercial applications. There are success stories, but in general, such
systems are not being adopted at the rates that their effectiveness would justify. As stated by Blessing,
Gilbert, Ourada & Ritter (2009) in a paper on authoring model-tracing cognitive tutors, “ITSs, including
model-tracing tutors, have not been widely adopted in educational or other settings, such as corporate
training. Perhaps the most successful deployment of model-tracing tutors is Carnegie Learning’s
Cognitive Tutors for math, which are in use by over 1300 school districts and by hundreds of thousands
of students each year. After this notable success, however, most educational and training software is not
of the ITS variety.”
Multiple factors could be impeding adoption, but two factors in particular stand out. The first is that ITSs
are designed to be standalone systems that do not communicate or interoperate with other systems used to
support learning, education, and training. The second is the sheer complexity of ITSs and the concomitant
effort it takes to develop them (Murray, 2003).
With this as motivation, this chapter explores how standardization might help ITSs fit into learning
ecosystems and simplify their design. Here we suggest focusing on standards for exchanging learner
information among systems, not on standards for internal components. We also point out that the
requirement to exchange learner information emphasizes the problem of determining which adaptations
are responsible for the positive learning effect sizes observed when using ITSs. We believe that the
suggested approach will enrich learner models, encourage developers to separate their innovative and
proprietary adaptation engines from the portions of ITSs that interoperate with other learning systems,
and ultimately, transform ITS architecture in ways that will make them easier to implement and adopt.

Standards
As Christensen & Raynor (2003) point out, interoperability and standardization enable competition and
the growth of supply chains. In learning technology, for example, the emergence of learning management
systems (LMSs) in the 1990s disrupted the print-based supply chain from authors to publishers to schools
to learners. Standards such as IMS Content Packaging (IMS Global Learning Consortium, 2004), AICC
Computer Managed Instruction (AICC, 2004), Sharable Content Object Reference Model (SCORM) 1.2
(Dodds, 2001), SCORM 2004 (ADL, 2006), and IMS Common Cartridge (IMS Global Learning
Consortium, 2011) reestablished much of the same chain by allowing courseware to be produced
independently and used by any compliant LMS. Arguably, the eLearning industry, which is a multibillion dollar industry today (Adkins, 2011; Bersin, 2012; Global Industry Analysts, 2013), would not
exist without these standards, and it is reasonable to assume that some standardization is needed to spur
the adoption of ITSs.
7

Design Recommendations for Intelligent Tutoring Systems - Volume 1: Learner Modeling
At the same time, many proposed learning technology standards have failed to be completed or failed to
achieve significant adoption. These proposals include standards for learner information and learner
models, architectures, repository interoperability, intelligent agent communication, competency
definitions, student identifiers, and many others. For example, the IEEE Learning Technology Standards
Committee web site from December 2000 (IEEE, 2000a) shows 17 standards projects, almost all of which
are still relevant today (including a standard for learner models). Less than half of these were completed
in any form, and almost none have achieved any significant adoption. In most cases, failed efforts were
driven more by an example and a vision rather than by a market/adopter pain point and real products.
These failed standards, as described in Robson (2006), were “innovation and research-driven standards”
rather than “market-driven standards.” If standards are to be developed to spur the adoption of intelligent
tutoring systems, they must be defined by market needs, or else they will have little effect.

Market Needs
What are those market needs? If we accept that ITSs are effective, the immediate market needs of
potential customers are to (1) implement them with as little integration effort and work disruption as
possible; and (2) obtain the data that are required for learning management and talent management and
that can be used to demonstrate pedagogical impact and monitor costs. Customers may tolerate some
inconvenience to get a more effective learning solution, but they are probably not willing to reconfigure
their entire learning infrastructure, retrain all of their users, radically alter their established workflows, or
give up on tracking grades and course completion status.
Unfortunately, ITSs make little or no attempt to exchange even basic results data with other systems. This
isolationism is typical of new learning technologies. Early LMS and assessment engine products were the
same, and so are massive open online courses (MOOCS) and the Khan Academy. The following
exchange from the Google Khan Academy Developers Group (Azevedo & Ojeda, 11/19/12) typifies the
reluctance of new product developers to address interoperability and the frustration of early adopters who
typically run their daily operations via institutional learning management systems. (Typos and spelling
corrected).
DA: Hello developers. Is Khan Academy / Khan Exercises SCORM Compliant?
MO: hi D. Sorry to say, we do not. Is there some definite advantage to supporting it other than
this graphic from scorm.com?” (Graphic shows reduced costs from using SCORM)
DA: Hello and thanks for the quick response. I find one big use for SCORM. If a student uses
multiple learning platforms it would be nice for grades and progress to be shared across the
platforms. One example would be: I’m attending a class in Khan Academy, like Algebra II, but I
find another site/course and I want to make Algebra III in that new site/course. If I could
export/import my certified data across platforms that would be very flexible for students. In the
long run people will like that the time spent on Khan or another site/course is certified and
flexible, like in real universities there is equivalence in subjects and grades/progress.1
The immediate market needs of potential customers are to track results and hold down implementation
costs. These are conditions for diffusion of the technology, but once diffusion starts, other requirements
will appear. For example, since ITSs provide individualized learning, it is likely that students will
frequently switch among different systems. If the one system has gathered data about a learner’s cognitive
or affective characteristics, other systems can make use of it. Existing standards such as IMS GLC
1

We note that one of the motivations for moving from SCORM to the Experience API is that SCORM does not address learners
working on multiple platforms.

8

Design Recommendations for Intelligent Tutoring Systems - Volume 1: Learner Modeling
Learner Information Package (IMS Global Learning Consortium, 2005) and the Europass (Cedefop,
2012) can be used for transcript data, curriculum vitae (CV) data, and high level competencies (e.g.,
language skills), but new standards will be needed for expressing and comparing learner information.
In addition, adoption will bring renewed demands for
avoiding lock-in. Even if the same content cannot be
plugged and played in multiple intelligent tutoring
systems, customers will want to leverage content
they have already acquired and will want to
outsource ITS development to multiple sources. To
achieve this requirement, there must be some
reasonable separation between the “system”
software, “content” such as text and multimedia
presented to the learner, and “interfaces,” including
the user interface and those that exchange data with
other learning systems. This is diagrammatically
shown in Figure 2-1.

Standards for Learner Information

Figure 2-1. System, Content and Interfaces

The first requirement for adoption is for ITSs to “play well” in current learning environments. This can be
done by requiring conformance to SCORM and the Experience API or Tin Can API (Advanced
Distributed Learning, 2013) or by adopting the IMS approach based on Learning Tools Interoperability
(IMS Global Learning Consortium, 2012). The important point is that ITS developers should not ignore
these standards if they want their systems to be adopted.
Requirements to exchange even very coarse data such as completion status and quiz results will naturally
lead developers to reexamine their system architectures and, hopefully, lead to a separation of
components such as that illustrated in Figure 2-1. For example, the Experience API, planned for the next
generation of SCORM, uses an “actor – verb – activity” paradigm and is designed for compatibility with
semantic inference engines (Poltrack, Hruska, Johnson & Haag, 2012). Using this application
programming interface (API), a score on a quiz can be reported as a series of statements such as “Student
completed quiz” and “Student scored 95.” An inference engine embedded in an LMS or other learning
system might additionally know that “Quiz assesses Quadratic Formula” and conclude that “Student
demonstrates competency in Quadratic Formula.” The requirement to generate such triples and support
the API will suggest using a similar structure to store and exchange other data, including learner
information. This may or may not be the optimal design choice for a particular ITS, but experience shows
that developers of new systems often use standards as guidelines for functionality and design (Devedzic,
Jovanovic & Gasevic, 2007).
Independent of how it is represented, the key question for standardization is what information should be
exchanged. In other words, what should an ITS be telling other systems, including other ITSs, other
enterprise learning systems, and other applications used by instructors, students, managers, and
researchers? Since ITSs are valuable because of their positive effects on learning, this information should
consist of the data responsible for attaining this effect. In other words, the question of what information
should be exchanged is the question of what student data are required to achieve near optimal adaptation.
This is a special case of the more general question, posed by Durlach (2012) and Ray Perez (personal
communication), of what functionality in ITSs has the most effect on learning outcomes and how much of
this functionality is needed in practice. We do not know the answer, but we can nonetheless make some
reasonable conjectures concerning the types of learner information might be included:
9

Design Recommendations for Intelligent Tutoring Systems - Volume 1: Learner Modeling


Educational records and high level competencies such as language skills and flight certifications.
Standards exist for these data. It is not clear what inferences an intelligent tutor can make directly
from them, but since human tutors find them valuable, there is an argument for including them.



Competencies (including Skills, Knowledge, Abilities, Outcomes, Objectives) and level of
competence. These have also been standardized and represent data that are crucial for talent
management, job placement, and many other applications. They are also used for AICC/SCORMtype sequencing, and at a fine level of granularity, can be equated with domain topics.



Data in affective, motivational, and social dimensions. Cognitive models are more common and
better understood, but many adaptations rely on data in these dimensions (Dimitrova, 2009). For
example, tutors may observe how students react to different types of stimuli and discover beliefs
and attitudes that other systems could use to select instructional content and strategies.



Goals, including learning goals and mission/task goals. These goals are also data that a human
tutor would want to know and that might be important for adaptation.



Physical adaptations, such as location, device capabilities, ambient light, and accessibility data.
Accessibility data have been standardized, and these other data can clearly be used in many ways.

As an historical note, we believe that the task of standardizing learner data was first taken up by the IEEE
Learning Technology Standards Committee. In 2000, the scope statement for the Learner Model working
group (IEEE, 2000b) read: “This standard will specify the syntax and semantics of a ‘Learner Model,’
which will characterize a learner (student or knowledge worker) and his or her knowledge/abilities. This
will include elements such as knowledge (from coarse- to fine-grained), skills, abilities, learning styles,
records, and personal information. This standard will allow these elements to be represented in multiple
levels of granularity, from a coarse overview, down to the smallest conceivable sub-element. The
standard will allow different views of the Learner Model (learner, teacher, parent, school, employer, etc.)
and will substantially address issues of privacy and security”
Its purpose consisted of five items:
1. To enable learners (students or knowledge workers) of any age, background, location, means, or
school/work situation to create and build a personal Learner Model, based on a national
standard, which they can utilize throughout their education and work life.
2. To enable courseware developers to develop materials that will provide more personalized and
effective instruction.
3. To provide educational researchers with a standardized and growing source of data.
4. To provide a foundation for the development of additional educational standards, and to do so
from a student-centered learning focus.
5. To provide architectural guidance to education system designers.
This project was known as “Personal and Private Information” (PAPI) and never turned into a standard
for reasons beyond the scope of this chapter. It did, however, have some important attributes. It
considered highly granular information, a consideration abandoned by the IMS Learner Information
Package and Europass standards. It had personalized learning and architectural guidance as important use
cases. Architectural guidance is important because the complexity of ITSs is a barrier to adoption.
Interchange standards do not dictate how data are stored or processed internally, but they tend to
10

Design Recommendations for Intelligent Tutoring Systems - Volume 1: Learner Modeling
influence how systems are designed. Devedzic et al. (2007), for example, give an extensive overview of
eLearning standards aimed at developers of web-based eLearning systems because “they establish highlevel principles for organizing learning resources and developing web-based education (WBE)
applications.” As things stand, an ITS constructs its internal model of a specific learner solely from its
interactions with that learner. The modeling framework is constructed from cognitive and learning
science, but the only data used to instantiate the models come from interactions with the ITS. The ability
to retrieve learner information from other systems will change that and encourage developers to separate
their innovative and proprietary adaptation engines from the portions of their systems that interoperate
with other learning technologies.

Models Should Not Be Standardized
We have suggested that learner information should be standardized but have avoided suggesting the same
for learner models. In fact, we do not believe that standardization is appropriate for any of the models
associated with the inner workings of ITSs.
In the abstract, ITSs adjust their instructional strategy based on a learner model, expert model domain
model, and pedagogical model (Durlach & Ray, 2011), but very few real-world ITSs use all these
components and the nature of each of these components can vary wildly. For example, the model-tracing
cognitive tutors built by Carnegie Learning (Anderson, Corbett, Koedinger & Pelletier, 1995) encode
expert knowledge in logical production rules and give hints to students when their input violates one of
the rules, whereas ALEKS (2011) tracks which topics a student has mastered and, using data gathered
from a large population of previous learners, infers which new topics the student is likely to be able to
master next. In contrast to cognitive and model-tracing tutors, the AutoTutor family (Graesser et al.,
2004; Hu et al., 2009) has less explicit encoding of domain knowledge. These tutors use semantic analysis
to determine how relevant student input is to content that has been presented, and in some cases, adjust
their strategies based on affective states determined by sensor data (D’Mello & Graesser, 2010; D’Mello
& Graesser, 2012). An examination of examples cited in sources (Graesser, Jeon & Dufty, 2008; Murray,
2003; Ohlsson & Mitrović, 2006; Woolf, 2009) reveals even further diversity in the way that ITSs
operate.
This diversity exists for a good reason. Motivated by the two-sigma effect size observed with one-on-one
human tutoring, ITSs attempt to replicate this experience with technology (Kulik & Kulik, 1991;
VanLehn, 2011). Each system’s technical approach to building and using models to achieve this goal is
its principal locus of innovation and a chief source of differentiation in the marketplace. There is a
significant difference between macro-adaptations, which persist over time and can be used by multiple
systems, and micro-adaptations, which are ephemeral and lie fully in the domain of a single system. The
candidate standardization categories in the previous section are all macro-adaptations. Since ITSs derive
their competitive advantage from micro-adaptations and the models that enable them, standardization of
these models would not be accepted and, if accepted, may hinder innovation.

Recommendations for GIFT and Future Research
GIFT (Sottilare, 2012) provides a testbed in which multiple ITSs can function. Its architecture
contemplates that ITSs will exchange data with an LMS. Adoption of GIFT would encourage ITS
developers to include LMS-compatible reporting mechanisms, which we argue is a necessary step for
market diffusion of ITS technology.

11

Design Recommendations for Intelligent Tutoring Systems - Volume 1: Learner Modeling
GIFT also includes a learner module, which we would view as learner information exchange service.
There are existing standards that can serve as the basis for constructing such a service, and any reasonable
candidate standard for learner information exchange will be extensible enough to evolve over time.
The key question, however, is what data ITSs actually need to exchange to enhance learning gains and
support the other intelligent systems and apps used by teachers and students. This is a difficult research
question, and the history of learning technology standardization teaches that abstract and theoretical
answers to questions of this nature do not lead to practical and adoptable standards. The best approach is
likely repeated cycles of experimentation and observation. That approach may require creating tutors that
implement well-controlled and limited functionality, measuring their effect on learning, and iteratively
refining standards for learner information exchange based on the results. A framework such as GIFT is
ideally suited as the “breadboard” for such experimentation.

References
Adkins, S. (2011). The U.S. Market for Self-paced eLearning Products and Services: 2010-2015 Forecast and
Aanalysis.
ADL. (2006). Sharable Content Object Reference Model (SCORM) 2004. (3rd ed.). Alexandria, VA: Advanced
Distributed Learning initiative.
Advanced Distributed Learning. (2013). Experience API, from http://www.adlnet.gov/capabilities/tla/experience-api
AICC. (2004). CMI Guidelines for Interoperability (Rev 4.0 ed.): Aviation Industry CBT Committee.
ALEKS. (2011). ALEKS Home Page Retrieved October 20, 2011, from http://www.aleks.com
Anderson, J. R., Corbett, A. T., Koedinger, K. R. & Pelletier, R. (1995). Cognitive tutors: Lessons learned. The
journal of the learning sciences, 4(2), 167-207.
Azevedo, D. & Ojeda, M. (11/19/12). SCORM, from
https://groups.google.com/forum/?fromgroups=#!msg/khanacademydevelopers/pEts3twbk5g/3Y9SL4sHgZoJ
Barr, A., Beard, M. & Atkinson, R. C. (1975). A rationale and description of a CAI program to teach the BASIC
programming language. Instructional Science, 4(1), 1-31.
Bersin, J. (2012). LMS 2013: The $1.9 Billion Market for Learning Management Systems: Deloitte.
Blessing, S. B., Gilbert, S. B., Ourada, S. & Ritter, S. (2009). Authoring model-tracing cognitive tutors.
International Journal of Artificial Intelligence in Education, 19(2), 189-210.
Cedefop. (2012). Europass Retrieved February, 2013, from http://europass.cedefop.europa.eu/en/home
Christensen, C. M. & Raynor, M. E. (2003). The innovator’s solution: Creating and sustaining successful growth:
Harvard Business Press.
Devedzic, V., Jovanovic, J. & Gasevic, D. (2007). The pragmatics of current e-learning standards. Internet
Computing, IEEE, 11(3), 19-27.
Dimitrova, V. (2009). Artificial Intelligence in Education: Building Learning Systems that Care: From Knowledge
Representation to Affective Modelling (Vol. 200): Ios PressInc.
Dodds, P. (2001). Sharable Content Object Reference Model v 1.2: Advanced Distributed Learning Initiative.
Dodds, P. & Fletcher, J. D. (2004). Opportunities for New “Smart” Learning Environments Enabled by NextGeneration Web Capabilities. Journal of Educational Multimedia and Hypermedia, 13(4), 391-404.
Durlach, P. J. (2012). Vanilla, Chocolate, or Chunky Monkey: Flavors of Adaptation in Instructional Technology.
iFest 2012, from http://www.adlnet.gov/wp-content/uploads/2012/08/Durlach_Adaption_in_IT_iFest2012.pdf
12

Design Recommendations for Intelligent Tutoring Systems - Volume 1: Learner Modeling
Durlach, P. J. & Ray, J. M. (2011). Designing adaptive instructional environments: Insights from empirical evidence
Army Research Institute Report. Arlington, VA.
Global Industry Analysts. (2013). Global eLearning Market to Reach $107.3 Billion by 2015. Reported on PRWeb.,
from http://www.prweb.com/releases/elearning/corporate_elearning/prweb4531974.htm
Graesser, A. C., Jeon, M. & Dufty, D. (2008). Agent technologies designed to facilitate interactive knowledge
construction. Discourse Processes, 45(4-5), 298-322.
Graesser, A. C., Lu, S., Jackson, G. T., Mitchell, H. H., Ventura, M., Olney, A. & Louwerse, M. M. (2004).
AutoTutor: A tutor with dialogue in natural language. Behavior Research Methods, 36(2), 180-192.
Hu, X., Cai, Z., Han, L., Craig, S. D., Wang, T. & Graesser, A. C. (2009). AutoTutor Lite.
IEEE. (2000a). IEEE Learning Technology Standards Committee (LTSC) Retrieved February, 2013, from
http://web.archive.org/web/20010217054051/http://grouper.ieee.org/LTSC/
IEEE. (2000b). IEEE P1484.2 Learner Model Working Group Retrieved February, 2013, from
http://web.archive.org/web/20010221110739/http://ltsc.ieee.org/wg2/index.html
IMS Global Learning Consortium. (2004). IMS content packaging specification v 1.1.4.
IMS Global Learning Consortium. (2005). IMS Learner Information Package.
IMS Global Learning Consortium. (2011). IMS Common Cartridge Specification.
IMS Global Learning Consortium. (2012). Learning Tools Interoperability, from
http://www.imsglobal.org/toolsinteroperability2.cfm
Kulik, C.-L. C. & Kulik, J. A. (1991). Effectiveness of computer-based instruction: An updated analysis. Computers
in human behavior, 7(1), 75-94.
Murray, T. (2003). An Overview of Intelligent Tutoring System Authoring Tools: Updated analysis of the state of
the art. Chapter 17 in Murray, T., Blessing, S. & Ainsworth, S. Artificial Intelligence.
Ohlsson, S. & Mitrović, A. (2006). Constraint-based knowledge representation for individualized instruction
Computer Science and Information Systems, 3(1), 1-22. doi: 10.2298/CSIS0601001S
Poltrack, J., Hruska, N., Johnson, A. & Haag, J. (2012). The Next Generation of SCORM: Innovation for the Global
Force. Paper presented at the The Interservice/Industry Training, Simulation & Education Conference
(I/ITSEC).
Robson, R. (2006). Globalization and the future of standardization. Computer, 39(7), 82-84. doi:
10.1109/mc.2006.231
Sleeman, D. & Brown, J. S. (1982). Intelligent tutoring systems.
Sottilare, R.A., Brawner, K.W., Goldberg, B.S. & Holden, H.K. (2012). The Generalized Intelligent Framework for
Tutoring (GIFT). Orlando, FL: U.S. Army Research Laboratory – Human Research & Engineering
Directorate (ARL-HRED).
VanLehn, K. (2011). The Relative Effectiveness of Human Tutoring, Intelligent Tutoring Systems, and Other
Tutoring Systems. Educational Psychologist, 46(4), 197-221.
Woolf, B. P. (2009). Building intelligent interactive tutors: Student-centered strategies for revolutionizing elearning. Burlington, MA: Morgan Kaufmann.

13

Design Recommendations for Intelligent Tutoring Systems - Volume 1: Learner Modeling

14

Design Recommendations for Intelligent Tutoring Systems - Volume 1: Learner Modeling

CHAPTER 3 ‒ Important Considerations for Learner Models:

Transfer Potential and Pedagogical Content Knowledge
1

Alan Lesgold1 and Arthur Graesser2
University of Pittsburgh; 2 University of Memphis

Introduction
In the age of the intelligent machine, training of humans inevitably requires training for transfer. If we
knew exactly what a person should do in a situation, we could program a computer or robot to do it
instead. However, no two situations are exactly the same so uncertainty and differences arise in matches
between the current situation and episodes of the past. Modeling transfer has traditionally meant getting
as close as possible to modeling the total expertise trainees were expected to have in handling episodes of
the past and future. We suggest that there may be more effective approaches today. One possibility is to
model knowledge (general or specific) that would permit a person to represent a situation in a form that
can be handled by diverse training and transfer trajectories. A second possibility would be to establish a
principled way to prioritize which knowledge should be overlearned so that it can be “stretched” to novel
situations not covered in training. A related approach would be to do both. For example, one could
perform situation modeling within the domain of interest along with cognitive load analyses of the
situation modeling examples drawn from the target domain. This chapter explores these options and
considers what task analysis and expert modeling is needed to capture them.
If we want people to carry out performance of a task in a precise way, we can conduct a task analysis and
then teach each of the elements of the performance and directly measure how well each element is
acquired. This approach to training worked remarkably well for the training of line workers in the
industrial age and of enlisted personnel in the military prior to the knowledge revolution (Collins &
Halverson, 2009). It assumed availability of supervisors or officers who would creatively handle
unexpected challenges and provide direction that allowed teams to adapt to emergent situations. It was
generally assumed that these leaders did not need carefully monitored complete training for their roles but
rather that they would be selected as being intelligent enough to prove a useful bridge from the formal
training of their subordinates to the actual situations their unit might encounter.
Now that we are entrenched in the knowledge revolution, however, routine performance can be specified
so precisely that we can teach a person or program an intelligent machine to perform these procedures
competently. Moreover, once empowered by intelligent tools, every worker or soldier has a role akin to
the leaders or officers of times past. Nevertheless, experience has shown that most workers in intelligent
work environments need substantial training, even if we cannot fully drill them on every aspect of the
performances we hope they will exhibit. This creates a need to consider transfer in the design of training.
That is, we are no longer merely preparing people by teaching them all the elements, rules, or procedures
that define perfect performance in every likely circumstance. Rather, they need to be prepared for
emergent situations that deviate from original training. Our hope is that our trainees will show transfer
from the specific prescribed training to ideal performance on emergent tasks.

Related Research
The profound challenge is that there typically is unspectacular transfer from training to new situations
(Banich & Caccamise, 2010). A classical study by Hayes and Simon (1977) had college students
attempting to solve a series of problems that had structurally identical solutions but varied in surface
characteristics, such as substituting names of characters and objects (Hobbits and Orcs vs. Monsters and
15

Design Recommendations for Intelligent Tutoring Systems - Volume 1: Learner Modeling
Globes). Transfer between four successive problems was near zero unless students were explicitly
instructed to make similarity connections between the problems. Gick and Holyoak (1980) investigated
whether the structure of a story about troops converging on a location would help college students solve a
radiation problem that had a direct structural isomorph to the story. The transfer was modest without
cognitive activities that intentionally tried to establish correspondences between the two representations.
At a more global practical level, the order of courses in a curriculum is rarely backed by substantiating
transfer data (Vuong, Nixon & Towle, 2011). For example, there is not an abundant body of empirical
evidence that calculus helps engineering or that physics helps students understand chemistry. Knowledge
and skills are highly constrained by specific characteristics of the subject matter.
The above examples present a dismal picture of transfer, but it can be countered by exemplars of
successful transfer. The literature is replete with evidence that training on particular tasks can facilitate
performance on similar tasks in the future. The devil is in the details of the similarity of the stimuli, tasks,
and associated cognitive representations (Gentner & Markman, 1997).
The transfer problem has enormous implications for the GIFT architecture (Sottilare, Brawner, Goldberg
& Holden, 2012), particularly with respect to the distinction between domain-independent and domaindependent modules. The current GIFT architecture specifies that the sensor and pedagogical modules are
domain-independent whereas the subject-matter knowledge is domain dependent. The implication of the
domain-independent modules needs to be clarified. Does that mean that there should be significant
transfer between tasks that have similar subject matters but different sensor and pedagogical modules?
Does that mean that the same pedagogical methods can be applied to a broad set of subject matters, as
opposed to the pedagogical modules being distinctively tailored for particular subject matters? Does
subject matter (domain knowledge) reign supreme over sensor and pedagogical modules? What are the
priorities among these GIFT modules on the matter of transfer?
We propose that a model of student2 learning has two categories of knowledge in an intelligent training
system. First, it knows how well the student has mastered the elements specifically being trained. Second,
it knows how prepared the student is to confront categories of situations that cannot be predicted and
cannot be rehearsed adequately. This latter requirement means that such systems must embody a theory of
transfer that can allow us to know how far a student’s knowledge might stretch.
Consider, for example, a football coach. The coach might train the team to execute a particular kind of
play, such as an option running play. The team can practice each of the two or three ball carrier options,
so that every team member knows who to pass the ball to, who to get the ball from, or where the players
position themselves to block the defense. However, the practice sessions only rehearse some of the
possibilities because the defensive players are intelligent entities and do not necessarily behave exactly as
the sham defense set up during the practice sessions. Yet the coach is hopeful that the team will perform
well in every case.
Some of the performance environments are predictable. By having the defense behave intelligently and
conform to known football best practices, the coach can create a variety of practice situations that
anticipate what will happen in an upcoming game. However, the game will have situations that deviate
from the ones on which practice occurred. A theory of practice is useful to the extent that it allows the
coach to predict game performance from practice performance, select the right practice situations, and
decide when there has been enough practice.

2

For simplicity of exposition, we use terms like “student model” and “students” even though the primary audience
for this volume are training developers and training system designers, not schoolteachers or professors.
16

Design Recommendations for Intelligent Tutoring Systems - Volume 1: Learner Modeling
Unpredictable situations will nevertheless occur. It might be icy or rainy. The field might be muddy or not
the same surface as that on which practice occurred. The players may change as wounds heal, injuries
occur, and so on. Consequently, there need to be global indicators of whether team members are ready for
a wider range of unpredictable occurrences. A decision is needed as to whether the current situation is
under the realm of status quo or an unusual case. An adequate model would identify specific
performances that a player can do well in prototypical situations as well as some estimates of which
categories of unpredictable cases can be expected to be adequately handled. The latter models the transfer
potential of what was explicitly learned. One way to address the problem of transfer is to consider the two
skill sets separately. That is, there are the specific performance skills that encompass expertise in the
target domain. There are also situation representation capabilities that allow a student to classify a novel
situation as one in which particular learned performances can be successful. These situation
representational skills mediate the availability of learned rules to be applied in emergent situations. Our
experiences in building intelligent training systems have convinced us that these two skill sets should be
handled in different ways.
The first author has developed and tested intelligent training systems for maintaining equipment in the
military and business sectors, including troubleshooting equipment failures (Gott & Lesgold, 2000; Gott,
Lesgold & Kane, 1997; Lesgold & Nahemow, 2001). For the specific performance skills, there was the
standard approach of building an expert model that was capable of solving the range of tasks that was
targeted for training. The initial assumption was that once each rule in the expert model was demonstrated
by the student, training was considered completed. However, our experience quickly revealed that this
was not entirely the case because the target domains we addressed were extremely complex. All of the
information needed to trigger each of the rules relevant for a given task was present in the task domain,
but there was so much information that a serious challenge remained in representing the situation at hand
sufficiently to trigger appropriate rules.
The challenge of system complexity required us to train the situation representation skills needed to
perceive the problem domain adequately to trigger expert rules. Our expert informants initially believed
that this required training on recognizing the various components of the complex target domain. For
example, in the case of avionics test stations, this would include the system modules and the components
of the modules. However, it was apparent that all of the students could recognize all of the modules and
their components, even at the beginning of training. The needed situation representation skills were more
subtle than our expert informants realized.
A reasonable training method would be to directly teach how to recognize which aspects of a situation are
important. This is done in football when players are taught the names and overall strategies behind
various types of plays. It also is done in medicine when physicians in training directly learn to recognize
various syndromes for which diagnostic rule sets are known. This approach to teaching for transfer can
work when there are recurrent examples of these various plays or syndromes. In football, that happens
because the coach has studied game films and knows the patterns that the opposing team is likely to use.
In medicine, it happens because various genetic and environmental factors predispose human bodies to
exhibit various syndromes or patterns of malfunction (e.g., lots of us overeat and under-exercise, with the
bodily response being pretty stereotypic).
Engineered systems generally are built and modified to adapt to the range of circumstances in which they
are deployed, but unfortunately they often do not have predictable patterns of breakdown. If those existed,
they would be engineered away. This makes it harder to teach situation representation by teaching how to
recognize syndromes; there are few if any recurrent syndromes to teach. This required us to take a
different approach that did not explicitly teach students to watch for specific patterns, at least for the most
part. Instead, students were provided a useful range of experiences that prompted them to construct rule-

17

Design Recommendations for Intelligent Tutoring Systems - Volume 1: Learner Modeling
based knowledge on top of their prior experiences with the task domain. This more flexible emphasis on
the range of the situations was expected to extend the students’ knowledge to new situations.
The scheme that was developed was called intelligent coached apprenticeship. Basically, we pushed
students to go beyond their overlearned knowledge to solve novel problems (Gott & Lesgold, 2000). We
expected that none of the problem tasks our training system provided would be solvable by the students
without help. Five levels of help were available. The lowest level listed the sequence of actions the
student already had taken to address the problem. More than half of the time, that level of hint was
sufficient to keep the student moving toward a solution to the problem. If the student was still stuck, there
would be more explicit levels of hint (e.g., where to look for information) until a final hint level that told
the student exactly what to do on the next step. For the population our training addressed, the desire to
learn was strong, and students seldom asked for more help than they needed. After completing the
solution for the problem, they also were able to study a comparison of their solution path to that of an
expert.
The central hypothesis in this research (Gott & Lesgold, 2000; Gott, Lesgold & Kane, 1997) is that
scaffolding for students to shape their own construction of situation representation skills would produce
the needed development of those skills, even if we could not explicitly list them all and even if it was not
feasible to build a complete set of situation representation rules and train on each. It was reassuring that
our efforts were highly successful. We had expert technicians develop a collection of far transfer tasks
that involved applying the expert rules we taught to new hardware. The transfer hardware was a fictitious
piece of hardware that could still be diagnosed using the expert rule set that our training system
embodied. The students we trained performed in about the same range as experts on those problems (Gott
& Lesgold, 2000; Gott, Lesgold & Kane, 1997).
It may eventually be possible to teach the needed situation representation skills directly rather than to
assume them and indirectly stimulate their construction by the students. For example, Forbus et al. (2007)
have made progress in specifying what a computer needs in order to learn similar kinds of
representational skills. Nevertheless, the scheme we developed is worthy of further exploitation. On the
one hand, it focuses on the expert rules needed to do the necessary problem solving, and on the other
hand, it does not ignore the reality that some level of further knowledge construction is needed if those
rules are to be available when needed in addressing novel tasks.
There will always be uncertainty that an individual student’s understanding of the task domain embodies
entirely the same constructs that the training designer might have had. This is especially the case in
complex technical domains, where the training designers often know much more basic science than the
technicians being trained. As an example, consider the terms used in expert rules for diagnosing failures
of an ion beam system for writing circuits on computer chips (Lesgold & Nahemow, 2001). A complete
account requires knowledge of quantum physics, silicon chemistry, and optics. Technicians two years out
of high school could learn the diagnosis rules well enough to apply them in transfer situations using the
training technique of providing difficult problems with scaffolding and post-problem reflection (Lesgold
& Nahemow, 2001).

Discussion
The intelligent coached apprenticeship was impressively effective, producing learning effect sizes
exceeding one standard deviation. These results are on par with or exceed intelligent tutoring systems that
have been developed and tested during the last decade (Graesser, Conley & Olney, 2012; VanLehn,
2011). However, questions remain about the information that is needed in the students’ situation

18

Design Recommendations for Intelligent Tutoring Systems - Volume 1: Learner Modeling
representation to assure adequate learning. There are also questions directly relevant to GIFT (Sottilare et
al., 2012).
The systems described in the previous section did not log information about the students’ situation
representation, and it did not retain its short-term estimations of student reasoning strategies. Instead, the
systems were validated by conducting far transfer studies. This is an expensive way to proceed. It is worth
considering simpler ways to assure adequate learning of rules that provide substantial transfer. Some of
these possibilities are presented in this section.
One important step would be to record information about the level of coaching/scaffolding that a student
requires. The systems described above did that, but the data were not retained beyond their presence in a
post-problem recapitulation of the student’s activity and a comparison of that to an expert solution. GIFT,
in contrast, routinely records in the log files the raw sensor data, computer-student interactions,
pedagogical strategies being implemented, expected student responses (both correct and misconceptions),
actual student responses, and other data that can be mined for system improvement. At any given moment
in solving a problem in the intelligent coached apprenticeship systems, the system kept track of what the
best next step for a student. Therefore, it would be straightforward to record data that indicates that the
student has a misconception or does not know what to do next (see, for example, the inference of
“malrules” by systems VanLehn [1981] has built). Informative episodes occur when the student asks for
coaching or performs a non-optimal next step. As the student starts taking appropriate actions, the past
records of incomplete learning could be retired and eventually there would be confidence that a rule has
been learned broadly. GIFT supports such data mining and machine learning activities.
Note that the approach is substantially different from direct training of a rule. In direct training, the
student is placed in circumstances where the rule should be triggered and then taught explicitly to apply it.
The problem with the direct training approach is that the rules flagged as learned may not be triggered in
circumstances where complex tasks are being performed without attention focused directly on possible
circumstances where the rule applies. We have known for almost a century (Whitehead, 1929) that
specifically learned bits of knowledge often are not used in broader circumstances where they should
apply. By focusing more of learning on explicitly stretching one’s knowledge, this problem has been
avoided. The question arises whether training of humans is needed at all, given that the intelligent
coached apprenticeship systems relied upon expert models to provide coaching. Expert models would
perhaps not be able to transfer to novel situations any more than novice technicians. We were able to
formally represent each problem in a way that permitted the expert system to solve it, but those
representations were only possible because of the additional knowledge of the experts who developed
them. That more implicit knowledge is exactly what intelligent coached apprenticeship endeavors to train.
Decisions will need to be made on how the above mechanisms in the intelligent coached apprenticeship
would be implemented in GIFT. The student-constructed situation representation is not merely stored in
the GIFT Domain Module, but rather is apparently derived in the Learner Module from a combination of
activities involving the Sensor Module, Pedagogical Module, the Domain Module, and history of the
logged data. Does this complex interactivity clash with the modular assumption that differentiates the
domain-dependent Domain Module from the remaining domain-independent components?
Another important step for implementing effective intelligent coached apprenticeship systems resides in
tracking mastery of rules in broad contexts. The problem sets presented to students need to be sufficiently
challenging and span a wide enough range of situations. They can never span all the situations a student
will encounter when applying what is being taught. However, they need to span a sufficiently wide range
that they force students to reflect on why each rule is applicable and the range of possible situations of
applicability. For the two rather different domains developed by Lesgold and his colleagues, the expert
technicians helped build problems that are sufficiently diverse that there was transfer to novel situations.
19

Design Recommendations for Intelligent Tutoring Systems - Volume 1: Learner Modeling
One situation involved diagnosis of failures in complex switching systems designed to test electronic
components from military aircraft. The range of problems experts helped us develop was sufficient to
produce far transfer performance that was about as good as that of domain experts (Gott, Kane &
Lesgold, 1995; Gott, Lesgold & Kane, 1997). The second domain in which we did similar training and
testing with similar results was the diagnosis of failures in machines used to make computer chips,
notably ion deposition systems that put layers on chips and ion beam implant systems that write circuits
on those layers (Lesgold & Nahemow, 2001). The switching systems had underlying knowledge
associated with simple electrical properties of circuit continuity and magnetics (relays were involved).
The chip-making systems involved underlying knowledge of electrical systems, heat distribution, silicon
gas compounds, movements by robots, and to some extent, even more complex basic science.
The same basic approach worked for both domains so it is at least a reasonable conjecture that it can work
much more widely. It would be worthwhile, however, to study the utility of logging indications of
uncertainty, lack of knowledge, or misconceptions at points where the rules in an expert system are not
applied routinely or accurately. Such information might allow better decisions on how much training is
needed to produce mastery. Once again, the logged data, data mining, and machine learning facilities of
GIFT are, in principle, equipped to implement this approach. Decisions will need to be made on the
division of labor among GIFT modules when accommodating changes to the system.

Recommendations and Future Research
The intelligent coached apprenticeship system described in this chapter underscores the importance of
capturing situation representations that are constructed by the students in order to handle a diverse range
of cases applicable to transfer situations. Domain experts will be needed to select the problems that
deviate from routine cases that can be trained explicitly. Research will therefore be needed to understand
how these cases/problems are selected, the mapping between training and transfer cases, the situation
representations that students construct, and their ability to identify unusual cases. A detailed analysis of
the log data should be helpful in these research efforts and also in modification of the systems in iterative
development.
There are two major recommendations directly relevant to GIFT. First, decisions will need to be made on
how the specific modules will participate in the intelligent coached apprenticeship system. It is not a
simple matter of storing content in the Domain Module. Second, decisions will need to be made on how
to represent the information stored in the log files and the various modules. It is not a simple matter of
storing everything. The features, content, and structures will need to be able to support new cases and
situation representations in addition to domain-dependent and domain-independent information.

References
Banich, M.T. & Caccamise, D. (2010)(Eds.) Generalization of knowledge: Multidisciplinary perspectives. New
York: Psychology Press. .
Collins, A. & Halverson, R. (2009). Rethinking education in the age of technology: The digital revolution and
schooling in America. New York: Teacher College Press.
Forbus, K., Riesbeck, C., Birnbaum, L., Livingston, K., Sharma, A. & Ureel, L. (2007). Integrating natural
language, knowledge representation and reasoning, and analogical processing to learn by reading.
Proceedings of AAAI-07: Twenty-second Conference on Artificial Intelligence. (1542-1547). Vancouver,
BC: AAAI.
Gentner, D. & Markman, A. (1997). Structure mapping in analogy and similarity. American Psychologist, 52, 45-56.

20

Design Recommendations for Intelligent Tutoring Systems - Volume 1: Learner Modeling
Gick, M.L. & Holyoak, K.J. (1980). Analogical problem solving. Cognitive Psychology, 12, 306-355.
Gott, S. P., Kane, R. S. & Lesgold, A. (1995, February). Tutoring for transfer of technical competence. Armstrong
Laboratory Technical Report AL/HR-TP-1995-0002. Brooks AFB, TX: Air Force Materiel Command.
Gott, S. P., Lesgold, A. M. & Kane, R. S. (1997). Promoting the transfer of technical competency. In S. Dijkstra, et
al. (Eds.), Instructional design: International perspectives (vol. 2). (221-250). Hillsdale, NJ: Erlbaum.
Gott, S. P. & Lesgold, A. M. (2000). Competence in the workplace: How cognitive performance models and situated
instruction can accelerate skill acquisition. In R. Glaser (Ed.), Advances in instructional psychology. (239327). Hillsdale, NJ: Erlbaum.
Graesser, A.C., Conley, M. & Olney, A. (2012). Intelligent tutoring systems. In K.R. Harris, S. Graham, and T.
Urdan (Eds.), APA Educational Psychology Handbook: Vol. 3. Applications to Learning and Teaching (pp.
451-473). Washington, DC: American Psychological Association.
Hayes, J.R. & Simon, H.A. (1977). Psychological differences among problem isomorphs. In J. Castellan, D.B.
Pisoni & G. Potts (Eds.), Cognitive theory, vol. 2. Hillsdale, NJ: Erlbaum.
Lesgold, A. & Nahemow, M. (2001). Tools to assist learning by doing: Achieving and assessing efficient technology
for learning. In D. Klahr & S. Carver (Eds.), Cognition and instruction: Twenty-five years of progress.
Mahwah, NJ: Erlbaum.
Sottilare, R.A., Brawner, K.W., Goldberg, B.S. & Holden, H.K. (2012). The Generalized Intelligent Framework for
Tutoring (GIFT). Orlando, FL: U.S. Army Research Laboratory – Human Research & Engineering
Directorate (ARL-HRED).
VanLehn, K. (1981). Bugs are not enough (Tech. Rep. No. CIS-11). Palo Alto, CA: Xerox Research Center.
VanLehn, K. (2011). The relative effectiveness of human tutoring, intelligent tutoring systems and other tutoring
systems. Educational Psychologist, 46, 4, 197-221.
Vuong, A., Nixon, T. & Towle, B. (2011). A method for finding prerequisites within a curriculum. In Pechenizkiy,
M., Calders, T., Conati, C., Ventura, S., Romero , C., and Stamper, J. (Eds.), Proceedings of the 4th
International Conference on Educational Data Mining (pp. 211-216).
Whitehead, A. N. (1929/1967). The aims of education and other essays. New York: The Free Press.

21

Design Recommendations for Intelligent Tutoring Systems - Volume 1: Learner Modeling

22

Design Recommendations for Intelligent Tutoring Systems - Volume 1: Learner Modeling

CHAPTER 4 ‒Matching Learner Models to Instructional

Strategies
Andrew M. Olney and Whitney L. Cade
University of Memphis

Introduction
Learner models represent key variables that guide instructional strategies during e-Learning. By
representing key variables about a particular student, learner models make adaptive instruction possible.
Without learner models, instruction cannot be individualized and instead is often calibrated to the average
ability learner or the lowest ability learner. Learner models enable instruction to focus on what the student
doesn’t know and adjust to the student’s abilities, motivation, and preferences. In other words, learner
models enable adaptive instructional strategies.
Given the tight correspondence between learner models and instructional strategies, they must be
considered in parallel when designing e-Learning systems. In fact, a strong argument could be made that
the choice of instructional strategy drives all other modeling decisions in e-Learning by placing
requirements on the learner model and other related models (see Pavlik et al., Chapter 5 in this volume,
for a review). Thus, one approach to studying learner models would be to conduct an analysis of all
possible instructional strategies and then examine what requirements they place on learner models.
However, the vast number of instructional strategies that have been proposed in the literature make this
approach somewhat impractical.
In this chapter, we explore an alternative approach to strategy-based analysis of learner models. The
Institute for Simulation and Training at the University of Central Florida has recently published an online
database known as the Instructional Strategies Indicator (ISI) (Tarr, 2012). The ISI contains 150
instructional strategies, indexed by when and where instruction occurs, the evidence for instructional
efficacy, the size of the group being instructed, the expertise of the learner, and the type of knowledge
being taught. The ISI represents an effort to organize the known instructional strategies into a
comprehensive framework, allowing for the optimal selection of an instructional strategy in a given
instructional setting.
The methodology used to create the ISI was qualitative and data-driven, using aspects of grounded theory
methodology to select relevant literature and create an analytic framework to describe the literature.
Vogel-Walcutt, Fiorella, and Malone (2012) conducted searches of PsychInfo, the Educational Resource
Information Center (ERIC), and Google Scholar using a predeﬁned set of search terms and restricting the
dates of studies to between 2000 and 2010. Of the 4,515 articles returned, only 771 were retained as
relevant to the ISI criteria, which included relevance to military training. In addition to being coded
according to the dimensions mentioned above (e.g., when and where instruction occurs), instructional
strategies in the retained articles were rated by judges on their associated evidence for efficacy. Strategies
were ranked on a 0–9 scale based on multiple criteria of evidence, including empirical results and quality
of study, with judges’ ratings being checked for inter-rater reliability (Vogel-Walcutt, Malone & Fiorella,
2012). This process yielded 150 different instructional strategies that were included in the ISI. However,
only 13 of these strategies were given the highest rating of 7–9, which was reserved for strategies backed
by multiple randomized experiments with moderate to large effect sizes (d ≥ 0.5; Cohen, 1992).
Table 1 presents these 13 strategies with a subset of the ISI dimensions. Missing dimensions include
hierarchical categorizations of the strategy type and the knowledge, skills, and abilities to be learned.
Included dimensions are (1) timing of instruction relative to the instructional event (pre/during/post), (2)
23

Design Recommendations for Intelligent Tutoring Systems - Volume 1: Learner Modeling
setting of instruction (e.g., computer-based, classroom, or live-training), (3) group size (e.g., individual,
small group, or large group), (4) learner’s level of expertise (novice/journeyman/expert), (5) knowledge
type targeted (declarat

[The evaluation harness truncated this reference: showing the first 120000 of 978441 characters.]
</reference>

<statements>
1. The classical architectural paradigm of an intelligent tutoring system consists of four interconnected core components: the domain model, the student model, the pedagogical or tutor model, and the user-interface model
2. The pedagogical model translates disparities between the student model and the domain model into instructional interventions
</statements>

Begin the assessment now. Output only the JSON list, without any conversational text or explanations.