You will be provided with a reference and some statements. Please determine whether each statement is 'supported', 'unsupported', or 'unknown' with respect to the reference. Please note:
First, assess whether the reference contains any valid content. If the reference contains no valid information, such as a 'page not found' message, then all statements should be considered 'unknown'.
If the reference is valid, for a given statement: if the facts or data it contains can be found entirely or partially within the reference, it is considered 'supported' (data accepts rounding); if all facts and data in the statement cannot be found in the reference, it is considered 'unsupported'.

You should return the result in a JSON list format, where each item in the list contains the statement's index and the judgment result, for example:
[
    {
        "idx": 1,
        "result": "supported"
    },
    {
        "idx": 2,
        "result": "unsupported"
    }
]

Below are the reference and statements:
<reference>
ON REINFORCEMENT LEARNING FOR
QUANTITATIVE INVESTMENT VIA
THE CONDITIONAL VALUE-AT-RISK

ALI ALAMEER

SYSTEMS & CONTROL ENGINEERING
May 2022

©Ali Alameer
2022

iii

To my beloved parents, Amal and Ahmed,
my wife Shaikha, and my daughter and son, Amal and Mahdi

iv

ACKNOWLEDGMENTS
I am grateful to my thesis advisor Professor Khaled Alshehri who made this journey
motivative and educative. During the work on this thesis, his guidance and support
have profoundly better shaped my thought process on academic, professional, and
personal levels. Professor Khaled introduced to me essential financial and mathematical tools that contributed to this work’s success, particularly convex optimization and
the Conditional Value-at-Risk. I would extend my gratefulness to Professor Magdi
Mahmoud who sharpened my mathematical analysis intuition during the Nonlinear
Control course in my Graduate studies. I would also like to express my appreciation
to Professor Sami Elferik who taught me the course Adaptive Control, which was a
primary inspiration for me to deploy adaptive learning in this research. I am also
appreciative to Professor Fouad Al-Sunni for his support in introducing me to investment through machine learning during my early days of joining the Systems & Control
Engineering MSc program.
My appreciation also goes for those who provided genuine feedback that enhanced
and improved this research along the journey:
• Dr. Haitham Saleh and Dr. Ahmad Baubaid (KFUPM)
• Dr. Fahad Al-Balawi, Dr. Hani Al-Mansouri,
and Dr. Huda Alamri (National Center for Artificial Intelligence)
v

• Hussain Al-Marzoug and Ahmed Al-Humoud (Saudi Aramco)
Last but not least, I am fortunate to be surrounded by beloved people, including
my parents, siblings, wife, daughter, and son, who all were supportive and motivative
along the journey and were a pivotal contribution to my success.

vi

TABLE OF CONTENTS
ACKNOWLEDGMENTS

v

LIST OF TABLES

x

LIST OF FIGURES

xi

LIST OF SYMBOLS

xiii

LIST OF ABBREVIATIONS

xv

ABSTRACT

xvii

ABSTRACT (ARABIC)

xix

CHAPTER 1 INTRODUCTION

1

1.1

Reinforcement Learning – Historical Aspect . . . . . . . . . . . . . . .

6

1.2

Investment Paradigms . . . . . . . . . . . . . . . . . . . . . . . . . . .

7

1.3

Thesis Statement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

8

1.4

Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

9

1.5

Contribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

1.6

Outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

CHAPTER 2 LITERATURE REVIEW

15

2.1

Trading with Critic and Actor-Critic Methods . . . . . . . . . . . . . . 16

2.2

Trading with Actor-Based Methods . . . . . . . . . . . . . . . . . . . . 20

vii

CHAPTER 3 INVESTMENT IN FINANCIAL MARKETS
3.1

24

Investment Objective . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
3.1.1

Conditional Value-at-Risk . . . . . . . . . . . . . . . . . . . . . 26

3.2

Technical, Fundamental, and Macroeconomic Variables Analysis . . . . 28

3.3

Investment Decisions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

3.4

The Investment Process: A Control Perspective . . . . . . . . . . . . . 32
3.4.1

Dynamics of Financial Markets . . . . . . . . . . . . . . . . . . 32

3.4.2

Controller Design Aspects: Stability, Optimality, and Adaptability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34

CHAPTER 4 REINFORCEMENT LEARNING

36

4.1

The Markov Decision Process – The Environment and The Agent . . . 37

4.2

Optimal Value Functions and Policies . . . . . . . . . . . . . . . . . . . 39

4.3

Reinforcement Learning Approaches . . . . . . . . . . . . . . . . . . . 40
4.3.1

Critic-Based Methods . . . . . . . . . . . . . . . . . . . . . . . 41

4.3.2

Actor-Based Methods (Direct Reinforcement Learning) . . . . . 44

4.3.3

Actor-Critic Methods . . . . . . . . . . . . . . . . . . . . . . . . 47

CHAPTER 5 CVAR-BASED INVESTING AGENTS

52

5.1

Agents’ Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54

5.2

Trading Policy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57

5.3

Reward Signal

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58

5.3.1

Single-Asset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58

5.3.2

Assets Allocation . . . . . . . . . . . . . . . . . . . . . . . . . . 59

5.3.3

Including Financial Markets Friction . . . . . . . . . . . . . . . 61

CHAPTER 6 COMPUTATION OF THE APPROXIMATE OPTIMAL POLICY

63

6.1

CVaR-Constrained Agent . . . . . . . . . . . . . . . . . . . . . . . . . 64

6.2

CVaR-Sensitive Agent . . . . . . . . . . . . . . . . . . . . . . . . . . . 69

viii

CHAPTER 7 NUMERICAL EXPERIMENTS
7.1

71

Trading a Single Asset . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
7.1.1

Agents’ Sensitivity to Risk Parameters . . . . . . . . . . . . . . 72

7.1.2

Agents’ Endurance to Trading Frictions . . . . . . . . . . . . . 79

7.1.3

Comparison with Various Trading Systems . . . . . . . . . . . . 80

7.2

Assets Allocation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84

7.3

Risk-Neutral, Constrained or Sensitive Agent? . . . . . . . . . . . . . . 87

CHAPTER 8 GENERALIZATIONS

88

CHAPTER 9 CONCLUSION

92

APPENDIX A PROOFS

95

REFERENCES

98

VITAE

120

ix

LIST OF TABLES
2.1

Summary of quantitative trading with RL – Critic-tabular methods . . 17

2.2

Summary of quantitative trading with RL – Critic methods . . . . . . 18

2.3

Summary of quantitative trading with RL – Actor-critic methods . . . 19

2.4

Summary of quantitative trading with RL – Actor methods . . . . . . 21

5.1

Summary of Conditional Value-at-Risk (CVaR)-based and risk-neutral
agents’ models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55

7.1

Description of out-of-sample data for assessing CVaR agents’ sensitivity
to risk parameters and transaction costs – Trading a single asset . . . . 73

7.2

Total profit of CVaR𝛾 -constrained agent in trading single asset against
various transaction costs . . . . . . . . . . . . . . . . . . . . . . . . . . 80

7.3

Total profit of CVaR𝛾 -sensitive agent in trading the SPDR against various transaction costs . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80

7.4

Comparison of total return for various trading systems under trading
the S&P500 Index – Trading a single asset . . . . . . . . . . . . . . . . 83

x

LIST OF FIGURES
3.1

A general block diagram of the investment process in financial markets

3.2

A general probability density for a loss with its respective VaR𝛾 and

25

CVaR𝛾 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
3.3

Two candlesticks with opening, low, high, and closing prices. When
the closing price of an interval is higher than its opening, the candle is
green (left); otherwise, it is red (right). . . . . . . . . . . . . . . . . . . 29

3.4

An illustrative example for a price signal and its Simple Moving Average
(SMA) that is calculated based on the closing price of 14 past time steps. 30

3.5

Two profitable trading scenarios. The green arrows indicate a profitable round-trip trade of long-sell, whereas the blue arrows show the
profitable short-cover scenario . . . . . . . . . . . . . . . . . . . . . . . 31

4.1

The agent-environment interaction in reinforcement learning [1]. . . . . 38

4.2

The RL actor-critic architecture [1]. . . . . . . . . . . . . . . . . . . . . 48

5.1

Proposed trading system’s block diagram. Solid signals represent sensing the environment and taking action during [𝑡, 𝑡 + 1). Dashed signals
represent receiving a reward at 𝑡 + 1 . . . . . . . . . . . . . . . . . . . . 56

7.1

Price signals of the various tradable assets for assessing CVaR agents’
sensitivity to risk parameters and transaction costs – Trading a single
asset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73

7.2

Sensitivity of CVaR-constrained agent to 𝛾 and 𝜇 𝛽 under trading the
SPY-ETF . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75

xi

7.3

Sensitivity of CVaR-constrained agent to 𝛾 and 𝜇 𝛽 under trading Brent
crude . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76

7.4

Sensitivity of CVaR-constrained agent to 𝛾 and 𝜇 𝛽 under trading the
EURUSD currency pair . . . . . . . . . . . . . . . . . . . . . . . . . . 76

7.5

Sensitivity of CVaR-sensitive agent to 𝛾 under trading the SPY-ETF . 79

7.6

Comparison of cumulative return for risk-neutral and CVaR agents under trading the S&P500 Index – Trading a single asset . . . . . . . . . 82

7.7

Out-of-sample data for assets allocation experiment from January 2000
to February 2022. SPY weekly closing and adjusted closing prices (top);
weekly rates of 3-month T-bill (bottom). . . . . . . . . . . . . . . . . . 85

7.8

Cumulative return and trading signals of the risk-neutral and CVaR
agents under the assets allocation problem. Brown and yellow plots
show the performance of Buy&Hold and Buy&Hold while reinvesting
dividends, respectively. . . . . . . . . . . . . . . . . . . . . . . . . . . . 86

xii

LIST OF SYMBOLS
𝑎

Agent (Controller) action

𝛼

Stochastic gradient step size

𝛼𝑎

Tabular actor learning step size

𝛼𝑐

Tabular critic learning step size

𝜇𝛽

CVaR constraint boundary

𝑊0

Initial investment’s capital

ĉ𝛾,𝑖

Estimate of CVaR with the confidence level 𝛾 at the 𝑖 th iteration.

𝛿

Half of the round-trip transaction cost

𝛾𝑑

Reward discount rate

B𝑇

Risk constraint over the investment horizon 𝑇

U𝑇

Financial objective over the investment horizon 𝑇

𝛾

Agent’s risk-aversion level

𝑇

Investment horizon

𝑖

Computation iteration step

𝜂

Log-barrier parameter

𝐿

Loss vector
xiii

𝜋

Agent (Controller) policy

𝜓

Probability density function

Ψ

Probability distribution function

𝑝𝑡

Asset’s price quote at time 𝑡

𝜆

Regularization constant

𝑟

Risky asset’s normalized return

𝑟𝑓

Risk-free asset’s normalized return

𝑅

Reward vector

𝑥

Environment (System) state

𝜅

Dimension of the state space

𝑡

Investment time step

v̂𝛾,𝑖

Estimate of VaR with the confidence level 𝛾 at the 𝑖 th iteration.

𝑊𝑇

Final wealth over the investment horizon 𝑇

xiv

LIST OF ABBREVIATIONS
A3C

Asynchronous Actor-Critic

CNN

Convolutional Neural Network

CVaR

Conditional Value-at-Risk

DDPG

Deep Deterministic Policy Gradient

DDQN

Double Deep Q-Nnetwork

DMAC

Dual Moving Average Crossover

DP

Dynamic Programming

DPG

Deterministic Policy Gradient

DQN

Deep Q-Network

DRL

Direct Reinforcement Learning

ETF

Exchange Traded Fund

FDDR

Fuzzy Deep Direct Reinforcement

GD

Gradient Descent

GRU

Gated Recurrent Unit

LSTM

Long Short-Term Memory

MDD

Maximum Drawdown
xv

MDP

Markov Decision Process

MPT

Modern Portfolio Theory

NN

Neural Network

PG

Policy Gradient

PPO

Proximal Policy Optimization

RDPG

Recurrent Deterministic Policy Gradient

RL

Reinforcement Learning

RRL

Recurrent Reinforcement Learning

RSI

Relative Strength Index

SAC

Soft Actor-Critic

SARSA

State-Action-Reward-State-Action

SG

Stochastic Gradient

SMA

Simple Moving Average

SPDR

Standard and Poor’s Depository Receipt

𝑇𝐶

Transaction Cost

TD

Temporal Difference

TD3

Twin Delayed Deep Deterministic Reinforcement Learning

VaR

Value-at-Risk

xvi

THESIS ABSTRACT
NAME:

Ali Alameer

TITLE OF STUDY:

ON REINFORCEMENT LEARNING FOR QUANTITATIVE INVESTMENT VIA THE CONDITIONAL
VALUE-AT-RISK

MAJOR FIELD:

Systems & Control Engineering

DATE OF DEGREE: May 2022

Financial markets behave in unforeseen patterns due to the investors’ anticipation
of the price action of financial assets. Therefore, one can view financial markets as
stochastic dynamical systems exhibiting nonstationary behavior under mathematical
engineering contexts. Reinforcement Learning, a machine learning-based optimal control tool, is an attractive approach to obtaining optimal policies under such complex
systems since it does not necessarily require knowledge about the system’s dynamics to find an approximated optimal policy. Furthermore, leveraging Reinforcement
Learning with the Conditional Value-at-Risk (CVaR) as a constitutional element in
the performance index allows the agent to learn investing while dealing with implicit
and explicit risks under such a model-free framework. This thesis proposes convex

xvii

CVaR-based investment agents under the Direct Reinforcement Learning framework
to address risks associated with investing in financial markets. Due to convexity, the
proposed approach can be compatible with the incremental learning technique. With
that, we enhance the agent’s adaptability in managing risks within financial markets
at an efficient computational cost with the merit of guaranteed policy convergence.
We consider experimenting with frictional investment in various financial markets to
compare a CVaR-constrained agent’s financial performance against an unconstrained
CVaR-sensitive one. Associated with the adaptive incremental learning approach, the
CVaR-constrained agent shows potency in improving investment return per unit of risk.
In contrast, the CVaR-sensitive agent exhibits robustness in handling intense financial
markets pullbacks. Both CVaR agents reveal enhanced risk management compared to a
risk-insensitive agent under two investment problems: single asset trading and assets
allocation. Moreover, with trading a single asset, our experimentation revealed that
the proposed convex CVaR-constrained agent could outperform the state-of-art Direct
Reinforcement Learning trading system. We finally show how our CVaR agents are
extendable to learn investing under the most general investment problem of optimizing
a multi-asset portfolio.

xviii

‫ملخص الرسالة‬
‫الاسم‪:‬‬

‫علي أحمد الأمير‬

‫عنوان الرسالة‪:‬‬

‫الاستثمار الكمي باستخدام التعلم الُم عَّزز والقيمة الُم ْح تَم لة للمخاطر‬

‫التخصص‪:‬‬

‫هندسة النظم و التحكم‬

‫تاريخ الدرجة العلمية‪:‬‬

‫مايو ‪٢٠٢٢‬‬

‫دائمًا ما تشهد أسواق المال حراك يصعب التنبؤ به بسبب اختلاف توقع المستثمرين لحركة أسعار الأصول المالية‪ .‬لذلك‬
‫يمكن للشخص أن ينظر إلى أسواق المال بعين هندسية على أنها أنظمة ديناميكية ذات عشوائية متغيرة مع الزمن‪ُ .‬يعّد التعُّلم‬
‫المعَّزز‪ ،‬وهو أداة مستوحاة من نظرية التحكم الأمثل وقائمة على تعلم الآلة‪ ،‬نهًج ا جذاًبا للوصول إلى استراتيجية تحكم مثالية‬
‫تحت ظل هذه الأنظمة المعقدة لأنه لا يتطلب بالضرورة معرفة ديناميكيات النظام للعثور على استراتيجية مقاربة ‪-‬رياضيًا‪-‬‬
‫للاستراتيجية المثالية‪ .‬علاوة على ذلك‪ ،‬فإن تطبيق التعلم المعزز مع الأخذ بعين الاعتبار "القيمة المحتملة للمخاطر" كمؤشر‬
‫تراقبه الآلة الذكية يجعلها قادرة على تقييم مدى فعالية استراتيجيتها في إدارة المخاطر أثناء الاستثمار ‪ ،‬وكل ذلك يتم في إطار‬
‫يخلو من تمثيل ديناميكية أسواق المال بنماذج رياضية‪ .‬تطرح هذه الرسالة إمكانية تَع ُّلم الآلة الذكية إدارة المخاطر المتعلقة‬
‫بالاستثمار في أسواق المال عن طريق مراقبتها وتحكمها في "القيمة المحتملة للمخاطر" لقراراتها الاستثمارية ضمن إطار‬
‫التعلم المعزز المباشر‪ .‬بسبب تحُّد ب الدالة الرياضية المقترحة في هذه الرسالة‪ ،‬يمكن لآلتنا الذكية أن تتعلم إدارة المخاطر‬
‫‪xix‬‬

‫وتطوير استراتيجياتها في ذلك من خلال تجاربها المتراكمة من غير تدخل بشري‪ .‬بذلك النهج‪ ،‬فإننا نعزز قدرة الآلة على‬
‫تَع ُّلم استراتيجيات جديدة في إدارة المخاطر تتكيف مع تقلبات أسواق المال الحادة و ذلك بتكلفة حوسبية قليلة مع ميزة‬
‫ضمان وصول الآلة لاستراتيجيات مثالية‪ .‬في هذه الأطروحة نعرض نتائج تجربتنا بخصوص الاستثمار في أسواق مالية مختلفة‬
‫لتقييم الأداء المالي لآلة تتعلم وتطور استراتيجيتها بحسب متوسط عوائدها الاستثماريه مع محاولتها تقنين "القيمة المحتملة‬
‫للمخاطر" بنفس الوقت‪ .‬و في المقابل قمنا بتجربة آلة أخرى تنظر فقط إلى تقليص "القيمة المحتملة للمخاطر" المتعلقة‬
‫بعوائد استثماراتها‪ .‬من خلال تلك التجارب‪ ،‬لاحظنا فعالية الآلة الأولى في تحسين عائد الاستثمار لكل وحدة قياس للخطر‪.‬‬
‫على الجانب الآخر‪ُ ،‬تظهر الآلة الثانية متانة في التعامل مع التراجعات الحادة في أسعار الأصول المالية‪ .‬كل تلك الآلات‬
‫الذكية المطروحه في رسالتنا أظهرت قدرة واضحة على إدارة المخاطر وبصورة أفضل عندما ُقورنت مع آلة أخرى غير معنّية‬
‫بتعلم إدارة المخاطر‪ ،‬وذلك في ظل تجربتنا لنوعين من أنواع الاستثمارات في أسواق المال‪ :‬المضاربة ‪ ،‬وتوزيع رأس المال‬
‫على أصلين بمخاطر متفاوتة (أسهم شركات ُم ْد رجة وسندات حكومية)‪ .‬علاوة على ذلك كشفت تجربتنا أن الآلة المقترحة‬
‫في هذه الرسالة يمكن أن تتفوق على أحدث آلة تداول مطروحة في الأوراق العلمية المتعلقة بالاستثمار باستخدام التعلم المعزز‬
‫المباشر‪ .‬أخيًر ا‪ ،‬عّم منا نظرًّيا إمكانية الآلات الذكية المقترحة في هذه الأطروحة أن تتعلم إدارة المحافظ الاستثمارية متعددة‬
‫الأصول‪.‬‬

‫‪xx‬‬

CHAPTER 1

INTRODUCTION
Financial markets exhibit unpredictable behavior centrally caused by investors’ sentiment toward the overall market movement, or more precisely, the price signals actions.
That sentiment is often not unified among investors and mainly develops based on
their perception of news, such as political, economic, or financial announcements of
public companies. Researchers are thus motivated by this tendency to continue investigating and devising practices for optimal risk management when investing. The
continuous evolution of mathematical engineering tools has consistently introduced
new potential risk management strategies, causing quantitative investment to remain
an active research area. Quantitative investment is a sub-field of quantitative finance
and is a vastly growing area, and it is fundamentally related to developing profitable investment strategies under mathematical engineering contexts. In the 1950s,
Markowitz introduced a central concept in investing where he formulated the problem
of portfolio management as a mathematical program under a single-stage optimization framework [2]. Nonetheless, developing engineering-based investment techniques
with consistent performance while adequately managing risk is complex since finan1

cial markets exhibit severe uncertainties with nonstationary behavior [3]. Under the
engineering context, financial markets can be viewed as stochastic systems with timevariant dynamics. To optimize a financial performance measure of a quantitative
investment strategy, that system, indeed, requires tools from control theory. Recent
developments in control provided vital tools that are yet to be utilized in quantitative finance. For example, market-making [4], cost-optimal trades execution [5],
and, specifically quantitative investment, assets allocation, and portfolio management
problems [6], [7], which all might be looked at through the lens of stochastic optimal
control.
The emergence of Reinforcement Learning (RL) [1], a machine learning paradigm
that intersects with optimal control theory has prospered optimal control tools to
approach real-life problems associated with stochastic dynamical systems. RL agents
have intelligibly proven success in such systems under practical setups. For instance,
in defeating world champions in the game of Go and Chess with RL algorithms such
as “AlphaGo” and “AlphaZero” [8], [9] along with the “MuZero” agent that showed
superior performance in a wide range of games including Shogi and Arcade [10]. That
success, indeed, motivated researchers to explore RL potency in the field of assets allocation and portfolio optimization. The inception of quantitative investment through
RL was by Neuneier in [11], [12] where he approached the problem of assets allocation
through Q-learning [13]. At about the same time, Moody et al. developed a policy
search RL-based system called Direct Reinforcement Learning (DRL) for trading a single asset. Also, they extended its application to the assets allocation problem [14]–[16].
Intelligibly, the adoption of RL into quantitative investment is attractive since the
2

controller can learn to trade amid interactions with financial markets.
Formulating the problem under the notion of RL composes of several main parts.
The first is the environment’s design that the agent interacts with for trial-and-error
experiential learning. The other part, i.e., the agent, takes actions following a policy based on the environment’s states1 . Lastly, we evaluate the policy pursued by the
agent through a “reinforcing signal,” or reward, that is received from the environment.
That signal provides feedback on the policy’s effectiveness in optimizing a predefined
performance measure. Following the DRL notion, the agent can learn a parametric policy without estimating value functions, unlike critic and actor-critic methods.
Instead, the policy can be represented by a function approximator that estimates
the optimal behavior of the agent concerning a predefined objective. As abundantly
discussed by Moody and Saffell [16], the paramount privilege of this method under
quantitative investment is that one can recursively update the policy at each time step
since the marginal impact on the accrued returns is measurable immediately after each
of the agent’s trading decisions.
We emphasize that most of the state-of-art RL algorithms are founded based on
maximizing the expectation of the agent’s cumulative reward, for example, Double
Deep Q-Nnetwork (DDQN) [17], and Deep Deterministic Policy Gradient (DDPG)
[18]. Although that risk-insensitive formulation would be practical under some applications, many real-life problems necessitate subtle risk management while making
decisions; indeed, quantitative investment is one application among others. There is
1 We will use the terms system and environment, controller and agent interchangeably in this

thesis.

3

afoot research related to RL to approach problems that require risk dimension during
the learning process, which researchers call it Safe RL [19]. Safe RL can introduce a
risk element during the learning course through designing a risk-sensitive performance
measure, for instance, the Markowitz approach of mean-variance optimization [2], [20],
[21], or the CVaR [22]. Another method is training a risk-constrained agent where
it learns to maximize the expectation while meeting a constraint on a specific risk
measure, for example, the reward variance or CVaR. Noteworthy, the mathematical
properties of performance measures play a crucial role in determining the quality of
the final obtained policy. That, in particular, motivates us to explore measures that
can introduce convexity into the problem. Specifically, we are interested in assessing
CVaR in the agent’s financial objective due to the computational attractiveness and
the mathematical guarantees one may achieve with that metric.
The pioneering work of Rockafellar and Uryasev was the first that introduced CVaR
optimization [23]. We can fundamentally define CVaR as the expected low-probable
but significant losses that may incur during the investment horizon. Mathematically,
CVaR is a statistical quantity that measures the expectation of large losses that fall
within the tail of the loss distribution. Due to the appealing mathematical properties
of CVaR, such as convexity [24], it has grabbed the attention of researchers in the field
of finance. In [25]–[27], the portfolio optimization problem was solved with CVaR being
an objective function or a constraint. Under leveraging RL with CVaR, Borkar and
Jain proposed an online Q-learning algorithm to solve CVaR-constrained RL problems
[28], whereas Chow et al. [29] exposed CVaR-constrained Policy Gradient (PG) and
actor-critic RL algorithms.
4

In quantitative investment with RL or machine learning in general, besides the
uncertainties of future price movement of financial assets, model parameters estimation
is an additional implicit risk since that estimate is probably inaccurate, especially
with the complex time-variant dynamics of the markets. Besides designing a riskaware agent to handle those risks, complex dynamics may necessitate unique learning
approaches for practical real-time trading. Although that RL can introduce online
learning, the trading system’s performance would likely worsen as time evolves if one
leaves updating the model after training, which is a challenge raised due to the nonstationary behavior of the markets. Data nonstationarity is sometimes called the
concept drift where it refers to the time-variation of the data distribution [30]–[32].
There are exposed learning algorithms that can handle the concept drift, such as
adaptive incremental learning [33]. Under that learning paradigm and quantitative
investment, the RL-based trading system can be updated in real-time based on the
most recent observation, or mini-batch past samples, eliminating the need for multiepoch training and validation.
In this thesis, we present an incremental-learner investing agent that follows the
DRL notion introduced in [16] considering CVaR as a primary element in the financial
performance measure. We specifically are interested in applying that system to two
quantitative investment problems: trading a single asset and the more general case of
assets allocation.
Before we move any further in our discussions, we need to emphasize that our
proposed trading system lies at the high level of the control hierarchy where we present
the Real Time Optimizer. In that sense, we mandate our system to optimize a financial
5

measure in which all subsystems, in practice2 , collaboratively operate to achieve that
objective.
Next, we endeavor to widen our and the reader’s insights by going back to the
historical aspect of RL and how it melted historically with optimal control theory.

1.1

Reinforcement Learning – Historical Aspect

The root of the RL concept is backed to 1948 when the so-called pleasure-pain system
was introduced by Turing in his report Intelligent Machinery [34]. In that report,
Turing discussed the potential of computer-based artificial intelligence inspired by the
biological theory of “reinforcing” animal behavior through a trial-and-error learning
approach. On the other side, the research in optimal control theory was progressing
extensively at that time. In the 1950s, Richard Bellman proposed a pioneering notion
in optimal control when he presented the utilization of system states dynamics and
control feedback signals to extract a control strategy such that a predefined function
is optimized [35]. Bellman also proposed solving this function, which is known as the
Bellman optimality equation, by Dynamic Programming (DP). At that time, computational intractability was imposing limitations in the use of DP, especially when
the number of state variables of a stochastic dynamical system is large, i.e., curse
of dimensionality. Another shortcoming in using DP for finding optimal behaviors
is that it requires complete knowledge about the system dynamics. These challenges
2 A real-time trading system requires several lower subsystems to achieve its objective.

For example, a cloud system is required to retrieve all the necessary data and a communication protocol
between the trading system, the cloud, and trading platforms. Obviously, the scope of this proposal
is limited to the design of the trading system only.

6

perhaps motivated melting the concept of learning with DP. Werbos first devised this
in 1977 in his efforts to develop advanced forecasting methods for coping with the
global crisis, and he, therefore, introduced the concept of heuristic DP that works as
an approximate approach for solving the problem [36]. About ten years later, Watkins
introduced a seminal work in RL, whose completely integrated DP and optimal control
with learning, and proposed the concept of model-free RL in his Ph.D. thesis [13]. In
that context, one does not need to know the dynamical behavior of a system to extract
an optimal controller’s policy, and it is instead learned through estimating the Bellman equation by experience. Since then, the research in RL has been extensive. That
led to various novel algorithms that grabbed the attention of other engineering fields
researchers, including quantitative investment. To this end, as an introduction, we
next conceptually discuss the various investment paradigms that a trader can follow
while investing in the financial markets.

1.2

Investment Paradigms

Investment in financial markets, generally speaking, can be categorized into three
main paradigms: trading a single asset, assets allocation, and portfolio management.
Transacting a single instrument is about buying and selling the asset repetitively
in a short period, usually with an intraday interval. When applying auto-trading
systems, one can also use High-Frequency Trading (HFT) strategies [37] where multitrade orders can be placed within seconds or even microseconds. From an engineering
perspective, the problem of assets allocation, where assets with various risk levels are
7

traded at different investment capital weights, extends the problem of transacting a
single asset. With that approach, one can consider re-balancing the investment capital
at each time interval among risky (e.g., public equity) or risk-less (e.g., fixed income)
assets. The reflected interval in assets allocation problems is usually significantly larger
than that of single-asset trading, where holding the assets may range from a week to
several months. For portfolio management problems, the investor can form a portfolio
consisting of an arbitrary number of assets where re-balancing the assets’ weights can
also occur at large time intervals. This thesis considers trading a single asset and
assets allocation; nonetheless, we extend our models to portfolio optimization but
leave experimenting with it for future work.

1.3

Thesis Statement

We have so far elucidated the necessary background to reveal the thesis statement:
This thesis endeavors to novelly address risk management of investing in
financial markets via quantitative methods, including methods from control
theory, finance, and machine learning.
We underscore that the statement will resurface in our discussions throughout the
thesis, for example, when discussing the trading controller’s design aspects, proposing
our CVaR models, experimenting these models with actual financial markets data, and
exposing the potential generalization of the proposed models. Next, with that statement in mind, we motivate our thesis by discussing the privileges of each quantitative
8

element in our approach and how melting those could lead to a convex-risk-aware
investing agent with efficient computational cost and the merit of mathematically
guaranteed convergence.

1.4

Motivation

The proposed methodology involves several inspirational concepts from control, finance, and machine learning communities in which implementing them may lead to
an online convex problem; those, in particular, are DRL, CVaR, and incremental learning.
RL, generally speaking, is an attractive tool to solve financial problems. That
is attributed to its ability to approximate the optimal policy of the controller under unknown dynamics, making it an attractive technique for systems with complex
dynamics like financial markets3 . Besides the motivation given in [16], [38], [39] for
selecting DRL rather than other RL approaches under financial trading systems, it
is noteworthy that DRL prompts the development of risk-aware agents. In a sense,
with DRL, one can allow flexible investment capital allocation since it conveniently
handles continuous control signals, opposite to critic RL, where it has discrete actions. Although actor-critic has continuous action space, it is more computationally
complex since it mandates approximating the agent’s policy and the value function,
which might be restrictive under our incremental learning approach requiring fast
computation.
3 We give general insights related to the investment problem when looking at it from control

perspectives in Chapter 3. That helps us to understand why RL is a suitable tool to solve such a
problem when we discuss its fundamental formulation in Chapter 4.

9

Our selection of CVaR is motivated by financial and mathematical aspects. Financially, searching for an agent’s policy based on CVaR prevents the agent from making
risky decisions that may cause significant losses while being aware of maximizing returns in case of imposing it as an agent’s constraint. Therefore, it learns a policy
that is aware of risk management under the severe uncertainties within the financial
market. From the mathematical aspect, under careful formulation using CVaR, we
can ensure convexity of the problem while keeping it general. The convex formulation
provides provision for finding an optimal global policy, thus, allowing for a fast online algorithm leading to faster convergence, which is indeed advantageous under our
proposed incremental learning approach. Adaptive control techniques and the fact
that financial markets exhibit time-variant dynamics inspired us to implement that
adaptive learning approach that aims to estimate the policy’s parameters frequently
based on the most recent observations and the agent’s financial performance.
To this end, we see that it is motivational to research the application of CVaR-based
RL through incremental learning under quantitative investment.

1.5

Contribution

Based on our literature review in Chapter 2, we can describe our contributions into
the field of developing investing systems under the notion of DRL as follows:
• We leverage DRL with CVaR that encourages the controller to learn an investment policy concerned about low probable but significant losses during
the trading course. We particularly introduce two CVaR-aware agents: CVaR10

constrained and unconstrained CVaR-sensitive agents.
• We provide sufficient conditions for the convexity of our model, which ensures superior computational performance that is compatible with incremental learning
and provides mathematical guarantees during searching for the optimal policy.
• We propose an online algorithm following the incremental learning technique
to actively discover and update a lucrative policy through a single run over
the data. In other words, our proposed system can learn without multi-epoch
training and validation.
• We experimentally assess the agents’ response to our models’ risk parameters
to disseminate a baseline for the expected agents’ investing behavior that can
serve future research. Our experiments also demonstrate superior performance
compared to state-of-the-art methods.
• We postulate clear future directions related to potential generalizations of our
approach and the open challenges we encountered during experimenting with
our trading agents.
We highlight that it seems convexity is of significant importance in the finance
literature. We noticed that the finance community has widespread research in formulating the investment problems, particularly portfolio optimization, under convex
setups. Since the work of Markowitz, several pioneering works considered convex formulation, see for example Cvitani et al. [40], Lobo et al. [41], Kraning et al. [42],
Moehle et al. [43], and Boyd et al. [44]. That is perhaps due to convexity attractive11

ness in terms of computational cost and the merit of guaranteed convergence. To this
end, we believe that our proposed convex trading system is foundational in bridging
that divide within the current research of RL-based trading systems when compared
to the common techniques proposed in the finance literature.

1.6

Outline

In Chapter 2, we initially discuss the literature related to investment with all modelfree RL methods, i.e., critic, actor, and actor-critic strategies. Although our research
focuses on DRL (i.e., actor methods), it is vital to review the practices and algorithms
implemented to solve the problem.
We then in Chapter 3 move to broadly discuss financial markets and the investment
process, which includes the investors’ objective of participating in those markets along
with the type of analysis and decisions they could take during the process. Chapter 3
also provides insights into how the dynamics of financial markets can be modeled with
stochastic differential equations. Accordingly, we discuss general aspects that need to
be considered to design a proper controller to meet the investment objective.
After that, we elucidate the RL notion and the stochastic dynamics of a Markov
Decision Process (MDP) in Chapter 4, including the distributions of states transitions
and the reward signal. Chapter 4 also discusses the different model-free RL methods,
including the RL frontier algorithms. Nonetheless, we give more focus on the DRL
since we specifically follow that model-free approach in this thesis.
We propose the CVaR-based models in Chapter 5 and their detailed formulation.
12

We initially frame the fundamental setup of optimizing those models, be CVaR as an
objective or a constraint in the agent’s investing policy. Then we discuss the models’
detailed design, including the policy and reward functions, considering the problems of
trading single asset and assets allocation. Based on the problem formulation, we state
our theorems related to the sufficient conditions for having convex trading systems.
Chapter 6 abundantly discusses the computational procedure for our proposed
models, including interactions of our agents within financial markets and learning the
optimal trading policy incrementally. We also expose how one would compute CVaR
adaptively to be compatible with our proposed incremental learning approach.
We carry out exhaustive numerical experiments on our agents considering solving
the problems of trading single asset and assets allocation in Chapter 7. We particularly
assess the agents’ sensitivity to the models’ risk parameters, including the level of risk
aversion and the constraint boundary. We do this under various financial markets
exhibiting different volatility levels to attain vaster insights. We also test the agents’
financial performance under conservative trading frictions to examine their endurance
against wild transaction costs. Moreover, we compare the performance of our proposed
agents against the foundational one of Moody and state-of-the-art Fuzzy Deep Direct
Reinforcement (FDDR) [39]. Finally, we experiment with the potential asset allocation
extension where the agent distributes its wealth between risky and risk-less assets.
In Chapter 8 we propose potential generalizations to our models, and we show
that they can be extended to solve the most general investment problem of portfolio
management. We also show our models’ flexibility in being externally connected with
learning systems that model the market’s volatility. In that sense, one can integrate
13

our models with such learning systems to establish adaptive risk-aversion agents.
Last but not least, we finally conclude our thesis in Chapter 9.

14

CHAPTER 2

LITERATURE REVIEW
The literature in quantitative investment with RL has been extended recently, particularly for the last decade with the emergence of deep learning. In fact, during our
review, we found that function approximation with deep learning is dominant under
all RL approaches. We also observed that risk-neutral performance measures are commonplace in critic and actor-critic methods, with more risk-aversion trading systems
proposed under actor-based methods. Further, the environment’s representation is
plethoric in using optimization algorithms that select the best states or market data
de-noising techniques. Meanwhile, few works focus on the reward function designs or
formulation of performance criteria.
Here in this chapter, we review the literature under the different RL approaches we
discuss later in Chapter 4. We dictate most of the discussion for actor-based methods
since it is one of the core elements in our proposed model. Nonetheless, we also discuss
other approaches, specifically the seminal works introducing recent RL algorithms in
quantitative trading. Our review is ample in which we deliberate different aspects of
the problem, considering mainly the MDP design and to which extent the formulation
15

is practical. Considering the MDP, we are concerned about financial markets states
representation, the action space, and the reward function design. Nevertheless, our
discussion focuses more on the performance criterion among all RL methods since that
represents a constitutional element in our proposed models.
In our review within the following sections, we combine critic and actor-critic
methods in one section, and we devote one separate section for actor-based RL with
a more comprehensive discussion.

2.1

Trading with Critic and Actor-Critic Methods

Under the critic approach with tabular methods, i.e., discrete state-action space, we
observed a limited number of works, most of which are in the early 2000s. That
perhaps is attributed to the complexity of financial markets, which requires a large
continuous state space for better representation. We, therefore, only sync up on some
of these works before we move to discuss the function approximation approach.
A summary of all reviewed papers that use RL with tabular methods is presented
in Table 2.11 . The work of Dempster et al. in [45] was instrumental in integrating
quantitative trading with critic RL through Q-learning algorithm. In their proposition,
the environment is represented by technical indicators2 as discrete states, and the
controller’s action follows a deterministic policy in a long-only3 setup. Moreover, the
performance design, i.e., the action-value function, is risk-neutral and governed by the
expected cumulative return. One significant observation in their numerical experiment
1 In Chapter 3, we discuss the market frictions and evaluation metrics stated in Table 2.1.

2 We discuss technical indicators as tools for investors to make trading decisions in Chapter 3.
3 The concepts of long and short trading are discussed in Chapter 3.

16

Table 2.1: Summary of quantitative trading with RL – Critic-tabular methods
Reference

Policy

RL
Deter.

Dempster
et al. [45]

Q-learning

Zhu et
al. [47]

Q-learning

Jangmin
et al. [49]

Q-learning

Yang et
al. [50]

Modelbased

Elder et
al. [46]
Chen et
al. [48]

Q-learning
SARSA

Value Function
Stoch.

RiskNeut.

Market Frictions

Assets

RiskAverse

Trans.

Spr.

Evaluation Metrics

Slip.

TP

SR

Sort.
R

MDD

*

*

*

*

*

*

*

*

*

Forex
[15m]

*

*

USETF
[days]

*

*

KSM
[1d]

*

*

USETF
[15m]

*

*

USI
[15m]

*

JSM
[1d]

*

*

*
*

*
*

*

*

*
*

*

*
*

*

Notes:
1.

US Exchange Traded Fund (USETF); Korea Stock Market (KSM); US Indexes (USI); Japan Stock Market (JSM)

2.

m = minute; d = day

3.

Deter. = Deterministic; Stoch. = Stochastic; Trans. = Transaction; Spr. = Spread; Slip. = Slippage; TP = Total Profit;
SR = Sharpe Ratio; Sort. R = Sortino Ratio; MDD= Max. Drawdown

is the high sensitivity of investment return to the applied transaction cost. Elder [46]
implemented Q-learning and proposed to represent the environment with the agent’s
internal states, for example, the agent’s current holdings and its achieved return.
Elder also proposed a risk-averse reward where it accounts for the expected return to
its movement. In Elder’s proposition, he did not include market frictions, although he
applied them while simulating the developed strategy. Zhu et al. in [47] presented Qlearning algorithm that trades multiple stocks from the Korean stock market with the
standard design of the reward function, which is the achieved profit by the trading
agent. Chen et al. [48] were different where they used the on-policy State-ActionReward-State-Action (SARSA) algorithm, and they applied their algorithm on Japan
stock market, where all transaction costs were neglected in their proposed model.
For comprehensive insights into the implementation of critic methods under function approximation RL within different markets, the interested reader may refer to
Table 2.2. One of the seminal works was introduced by Neuneier [12] where he ap-

17

Table 2.2: Summary of quantitative trading with RL – Critic methods
Reference

Policy

RL
Deter.

Value Function
Stoch.

RiskNeut.
*

Lee at
el. [51]

Q-learning
with NN

𝜀greedy

Bertoluzzo
and
Corazza
[52]

Q-learning
with linear
function

𝜀greedy

Eilers et
al. [53]

Q-learning
with NN

𝜀greedy

Cumming
[54]

LeastSquare
Temporal
Difference
[55]

𝜀greedy

Corazza
and
Sangalli
[56]

Q-learning
and
SARSA
with linear
function

𝜀greedy

Carapucco [57]

Q-learning
with NN

𝜀greedy

Huang
[58]

DQN with
LSTM

Gao [59]

DQN with
LSTM,
GRN,
CNN

Chen and
Gao [60]

Market Frictions

Assets

RiskAverse

Trans.
KSM
[1d]

Spr.

*

Evaluation Metrics

Slip.

TP

*

*

ISM
[1d]

*

*

USI,
GI
[1mo]

*

*

Forex
[1m]

*

*

ISM
[1d]

*

*

Forex
[2h]

*

*

*

Forex
[15m]

*

*

𝜀greedy

*

APS

*

DQN,
DQN with
LSTM

𝜀greedy

*

USETF
[1d]

*

Lucarelli
and
Borrotti
[61]

DDQN
and
Duelling
DDQN
with CNN

𝜀greedy

Sornmayura
[62]

DQN

𝜀greedy

*

Forex
[1d]

Tsantekidis et
al. [63]

DDQN
with
LSTM

𝜀greedy

*

Forex
[60m]

Carta et
al. [64]

Duelling
DDQN

𝜀greedy

*

Theate
and
Ernst [65]

DDQN

𝜀greedy

*

Crypto
[1m]

*

*

Sort.
R

MDD

*

*

*

*

*

SR

*
*

*

*

*

*

*

*

*

USSM
[60m]

*

*

VSM
[1d]

*

*

*

*

*

*

*

*

*

Notes:
1.

US Stock Market (USSM); Various Stock Markets (VSM) ;Korea Stock Market (KSM); Italy Stock Market (ISM); German
Indexes (GI); US Indexes (USI); US Exchange Traded Fund (USETF); Artificial Price Signal (APS)

2.

m = minute; h = hour; d = day

3.

Deter. = Deterministic; Stoch. = Stochastic; Trans. = Transaction; Spr. = Spread; Slip. = Slippage; TP = Total Profit;
SR = Sharpe Ratio; Sort. R = Sortino Ratio; MDD= Max. Drawdown

proximated the action values with Neural Network (NN) to manage assets allocation.
Neuneier also introduced a risk-adjusted performance measure to promote the agent
to learn a risk-aware policy. Cumming used a linear function to approximate the
state-value function and to learn the optimal policy simultaneously [54]. In his work,
he implemented the trading system on the Forex market, where his reward design considers the return resulting from the agent’s actions. Interestingly, the authors in [51]
18

Table 2.3: Summary of quantitative trading with RL – Actor-critic methods
Reference

Policy

RL
Deter.

Value Function

Market Frictions

Assets

RiskAverse

Evaluation Metrics

Stoch.

RiskNeut.

*

*

JSM
[5d]
*

*

*

Trans.

Spr.

Slip.

TP

SR

Mabu et
al. [66]

GNP

Bekiros
[67]

Adaptive
Fuzzy

*

*

VSM
[1d]

Li and
Shi [68]

On-policy
RDPG
with
LSTM

*

*

CSM

*

*

*

Ponomarev et
al. [69]

A3C with
LSTM

*

RSM
[1m]

*

*

*

Liu et
al. [70]

Off-policy
RDPG
with GRU

CSM
[1m]

*

*

*

Briola et
al. [71]

PPO with
NN

Notes:

*

*

*
*

USSM
[tick]

*

Sort.
R

MDD

*

*
*

*

*

1.

Japan Stock Market (JSM); Various Stock Markets (VSM); China Stock Market (CSM); Russia Stock Market (RSM); US
Stock Market (USSM)

2.

m = minute; d = day

3.

Deter. = Deterministic; Stoch. = Stochastic; Trans. = Transaction; Spr. = Spread; Slip. = Slippage; TP = Total Profit;
SR = Sharpe Ratio; Sort. R = Sortino Ratio; MDD= Max. Drawdown

proposed a multi-agent Q-learning trading system in which each is designed to handle
a specific task within the trading process. Specifically, two agents have the role of analyzing the market and recommending opening or closing a position, while the other
agents learn to execute the transactions at optimal timing. To advance the research
further under approximating value functions, Carapucco in [57] used Deep Q-Network
(DQN) to build a controller that trades within the Forex market as well where the
states representing the system are the ask and bid prices and the market volume.
As known for critic-based methods, Cumming and Carapucco used a discrete action
space that allows for short selling. Following the concept of a multi-agent trading
system proposed by [51], Carta et al. in [64] exposed the DQN architecture where the
final trading execution depends on the voting by several trading agents to enhance
the system robustness.
As summarized in Table 2.3, we noticed that most of the works that build trading systems with actor-critic methods use function approximation, particularly deep
19

learning architectures. Bekiros in [67] implemented fuzzy learning [72] to model the
environment and approximate the value and policy functions. Bekiros has a unique
design of the reward signal, where he defined it as the price signal prediction error.
Li and Shi [68] used on-policy Recurrent Deterministic Policy Gradient (RDPG) [73]
with Long Short-Term Memory (LSTM) as their function approximator [74]. They
first de-noised the raw price signal by using auto-encoders in their work. Then, it
is passed to the LSTM architecture to capture time-dependency within the filtered
price signal, where its outputs are the final states to represent the environment. The
goal of their agent is to maximize the total profit by using it as a reward, while
their action space is discretized and represented by the number of shares the agent is
willing to trade. Liu et al. [70] used the notion of differential Sharpe ratio [15] as a
risk-adjusted measure under applying Gated Recurrent Unit (GRU) [75] as a function
approximation in the RDPG algorithm, where they used the historical trajectories of
the MDP for the hidden unit. Further, Ponomarev [69] assessed the performance of
Asynchronous Actor-Critic (A3C) architecture with LSTM on the Russia stock market. We found the work of Zhang et al. [76] to be instrumental where they tested
their deep actor-critic trading system on a wide range of financial markets, including
commodities, equity indices, and Forex.

2.2

Trading with Actor-Based Methods

This section focuses on the performance measure since it is one of the core elements
in our contribution. We summarize our review of this method in Table 2.4.
20

Table 2.4: Summary of quantitative trading with RL – Actor methods
Reference

Policy

RL
Deter.

Moody et
al. [16]
Gold [77]
Dempster
et al. [38]
Bertoluzzo
et al. [78]
Maringer
et al. [79]

RRL
RRL
RRL
RRL
RRL

Performance Function
Stoch.

RiskNeut.

*
*
*
*
*

*

*

*

*

*

*

Forex
[1m]

*

*

*

VSM
[1d]

*

*

*

USSM
[1d]

*

*

USI
[1d,
1w,
1mo]

*

*

*

*

*

*

*

*

USSM
[1d]

Gabrielsson et
al. [83]

RRL

*

*

USIF
[1m]

Deng et
al. [84]

RRL

*

Lei et
al. [86]
Weng et
al. [87]
Sattarov
et al. [88]
Fengqian
et al. [89]

Deep CNN
Deep NN
Deep NN

*

*
*
*
*

*
*

*

*

*

*

*

*

CIF,
COMF
[1m]

*

*

*

*

*

*

Forex
[30m]

*

*

USSM
[1d]

*

Crypto
[30m]

*

Crypto
[60m]

*

*

COMF,
CFI
[1m]

*

*

*

*

*

*

*
*

*

1.

Various Stock Markets (VSM); US Stock Market (USSM); US Indexes (USI); US Indexes Futures (USIF); China Indexes
Futures (CFI), Commodities Futures (COMF); US Exchage Traded Fund (USETF)

2.

s = second; m = minute; d = day; w = week; mo = month

3.

Deter. = Deterministic; Stoch. = Stochastic; Trans. = Transaction; Spr. = Spread; Slip. = Slippage; TP = Total Profit;
SR = Sharpe Ratio; Sort. R = Sortino Ratio; MDD= Max. Drawdown

To the best of our knowledge, Moody and Saffel were the first who introduced the
concept of policy search RL into trading problems under the name of DRL [16]. Their
controller also has a recurrent structure, which refers to Recurrent Reinforcement
Learning (RRL), where the controller decision is influenced by the preceded one in
which the agent learns a path-dependent policy, hence reducing the turnover rate of its
transactions. They endeavored to design subtle performance measures that enhance
the controller’s risk awareness while trading. Their first measure is the Sharpe ratio,
which is defined as the mean of achieved excess returns over its standard deviation

21

MDD

*

CIF
[1/2s]

Notes:

Sort.
R

*

*

*

Deep GRU

SR

*

*

Lu [85]

TP

*

RRL

*

*

Slip.

Forex
[30m]

Zhang et
al. [81],
[82]

Deep
LSTM

Spr.

Forex
[30m]

RRL

Fuzzy
deep RRL

Trans.

Evaluation Metrics

*

Gorse [80]

Deng et
al. [39]

*

Market Frictions

Assets

RiskAverse

*

[90]. In that sense, the Sharpe ratio is a symmetric risk-performance measure since
it counts for the variance of both upside and downside returns. They, therefore,
developed an asymmetric risk measure that counts for the downside returns, which
they called the Downside Deviation ratio. Moody and Saffell implemented a discrete
action space rather than a continuous, although their problem formulation can handle
actions with a portion of the investment capital. Bertoluzzo and Corazza [78], on
the other side, explored a performance measure that maximizes the upside returns
while minimizing the downside risk. In that sense, the learned policy is eager to
maximize the positive returns while simultaneously avoiding decisions that may result
in losses. In [39], [84], total return and Sharpe ratio as the system’s performance
measures were implemented using a sparse-coding algorithm and deep fuzzy learning,
respectively. Almahdi and Yang [91] designed the performance function following
the concept of Calmar ratio, which is defined as the ratio of expected returns to the
expected Maximum Drawdown (MDD), i.e., the most extensive observed loss from a
top of wealth to a trough [92]. Noteworthy, total return and Sharpe ratio are the most
common performance measures among others, and they were extensively scouted with
different deep learning architectures [85]–[87], [93].
In light of that literature review, we believe that our work further advances the
research of trading systems with the DRL notion. We introduce CVaR as a constitutional element in the agent’s financial measure. That leads to a convex formulation
for the problem with the merits of computational efficiency and guaranteed policy
convergence, allowing for a competent implementation of incremental learning. As we
discussed earlier, apart from incremental learning, an additional inspiration to exploit
22

convexity is that the finance community has widespread research in formulating the
investment problems, particularly portfolio optimization, under convex setups.

23

CHAPTER 3

INVESTMENT IN FINANCIAL
MARKETS
Investment in financial markets is generally about trading financial assets, e.g., stocks,
bonds, currency pairs, or commodities, to maximize wealth while properly controlling
risk over time. Different investment approaches can lead to achieving that goal, yet,
all methods involve an iterative process of analyzing the financial market and making
decisions accordingly, that is, buying or selling assets at the right time. Noteworthy,
the described investment process, and other financial problems that require modeling,
can be engineered by using appropriate mathematical and computational tools, which
eventually led to the emergence of the quantitative finance field [94]. Nevertheless,
to approach those problems, one may develop mathematical insights related to the
dynamics of financial markets. Generally, the complex dynamics of financial markets
can be modeled through stochastic differential equations, and that is common among
quantitative finance researchers, as we further discuss in Section 3.4.

24

Investment returns

Objective
Feedback

Financial
Markets

Data

Analysis

Information

Decisions

Investor

Transactions
Figure 3.1: A general block diagram of the investment process in financial markets
In this chapter, we discuss several well-known financial performance measures. We
give particular focus on the CVaR concept with detailed mathematical formulation
since it is one of the core concepts in our research. We also discuss the two main steps
of the investment process, market analysis and decision making. After developing that
general understanding of the investment process, we discuss markets dynamics and
their general model using stochastic differential equations, which would reveal why
applying RL to solve the investment problem is an attractive approach.
Before we proceed to discuss the fundamental steps of the investment process in
the remainder of this chapter, we visualize this process in the block diagram depicted
in Figure 3.1.

3.1

Investment Objective

Under all paradigms we discussed in Section 1.2, the investor’s ultimate objective is
to optimize a financial performance measure that usually accounts for the potential
profit against any associated risks with that investment. Under the Modern Port25

folio Theory (MPT) proposed by Markowitz [95], the main objective is to maximize
the expectation of investment return while minimizing its variance, which he relates
the latter to an investment potential risks [2]. Since then, associating returns variance with investment risks has become common among researchers and practitioners.
For example, Sharpe introduced the idea of Sharpe ratio as a performance measure,
which is defined as the ratio of a risky asset’s excess return to its standard deviation [96]. That measure, however, is symmetric since it counts for both the upside
and downside returns variance as an investment risk. An investor is usually concerned
about the variance of the investment downside returns rather than the overall. Hence,
the concept of downside risk as a performance measure was introduced by Sortino
in [97]. After that, Rockafellar and Uryasev [98] introduced the notion CVaR that is
only concerned about the extreme downside returns instead of the overall as proposed
by Sortino. In the following, we further discuss the concept of CVaR since it is our
considered performance measure in this work.

3.1.1

Conditional Value-at-Risk

We are interested in utilizing the well-known risk measure CVaR due its appealing
mathematical properties, such as coherency and convexity [24]. At a probability
level 𝛾, CVaR𝛾 is the conditional mean of all losses that exceed the Value-at-Risk
(VaR), where VaR specifies the lowest loss 𝛽 that will not be exceeded at a confidence level 𝛾, and denoted by VaR𝛾 . In accordance, we have the following definitions.

26

Figure 3.2: A general probability density for a loss with its respective VaR𝛾 and CVaR𝛾
Definition 3.1 Considering a continuous random loss denoted by 𝑍 with a probability
distribution function Ψ 𝑍 (𝑧), then VaR at a probability level 𝛾 is given by

VaR𝛾 (𝑍) = min{𝛽 ∈ R : Ψ𝑍 (𝑧) ≥ 𝛾}

(3.1)

Definition 3.2 Considering a continuous random loss denoted by 𝑍 with a probability
density function 𝜓 𝑍 (𝑧), then CVaR at a probability level 𝛾 is given by:

1
CVaR𝛾 (𝑍) =
1−𝛾

∫
𝑧𝜓 𝑍 (𝑧)𝑑𝑧

(3.2)

𝑧≥VaR𝛾

Fig.3.2 illustrates a general example for 𝜓 𝑍 (𝑧) with its VaR𝛾 and CVaR𝛾 . We emphasize
that the confidence level 𝛾 also plays the role or risk-aversion parameter where the
27

controller is more risk-averse as 𝛾 → 1, and less risk-averse as 𝛾 → 0 with the special
case of CVaR𝛾=0 (𝑍) = E[𝑍], where E[·] is the expectation operator.
With insights, one can observe an additional privilege of CVaR𝛾 where the agent’s
risk-level can be tuned through 𝛾 externally, for example, by an auxiliary learning
system as we discuss further and propose for future work in Chapter 8.

3.2

Technical, Fundamental, and Macroeconomic
Variables Analysis

All investment paradigms involve an iterative process of analyzing the market and
making decisions. We note that the type of analysis differs depending on the followed
investment approach. For trading a single asset, the preferred type of analysis among
practitioners is the so-called technical analysis [99]; meanwhile, fundamental analysis
and macroeconomic variables may also be used for markets assessment in the case of
assets allocation and portfolio management.
In its simplest definition, technical analysis is about generating temporal signals
that are a function of the asset opening, low, high, and closing prices at a predefined
number of past time intervals. Those prices, within a single time interval, form the
well-known price candlestick shown in Figure 3.3 [100]. Then, technicians use these
temporal signals to form a prediction about the potential future movement of the
asset’s price and decide to either buy or sell the asset accordingly. One of the most
common technical indicators is the Simple Moving Average (SMA), representing the
closing price’s mean over predefined past time steps; hence, it can be considered as a
28

High
High

Close
Open

Open
Close
Low

Low

Figure 3.3: Two candlesticks with opening, low, high, and closing prices. When
the closing price of an interval is higher than its opening, the candle is green (left);
otherwise, it is red (right).
filtering technique for the noisy price signal. Also, the Relative Strength Index (RSI)
is commonplace among practitioners. The RSI is a bounded measure, and it indicates
that the traded asset might be oversold (respectively overbought), which suggests a
potential price increase (respectively decrease) over the near future. In Figure 3.4,
we illustrate a raw price signal along with its associated SMA. For other common
technical indicators and technical analysis approaches, we refer the reader to these
references [101]–[103].
Fundamental analysis is more or less associated with investing in stock markets
in particular. That type of analysis considers generating signals related to firms’
performances, such as changes in inventories, accounts receivables, and gross margins,
to predict their earnings and subsequently predict the stock prices’ future movement
[104], [105].
Macroeconomic variables are also temporal signals, but they represent the eco29

Price

Price
SMA

Time
Figure 3.4: An illustrative example for a price signal and its Simple Moving Average
(SMA) that is calculated based on the closing price of 14 past time steps.
nomic trend rather than the financial market’s [106]. When macroeconomic variables
are used for markets analysis, for example, consumer price index, money supply, and
long-term interest rates, one may consider assessing the variables that could influence
the asset’s returns. We refer to these studies in which they evaluate the relationship
of different macroeconomic variables on a variety of financial markets [107], [108].

3.3

Investment Decisions

After analyzing the market and forming a prediction about the asset’s price movement, the investor has to make trading decisions based on that prediction. Under any
investment approach, initiating a trade includes two possible decisions: to open long
or short positions. A long position is about buying the asset with predicting the price
to move higher over time. On the other hand, short selling is related to borrowing
the asset and selling it to a third party when predicting a decline in its price. Later
on, the asset is repurchased and returned to the lender in exchange for cash. In that
30

Short
Sell

Long
Cover

Figure 3.5: Two profitable trading scenarios. The green arrows indicate a profitable
round-trip trade of long-sell, whereas the blue arrows show the profitable short-cover
scenario .
sense, a short position is profitable if the asset’s price drops. We note that even closing
the initiated position requires thorough analysis to ensure optimal closing time. Intuitively, the trader sells the asset to close a long position, whereas the trader “covers”
its selling position. To exemplify all those trading decisions, from opening to closing,
we indicate two potential profitable scenarios for each round-trip trade (long-sell and
short-cover) in Figure 3.5. Note that each decision in the process involves transaction
costs, which are called market frictions. The clearest one is the commission a broker
charges the investor for transacting assets, usually a percentage of the traded amount.
The other charge is the spread cost that results from the difference between the bid
and ask prices of an asset. Another implicit cost is the slippage, where usually the
actual price at closing a position has a “slip” with the target closing price of a trader
due to delays in orders execution.
We emphasize that the decision process may also involve selecting the weight of
investment capital to trade with, as we consider in this thesis, especially in the case of
31

assets allocation or portfolio management where the investor can distribute the capital
among various assets. Further, the process could be even more complex in portfolio
management, where the investor can choose only a predefined number of assets out
of a universe of financial instruments, which is referred to Cardinality Constraint in
portfolio optimization problems [109].

3.4

The Investment Process: A Control Perspective

With the overview of the investment process and its objective discussed earlier, we
shall develop insights related to that process along with its dynamics and how it
can generally be linked with control theory and RL specifically. One can see that
the controller here takes the place of an investor as shown in Figure 3.1 while it
interacts with financial markets through placing trading orders. Considering that
general understanding, we next discuss the dynamics of financial markets and the
associated implications on the controller general design aspects we consider in this
thesis to address managing risks quantitatively while investing in financial markets.

3.4.1

Dynamics of Financial Markets

We stated earlier that the dynamics of financial markets have a nonstationary stochastic nature. We now discuss that behaviour under mathematical context.
It is common in the finance literature to represent the market dynamics using
stochastic differential equations. For instance, the problem of portfolio optimization
32

was approached under stochastic setup in [6], [7], and see here for financial modelling applications in general [110]–[112]. Accordingly, let us represent the state of our
system, i.e., financial markets, with a stochastic process denoted by 𝑥𝑡 . Before proceeding to model the dynamics of 𝑥𝑡 , let us state the following definition of a stationary
stochastic process as remarked by Åström [113]
Definition 3.3 Over a finite horizon of 𝑡1 , 𝑡 2 , . . . , 𝑡 𝑘 , if the distribution of
𝑥𝑡1 , 𝑥𝑡2 , . . . , 𝑥𝑡 𝑘 is identical to the distribution of 𝑥𝑡1 +𝜏 , 𝑥𝑡2 +𝜏 , . . . , 𝑥𝑡 𝑘 +𝜏 ∀𝜏 such that
𝑡 𝑗 , 𝑡 𝑗 + 𝜏 ∈ 𝑇 for 𝑗 = 1, 2, . . . , 𝑘 then the stochastic process 𝑥𝑡 is said to be stationary. If only the first and second moments of the distributions are the same, the
process is then weakly stationary.
In our case, a fine selection for the state 𝑥𝑡 is the financial asset’s price signal. Hence,
to understand the dynamics of the financial markets, one may consider modeling the
time-evolution of the price signal using the general structure of a stochastic differential
equation given by Åström in [113]

𝑑𝑥𝑡 = 𝜁 (𝑥𝑡 , 𝑡) + 𝜎(𝑥𝑡 , 𝑡)𝑑𝑤

(3.3)

The general model in (3.3) implies that the state evolution is governed by the deterministic state dynamics 𝜁 (·) and the stochastic function 𝜎(·) where 𝑤 denotes the
well-know Wiener process [114]. Since the Wiener process evolution is governed by a
normal distribution, this in turn leads to a normally distributed time-evolvement of
the state 𝑥𝑡 at an average determined by 𝜁 (·) and a variance of 𝜎(·) [113].
Remark 3.1 Based on Definition 3.3, if 𝜁 (·) and 𝜎(·) in (3.3) are time invariant,
33

then the stochastic process 𝑥𝑡 is stationary; otherwise, the process is nonstationary
[115].
Based on the discussion above and the fact that the price signal evolution represents
the investment return, i.e., 𝑑𝑥𝑡 ≈ 𝑟 𝑡 , we can derive insights related to the dynamics
of the financial markets. That is, as statistically investigated in [3], the return of
financial assets exhibit nonstationary distribution, which is also not necessary to be
Gaussian, and that is also proven in practice [116]. To this end, one can conclude that
the market dynamics are nonstationary stochastic.

3.4.2

Controller Design Aspects: Stability, Optimality, and
Adaptability

With the fundamental insights related to the dynamics of financial markets, we can
derive the required controller’s design aspects for addressing risks associated with investing in financial markets. Concerning the system’s stability with RL, we emphasize
that RL differs from classical optimal control in that stability of RL-optimal control
policy is still an open challenge [117]. We nonetheless design our investment controller
concerning optimizing a risk-based objective function while dropping any concerns related to the stability of the financial markets. That is a practical design consideration
under our case since we assume that the agent’s trading volume is much less than the
overall market, i.e., the control signal has a negligible impact on the price dynamics.
Nevertheless, finding an optimal and risk-aware controller under a stochastic dynamical system is tedious. Although that DP, i.e., model-based RL, can find a stochastic
34

optimal control policy for such systems, it may become more difficult and impractical
to implement it on nonstationary systems like financial markets. That is because DP
requires knowledge about the stochastic dynamics of the system, which in fact, would
evolve with time under nonstationary environments. To this end, we believe in the
value of model-free RL that learns an approximated optimal policy under complex
systems like financial markets without prior knowledge about the market dynamics.
Furthermore, the RL technique has an adaptive element, and in our work, we enhance
that design aspect by implementing the incremental learning approach. Over the next
chapter, we discuss the general formulation of RL theory considering the different
model-free approaches.

35

CHAPTER 4

REINFORCEMENT LEARNING
Reinforcement Learning is an experience-based and goal-oriented learning system that
aims to optimize an agent’s behavior within an environment, clearly consistent with
the objective of classical optimal control with the differentiation in the approach to
achieving that objective. With that, RL finds the optimal control signal through
a data-driven approach, rather than prior knowledge of the system’s dynamics as in
classical control. Indeed, that makes RL an attractive tool to be applied to investment
problems, as we discussed earlier. Before we proceed in discussing the different modelfree RL techniques, we need to stress that RL can be categorized as a machine learning
paradigm other than supervised and unsupervised learning approaches [1]. Trading
through supervised machine learning can be challenging since one trains it to predict
the price to minimize prediction error rather than maximize a financial performance
function [118]. To this end, one of the vital privileges that RL could introduce into
solving investment problems is its potential in performing the two main investment
steps, market analysis and making decisions, without explicitly predicting the future
price movement. More importantly, all of that may be formulated under a multi-stage
36

optimization framework to meet the long-term investment objectives.
This chapter deliberates the fundamental elements of the RL framework, i.e., the
environment and the agent and their interaction under the MDP. We then discuss
and provide the general formulations of the different model-free RL approaches encompassing actor-, critic-, and actor-critic-based methods.

4.1

The Markov Decision Process – The Environment and The Agent

The RL notion is fundamentally based on the concept of MDPs [35], [119]. An MDP
composes an environment and an agent where the interaction produces a tuple of
temporal trajectories. When these interactions are endless in time, we classify the
MDP as an infinite process. However, a finite MDP has a terminal state and is called
episodic MDP. While states describe the environment, the agent is represented by its
policy that maps those states into actions. In that sense, the agent selects an action
𝑎, from a set of actions A, based on the observed state 𝑥∈ X following a stochastic
policy denoted by 𝜋∼ 𝜓(𝑎|𝑥), where 𝜓(·|·) denotes a conditional probability density,
or a deterministic policy 𝜋(𝑥). The agent then receives a reward 𝑟 that represents a
“reinforcing signal” showing the action’s effectiveness at 𝑥, and then the agent observes
a new state 𝑥 0 as a subsequent of its action 𝑎 at 𝑥. An observed trajectory can represent
that whole sequence, denoted by 𝜏 = {𝑥, 𝑎, 𝑟, 𝑥 0 }. Those interactions between the agent
and the environment are illustrated in Figure 4.1.
The transition between states in the environment is stochastically modeled with a
37

x

r

a

r’
𝑥’

Figure 4.1: The agent-environment interaction in reinforcement learning [1].
probability density conditioned on the joint probability of 𝑥 and 𝑎, i.e., 𝑥 0 ∼ 𝜓(𝑥 0 |𝑥, 𝑎).
Like 𝑥 0, the observed reward can also be modeled as 𝑟 ∼ 𝜓(𝑟 |𝑥, 𝑎). Combining the
probability density of both 𝑥 0 and 𝑟 results in a joint probability density that represents
the environment’s one-step-ahead stochastic dynamics

𝜓 𝑋 0,𝑅 (𝑥 0, 𝑟 |𝑥, 𝑎)

(4.1)

We note here that the RL algorithm is model-based when the stochastic dynamics of
the environment described in (4.1) are known. With that, one can find an optimal
policy by solving the well-known Bellman equation using DP techniques. However,
we leave the discussion of obtaining optimal policies with DP under model-based RL
since it is out of the thesis scope; we instead refer the interested readers to related
discussions that can be found in [1], [35]. If it is complex to know the stochastic
dynamics of the MDP, like financial markets, one can solve the problem through
different model-free RL approaches. Before discussing those approaches, let us see the
standard formulation for finding optimal policies within a model-free MDP.

38

4.2

Optimal Value Functions and Policies

The main objective of the learning process, i.e., trial-and-error under model-free MDP,
is to find a policy 𝜋 that maps states into actions while simultaneously optimizes
an action-value function1 that counts for the policy’s performance measure, and we
denote it by 𝑄 𝜋 (𝑥, 𝑎). Let us consider a finite MDP with a length of 𝑇 and time steps
indicated by 𝑡, then 𝑄 𝜋 (𝑥, 𝑎) can be described by2

𝑄 𝜋 (𝑥, 𝑎) = E[𝑅𝑡 |𝑥 = 𝑥𝑡 , 𝑎 = 𝑎 𝑡 ]

(4.2)

With the cumulative reward 𝑅𝑡 being defined as

𝑅𝑡 =

𝑇−1
Õ

𝑗

𝛾𝑑 𝑟 𝑡+𝑘+1

(4.3)

𝑗=0

Where 𝛾𝑑 ∈ [0, 1] is a discount factor for each. We now need to emphasize several
notes. First, the action-value function governed by (4.2) and (4.3) indicate that the
state-action value is the expected discounted cumulative reward when taking action
𝑎 𝑡 at 𝑥𝑡 following the policy 𝜋 from that time step and onward. To determine how
farsighted the agent is, one can use the discount factor 𝛾𝑑 that tunes the observed
reward’s weights that contributes to the cumulative reward 𝑅𝑡 . In other words, under
the extreme cases of 𝛾𝑑 , at 𝛾𝑑 = 1 all rewards contribute equally into 𝑅𝑡 , while at
1 The value can also be a function of states only. That is precisely the case with model-based RL

where the optimal policy is derived using DP. However, with model-free RL, it is better to represent
the values with the state-action pairs, refer to [1].
2 In analogy, later on in Chapter 5 when we propose our models, we use 𝑇 and 𝑡 to denote the
investment horizon and time-step, respectively.

39

𝛾𝑑 = 0 the cumulative reward is only the next immediate reward, i.e., 𝑅𝑡 = 𝑟 𝑡+1 .
Finally, a policy is called optimal and denoted by 𝜋 ∗ if it is associated with an
optimal action-value function, that is

∗

𝑄 𝜋 = max 𝑄 𝜋 (𝑥, 𝑎);

∀𝑥 ∈ X, 𝑎 ∈ A

𝜋

(4.4)

Where the controller can reach 𝜋 ∗ through iterative experiential learning that can
be performed with different model-free RL approaches, which we discuss in the next
section.

4.3

Reinforcement Learning Approaches

As we discussed earlier, different RL approaches can lead to the same objective of
having an optimal policy, i.e., critic-, actor-, and actor-critic-based methods [120].
Indeed, each of these approaches has its strengths and weaknesses. Therefore, the
selected approach solely depends on the problem to be solved. One can refer back to
our motivation discussed in Section 1.4 for that particular selection of actor RL, i.e.,
DRL, under our application.
Next, we discuss how one can reach the optimal policy 𝜋 ∗ through iterative experiential learning under the different model-free RL approaches.

40

4.3.1

Critic-Based Methods

Following this approach, the action-value function under policy 𝜋 can be estimated
by either tabular or function approximation methods [1]. With tabular methods, each
state-action pair forms an entry to a table and is assigned a value. Clearly, with this
method, the MDP can only be represented by discrete state and action spaces. On
the other hand, function approximation methods allow for continuous state spaces
where a vector of parameters approximates the values. The optimal value function,
with tabular methods, can be obtained by SARSA or Q-learning algorithms. Both of
those approximate the solution of the Bellman equation by trial-and-error experience,
which their proof of convergence to an optimal policy is deliberated in [121] and [122],
respectively. Both algorithms are fundamentally based on the Temporal Difference
(TD) notion [123]. However, they differ in the learning approach where SARSA learns
with the so-called on-policy learning while Q-learning is off-policy. To this end, with
the SARSA algorithm, learning the value function depends on the pair value of the
currently taken action and the next observed state. That is, the agent learns an
optimal policy 𝜋 ∗ with its associated optimal value function 𝑄 𝜋 (𝑥, 𝑎) through an
∗

iterative one-step update rule given by



𝑄 𝜋 (𝑥𝑡 , 𝑎 𝑡 ) ← 𝑄 𝜋 (𝑥𝑡 , 𝑎 𝑡 ) + 𝛼𝑐 𝑟 𝑡+1 + 𝛾𝑑 𝑄 𝜋 (𝑥𝑡+1 , 𝑎 𝑡 ) − 𝑄 𝜋 (𝑥𝑡 , 𝑎 𝑡 )

(4.5)

Where 𝛼𝑐 ∈ (0, 1) is the critic learning step size. On the other hand, with Q-learning
that follows the off-policy approach, the action-value update is carried out considering
the action that would reveal maximum value, i.e., greedy, at the next observed state
41

regardless of the currently taken action; the update rule is therefore given by



𝑄 𝜋 (𝑥𝑡 , 𝑎 𝑡 ) ← 𝑄 𝜋 (𝑥𝑡 , 𝑎 𝑡 ) + 𝛼𝑐 𝑟 𝑡+1 + 𝛾𝑑 max 𝑄 𝜋 (𝑥𝑡+1 , 𝑎) − 𝑄 𝜋 (𝑥𝑡 , 𝑎 𝑡 )
𝑎

(4.6)

We underscore the dominance of Q-learning utilization over SARSA in the literature. That perhaps is attributed to the data efficiency of Q-learning when it comes to
practical applications that usually have less data complexity. In a sense, Q-learning
directly updates the value function based on the greedy action rather than the current one, leading to faster convergence. Nevertheless, under both algorithms, we note
that all state-action pairs shall be visited frequently to ensure continuous updates for
proper convergence [1]. Clearly, with that, the data and computational power required
to find the optimal policy and have a satisfactory generalization performance grows
significantly with the increase of state-action pairs, which leads to the well-known
chronic dilemma of curse of dimensionality. To this end, function approximation has
been introduced to improve the generalization performance in large state spaces since
the agent can approximate the value of non-visited state-action pairs to nearby ones
that were visited before. With a parameter vector Θ ∈ R𝜘 , one can approximate the
optimal value function as
∗

𝑄 𝜋 (𝑥, 𝑎; Θ) ≈ 𝑄 𝜋 (𝑥, 𝑎)

(4.7)

One can reach an approximation using linear functions or non-linear ones, similar to
the pioneering work of Mnih et al. [124] where they introduced the concept of DQN
by utilizing a deep NN as the function approximation method. Along with these
approximations, one can iteratively learn the parameter vector Θ by minimizing the
42

difference between an online and target networks in an off-policy learning approach,
i.e.,
𝐿 𝑖 (Θ𝑖 ) = E

h

2
𝑦𝑖𝐷𝑄𝑁 − 𝑄(𝑥, 𝑎; Θ𝑖 )

i

(4.8)

Where
𝑦𝑖𝐷𝑄𝑁 = 𝑟 𝑡+1 + 𝛾𝑑 max 𝑄(𝑥𝑡+1 , 𝑎; Θ− )
𝑎

𝐿 𝑖 and Θ𝑖 count for the loss and the online network parameter vector, respectively, at
the 𝑖 th iteration, while Θ− is the target network parameter vector. Then, as proposed
in [124], one can update the parameters in the direction that minimizes the loss in
(4.8) using Gradient Descent (GD) methods [125]

ΔΘ = −𝛼∇Θ𝑖 𝐿 𝑖 (Θ𝑖 )

(4.9)

Where 𝛼∈ (0, 1) is the gradient step size. To enhance the data efficiency for the
agent’s learning process, Mnih implemented the concept of memory replay [126]. In a
sense, the agent does not only learn from recent observations but also learns from old
experiences that are stored in a memory buffer and used as mini-batches to update
the online network. Moreover, Schaul et al. proposed an algorithm to enhance the
use of the memory replay further, and that is by using prioritized experiences along
the learning process (see [127]).
Hasslet observed that an overoptimistic value estimation is usually attained with
DQN [128]. He found out that is attributed to the fact that the same maximum
operator is used twice during action evaluation, as shown in (4.8), and executing
43

actions by the agent. Hasslet, as a consequence, proposed the concept of DDQN
in [17]. The DDQN algorithm is quite similar to DQN except for 𝑦𝑖𝐷𝑄𝑁 in (4.8) where
it becomes as follows in DDQN architecture

𝑦𝑖𝐷𝐷𝑄𝑁 = 𝑟 𝑡+1 + 𝛾𝑑 𝑄(𝑥𝑡+1 , max 𝑄(𝑥𝑡+1 , 𝑎; Θ𝑖 ); Θ− )
𝑎

Noteworthy, under either tabular or function approximation methods, 𝜀-greedy
policies are preferred when the agent’s environment is nonstationary [1], that is precisely the financial market dynamics behavior. With that exploratory attitude, the
agent takes greedy actions, i.e., 𝑎 𝑡 = max𝑎 𝑄 𝜋 (𝑥𝑡 , 𝑎), with probability 1 − 𝜀, meanwhile
it explores the financial market by taking random decisions with probability 𝜀. Thus,
selecting 𝜀 is crucial in having a fair balance between exploitative and exploratory
decisions. In fact, exploration versus exploitation is one of the main challenges underlying the practical implementation of RL [1].
We can notice that, although function approximations under critic methods can
circumvent the curse of dimensionality associated with tabular methods, it still cannot
handle continuous action spaces. On the other hand, one of the critical privileges of
actor-based methods is their potency in handling continuous MDPs, i.e., state and
action spaces, which we will discuss next.

4.3.2

Actor-Based Methods (Direct Reinforcement Learning)

The actor-based method is one of the constituent elements in our proposed model in
this thesis. These methods aim to directly learn a policy that optimizes an objective
44

function independently from action-value estimation. The learning process involves a
parametric policy whose parameters are updated to optimize the objective function.
We refer the reader to the pivotal work of Williams that originally introduced these
methods, see [129], [130]. Notwithstanding, here we discuss the formulation of the
actor-based approach following Moody and Saffell [16], and we call it DRL to be
consistent with the literature.
Under that approach, let
𝜋(𝑥; Θ)

(4.10)

denote a deterministic policy function that has Θ as its decision vector. One can
represent that function by a linear or any non-linear universal approximators, for
example, feed-forward NN, LSTM, or even Convolutional Neural Network (CNN)
[131]. Note that the policy in (4.10) is a function of the system’s states. Obviously,
with that formulation, the policy’s actions are based on the sensed state of the system.
When DRL is applied to trading problems, one can evaluate the action efficacy,
at each time step, by observing the achieved investment return due to that action.
Hence, we can define the reward with neglecting the discount factor, i.e., 𝛾𝑑 = 0
and consequently 𝑅𝑡 = 𝑟 𝑡+1 . Based on that, let us now define a general performance
measure that is a function of the received rewards within 𝑇, indicated by U𝑇 (𝑅); then
one can set up the problem as
max

U𝑇 (𝑅)

(4.11)

U𝑇 (𝑅)

(4.12)

𝜋(𝑥;Θ)

Or shortly
max
Θ

45

Where 𝑅∈ R𝑇 is the reward vector, and the objective is to find the decision vector Θ
that maximizes3 our performance measure U𝑇 (𝑅).
To improve the policy with respect to the performance criterion in (4.12), the
decision vector Θ can be updated in an offline manner. That is, after a full sweep on
all trajectories in an episode, one can apply gradient ascent as follows

ΔΘ = 𝛼∇Θ U𝑇 (𝑅)

(4.13)

However, due to the market nonstationarity, one may be interested in having an online
and adaptive algorithm where the parameters are updated recursively at each time
step 𝑡. That can be attained by redesigning the performance criterion to be a function
of the received rewards up to and including time 𝑡 + 1, i.e., U𝑡 (𝑅1 , 𝑅2 , ..., 𝑅𝑡+1 ). In that
sense, the policy is now a function of Θ𝑡 , denoted by 𝜋𝑡 (𝑥𝑡 ; Θ𝑡 ), and the parameter
vector can be updated through Stochastic Gradient (SG) methods [132]

ΔΘ = 𝛼∇Θ𝑡 U𝑡 (𝑅1 , 𝑅2 , ..., 𝑅𝑡+1 )

(4.14)

Where now ΔΘ = Θ𝑡+1 − Θ𝑡 . We need to point out that optimization with SG has
an attractive privilege. Since SG provides “noisy” gradient information, it enhances
the exploratory behavior of the agent. That behavior can be altered by tuning the
gradient step size 𝛼. Still, that online algorithm described in (4.14) may lead to a
computational burden, particularly true when considering a long investment horizon
𝑇 or even an infinite horizon. To enhance the computation efficiency, one may design
3 It also can be a minimization problem. That all depends on the defined performance measure.

46

a marginal performance criterion by assessing the impact of the most recent reward on
the performance function and then update the parameter vector accordingly, see for
example [15], [39]. Those works, however, implemented a rolling training/validation
window to update the entire model after predefined time steps. Instead, we implement the incremental learning technique where multi-epoch training and validation
are eliminated in our method.

4.3.3

Actor-Critic Methods

The actor-critic method is an approach that is between critic- and actor-based methods. Under that paradigm, the agent learns an estimate for the value function while
simultaneously follows a policy represented by a function. In that sense, the estimation error, i.e., the TD error, of the value function represents the critic part, whereas
the actor part is used to update the agent’s policy based on that error. The reader can
refer to Figure 4.2 that illustrates the learning process under this RL method. Following the fundamental formulation of this approach in [1], consider the action-value
in (4.6) under off-policy learning, then one can define the value estimation error as

𝑒 𝑡 = 𝑟 𝑡+1 + 𝛾𝑑 max 𝑄 𝜋 (𝑠𝑡+1 , 𝑎) − 𝑄 𝜋 (𝑠𝑡 , 𝑎 𝑡 )
𝑎

(4.15)

Also, considering a stochastic policy that, for example, is given by a Gibbs softmax
function
𝑒 𝑦(𝑥,𝑎)
𝜋𝑡 ∼ 𝜓(𝑎|𝑥) = Í 𝑦(𝑥,𝑏)
𝑏𝑒

47

(4.16)

Figure 4.2: The RL actor-critic architecture [1].
Then one can update the value and policy functions independently

𝑄 𝜋 (𝑥𝑡 , 𝑎 𝑡 ) ← 𝑄 𝜋 (𝑥𝑡 , 𝑎 𝑡 ) + 𝛼𝑐 𝛿𝑡

(4.17)

𝑦(𝑥, 𝑎) ← 𝑦(𝑥, 𝑎) + 𝛼𝑎 𝑒 𝑡
Where 𝛼𝑎 ∈ (0, 1) is the learning rate for the actor part, and 𝑦(𝑥, 𝑎) are modifiable
values of the actor. With that, the tendency of taking 𝑎 at 𝑥𝑡 increases or decreases
depending on the value of the error 𝑒 𝑡 . In critic-based methods, besides their limited
applications to discrete action-space problems, a small perturbation in the value estimation results in a significant change in the followed policy, which imposes serious
convergent intricacies in many practical applications. The actor-critic algorithm can
improve that unstable behavior by approximating the policy with a function whose parameters are updated with SG methods to improve the performance measure, i.e., the
estimation of the value function. These methods have been extended to function ap48

proximation where separate parameter vectors approximate the critic and actor parts
independently; for example, see the work of Konda and Tsitsiklis [120]. Further, Sutton et al. [133] proposed an actor-critic approach under function approximation that
is based on stochastic policy gradient. Recently, Silver et al. [134] proposed the Deterministic Policy Gradient (DPG) algorithm that is significantly more sample-efficient
than its stochastic counter algorithm, especially in problems with large continuous
action spaces. Gradient computation in DPG only requires integration over the state
space, whereas the stochastic algorithm necessitates integration over both state and
action space. Note here that even policy gradient algorithms do have on-policy and
off-policy learning, and the details of those can be found here [134]. Let us consider,
under on-policy learning, a performance measure associated with a parametric policy

𝜋(𝑥; Θ), denoted by 𝐽 𝜋(𝑥; Θ) , that counts for the reward expectation, as the right
hand side of (4.2), then the policy gradient with respect to Θ considering a stochastic
policy can be given by [134]




∇Θ 𝐽 𝜋(𝑥; Θ) = E𝑥 0∼𝜓(𝑥 0 |𝑥,𝑎),𝑎∼𝜓(𝑥|𝑎) ∇Θ log 𝜓(𝑎|𝑥; Θ)𝑄 𝜋 (𝑥, 𝑎)

(4.18)

While if one considers a deterministic policy, then the gradient would be




∇Θ 𝐽 𝜋(𝑥; Θ) = E𝑥 0∼𝜓(𝑥 0 |𝑥,𝑎) ∇Θ 𝜋(𝑥; Θ)∇𝑎 𝑄 𝜋 (𝑥, 𝑎)| 𝑎=𝜋

(4.19)

While the critic can be updated through (4.9), one can then update the policy’s
parameter Θ in the ascent direction of the performance measure using (4.18) and

49

(4.19) for stochastic and deterministic policies, respectively. Note that for computing
the gradients in (4.18) and (4.19) one would need to estimate the value functions,
similar to what we described in (4.7)4 .
Noteworthy, various advancements related to actor-critic were exposed in the literature since the work of Silver, mainly inspired by the successful enhancements implemented on DQN. For instance, Lillicrap et al. [18] extended the concepts of memory
replay and target networks to the DPG framework, which he called DDPG. Moreover,
Mnih et al. [135] proposed different asynchronous deep RL algorithms, and the most
remarkable performance was found in the asynchronous actor-critic, which they called
A3C. The A3C is an on-policy learning method that, to some extent, has a similar
impact to memory replay on learning from data. During the learning process in A3C,
instead of memory replay, A3C asynchronously executes multiple agents in parallel on
various states of each. In that sense, the de-correlation of samples is evident, similar
to the impact of the memory replay concept while enhancing the use of data. Further,
DDPG is similar to DQN in the overestimation of the value function since the evaluation and execution of actions are based on the greedy policy. Therefore, Fujimoto et
al. [136] overcome this challenge by implementing double DDPG, which they called
Twin Delayed Deep Deterministic Reinforcement Learning (TD3). The most current
state-of-art actor-critic method under function approximation is the Proximal Policy
Optimization (PPO). Instead of learning from one sample at each gradient step in
DPG, PPO offers a novel performance function that allows multi-epoch of mini-batch
4 The critic and actor parts are usually estimated with different parameter vectors.

the same notation for the two for brevity purposes only.

50

We here use

updates; it thus has better empirical sample complexity [137]. Moreover, Haarnoja
et al. [138] proposed a Soft Actor-Critic (SAC) algorithm that deploys the concept
of maximum entropy RL [139] under the actor-critic framework. That was shown to
boost data efficiency during the learning process due to enhancing the exploratory
behavior of the agent. At the same time, it also uses target networks and memory
replay over the learning process.
It is now clear how the formulation of most state-of-art RL algorithms is founded
based on maximizing the expectation of the cumulative reward, which makes them
risk-insensitive. In line with the current research direction of Safe RL [19], we propose
CVaR-based RL to help the investing agent learn risk management during interaction
with financial markets, especially under the severe uncertainties they exhibit, as we
discussed earlier. In the next chapter, we propose our agents, specifically, the CVaRconstrained and CVaR-sensitive agents considering the two investment problems of
trading single asset and assets allocation.

51

CHAPTER 5

CVAR-BASED INVESTING
AGENTS
Over this chapter, we abundantly discuss and formulate our proposed agents that
we see as the primary contribution of this thesis. Throughout our formulation, we
endeavor to design agents that can inherently manage risk while investing. With
that purpose in mind, we initially explicate the proposed model by elucidating the
necessary preliminaries. We then discuss the models and their formulation, including
the policy and reward designs while embedding trading frictions. We also postulate
our theorems related to our models’ convexity under the two investment problems of
trading single asset and assets allocation.
Let 𝑝 0 , 𝑝 1 , ...,𝑝 𝑡 denote the price quotes of the financial asset, which are released by
the exchange centers. We let the financial market represent the agent’s environment,
where its states are functions of the past 𝑛 price samples, and are denoted by the
vector 𝑥𝑡 := ℎ𝑡 ( 𝑝 𝑡 , 𝑝 𝑡−1 , ..., 𝑝 𝑡−𝑛 ) ∈ R𝜅+1 , where ℎ𝑡 is continuous function for each 𝑡. In

52

that sense, we represent the states with technical indicators [99], or the well-known
price rate of change used in quantitative investment literature [3], [140]–[143]. When
the agent senses the environment’s state 𝑥𝑡 , it takes action following a time-variant
deterministic policy denoted by 𝜋𝑡 ∈ [−1, 1], and any 𝜋𝑡 > 0 indicates a long position
while 𝜋𝑡 < 0 indicates a short position. Based on the action 𝜋𝑡 at a state 𝑥𝑡 , the agent
receives a reward signal from the environment that represents the achieved return at
the immediate next time step1 , denoted by 𝑅𝑡+1 . The investment process of sensing
the environment state, taking action, and receiving a reward signal repeats at each
time step for 𝑡 = 0, 1, 2, 3, ..., 𝑇 where 𝑇 is the investment horizon.
Considering an investment’s capital (or wealth) 𝑊0 ∈ R+ at the initial time 𝑡 = 0,
the agent’s main objective is to learn a policy that optimizes a financial measure constrained on a risk metric. Those are functions of the agent’s initial capital, the rewards,
and the losses over the investment horizon, denoted by U𝑇 (𝑊0 , 𝑅) and B𝑇 (𝑊0 ,𝐿), respectively, where 𝑅, 𝐿 ∈ R𝑇 are vectors representing received rewards and losses over
the entire horizon. In our research, we are interested in introducing the well-known
risk measure CVaR as a main element in the financial performance measure. Specifically, we are interested in assessing unconstrained CVaR-sensitive agent and the case
of CVaR-constrained agent, which we formulate their objective functions next.
1 Under our notation, the reward also represents the change of wealth at each time step.

53

5.1

Agents’ Models

Considering the optimization of a general financial objective constrained on a risk
metric over the investment horizon of 𝑡 = 0, 1, 2, ..., 𝑇, one could have

max

U𝑇 (𝑊0 , 𝑅)

s.t.

B𝑇 (𝑊0 , 𝐿) ≤ 𝜇 𝛽

(5.1)

Where 𝜇 𝛽 is a strictly positive constant. Our CVaR-based models along with a riskneutral agent2 are special cases of the general model stated in (5.1), which are all
summarized in Table 5.1.
For what we call the CVaR-constrained agent, we consider maximizing the expectation of the reward constrained on CVaR of the loss 𝐿 = −𝑅 at a confidence level 𝛾,
i.e., CVaR𝛾 [−𝑊0 𝑅]. A parametric policy altogether controls that with a decision vector Θ ∈ R𝜅+1 where 𝜅 represents the dimension of the system’s state. Thus, Problem
(5.1) becomes
max

E[𝑊0 𝑅(Θ)]

s.t.

CVaR𝛾 [−𝑊0 𝑅(Θ)] ≤ 𝜇 𝛽

Θ

(5.2)

Where 𝜇 𝛽 in Problem (5.2) represents the investor’s tolerable expected loss that may
exceed VaR𝛾 within the investment horizon 𝑇.
For the design of what we call the CVaR-sensitive agent, it is equivalent to consid2 We only consider the risk-neutral model as one of the benchmarks when we assess our CVaR-based

models in Chapter 7.

54

Table 5.1: Summary of CVaR-based and risk-neutral agents’ models
CVaR-Constrained

max
Θ

s.t.

CVaR-Sensitive

E[𝑊0 𝑅(Θ)]

min
Θ

CVaR𝛾 [−𝑊0 𝑅(Θ)] ≤ 𝜇 𝛽

CVaR𝛾 [−𝑊0 𝑅(Θ)]

Risk-Neutral

max
Θ

E[𝑊0 𝑅(Θ)]

ering U𝑇 as CVaR𝛾 and 𝜇 𝛽 → ∞ in (5.1); thus our risk-sensitive model is

min
Θ

CVaR𝛾 [−𝑊0 𝑅(Θ)]

(5.3)

It is worth noting that for the special case when 𝛾 = 0 in (5.3), the agent is risk-neutral,
and Problem (5.3) becomes equivalent to

max
Θ

E[𝑊0 𝑅(Θ)]

(5.4)

By the discussion in [144], we state the following Lemma.
Lemma 5.1 If 𝑅(Θ) is concave, then, Problems (5.2) and (5.3) are convex.
Using CVaR, we have a risk-aware agent that takes into account generating profits and prevents losses during the investment horizon 𝑇. As discussed previously, we
generally consider a single-asset trader assuming negligible temporary and permanent
market impacts of the trader’s positions on the asset’s price, i.e., trading volume is
significantly less than the overall market. We also consider the assets allocation problem where the agent re-balances the capital among risky assets (e.g., public equity)
and risk-free assets (e.g., fixed income) at each time step. Our goal is to search for
55

Figure 5.1: Proposed trading system’s block diagram. Solid signals represent sensing
the environment and taking action during [𝑡, 𝑡 + 1). Dashed signals represent receiving
a reward at 𝑡 + 1
a decision vector Θ that solves the risk-aware agent Problems (5.2) and (5.3) under
both investment cases we described.
In Figure 5.1, we visualize the sequence of decisions and flow of information. At
each time step 𝑡, the agent senses the environment through 𝑥𝑡 , generates a trading
signal following the deterministic policy 𝜋𝑡 , and receives a reinforcing signal 𝑅𝑡+1 at
the immediate next time step. Over the following sections, we elucidate our design
for the policy and two reward signals considering the cases of single asset and assets
allocation.

56

5.2

Trading Policy

For the time-variant policy 𝜋𝑡 that is followed by our agent, we model it by a single-unit
neural network that is defined by

𝜋𝑡 = 𝑓 [𝑔𝑡 (𝑥𝑡 ; Θ)]

(5.5)

𝑔𝑡 (𝑥𝑡 ; Θ) = ΘT 𝑥𝑡

(5.6)

Where

𝑓 (·) is now an activation function of a neural network’s neuron,

𝑥𝑡 = [𝑥 1,𝑡

𝑥 2,𝑡

···

𝑥 𝜅,𝑡

1] ∈ R𝜅+1

represents the system with 𝜅 states at time 𝑡, and

Θ = [𝜃 1

𝜃2

···

𝜃𝜅

𝜃 𝜅+1 ]

is the decision vector where 𝜃 𝜅+1 is the bias parameter. We note that the policy in
(5.5) follows the foundational design of Moody and Safell in [16]. Besides that, as
we show later, we preferentially select that simplified function approximator to have
a thorough mathematical analysis for the model. We accordingly can find sufficient
conditions for a convex system with a fast and guaranteed convergence that can be
leveraged with incremental learning.

57

5.3

Reward Signal

As we discussed earlier, in general, we model the reward with the next immediate
observed return by the agent. In accordance, we design different rewards for the cases
of trading an asset and assets allocation since, in the latter case, the agent would also
realize a risk-free return when investing in a fixed income asset.

5.3.1

Single-Asset

We follow the reward design of Moody [14] under the case of trading a single asset.
Thus, the reward signal at each step can be described by

𝑅𝑡+1 = 𝜋𝑡 𝑟 𝑡+1

(5.7)

Where in our particular application 𝑟 𝑡+1 = 𝑝 𝑡+1 /𝑝 𝑡 − 1 is the normalized price return of
an asset. With the structure in (5.7), and the fact that we have a continuous policy
𝜋𝑡 allowing short sells, the agent is rewarded based on its trade size and the asset
price movement. Although that reward design does not reflect any trading frictions,
we take a distinct approach to model it for mathematical reasons we discuss later in
Section 5.3.3. Following the motivative argument of Moody and Safell [16] related to
immediate against future rewards under financial systems, we neglect the discount rate
𝛾𝑑 and hence consider the direct reward case. From the reward (5.7) and the policy

58

in (5.5), we can describe 𝑅𝑡+1 explicitly as a function of Θ and the asset’s return as

𝑅𝑡+1 (Θ) = 𝑓 [𝑔𝑡 (𝑥𝑡 ; Θ)]𝑟 𝑡+1

(5.8)

The following result, whose proof can be found in the Appendix, provides sufficient
conditions for the convexity of Problems (5.2) and (5.3) under the reward in (5.7) as
long as trading a single asset is concerned.
Theorem 5.1 Problems (5.2) and (5.3) are convex in Θ if, for each 𝑡, one of the
following conditions hold:
• 𝑟 𝑡+1 ≥ 0 and 𝑓 is concave.
• 𝑟 𝑡+1 ≤ 0 and 𝑓 is convex.

5.3.2

Assets Allocation

Under assets allocation, we shall slightly modify the reward signal in (5.7). In this
case, the agent decides the proportion of the capital or cumulative wealth invested in
the risky asset while investing the remaining in the risk-free instrument. We thus can
redesign the reward signal as

𝑓

𝑅𝑡+1 = 𝜋𝑡 𝑟 𝑡+1 + (1 − 𝜋𝑡 )𝑟 𝑡+1

(5.9)

Where 𝑟 𝑡+1 and 𝑟 𝑓 𝑡+1 represent the risky and risk-less asset normalized returns, respectively. Noteworthy, the expression in (5.9) is valid under long-only portfolios. To
59

this end, for the assets allocation problem, we restrict the policy range, i.e., 𝜋𝑡 ∈ [0, 1]
where the outermost value of 1 indicates the whole capital is invested in risky assets,
whereas 0 implies the agent fully invests in the risk-free instrument. Again, following
the reward in (5.9) and the policy in (5.5), one can formulate 𝑅𝑡+1 explicitly as a
function of the decision vector Θ and the assets’ returns

𝑓

𝑓

𝑅𝑡+1 (Θ) = 𝑓 [𝑔𝑡 (𝑥𝑡 ; Θ)] (𝑟 𝑡+1 − 𝑟 𝑡+1 ) + 𝑟 𝑡+1

(5.10)

We next state our theorem concerning the case of assets allocation under the reward
design in (5.9) whose proof is exposed in the Appendix.
Theorem 5.2 With the reward function in (5.9) in a long-only setup, Problems (5.2)
and (5.3) are convex in Θ if, for each 𝑡, one of the following conditions is true:
• 𝑟 𝑡+1 − 𝑟 𝑡+1 ≥ 0 and 𝑓 is concave.
𝑓

• 𝑟 𝑡+1 − 𝑟 𝑡+1 ≤ 0 and 𝑓 is convex.
𝑓

Remark 5.1 Theorem 5.1 and Theorem 5.2 elucidate that the convexity of the problem
depends on the asset price returns and the convexity of the selected activation. In
particular, to ensure that one of the conditions hold at each 𝑡, one must have an
activation that can be either concave or convex, depending on the sign of 𝑟 𝑡+1 or
𝑟 𝑡+1 − 𝑟 𝑡+1 under assets allocation. The linear activation is an obvious selection that
𝑓

achieves those requirements and is suitable for trading.
Remark 5.2 Our definition of the reward forms the basis of a “batch” algorithm
where the decision vector can only be updated after a full sweep over the data within
60

the investment horizon 𝑇. Similar to Moody and Saffell in [16], we do online parameter updates for consistency with real finance applications. Thus, to denote time
dependency, we henceforth subscript the decision vector with 𝑡, i.e., Θ𝑡 .
Remark 5.3 To improve handling markets uncertainties, we have an adaptive investing agent that learns incrementally to robustly detect financial market regime-switching
instead of the rolling training/validation window proposed by Moody [16] and Deng et
al. [39]. Incremental learning is made possible with efficient cost through our convex
setup.
Noteworthy, so far, our formulation of the reward function considers the ideal case
in which all associated trading frictions, e.g., transaction costs, are neglected. Incorporating such factors into the model requires perturbations of the reward function.
Next, we discuss adding those frictions into the reward signal.

5.3.3

Including Financial Markets Friction

In the finance literature, trading frictions, such as transaction costs, can be incorporated into the problem in several ways. For example, Davis and Norman in [7]
modeled the transaction charges as a linear function of the transacted assets amount
in a portfolio selection problem. Morton and Pliska in [145], on the other hand, implemented a fixed cost that depends on the portfolio value. Here we follow the approach
in [7] since it is consistent with the pioneering DRL trading systems [16], [39]. To this
end, one can model the Transaction Cost (𝑇𝐶) by adding the following term to either

61

reward in (5.7) or (5.9), depending on the trading system application

𝑇𝐶 = −𝛿|𝜋𝑡 − 𝜋𝑡−1 |

(5.11)

Where 𝛿 is a fixed percentage that counts for half of the round-trip cost (only opening
or closing a position), the argument inside the absolute value represents the traded
amount of the asset at time 𝑡.
Remark 5.4 Incorporating the 𝑇𝐶 within the rewards in (5.7) and (5.9) harms the differentiability of our optimization Problems due the absolute value in (5.11). Nonetheless, with the use of linear activation, the argument within the absolute value is linear
in Θ𝑡 . Hence, the transaction costs can be modeled by reformulating Problems (5.2)
and (5.3) via additional linear constraints and an auxiliary variable.
Having our models proposed in this chapter, we next discuss the computational
aspects of these models and how one would compute an approximate optimal trading
policy under our formulation.

62

CHAPTER 6

COMPUTATION OF THE
APPROXIMATE OPTIMAL
POLICY
The computational procedure discussed in this Chapter can be applied to both investment cases under consideration, uni-trading and assets allocation; it only depends on
the reward design for each investment case. To properly handle uncertainties, we follow the incremental learning approach [33] where the policy decision vector is updated
at each time step with eliminating multi-epoch training and validation.
Here, for the states, we model the market environment by technical indicators,
namely the lag asset’s returns. To this end, we consider representing it by a series
of past price returns, i.e., 𝑥𝑡 = [𝑟 𝑡 , 𝑟 𝑡−1 , . . . 𝑟 𝑡−(𝜅−1) , 1] T for single asset trading and
𝑥𝑡 = [𝑟 𝑡 , 𝑟 𝑡 , 𝑟 𝑡−1 , 𝑟 𝑡−1 . . . 𝑟 𝑡−(𝜅/2−1) , 𝑟 𝑡−(𝜅/2−1) , 1] T for assets allocation where in the latter
𝑓

𝑓

𝑓

case 𝜅 is any 2 𝑗∀ 𝑗 ∈ N+ .

63

As discussed, convexity is guaranteed with linear activation while not with the
hyperbolic tangent and sigmoid functions, making the system susceptible to initialization. Thus, for the policy activation 𝑓 (·), we preferentially use a bounded linear
function to represent the allowable range of the trading signal, i.e., [−1, 1] for trading
a single asset and [0, 1] for the case of assets allocation. Furthermore, as we discussed
in Chapter 5, this allows us to consider trading frictions while preserving differentiability. We use a unity investment capital 𝑊0 = 1; hence, the received reward signal by
the agent is the normalized return of the traded asset scaled by the trading signal.
With that, the final wealth at the end of the investment horizon 𝑇 is 𝑊𝑇 = 𝑊0 (1 + 𝑅 T 1)
where 1 is unity vector with proper dimension.
We aim to learn the optimal policy in an online manner where the decision 𝜋𝑡
is immediately evaluated when receiving 𝑅𝑡+1 . We next show how to compute the
objective in Problems (5.2) and (5.3) incrementally at each time step and how we
recursively learn the optimal policy. We use the subscript 𝑖 to represent our iterative
computation and distinguish it from the market time step 𝑡.

6.1

CVaR-Constrained Agent

Our convex setup for the agent gives privilege in computing the approximate optimal
policy while convergence is guaranteed. Let us consider the expectation of the observed
rewards up to 𝑡 + 1, i.e., E𝑡+1 [𝑊0 𝑅], where now 𝑅 ∈ R𝑡+1 , with general constraints
denoted by 𝐺 𝑙 for 𝑙 = 1, 2, 3 to count for the convex risk constraint along with modeling
the transaction cost as linear constraints. Then we can unfold our Problem (5.2) using
64

the well-known log barrier method [146] as follows
3

max
Θ𝑡 ,𝜗𝑡


1Õ
E𝑡+1 [𝑊0 𝑅] +
log 𝐺 𝑙
𝜂 𝑙=1

(6.1)

Where we have
𝐺 1 = CVaR𝛾,𝑡+1 [−𝑊0 𝑅(Θ𝑡 )] − 𝜇 𝛽
𝐺 2 = 𝜋𝑡 − 𝜋𝑡−1 − 𝜗𝑡

(6.2)

𝐺 3 = −𝜋𝑡 + 𝜋𝑡−1 − 𝜗𝑡
Where 𝜂∈ R+ is a parameter that determines the accuracy of the approximation to the
original Problem (5.2), and it should grow with the steps of computing the solution,
i.e., 1/𝜂 → 0 as 𝑖 → ∞. The auxiliary variable 𝜗𝑡 ∈ R models the absolute value
argument in (5.11). The unfolded objective described in (6.1) mandates computing
CVaR𝛾 to solve it. Since the reward samples arrive sequentially during the investment
process, we can estimate CVaR𝛾 at each time step using all captured rewards up to
𝑡 + 1, i.e., 𝑅1 , 𝑅2 , ..., 𝑅𝑡+1 and we denote it by CVaR𝛾,𝑡+1 as in (6.2). Our estimation
follows the Monte Carlo approach proposed by Hong and Liu in [147], which proposed
the following convergent Monte Carlo estimation of CVaR𝛾,𝑡+1 [−𝑊0 𝑅(Θ𝑡 )], denoted by
ĉ𝛾,𝑖 (we drop the constant 𝑊0 for brevity)

ĉ𝛾,𝑖 = v̂𝛾,𝑖 +

𝑡+1
Õ
1
[−𝑅 𝑗 − v̂𝛾,𝑖 ] +
(𝑡 + 1)(1 − 𝛾) 𝑗=1

65

(6.3)

Where [𝑦] + = max(𝑦, 0) and v̂𝛾,𝑖 is the estimation of VaR𝛾,𝑡+1 [−𝑊0 𝑅(Θ𝑡 )], which is
given by
(6.4)

v̂𝛾,𝑖 = −𝑅 d(𝑡+1)𝛾e:𝑡+1

Where −𝑅 d𝑘:(𝑡+1)e is the 𝑘 th order statistic from the (𝑡+1) samples. Once the estimation
in (6.3) is computed, we can directly apply stochastic gradient ascent [132] to the
augmeneted objective in (6.1) and update the policy’s parameters Θ𝑖 in its ascent
direction1
3



1Õ
Θ𝑖+1 = Θ𝑖 + 𝛼∇Θ𝑖 E𝑡+1 [𝑊0 𝑅] +
log(𝐺 𝑙 )
𝜂 𝑙=1



(6.5)

While the auxiliary variable can be updated similarly as


3
𝜕
1Õ
𝜗𝑖+1 = 𝜗𝑖 + 𝛼
E𝑡+1 [𝑊0 𝑅] +
log(𝐺 𝑙 )
𝜕𝜗𝑖
𝜂 𝑙=1

(6.6)

Where 𝛼∈ (0, 1) is the learning rate, and ensure replacing CVaR𝛾 in (6.2) with its
estimate in (6.3). However, the online update of Θ𝑡 governed by (6.3), (6.4), (6.5),
and (6.6) has two challenges. The first challenge is exploding gradient at high risk1
aversion level 𝛾 due to the term (𝑡+1)(1−𝛾)
in (6.3). That problem becomes even worse

as 𝛾 → 1, which leads to computational burden and the policy would probably have a
fixed trading position. To overcome the first challenge, we add the second norm of Θ𝑡
as a regularization term to Problem (6.1) to penalize large policy parameters; thus,
1 With a slight abuse of notation.

66

the problem becomes
3

max
Θ𝑡 ,𝜗𝑡

E𝑡+1 [𝑊0 𝑅] + 𝜆||Θ𝑡 || 2 +


1Õ
log 𝐺 𝑙
𝜂 𝑙=1

(6.7)

Where 𝜆∈ [0, 1] is the regularization constant. The regularization term, as stated
by Bishop in [148], enhances the generalization performance of learning systems with
objectives that do not involve error minimization. The second challenge we have is
a biased estimate ĉ𝛾,𝑖 as 𝑡 increases. Having a biased estimate is problematic in our
particular application since we expect the distribution of the reward to vary over time
due to the nonstationary behavior of the financial market. To circumvent the second
issue and to align our algorithm with incremental learning methods, we propose an
adaptive estimation for CVaR𝛾,𝑡+1 . In that sense, the estimate at each step is derived
using only a predetermined 𝑁 + 2 past samples of the reward. Thus, the estimates in
(6.3) and (6.4) become

Where 𝑅¯ = [𝑅𝑡−𝑁

𝑡+1
Õ
1
ĉ𝛾,𝑖 = v̂𝛾,𝑖 +
[−𝑅 𝑗 − v̂𝛾,𝑖 ] +
(𝑁 + 2)(1 − 𝛾) 𝑗=𝑡−𝑁

(6.8)

v̂𝛾,𝑖 = − 𝑅¯d(𝑁+2)𝛾e:𝑁+2

(6.9)

𝑅𝑡−𝑁−1

···

𝑅𝑡+1 ]. With that approach, we ensure the estimates

are adaptive and reflect any potential market-regime switching. In addition, the proposed approach enhances the computational efficiency of deriving the estimates since
with the original formulation the computational cost increases with 𝑡. We now can

67

Algorithm 1 CVaR-Constrained Optimal Policy – Incremental Learning
Select: 𝑊0 , 𝛾, 𝜇 𝛽 , 𝛼, 𝜆, 𝑁, 𝑛, 𝐼
Initialize: Θ0 , 𝜗0 , 𝜂0 and 𝑡 = 0, 𝑖 = 1
for 𝑡 < 𝑇 do
Environment
Sense the environment: 𝑥𝑡 ← ℎ𝑡 ( 𝑝 𝑡 , ..., 𝑝 𝑡−𝑛 )
Take action: 𝜋𝑡 ← 𝑓 (Θ𝑡T 𝑥𝑡 )
Observe reward: 𝑅𝑡+1 ← (5.7) or (5.9)
Computation
for 𝑖 ≤ 𝐼 do
¯
Compute E𝑡+1 [𝑊0 𝑅]
Compute VaR𝛾,𝑡+1 : v̂𝛾,𝑖 ← (6.9)
Compute CVaR𝛾,𝑡+1 : ĉ𝛾,𝑖 ← (6.8)
Update decision vector: Θ𝑖 ← (6.10)
Update aux. decision variable: 𝜗𝑖 ← (6.10)
𝜂𝑖 ← 𝜂𝑖−1 /𝑖
end for
Θ𝑡+1 ← Θ 𝐼
𝜗𝑡+1 ← 𝜗𝐼
end for
update Θ𝑡 and 𝜗𝑡 recursively as follows

3
1 Õ
¯
Θ𝑖+1 = Θ𝑖 + 𝛼∇Θ𝑖 E𝑡+1 [𝑊0 𝑅] + 𝜆||Θ𝑡 || 2 +
log(𝐺 𝑙 )
𝜂𝑖 𝑙=1


3
𝜕
1 Õ
¯
𝜗𝑖+1 = 𝜗𝑖 + 𝛼
E𝑡+1 [𝑊0 𝑅] + 𝜆||Θ𝑡 || 2 +
log(𝐺 𝑙 )
𝜕𝜗𝑖
𝜂𝑖 𝑙=1


(6.10)

The estimates in (6.8), (6.9) and the update rule in (6.10) form the basis of our online
incremental learning algorithm that optimizes Problem (5.2) under practical setup of
considering trading frictions, which is depicted in Algorithm 1.

68

6.2

CVaR-Sensitive Agent

For the CVaR-sensitive agent in Problem (5.3), it almost follows a similar computational procedure of the constrained version, except for slight differences we discuss
next. We again consider adaptive estimation for CVaR𝛾,𝑡+1 using 𝑅¯ along the the second norm of Θ𝑡 for regularization as well as the constraints 𝐺 𝑙 for 𝑙 = 2, 3. Considering
this, the unfolded version of (5.3) using the log-barrier method becomes
3

min

Θ𝑡 ,𝜗𝑡

CVaR𝛾,𝑡+1 [−𝑊0 𝑅(Θ𝑡 )] + 𝜆||Θ𝑡 || 2 +


1Õ
log 𝐺 𝑙
𝜂 𝑙=2

(6.11)

One now can estimate CVaR𝛾,𝑡+1 and VaR𝛾,𝑡+1 as in (6.8) and (6.9), respectively. Finally, the policy’s decision vector along with the auxiliary variable are iteratively
updated in the descent direction of the objective function

3
1 Õ
Θ𝑖+1 = Θ𝑖 − 𝛼∇Θ𝑖 ĉ𝛾,𝑖 + 𝜆kΘ𝑖 k 2 +
log(𝐺 𝑙 )
𝜂𝑖 𝑙=2


3
𝜕
1 Õ
𝜗𝑖+1 = 𝜗𝑖 − 𝛼
ĉ𝛾,𝑖 + 𝜆kΘ𝑖 k 2 +
log(𝐺 𝑙 )
𝜕𝜗𝑖
𝜂𝑖 𝑙=2


(6.12)

The estimates in (6.8) and (6.9) with the update rules in (6.12) form the basis of
our online algorithm of the CVaR-sensitive agent under frictional market, which is
depicted in Algorithm 2.
To the best of our knowledge, this is the first work presenting a trading system
under the DRL framework that learns on the go with well-performance. The convexity
of our CVaR𝛾 problems and the incremental policy learning demonstrate superior
performance during our experimentation that we discuss over Chapter 7. In that
69

Algorithm 2 CVaR-Sensitive Online Optimal Policy – Incremental Learning
Select: 𝑊0 , 𝛾, 𝛼, 𝜆, 𝑁, 𝑛, 𝐼
Initialize: Θ0 , 𝜗0 , 𝜂0 and 𝑡 = 0, 𝑖 = 1
for 𝑡 < 𝑇 do
Environment
Sense the environment: 𝑥𝑡 ← ℎ𝑡 ( 𝑝 𝑡 , ..., 𝑝 𝑡−𝑛 )
Take action: 𝜋𝑡 ← 𝑓 (Θ𝑡T 𝑥𝑡 )
Observe reward: 𝑅𝑡+1 ← (5.7) or (5.9)
Computation
for 𝑖 ≤ 𝐼 do
Compute VaR𝛾,𝑡+1 : v̂𝛾,𝑖 ← (6.9)
Compute CVaR𝛾,𝑡+1 : ĉ𝛾,𝑖 ← (6.8)
Update decision vector: Θ𝑖 ← (6.12)
Update aux. decision variable: 𝜗𝑖 ← (6.12)
𝜂𝑖 ← 𝜂𝑖−1 /𝑖
end for
Θ𝑡+1 ← Θ 𝐼
𝜗𝑡+1 ← 𝜗𝐼
end for
sense, the agents reveal their capability in learning adaptively and discovering a riskaware lucrative strategy through a single run over the data. Besides that, the proposed
model is more efficient than other methods in computing the solution, which is vital
under the incremental learning technique.

70

CHAPTER 7

NUMERICAL EXPERIMENTS
In this chapter, we perform exhaustive experimentation to gain insights related to the
performance of our proposed trading agents. We generally divide our experiments into
three main parts, each with a specific purpose and objective with its own considered
dataset. First, we assess the CVaR-constrained and CVaR-sensitive agents’ response
to the different risk parameters in our Problems (5.2) and (5.3), respectively. We
also investigate the systems’ endurance against various trading frictions. For vaster
insights, we do that assessment under different financial markets that exhibit various
volatility levels over the out-of-sample data, i.e., we test under trading an Exchange
Traded Fund (ETF), commodity, and the forex market. After that, we move on to
compare the performance of our proposed agents to the foundational one proposed by
Moody and Saffell [16] and the state-of-art fuzzy deep RL trading system proposed by
Deng et al. in [39]. We also compare the performance against baseline trading strategies, for example, the Buy&Hold and the so-called Dual Moving Average Crossover
(DMAC) [149]. We note that the first two experiments are performed under the case
of trading a single asset. Finally, we consider the application of assets allocation where
71

the investing agent re-balances the capital between ETF and 3-month Treasury bill
(T-bill) at each time step.

7.1

Trading a Single Asset

This section considers assessing our agents against different risk parameters, transaction costs, and other well-known trading systems. That is all under trading a single
asset application considering the reward signal in (5.7).

7.1.1

Agents’ Sensitivity to Risk Parameters

For a comprehensive experiment, we do the sensitivity assessment under different markets with the datasets described in Table 7.1. We demonstrate the price signal trend
of each considered asset in Figure 7.1. As we can observe from the price signals and
Table 7.1, the Brent crude oil has the most severe volatility compared to the SPY
(Standard and Poor’s Depository Receipt (SPDR)) and EURUSD pair. In contrast,
the latter has the most stable price action during the testing period, with all assets
experiencing several primary price trends during the selected period. Therefore, we
believe that our selection for those datasets would reveal comprehensive insights into
the agent adaptability in risk management under trading in different market characteristics. Nonetheless, we note that we only assess the CVaR-sensitive agent under
trading the SPY-ETF.

72

SPY-ETF Price ($)

500

400

300

200
0

2

4

6

8 10 12 14 16 18 20 22
×103
Ticks- 15 mins

(a) US SPDR trust fund price signal

EURUSD

Brent Crude Price ($)

1.3
80
60
40

1.2

1.1

20
0

10

20

30
40
Ticks- 15 mins

50

1.0
0

60
×103

(b) Brent crude oil price signal

10

20

30
40
50
Ticks- 15 mins

60

70
×103

(c) EURUSD pair signal

Figure 7.1: Price signals of the various tradable assets for assessing CVaR agents’
sensitivity to risk parameters and transaction costs – Trading a single asset
Table 7.1: Description of out-of-sample data for assessing CVaR agents’ sensitivity to
risk parameters and transaction costs – Trading a single asset
Asset

Start Date

End Date

Trading
Window

Investment
Horizon (𝑇)

Annualized
Volatility

SPDR trust fund
(US-ETF)

06/08/2018

12/11/2021

15-minute

∼ 21, 300

21.43%

Brent crude oil
(Commodity)

04/09/2018

03/09/2021

15-minute

∼ 65, 000

49.39%

EUR-USD current pair
(forex)

03/12/2018

02/12/2021

15-minute

∼ 75, 000

6.54%

CVaR-Constrained Agent’s Response to Risk-Aversion Level (𝛾)
and Constraint Boundary (𝜇 𝛽 )
For the sensitivity to the constraint bound 𝜇 𝛽 , we selected the range from 0.01% to
0.1% for the SPY and EURUSD, while we extended the boundary for the Brent crude
to 0.25% to observe how the agent would respond to a more relaxed constraint. Note
that the boundary value 𝜇 𝛽 governs the tolerable expected loss beyond VaR𝛾 at a
73

selected confidence level 𝛾 within about a single trading day. A common selection for
𝛾 is in the range of 0.90 − 0.99 depending on the preferred risk aversion of the trader
[23]; with that, we consider 𝛾 = 0.90 for the 𝜇 𝛽 sensitivity assessment experiment.
Nonetheless, when we test the sensitivity to 𝛾, we assess higher levels to see how the
agent would react to a highly conservative constraint while looking for the expected
return; we tested the range of 0.80 − 0.975 at a fixed 𝜇 𝛽 for each asset. We visualize
the results associated with risk sensitivity of our agent with considering a round-trip
transaction cost (2𝛿) of 0.1% under trading the SPY, Brent crude, and EURUSD in
Figure 7.2, Figure 7.3, and Figure 7.4, respectively.
One can notice that overall the total return and Sharpe ratio improve with 𝜇 𝛽
under all assets, except for the Brent crude, where they start declining at 𝜇 𝛽 > 0.1%
as shown in Figure 7.3. We also see that, under all assets, the total return and
Sharpe ratio follow the same pattern with increasing 𝜇 𝛽 . Since the Sharpe ratio, by
definition, is the expected return per unit of risk, we believe that pattern, considering
all assets with 𝜇 𝛽 ≤ 0.1% in the figures, indicates improvement in the agent’s risk
management at higher values of 𝜇 𝛽 as its ability to generate more profits at a steady
return standard deviation is evident. However, at more and more relaxed val

[The evaluation harness truncated this reference: showing the first 120000 of 179806 characters.]
</reference>

<statements>
1. Supervised ML models (trees, random forests, neural nets, AutoML) learn non‑linear mappings from predictors to returns or risk premia; deep RL models directly learn allocation policies that optimize risk‑adjusted reward under dynamic, multi‑period settings.
2. RL frameworks embed risk directly in the objective or constraints, e.g. Sharpe ratio (mean/variance), maximum drawdown, or CVaR‑based penalties; these can reduce left‑tail risk and drawdowns more effectively than basic mean–variance.
3. “Safe RL” approaches design reward functions or constraints explicitly in terms of risk measures (variance, CVaR), allowing agents to learn policies that manage tail losses as well as average returns.
4. Direct RL policies: Deep RL agents map market states to portfolio weights and learn strategies that maximize risk-adjusted returns (Sharpe, drawdown-adjusted measures, CVaR-sensitive rewards) under realistic transaction costs and dynamics.
5. Empirical comparisons show RL-based allocations can achieve higher Sharpe ratios, lower maximum drawdowns, and reduced left-tail risks than static mean-variance benchmarks.
6. Deep RL for allocation: Risk measure / handling: Risk coded into reward or constraints (Sharpe, drawdown, CVaR, variance penalties).
7. Deep RL for allocation: Main limitations: Hard to interpret; training instability; data‑hungry; harder to align strictly with economic theory.
8. Replace pure variance with a mean-risk formulation using CVaR or downside measures, especially for tail-risk-sensitive mandates.
9. Estimate risk via a combination of historical covariances, scenario analysis, and ML-based risk models (e.g. deep volatility or drawdown predictors), but keep the final risk measure simple and well-understood (variance + CVaR constraints, or maximum drawdown limits).
10. Use RL to learn rebalancing and timing rules on top of the baseline optimizer, e.g. controlling how aggressively to move toward the optimizer’s target weights given transaction costs and regime shifts.
11. Embed risk-adjusted rewards (Sharpe, CVaR penalties, drawdown constraints) so the policy optimizes long-run risk-return trade-offs rather than myopic one-period objectives.
12. RL and risk‑aware agents reduce left‑tail risks and improve risk‑adjusted returns versus static MV benchmarks.
</statements>

Begin the assessment now. Output only the JSON list, without any conversational text or explanations.