You will be provided with a reference and some statements. Please determine whether each statement is 'supported', 'unsupported', or 'unknown' with respect to the reference. Please note:
First, assess whether the reference contains any valid content. If the reference contains no valid information, such as a 'page not found' message, then all statements should be considered 'unknown'.
If the reference is valid, for a given statement: if the facts or data it contains can be found entirely or partially within the reference, it is considered 'supported' (data accepts rounding); if all facts and data in the statement cannot be found in the reference, it is considered 'unsupported'.

You should return the result in a JSON list format, where each item in the list contains the statement's index and the judgment result, for example:
[
    {
        "idx": 1,
        "result": "supported"
    },
    {
        "idx": 2,
        "result": "unsupported"
    }
]

Below are the reference and statements:
<reference>
Deep Reinforcement Learning for Portfolio Optimization using Latent
Feature State Space (LFSS) Module

arXiv:2102.06233v1 [q-fin.PM] 11 Feb 2021

Kumar Yashaswi1
Abstract— Dynamic Portfolio optimization is the process of
distribution and rebalancing of a fund into different financial
assets such as stocks, cryptocurrencies, etc, in consecutive
trading periods to maximize accumulated profits or minimize
risks over a time horizon. This field saw huge developments
in recent years, because of the increased computational power
and increased research in sequential decision making through
control theory. Recently Reinforcement Learning(RL) has been
an important tool in the development of sequential and dynamic
portfolio optimization theory. In this paper, we design a Deep
Reinforcement Learning (DRL) framework as an autonomous
portfolio optimization agent consisting of a Latent Feature State
Space(LFSS) Module for filtering and feature extraction of
financial data which is used as a state space for deep RL
model. We develop an extensive RL agent with high efficiency
and performance advantages over several benchmarks and
model-free RL agents used in prior work. The noisy and nonstationarity behaviour of daily asset prices in the financial
market is addressed through Kalman Filter. Autoencoders,
ZoomSVD, and restricted Boltzmann machines were the models
used and compared in the module to extract relevant time series
features as state space. We simulate weekly data, with practical
constraints and transaction costs, on a portfolio of S&P 500
stocks. We introduce a new benchmark based on technical
indicator Kd-Index and Mean-Variance Model as compared to
equal weighted portfolio used in most of the prior work. The
study confirms that the proposed RL portfolio agent with state
space function in the form of LFSS module gives robust results
with an attractive performance profile over baseline RL agents
and given benchmarks.
Keywords- Portfolio Optimization; Reinforcement learning;
Deep Learning; Kalman Filter; Autoencoders; ZoomSVD;
RBM; Markowitz Model; KD-Index

I. INTRODUCTION
Portfolio Optimization/Management problem is to optimize the allocation of capital across various financial
assets such as bonds, stocks or derivatives to optimize
a preferred performance metric, like maximize expected
returns or minimize risk. Dynamic portfolio optimization
involves sequential decision making of continuously reallocating funds (which has roots in control theory) into
assets in consecutive balancing periods based on real-time
financial information to achieve desired performance. In
financial markets, an investor’s success heavily relies on
maintaining a well balanced portfolio. Engineering methods
like signal processing [1], control theory [2], data mining
[3] and advanced machine learning [4,5] are routinely used
*This work was supported by Department of Mathematics, Indian Institute of Technology Kharagpur
1 K. Yashaswi- Department of Mathematics, Indian Institute of Technology Kharagpur- kyashaswi@iitkgp.ac.in

in financial market applications. Researchers are constantly
working on machine learning techniques that have proven
so successful in computer vision, NLP or beating humans
in chess, etc, in the domain of dynamic financial markets
environment.
Before the advent of machine learning, most portfolio
models were based on variations of Modern Portfolio Theory(MPT) by Markowitz [6]. These had the drawbacks of
being static and linear in computation. The dynamic methods used for this problem like dynamic programming and
convex optimization, required discrete action space based
models and thus were not so efficient in capturing market
information [7,8]. To address this issue we work with deep
reinforcement learning model. Reinforcement Learning(RL)
is an area in artificial intelligence which focuses on how
software agents take action in a dynamic environment to
maximize cumulative performance metric or reward [9].
Reinforcement learning is appropriate in dynamical systems
requiring optimal controls, like robotics [10], self-driving
cars [11] and gaming [12], with performance exceeding other
models.
With the breakthrough of deep learning, the combination
of RL and neural networks(DRL) has enabled the algorithm to give optimal results on more complex tasks and
solving many constraints posed by traditional models. In
portfolio optimization, deep reinforcement learning helps in
sequentially re-balancing the portfolio throughout the trading
period and has continuous action space approximated by a
neural network framework which circumvents the problem
of discrete action space.
RL has been widely used in financial domain [13,14] like
algorithmic trading and execution algorithms, though it has
not been used to that extent for portfolio optimization. Some
major works [15,16,17,18,19] gave state-of-the-art performance on their inception and most of the modern works have
been inspired by these research. While [18,19] considered
discrete action spaces using RL, [15,16,17] leveraged use of
deep learning for continuous action space. They considered
a model-free RL approach which modelled the dynamics
of market through exploration. The methods proposed in
[15,16,20,21,17,9] considered constraints like transaction
cost and suitable reward function, though they suffered from
a major drawback of state space modelling. They did not
take into account the risk and dimensionality issues caused
by volatile, noisy and non-stationary market environment.
Asset prices are highly fluctuating and time-varying. Sudden
fluctuations in asset price due to many factors like market
sentiments, internal company conflict, etc cause the prices to

deviate from their true value based on actual fundamentals
of the asset. Many other factors like economic conditions,
correlation to market, etc have an adverse effect on the prices
but due to limited data and model complexity, they cannot
be incorporated in modelling phase.
In this paper, we propose a model-free Deep RL agent
with a modified state space function over the previous works
which filters the fluctuations in asset price and derives a
compressed time-series representation containing relevant
information for modelling. We introduce the Latent Feature
State Space Module(LFSS) as a state space to our deep
RL architecture. This gives a compressed latent state space
representation of filtered time series in our trading agent. It
consists of 2 units:
• Filtering Unit
• Latent Feature Extractor Unit
The filtering unit first takes the raw, unfiltered signal from
asset prices and outputs a filtered signal using Kalman Filter
as described in [22]. After the filtered signal is obtained the
Latent Feature Extractor Unit extracts a lower dimensional
latent feature space using 3 models proposed by us:
• Autoencoders
• ZoomSVD
• Restricted Boltzmann Machine
All these 3 models are applied independently and their
performance is compared to judge which is the most suitable
to get a compressed latent feature space of the price signal.
Autoencoders and Restricted Boltzmann Machine are selfsupervised deep learning architecture that are widely used in
dimensionality reduction [23,24] to learn a lower dimension
representation for an input space, by training the network
to ignore input noise. Svd(Singular Value Decomposition)
is a matrix factorization procedure used widely in machine
learning and signal processing to get a linear compressed representation of a time series. We employ ZoomSvd procedure
which is a fast and memory efficient method for extracting
SVD in an arbitrary query time range proposed recently in
[25].
Asset price data are usually processed in OHLC(open,
high, low, close) form and RL state space are built upon the
same. In deep-RL models used in [15,16,20,21,9], returns or
price ratio(eg High/close) are used in unprocessed form as
state space with a backward time step which is the hypothesis
that how much the portfolio weights are determined by the
assets previous prices. Although most of the RL agents
applied in research use OHLC State Space, there are still
several challenges that LFSS Module solves:
• Financial Data is considered very noisy with distorted
observations due to turbulences caused by the information published daily along with those sudden market behaviours that change asset prices. These intraday noises
are not the actual representative of the asset price and
the real underlying state based on actual fundamentals
of the asset are corrupted by noise. Filtering techniques
help approximate the real state from noisy state and
help’s improve quality of data.

OHCL Data space is highly dimensional and, as such,
models that try to extract relevant patterns in the raw
price data can suffer from the so-called curse of dimensionality [26]. We will explore the potential of LFSS
module to extract relevant features from the dynamic
asset price information in a lower dimensional feature
space. This latent space can hypothetically incorporate
features such as economic conditions or asset fundamentals.
Rest of the framework is roughly similar to [1] in terms of
action space, reward function, etc. We use deep deterministic
policy gradients (DDPG) algorithm [27] as our RL Model
with convolutional neural network(CNN) as deep learning
architecture for prediction framework.
Our dataset was stocks constituting S&P 500 with data
from 2007-2015 considered for training data and 2016-2019
used for testing. We design a new benchmark based on
technical indicator Stochastic Kd-Index and Mean-Variance
model which is a variation of equal-weighted portfolio
benchmark and performs much better than the same. This
benchmark is based on the works in [28].
To the best of our knowledge, this is the first of many work
that leverages the use of latent features as state space, and
further integrates with already known deep RL Framework
in portfolio optimization domain. The main aim of our work
is to investigate the effectiveness of Latent Feature State
Space(LFSS) module added to our RL agent, on the process
of portfolio optimization to get improved result over existing
deep RL framework.
•

A. Related Work
With the increased complexity of deep RL models, their
application in Portfolio Optimization has increased widely
over the past years. As we discussed in the last section,
[15,16,17,18] were one of the breakthrough papers in this
domain. Our work borrowed many methods from [15,16],
which had the advantage of using continuous action space.
They used state of the art RL algorithms like DQN, DDPG,
PPO, etc for various markets. [20] makes a comparison of
model-based RL agent and model-free deep RL agent, thus
showing dominance and robustness of model-free Approach.
[15] showed CNN architecture performed much better than
LSTM networks for the task of dynamic optimization of
cryptocurrency portfolios, even though LSTM is more beneficial for time-series data. [9] used a modified reward
function that prevented large portfolio weights in a single
asset.
[26,29] were the first to use the concept of an added
module for improved performance to already existing architectures. [29] added a State Augmented Reinforcement
Learning(SARL) module to augment the asset information
which leveraged the power of Natural Language Processing(NLP) for price movement prediction. They used financial
news data to represent the external information to be encoded
and augmented to the final state. [26] used combination
of three modules infused prediction module(IPM), a generative adversarial data augmentation module (DAM) and a

behaviour cloning module (BCM). IPM forecast the future
price movements of each asset, using historical data . DAM
solves the problem of availability of large financial datasets
by increasing the dataset size by making use of GANs and
BCM uses a greedy strategy to reduce volatility in changes
in portfolio weight’s hence reducing transaction cost[26].
II. BACKGROUND
A. Portfolio Optimization
A portfolio is a collection of multiple financial assets, and
is characterized by its:
• Component: n assets that comprise it. In our case n =
15
th
• Portfolio vector, wt : the i
index illustrates the proportion of the funds allocated to the ith asset
wt = [w1,t , w2,t , w3,t , ...wn,t ] ∈ Rn
For w1,t < 0 for any n, implies short selling is allowed.
We add a risk-free asset to our portfolio for the case if all
the wealth is to be allocated to a risk-free asset. The weight
vector gets modified to
wt = [w0,t , w1,t , w2,t , w3,t , ...wn,t ]

(1)

The closing price is defined as vi,t for asset i at time t. The
price vector vt = [v0,t , v1,t , v2,t , v3,t , ...vn,t ], consists of the
market prices of the n assets, taken as the closing price of
the day in this case and constant price of risk-free asset v0,t .
Similarly, vthi and vtlo denote the highest prices and the lowest price vector at time step t, respectively. Over time, asset
price’s change, therefore we denote yt as the relative price
v1,t+1 v2,t+1 v3,t+1
vn,t+1 T
vector equal to vt+1
vt = (1, v1,t , v2,t , v3,t , .... vn,t ) .
To reduce risk, portfolios with many assets are preferable
over holding single assets. We assume readers have knowledge of important concepts of portfolio namely portfolio
value, asset returns, portfolio variance, Sharpe ratio, meanvariance model, etc as we will not go in detail of these topics.
These topics are well described in thesis reports [20,9].
B. Deep RL Model as a Markov Desicion Process
We briefly review the concepts of DRL and introduce the
mathematics of the RL agent. Reinforcement learning is a
self-learning method, in which the agent interacts with the
environment with no defined model and less prior information, learning from the environment by exploration while at
the same time, optimally updating its strategy to maximize a
performance metric. RL consist of an agent that receives the
controlled state of the system and a reward associated with
the last state transition. It then calculates an action which
is sent back to the system. In response, the system makes a
transition to a new state and the cycle is repeated as described
in fig 1.
The goal is to learn a set of actions taken for each state
(policy) so as to maximize the cumulative reward in dynamic
environment which in our case is the market. RL is modelled on the concepts of Markov Decision Process(MDP), a
stochastic model in discrete time for decision making [9].
They work on the principle of Markov chain.

Fig. 1.

Reinforcement Learning Setting [30]

A MDP is defined as a 5-tuple (S, A, P, r, γ) described
below with t being time horizon over portfolio periodS
• S =
t St is a collection of finite dimensional continuous S
state space of the RL model.
• A =
t At is the finite dimensional continuous action
space as we consider the market environment as an
infinite Markov Decision Process (IMDP).
• P : S × A × S → [0, 1] is the state transition probability
function
• r : S × A → R is the instant or the expected instant
reward at each time index obtained for taking an action
in a particular state.
• γ ∈ (0, 1] is a discount factor. When γ=1 a reward
maintains its full value at each future time index independent of the relative time from present time step. As γ
decreases, the effect of reward in the future is declined
exponentially by γ
C. Policy Function
Policy function(π) indicate the behaviour of a RL agent
in a given state. π is a “state to action” mapping function,
π : S → A. A policy is either Deterministic At+1 = π(St )
or Stochastic: π(S|α). We deal with the mapping to be
deterministic. Since we are dealing with the case of infinite
Markov decision process (IMDP), the policy is modelled
as a parameterized function with respect to parameter θ
and policy mapping becomes At = πθ (St ). Policy function
parameterized with θ is determined solved by using DRL
models like DDPG, DQN, etc which derived from concepts
of Q-Learning described in the next subsection.
D. Q-Learning
Q-learning is an approach in reinforcement learning which
helps in learning optimal policy function, using the concept
of Q-value function. Q-value function is defined as the
expected total reward when executing action A in state S
and henceforth follow a policy π for future time steps.
Qπ (St , At ) = Eπ (rt |St , At )

(2)

Since we are using a deterministic policy approach, we can
simplify the Q-function using Bellman Equation as:
Qπ (St , At ) = Eπ (Rt+1 + γQπ (St+1 , At+1 )|St , At )

(3)

The optimal policy for Qπ is the one which gives maximum
Q-function over all policies:
π(S) = argmaxQπ (S, A)

(4)

To speed up the convergence to the optimal policy, concept of
replay buffer is used and a target network is used to move
the relatively unstable problem of learning the Q function
closer to the case of supervised learning[16].
E. Deep Deterministic Policy Gradient(DDPG)
DDPG is a model-free algorithm combining Deep QNetwork (DQN) with Deep Policy Gradient (DPG). DQN
(Deep Q-Network) stabilizes Q-function learning by replay
buffer and the frozen target network [9]. In case of DPG, the
deterministic target policy function is constructed by a deep
learning model and the optimal policy is reached by using
gradient descent algorithm. The action is deterministically
outputted by the policy network from the given state. DDPG
works on actor-critic framework, where the actor network
which outputs continuous action, and then the actor performance is improved according to critic framework which
consist of an appropriate objective function.
The P
returns from the agent at a time step t is defined as
∞
k−t
rt =
r(Sk , Ak ) where r is reward function, γ
k=t γ
is the discount factor and return rt is defined as the total
discounted reward from time step t to final time period[15].
The performance metric of πθ for time interval [0, tf inal] is
defined as the corresponding reward function:
J[0,tf inal ] (πθ ) = E(r, Sπθ )
=

T
X

γ t r(St , πθ (St ))

(5)

t=1

The network is assigned random weights initially. With
the gradient descent algorithm (Eq 6) weights are constantly
updated to give the best expected performance metric.
θ ← θ + λ∇θ J[0,tf inal ] (πθ )

(6)

λ is the learning rate. For the portfolio optimization task, we
modify our reward function to be similar to that described
in [15] and we shall describe it in section IV. For training
methodology also we borrow the works of [15] as we use the
concepts of Portfolio Vector Memory(PVM) and stochastic
batch optimization. We have only worked with and described
in brief DDPG algorithm but other algorithms like Proximal
Policy Optimization(PPO) are widely used.

The filtering unit uses Kalman Filter to obtain a linear
hidden state of the returns with the noise being considered
Gaussian. The Latent Feature Extractor Unit compares 3
models Autoencoders, ZoomSVD and RBM, each being used
independently with the filtering unit to extract latent feature
space which is the state space of our deep RL architecture.
LFSS module structure can be visualized from figure 2.
A. Kalman Filter
Is an algorithm built on the framework of recursive
Bayesian estimation, that given a series of temporal measurements containing statistical noise, outputs state estimates
of hidden underlying states that tend to be more accurate
representation of underlying process than those based on
an observed measurement [31]. Theory of Bayesian filtering
is based on hidden Markov model. Kalman Filter assumes
linear modelling and gaussian error (Eq. 6,7). Kalman Filter
is a class of Linear-Quadratic Gaussian Estimator(LQG)
which have closed-form solution as compared to other
filtering methods having approximate solutions(eg Particle
Filtering)[32]. We used Kalman Filter to clean our return
time-series of any Gaussian noise present due to market
dynamics. The advantages of Kalman Filter are:
• Gives simple but effective state estimates[18] which is
computationally inexpensive
• Market’s are highly fluctuating and non-stationary, but
the Kalman Filter gives optimal results despite being a
LQG estimator.
• It is dynamic and works sequentially in real time,
making predictions only using current measurement
and previously estimated state on predefined model
dynamics.
Though markets may not always be modelled in linear fashion and noises may not be Gaussian, Kalman filter is widely
used and various experiments have proved to be effective
for state estimation [31,22]. The mathematical explanation
of the filter is given from eq 7-15 [22].
xt+1 = Ft xt + wt , wt ∼ N (0, Q)

(7)

yt = Gt xt + vt , vt ∼ N (0, R)

(8)

where xt and yt are the state estimate and measurements at
time index t respectively. Ft is the state transition matrix,
Gt is measurement function. White noises wt (state noise)
and vt (process noise) are independent, normally distributed
with zero mean and constant covariance matrices Q and R
respectively.

III. L ATENT F EATURE S TATE S PACE M ODULE
Price and returns are usually derived from Open, High,
Low, Close(OHLC) data. These are unprocessed and contain
high level of noise. Using OHLC data based state space
[15,16,20,21,9] also leads to high dimensionality and the use
of extra irrelevant data. LFSS module helps in tackling such
problems by using a filtering unit to reduce noise in the data
and a Latent Feature Extractor Unit which helps in obtaining
a set of compressed latent features which represent the assetprices in a more suitable and relevant form.

L
ML
xM
t+1|t+1 = Ft xt|t + Kt+1 εt+1

(9)

L
εt+1 = yt+1 − Gt xM
t+1|t

(10)

Kt+1 = Pt+1|t GTt [Gt Pt+1|t GTt + R]−1

(11)

Pt+1|t = Ft Pt|t FtT + Q

(12)

Pt+1|t+1 = [I − Kt+1 G]Pt+1|t

(13)

L
xM
0|0 = µ0

(14)

Fig. 2. LFSS Module Structure: (i) The stock price series is passed through a filtering layer which uses Kalman Filter to clean the signal of any Gaussian
noise. (ii) The filtered signal is passed through a Latent Feature Extractor Unit which consists of one of 3 feature extractor methods a) Autoencoders b)
ZoomSVD c) RBM, each applied individually and independently to obtain a lower dimensional signal

P0|0 = P0

(15)

Eq 9-15 give the closed-form solution of Kalman filter using
maximum likelihood estimation(ML). [32] gives a detailed
derivation of closed-form solution using ML and Maximum
a posteriori (MAP) estimate. εt is the error estimate defined
in Eq-10 and Kt is called the Kalman gain and defined in
Eq-11. Eq 14-15 gives the initial state assumption of the
process. In the experiment section, we use the Kalman filter
for our asset price signal and input the filtered signal to the
Latent Feature Extractor Unit of LFSS module.

output similar to the input. The mean squared norm loss
is used(equation 18,19) as loss function L.
φ, Ω = argminφ,Ω kX − (φ ◦ Ω)Xk2

(18)

L = kX − X 0 k2

(19)

For our methodology we compare and select the most
suitable of CNN autoencoder(fig 3), LSTM autoencoders and
DNN autoencoders.

B. Autoencoders
Its a state of the art model from non-linear feature extraction of time series leveraging the use of deep learning. An
autoencoder is a neural network architecture in which the
output data is the same as input data during training, thus
learns a compact representation of the input, with no need
for labels [23]. Since no output exists i.e. requires no human
intervention such as data labelling it is a domain of selfsupervised learning [33]. The architecture of an autoencoder
uses two parts in this transformation[34]
• Encoder- by which it transforms its high dimensional
inputs into a lower dimension space while keeping the
most important features
φ(X) → E
•

(16)

Decoder- which tries to reconstruct the original input
from the output of the encoder
Ω(E) → X

(17)

The output of the encoder is the latent-space representation
which is of interest to us. It is a compressed form of
the input data in which the most influential and significant
features are kept. The key objective of autoencoders is to
not directly replicate the input into the output [34] i.e learn
an identity function. The output is the original input with
certain information loss. Encoder should map the input to a
lower dimension than input as a larger dimension may lead
to learning an identity function.
The encoder-decoder weights are learnt using backpropagation similar to supervised learning approach but with

Fig. 3.
Convolutional-Autoencoder Explained- (i) First the encoder
consisting of Convolution and Pooling Layer, compresses the input time
series to lower dimension. (ii) Decoder reconstructs the input space using
upsampling and Convolution layer [35]

C. ZoomSVD: Fast and Memory Efficient Method for Extracting SVD of a time series in Arbitrary Time RangeBased on the works in [25], this method proposes an
efficient way of calculating the Singular Value Decomposition(SVD) of time series in any particular range. The SVD
for a matrix is as defined in Eq 20.
A = U ΣV T

(20)

If A has dimensions M × N (for time series case, M is
the length of time period and N is the number of assets)
then for case of compact-SVD(form of SVD used) U is
M × R and V is N × R orthogonal matrices such that
U U T = IRxR and V V T = IRxR and R ≤ min[M, N ] is
the rank of A. U and V are called left-singular vectors and
right-singular vectors of A. Σ=diag(d1 , d2 , d3 , ...dR ) is a
square diagonal of size R × R with d1 ≥ d2 ≥ d3 , ... ≥

Fig. 4. ZoomSVD representation from [13]- (i) In the Storage-Phase the original asset-price series matrix is divided in blocks and compressed to lower
dimensional SVD form using incremental-SVD. (ii) In Query-Phase, SVD in reconstructed from the compressed block structure in storage phase, for a
particular time-query [ti , tf ] using Partial-SVD and Stitched-SVD [25]

dR ≥ 0 and are called singular values of A. For time
series data, A = [A1 ; A2 ; A3 ; ....AN ] where Ai is column
matrix of time series values within a range and A is the
vertical concatenation of each Ai . The SVD of a matrix can
be calculated by many numerical methods known in linear
algebra. Some of the methods are described in [36].
SVD is a widely used numerical method to discover
hidden/latent factors in multiple time series data [25], and
in many other applications including principal component
analysis, signal processing, etc. While autoencoders extract
non-linear features from high dimensional time series, SVD
is much simpler and finds linear patterns in the dimension
of time series thus leading to less overfitting in many cases.
Zoom-SVD incrementally and sequentially compresses time
series matrix in a block by block approach to reduce the
space cost in storage phase, and for a given time range query
computes singular value decomposition (SVD) in query
phase by methodologically stitching stored SVD results [25].
ZoomSVD is divided in 2 phases• Storage Phase of Zoom-SVD
– Block Matrix
– Incremental SVD
• Query Phase of Zoom-SVD
– Partial-SVD
– Stitched-SVD
Figure 4 explains the original structure proposed in [25].
In storage phase, time series matrix A is divided into blocks
of size b decided beforehand. SVD of each block is computed
sequentially and stored discarding original matrix A. In the
query phase, an input query range [ti ,tf ] is passed and
using the block structure the blocks containing the range are
considered. From the selected blocks, initial and final blocks
contain partial time ranges. Partial-SVD is used to compute
SVD for initial and final blocks and is merged with SVD of
complete blocks using Stitched-SVD to give final SVD for
query range [ti ,tf ]. Further explanation of the mathematical
formulation of the storage and query phase are described in
[25].
ZoomSVD solves the problem of expensive computational
cost and large storage space in some cases. In comparison to
traditional SVD methods, ZoomSVD computes time range
queries up to 15x faster and requires 15x less space than

other methods[25]. For the case of portfolio optimization
problem, we use ΣV T as our state space. The right singular
vector represents the characteristics of the time series matrix
A and the singular value’s represents the strength of the
corresponding right singular vector [37]. U is not used
because of high dimensionality due to the large time horizon
required by the RL agent.
D. Restricted Boltzmann machine: Probabilistic representation of the training data
Restricted Boltzmann machine is based on the works of
[38], and has similar utility to the above mentioned methods
in fields like collaborative filtering, dimensionality reduction,
etc. A restricted Boltzmann machine (RBM) is a form of
artificial neural network that trains to learn a probability
distribution over its set of inputs and is made up of two
layers: the visible and the hidden layer. The visible layer
represents the input data and the hidden layer tries to learn
feature-space from the visible layer aiming to represent a
probabilistic distribution of the data [39]. Its an energy-

Fig. 5. RBM structure consisting of 3 visible units and 2 hidden units with
weights w and bias a,b, forming a bipartite graph structure [39]

based model implying that the probability distribution over
the variables v(visible layer) and h(hidden layer) is defined
by an entropy function in vector form(Eq 21).
E(v, h) = −hT W v − aT v − bT h

(21)

where W is the weights, a and b are bias added.
RBM’s have been mainly used for binary data, but there
are some works [39,40] which present new variations for

dealing with continuous data using a modification to RBM
called Gaussian-Bernoulli RBM (GBRBM). We further add
extension of RBM, conditional RBM(cRBM) [24,41]. The
cRBM has auto-regressive weights that model short-term
temporal dependencies and hidden layer unit that model
long-term temporal structure in time-series data [24]. A
cRBM is similar to RBM’s except that the bias vector for
both layers is dynamic and depends on previous visible layers
[24]. We will further refer to RBM as cRBM in the paper.
Restricted Boltzmann machines uses contrastive divergence (CD) algorithm to train the network to maximize input
probability. We refer to [39,24] for further explanation of
mathematics of RBM’s and cRBM’s.
IV. P ROBLEM F ORMULATION
In the previous section, we had described RL agent formulation in terms of Markov decision process defined using 5
tuple (S, A, P, r, γ). In this section we describe each element
of tuple with respect to our portfolio optimization problem
along with the dataset used and the network architecture used
as our policy network.
A. Market Assumptions
In a market environment, some assumptions are considered
close to reality if the trading volume of the asset in a market
is high. Since we are dealing with stocks from S&P 500
the trading volume is significantly high(high liquidity) to
consider 2 major assumptions:
Zero Slippage- Due to high liquidity, each trade can be
carried out immediately at the last price when an order
is placed
• Zero Market Impact- The wealth invested by our agent
insignificant to make an influence on the market

•

C. Action Space
To solve the dynamic asset allocation task, the trading
agent must at every time step t be able to regulate the
portfolio weights wt . The action at at the time t is the
portfolio vector wt at the time t:
at ≡ wt = [w1,t , w2,t , w3,t , ...wn,t ]

The action space A is thus a subset of the continuous Rn
real n-dimensional space:
at ∈ A ⊆ R n ,

The S&P 500 is an American stock market index based
on the market capitalization of 500 large companies. There
are about 500 tradable stocks constituting S&P 500 index
however, we will only be using a subset of 15 randomly
selected stocks for our portfolio. The stocks were- Apple,
Amex, Citi bank, Gilead, Berkshire Hathaway, Honeywell,
Intuit, JPMC, Nike, NVDIA, Oracle, Procter & Gamble,
Walmart, Exxon Mobile and United Airlines Holdings. We
took stocks from various sectors to diversify our portfolio.
We took the data from January 2007 to December 2019,
that made up a dataset of total size of 3271 rows. 70 %
(2266 prices, 2007-2015) of data was considered as training
set and 30% as testing set(1005 prices, 2016-2019). This
large span of time subjects our RL agent to different types of
market scenarios and learns optimal strategies in both bearish
and bullish market. For our risk-free asset to invest in case
investing in stocks is risky, we use a bond with a constant
return of 0.05 % annually.

n
X

ai,t = 1, ∀t ≥ 0

(23)

i=1

The action space is continuous (infinite) and therefore consider the stock market as an infinite decision-making process
for MDP(IMDP) [9].
D. OHLC Based State Space
Before we describe the novel state space structure used by
our RL agent, we describe the state space based on OHLC
data being used in previous works. In later sections, we
showcase the superior performance of our novel state space
based RL agent.
Using the data in OHLC form, [15] considered closing,
high and low prices to form input tensor for state space of RL
agent. The dimension of input is (n,m,3)(some works may
define dimensions in (3,m,n) form), where n is the number
of assets, m is the window size and 3 denotes the number of
price forms. All the prices are normalized by closing price
vt at time t. The individual price tensors can be written as:
Vt = [vt−m+1 vt |vt−m+2 vt |vt−m+3 vt |..|vt vt ] (24)
(hi)

= [vt−m+1

(hi)

vt |vt−m+2

(hi)

vt |vt−m+3

(hi)

vt |..|vt

(lo)

= [vt−m+1

(lo)

vt |vt−m+2

(lo)

vt |vt−m+3

(lo)

vt |..|vt

Vt
B. Dataset

(22)

Vt

(hi)

vt ]
(25)

(lo)

vt ]
(26)
(hi) (lo)
Where vt , vt , vt represents the asset close, high and
low prices at time t,
represents element-wise division
operator and | represents horizontal concatenation. Vt is
defined individually for each asset as
Vt = [V1,t ; V2,t ; V3,t ; ..; Vn,t ]

(27)

where Vi,t is close price vector for asset i and ; represents
(hi)
(lo)
vertical concatenation. Similar is defined for Vt
and Vt .
Xt is the final state space formed by stacking layers Vt ,
(hi)
(lo)
as seen in figure 6.
Vt , Vt
In order to incorporate constraints like transaction cost,
the state space Xt at time t is combined with previous
time step portfolio weight wt−1 to form final state space
St = (Xt , wt−1 ). The addition of wt−1 to state space has
been widely used in other works [15,16,20,21,9] to model
transaction cost of portfolio.

account practical constraints, its near-optimality with respect
to portfolio performance and should be easily differentiable.
Applying the reward function used in [15] satisfying all of
the above conditions, the reward function at time t is :
rt = r(st , at ) = ln(at · yt − c

n
X

|ai,t − wi,t |)

(28)

i=1

As explained by Eq 4, the performance objective function is
defined as:
T
1 X t
J[0,tf inal ] (πθ ) =
γ r(St , πθ (St ))
T t=1

Fig. 6.

OHLC state space in 3-dimensional input structure [15]

E. LFSS Module Based State Space
LFSS module based state space consisted of two asset
features that were processed individually through different
neural networks and merged after processing. The state space
considered was based on time-series features of asset returns
for 15 assets in our portfolio. The asset feature extracted
were• Covariance Matrix- the symmetric matrix for 15 assets
was calculated. This part of the state space signifies the
risk/volatility and dependency part of each asset with
one another. 3 matrix for close, high and low prices
were computed giving a 15 × 15 × 3inputmatrix.
• LFSS Module Features- We described Filtering unit to
clean the time series followed by 3 different methods
Autoencoder, ZoomSVD, RBM in Latent Feature Extractor Unit to extract latent features. Close, high and
low prices were individually processed by LFSS module
to form 3 dimensional structure similar to figure 6 but
in a lower-dimensional feature space.
Similar to technique mentioned in last sub-section, previous time step portfolio weight wt−1 is added to form final
state space St = (Xt , wt−1 ), where Xt is merged network
result.
F. Reward Function
A key challenge in portfolio optimization is controlling
transaction costs(brokers’ commissions, tax, spreads, etc)
which is a practical constraint to consider to not get bias
in estimating returns. Whenever the portfolio is re-balanced,
the corresponding transaction cost is deducted from the
portfolio return. Many trading strategies work on the basic
assumption on neglecting transaction cost, hence are not
suitable for real-life trading and only obtain optimality using
greedy algorithm(e.g allocating all the wealth into the asset
which has the most promising expected growth rate) without
considering transaction cost. Going by the rule of thumb
the transaction commission is taken as 0.2% for every buy
and sell re-balancing activity i.e c = cb = cs = 0.2. We
have to consider such a reward function which takes into

(29)

where γ is discount factor and r(St , πθ (St )) is the immediate
reward at time t with policy function πθ (St ). We modify Eq
5 by dividing it by T which is used to normalize the objective
function for time periods of different length. The main
objective involves maximizing performance objective based
on policy function πθ (St ) parametrized by θ. The network is
assigned random weights initially. With the gradient descent
algorithm(Eq 6) weights are constantly updated to give the
best expected performance metric, based on the actor-critic
network.
θ ← θ + λ∇θ J[0,tf inal ] (πθ )

(30)

G. Input Dimensions
LFSS Module used considered two deep learning techniques Autoencoders and cRBM for feature extraction of
time-series. While ZoomSVD has a static method for representation and we kept fixed size of cRBM structure, we
compared different network topology’s used for training
autoencoders. 3 types of neural networks- Convolutional
Neural Network(CNN), Long-Short Term Memory(LSTM)
and Deep Neural Network(DNN) were compared, with the
corresponding results shown in the experiments section. Our
input signal consisted of considering window-size of 60 days
or roughly 2 months price data, giving sufficient information
at each time step t. The window-size is adjustable as this
value is experiment based and there is no theoretical backing
of choosing 60 days window. Since we are using 15 assets
in our portfolio the state space concept used by [15] based
on OHLC data, has input size of 15×60×3, where 60 is the
window size, 15 is the number of assets and 3 representing
3 prices close, high and low. We added the asset covariance
matrix in our state space, along with LFSS module features,
which has dimensions of 15×15×1. The LFSS module
feature space for individual methods had dimensions as
followsAutoencoder - 15×30×3
ZoomSVD - 15×15×3
• cRBM - 15×30×3

•

•

Autoencoders and cRBM provide flexibility of reducing the
original price input into variable-sized feature space, while
ZoomSVD has drawback of fixed-sized mapping.

Fig. 7. The framework of our proposed LFSS Module based RL Agent. Asset prices are structured in 3-dimensional close, high and low price and then
the state space S1,t and S2,t are extracted from state-covariance and LFSS module, and a combined state Xt is formed from a merged Neural Network.
Combined with previous timestep action space at−1 , a new portfolio vector is formed for time t through the policy network

H. Network Architecture
Our Deterministic target policy function is constructed
by a neural network architecture used by the agent. Several variations of architecture are used for building policy
function revolving around CNN, LSTM, etc. To establish
the performance of Latent Feature State Space compared to
OHLC state space, we work with single model architecture
for better judgement of performance of various state-spaces.
Considering the results in [15,20,21], CNN based network
outperformed RNN and LSTM for approximating policy
function, due to its ease in handling large amounts of
multidimensional data, though this result is empirical as
sequential neural network like RNN and LSTM should model
price data better than CNN[15].
Motivated by [15] we used similar techniques, namely
Identical Independent Evaluators(IIE) and Portfolio-Vector
Memory(PVM) for our trading agent. In IIE, the policy
network propagates independently for the n+1 assets with
common network parameters shared between each flow. The
scalar output of each identical stream is normalized by
the softmax layer and compressed into a weight vector as
the next period’s action. PVM allows simultaneous minibatch training of the network, enormously improving training
efficiency [15].
Our state space consists of 3 parts- LFSS Module Features,
covariance matrix and the portfolio vector from the previous
rebalancing step(wt−1 ). LFSS and covariance matrix were
individually processed through CNN architecture, and concatenated together into a single vector Xt after individual
layers of processing. The previous actions, wt−1 , is concatenated to this vector to form final stage state St = (Xt , wt−1 ).
This is passed through a deep neural network, with IIE and
PVM setup, to output portfolio vector wt with the training
process as described in the previous subsection. The basic
layout of our RL agent is as given in figure 7.
I. Benchmarks
•

Equal Weight Baseline(EW) - is a trivial baseline which
assigns equal weight to all assets in the portfolio.

S&P 500 - the stock-market index which depicts the
macro-level movement of the market as a whole.
• OHLC State-Space Rl Agen(Baseline-RL) - uses unprocessed asset prices as state space and trains the agent
with a CNN approximated policy function. It is the most
widely used RL agent structure for dynamic portfolio
optimization.
• Weighted Moving Average Mean Reversion(WMAMR)
- is a method which captures information from past period price relatives using moving averaged loss function
to achieve an optimal portfolio.
• Integrated Mean-Variance Kd-Index Baseline(IMVK)derived from [28], this works on selecting a subset of
given portfolio asset based on technical-indicator KdIndex and mean-variance model[6], and allocates equalweights to the subset at each time-step. For calculating
Kd-Index first a raw stochastic value(RSV) is calculated
as follows:
•

RSVt =

(vt − vmin )
× 100
(vmax − vmin )

(31)

vt , vmax , vmin represent the closing price at time t,
highest closing price and lowest closing price, respectively. Kd-Index is computed as:
2
1
+ Kt−1 ×
3
3

(32)

1
2
+ Dt−1 ×
3
3

(33)

Kt = RSVt ×

Dt = Kt ×

where K1 = D1 = 50. Whenever Kt−1 ≤ Dt−1
and Kt > Dt , a buy signal is created. Two strategies,
Moderate and aggressive, are formed upon Kd-Index.
Moderate strategy works on selecting an asset only if
Mean-Variance model weights for the asset is positive
and Kd-Index indicated a buy signal at that time-step.
Aggressive strategy selects an asset only based on KdIndex indicating a buy signal. Equal distribution of
wealth is done on the subset.

V. E XPERIMENTAL R ESULTS
A. Filtering Unit
As described before, LFSS Module has an added Filtering
Unit consisting of a Kalman Filter to filter out noise from
price data of each asset. Since deep learning model are
sensitive to the quality of data used, many complex deep
learning models may overfit the data, learning noise in the
data as well, thus not generalizing well to different datasets.
Each of the asset prices was filtered before passing to the
Latent Feature Extractor Unit.

Fig. 8.
Signal

Autoencoder Training Summary
LSTM
CNN
Training set Mse
0.0610
0.0567
Test set Mse
0.0592
0.0531
Epochs
125
100

DNN
0.0593
0.0554
100

TABLE I

cRBM Space- We use a cRBM network stacked of 2
layers(1 visible layer, 1 hidden layer). Visible unit corresponding to the asset price window(60) and 30 neurons in
the hidden layer for extracted feature space.

Filtering Unit: Filtered Berkshire Hathaway Price and Returns
Fig. 9.
Extracted Features from Input Price Window. Feature-1, 2, 3
corresponds to CNN-Autoencoder, ZoomSVD and cRBM extracted space,
respectively

B. Latent Feature Extractor Unit
Autoencoder Space- We build a time-series price encoder
based on 3 encoder-decoder models- LSTM, CNN and DNN
based autoencoders. We used a window size of 60 and the
input consisted of price windows for each asset. To get
better scalability of data, log returns were normalized by the
min-max scalar. The data was divided into 80-20% trainingtesting dataset.
The training process summary is as given in table 1,
for both training and testing data. Number of epochs were
decided on the convergence of mean-squared error of the
autoencoder i.e L = kX − X 0 k2 .
ZoomSVD Space- Storage-Phase of ZoomSVD required
dividing the input price data into blocks, and storing the
initial asset-price matrix in compressed SVD form. The block
parameter b was taken as 60, but is usually decided on
storage power of systems for large datasets. SVD of each
block is stored by incremental-SVD. As our time-range query
is passed by our RL agent, the SVD based state-space is
computed by Partial and stitched-SVD.

C. Training Step
For each individual RL agent, number of episodes were
decided on the convergence of percentage-distance from
equal-weighted portfolio returns. A stable learning rate of
0.01 of adam optimizer was used, with 32 as batch size. The
exploration probability is 20% initially and decreases after
every episode.
D. Comparison of LFSS module based RL Agent with
Baseline-RL agent
After setting up LFSS module and integrating it to policy
network, we backtested the results on given stock portfolio.
Table 10 gives comparison summary of different feature
extraction methods with one another and baseline-RL agent.
Initial portfolio investment is 10000$.
As shown in Table 10, adding cRBM and CNNAutoencoder based LFSS module to the agent leads
to a significant increase in different performance metric, while DNN-Autoencoder based LFSS module gives
slightly better performance than baseline-RL. ZoomSVD and

LSTM-Autoencoder state space performs roughly similar to
baseline-RL. Individual performance metric for each asset is
described below• Portfolio-Value(PV)- In terms of portfolio value over a
span of 4 years on training-set an initial investment of
10000$ reached roughly 39000$ for test-set for CNNAutoencoder and cRBM RL-agent, which performed
significantly better than baseline RL-agent. The addition of autoencoder and RBM based LFSS module
models the asset-market environment in a better way
than raw price data for the RL-agent. Deep-learning
based feature selection methods could model non-linear
features in asset-price data as compared to linear space
in ZoomSVD. Though cRBM RL agent did not give
high returns in the initial years, it had a sharp increase
in portfolio value in later years, performing much better
than other methods during this period.
• Sharpe-Ratio- Most of the models performed similarly
in terms of volatility, with returns varying. There was
no statistically significant difference in volatilities. This
implies the LFSS module added is in not learning
anything new to minimize risk in the model. This may
be due to the simplicity of reward function used, and
more work will be required to incorporate risk-adjusted
reward to handle volatility, which fits in with DDPG
algorithm.
• Sortino Ratio- Addition of LFSS module, reduces downside deviation to some extent to the RL agent, but
overall a more effective reward function can minimize
downward deviation further.

Fig. 10.

Baseline RL
DNNAutoencoder
RL
cRBM RL
ZoomSVD
RL
CNNAutoencoder
RL
LSTMAutoencoder
RL

Performance Summary
Portfolio- Annual
Annual
Value($) Returns
Volatility
29440
0.486
0.254
32960
0.575
0.252

SharpeRatio

SortinoRatio

1.91
2.28

2.12
2.51

37368
28343

0.684
0.458

0.256
0.257

2.67
1.78

2.92
2.03

38864

0.722

0.254

2.84

3.04

30121

0.503

0.255

1.97

2.13

TABLE II
P ERFORMANCE METRIC FOR DIFFERENT RL AGENTS

Portfolio-Value- In terms of portfolio value our RL
agent and the baseline-RL performs much better than all
the benchmarks used. This shows the superiority of deep
RL based methods for the task of portfolio optimization.
For the test set, equal-weighted portfolio gave better
results than all our benchmarks. IMVK based strategies
performed poorly on test set, mostly due to high change
in weights leading to high transaction cost. If we
consider the overall time period from 2007-2019 (fig
13), IMVK based strategies perform much better than
equal-weighted portfolio till 2017, followed by a large
drop in portfolio performance.
• Sharpe Ratio- Similar to high returns given by RL
agents, deep RL based approach gave much better
Sharpe Ratio than the benchmarks. Though there was
high difference in volatility, the risk was compensated
by the high returns.
• Sortino Ratio- Similar to Sharpe ratio, RL agents performed much better to all other benchmarks.
•

Comparison of various RL agents

E. Performace Measure w.r.t to Benchmarks
We used 5 non-RL based benchmarks for comparison of
our agent with standard portfolio optimization techniques.
The 5 benchmarks used were EW, S&P 500, WMAMR,
IMVK(Moderate and Aggressive Strategy) as explained in
the last section. We compare these benchmarks to one other
and to our RL-agent. The performance summary for all the
methods is as described in table 3.

Fig. 11.

Comparison of RL agent with various benchmarks

Portfolio
Value($)
Annual
Returns
Annual
Volatility
Sharpe
Ratio
Sortino
Ratio

Baseline- DNNcRBM
RL
Autoencoder RL
RL
29440
32960
37368

ZoomSVD
RL
28343

Performance Summary
CNNLSTMEW
Autoencoder Autoencoder
RL
RL
38864
30121
19456

0.486

0.575

0.684

0.458

0.722

0.503

0.254

0.252

0.256

0.257

0.254

1.91

2.28

2.67

1.78

2.12

2.51

2.92

2.03

Aggressive Moderate
Strategy
Strategy

S&P
500

WMAMR

13604

13144

15970

15524

0.235

0.0953

0.0785

0.149

0.152

0.255

0.144

0.162

0.154

0.128

0.144

2.84

1.97

1.63

0.59

0.51

1.16

1.06

3.04

2.13

1.84

0.78

0.69

1.37

1.26

TABLE III
P ERFORMANCE METRIC FOR ALL THE PROPOSED METHODS

Fig. 12.

Backtest Results for all methods proposed

VI. CONCLUSIONS
In this paper, we propose a RL agent for portfolio optimization with an added LFSS Module to obtain a novel state
space. We compared the performances of our RL agent with
already existing approaches for a portfolio of 15 S&P 500
stocks. Our agent performed much better than baseline-RL
agent and other benchmarks in terms of different metrics like
portfolio value, Sharpe Ratio, Sortino Ratio, etc. The main
catalyst for improvement in returns was due to the addition
of LFSS module which consisted of a time series filtering
unit and latent feature extractor unit, to get a compressed
latent feature state space for the policy network. We further
compared different feature space extracting methods like
autoencoders, ZoomSVD and cRBM. Deep learning based
techniques like CNN-Autoencoder and cRBM performed
effectively in obtaining a state-space for our RL agent,
that gave high returns, while minimizing risk. Additionally, we also introduced a new benchmark based on KdIndex technical-indicator, that gave comparable performance
to equal-weighted portfolio, and can be further used as a
benchmark for more portfolio optimization problems.
The work done in this paper can provide a flexible

Fig. 13. Comparison of Aggressive and Moderate Strategy IMVK with
Equal-Weighted Portfolio

modelling framework for various different state space with
more sophisticated deep learning models and with the advancements in RL algorithms our RL agent can be further
improved upon.
VII. F UTURE W ORK
Due to the flexible approach in state space modelling, there
are many ways to improve upon the performance of the RL
agent. To improve upon the quality of data through filtering
layer, non-linear filters can be applied like EKF, particle
filter, etc[32], which take into account the non-linearity in
the data or use an adaptive filter[452] that at each time
step uses most appropriate filter out of a set of filters. For
extracting latent features, cRBM can be further extended to
deep belief networks[43], which goes deeper than cRBM and
is trained using a similar approach . Since RL is highly
sensitive to the data used, our agent should be trained on
more datasets, which are more volatile than American stock
exchange data, as this will give a more robust performance.
Additionally, we can add data augmentation module[26] to
generate synthetic data, hence exposing the agent to more
data. Text data based embedding can also be added to the

state space[29], leveraging the use of NLP techniques.
In terms of deep RL algorithms, PPO was not used in this
work. Performance of LFSS module with PPO algorithm can
be further explored. This framework can be integrated with
appropriate risk-adjusted reward-function that minimizes the
risk of our asset allocation.
VIII. ACKNOWLEDGEMENT
This work was supported by Professor Nitin Gupta and
Professor Geetanjali Panda- Department of Mathematics. I
would like to thank them for helpful insights on financial
portfolio optimization, MDP and non-linear optimization,
which was essential in the progress of the project.
R EFERENCES
[1] Y. Feng and D. P. Palomar, “A Signal Processing Perspective
of Financial Engineering”, Foundations and Trends in Signal
Processing, vol. 9, no. 1-2, pp. 1-231, 2015.
[2] M. Japundžić, D. Jočić and I. Pavkov, "Application of stochastic
control theory to the optimal portfolio selection problem,"
2012 IEEE 10th Jubilee International Symposium on Intelligent
Systems and Informatics, Subotica, pp. 85-88, 2012.
[3] Kovalerchuk B., Vityaev E., Data Mining for Financial Applications. In: Maimon O., Rokach L. (eds) Data Mining and
Knowledge Discovery Handbook. Springer, Boston, MA, 2005.
[4] Heaton, J.; Polson, N.; and Witte, J. H., Deep learning for
finance: deep portfolios. Applied Stochastic Models in Business
and Industry 33(1):3–12, 2017.
[5] R. Asadi and A. Regan, “A spatial-temporal decomposition
based deep neural network for time series forecasting”, arXiv
preprint arXiv:1902.00636, 2019.
[6] Markowitz, H. M., Portfolio selection: Efficient diversification
of investment, 344 p, 1959.
[7] Haugh, M.B., L. Kogan and Z. Wu., Portfolio Optimization with
Position Constraints: an Approximate Dynamic Programming
Approach. Working paper. Columbia University and MIT, 2006.
[8] Cai, Y., K. L. Judd, and R. Xu., Numerical solution of dynamic
portfolio optimization with transaction costs, NBER Working
Paper No. w18709, 2013.
[9] Antonios Vogiatzis, Reinforcement Learning for Financial Portfolio Management- Diploma Thesis, 2019.
[10] Kober, J., Bagnell, J. A., Peters, J. Reinforcement learning
in robotics: A survey, The International Journal of Robotics
Research 32, 1238–1274, 2013.
[11] Y. Zhu, R. Mottaghi, E. Kolve, J. J. Lim, A. Gupta, L.
FeiFei, and A. Farhadi, Target-driven visual navigation in indoor scenes using deep reinforcement learning, arXiv preprint
arXiv:1609.05143, 2016.
[12] Szita I., Reinforcement Learning in Games, In: Wiering M., van
Otterlo M. (eds) Reinforcement Learning. Adaptation, Learning,
and Optimization, vol 12. Springer, Berlin, Heidelberg, 2012.
[13] Dixon M.F., Halperin I., Bilokon P., Applications of Reinforcement Learning. In: Machine Learning in Finance. Springer,
Cham, 2020.
[14] J. Ahmet Murat Ozbayoglu, Mehmet Ugur Gudelek, and Omer
Berat Sezer, Deep learning for financial applications: A survey.
working paper, 2019.
[15] Jiang, Zhengyao, Dixing Xu, and Jinjun Liang, ”A deep reinforcement learning framework for the financial portfolio management problem.” arXiv preprint arXiv:1706.10059, 2017.
[16] Liang, Z., Chen, H., et al, Adversarial Deep Reinforcement
Learning in Portfolio Management. arXiv:1808.09940, 2018.
[17] C. [Almahdi and Yang 2017] Almahdi, S., and Yang, S. Y.,
An adaptive portfolio trading system: A risk-return portfolio
optimization using recurrent reinforcement learning with expected maximum drawdown. Expert, Systems with Applications
87:267–279, 2017.
[18] J. Moody, M. Saffell, Y. Liao. Reinforcement learning for
trading systems and portfolios, In SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 129-140,
1998.

[19] Moody, J. and Saffell, M., Learning to trade via direct reinforcement, IEEE transactions on neural networks, 12 (4):875–889,
2001.
[20] Filos A., Reinforcement Learning for Portfolio Management[J].
arXiv preprint arXiv:1909.09571, 2019.
[21] J. Kanwar, N., et al, Deep Reinforcement Learning-based Portfolio Management. Ph.D. Dissertation, 2019.
[22] Y. Xu and G. Zhang, “Application of kalman filter in the prediction of stock price,” in International Symposium on Knowledge
Acquisition and Modeling (KAM), Atlantis press, pp. 197–198,
2015.
[23] G.E. Hinton and R.R. Salakhutdinov, Reducing the dimensionality of data with neural networks. Science, 313(5786):504–507,
2006.
[24] Langkvist, M., Modeling Time-Series with Deep Networks,
Ph.D. Thesis, Orebro University, Orebro, Sweden, 2014.
[25] J.-G. Jang, D. Choi, J. Jung and U. Kang, "Zoom-svd: Fast
and memory efficient method for extracting key patterns in an
arbitrary time range", CIKM, pp. 1083-1092, 2018.
[26] Yu, P., Lee, J. S., Kulyatin, I., Shi, Z., and Dasgupta S,
Model-based deep reinforcement learning for dynamic portfolio
optimization. arXiv preprint arXiv:1901.08740, 2019.
[27] Silver, David, et al., ”Deterministic policy gradient algorithms”,
ICML, 2014.
[28] Lin JL., Wu CH., Khomnotai L, Integration of Mean–Variance
Model and Stochastic Indicator for Portfolio Optimization. In:
Juang J., Chen CY., Yang CF. (eds) Proceedings of the 2nd
International Conference on Intelligent Technologies and Engineering Systems (ICITES2013), Lecture Notes in Electrical
Engineering, vol 293. Springer, Cham, 2014.
[29] Ye, Yunan, Hengzhi Pei, Boxin Wang, Pin-Yu Chen, Yada Zhu,
Jun Xiao, and Bo Li, Reinforcement-learning based portfolio
management with augmented asset movement prediction states,
AAAI, 2020.
[30] Sutton, R. and Barto, A. (1998). Reinforcement learning: An
introduction. Cambridge, MA, MIT Press, 1998.
[31] James Martin Rankin, Kalman filtering approach to market price
forecasting. PhD thesis, Iowa State University, 1986.
[32] Chen, Z., Bayesian filtering: From Kalman filters to particle
filters, and beyond. Tech. Rep. McMaster Univ., Canada, 2003.
[33] Vincent, P., Larochelle, H., Lajoie, I., Bengio, Y., Manzagol,
P.-A. Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion. J
Mach Learn Res 11, 3371–3408, 2010.
[34] Nigel Bosch, Unsupervised Deep Autoencoders for Feature Extraction with Educational Data, In Proceedings of the EDM 2017
Workshops and Tutorials co-located with the 10th International
Conference on Educational Data Mining, EDM, Urbana, IL,
USA, 2017.
[35] Vladimir Puzyrev. "Deep convolutional autoencoder for cryptocurrency market analysis," Papers 1910.12281, arXiv.org,
2019.
[36] Zhang W., Arvanitis A. and Al-Rasheed A., “Singular Value Decomposition and its numerical computations,” Michigan Technological University, Michigan, 2012.
[37] Hayashi, Isao and Jiang, Yinlai and Wang, Shuoyu, SVDBased Feature Extraction from Time-Series Motion Data and
Its Application to Gesture Recognition, Proceedings of the
8th International Conference on Bioinspired Information and
Communications Technologies:386-387, 2014.
[38] Salakhutdinov, R., Mnih, A., and Hinton, G. E. (2007), Restricted Boltzmann machines for collaborative filtering, In ICML
2007.
[39] R. Hrasko, A. G. C. Pacheco, and R. A. Krohling, “Time
Series Prediction Using Restricted Boltzmann Machines and
Backpropagation,” Procedia Computer Science, vol. 55, no.
Itqm, pp. 990–999, 2015.
[40] Kuremoto T., Kimura S., Kobayashi K., Obayashi M., Time
Series Forecasting Using Restricted Boltzmann Machine, In:
Huang DS., Gupta P., Zhang X., Premaratne P. (eds) Emerging
Intelligent Computing Technology and Applications, ICIC 2012,
Communications in Computer and Information Science, vol 304.
Springer, Berlin, Heidelberg, 2012.
[41] G. W. Taylor and G. E. Hinton, Factored conditional restricted
Boltzmann machines for modeling motion style, In ICML, 2009.

[42] Tulsyan A, Khare S, Huang B, Gopaluni B, Forbes F, A switching strategy for adaptive state estimation, Signal Processing
2018, 143:371–380, 2018.
[43] Y. Ren, J. Mao, Y. Liu, and Y. Li., ”A Novel DBN Model
for Time Series Forecasting.” ,IAENG International Journal of
Computer Science, vol. 44, no. 1, 2017.
[44] THaoran Wang and Xun Yu Zhou, Continuous-time meanvariance portfolio selection: A reinforcement learning framework. SSRN 3382932, 2019.
[45] Ta, V.-D.; Liu, C.-M.; Tadesse, D.A. Portfolio OptimizationBased Stock Prediction Using Long-Short Term Memory Network in Quantitative Trading, Appl. Sci. 10, 437, 2020.
</reference>

<statements>
1. memory-augmentation (appending lagged weights) improves stability under path-dependent costs.
</statements>

Begin the assessment now. Output only the JSON list, without any conversational text or explanations.