You will be provided with a reference and some statements. Please determine whether each statement is 'supported', 'unsupported', or 'unknown' with respect to the reference. Please note:
First, assess whether the reference contains any valid content. If the reference contains no valid information, such as a 'page not found' message, then all statements should be considered 'unknown'.
If the reference is valid, for a given statement: if the facts or data it contains can be found entirely or partially within the reference, it is considered 'supported' (data accepts rounding); if all facts and data in the statement cannot be found in the reference, it is considered 'unsupported'.

You should return the result in a JSON list format, where each item in the list contains the statement's index and the judgment result, for example:
[
    {
        "idx": 1,
        "result": "supported"
    },
    {
        "idx": 2,
        "result": "unsupported"
    }
]

Below are the reference and statements:
<reference>
[Literature Review] Combining Transformer based Deep Reinforcement Learning with
  Black-Litterman Model for Portfolio Optimization

This paper presents a novel portfolio optimization model that combines **Deep Reinforcement Learning (DRL)** with the **Black-Litterman (BL) model**. The core idea is to train a **DRL** agent to intelligently apply the **BL model** for determining optimal portfolio weights, especially in markets allowing short selling. This approach aims to enable the **DRL** agent to learn and exploit the dynamic correlations between asset returns, a crucial aspect often overlooked in traditional **DRL** portfolio optimization frameworks.

Here's a breakdown of the paper's key aspects with more technical detail:

**1. Problem Statement and Motivation:**

*   Traditional **DRL** agents for portfolio optimization typically learn a policy to directly determine portfolio weights based on an environmental reward function.  This often neglects the dynamic correlations between asset returns, which are essential for effective long/short strategies.
*   **Markowitz's Mean-Variance Model** considers these correlations, but can suffer from "error maximization," leading to overweighting of assets with estimation errors.
*   The **BL model**, a Bayesian model, offers a solution by combining subjective investor views with historical data to derive a more robust posterior distribution of asset returns.

**2. Proposed Solution (BDA - Black-Litterman Model Based Deep Reinforcement Learning Agent):**

*   The paper proposes training a **DRL** agent to *use* the **BL model** to determine portfolio weights. The **DRL** agent learns a policy function to provide the subjective views (expected returns and risk aversion) required by the **BL model**.
*   The **DRL** agent uses a **Transformer neural network** and a convolutional neural network to output the return expectation and risk aversion in the **BL model**.  The prior distribution parameters in the **BL model** are calculated based on historical data. By removing the position encoding module, we let our **BDA** concentrate on learning the nonlinear correlation from multiple concurrent series of portfolio asset returns when determining the subjective view for the expected return.
*   The **BL model** then combines these subjective views with historical return data to generate a posterior distribution of asset returns.  This posterior distribution is used to solve a mean-variance optimization problem, resulting in the target portfolio weights.
*   This design allows the **DRL** agent to indirectly learn the dynamic correlations between asset returns and implement effective long/short strategies based on these correlations.

**3. Core Methodology (Detailed):**

*   **Markov Decision Process (MDP) Formulation:** The portfolio optimization problem is framed as a finite **MDP**, defined by the tuple `<S, A, P, R>`.

    *   *S* (State Space): The state *s\_t* at time *t* includes the portfolio weights from the previous period and a historical return tensor. The historical return tensor contains price fluctuation data over several past trading periods.
    *   *A* (Action Space): The action *a\_t* is defined as the target portfolio weights, representing the proportion of investment in each asset.  Since the **DRL** agent is controlling the parameters *within* the **BL model**, the final portfolio weights are indirectly determined by the action.
    *   *P* (Transition Dynamics): *P(s\_t+1 | s\_t, a\_t)* describes the probability of transitioning to the next state *s\_t+1* given the current state *s\_t* and action *a\_t*.  This is governed by the market dynamics and asset price movements.
    *   *R* (Reward Function): The reward function *R(s\_t+1, a\_t)* is designed to encourage profitable and risk-aware trading.  It is based on the daily portfolio return, adjusted by the portfolio's variance and the transaction scale.  This encourages the agent to balance return with risk and minimize transaction costs.

*   **Black-Litterman Model Integration:** The **BL model** is used as a core component of the action selection process.

    *   *Prior Distribution:* A prior distribution of excess returns is established based on historical data. The expected value and covariance matrix of the prior distribution are calculated based on historical data and the inverse optimization method, ensuring a reasonable starting point.
    *   *Subjective Views:* The **DRL** agent's neural network outputs subjective views on the expected returns of the assets.  These views are expressed as linear equations, representing the agent's belief about the expected performance of specific assets or portfolios.
    *   *Posterior Distribution:* The **BL model** combines the prior distribution with the subjective views using Bayes' theorem to create a posterior distribution of expected returns. This posterior distribution reflects a more informed and robust estimate of asset returns.
    *   *Optimization:* Using the posterior distribution of expected excess returns derived by the **BL model**, a mean-variance optimization problem is solved to determine the optimal portfolio weights.

*   **Policy Gradient Optimization (Modified):** The paper uses a policy gradient method to train the **DRL** agent, but with a key modification to address challenges related to the high-dimensional action space and overfitting.

    *   *Direct Objective Function Formulation:* Instead of training a critic network (as in traditional actor-critic methods), the paper directly formulates an objective function based on the environment's reward function.  This avoids the "curse of dimensionality" that can plague critic network training in high-dimensional spaces.
    *   *Target Value Setting:* The objective function is *not* directly maximized.  Instead, a *target value* is set for an evaluation function derived from the reward function. The training goal is for the agent's policy to reach this target value.
    *   *Theoretical Optimal Portfolio Weights:* We compute the theoretically optimal portfolio weights under the unconstrained portfolio optimization problem with risk aversion to calculate the target value.
    *   *Policy Update:* The policy function's parameters are updated by propagating the *analytic gradients* of the training objective function back into the neural network.  This direct gradient propagation ensures efficient and stable learning.

*   **Neural Network Architecture:**

    *   *Transformer Network:* A **Transformer** network (without positional encoding) is used to learn the agent's subjective views. The Transformer architecture is well-suited for capturing long-range dependencies in the time series data of asset returns.
    *   *Convolutional Network (CNN):* A **CNN** is used to determine the agent's risk aversion.

**4. Experiments and Results:**

*   The proposed **BDA** is tested on real-world US stock market data, using the constituent stocks of the **Dow Jones Industrial Average (DJIA)**.
*   The performance of the **BDA** is compared against various portfolio choice strategies, including:

    *   Traditional financial strategies (e.g., constant rebalanced portfolios, mean-variance optimization).
    *   Deep learning strategies (e.g., models that predict returns and use a TopK-Drop strategy).
    *   Other DRL strategies (e.g., DDPG, PPO, SAC, A2C, TD3).

*   The results demonstrate that the **BDA** significantly outperforms the comparison strategies in terms of accumulated return and return per unit of risk (Sharpe Ratio, Sortino Ratio).

**5. Key Contributions:**

*   **Novel Combination:** The paper presents a novel and effective combination of **DRL** and the **BL model** for portfolio optimization, particularly in markets allowing short selling.
*   **Improved Risk-Adjusted Returns:** The **BDA** is shown to learn the dynamic correlations between asset returns and implement effective long/short strategies, leading to higher risk-adjusted returns.
*   **Efficient Training Method:** The paper introduces a modified policy gradient approach that avoids the limitations of traditional actor-critic methods in high-dimensional action spaces.
*   **Generalization Ability:** The use of a Transformer network without positional encoding helps to improve the generalization ability of the **DRL** agent's policy, mitigating overfitting.

**Technical Terms:**

*   **DRL:** Deep Reinforcement Learning
*   **BL Model:** Black-Litterman Model
*   **MDP:** Markov Decision Process
*   **Sharpe Ratio (SR):** Risk-adjusted measure of return.
*   **Sortino Ratio (STR):**  Measure of risk-adjusted return that only considers downside risk.
*   **DJIA:** Dow Jones Industrial Average
*   **DDPG:** Deep Deterministic Policy Gradient
*   **PPO:** Proximal Policy Optimization
*   **SAC:** Soft Actor-Critic
*   **A2C:** Advantage Actor Critic
*   **TD3:** Twin Delayed Deep Deterministic Policy Gradient
*   **CNN:** Convolutional Neural Network
*   **Transformer Network:** A type of neural network architecture particularly well-suited for sequential data.

In essence, this paper proposes a clever way to inject financial expertise (through the **BL model**) into a **DRL** agent, allowing it

open navigation menu

Features
Pricing
FAQ
Blog
Explore Literature
Events
open navigation menu

Features
Pricing
FAQ
Blog
Explore Literature
Events
EN
Upload Paper
Get Started Free
Sign in
This page provides the most accurate and concise summary worldwide for the paper titled Combining Transformer based Deep Reinforcement Learning with
Black-Litterman Model for Portfolio Optimization. With Moonlight, your AI research colleague, you can effortlessly and quickly grasp all the papers you read. Install the Chrome extension from https://www.themoonlight.io/ or directly upload files on the web. Moonlight offers features tailored to your needs: - Text Explanation: AI simplifies complex concepts and paragraphs. - Image Explanation: One-click explanations of images, tables, and formulas. - AI Chat: Engage with AI to dive deeper into paper discussions. - Smart Citations: Instantly view information (title, author, summary) of cited papers without scrolling to the References. - Translation: Quickly translate unfamiliar words, sentences, or the entire page. - Auto Highlight: AI highlights key points automatically, helping you identify originality, methods, and results swiftly. - External Link Explanation: AI analyzes external sources and explains their relevance to your document. - Markup: Highlight important sentences and annotate to create personalized research notes. - Save and Share: Store documents in your library and easily share them. - Scholar Deep Search: Receive recommendations of relevant papers based on your stored documents.
Original Paper
Review
Explore Literature
Computer Science
[Literature Review] Combining Transformer based Deep Reinforcement Learning with
Black-Litterman Model for Portfolio Optimization
Ruoyu Sun, Angelos Stefanidis + 2 more
|
February 2024
TL;DR
Master dynamic portfolio optimization: This novel DRL-Black-Litterman fusion learns correlations for superior risk-adjusted returns.
Open with AI Viewer
This paper presents a novel portfolio optimization model that combines
Deep Reinforcement Learning (DRL)
with the
Black-Litterman (BL) model
. The core idea is to train a
DRL
agent to intelligently apply the
BL model
for determining optimal portfolio weights, especially in markets allowing short selling. This approach aims to enable the
DRL
agent to learn and exploit the dynamic correlations between asset returns, a crucial aspect often overlooked in traditional
DRL
portfolio optimization frameworks.

Here's a breakdown of the paper's key aspects with more technical detail:

1. Problem Statement and Motivation:

Traditional
DRL
agents for portfolio optimization typically learn a policy to directly determine portfolio weights based on an environmental reward function. This often neglects the dynamic correlations between asset returns, which are essential for effective long/short strategies.

Markowitz's Mean-Variance Model
considers these correlations, but can suffer from "error maximization," leading to overweighting of assets with estimation errors.

The
BL model
, a Bayesian model, offers a solution by combining subjective investor views with historical data to derive a more robust posterior distribution of asset returns.

2. Proposed Solution (BDA - Black-Litterman Model Based Deep Reinforcement Learning Agent):

The paper proposes training a
DRL
agent to
use
the
BL model
to determine portfolio weights. The
DRL
agent learns a policy function to provide the subjective views (expected returns and risk aversion) required by the
BL model
.

The
DRL
agent uses a
Transformer neural network
and a convolutional neural network to output the return expectation and risk aversion in the
BL model
. The prior distribution parameters in the
BL model
are calculated based on historical data. By removing the position encoding module, we let our
BDA
concentrate on learning the nonlinear correlation from multiple concurrent series of portfolio asset returns when determining the subjective view for the expected return.

The
BL model
then combines these subjective views with historical return data to generate a posterior distribution of asset returns. This posterior distribution is used to solve a mean-variance optimization problem, resulting in the target portfolio weights.

This design allows the
DRL
agent to indirectly learn the dynamic correlations between asset returns and implement effective long/short strategies based on these correlations.

3. Core Methodology (Detailed):

Markov Decision Process (MDP) Formulation:
The portfolio optimization problem is framed as a finite
MDP
, defined by the tuple
<S, A, P, R>
.

S
(State Space): The state
s_t
at time
t
includes the portfolio weights from the previous period and a historical return tensor. The historical return tensor contains price fluctuation data over several past trading periods.

A
(Action Space): The action
a_t
is defined as the target portfolio weights, representing the proportion of investment in each asset. Since the
DRL
agent is controlling the parameters
within
the
BL model
, the final portfolio weights are indirectly determined by the action.

P
(Transition Dynamics):
P(s_t+1 | s_t, a_t)
describes the probability of transitioning to the next state
s_t+1
given the current state
s_t
and action
a_t
. This is governed by the market dynamics and asset price movements.

R
(Reward Function): The reward function
R(s_t+1, a_t)
is designed to encourage profitable and risk-aware trading. It is based on the daily portfolio return, adjusted by the portfolio's variance and the transaction scale. This encourages the agent to balance return with risk and minimize transaction costs.

Black-Litterman Model Integration:
The
BL model
is used as a core component of the action selection process.

Prior Distribution:
A prior distribution of excess returns is established based on historical data. The expected value and covariance matrix of the prior distribution are calculated based on historical data and the inverse optimization method, ensuring a reasonable starting point.

Subjective Views:
The
DRL
agent's neural network outputs subjective views on the expected returns of the assets. These views are expressed as linear equations, representing the agent's belief about the expected performance of specific assets or portfolios.

Posterior Distribution:
The
BL model
combines the prior distribution with the subjective views using Bayes' theorem to create a posterior distribution of expected returns. This posterior distribution reflects a more informed and robust estimate of asset returns.

Optimization:
Using the posterior distribution of expected excess returns derived by the
BL model
, a mean-variance optimization problem is solved to determine the optimal portfolio weights.

Policy Gradient Optimization (Modified):
The paper uses a policy gradient method to train the
DRL
agent, but with a key modification to address challenges related to the high-dimensional action space and overfitting.

Direct Objective Function Formulation:
Instead of training a critic network (as in traditional actor-critic methods), the paper directly formulates an objective function based on the environment's reward function. This avoids the "curse of dimensionality" that can plague critic network training in high-dimensional spaces.

Target Value Setting:
The objective function is
not
directly maximized. Instead, a
target value
is set for an evaluation function derived from the reward function. The training goal is for the agent's policy to reach this target value.

Theoretical Optimal Portfolio Weights:
We compute the theoretically optimal portfolio weights under the unconstrained portfolio optimization problem with risk aversion to calculate the target value.

Policy Update:
The policy function's parameters are updated by propagating the
analytic gradients
of the training objective function back into the neural network. This direct gradient propagation ensures efficient and stable learning.

Neural Network Architecture:

Transformer Network:
A
Transformer
network (without positional encoding) is used to learn the agent's subjective views. The Transformer architecture is well-suited for capturing long-range dependencies in the time series data of asset returns.

Convolutional Network (CNN):
A
CNN
is used to determine the agent's risk aversion.

4. Experiments and Results:

The proposed
BDA
is tested on real-world US stock market data, using the constituent stocks of the
Dow Jones Industrial Average (DJIA)
.

The performance of the
BDA
is compared against various portfolio choice strategies, including:

Traditional financial strategies (e.g., constant rebalanced portfolios, mean-variance optimization).

Deep learning strategies (e.g., models that predict returns and use a TopK-Drop strategy).

Other DRL strategies (e.g., DDPG, PPO, SAC, A2C, TD3).

The results demonstrate that the
BDA
significantly outperforms the comparison strategies in terms of accumulated return and return per unit of risk (Sharpe Ratio, Sortino Ratio).

5. Key Contributions:

Novel Combination:
The paper presents a novel and effective combination of
DRL
and the
BL model
for portfolio optimization, particularly in markets allowing short selling.

Improved Risk-Adjusted Returns:
The
BDA
is shown to learn the dynamic correlations between asset returns and implement effective long/short strategies, leading to higher risk-adjusted returns.

Efficient Training Method:
The paper introduces a modified policy gradient approach that avoids the limitations of traditional actor-critic methods in high-dimensional action spaces.

Generalization Ability:
The use of a Transformer network without positional encoding helps to improve the generalization ability of the
DRL
agent's policy, mitigating overfitting.

Technical Terms:

DRL:
Deep Reinforcement Learning

BL Model:
Black-Litterman Model

MDP:
Markov Decision Process

Sharpe Ratio (SR):
Risk-adjusted measure of return.

Sortino Ratio (STR):
Measure of risk-adjusted return that only considers downside risk.

DJIA:
Dow Jones Industrial Average

DDPG:
Deep Deterministic Policy Gradient

PPO:
Proximal Policy Optimization

SAC:
Soft Actor-Critic

A2C:
Advantage Actor Critic

TD3:
Twin Delayed Deep Deterministic Policy Gradient

CNN:
Convolutional Neural Network

Transformer Network:
A type of neural network architecture particularly well-suited for sequential data.

In essence, this paper proposes a clever way to inject financial expertise (through the
BL model
) into a
DRL
agent, allowing it

Was this review helpful?
Helpful
Not helpful
Open with AI Viewer
Save
View More Reviews
Related Papers
Go to library
Time Series Supplier Allocation via Deep Black-Litterman Model
Solve complex supplier allocation: the first to fuse finance models with STGNNs for robust, data-sparse decisions.
View Review
·
Open original
Deep Reinforcement Learning for Reliability Based Bi-Objective Portfolio Optimization
Explore how integrating Extreme Value Theory and DRL enables robust portfolio management under probabilistic reliability constraints in volatile markets.
View Review
·
Open original
Enhancing Portfolio Optimization with Deep Learning Insights
Transformer networks unlock robust portfolio optimization, decisively outperforming LSTMs and benchmarks even in volatile market regimes.
View Review
·
Open original
A novel multi-agent dynamic portfolio optimization learning system based
on hierarchical deep reinforcement learning
Unlock superior risk-adjusted portfolio returns by conquering DRL's challenges with this novel hierarchical learning system.
View Review
·
Open original
Latent Variable Estimation in Bayesian Black-Litterman Models
Revolutionize portfolio optimization: Learn latent investor views from market data to achieve 50% higher Sharpe ratios.
View Review
·
Open original
AI One-Click Summary
Real-time Translation
AI Q&A
With Free AI PDF Viewer
Revolutionize Your Paper Reading
Open This Paper in AI Viewer Now

Terms of Use
Privacy Policy
Medium
GitHub
LinkedIn
Email
Corca, Inc. / CEO Younghyun Chung / Business Registration Number 271-86-02206
6F, 11-8 Teheran-ro 77-gil, Gangnam-gu, Seoul, Republic of Korea, 06159
Contact 02-6925-6978 E-mail: moonlight@corca.ai
© 2026 Corca, Inc. All rights reserved.
</reference>

<statements>
1. Concurrently, Deep Reinforcement Learning (DRL) formalizes portfolio selection as a sequential Markov Decision Process (MDP)
2. An autonomous agent observes a state vector \(s_t\) composed of asset price series, order book dynamics, and economic covariates, and directly maps these observations to an action vector representing portfolio weights \(a_t = w_t \in \Delta^N\) using parameterized policy networks
3. The policy parameters are optimized via policy gradient methods, Actor-Critic variants, or Soft Actor-Critic algorithms to maximize cumulative expected reward functions parameterized by downstream financial objectives, such as the differential Sharpe ratio or risk-adjusted wealth accumulation
4. Despite their capacity to model non-linear relationships, pure deep learning allocators face significant practical hurdles
5. In direct policy parameterization, deep neural networks or reinforcement learning agents output portfolio weight vectors directly via normalized Softmax output layers
6. In the Pure Deep Learning (DRL / DNN) framework, the Asset Allocation Engine uses direct parameterized policy networks (Softmax) or actor-critic agents
7. In the Pure Deep Learning (DRL / DNN) framework, Tail Risk & Non-Linearity captures complex non-linear interactions, but risks uncalibrated tail behavior
8. An agent trained predominantly during a sustained bull market with low macroeconomic volatility learns aggressive long-bias policies
9. When confronted with structural shifts—such as sudden liquidity contractions, unexpected rate hikes, or flight-to-safety episodes—the learned policy network often degrades, incurring severe drawdowns because it lacks the baseline conservatism provided by equilibrium anchors
10. These representations are concatenated with normalized numerical features and passed into a spatio-temporal attention network that combines a Transformer with a Spatio-Temporal Graph Neural Network
</statements>

Begin the assessment now. Output only the JSON list, without any conversational text or explanations.