DEEP ACTOR–CRITIC METHODS FOR OPTIMAL TRADE EXECUTION UNDER UNCERTAIN MARKET IMPACT

Ayush Ojha

doi:10.5281/zenodo.23291906

Abstract

Market-impact misspecification can change the preferred distribution of trades over a liquidation horizon. We compare
deep deterministic policy gradient and twin delayed deep deterministic policy gradient under nominal and parameterrandomized training in a constrained single-asset simulator. Ten training seeds per group are evaluated against timeweighted execution and a fixed quadratic schedule. The study comprises forty training runs, 800,000 transitions and
344,064 held-out episode evaluations. Across seven prespecified nonnominal conditions, the randomized-minusnominal TD3 objective difference is −0.000658 (−0.17%); its paired training-seed 95% interval is [−0.01057, 0.00925]
and includes zero. The initial five-seed estimate changes substantially when five additional fixed seeds are included.
Learned policies approach the fixed quadratic schedule and improve substantially over time-weighted execution in the
nominal setting. Under the largest joint impact and liquidity shift, randomized TD3 improves on nominal TD3 but
remains worse than time-weighted execution. All evaluated policies complete liquidation. These results do not establish
a systematic benefit from parameter randomization or superiority to analytical control. The study uses synthetic paths,
immediate fills and a chosen price-risk penalty; it makes no claim about attainable market returns.

Keywords

Optimal execution; deep reinforcement learning; actor–critic methods; market impact; parameter uncertainty; financial technology.

References

1) Dimitris Bertsimas and Andrew W. Lo, “Optimal Control of Execution Costs,” Journal of Financial
Markets, vol. 1, no. 1, pp. 1–50, 1998. https://doi.org/10.1016/S1386-4181(97)00012-8
2) Robert Almgren and Neil Chriss, “Optimal Execution of Portfolio Transactions,” The Journal of Risk, vol. 3,
no. 2, pp. 5–39, 2001. https://doi.org/10.21314/jor.2001.041
3) Robert Almgren, “Optimal Trading with Stochastic Liquidity and Volatility,” SIAM Journal on Financial
Mathematics, vol. 3, no. 1, pp. 163–181, 2012. https://doi.org/10.1137/090763470
4) Yuriy Nevmyvaka, Yi Feng, and Michael Kearns, “Reinforcement Learning for Optimized Trade
Execution,” in Proceedings of the 23rd International Conference on Machine Learning (ICML ’06), pp.
673–680, 2006. https://doi.org/10.1145/1143844.1143929
5) Jean-Philippe Bouchaud, J. Doyne Farmer, and Fabrizio Lillo, “How Markets Slowly Digest Changes in
Supply and Demand,” in Handbook of Financial Markets: Dynamics and Evolution, pp. 57–160, 2009.
https://doi.org/10.1016/B978-012374258-2.50006-3
6) J. Doyne Farmer, Austin Gerig, Fabrizio Lillo, and Szabolcs Mike, “Market Efficiency and the LongMemory of Supply and Demand: Is Price Impact Variable and Permanent or Fixed and Temporary?”
Quantitative Finance, vol. 6, no. 2, pp. 107–112, 2006. https://doi.org/10.1080/14697680600668048
7) Bence Tóth, Yves Lempérière, Cyril Deremble, Joachim de Lataillade, Julien Kockelkoren, and JeanPhilippe Bouchaud, “Anomalous Price Impact and the Critical Nature of Liquidity in Financial Markets,”
Physical Review X, vol. 1, no. 2, article 021006, 2011. https://doi.org/10.1103/PhysRevX.1.021006
8) Elia Zarinelli, Michele Treccani, J. Doyne Farmer, and Fabrizio Lillo, “Beyond the Square Root: Evidence
for Logarithmic Dependence of Market Impact on Size and Participation Rate,” Market Microstructure and
Liquidity, vol. 1, no. 2, article 1550004, 2015. https://doi.org/10.1142/S2382626615500045
9) Emmanuel Bacry, Adrian Iuga, Matthieu Lasnier, and Charles-Albert Lehalle, “Market Impacts and the Life
Cycle of Investors Orders,” Market Microstructure and Liquidity, vol. 1, no. 2, article 1550009, 2015.
https://doi.org/10.1142/S2382626615500094
10) Dieter Hendricks and Diane Wilcox, “A Reinforcement Learning Extension to the Almgren-Chriss
Framework for Optimal Trade Execution,” in 2014 IEEE Conference on Computational Intelligence for
Financial Engineering & Economics (CIFEr), pp. 457–464, 2014.
https://doi.org/10.1109/CIFER.2014.6924109
11) David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller,
“Deterministic Policy Gradient Algorithms,” in Proceedings of the 31st International Conference on
Machine Learning, vol. 32, pp. 387–395, 2014. https://proceedings.mlr.press/v32/silver14.html
12) Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David
Silver, and Daan Wierstra, “Continuous Control with Deep Reinforcement Learning,” in Proceedings of the
4th International Conference on Learning Representations (ICLR), 2016. https://arxiv.org/abs/1509.02971v5
13) Scott Fujimoto, Herke van Hoof, and David Meger, “Addressing Function Approximation Error in ActorCritic Methods,” in Proceedings of the 35th International Conference on Machine Learning, vol. 80, pp.
1587–1596, 2018. https://proceedings.mlr.press/v80/fujimoto18a.html
14) Arnab Nilim and Laurent El Ghaoui, “Robust Control of Markov Decision Processes with Uncertain
Transition Matrices,” Operations Research, vol. 53, no. 5, pp. 780–798, 2005.
https://doi.org/10.1287/opre.1050.0216
15) Aravind Rajeswaran, Sarvjeet Ghotra, Balaraman Ravindran, and Sergey Levine, “EPOpt: Learning Robust
Neural Network Policies Using Model Ensembles,” in Proceedings of the 5th International Conference on
Learning Representations (ICLR), 2017. https://openreview.net/forum?id=SyWvgP5el
16) Xue Bin Peng, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel, “Sim-to-Real Transfer of
Robotic Control with Dynamics Randomization,” in 2018 IEEE International Conference on Robotics and
Automation (ICRA), pp. 3803–3810, 2018. https://doi.org/10.1109/ICRA.2018.8460528
17) Gerardo Hernández-del-Valle and Carlos Pacheco-González, “Optimal Execution of Portfolio Transactions
with Geometric Price Process,” arXiv:0908.1211, 2009. https://arxiv.org/abs/0908.1211v2
18) Takashi Kato, “An Optimal Execution Problem with Market Impact,” Finance and Stochastics, vol. 18, no.
3, pp. 695–732, 2014. https://doi.org/10.1007/s00780-014-0232-0
19) Anna A. Obizhaeva and Jiang Wang, “Optimal Trading Strategy and Supply/Demand Dynamics,” Journal of
Financial Markets, vol. 16, no. 1, pp. 1–32, 2013. https://doi.org/10.1016/j.finmar.2012.09.001
20) Aurélien Alfonsi, Antje Fruth, and Alexander Schied, “Optimal Execution Strategies in Limit Order Books
with General Shape Functions,” Quantitative Finance, vol. 10, no. 2, pp. 143–157, 2010.
https://doi.org/10.1080/14697680802595700
21) Weibing Huang, Charles-Albert Lehalle, and Mathieu Rosenbaum, “Simulating and Analyzing Order Book
Data: The Queue-Reactive Model,” Journal of the American Statistical Association, vol. 110, no. 509, pp.
107–122, 2015. https://doi.org/10.1080/01621459.2014.982278
22) Takashi Kato, “VWAP Execution as an Optimal Strategy,” JSIAM Letters, vol. 7, pp. 33–36, 2015.
https://doi.org/10.14495/jsiaml.7.33
23) Takashi Kato, “An Optimal Execution Problem in the Volume-Dependent Almgren–Chriss Model,”
Algorithmic Finance, vol. 7, nos. 1–2, pp. 1–14, 2018. https://doi.org/10.3233/AF-170198
24) Álvaro Cartea and Sebastian Jaimungal, “Incorporating Order-Flow into Optimal Execution,” Mathematics
and Financial Economics, vol. 10, no. 3, pp. 339–364, 2016. https://doi.org/10.1007/s11579-016-0162-z
25) Olivier Guéant, Charles-Albert Lehalle, and Joaquin Fernandez-Tapia, “Optimal Portfolio Liquidation with
Limit Orders,” SIAM Journal on Financial Mathematics, vol. 3, no. 1, pp. 740–764, 2012.
https://doi.org/10.1137/110850475
26) Rama Cont, Arseniy Kukanov, and Sasha Stoikov, “The Price Impact of Order Book Events,” Journal of
Financial Econometrics, vol. 12, no. 1, pp. 47–88, 2014. https://doi.org/10.1093/jjfinec/nbt003
27) Zoltán Eisler, Jean-Philippe Bouchaud, and Julien Kockelkoren, “Models for the Impact of All Order Book
Events,” in Market Microstructure, pp. 113–135, 2012. https://doi.org/10.1002/9781118673553.ch5
28) Gianbiagio Curato, Jim Gatheral, and Fabrizio Lillo, “Optimal Execution with Non-Linear Transient Market
Impact,” Quantitative Finance, vol. 17, no. 1, pp. 41–54, 2017.
https://doi.org/10.1080/14697688.2016.1181274
29) J. Donier, J. Bonart, I. Mastromatteo, and J.-P. Bouchaud, “A Fully Consistent, Minimal Model for NonLinear Market Impact,” Quantitative Finance, vol. 15, no. 7, pp. 1109–1121, 2015.
https://doi.org/10.1080/14697688.2015.1040056
30) M. Schneider and F. Lillo, “Cross-Impact and No-Dynamic-Arbitrage,” Quantitative Finance, advance
online publication, 2018. https://doi.org/10.1080/14697688.2018.1467033
31) Thomas Spooner, John Fearnley, Rahul Savani, and Andreas Koukorinis, “Market Making via
Reinforcement Learning,” in Proceedings of the 17th International Conference on Autonomous Agents and
Multiagent Systems (AAMAS), pp. 434–442, 2018.
https://www.ifaamas.org/Proceedings/aamas2018/pdfs/p434.pdf
32) Ayush Ojha, “Learned Review Policies Versus Score Thresholds for Budgeted Fraud Alert Allocation,”
International Journal of Engineering Technology Research & Management, vol. 2, no. 2, pp. 98–109,
February 2018. https://doi.org/10.5281/zenodo.23274871
33) Ayush Ojha, “Transaction Costs and Downside Risk in Deep Reinforcement Learning for Portfolio
Allocation,” International Journal of Engineering Technology Research & Management, vol. 1, no. 9, pp.
19–31, September 2017. https://doi.org/10.5281/zenodo.23212855
34) Alexander Schied, “Robust Strategies for Optimal Order Execution in the Almgren–Chriss Framework,”
Applied Mathematical Finance, vol. 20, no. 3, pp. 264–286, 2013.
https://doi.org/10.1080/1350486X.2012.683963
35) Álvaro Cartea, Ryan Donnelly, and Sebastian Jaimungal, “Algorithmic Trading with Model Uncertainty,”
SIAM Journal on Financial Mathematics, vol. 8, no. 1, pp. 635–671, 2017.
https://doi.org/10.1137/16M106282X
36) Richard S. Sutton, “Learning to Predict by the Methods of Temporal Differences,” Machine Learning, vol.
3, no. 1, pp. 9–44, 1988. https://doi.org/10.1007/BF00115009
37) Christopher J. C. H. Watkins and Peter Dayan, “Q-Learning,” Machine Learning, vol. 8, nos. 3–4, pp. 279–
292, 1992. https://doi.org/10.1007/BF00992698
38) Long-Ji Lin, “Self-Improving Reactive Agents Based on Reinforcement Learning, Planning and Teaching,”
Machine Learning, vol. 8, no. 3, pp. 293–321, 1992. https://doi.org/10.1007/BF00992699
39) Volodymyr Mnih et al., “Human-Level Control Through Deep Reinforcement Learning,” Nature, vol. 518,
no. 7540, pp. 529–533, 2015. https://doi.org/10.1038/nature14236
40) Ronald J. Williams, “Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement
Learning,” Machine Learning, vol. 8, no. 3, pp. 229–256, 1992. https://doi.org/10.1007/BF00992696
41) Richard S. Sutton, David McAllester, Satinder Singh, and Yishay Mansour, “Policy Gradient Methods for
Reinforcement Learning with Function Approximation,” in Advances in Neural Information Processing
Systems, vol. 12, 1999.
https://proceedings.neurips.cc/paper/1999/hash/464d828b85b0bed98e80ade0a5c43b0f-Abstract.html
42) Vijay R. Konda and John N. Tsitsiklis, “Actor-Critic Algorithms,” in Advances in Neural Information
Processing Systems, vol. 12, 1999. https://papers.neurips.cc/paper/1786-actor-critic-algorithms
43) Hado van Hasselt, “Double Q-Learning,” in Advances in Neural Information Processing Systems, vol. 23,
2010. https://proceedings.neurips.cc/paper/2010/hash/091d584fced301b442654dd8c23b3fc9-Abstract.html
44) Hado van Hasselt, Arthur Guez, and David Silver, “Deep Reinforcement Learning with Double QLearning,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 30, no. 1, pp. 2094–2100,
2016. https://doi.org/10.1609/aaai.v30i1.10295
45) John Schulman, Philipp Moritz, Sergey Levine, Michael I. Jordan, and Pieter Abbeel, “High-Dimensional
Continuous Control Using Generalized Advantage Estimation,” in Proceedings of the 4th International
Conference on Learning Representations (ICLR), 2016. https://arxiv.org/abs/1506.02438v5
46) John Schulman, Sergey Levine, Philipp Moritz, Michael Jordan, and Pieter Abbeel, “Trust Region Policy
Optimization,” in Proceedings of the 32nd International Conference on Machine Learning, vol. 37, pp.
1889–1897, 2015. https://proceedings.mlr.press/v37/schulman15.html
47) John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov, “Proximal Policy
Optimization Algorithms,” arXiv:1707.06347, 2017. https://arxiv.org/abs/1707.06347v2
48) Volodymyr Mnih et al., “Asynchronous Methods for Deep Reinforcement Learning,” in Proceedings of the
33rd International Conference on Machine Learning, vol. 48, pp. 1928–1937, 2016.
https://proceedings.mlr.press/v48/mniha16.html
49) Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine, “Reinforcement Learning with Deep
Energy-Based Policies,” in Proceedings of the 34th International Conference on Machine Learning, vol. 70,
pp. 1352–1361, 2017. https://proceedings.mlr.press/v70/haarnoja17a.html
50) Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine, “Soft Actor-Critic: Off-Policy
Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor,” in Proceedings of the 35th
International Conference on Machine Learning, vol. 80, pp. 1861–1870, 2018.
https://proceedings.mlr.press/v80/haarnoja18b.html
51) Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel, “Benchmarking Deep
Reinforcement Learning for Continuous Control,” in Proceedings of the 33rd International Conference on
Machine Learning, vol. 48, pp. 1329–1338, 2016. https://proceedings.mlr.press/v48/duan16.html
52) Diederik P. Kingma and Jimmy Lei Ba, “Adam: A Method for Stochastic Optimization,” in Proceedings of
the 3rd International Conference on Learning Representations (ICLR), 2015.
https://arxiv.org/abs/1412.6980v9