Solving long-run average reward robust MDPs via stochastic games

conference paper Solving long-run average reward robust MDPs via stochastic games published yes Krishnendu Chatterjee author 2E5DCA20-F248-11E8-B48F-1D18A9856A870000-0002-4561-241X Ehsan Kafshdar Goharshadi author 103b4fa0-896a-11ed-bdf8-87b697bef40d0000-0002-8595-0587 Mehrdad Karrabi author 67638922-f394-11eb-9cf6-f20423e08757 Petr Novotný author 3CC3B868-F248-11E8-B48F-1D18A9856A87 Dorde Zikelic author 294AA7A6-F248-11E8-B48F-1D18A9856A870000-0002-4681-1699 KrCh department IJCAI: International Joint Conference on Artificial Intelligence Formal Methods for Stochastic Models: Algorithms and Applications project Markov decision processes (MDPs) provide a standard framework for sequential decision making under uncertainty. However, MDPs do not take uncertainty in transition probabilities into account. Robust Markov decision processes (RMDPs) address this shortcoming of MDPs by assigning to each transition an uncertainty set rather than a single probability value. In this work, we consider polytopic RMDPs in which all uncertainty sets are polytopes and study the problem of solving long-run average reward polytopic RMDPs. We present a novel perspective on this problem and show that it can be reduced to solving long-run average reward turn-based stochastic games with finite state and action spaces. This reduction allows us to derive several important consequences that were hitherto not known to hold for polytopic RMDPs. First, we derive new computational complexity bounds for solving long-run average reward polytopic RMDPs, showing for the first time that the threshold decision problem for them is in NP∩CONP and that they admit a randomized algorithm with sub-exponential expected runtime. Second, we present Robust Polytopic Policy Iteration (RPPI), a novel policy iteration algorithm for solving long-run average reward polytopic RMDPs. Our experimental evaluation shows that RPPI is much more efficient in solving long-run average reward polytopic RMDPs compared to state-of-the-art methods based on value iteration. International Joint Conferences on Artificial Intelligence2024Jeju, South Korea eng 33rd International Joint Conference on Artificial Intelligence 1045-0823 9781956792041 2312.1391210.24963/ijcai.2024/741 6707-6715 Chatterjee K, Goharshady E, Karrabi M, Novotný P, Zikelic D. Solving long-run average reward robust MDPs via stochastic games. In: 33rd International Joint Conference on Artificial Intelligence. International Joint Conferences on Artificial Intelligence; 2024:6707-6715. doi:<a href="https://doi.org/10.24963/ijcai.2024/741">10.24963/ijcai.2024/741</a> Chatterjee, Krishnendu, Ehsan Goharshady, Mehrdad Karrabi, Petr Novotný, and Dorde Zikelic. “Solving Long-Run Average Reward Robust MDPs via Stochastic Games.” In 33rd International Joint Conference on Artificial Intelligence, 6707–15. International Joint Conferences on Artificial Intelligence, 2024. <a href="https://doi.org/10.24963/ijcai.2024/741">https://doi.org/10.24963/ijcai.2024/741</a>. Chatterjee, K., Goharshady, E., Karrabi, M., Novotný, P., & Zikelic, D. (2024). Solving long-run average reward robust MDPs via stochastic games. In 33rd International Joint Conference on Artificial Intelligence (pp. 6707–6715). Jeju, South Korea: International Joint Conferences on Artificial Intelligence. <a href="https://doi.org/10.24963/ijcai.2024/741">https://doi.org/10.24963/ijcai.2024/741</a> K. Chatterjee, E. Goharshady, M. Karrabi, P. Novotný, D. Zikelic, in:, 33rd International Joint Conference on Artificial Intelligence, International Joint Conferences on Artificial Intelligence, 2024, pp. 6707–6715. Chatterjee K, Goharshady E, Karrabi M, Novotný P, Zikelic D. 2024. Solving long-run average reward robust MDPs via stochastic games. 33rd International Joint Conference on Artificial Intelligence. IJCAI: International Joint Conference on Artificial Intelligence, 6707–6715. Chatterjee, Krishnendu, et al. “Solving Long-Run Average Reward Robust MDPs via Stochastic Games.” 33rd International Joint Conference on Artificial Intelligence, International Joint Conferences on Artificial Intelligence, 2024, pp. 6707–15, doi:<a href="https://doi.org/10.24963/ijcai.2024/741">10.24963/ijcai.2024/741</a>. K. Chatterjee, E. Goharshady, M. Karrabi, P. Novotný, and D. Zikelic, “Solving long-run average reward robust MDPs via stochastic games,” in 33rd International Joint Conference on Artificial Intelligence, Jeju, South Korea, 2024, pp. 6707–6715. 181602024-09-29T22:01:39Z2025-04-14T07:52:46Z