A Markovian Decision Process

By Richard Bellman. 1957

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Game & Decision Theory

Citations in the corpus

Off-Policy Actor-Critic with Shared Experience Replay (2019, background): “In the fully observable case, the RL problem is formalized as a Markov Decision Process (Bellman, 1957): a tuple (S,A, p, γ), where S,A denotes finite sets of states Figure 1.”
Kernel Mean Embedding of Distributions: A Review and Beyonds (2016, methodology): “While the ordinary Bellman operator (4.24) is contractive and isotonic which guarantee that the value iteration converges monotonically (Porta et al. 2006), the kernel Bellman operator in (4.25) may not have these two properties due to the unnormalized coefficients of the conditional mean embedding ”
Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models (2018, methodology): “Adhering to the Markov decision process formulation [42], we denote the state s ∈ Rs and the actions a ∈ Ra of the system, the reward function r(s,a), and we consider the dynamic systems governed by the transition function fθ : R ds+da 7→ Rs such that given the current state st and current input at,”
Same State, Different Task: Continual Reinforcement Learning without Interference (2021, background): “A Markov Decision Process (MDP, (Bellman 1957)) is a tuple (S,A, P,R, γ).”
Trajectory-wise Control Variates for Variance Reduction in Policy Gradient Methods (2019, background): “We show that when dynamics is the MDP is unknown, an ordering is feasible to implement, if and only if,Rk appears before Sk+1..h for all t ≤ k h.”
Safe Policy Improvement with Baseline Bootstrapping (2017, methodology): “A proper introduction to Markov Decision Processes (Bell-man, 1957, MDPs) and Reinforcement Learning (Sutton & Barto, 1998, RL) is available in Appendix A.1.”
Acme: A Research Framework for Distributed Reinforcement Learning (2020, methodology): “(2) This construction is commonly referred to as the Bellman equation (Bellman, 1957) and a similar approach can also be used for V.”
Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels (2020, methodology): “Reinforcement Learning from Images We formulate image-based control as a ﬁnite-horizon partially observable Markov decision process (POMDP) [6, 27], instantiated by an episodic environment with discrete time step t ∈ [1 , T ] , where T is ﬁxed.”
Cognitive Mapping and Planning for Visual Navigation (2017, background): “…based on value iteration networks proposed by Tamar et al. (2016), who observed that a particular type of planning algorithm called value iteration (Bellman 1957) can be implemented as a neural networkwith alternating convolutions and channel-wise max pooling operations, allowing the planner to be…”
Multi-task Deep Reinforcement Learning with PopArt (2018, background): “The RL problem is often formalised as a Markov decision process (Bellman 1957): a tuple ( S , A , p, γ ) , where S , A are ﬁnite sets of states and actions , p denotes the dynamics, such that p ( r, s (cid:48) | s, a ) is the probability of observing reward r and state s (cid:48) when executing…”
A survey and critique of multiagent deep reinforcement learning (2018, neutral): “ected discounted sum of rewards. There are dierent techniques for solving MDPs assuming a complete description of all its elements. One of the most common techniques is the value iteration algorithm [44], which requires a complete and accurate representation of states, actions, rewards, and transiti”
Averaged-DQN: Variance Reduction and Stabilization for Deep Reinforcement Learning (2016, methodology): “This algorithm is based on a simple value iteration update (Bellman, 1957), directly estimating the optimal value function Q ∗ .”
Horizon: Facebook's Open Source Applied Reinforcement Learning Platform (2018, background): “gorithms are theoretically designed on the Markov Decision Process (MDP) framework [6] where some sort of long-term reward is optimized in a sequential setting.”
Programmation et apprentissage bayésien pour les jeux vidéo multi-joueurs, application à l'intelligence artificielle de jeux de stratégies temps-réel. (Bayesian Programming and Learning for Multi-Player Video Games, Application to RTS AI) (2012, background): “MDP can be solved through dynamic programming or the Bellman value iteration algorithm [Bellman, 1957].”
From Data to Optimal Decision Making: A Data-Driven, Probabilistic Machine Learning Approach to Decision Support for Patients With Sepsis (2015, unknown): “Some of the most powerful methods for modeling decision making in clinical decision support are those that treat the learning problem as a Markov Decision Process (MDPs) [13].”
A Survey of Learning in Multiagent Environments: Dealing with Non-Stationarity (2017, methodology): “One of the most common techniques is the value iteration algorithm (Bellman, 1957) which is based on the Bellman equation: V π(s) = ∑ a∈A π(s, a) ∑ s′∈S T (s, a, s′)[R(s, a, s′) + γV π(s′)], (1) with γ ∈ [0, 1).”
Attention-Aware Deep Reinforcement Learning for Video Face Recognition (2017, methodology): “In our work, we formulate the process of finding attentions in video pairs as a Markov decision process (MDP) and introduce a reinforcement learning method to train the evaluation network.”
Transfer of Deep Reactive Policies for MDP Planning (2018, background): “Probabilistic planners can use the model to perform a full Bellman backup, i.e., expectation over all next outcomes from an action (e.g., in Value Iteration [Bellman, 1957]), whereas RL agents can only backup from a single sampled next state (e.g., in Q Learning [Sutton and Barto, 1998]).”