Value-Aware Loss Function for Model-based Reinforcement Learning

By Amir-massoud Farahmand, André Barreto, Daniel Nikolaev Nikovski. 2017

133 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

When to Trust Your Model: Model-Based Policy Optimization (2019, methodology): “Alternate analyses have been carried out by incorporating the structure of the value function into the model learning (Farahmand et al., 2017) or by regularizing the model by controlling its Lipschitz constant (Asadi et al., 2018).”
Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guarantees (2018, methodology): “Our discrepancy bound in Section 4 is closely related to the work (Farahmand et al., 2017) on the value-aware model loss.”
Iterative Value-Aware Model Learning (2018, methodology): “Therefore, it minimizes c 22 ,ν ( ˆ P , P ∗ ) = d ν ( x, a ) sup As argued by Farahmand et al. [2017a], this is still a tighter objective to minimize than the KL - divergence.”
Planning with Diffusion for Flexible Behavior Synthesis (2022, methodology): “…latent-space model for reward prediction (Tamar et al., 2016; Oh et al., 2017; Schrittwieser et al., 2019); weighing model training objectives by state values (Farahmand et al., 2017); and applying collocation techniques to learned single-step energies (Du et al., 2019; Rybkin et al., 2021).”
The Value Equivalence Principle for Model-Based Reinforcement Learning (2020, methodology): “However, we argue that ignoring the future use of r̃ and p̃ may not always be the best choice [22, 15].”
FLAMBE: Structural Complexity and Representation Learning of Low Rank MDPs (2020, methodology): “On the theoretical side, in the reward-sensitive setting, can we avoid learning the entire model, perhaps by utilizing “value-aware” methods (Farahmand et al., 2017; Ayoub et al., 2020)? Can we remove realizability conditions on μ0:H 1 and develop provably efficient model-free algorithms for represe”
Model-based RL in Contextual Decision Processes: PAC bounds and Exponential Improvements over Model-free Approaches (2018, methodology): “Farahmand et al. (2017) also use IPMs to detect model error but their analysis is restricted to test functions that form a ball in an RKHS, and they do not address exploration issues.”
Applications of reinforcement learning in energy systems (2021, unknown): “…into the following three groups: (i) value-based, e.g., Dyna-Q algorithm [42], Deep Dyna-Q algorithm [43], and Value-Aware Model Learning (VAML) [44], (ii) policy-based, e.g., Model-Based Policy Gradient (MBPG) [45], ME-TRPO, SLBO, and Policy-Aware Model Learning (PAML) [46], and (iii)…”