Safe Reinforcement Learning

By Philip Thomas. 2015

162 citations recorded by the source corpus.

Subfields: Reinforcement Learning, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

In Safety, Robustness & Interpretability

Citations in the corpus

Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learning (2016, methodology): “We used three domains: a 4 × 4 gridworld previously constructed speciﬁcally for evaluating OPE meth-ods (Thomas, 2015b, Section 2.5), as well as two simple domains that we developed to exemplify the settings where different methods excel and fail.”
Doubly Robust Off-policy Value Evaluation for Reinforcement Learning (2015, background): “However, for complex real-world problems, it is often hard to specify a function class in regression that is efﬁciently learnable with limited data while at the same time has a small approximation error.”
MOPO: Model-based Offline Policy Optimization (2020, background): “However, such online data collection can incur high sample complexity [45, 55, 56], limit the power of generalization to unseen random initialization [8, 74, 4], and pose risks in safety-critical settings [67].”
MOReL : Model-Based Offline Reinforcement Learning (2020, background): “like healthcare and industrial automation where careful oversight by a domain expert is necessary for taking exploratory actions or deploying new policies [13, 14].”
D4RL: Datasets for Deep Data-Driven Reinforcement Learning (2020, methodology): “ing, issues such as distribution shift can cause undesirable performance in practice, as mentioned previously. Thus, approaches to ofﬂine RL have focused on techniques such as safe policy improvement [33] or adding regularizers in to mitigate effects of distribution shift [37]. In order to be effect”
Safe Policy Improvement with Soft Baseline Bootstrapping (2019, methodology): “…generated finite MDPs, where the SoftSPIBB algorithms are compared to several Batch RL competitors: basic RL, High Confidence Policy Improvement (Thomas, 2015, HCPI), Reward-Adjusted MDPs (Petrik et al., 2016, RaMDP), Robust MDPs (Iyengar, 2005; Nilim and El Ghaoui, 2005), and to Soft-SPIBB…”
Safe Policy Improvement with Baseline Bootstrapping (2017, background): “ARBRM [24] no yes yes no Robust MDP [24] no yes yes no Reward-Adjusted MDP [24] no yes no no High-Confidence PI [32] no yes no no Qb -SPIBB-1 novel Th.”
Bootstrapping with Models: Confidence Intervals for Off-Policy Evaluation (2016, neutral): “Unfortunately, these approachs requires prohibitive amounts of data and were shown to be far less data-efficient than bootstrapping with IS [18, 19].”
Offline Reinforcement Learning with Fisher Divergence Critic Regularization (2021, background): “However, in many practical scenarios, where deploying a new policy to interact with the live environment is expensive or associated with risks or safety concerns (Thomas, 2015), it is more common to have only ofﬂine access to the environment.”
Towards Optimal Off-Policy Evaluation for Reinforcement Learning with Marginalized Importance Sampling (2019, background): “1 Generic IS-Based Estimators Setup The IS-based estimators usually provide an unbiased or consistent estimate of the value of target policy π [Thomas, 2015].”
Bias Correction of Learned Generative Models using Likelihood-Free Importance Weighting (2019, methodology): “In model-based off-policy policy evaluation (henceforth MBOPE), a learned dynamics model is used to simulate and evaluate a target policy without real-world deployment [2], which is especially valuable for risk-sensitive applications [3].”
Importance Sampling Policy Evaluation with an Estimated Behavior Policy (2018, methodology): “The domain has been used in prior off-policy policy evaluation work (Thomas, 2015; Thomas & Brunskill, 2016a; Hanna et al., 2017; Farajtabar et al., 2018).”
Towards Safe Policy Improvement for Non-Stationary MDPs (2020, background): “, E[ρ̂(π, i)] = ρ(π, i), under the the following assumption [58], which can typically be satisfied using an entropy-regularized policy βi.”
High Confidence Off-Policy Evaluation with Models (2016, background): “Also, as in previous work on importance sampling, we shorten the horizon of the problem by holding action at constant for 4 updates of the environment state [7, 19].”
Doubly Robust Off-policy Evaluation for Reinforcement Learning (2015, background): “In robotics and business/marketing applications, for instance, it is often risky (thus expensive) to run a policy without an estimate of the policy's quality (Li et al., 2011; Bottou et al., 2013; Thomas et al., 2015a).”
Dead-ends and Secure Exploration in Reinforcement Learning (2019, background): “Safety in RL (Garcı́a & Fernández, 2015) is an overloaded term, as it may be considered with respect to parametric uncertainty (Thomas, 2015; Petrik et al., 2016; Laroche et al., 2019), internal uncertainty (Altman, 1999; Carrara et al., 2019), or interruptibility (Orseau & Armstrong, 2016;…”
Multi-step Off-policy Learning Without Importance Sampling Ratios (2017, background): “When safety and data frugality is paramount, the ability to evaluate a policy without actually following it can be invaluable (cf. Thomas 2015).”
Safe Exploration in Continuous Action Spaces (2018, background): “Safe exploration, as depicted above, traditionally requires access to data generated with some known behavior policy upon which gradual safe updates are performed; see (Thomas, 2015) for a comprehensive study of such off-policy methods.”
POPCORN: Partially Observed Prediction COnstrained ReiNforcement Learning (2020, methodology): “In this work, the specific OPE technique we use is consistent weighted per-decision importance sampling (CWPDIS, (Thomas, 2015)) which estimates the value of a policy ⇡ ✓ as: V CWPDIS (⇡ ✓ ) , TX t=1 t P n2D rnt⇢nt(⇡✓)P n2D ⇢nt(⇡✓) , (2) ⇢ nt (⇡ ✓ ) , tY s=0 ⇡ ✓ (a ns”