Value Function Approximation in Reinforcement Learning Using the Fourier Basis

By George Dimitri Konidaris, Sarah Osentoski, Philip S. Thomas. 2011

333 citations recorded by the source corpus.

We describe the Fourier basis, a linear value function approximation scheme based on the Fourier series. We empirically demonstrate that it performs well compared to radial basis functions and the polynomial basis, the two most popular fixed bases for linear value function approximation, and is competitive with learned proto-value functions.

Subfields: Artificial Intelligence, Electrical and Electronic Engineering, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Electrical and Electronic Engineering

In Reinforcement Learning

Citations in the corpus

Investigating Contingency Awareness Using Atari 2600 Games (2012, methodology): “While automatic feature construction methods (Parr et al. 2007; Konidaris, Osentoski, and Thomas 2011) and generic feature sets (Schweitzer and Seidmann 1985) have been proposed, current techniques do not explicitly incorporate any notion of contingency awareness .”
Learning Parameterized Skills (2012, neutral): “Popular choices for basis functions include Radial Basis Functions and the Fourier Basis [39].”
Reinforcement learning in robotics: A survey (2013, methodology): “Konidaris et al. (2011a, 2012) proposes an approach that constructs a skill tree from human demonstrations.”
Improved empirical methods in reinforcement-learning evaluation (2015, methodology): “The state-action value functions evaluated were trained using batch reinforcement-learning algorithms, namely: Least-Squares Policy Iteration (LSPI) (Lagoudakis and Parr, 2003), with either a grid-like or Fourier basis (Konidaris et al., 2008), and Q-learning with experience replay (Lin, 1992).”
Learning Action Representations for Reinforcement Learning (2019, methodology): “State features were represented using the 3rd order coupled Fourier basis (Konidaris et al., 2011).”
Hierarchical Relative Entropy Policy Search (2014, methodology): “In our experiments we used either kernel based features or features based on a Fourier transformation as shown by (Konidaris et al., 2011).”
Sparse Q-learning with Mirror Descent (2012, background): “Thanks to the ﬂexible interpolation capability between multiplicative and additive gradient updates, the p -norm link function is more robust and applicable to various basis functions, such as polynomial, radial basis function (RBF), Fourier basis [KOT08], proto-value functions (PVFs), etc.”
RLLib: C++ Library to Predict, Control, and Represent Learnable Knowledge Using On/Off Policy Reinforcement Learning (2015, background): “2] or compact features [12], suitable for the problem.”
Design Principles of the Hippocampal Cognitive Map (2014, background): “Spectral features also serve as generally useful representations [26, 22], and high frequency components are important for representing detail in the value function.”
Preventing undesirable behavior of intelligent machines (2019, methodology): “A more conventional RL approach would be to maintain a single stochastic policy, perhaps using the Fourier basis [199] and softmax action selection [47], rather than a distribution over deterministic policies.”
A Review of Robot Learning for Manipulation: Challenges, Representations, and Algorithms (2019, background): “independent Fourier basis [207]), or more general decisions regarding inductive bias and representational power (e.”
High Confidence Policy Improvement (2015, methodology): “We used CMA-ES (Hansen, 2006) to solve all arg max π , where π was parameterized by a vector of policy parameters using linear softmax action selection (Sutton & Barto, 1998) with the Fourier basis (Konidaris et al., 2011).”
Learning grounded finite-state representations from unstructured demonstrations (2015, methodology): “In this work, we use the univariate Fourier basis (Konidaris et al., 2011) for a function approximator, although others have used normalized radial basis functions (Pastor et al.”
Constructing Skill Trees for Reinforcement Learning Agents from Demonstration Trajectories (2010, methodology): “01) using a 5th-order Fourier basis [13] with α = 0.”
Regularized Policy Iteration with Nonparametric Function Spaces (2016, background): “11 These basis functions might be predefined (e.g., Fourier (Konidaris et al., 2011) or wavelets) or constructed datadependently by one of already mentioned feature generation methods.”
TD_gamma: Re-evaluating Complex Backups in Temporal Difference Learning (2011, methodology): “A 5th order Fourier basis [9] was used as the function approximator for the 3 continuous domains.”
Robot learning from demonstration by constructing skill trees (2012, background): “More details are available in Konidaris et al. (2011a).”