Applying reinforcement learning towards automating resource allocation and application scalability in the cloud

By Enda Barrett, Enda Howley, Jim Duggan. 2013

238 citations recorded by the source corpus.

Public Infrastructure as a Service (IaaS) clouds such as Amazon, GoGrid and Rackspace deliver computational resources by means of virtualisation technologies. These technologies allow multiple independent virtual machines to reside in apparent isolation on the same physical host. Dynamically scaling applications running on IaaS clouds can lead to varied and unpredictable results because of the performance interference effects associated with co‐located virtual machines. Determining appropriate scaling policies in a dynamic non‐stationary environment is non‐trivial. One principle advantage exhibited by IaaS clouds over their traditional hosting counterparts is the ability to scale resources on‐demand. However, a problem arises concerning resource allocation as to which resources should be added and removed when the underlying performance of the resource is in a constant state of flux. Decision theoretic frameworks such as Markov Decision Processes are particularly suited to decision making under uncertainty. By applying a temporal difference, reinforcement learning algorithm known as Q‐learning, optimal scaling policies can be determined. Additionally, reinforcement learning techniques typically suffer from curse of dimensionality problems, where the state space grows exponentially with each additional state variable. To address this challenge, we also present a novel parallel Q‐learning approach aimed at reducing the time taken to determine optimal policies whilst learning online. Copyright © 2012 John Wiley & Sons, Ltd.

Subfields: Computer Networks and Communications, Information Systems

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Networks and Communications

In Information Systems

Citations in the corpus

Consistent On-Line Off-Policy Evaluation (2017, methodology): “Reinforcement Learning (RL) techniques were successfully applied in fields such as robotics, games, marketing and more (Kober et al., 2013; Al-Rawi et al., 2015; Barrett et al., 2013).”
A Review of Auto-scaling Techniques for Elastic Applications in Cloud Environments (2014, background): “[42] create an agent per VM, that keeps its own, small lookup table.”
Elasticity in Cloud Computing: State of the Art and Research Challenges (2018, methodology): “For example, [72], [96], [102] use RL in reactive mode while [113],”
A Hierarchical Framework of Cloud Resource Allocation and Power Management Using Deep Reinforcement Learning (2017, methodology): “As a result, previous works only used RL to dynamically adjust the power status of a single physical server [7] or the number of homogeneous VMs to an application [5,8], in order to restrict the state and action spaces.”
MArk: Exploiting Cloud Services for Cost-Effective, SLO-Aware Machine Learning Inference Serving (2019, background): “As summarized in Table 1, they only consider provisioning over homogeneous instances in IaaS [9, 25, 42, 70].”
Swayam: distributed autoscaling to meet SLAs of machine learning inference services with resource efficiency (2017, background): “Prior autoscaling and prediction solutions for stateless web services [10, 12, 16, 19, 31] typically perform centralized monitoring of”