Application of the Actor-Critic Architecture to Functional Electrical Stimulation Control of a Human Arm

By Philip S. Thomas, A. V. Bogert, Kathleen M. Jagodnik, Michael S. Branicky. 2009

31 citations recorded by the source corpus.

Clinical tests have shown that the dynamics of a human arm, controlled using Functional Electrical Stimulation (FES), can vary significantly between and during trials. In this paper, we study the application of the actor-critic architecture, with neural networks for the both the actor and the critic, as a controller that can adapt to these changing dynamics of a human arm. Development and tests were done in simulation using a planar arm model and Hill-based muscle dynamics. We begin by training it using a Proportional Derivative (PD) controller as a supervisor. We then make clinically relevant changes to the dynamics of the arm and test the actor-critic’s ability to adapt without supervision in a reasonable number of episodes. Finally, we devise methods for achieving both rapid learning and long-term stability.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

1. Chao Yu
2. Bo Liu
3. Jiming Liu
4. Ji Liu
5. Shamim Nemati
6. Sridhar Mahadevan
7. Will Dabney
8. Stephen Giguere

Citations in the corpus

• Reinforcement Learning in Healthcare: A Survey (2019, background): “o calculate stimulation patterns to efficiently adapt the control strategy to a wide range of time varying situations in patients’ preferences and reaching dynamics. AC-based control strategies [294], [295], [290] were proposed to evaluate targetoriented task performed using a planar musculoskeletal ”
• Training an Actor-Critic Reinforcement Learning Controller for Arm Movement Using Human-Generated Rewards (2017, methodology): “[20], [22] applied RL control to a planar arm simulation, achieving good performance within a few hundred episodes of training.”
• Proximal Reinforcement Learning: A New Theory of Sequential Decision Making in Primal-Dual Spaces (2014, background): “Researchers have shown that policy search algorithms are a viable approach to creating controllers that can automatically adapt to an individual’s arm by training on a few hundred two-second reaching movements [86].”