Bootstrapped Policy Gradient for Difficulty Adaptation in Intelligent Tutoring Systems

By Yaqian Zhang, Wooi-Boon Goh. 2019

4 citations recorded by the source corpus.

One challenge for an intelligent interactive tutoring agent is to autonomously determine the difficulty levels of the questions presented to the users. This difficulty adaptation problem can be formulated as a sequential decision making problem which can be solved by Reinforcement learning (RL) methods. However, the cost of taking an action is an important consideration when applying RL in real-time responsive application involving human in the loop. Sample efficient algorithms are therefore critical for such applications, especially when the action space is large. This paper proposes a bootstrapped policy gradient (BPG) framework, which can incorporate prior knowledge into policy gradient to enhance sample efficiency. The core idea is to update the summed probability of a set of related actions rather than a single action at gradient estimation sample. A sufficient condition for unbiased convergence is provided and proved. We apply the BPG to solve the difficulty adaptation problem in a challenging environment with large action space and short horizon, and it achieves fast and unbiased convergence both in theory and in practice. We also generalize BPG to multi-dimensional continuous action domain in general actor-critic reinforcement learning algorithms with no prior knowledge required.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Repeated Augmented Rehearsal: A Simple but Strong Baseline for Online Continual Learning (2022, methodology): “To address the sample efﬁciency issue, we employ the multi-armed bandit framework and apply bootstrapped policy gradient (BPG) [Zhang and Goh, 2019].”