Model-Free Real-Time EV Charging Scheduling Based on Deep Reinforcement Learning

By Zhiqiang Wan, Hepeng Li, Haibo He, Danil V. Prokhorov. IEEE Transactions on Smart Grid, 2019

457 citations recorded by the source corpus.

Driven by the recent advances in electric vehicle (EV) technologies, EVs have become important for smart grid economy. When EVs participate in demand response program which has real-time pricing signals, the charging cost can be greatly reduced by taking full advantage of these pricing signals. However, it is challenging to determine an optimal charging strategy due to the existence of randomness in traffic conditions, user’s commuting behavior, and the pricing process of the utility. Conventional model-based approaches require a model of forecast on the uncertainty and optimization for the scheduling process. In this paper, we formulate this scheduling problem as a Markov Decision Process (MDP) with unknown transition probability. A model-free approach based on deep reinforcement learning is proposed to determine the optimal strategy for this problem. The proposed approach can adaptively learn the transition probability and does not require any system model information. The architecture of the proposed approach contains two networks: a representation network to extract discriminative features from the electricity prices and a Q network to approximate the optimal action-value function. Numerous experimental results demonstrate the effectiveness of the proposed approach.

Subfields: Automotive Engineering, Electrical and Electronic Engineering

View Paper: https://www.semanticscholar.org/paper/59021ecd3fd15c59b8b774c87ae974c9fffe9fa5

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Automotive Engineering

1. Dacheng Tao
2. Jing Zhang
3. Frede Blaabjerg

In Electrical and Electronic Engineering

1. Dacheng Tao
2. Jing Zhang
3. Na Li
4. Steven H. Low
5. Yujie Tang
6. Zhe Chen
7. Frede Blaabjerg
8. Yan Du

Citations in the corpus

• A Multi-Agent Reinforcement Learning-Based Data-Driven Method for Home Energy Management (2020, unknown): “[14], Wan et al. use deep RL algorithms to determine the optimal solutions of the EV charging/discharging scheduling problem.”
• Empowering Things With Intelligence: A Survey of the Progress, Challenges, and Opportunities in Artificial Intelligence of Things (2020, methodology): “For load monitoring and electric vehicle charging scheduling, TL [200] and deep RL [215] methods are proposed.”
• Intelligent Multi-Microgrid Energy Management Based on Deep Neural Network and Model-Free Reinforcement Learning (2020, background): “The authors in [23] proposed the application of deep RL to optimize the real-time electric vehicle charging schedule with the consideration of future electricity price.”
• Constrained EV Charging Scheduling Based on Safe Deep Reinforcement Learning (2020, methodology): “If we maximize the lower bound to generate the policy π k + 1 when at the k th iteration, we are guaranteed that an improving policy can be obtained according to Wan et al. [16], i.e., Now, we can use the update rule in (38) to improve the policy π k to maximize the return J (π k ) .”
• Reinforcement Learning and Its Applications in Modern Power and Energy Systems: A Review (2020, methodology): “Reference [80] proposes a DQN-based approach for the optimization of EV charging while [81] extends this work by integrating the DQN algorithm with the long-short term memory (LSTM) NN.”
• Reinforcement Learning for Selective Key Applications in Power Systems: Recent Advances and Future Challenges (2021, background): “In [121]–[123], various RL techniques are studied to design EV charging policies to deal with the randomness in the arrival and departure time of an EV.”