Load Frequency Control Strategy Based on Deep Q Learning for Island Microgrid with Electric Vehicles

doi:10.12204/j.issn.1000-7229.2022.04.010

Abstract

Abstract:

The load frequency control is of vital importance to maintain the stable operation of the island microgrid. Aiming at the frequency control problem when the microgrid is subjected to strong random disturbances and network topology parameters change, this paper proposes a load frequency control strategy based on deep Q-learning (DQN) for island microgrid with electric vehicles. Firstly, a frequency control model of electric vehicle considering the randomness of user charging behavior is established, and a load frequency control (LFC) model for microgrid including photovoltaic power, wind power, micro gas turbines, electric vehicles and their random power increment constraints is built. Secondly, the structure of frequency controller based on DQN is designed, and the definitions of state space, action space and reward function are completed in turn, and the optimal hyperparameters is obtained through adjustment. Finally, the simulation results show that, compared with PI control and FUZZY control, the DQN controller proposed in this paper has the ability of online learning and experience playback, which can deal with the LFC problem of microgrid with strong randomness more effectively, and it can better adapt to the complex operating conditions of system parameters and structure changes.

Key words: island microgrid, electric vehicles, frequency control, deep Q-learning

CLC Number:

TM73

FAN Peixiao, YANG Jun, XIAO Jinxing, XU Bingyan, YE Ying, LI Yonghui, LI Rui. Load Frequency Control Strategy Based on Deep Q Learning for Island Microgrid with Electric Vehicles[J]. ELECTRIC POWER CONSTRUCTION, 2022, 43(4): 91-99.

Figures/Tables 19

Fig.1 Load frequency control response model of micro gas turbine

Fig.2 Boundary of charging and discharging constraints of electric vehicle

Fig.3 Distribution of control commands in charging stations

Fig.4 Output power increment constraint of single EV

Fig.5 Frequency control model of electric vehicle under random power increment constraint

Fig.6 Load frequency control model of microgrid with electric vehicle

Fig.7 The main process of deep Q learning

Fig.8 Microgrid LFC controller structure based on DQN

Table 1 Convergence test results under different parameters

Fig.9 LFC structure of island microgrid with electric vehicles

Table 2 System parameters of microgrid LFC model

Fig.10 Random perturbation function in the pre-learning phase

Fig.11 Pre-learning process of agent

Fig.12 Strong random disturbance to island microgrid

Fig.13 Frequency deviation of microgridunder strong random disturbance

Fig.14 Output power increment of MT and EV under strong random disturbance

Table 3 Simulation results under strong random disturbance

Fig.15 Frequency deviation of microgrid under fault condition of FM unit

Table 4 Simulation results under fault condition of FM unit

References 24

[1]	米阳, 郎中杰, 何星瑭, 等. 孤岛微电网鲁棒自适应协调控制[J]. 控制工程, 2021, 28(5):924-930.
	MI Yang, LANG Zhongjie, HE Xingtang, et al. Coordinated control for isolated microgrid based on robust adaptive method[J]. Control Engineering of China, 2021, 28(5):924-930.
[2]	KARKEVANDI A E, DARYANI M J, USTA Ö. ANFIS-based Intelligent PI controller for secondary frequency and voltage control of microgrid[C]// 2018 IEEE PES Innovative Smart Grid Technologies Conference Europe (ISGT-Europe). IEEE, 2018.
[3]	MAHDI M M, AHMAD A Z. Load frequency control in microgrid using fuzzy logic table control[C]// 2017 11th IEEE International Conference on Compatibility, Power Electronics and Power Engineering. Cadiz: IEEE, 2017: 318-323.
[4]	CHAE S H, KIM G H, CHOI Y J, et al. Design of isolated microgrid system considering controllable EV charging demand[J]. Sustainability, 2020, 12(22):9746.
[5]	李航, 李国杰, 汪可友. 基于深度强化学习的电动汽车实时调度策略[J]. 电力系统自动化, 2020, 44(22):161-167.
	LI Hang, LI Guojie, WANG Keyou. Real-time dispatch strategy for electric vehicles based on deep reinforcement learning[J]. Automation of Electric Power Systems, 2020, 44(22):161-167.
[6]	林婷, 高亮. 含电动汽车的微电网规划研究[J]. 电力科学与技术学报, 2021, 36(4):53-58.
	LIN Ting, GAO Liang. Planning research of micro-grid with electric vehicles[J]. Journal of Electric Power Science and Technology, 2021, 36(4):53-58
[7]	ZIRAS C, PROSTEJOVSKY A M, BINDNER H W., et al. Decentralized and discretized control for storage systems offering primary frequency control[J]. Elsevier, 2019, 177(C).
[8]	YANG Jun, ZENG Zhili, TANG Yufei, et al. Load frequency control in isolated micro-grids with electrical vehicles based on multivariable generalized predictive theory[J]. Energies, 2015(8):2145-2164.
[9]	RAO Yingqing, YANG Jun, XIAO Jinxing, et al. A frequency control strategy for multimicrogrids with V2G based on the improved robust model predictive control[J]. Energy, 2021(222):119963,1-13.
[10]	万千, 夏成军, 管霖, 等. 含高渗透率分布式电源的独立微网的稳定性研究综述[J]. 电网技术, 2019, 43(2):598-612.
	WAN Qian, XIA Chengjun, GUAN Lin, et al. Review on stability of isolated microgrid with highly penetrated distributed generations[J]. Power System Technology, 2019, 43(2):598-612.
[11]	龙英文, 陈洁, 余粟. 一种分布式系统的孤岛微电网无功功率分配算法[J]. 控制工程, 2021, 28(7):1354-1359.
	LONG Yingwen, CHEN Jie, YU Su. A reactive power sharing algorithm for islanded micro-grid based on distributed system[J]. Control Engineering of China, 2021, 28(7):1354-1359.
[12]	程乐峰, 余涛, 张孝顺, 等. 机器学习在能源与电力系统领域的应用和展望[J]. 电力系统自动化, 2019, 43(1):15-31.
	CHENG Lefeng, YU Tao, ZHANG Xiaoshun, et al. Machine learning for energy and electric power systems: State of the art and prospects[J]. Automation of Electric Power Systems, 2019, 43(1):15-31.
[13]	张孝顺, 李清, 余涛, 等. 基于协同一致性迁移Q学习算法的虚拟发电部落AGC功率动态分配[J]. 中国电机工程学报, 2017, 37(5):1455-1467.
	ZHANG Xiaoshun, LI Qing, YU Tao, et al. Collaborative consensus transfer Q-learning based dynamic generation dispatch of automatic generation control with virtual generation tribe[J]. Proceedings of the CSEE, 2017, 37(5):1455-1467.
[14]	余涛, 梁海华, 周斌. 基于R (λ)学习的孤岛微电网智能发电控制[J]. 电力系统保护与控制, 2012, 40(13):7-13.
	YU Tao, LIANG Haihua, ZHOU Bin. Smart power generation control for microgrids islanded operation based on R (λ) learning[J]. Power System Protection and Control, 2012, 40(13):7-13.
[15]	唐捷, 张泽宇, 程乐峰, 等. 基于CEQ(λ)强化学习算法的微电网智能发电控制[J]. 电测与仪表, 2017, 54(1):39-45.
	TANG Jie, ZHANG Zeyu, CHENG Lefeng, et al. Smart generation control for micro-grids based on correlated equilibrium Q(λ) learning algorithm[J]. Electrical Measurement & Instrumentation, 2017, 54(1):39-45.
[16]	TAN C X, HAN R J, YE R G, et al. Adaptive learning recommendation strategy based on deep Q-learning[J]. Applied Psychological Measurement, 2020, 44(4):251-266. doi: 10.1177/0146621619858674 URL
[17]	华昊辰, 李宇童, 王同贺, 等. 一种基于混合随机H₂/H_∞方法的能源互联网边缘计算系统控制策略[J]. 中国电机工程学报, 2020, 40(21):6875-6885.
	HUA Haochen, LI Yutong, WANG Tonghe, et al. A novel stochastic mixed H₂/h_∞ control strategy for energy Internet edge computing system[J]. Proceedings of the CSEE, 2020, 40(21):6875-6885.
[18]	李晓萌, 贾宏杰, 穆云飞, 等. 时滞环境下基于电动汽车与电热泵的协调频率控制[J]. 电力自动化设备, 2020, 40(4):88-95, 110.
	LI Xiaomeng, JIA Hongjie, MU Yunfei, et al. Coordinated frequency control based on electric vehicles and heat pumps considering time-delay[J]. Electric Power Automation Equipment, 2020, 40(4):88-95, 110.
[19]	李军军, 吴政球, 谭勋琼, 等. 风力发电及其技术发展综述[J]. 电力建设, 2011, 32(8):64-72.
	LI Junjun, WU Zhengqiu, TAN Xunqiong, et al. Review of wind power generation and relative technology development[J]. Electric Power Construction, 2011, 32(8):64-72.
[20]	刘全, 翟建伟, 章宗长, 等. 深度强化学习综述[J]. 计算机学报, 2018, 41(1):1-27.
	LIU Quan, ZHAI Jianwei, ZHANG Zongzhang, et al. A survey on deep reinforcement learning[J]. Chinese Journal of Computers, 2018, 41(1):1-27.
[21]	刘建伟, 高峰, 罗雄麟. 基于值函数和策略梯度的深度强化学习综述[J]. 计算机学报, 2019, 42(6):1406-1438.
	LIU Jianwei, GAO Feng, LUO Xionglin. Survey of deep reinforcement learning based on value function and policy gradient[J]. Chinese Journal of Computers, 2019, 42(6):1406-1438.
[22]	刘俊峰, 陈剑龙, 王晓生, 等. 基于深度强化学习的微能源网能量管理与优化策略研究[J]. 电网技术, 2020, 44(10):3794-3803.
	LIU Junfeng, CHEN Jianlong, WANG Xiaosheng, et al. Energy management and optimization of multi-energy grid based on deep reinforcement learning[J]. Power System Technology, 2020, 44(10):3794-3803.
[23]	风特征数据库[DB/OL]. [2014-03-03]. http://www.wind -data.com/.
[24]	赵伟, 李雄, 乔仁飞, 等. 基于混合储能的孤岛微网VSG控制策略[J]. 电力系统保护与控制, 2021, 49(12):33-40.
	ZHAO Wei, LI Xiong, QIAO Renfei, et al. VSG control strategy of an isolated microgrid based on hybrid energy storage[J]. Power System Protection and Control, 2021, 49(12):33-40.