Multi-source Coordinated Frequency Control Method Based on Reinforcement Learning for Island Microgrid

doi:10.12204/j.issn.1000-7229.2020.09.008

Abstract

Abstract:

In order to improve the frequency anti-interference of the island microgrid, a coordinated frequency control method based on reinforcement learning for island microgrid is proposed. The proposed control method performs Q-learning on the basis of the frequency deviation of the microgrid, and dynamically adjusts the droop control parameters of multiple distributed power sources to change their output power, to realize multi-source coordinated active frequency control in the microgrid. Firstly, the principle of Q-learning algorithm is introduced. Secondly, a frequency recovery control method based on Q-learning is proposed, and a controller based on the Q-learning algorithm is set up. The Q-learning algorithm is used to dynamically correct the droop parameters and coordinate multiple distributed power sources in the microgrid for frequency recovery control. Finally, MATLAB is used to establish a typical microgrid simulation model, and on the basis of S-function, a self-defined reinforcement learning controller is established to verify the effectiveness and adaptability of the proposed method from the aspects of the Q-learning training process and frequency control response characteristics.

Key words: reinforcement learning, island microgrid, frequency control, Q-learning

CLC Number:

TM712

YAO Jianhua, HU Sheng, WANG Guan, SHEN Yun, JIANG Linlin, FENG Yuli, GONG Chengya, ZHANG Zhaoxuan, LIU Wei. Multi-source Coordinated Frequency Control Method Based on Reinforcement Learning for Island Microgrid[J]. ELECTRIC POWER CONSTRUCTION, 2020, 41(9): 69-75.

Figures/Tables 13

Fig.1 Basic framework of reinforcement learning

Fig.2 Parallel connection structure of distributed power sources

Fig.3 Frequency control method based on Q-learning

Fig.4 Flowchart of Q table training

Fig.5 Structure the simulation microgrid

Fig.6 Pre-learning process of Q-learning

Fig.7 Distributed power output in pre-learning scenarios

Fig.8 System frequency in pre-learning scenarios

Fig.9 The pre-learning process of Q-learning after the access of distributed power

Fig.10 Power changes after the access of distributed power

Fig.11 System frequency change after the access of distributed power

Fig.12 Change of Q-learning control adjustment amount in load fluctuation scenarios

Fig.13 System frequency change in load fluctuation scenarios

References 23

[1]	刘开俊, 高艺, 宋福龙 . 多适应性电网规划风险评估框架设计及应用展望[J]. 电力建设, 2014,35(12):1-5. doi: 10.3969/j.issn.1000-7229.2014.12.001 URL
	LIU Kaijun, GAO Yi, SONG Fulong . Risk assessment framework design on flexibility power grid planning and its application prospect[J]. Electric Power Construction, 2014,35(12):1-5. doi: 10.3969/j.issn.1000-7229.2014.12.001 URL
[2]	韩丰, 高艺, 宋福龙 , 等. 电网规划评估方法及实用化技术[J]. 电力建设, 2014,35(12):6-13. doi: 10.3969/j.issn.1000-7229.2014.12.002 URL
	HAN Feng, GAO Yi, SONG Fulong , et al. Evaluation method and practical technology on power grid planning[J]. Electric Power Construction, 2014,35(12):6-13. doi: 10.3969/j.issn.1000-7229.2014.12.002 URL
[3]	刘梦, 杨作梁 . 考虑网架结构和分布式电源的主动配电系统双层规划[J]. 电测与仪表, 2018,55(14):46-50, 81.
	LIU Meng, YANG Zuoliang . Bi-level programming of active distribution system considering grid structure and distributed generation[J]. Electrical Measurement & Instrumentation, 2018,55(14):46-50, 81.
[4]	杨鹏龙, 董海鹰 . 基于改进型同步逆变器技术的微电网变换器[J]. 电机与控制应用, 2018,45(1):7-11.
	YANG Penglong, DONG Haiying . A grid-friendly converter based on improved synchronous inverter technology[J]. Electric Machines & Control Application, 2018,45(1):7-11.
[5]	赵晶晶, 吕雪, 符杨 , 等. 基于可变系数的双馈风机虚拟惯量与超速控制协调的风光柴微电网频率调节技术[J]. 电工技术学报, 2015,30(5):59-68.
	ZHAO Jingjing, LÜ Xue, FU Yang , et al. Frequency regulation of the wind/photovoltaic/diesel microgrid based on DFIG cooperative strategy with variable coefficients between virtual inertia and over-speed control[J]. Transactions of China Electrotechnical Society, 2015,30(5):59-68.
[6]	陈萌, 肖湘宁, 罗超 . 基于虚拟同步发电机的微电网延时补偿二次频率控制[J]. 电工技术学报, 2018,33(16):3845-3854.
	CHEN Meng, XIAO Xiangning, LUO Chao . Secondary frequency control including delay compensation in microgrids based on virtual synchronous generator[J]. Transactions of China Electrotechnical Society, 2018,33(16):3845-3854.
[7]	王金玉, 潘阳阳, 钱勇 . 适用于岛屿环境的微电网发电系统设计[J]. 电力学报, 2019,34(4):395-398, 410.
	WANG Jinyu, PAN Yangyang, QIAN Yong . Design of microgrid power generation system for island environment[J]. Journal of Electric Power, 2019,34(4):395-398, 410.
[8]	赵卓立, 杨苹, 许志荣 , 等. 多源多变换微电网系统建模与动态稳定分析方法[J]. 控制理论与应用, 2017,34(8):1062-1073.
	ZHAO Zhuoli, YANG Ping, XU Zhirong , et al. Modeling and dynamic stability analysis method of microgrids with multiple energy resources and multiple conversions[J]. Control Theory & Applications, 2017,34(8):1062-1073.
[9]	王波, 张保会, 郝治国 . 基于功率监测和频率变化率的孤岛微电网紧急切负荷控制[J]. 电力系统自动化, 2015,39(8):33-37.
	WANG Bo, ZHANG Baohui, HAO Zhiguo . Emergency load shedding control for islanded microgrid based on power monitoring and rate of frequency change[J]. Automation of Electric Power Systems, 2015,39(8):33-37.
[10]	洪灏灏, 顾伟, 黄强 , 等. 微电网中多虚拟同步机并联运行有功振荡阻尼控制[J]. 中国电机工程学报, 2019,39(21):6247-6255.
	HONG Haohao, GU Wei, HUANG Qiang , et al. Power oscillation damping control for microgrid with multiple VSG units[J]. Proceedings of the CSEE, 2019,39(21):6247-6255.
[11]	张羽, 李咸善 . 基于频率调整策略的微电网多目标优化自愈控制[J]. 电网技术, 2017,41(3):831-839.
	ZHANG Yu, LI Xianshan . Multi-objective optimization for microgrid self-healing control based on frequency regulation strategy[J]. Power System Technology, 2017,41(3):831-839.
[12]	赵晶晶, 吕雪, 符杨 , 等. 基于双馈感应风力发电机虚拟惯量和桨距角联合控制的风光柴微电网动态频率控制[J]. 中国电机工程学报, 2015,35(15):3815-3822. doi: 10.13334/j.0258-8013.pcsee.2015.15.009 URL
	ZHAO Jingjing, LÜ Xue, FU Yang , et al. Dynamic frequency control strategy of wind/photovoltaic/diesel microgrid based on DFIG virtual inertia control and pitch angle control[J]. Proceedings of the CSEE, 2015,35(15):3815-3822.
[13]	张春雪, 黎灿兵, 冯伟 , 等. 孤立运行光/储微电网中储能变流器暂态功率波动协调抑制策略[J]. 中国电机工程学报, 2018,38(8):2302-2314, 2540.
	ZHANG Chunxue, LI Canbing, FENG Wei , et al. A coordinated transient power fluctuation suppression strategy for power conversion system in islanded PV/storage microgrid[J]. Proceedings of the CSEE, 2018,38(8):2302-2314, 2540.
[14]	ZHAI H, YI H, ZHUO F, et al. Online control of SVG for network unbalance suppression [C]//2016 IEEE 8th International Power Electronics and Motion Control Conference (IPEMC-ECCE Asia). IEEE, 2016: 137-141.
[15]	WANG Z X, YI H, ZHUO F, et al. A self-adaptive power balance control strategy for PV inverters in islanded microgrids [C]//2016 IEEE Applied Power Electronics Conference and Exposition (APEC). IEEE, 2016: 3358-3363.
[16]	姚玮, 陈敏, 牟善科 , 等. 基于改进下垂法的微电网逆变器并联控制技术[J]. 电力系统自动化, 2009,33(6):77-80, 94.
	YAO Wei, CHEN Min, MOU Shanke , et al. Paralleling control technique of microgrid inverters based on improved droop method[J]. Automation of Electric Power Systems, 2009,33(6):77-80, 94.
[17]	杨志淳, 刘开培, 乐健 , 等. 孤岛运行微电网中模糊PID下垂控制器设计[J]. 电力系统自动化, 2013,37(12):19-23, 68. doi: 10.7500/AEPS201203279 URL
	YANG Zhichun, LIU Kaipei, LE Jian , et al. Design of fuzzy PID droop controllers for islanded microgrids[J]. Automation of Electric Power Systems, 2013,37(12):19-23, 68. doi: 10.7500/AEPS201203279 URL
[18]	LI P, WANG W, YANG X L, et al. A droop control method of microsources based on divided self-adjusting slope coefficient [C]//2010 International Conference on Power System Technology. IEEE, 2010: 1-6.
[19]	周贤正, 荣飞, 吕志鹏 , 等. 低压微电网采用坐标旋转的虚拟功率V/f下垂控制策略[J]. 电力系统自动化, 2012,36(2):47-51, 63.
	ZHOU Xianzheng, RONG Fei, LÜ Zhipeng , et al. A coordinate rotational transformation based virtual power V/f droop control method for low voltage microgrid[J]. Automation of Electric Power Systems, 2012,36(2):47-51, 63.
[20]	郜登科, 姜建国, 张宇华 . 使用电压-相角下垂控制的微电网控制策略设计[J]. 电力系统自动化, 2012,36(5):29-34.
	GAO Dengke, JIANG Jianguo, ZHANG Yuhua . Design of microgrid control strategy using voltage amplitude and phase angle droop control[J]. Automation of Electric Power Systems, 2012,36(5):29-34.
[21]	MAJUMDER R, LEDWICH G, GHOSH A , et al. Droop control of converter-interfaced microsources in rural distributed generation[J]. IEEE Transactions on Power Delivery, 2010,25(4):2768-2778. doi: 10.1109/TPWRD.2010.2042974 URL
[22]	王健, 张平陆, 赵忠英 , 等. 结合神经网络和Q(λ)-learning的路径规划方法[J]. 自动化与仪表, 2019,34(9):1-4.
	WANG Jian, ZHANG Pinglu, ZHAO Zhongying , et al. Path planning method based on neural network and Q(λ)-learning[J]. Automation & Instrumentation, 2019,34(9):1-4.
[23]	薛晗, 邵哲平, 方琼林 , 等. 基于强化学习的倒立摆分数阶梯度下降RBF控制[J]. 控制与决策, 2019. http://kzyjc.alljournals.cn/ch/reader/view_abstract.aspx?file_no=201906090000002&flag=2. URL pmid: 26236754
	XUE Han, SHAO Zheping, FANG Qionglin , et al. Reinforcement learning based fractional gradient descent RBF neural network control of inverted pendulum [J]. Control and Decision, 2019. http://kzyjc.alljournals.cn/ch/reader/view_abstract.aspx?file_no=201906090000002&flag=2. doi: 10.1037/dec0000030 URL pmid: 26236754