基于强化学习的孤岛微电网多源协调频率控制方法

doi:10.12204/j.issn.1000-7229.2020.09.008

电力建设 ›› 2020, Vol. 41 ›› Issue (9): 69-75.doi: 10.12204/j.issn.1000-7229.2020.09.008

基于强化学习的孤岛微电网多源协调频率控制方法

姚建华¹, 胡晟¹, 王冠¹, 沈云¹, 姜林林¹, 冯宇立¹, 龚成亚¹, 张照轩², 柳伟²

1.国网浙江嘉善县供电有限公司,浙江省嘉兴市 314100
2.南京理工大学自动化学院,南京市 210094

收稿日期:2019-12-26 出版日期:2020-09-01 发布日期:2020-09-03
通讯作者: 柳伟
作者简介:姚建华(1981),男,本科,工程师,主要从事综合能源工作;|胡晟(1986),男,硕士,工程师,主要研究方向为电网规划;|王冠(1991),男,本科,助理工程师,主要研究方向为电力系统运行管理;|沈云(1993),男,本科,助理工程师,主要研究方向为电力营销;|姜林林(1992),男,硕士,助理工程师,主要研究方向为电力营销;|冯宇立(1990),男,本科,工程师,主要研究方向为电力基建和运检;|龚成亚(1988),男,硕士,工程师,主要研究方向为电力营销;|张照轩(1996),男,硕士研究生,研究方向为人工智能技术在电力系统中的应用。
基金资助:
国网浙江省电力有限公司集体企业科技项目(2019-LHKJ-011)

Multi-source Coordinated Frequency Control Method Based on Reinforcement Learning for Island Microgrid

YAO Jianhua¹, HU Sheng¹, WANG Guan¹, SHEN Yun¹, JIANG Linlin¹, FENG Yuli¹, GONG Chengya¹, ZHANG Zhaoxuan², LIU Wei²

1. State Grid Zhejiang Jiashan Power Supply Co., Ltd., Jiashan 314100, Zhejiang Province,China
2. College of Automation,Nanjing University of Science and Technology, Nanjing 210094, China

Received:2019-12-26 Online:2020-09-01 Published:2020-09-03
Contact: LIU Wei
Supported by:
State Grid Zhejiang Electric Power Supply Co., Ltd. Collective Enterprise Technology Project(2019-LHKJ-011)

摘要/Abstract

摘要：

为了提升孤岛微电网频率抗干扰性,提出一种基于强化学习的孤岛微电网多源协调频率控制方法。针对微电网频率偏差进行Q学习,动态调节多个分布式电源的下垂控制参数以改变其输出功率,实现微电网内多源协调有功频率控制。首先,介绍了Q学习算法的基本原理;其次,提出基于Q学习的频率恢复控制方法,并设计基于Q学习算法的控制器,利用Q学习算法动态修正下垂参数,协调微电网多个分布式电源进行频率恢复控制;最后,利用MATLAB建立典型微电网仿真模型,并基于S-function自定义建立强化学习控制器,从Q学习训练过程、频率控制响应特性多个方面验证了所提方法的有效性和适应性。

关键词: 强化学习, 孤岛微电网, 频率控制, Q学习

Abstract:

In order to improve the frequency anti-interference of the island microgrid, a coordinated frequency control method based on reinforcement learning for island microgrid is proposed. The proposed control method performs Q-learning on the basis of the frequency deviation of the microgrid, and dynamically adjusts the droop control parameters of multiple distributed power sources to change their output power, to realize multi-source coordinated active frequency control in the microgrid. Firstly, the principle of Q-learning algorithm is introduced. Secondly, a frequency recovery control method based on Q-learning is proposed, and a controller based on the Q-learning algorithm is set up. The Q-learning algorithm is used to dynamically correct the droop parameters and coordinate multiple distributed power sources in the microgrid for frequency recovery control. Finally, MATLAB is used to establish a typical microgrid simulation model, and on the basis of S-function, a self-defined reinforcement learning controller is established to verify the effectiveness and adaptability of the proposed method from the aspects of the Q-learning training process and frequency control response characteristics.

Key words: reinforcement learning, island microgrid, frequency control, Q-learning

中图分类号:

TM712

姚建华, 胡晟, 王冠, 沈云, 姜林林, 冯宇立, 龚成亚, 张照轩, 柳伟. 基于强化学习的孤岛微电网多源协调频率控制方法[J]. 电力建设, 2020, 41(9): 69-75.

YAO Jianhua, HU Sheng, WANG Guan, SHEN Yun, JIANG Linlin, FENG Yuli, GONG Chengya, ZHANG Zhaoxuan, LIU Wei. Multi-source Coordinated Frequency Control Method Based on Reinforcement Learning for Island Microgrid[J]. ELECTRIC POWER CONSTRUCTION, 2020, 41(9): 69-75.

图/表 13

图1 强化学习基本框架

Fig.1 Basic framework of reinforcement learning

图2 分布式电源并联连接结构

Fig.2 Parallel connection structure of distributed power sources

图3 基于Q学习的频率控制方式

Fig.3 Frequency control method based on Q-learning

图4 Q表训练流程

Fig.4 Flowchart of Q table training

图5 微电网仿真结构

Fig.5 Structure the simulation microgrid

图6 Q学习预学习过程

Fig.6 Pre-learning process of Q-learning

图7 预学习场景分布式电源输出功率

Fig.7 Distributed power output in pre-learning scenarios

图8 预学习场景系统频率

Fig.8 System frequency in pre-learning scenarios

图9 分布式电源接入后的Q学习预学习过程

Fig.9 The pre-learning process of Q-learning after the access of distributed power

图10 分布式电源接入后的功率变化

Fig.10 Power changes after the access of distributed power

图11 分布式电源接入后系统频率变化

Fig.11 System frequency change after the access of distributed power

图12 负荷波动场景Q学习控制调节量变化

Fig.12 Change of Q-learning control adjustment amount in load fluctuation scenarios

图13 负荷波动场景系统频率变化

Fig.13 System frequency change in load fluctuation scenarios

参考文献 23

[1]	刘开俊, 高艺, 宋福龙 . 多适应性电网规划风险评估框架设计及应用展望[J]. 电力建设, 2014,35(12):1-5. doi: 10.3969/j.issn.1000-7229.2014.12.001 URL
	LIU Kaijun, GAO Yi, SONG Fulong . Risk assessment framework design on flexibility power grid planning and its application prospect[J]. Electric Power Construction, 2014,35(12):1-5. doi: 10.3969/j.issn.1000-7229.2014.12.001 URL
[2]	韩丰, 高艺, 宋福龙 , 等. 电网规划评估方法及实用化技术[J]. 电力建设, 2014,35(12):6-13. doi: 10.3969/j.issn.1000-7229.2014.12.002 URL
	HAN Feng, GAO Yi, SONG Fulong , et al. Evaluation method and practical technology on power grid planning[J]. Electric Power Construction, 2014,35(12):6-13. doi: 10.3969/j.issn.1000-7229.2014.12.002 URL
[3]	刘梦, 杨作梁 . 考虑网架结构和分布式电源的主动配电系统双层规划[J]. 电测与仪表, 2018,55(14):46-50, 81.
	LIU Meng, YANG Zuoliang . Bi-level programming of active distribution system considering grid structure and distributed generation[J]. Electrical Measurement & Instrumentation, 2018,55(14):46-50, 81.
[4]	杨鹏龙, 董海鹰 . 基于改进型同步逆变器技术的微电网变换器[J]. 电机与控制应用, 2018,45(1):7-11.
	YANG Penglong, DONG Haiying . A grid-friendly converter based on improved synchronous inverter technology[J]. Electric Machines & Control Application, 2018,45(1):7-11.
[5]	赵晶晶, 吕雪, 符杨 , 等. 基于可变系数的双馈风机虚拟惯量与超速控制协调的风光柴微电网频率调节技术[J]. 电工技术学报, 2015,30(5):59-68.
	ZHAO Jingjing, LÜ Xue, FU Yang , et al. Frequency regulation of the wind/photovoltaic/diesel microgrid based on DFIG cooperative strategy with variable coefficients between virtual inertia and over-speed control[J]. Transactions of China Electrotechnical Society, 2015,30(5):59-68.
[6]	陈萌, 肖湘宁, 罗超 . 基于虚拟同步发电机的微电网延时补偿二次频率控制[J]. 电工技术学报, 2018,33(16):3845-3854.
	CHEN Meng, XIAO Xiangning, LUO Chao . Secondary frequency control including delay compensation in microgrids based on virtual synchronous generator[J]. Transactions of China Electrotechnical Society, 2018,33(16):3845-3854.
[7]	王金玉, 潘阳阳, 钱勇 . 适用于岛屿环境的微电网发电系统设计[J]. 电力学报, 2019,34(4):395-398, 410.
	WANG Jinyu, PAN Yangyang, QIAN Yong . Design of microgrid power generation system for island environment[J]. Journal of Electric Power, 2019,34(4):395-398, 410.
[8]	赵卓立, 杨苹, 许志荣 , 等. 多源多变换微电网系统建模与动态稳定分析方法[J]. 控制理论与应用, 2017,34(8):1062-1073.
	ZHAO Zhuoli, YANG Ping, XU Zhirong , et al. Modeling and dynamic stability analysis method of microgrids with multiple energy resources and multiple conversions[J]. Control Theory & Applications, 2017,34(8):1062-1073.
[9]	王波, 张保会, 郝治国 . 基于功率监测和频率变化率的孤岛微电网紧急切负荷控制[J]. 电力系统自动化, 2015,39(8):33-37.
	WANG Bo, ZHANG Baohui, HAO Zhiguo . Emergency load shedding control for islanded microgrid based on power monitoring and rate of frequency change[J]. Automation of Electric Power Systems, 2015,39(8):33-37.
[10]	洪灏灏, 顾伟, 黄强 , 等. 微电网中多虚拟同步机并联运行有功振荡阻尼控制[J]. 中国电机工程学报, 2019,39(21):6247-6255.
	HONG Haohao, GU Wei, HUANG Qiang , et al. Power oscillation damping control for microgrid with multiple VSG units[J]. Proceedings of the CSEE, 2019,39(21):6247-6255.
[11]	张羽, 李咸善 . 基于频率调整策略的微电网多目标优化自愈控制[J]. 电网技术, 2017,41(3):831-839.
	ZHANG Yu, LI Xianshan . Multi-objective optimization for microgrid self-healing control based on frequency regulation strategy[J]. Power System Technology, 2017,41(3):831-839.
[12]	赵晶晶, 吕雪, 符杨 , 等. 基于双馈感应风力发电机虚拟惯量和桨距角联合控制的风光柴微电网动态频率控制[J]. 中国电机工程学报, 2015,35(15):3815-3822. doi: 10.13334/j.0258-8013.pcsee.2015.15.009 URL
	ZHAO Jingjing, LÜ Xue, FU Yang , et al. Dynamic frequency control strategy of wind/photovoltaic/diesel microgrid based on DFIG virtual inertia control and pitch angle control[J]. Proceedings of the CSEE, 2015,35(15):3815-3822.
[13]	张春雪, 黎灿兵, 冯伟 , 等. 孤立运行光/储微电网中储能变流器暂态功率波动协调抑制策略[J]. 中国电机工程学报, 2018,38(8):2302-2314, 2540.
	ZHANG Chunxue, LI Canbing, FENG Wei , et al. A coordinated transient power fluctuation suppression strategy for power conversion system in islanded PV/storage microgrid[J]. Proceedings of the CSEE, 2018,38(8):2302-2314, 2540.
[14]	ZHAI H, YI H, ZHUO F, et al. Online control of SVG for network unbalance suppression [C]//2016 IEEE 8th International Power Electronics and Motion Control Conference (IPEMC-ECCE Asia). IEEE, 2016: 137-141.
[15]	WANG Z X, YI H, ZHUO F, et al. A self-adaptive power balance control strategy for PV inverters in islanded microgrids [C]//2016 IEEE Applied Power Electronics Conference and Exposition (APEC). IEEE, 2016: 3358-3363.
[16]	姚玮, 陈敏, 牟善科 , 等. 基于改进下垂法的微电网逆变器并联控制技术[J]. 电力系统自动化, 2009,33(6):77-80, 94.
	YAO Wei, CHEN Min, MOU Shanke , et al. Paralleling control technique of microgrid inverters based on improved droop method[J]. Automation of Electric Power Systems, 2009,33(6):77-80, 94.
[17]	杨志淳, 刘开培, 乐健 , 等. 孤岛运行微电网中模糊PID下垂控制器设计[J]. 电力系统自动化, 2013,37(12):19-23, 68. doi: 10.7500/AEPS201203279 URL
	YANG Zhichun, LIU Kaipei, LE Jian , et al. Design of fuzzy PID droop controllers for islanded microgrids[J]. Automation of Electric Power Systems, 2013,37(12):19-23, 68. doi: 10.7500/AEPS201203279 URL
[18]	LI P, WANG W, YANG X L, et al. A droop control method of microsources based on divided self-adjusting slope coefficient [C]//2010 International Conference on Power System Technology. IEEE, 2010: 1-6.
[19]	周贤正, 荣飞, 吕志鹏 , 等. 低压微电网采用坐标旋转的虚拟功率V/f下垂控制策略[J]. 电力系统自动化, 2012,36(2):47-51, 63.
	ZHOU Xianzheng, RONG Fei, LÜ Zhipeng , et al. A coordinate rotational transformation based virtual power V/f droop control method for low voltage microgrid[J]. Automation of Electric Power Systems, 2012,36(2):47-51, 63.
[20]	郜登科, 姜建国, 张宇华 . 使用电压-相角下垂控制的微电网控制策略设计[J]. 电力系统自动化, 2012,36(5):29-34.
	GAO Dengke, JIANG Jianguo, ZHANG Yuhua . Design of microgrid control strategy using voltage amplitude and phase angle droop control[J]. Automation of Electric Power Systems, 2012,36(5):29-34.
[21]	MAJUMDER R, LEDWICH G, GHOSH A , et al. Droop control of converter-interfaced microsources in rural distributed generation[J]. IEEE Transactions on Power Delivery, 2010,25(4):2768-2778. doi: 10.1109/TPWRD.2010.2042974 URL
[22]	王健, 张平陆, 赵忠英 , 等. 结合神经网络和Q(λ)-learning的路径规划方法[J]. 自动化与仪表, 2019,34(9):1-4.
	WANG Jian, ZHANG Pinglu, ZHAO Zhongying , et al. Path planning method based on neural network and Q(λ)-learning[J]. Automation & Instrumentation, 2019,34(9):1-4.
[23]	薛晗, 邵哲平, 方琼林 , 等. 基于强化学习的倒立摆分数阶梯度下降RBF控制[J]. 控制与决策, 2019. http://kzyjc.alljournals.cn/ch/reader/view_abstract.aspx?file_no=201906090000002&flag=2. URL pmid: 26236754
	XUE Han, SHAO Zheping, FANG Qionglin , et al. Reinforcement learning based fractional gradient descent RBF neural network control of inverted pendulum [J]. Control and Decision, 2019. http://kzyjc.alljournals.cn/ch/reader/view_abstract.aspx?file_no=201906090000002&flag=2. doi: 10.1037/dec0000030 URL pmid: 26236754

基于强化学习的孤岛微电网多源协调频率控制方法

Multi-source Coordinated Frequency Control Method Based on Reinforcement Learning for Island Microgrid

RichHTML

PDF

可视化

摘要/Abstract

引用本文

使用本文

图/表 13

参考文献 23

相关文章 9

编辑推荐

Metrics

本文评价

[1]	郝旭东, 孙伟, 程定一, 张国强, 匡洪辉. 基于Q强化学习的综合能源服务商现货市场申报策略研究[J]. 电力建设, 2020, 41(9): 132-138.
[2]	陆秋瑜，杨银国，王中冠，谭嫣，夏天，伍双喜. 基于次梯度投影分布式控制法的虚拟电厂经济性一次调频[J]. 电力建设, 2020, 41(3): 79-85.
[3]	史景坚，周文涛，张宁，陈桥，刘金涛，曹振博，陈懿，宋航，刘友波. 含储能系统的配电网电压调节深度强化学习算法[J]. 电力建设, 2020, 41(3): 71-78.
[4]	刘东明，杨杨，王军波，史海涛，马鸿君，张虹. 基于事件触发的互联电网负荷频率模型预测输出反馈控制[J]. 电力建设, 2020, 41(2): 108-117.
[5]	钟永洁，翟苏巍，孙永辉. 孤岛模式下互联微电网的自适应同步频率控制[J]. 电力建设, 2019, 40(10): 75-83.
[6]	朱全胜,常烨骙，张振安，赵华，晋萃萃，吴玉琼，刘娆. 面向各类功率扰动的频率控制及评估问题探讨[J]. 电力建设, 2018, 39(9): 129-136.
[7]	瞿艳霞，马覃峰，时伯年. 柔性直流互联电网的孤岛频率快速控制技术 [J]. 电力建设, 2017, 38(6): 73-.
[8]	孟明, 苏亚慧，吴亚帆. 基于旋转电容的直流电网软开关DC-DC变换器设计[J]. 电力建设, 2017, 38(11): 32-.
[9]	徐攀腾. 云广特高压直流输电工程送端孤岛频率控制分析[J]. 电力建设, 2011, 32(11): 48-50.