基于深度Q学习的含电动汽车孤岛微电网负荷频率控制策略

doi:10.12204/j.issn.1000-7229.2022.04.010

电力建设 ›› 2022, Vol. 43 ›› Issue (4): 91-99.doi: 10.12204/j.issn.1000-7229.2022.04.010

基于深度Q学习的含电动汽车孤岛微电网负荷频率控制策略

范培潇¹(), 杨军¹(), 肖金星², 徐冰雁², 叶影², 李勇汇¹, 李蕊¹()

1.武汉大学电气与自动化学院,武汉市 430072
2.国网上海市电力公司, 上海市 200122

收稿日期:2021-05-31 出版日期:2022-04-01 发布日期:2022-03-24
通讯作者: 李蕊 E-mail:whufpx0408@163.com;JYang@whu.edu.cn;lirui@whu.edu.cn
作者简介:范培潇(1999),男,硕士研究生,主要研究方向为深度强化学习、微电网控制,E-mail: whufpx0408@163.com;
杨军(1977),男,教授,博士生导师,主要研究方向为电动汽车、电力系统运行安全与稳定等,E-mail: JYang@whu.edu.cn;
肖金星(1983),男,硕士,高级工程师,主要研究方向为电力经济;
徐冰雁(1980),男,硕士,高级工程师,主要研究方向为电力系统;
叶影(1985),男,硕士,高级工程师,主要研究方向为电力系统;
李勇汇(1973),男,博士,副教授,主要研究方向为电力系统运行控制。
基金资助:
国家电网公司科技项目(52093220000H)

Load Frequency Control Strategy Based on Deep Q Learning for Island Microgrid with Electric Vehicles

FAN Peixiao¹(), YANG Jun¹(), XIAO Jinxing², XU Bingyan², YE Ying², LI Yonghui¹, LI Rui¹()

1. School of Electrical and Automation, Wuhan University, Wuhan 430072, China
2. State Grid Shanghai Municipal Electric Power Company, Shanghai 200122, China

Received:2021-05-31 Online:2022-04-01 Published:2022-03-24
Contact: LI Rui E-mail:whufpx0408@163.com;JYang@whu.edu.cn;lirui@whu.edu.cn
Supported by:
State Grid Corporation of China Research Program(52093220000H)

摘要/Abstract

摘要：

负荷频率控制对维持孤岛微电网的稳定运行有着至关重要的意义。针对微电网受到强随机扰动和网络拓扑参数改变时的频率控制问题,文章提出了基于深度Q学习(deep Q-learning, DQN)的含电动汽车孤岛微电网负荷频率控制策略。首先,建立了考虑用户充电行为随机性的集群电动汽车频率控制模型,从而搭建出包含光伏、风电、微型燃气轮机、电动汽车及其随机功率增量约束的微电网负荷频率控制(load frequency control, LFC)模型。其次,设计了基于DQN的频率控制器结构,并依次完成了状态空间、动作空间以及奖励函数的定义,并通过调节得到了最优超参数。最后,仿真结果表明,与PI控制、FUZZY控制相比,文章所提出的DQN控制器具备在线学习和经验回放能力,能更有效地应对强随机性的微电网LFC问题,同时也能更好地适应系统网络拓扑参数与结构改变的复杂运行工况。

关键词: 孤岛微电网, 电动汽车, 频率控制, 深度Q学习算法

Abstract:

The load frequency control is of vital importance to maintain the stable operation of the island microgrid. Aiming at the frequency control problem when the microgrid is subjected to strong random disturbances and network topology parameters change, this paper proposes a load frequency control strategy based on deep Q-learning (DQN) for island microgrid with electric vehicles. Firstly, a frequency control model of electric vehicle considering the randomness of user charging behavior is established, and a load frequency control (LFC) model for microgrid including photovoltaic power, wind power, micro gas turbines, electric vehicles and their random power increment constraints is built. Secondly, the structure of frequency controller based on DQN is designed, and the definitions of state space, action space and reward function are completed in turn, and the optimal hyperparameters is obtained through adjustment. Finally, the simulation results show that, compared with PI control and FUZZY control, the DQN controller proposed in this paper has the ability of online learning and experience playback, which can deal with the LFC problem of microgrid with strong randomness more effectively, and it can better adapt to the complex operating conditions of system parameters and structure changes.

Key words: island microgrid, electric vehicles, frequency control, deep Q-learning

中图分类号:

TM73

范培潇, 杨军, 肖金星, 徐冰雁, 叶影, 李勇汇, 李蕊. 基于深度Q学习的含电动汽车孤岛微电网负荷频率控制策略[J]. 电力建设, 2022, 43(4): 91-99.

FAN Peixiao, YANG Jun, XIAO Jinxing, XU Bingyan, YE Ying, LI Yonghui, LI Rui. Load Frequency Control Strategy Based on Deep Q Learning for Island Microgrid with Electric Vehicles[J]. ELECTRIC POWER CONSTRUCTION, 2022, 43(4): 91-99.

图/表 19

图1 微型燃气轮机负荷频率控制响应模型

Fig.1 Load frequency control response model of micro gas turbine

图2 电动汽车的充放电约束边界

Fig.2 Boundary of charging and discharging constraints of electric vehicle

图3 控制指令在充电站内的分布

Fig.3 Distribution of control commands in charging stations

图4 单体EV的输出功率增量约束

Fig.4 Output power increment constraint of single EV

图5 随机功率增量约束下的电动汽车频率控制模型

Fig.5 Frequency control model of electric vehicle under random power increment constraint

图6 含电动汽车的微电网负荷频率控制模型

Fig.6 Load frequency control model of microgrid with electric vehicle

图7 深度Q学习的主要流程

Fig.7 The main process of deep Q learning

图8 基于DQN的微电网LFC控制器结构

Fig.8 Microgrid LFC controller structure based on DQN

表1 不同参数设置下的收敛测试结果

Table 1 Convergence test results under different parameters

图9 含电动汽车的孤岛微电网LFC结构

Fig.9 LFC structure of island microgrid with electric vehicles

表2 微电网LFC模型的系统参数

Table 2 System parameters of microgrid LFC model

图10 预学习阶段的随机输入函数

Fig.10 Random perturbation function in the pre-learning phase

图11 智能体的预学习过程

Fig.11 Pre-learning process of agent

图12 孤岛微电网受到的强随机扰动

Fig.12 Strong random disturbance to island microgrid

图13 强随机扰动下的微电网频率偏差

Fig.13 Frequency deviation of microgridunder strong random disturbance

图14 强随机扰动下的MT、EV输出功率增量

Fig.14 Output power increment of MT and EV under strong random disturbance

表3 强随机扰动下的频率偏差仿真结果

Table 3 Simulation results under strong random disturbance

图15 调频机组故障情况下的微电网频率偏差

Fig.15 Frequency deviation of microgrid under fault condition of FM unit

表4 调频机组故障情况下的频率偏差仿真结果

Table 4 Simulation results under fault condition of FM unit

参考文献 24

[1]	米阳, 郎中杰, 何星瑭, 等. 孤岛微电网鲁棒自适应协调控制[J]. 控制工程, 2021, 28(5):924-930.
	MI Yang, LANG Zhongjie, HE Xingtang, et al. Coordinated control for isolated microgrid based on robust adaptive method[J]. Control Engineering of China, 2021, 28(5):924-930.
[2]	KARKEVANDI A E, DARYANI M J, USTA Ö. ANFIS-based Intelligent PI controller for secondary frequency and voltage control of microgrid[C]// 2018 IEEE PES Innovative Smart Grid Technologies Conference Europe (ISGT-Europe). IEEE, 2018.
[3]	MAHDI M M, AHMAD A Z. Load frequency control in microgrid using fuzzy logic table control[C]// 2017 11th IEEE International Conference on Compatibility, Power Electronics and Power Engineering. Cadiz: IEEE, 2017: 318-323.
[4]	CHAE S H, KIM G H, CHOI Y J, et al. Design of isolated microgrid system considering controllable EV charging demand[J]. Sustainability, 2020, 12(22):9746.
[5]	李航, 李国杰, 汪可友. 基于深度强化学习的电动汽车实时调度策略[J]. 电力系统自动化, 2020, 44(22):161-167.
	LI Hang, LI Guojie, WANG Keyou. Real-time dispatch strategy for electric vehicles based on deep reinforcement learning[J]. Automation of Electric Power Systems, 2020, 44(22):161-167.
[6]	林婷, 高亮. 含电动汽车的微电网规划研究[J]. 电力科学与技术学报, 2021, 36(4):53-58.
	LIN Ting, GAO Liang. Planning research of micro-grid with electric vehicles[J]. Journal of Electric Power Science and Technology, 2021, 36(4):53-58
[7]	ZIRAS C, PROSTEJOVSKY A M, BINDNER H W., et al. Decentralized and discretized control for storage systems offering primary frequency control[J]. Elsevier, 2019, 177(C).
[8]	YANG Jun, ZENG Zhili, TANG Yufei, et al. Load frequency control in isolated micro-grids with electrical vehicles based on multivariable generalized predictive theory[J]. Energies, 2015(8):2145-2164.
[9]	RAO Yingqing, YANG Jun, XIAO Jinxing, et al. A frequency control strategy for multimicrogrids with V2G based on the improved robust model predictive control[J]. Energy, 2021(222):119963,1-13.
[10]	万千, 夏成军, 管霖, 等. 含高渗透率分布式电源的独立微网的稳定性研究综述[J]. 电网技术, 2019, 43(2):598-612.
	WAN Qian, XIA Chengjun, GUAN Lin, et al. Review on stability of isolated microgrid with highly penetrated distributed generations[J]. Power System Technology, 2019, 43(2):598-612.
[11]	龙英文, 陈洁, 余粟. 一种分布式系统的孤岛微电网无功功率分配算法[J]. 控制工程, 2021, 28(7):1354-1359.
	LONG Yingwen, CHEN Jie, YU Su. A reactive power sharing algorithm for islanded micro-grid based on distributed system[J]. Control Engineering of China, 2021, 28(7):1354-1359.
[12]	程乐峰, 余涛, 张孝顺, 等. 机器学习在能源与电力系统领域的应用和展望[J]. 电力系统自动化, 2019, 43(1):15-31.
	CHENG Lefeng, YU Tao, ZHANG Xiaoshun, et al. Machine learning for energy and electric power systems: State of the art and prospects[J]. Automation of Electric Power Systems, 2019, 43(1):15-31.
[13]	张孝顺, 李清, 余涛, 等. 基于协同一致性迁移Q学习算法的虚拟发电部落AGC功率动态分配[J]. 中国电机工程学报, 2017, 37(5):1455-1467.
	ZHANG Xiaoshun, LI Qing, YU Tao, et al. Collaborative consensus transfer Q-learning based dynamic generation dispatch of automatic generation control with virtual generation tribe[J]. Proceedings of the CSEE, 2017, 37(5):1455-1467.
[14]	余涛, 梁海华, 周斌. 基于R (λ)学习的孤岛微电网智能发电控制[J]. 电力系统保护与控制, 2012, 40(13):7-13.
	YU Tao, LIANG Haihua, ZHOU Bin. Smart power generation control for microgrids islanded operation based on R (λ) learning[J]. Power System Protection and Control, 2012, 40(13):7-13.
[15]	唐捷, 张泽宇, 程乐峰, 等. 基于CEQ(λ)强化学习算法的微电网智能发电控制[J]. 电测与仪表, 2017, 54(1):39-45.
	TANG Jie, ZHANG Zeyu, CHENG Lefeng, et al. Smart generation control for micro-grids based on correlated equilibrium Q(λ) learning algorithm[J]. Electrical Measurement & Instrumentation, 2017, 54(1):39-45.
[16]	TAN C X, HAN R J, YE R G, et al. Adaptive learning recommendation strategy based on deep Q-learning[J]. Applied Psychological Measurement, 2020, 44(4):251-266. doi: 10.1177/0146621619858674 URL
[17]	华昊辰, 李宇童, 王同贺, 等. 一种基于混合随机H₂/H_∞方法的能源互联网边缘计算系统控制策略[J]. 中国电机工程学报, 2020, 40(21):6875-6885.
	HUA Haochen, LI Yutong, WANG Tonghe, et al. A novel stochastic mixed H₂/h_∞ control strategy for energy Internet edge computing system[J]. Proceedings of the CSEE, 2020, 40(21):6875-6885.
[18]	李晓萌, 贾宏杰, 穆云飞, 等. 时滞环境下基于电动汽车与电热泵的协调频率控制[J]. 电力自动化设备, 2020, 40(4):88-95, 110.
	LI Xiaomeng, JIA Hongjie, MU Yunfei, et al. Coordinated frequency control based on electric vehicles and heat pumps considering time-delay[J]. Electric Power Automation Equipment, 2020, 40(4):88-95, 110.
[19]	李军军, 吴政球, 谭勋琼, 等. 风力发电及其技术发展综述[J]. 电力建设, 2011, 32(8):64-72.
	LI Junjun, WU Zhengqiu, TAN Xunqiong, et al. Review of wind power generation and relative technology development[J]. Electric Power Construction, 2011, 32(8):64-72.
[20]	刘全, 翟建伟, 章宗长, 等. 深度强化学习综述[J]. 计算机学报, 2018, 41(1):1-27.
	LIU Quan, ZHAI Jianwei, ZHANG Zongzhang, et al. A survey on deep reinforcement learning[J]. Chinese Journal of Computers, 2018, 41(1):1-27.
[21]	刘建伟, 高峰, 罗雄麟. 基于值函数和策略梯度的深度强化学习综述[J]. 计算机学报, 2019, 42(6):1406-1438.
	LIU Jianwei, GAO Feng, LUO Xionglin. Survey of deep reinforcement learning based on value function and policy gradient[J]. Chinese Journal of Computers, 2019, 42(6):1406-1438.
[22]	刘俊峰, 陈剑龙, 王晓生, 等. 基于深度强化学习的微能源网能量管理与优化策略研究[J]. 电网技术, 2020, 44(10):3794-3803.
	LIU Junfeng, CHEN Jianlong, WANG Xiaosheng, et al. Energy management and optimization of multi-energy grid based on deep reinforcement learning[J]. Power System Technology, 2020, 44(10):3794-3803.
[23]	风特征数据库[DB/OL]. [2014-03-03]. http://www.wind -data.com/.
[24]	赵伟, 李雄, 乔仁飞, 等. 基于混合储能的孤岛微网VSG控制策略[J]. 电力系统保护与控制, 2021, 49(12):33-40.
	ZHAO Wei, LI Xiong, QIAO Renfei, et al. VSG control strategy of an isolated microgrid based on hybrid energy storage[J]. Power System Protection and Control, 2021, 49(12):33-40.

基于深度Q学习的含电动汽车孤岛微电网负荷频率控制策略

Load Frequency Control Strategy Based on Deep Q Learning for Island Microgrid with Electric Vehicles

RichHTML

PDF

可视化

摘要/Abstract

引用本文

使用本文

图/表 19

参考文献 24

相关文章 15

编辑推荐

Metrics

本文评价

[1]	张祥成, 刘飞, 田旭, 王世斌, 杜亚彬, 李彬. 考虑竞争偏好的数据中心电动汽车用能联合优化策略[J]. 电力建设, 2023, 44(3): 85-92.
[2]	姚寅, 朱烨冬, 李东东, 周波, 林顺富. 基于决策实验室算法对抗解释结构模型的电动汽车多场景需求响应策略分析[J]. 电力建设, 2023, 44(3): 93-104.
[3]	付卓铭, 胡俊杰, 马文帅, 姚丽. 规模化电动汽车参与电力系统二次调频研究综述[J]. 电力建设, 2023, 44(2): 1-14.
[4]	王婷, 陈晨, 谢海鹏. 配电网对分布式电源和电动汽车的承载力评估及提升方法综述[J]. 电力建设, 2022, 43(9): 12-24.
[5]	任鑫芳, 张志朝, 许李天伦, 王诗超, 刘展志, 许方圆. 计及电动汽车与温控负荷需求响应的分层能源系统优化调度[J]. 电力建设, 2022, 43(9): 77-86.
[6]	王萍萍, 许建中, 闫庆友, 林宏宇. 计及灵活性负荷资源需求响应和不确定性的楼宇微网调度双层优化模型[J]. 电力建设, 2022, 43(6): 128-140.
[7]	雷霄, 夏勇军, 杨尚瑾, 王薇薇, 陈堃, 李新年. LCC/VSC混联系统多区域间惯性支撑协调控制策略与仿真[J]. 电力建设, 2022, 43(5): 109-116.
[8]	邢超, 奚鑫泽, 何廷一, 李胜男, 刘明群. 计及风电调频的备用容量滚动优化方法[J]. 电力建设, 2022, 43(2): 117-125.
[9]	严欢, 胡俊杰, 黄旦莉, 张展宇, 岳园园. 考虑电动汽车虚拟电厂灵活性和高比例光伏接入的配电网规划[J]. 电力建设, 2022, 43(11): 14-23.
[10]	王若谷, 陈果, 王秀丽, 钱涛, 高欣. 计及风电与电动汽车随机性的两阶段机组组合研究[J]. 电力建设, 2021, 42(8): 63-70.
[11]	刘洋, 李立生, 刘志伟, 苗世洪, 张世栋, 张林利. 考虑广义储能集群参与的配电网协同控制策略[J]. 电力建设, 2021, 42(8): 89-98.
[12]	赵佳, 孟润泉, 魏斌, 王磊, 韩肖清. 计及电动汽车用户需求的直流微电网经济调度策略[J]. 电力建设, 2021, 42(7): 39-47.
[13]	范磊, 陈良亮, 罗雯茜, 董晓霄, 袁越. 考虑城市不同功能区的电动汽车负荷时空分布建模方法[J]. 电力建设, 2021, 42(6): 67-75.
[14]	薛溟枫, 毛晓波, 潘湧涛, 杨艳红, 赵振兴, 李彦君. 基于LSTM神经网络的电动汽车充电站需求响应特性封装及配电网优化运行[J]. 电力建设, 2021, 42(6): 76-85.
[15]	林铭蓉, 胡志坚, 高明鑫, 陈锦鹏. 考虑需求响应和电动汽车负荷路-电耦合特性的配电网可靠性评估[J]. 电力建设, 2021, 42(6): 86-95.