多智能体深度强化学习驱动的跨园区能源交互优化调度

doi:10.12204/j.issn.1000-7229.2024.05.007

电力建设 ›› 2024, Vol. 45 ›› Issue (5): 59-70.doi: 10.12204/j.issn.1000-7229.2024.05.007

• 大模型小样本条件下新能源规划设计与优化运行技术·栏目主持葛磊蛟副教授、孙铭阳教授、郑锋副教授、黄文焘副教授· • 上一篇下一篇

多智能体深度强化学习驱动的跨园区能源交互优化调度

李扬¹(), 马文捷¹, 卜凡金², 杨震³, 王彬⁴, 韩猛²

1.东北电力大学电气工程学院,吉林省吉林市 132012
2.国网淄博供电公司,山东省淄博市 255022
3.国网北京市电力公司,北京市 100032
4.国网济宁供电公司,山东省济宁市 272000

收稿日期:2023-07-19 出版日期:2024-05-01 发布日期:2024-04-29
通讯作者: 李扬(1980),男,教授,博士生导师,主要研究方向为综合能源系统优化调度、电力系统稳定评估,E-mail:liyang@neepu.edu.cn。
作者简介:马文捷(1999),男,硕士研究生,主要研究方向为人工智能在综合能源系统中的应用;
卜凡金(1996),男,硕士,主要研究方向为人工智能在综合能源系统中的应用;
杨震(1992),男,硕士,工程师,主要研究方向为电力系统优化运行与控制;
王彬(1995),男,硕士,工程师,主要研究方向为电力系统优化运行与控制;
韩猛(1996),男,硕士,工程师,主要研究方向为电力系统优化运行与控制。
基金资助:
吉林省自然科学基金项目(YDZJ202101ZYTS149)

Deep Reinforcement Learning-driven Cross-Community Energy Interaction Optimal Scheduling

LI Yang¹(), MA Wenjie¹, BU Fanjin², YANG Zhen³, WANG Bin⁴, HAN Meng²

1. School of Electrical Engineering, Northeast Electric Power University, Jilin 132012, Jilin Province, China
2. State Grid Zibo Power Supply Company, Zibo 255022, Shandong Province, China
3. State Grid Beijing Electric Power Company, Beijing 100032, China
4. State Grid Jining Power Supply Company, Jining 272000, Shandong Province, China

Received:2023-07-19 Published:2024-05-01 Online:2024-04-29
Supported by:
Natural Science Foundation of Jilin Province(YDZJ202101ZYTS149)

摘要/Abstract

摘要：

为协调多园区综合能源系统各个园区之间的能量交互,多能源子系统之间的能源转换,实现综合能源系统整体优化调度,提出一种利用多智能体深度强化学习算法学习不同园区的负荷特征,并在此基础上进行决策的综合调度模型。该模型将多园区综合能源系统的调度问题转化为马尔科夫决策过程,并利用深度强化学习算法进行求解,避免了对多园区、多能源子系统之间复杂的能量耦合关系进行建模。仿真结果表明,所提方法可以很好地捕捉到不同园区的负荷特性,并利用其中的互补特性协调不同园区之间进行合理的能量交互,可以实现弃风率由16.3%降低至0,并可以使总运行成本降低5 445.6元,具有良好的经济效益和环保效益。

关键词: 多智能体深度强化学习, 综合能源系统, 优化调度, 可再生能源消纳, 负荷特征学习, 多园区能量交互

Abstract:

In order to coordinate energy interactions among various communities and energy conversions among multi-energy subsystems within the multi-community integrated energy system under uncertain conditions, and achieve overall optimization and scheduling of the comprehensive energy system, this paper proposes a comprehensive scheduling model that utilizes a multi-agent deep reinforcement learning algorithm to learn load characteristics of different communities and make decisions based on this knowledge. In this model, the scheduling problem of the integrated energy system is transformed into a Markov decision process and solved using a data-driven deep reinforcement learning algorithm, which avoids the need for modeling complex energy coupling relationships between multi-communities and multi-energy subsystems. The simulation results show that the proposed method effectively captures the load characteristics of different communities and utilizes their complementary features to coordinate reasonable energy interactions among them. This leads to a reduction in wind curtailment rate from 16.3% to 0% and lowers the overall operating cost by 5445.6 Yuan, demonstrating significant economic and environmental benefits.

Key words: multi-agent deep reinforcement learning, integrated energy system, optimal scheduling, renewable energy consumption, load characteristic learning, energy interaction among communities

中图分类号:

TM732

李扬, 马文捷, 卜凡金, 杨震, 王彬, 韩猛. 多智能体深度强化学习驱动的跨园区能源交互优化调度[J]. 电力建设, 2024, 45(5): 59-70.

LI Yang, MA Wenjie, BU Fanjin, YANG Zhen, WANG Bin, HAN Meng. Deep Reinforcement Learning-driven Cross-Community Energy Interaction Optimal Scheduling[J]. ELECTRIC POWER CONSTRUCTION, 2024, 45(5): 59-70.

图/表 13

图1 多园区综合能源系统结构

Fig.1 Structure of multi-community IES

图2 居民园区系统结构

Fig.2 System structure of residential community

图3 水电联产机组结构图

Fig.3 Structure diagram of hydropower cogeneration unit

图4 淡水加热器结构原理示意图

Fig.4 Structure diagram of fresh water heater

图5 MAPPO网络结构框架

Fig.5 MAPPO network structure framework

图6 所提方法流程图

Fig.6 Flow chart of the proposed method

图7 测试系统结构图

Fig.7 Test system structure diagram

表1 各主要设备的运行参数

Table 1 Operation parameters of each main device

图8 训练过程奖励变化曲线

Fig.8 Reward curve during training

表2 不同模式运行成本对比元

Table 2 Comparison table of operating costs in different modes

图9 不同模式各个园区电力调度图

Fig.9 Power dispatching graph of each community in different modes

图10 不同模式各个园区热力调度图

Fig.10 Heat dispatching graph of each community in different modes

图11 不同模式下可再生能源消纳情况

Fig.11 Consumption of renewable energy under different modes

参考文献 41

[1]	LI Y, HAN M, YANG Z, et al. Coordinating flexible demand response and renewable uncertainties for scheduling of community integrated energy systems with an electric vehicle charging station: a bi-level approach[J]. IEEE Transactions on Sustainable Energy, 2021, 12(4): 2321-2331. doi: 10.1109/TSTE.2021.3090463 URL
[2]	邵磊, 多增森, 柴嘉启, 等. 抽蓄-风-光-火联合系统日前优化调度研究[J]. 电网与清洁能源, 2023, 39(6): 108-114.
	SHAO Lei, DUO Zengsen, CHAI Jiaqi, et al. A study on the day-ahead optimal scheduling of the pumped storage-wind-solar-thermal combined system[J]. Power System and Clean Energy, 2023, 39(6): 108-114.
[3]	YAZDANIAN M, MEHRIZI-SANI A. Distributed control techniques in microgrids[J]. IEEE Transactions on Smart Grid, 2014, 5(6): 2901-2909. doi: 10.1109/TSG.2014.2337838 URL
[4]	ZECCHINO A, PROSTEJOVSKY A M, ZIRAS C, et al. Large-scale provision of frequency control via V2G: the Bornholm power system case[J]. Electric Power Systems Research, 2019, 170: 25-34. doi: 10.1016/j.epsr.2018.12.027 URL
[5]	赵波, 张雪松, 李鹏, 等. 储能系统在东福山岛独立型微电网中的优化设计和应用[J]. 电力系统自动化, 2013, 37(1): 161-167.
	ZHAO Bo, ZHANG Xuesong, LI Peng, et al. Optimal design and application of energy storage system in dongfushan island stand-alone microgrid[J]. Automation of Electric Power Systems, 2013, 37(1): 161-167.
[6]	SIDDIQUI O, DINCER I. Design and analysis of a novel solar-wind based integrated energy system utilizing ammonia for energy storage[J]. Energy Conversion and Management, 2019, 195: 866-884. doi: 10.1016/j.enconman.2019.05.001 URL
[7]	余晓丹, 徐宪东, 陈硕翼, 等. 综合能源系统与能源互联网简述[J]. 电工技术学报, 2016, 31(1): 1-13.
	YU Xiaodan, XU Xiandong, CHEN Shuoyi, et al. A brief review to integrated energy system and energy internet[J]. Transactions of China Electrotechnical Society, 2016, 31(1): 1-13.
[8]	马喜平, 杨燕静, 沈渭程, 等. 基于改进量子粒子群算法的微能源网优化运行[J]. 电网与清洁能源, 2022, 38(7): 47-53.
	MA Xiping, YANG Yanjing, SHEN Weicheng, et al. Optimized operation of micro energy grid based on improved quantum particle swarm optimization (IQPSO) algorithm[J]. Power System and Clean Energy, 2022, 38(7): 47-53.
[9]	孙一凡, 许子芸, 王林. 计及多时间尺度灵活性的电-气互联系统优化调度方法[J]. 浙江电力, 2023, 42(7): 9-17.
	SUN Yifan, XU Ziyun, WANG Lin. Optimal scheduling method for electricity-gas interconnected system considering multi-time-scale flexibility[J]. Zhejiang Electric Power, 2023, 42(7): 9-17.
[10]	LI Y, WANG C L, LI G Q, et al. Improving operational flexibility of integrated energy system with uncertain renewable generations considering thermal inertia of buildings[J]. Energy Conversion and Management, 2020, 207: 112526. doi: 10.1016/j.enconman.2020.112526 URL
[11]	LI P, YU D W, YANG M, et al. Flexible look-ahead dispatch realized by robust optimization considering CVaR of wind power[J]. IEEE Transactions on Power Systems, 2018, 33(5): 5330-5340. doi: 10.1109/TPWRS.59 URL
[12]	LIANG R H, LIAO J H. A fuzzy-optimization approach for generation scheduling with wind and solar energy systems[J]. IEEE Transactions on Power Systems, 2007, 22(4): 1665-1674. doi: 10.1109/TPWRS.2007.907527 URL
[13]	雷嘉明, 姜爱华, 吴新飞, 等. 计及源荷不确定性的综合能源系统近端策略优化调度[J]. 电力科学与技术学报, 2023, 38(5): 1-11.
	LEI Jiaming, JIANG Aihua, WU Xinfei, et al. Proximal policy optimization dispatch of integrated energy system considering source-load uncertainty[J]. Journal of Electric Power Science and Technology, 2023, 38(5): 1-11.
[14]	叶琳, 项中明, 张静, 等. 基于多强化学习智能体架构的电网运行方式调节方法[J]. 浙江电力, 2022, 41(6): 1-7.
	YE Lin, XIANG Zhongming, ZHANG Jing, et al. An operating condition adjustment method for power grid using multi-DRL-agent architecture[J]. Zhejiang Electric Power, 2022, 41(6): 1-7.
[15]	CAO J, HARROLD D, FAN Z, et al. Deep reinforcement learning-based energy storage arbitrage with accurate lithium-ion battery degradation model[J]. IEEE Transactions on Smart Grid, 2020, 11(5): 4513-4521. doi: 10.1109/TSG.5165411 URL
[16]	LI H P, WAN Z Q, HE H B. Constrained EV charging scheduling based on safe deep reinforcement learning[J]. IEEE Transactions on Smart Grid, 2020, 11(3): 2427-2439. doi: 10.1109/TSG.5165411 URL
[17]	BUI V H, HUSSAIN A, KIM H M. Double deep Q-learning-based distributed operation of battery energy storage system considering uncertainties[J]. IEEE Transactions on Smart Grid, 2020, 11(1): 457-469. doi: 10.1109/TSG.5165411 URL
[18]	YAN L F, CHEN X, ZHOU J Y, et al. Deep reinforcement learning for continuous electric vehicles charging control with dynamic user behaviors[J]. IEEE Transactions on Smart Grid, 2021, 12(6): 5124-5134. doi: 10.1109/TSG.2021.3098298 URL
[19]	BALDI S, KORKAS C D, LV M L, et al. Automating occupant-building interaction via smart zoning of thermostatic loads: a switched self-tuning approach[J]. Applied Energy, 2018, 231: 1246-1258. doi: 10.1016/j.apenergy.2018.09.188 URL
[20]	GAO J K, LI Y, WANG B, et al. Multi-microgrid collaborative optimization scheduling using an improved multi-agent soft actor-critic algorithm[J]. Energies, 2023, 16(7): 3248. doi: 10.3390/en16073248 URL
[21]	蔺伟山, 王小君, 孙庆凯, 等. 不确定性环境下基于深度强化学习的综合能源系统动态调度[J]. 电力系统保护与控制, 2022, 50(18): 50-60.
	LIN Weishan, WANG Xiaojun, SUN Qingkai, et al. Dynamic dispatch of an integrated energy system based on deep reinforcement learning in an uncertain environment[J]. Power System Protection and Control, 2022, 50(18): 50-60.
[22]	翁利国, 练德强, 张阳辉, 等. 考虑网络重构的多园区储能多目标优化配置[J]. 电力科学与技术学报, 2023, 38(3): 54-64.
	WENG Liguo, LIAN Deqiang, ZHANG Yanghui, et al. Multi-objective optimal configuration of multi-park energy storage considering network re-configuration[J]. Journal of Electric Power Science and Technology, 2023, 38(3): 54-64.
[23]	肖婷予, 刘朝, 刘浪, 等. 基于热渗透原理的水电联产系统性能分析[J]. 工程热物理学报, 2022, 43(12): 3129-3136.
	XIAO Tingyu, LIU Chao, LIU Lang, et al. Performance analysis of a thermo-osmotic water and electricity cogeneration system[J]. Journal of Engineering Thermophysics, 2022, 43(12): 3129-3136.
[24]	ALMULLA A, HAMAD A, GADALLA M. Integrating hybrid systems with existing thermal desalination plants[J]. Desalination, 2005, 174(2):171-192. doi: 10.1016/j.desal.2004.08.041 URL
[25]	LIN S H, ELIMELECH M. Staged reverse osmosis operation: configurations, energy efficiency, and application potential[J]. Desalination, 2015, 366: 9-14. doi: 10.1016/j.desal.2015.02.043 URL
[26]	WANG Q, YANG L, HUANG K. Fast prediction and sensitivity analysis of gas turbine cooling performance using supervised learning approaches[J]. Energy, 2022, 246: 123373. doi: 10.1016/j.energy.2022.123373 URL
[27]	LI Y M, PAN W B, XIA J J, et al. Combined heat and water system for long-distance heat transportation[J]. Energy, 2019, 172: 401-408. doi: 10.1016/j.energy.2019.01.139
[28]	吴放, 马元华, 缪正强, 等. 一种基于反渗透海水淡化和核能供热的水热同传系统: CN112551752A[P]. 2021-03-26.
[29]	LI Y, BU F J, LI Y Z, et al. Optimal scheduling of island integrated energy systems considering multi-uncertainties and hydrothermal simultaneous transmission: a deep reinforcement learning approach[J]. Applied Energy, 2023, 333: 120540. doi: 10.1016/j.apenergy.2022.120540 URL
[30]	LI Y, WANG J L, ZHAO D B, et al. A two-stage approach for combined heat and power economic emission dispatch: combining multi-objective optimization with integrated decision making[J]. Energy, 2018, 162: 237-254. doi: 10.1016/j.energy.2018.07.200 URL
[31]	原希尧, 王关涛, 朱若源, 等. 碳-绿色证书交易机制下考虑回收P2G余热和需求响应的PIES优化调度[J]. 电力建设, 2023, 44(3): 25-35. doi: 10.12204/j.issn.1000-7229.2023.03.003
	YUAN Xiyao, WANG Guantao, ZHU Ruoyuan, et al. Optimal scheduling of park integrated energy system with P2G waste heat recovery and demand response under carbon-green certificate trading mechanism[J]. Electric Power Construction, 2023, 44(3): 25-35. doi: 10.12204/j.issn.1000-7229.2023.03.003
[32]	匡萃浙, 张勇军, 王群, 等. 基于区域能源中心的居民电-热用能多目标优化[J]. 电力建设, 2021, 42(2): 58-67. doi: 10.12204/j.issn.1000-7229.2021.02.008
	KUANG Cuizhe, ZHANG Yongjun, WANG Qun, et al. District energy center-based multi-objective optimization of resident electricity-heat energy consumption[J]. Electric Power Construction, 2021, 42(2): 58-67. doi: 10.12204/j.issn.1000-7229.2021.02.008
[33]	牟晨璐, 丁涛, 李立, 等. 基于分层分布式调度的多园区服务商与综合能源供应商两级协调优化运行模型[J]. 电网技术, 2021, 45(11): 4336-4348.
	MU Chenlu, DING Tao, LI Li, et al. Coordinated optimization operation model of multiple load-serving entities and energy supply companies based on two-level distributed scheduling[J]. Power System Technology, 2021, 45(11): 4336-4348.
[34]	KIRAN B R, SOBH I, TALPAERT V, et al. Deep reinforcement learning for autonomous driving: a survey[J]. IEEE Transactions on Intelligent Transportation Systems, 2022, 23(6): 4909-4926. doi: 10.1109/TITS.2021.3054625 URL
[35]	BELLMAN R. A Markovian decision process[J]. Indiana University Mathematics Journal, 1957, 6(4): 679-684. doi: 10.1512/iumj.1957.6.56038 URL
[36]	PUTERMAN M L. Markov decision processes[J]. Handbooks in Operations Research and Management Science, 1990, 2: 331-434.
[37]	张沛, 朱驻军, 谢桦. 基于深度强化学习近端策略优化的电网无功优化方法[J]. 电网技术, 2023, 47(2): 562-572.
	ZAHNG Pei, ZHU Zhujun, XIE Hua. Reactive power optimization based on proximal policy optimization of deep reinforcement learning[J]. Power System Technology, 2023, 47(2): 562-572. doi: 10.52783/pst.226 URL
[38]	ZHOU S, HU Z, GU W, et al. Combined heat and power system intelligent economic dispatch:a deep reinforcement learning approach[J]. International Journal of Electrical Power & Energy Systems, 2020, 120:106016. doi: 10.1016/j.ijepes.2020.106016 URL
[39]	巨云涛, 陈希. 基于双层多智能体强化学习的微网群分布式有功无功协调优化调度[J]. 中国电机工程学报, 2022, 42(23): 8534-8548.
	JU Yuntao, CHEN Xi. Distributed active and reactive power coordinated optimal scheduling of networked microgrids based on two-layer multi-agent reinforcement learning[J]. Proceedings of the CSEE, 2022, 42(23): 8534-8548.
[40]	孙长银, 穆朝絮. 多智能体深度强化学习的若干关键科学问题[J]. 自动化学报, 2020, 46(7): 1301-1312.
	SUN Changyin, MU Chaoxu. Important scientific problems of multi-agent deep reinforcement learning[J]. Acta Automatica Sinica, 2020, 46(7): 1301-1312.
[41]	李柏堉, 赵津蔓, 韩肖清, 等. 基于双智能体深度强化学习的电力系统静态安全预防控制方法[J]. 中国电机工程学报, 2023, 43(5): 1818-1831.
	LI Baiyu, ZHAO Jinman, HAN Xiaoqing, et al. Static security oriented preventive control of power system based on double deep reinforcement learning[J]. Proceedings of the CSEE, 2023, 43(5): 1818-1831.

多智能体深度强化学习驱动的跨园区能源交互优化调度

Deep Reinforcement Learning-driven Cross-Community Energy Interaction Optimal Scheduling

RichHTML

PDF

可视化

摘要/Abstract

引用本文

使用本文

图/表 13

参考文献 41

相关文章 15

编辑推荐

Metrics

本文评价

[1]	周祥, 李晓露, 柳劲松, 林顺富. 考虑区域间辅助奖励的配电网电压优化控制[J]. 电力建设, 2024, 45(5): 80-93.
[2]	胡枭, 李少伦, 王丹, 陈奇成, 喻洁, 杨金铎. 㶲在综合能源系统中的理论、模型、应用综述及展望[J]. 电力建设, 2024, 45(3): 1-15.
[3]	张志一, 窦震海, 于润泽, 胡亚春, 陈佳佳, 尹文良. 考虑电-热等效虚拟储能的综合能源系统低碳经济调度[J]. 电力建设, 2024, 45(3): 16-26.
[4]	李志川, 杨季平, 孙兆恒, 李子航, 刘静, 刘龙飞, 徐宪东. 计及风电接入的海上油气田能源系统分层调控[J]. 电力建设, 2024, 45(3): 69-77.
[5]	王利猛, 刘雪梦, 李扬, 常铎, 任星. 阶梯式碳交易机制下考虑需求响应的综合能源系统低碳优化调度[J]. 电力建设, 2024, 45(2): 102-114.
[6]	韩宝慧, 陆玲霞, 包哲静, 于淼. 基于多头概率稀疏自注意力模型的综合能源系统多元负荷短期预测[J]. 电力建设, 2024, 45(2): 127-136.
[7]	寇洋, 武家辉, 江欢, 张华, 杨健. 计及碳捕集和旋转备用配置的低碳优化运行[J]. 电力建设, 2024, 45(1): 102-111.
[8]	何义琼, 李鑫, 贾海清, 刘长玺, 刘茹, 谷旭东, 雷霞. 考虑风电时空特性和相对波动率的平衡成本分摊方法[J]. 电力建设, 2024, 45(1): 157-166.
[9]	王佳蕊, 孙勇, 胡枭, 李德鑫, 吕项羽, 李宝聚, 李少伦, 陈厚合. 基于MICP的多能耦合综合能源系统可再生能源消纳能力研究[J]. 电力建设, 2023, 44(8): 157-170.
[10]	杨玉龙, 王淞, 陈韬, 金荣, 董小瑞, 程芬, 张鹏. 基于蓄热水箱温度可行域模糊确定的电锅炉优化调度方法[J]. 电力建设, 2023, 44(7): 111-120.
[11]	宁毕武, 张冠锋, 王海鑫, 颜宁. 考虑梯次利用储能系统安全阈值的综合能源系统低碳经济调度方法[J]. 电力建设, 2023, 44(6): 1-11.
[12]	赵海祺, 董树锋, 南斌, 张祥龙. 考虑电炉短流程的钢铁工业园区综合能源系统低碳运行模式分析[J]. 电力建设, 2023, 44(6): 23-32.
[13]	邱彬, 姚铭承, 王凯, 刘宏志, 慕会宾, 张志超. 计及碳抵消-碳交易下天然气管网压力能发电的IES低碳经济优化调度[J]. 电力建设, 2023, 44(6): 41-52.
[14]	王永利, 韩煦, 刘晨, 蔡成聪, 周泯含, 马子奔. 基于生-光耦合利用的乡村电-热综合能源系统规划[J]. 电力建设, 2023, 44(3): 1-14.
[15]	原希尧, 王关涛, 朱若源, 白星振, 葛磊蛟, 李顾辉. 碳—绿色证书交易机制下考虑回收P2G余热和需求响应的PIES优化调度[J]. 电力建设, 2023, 44(3): 25-35.