A Novel Energy Management Method Based on Modified Deep Q Network Algorithm for Multi-park Integrated Energy System

doi:10.12204/j.issn.1000-7229.2022.12.009

Abstract

Abstract:

Multi-park integrated energy system can significantly improve the operation economy by complementing each other with multiple energy sources. However, the complex interactions between parks and multi-energy coupling decisions can bring challenging problems such as large decision space and difficult convergence of algorithms to the energy management of multi-park integrated energy system. To solve the above problems, an energy management method based on modified deep Q network (MDQN) algorithm for multi-park integrated energy systems is proposed. Firstly, the external meteorological data and historical interactive power data independent of the park are used to construct a long short-term memory (LSTM) deep network-based external interactive environmental equivalence model for each park integrated energy system, which reduces the computational complexity of the reinforcement learning reward function. Secondly, an improved DQN algorithm based on k-first sampling strategy is proposed to replace the greedy strategy with k-first sampling strategy to overcome the inefficiency of exploration in large-scale action spaces. Finally, the results are validated in an algorithm containing three integrated energy systems in the park, and show that the MDQN algorithm has better convergence and stability compared with the original DQN algorithm, while it can improve the economic efficiency of the park by 29.16%.

Key words: park integrated energy system, deep reinforcement learning, energy management, modified deep Q network (MDQN) algorithm

CLC Number:

TM734

XUE Mingfeng, MAO Xiaobo, XIAO Hao, PU Xiaowei, PEI Wei. A Novel Energy Management Method Based on Modified Deep Q Network Algorithm for Multi-park Integrated Energy System[J]. ELECTRIC POWER CONSTRUCTION, 2022, 43(12): 83-93.

Figures/Tables 11

Fig.1 Overall structure of the park-level integrated energy system

Fig.2 Diagram of TCL control flow

Fig.3 Equivalent package model of park-level integrated energy system

Fig.4 MDQN method

Fig.5 Simulation structure of the park-level integrated energy system

Table 1 Simulation parameters

Fig.6 Trend of loss function of LSTM network

Fig.7 Comparison of reward values of MDQN and DQN algorithms

Fig.8 Energy management by DQN agent

Fig.9 Energy management by MDQN agent

Fig.10 Rewards comparison of MDQN and DQN algorithms in different scenarios

References 25

[1]	CHENG Y H, ZHANG N, WANG Y, et al. Modeling carbon emission flow in multiple energy systems[J]. IEEE Transactions on Smart Grid, 2019, 10(4): 3562-3574. doi: 10.1109/TSG.2018.2830775 URL
[2]	孔力, 裴玮, 饶建业, 等. 建设新型电力系统促进实现碳中和[J]. 中国科学院院刊, 2022, 37(4): 522-528.
	KONG Li, PEI Wei, RAO Jianye, et al. Build new power system to promote carbon neutrality[J]. Bulletin of Chinese Academy of Sciences, 2022, 37(4): 522-528.
[3]	李俊峰. “双碳”目标为能源电力发展指明方向[J]. 国家电网, 2021(3): 46-47.
[4]	XIAO H, PEI W, DONG Z M, et al. Bi-level planning for integrated energy systems incorporating demand response and energy storage under uncertain environments using novel metamodel[J]. CSEE Journal of Power and Energy Systems, 2018, 4(2): 155-167. doi: 10.17775/CSEEJPES.2017.01260 URL
[5]	肖浩, 裴玮, 孔力. 基于模型预测控制的微电网多时间尺度协调优化调度[J]. 电力系统自动化, 2016, 40(18): 7-14, 55.
	XIAO Hao, PEI Wei, KONG Li. Multi-time scale coordinated optimal dispatch of microgrid based on model predictive control[J]. Automation of Electric Power Systems, 2016, 40(18): 7-14, 55.
[6]	边辉, 陈丽娜, 马凡琳, 等. 基于中枢解耦与演化博弈的多农业园区综合能源系统优化运行[J]. 电力建设, 2022, 43(2): 26-36. doi: 10.12204/j.issn.1000-7229.2022.02.004
	BIAN Hui, CHEN Lina, MA Fanlin, et al. Optimal operation strategy based on central decoupling and evolutionary game for multiple agricultural integrated energy systems[J]. Electric Power Construction, 2022, 43(2): 26-36. doi: 10.12204/j.issn.1000-7229.2022.02.004
[7]	林文智, 杨苹, 陈芯羽, 等. 计及需求响应不确定性的园区综合能源系统日前经济优化调度[J]. 电力建设, 2021, 42(12): 9-20. doi: 10.12204/j.issn.1000-7229.2021.12.002
	LIN Wenzhi, YANG Ping, CHEN Xinyu, et al. Day-ahead optimal economic dispatch of park integrated energy system considering uncertainty of demand response[J]. Electric Power Construction, 2021, 42(12): 9-20. doi: 10.12204/j.issn.1000-7229.2021.12.002
[8]	刘玉成, 段炼, 班晓萌, 等. 基于区块链技术的园区级综合能源系统能量交互[J]. 电力建设, 2021, 42(12): 30-38. doi: 10.12204/j.issn.1000-7229.2021.12.004
	LIU Yucheng, DUAN Lian, BAN Xiaomeng, et al. Blockchain technology for distributed energy scheduling method of district-level integrated energy system[J]. Electric Power Construction, 2021, 42(12): 30-38. doi: 10.12204/j.issn.1000-7229.2021.12.004
[9]	李振, 赵鹏翔, 王楠, 等. 基于强化学习的园区综合能源系统多主体运行优化[J]. 电力系统及其自动化学报, 2021, 33(12): 60-68.
	LI Zhen, ZHAO Pengxiang, WANG Nan, et al. Multi-agent operation optimization of district integrated energy system based on reinforcement learning[J]. Proceedings of the CSU-EPSA, 2021, 33(12): 60-68.
[10]	刘翕铭, 于永进, 杨洋. 基于改进麻雀算法的园区综合能源系统优化研究[J]. 智慧电力, 2021, 49(6): 9-16, 82.
	LIU Ximing, YU Yongjin, YANG Yang. Optimization of regional integrated energy system based on improved sparrow algorithm[J]. Smart Power, 2021, 49(6): 9-16, 82.
[11]	邓杰, 姜飞, 王文烨, 等. 考虑动态能效模型的园区综合能源系统梯级优化运行[J]. 电网技术, 2022, 46(3): 1027-1039.
	DENG Jie, JIANG Fei, WANG Wenye, et al. Study on cascade optimization operation of park-level integrated energy system considering dynamic energy efficiency model[J]. Power System Technology, 2022, 46(3): 1027-1039.
[12]	顾伟, 陆帅, 王珺, 等. 多区域综合能源系统热网建模及系统运行优化[J]. 中国电机工程学报, 2017, 37(5): 1305-1316.
	GU Wei, LU Shuai, WANG Jun, et al. Modeling of the heating network for multi-district integrated energy system and its operation optimization[J]. Proceedings of the CSEE, 2017, 37(5): 1305-1316.
[13]	刘一兵, 吴文传, 张伯明, 等. 基于混合整数二阶锥规划的三相有源配电网无功优化[J]. 电力系统自动化, 2014, 38(15): 58-64.
	LIU Yibing, WU Wenchuan, ZHANG Boming, et al. Reactive power optimization for three-phase distribution networks with distributed generators based on mixed integer second-order cone programming[J]. Automation of Electric Power Systems, 2014, 38(15): 58-64.
[14]	周灿煌, 郑杰辉, 荆朝霞, 等. 面向园区微网的综合能源系统多目标优化设计[J]. 电网技术, 2018, 42(6): 1687-1697.
	ZHOU Canhuang, ZHENG Jiehui, JING Zhaoxia, et al. Multi-objective optimal design of integrated energy system for park-level microgrid[J]. Power System Technology, 2018, 42(6): 1687-1697.
[15]	郭力, 许东, 王成山, 等. 冷电联供分布式供能系统能量优化管理[J]. 电力系统自动化, 2009, 33(19): 96-100.
	GUO Li, XU Dong, WANG Chengshan, et al. Energy optimization and management of combined cooling and power distributed energy supply system[J]. Automation of Electric Power Systems, 2009, 33(19): 96-100.
[16]	于波, 吴亮, 卢欣, 等. 区域综合能源系统优化调度方法[J]. 电力建设, 2016, 37(1): 70-76. doi: 10.3969/j.issn.1000-7229.2016.01.011
	YU Bo, WU Liang, LU Xin, et al. Optimal dispatching method of integrated community energy system[J]. Electric Power Construction, 2016, 37(1): 70-76. doi: 10.3969/j.issn.1000-7229.2016.01.011
[17]	高阳, 陈世福, 陆鑫. 强化学习研究综述[J]. 自动化学报, 2004, 30(1): 86-100.
	GAO Yang, CHEN Shifu, LU Xin. Research on reinforcement learning technology: a review[J]. Acta Automatica Sinica, 2004, 30(1): 86-100.
[18]	杨挺, 赵黎媛, 刘亚闯, 等. 基于深度强化学习的综合能源系统动态经济调度[J]. 电力系统自动化, 2021, 45(5): 39-47.
	YANG Ting, ZHAO Liyuan, LIU Yachuang, et al. Dynamic economic dispatch for integrated energy system based on deep reinforcement learning[J]. Automation of Electric Power Systems, 2021, 45(5): 39-47.
[19]	陈明昊, 孙毅, 谢志远. 基于双层深度强化学习的园区综合能源系统多时间尺度优化管理[J/OL]. 电工技术学报, 2022: 1-16 (2022-06-01) [2022-06-24]. https://doi.org/10.19595/j.cnki.1000-6753.tces.211879.
	CHEN Minghao, SUN Yi, XIE Zhiyuan. The multi-time-scale management optimization for park integrated energy system based on the bi-layer deep reinforcement learning[J/OL]. Transactions of China Electrotechnical Society, 2022: 1-16 (2022-06-01) [2022-06-24]. https://doi.org/10.19595/j.cnki.1000-6753.tces.211879.
[20]	冯昌森, 张瑜, 文福拴, 等. 基于深度期望Q网络算法的微电网能量管理策略[J]. 电力系统自动化, 2022, 46(3): 14-22.
	FENG Changsen, ZHANG Yu, WEN Fushuan, et al. Energy management strategy for microgrid based on deep expected Q network algorithm[J]. Automation of Electric Power Systems, 2022, 46(3): 14-22.
[21]	杨楠, 黄禹, 董邦天, 等. 基于多主体博弈的电力-天然气综合能源系统联合规划方法研究[J]. 中国电机工程学报, 2019, 39(22): 6521-6533.
	YANG Nan, HUANG Yu, DONG Bangtian, et al. Research on the joint planning method of electricity-gas integrated energy system based on multi-agent game[J]. Proceedings of the CSEE, 2019, 39(22): 6521-6533.
[22]	顾洁, 白凯峰, 时亚军. 基于多主体主从博弈优化交互机制的区域综合能源系统优化运行[J]. 电网技术, 2019, 43(9): 3119-3134.
	GU Jie, BAI Kaifeng, SHI Yajun. Optimized operation of regional integrated energy system based on multi-agent master-slave game optimization interaction mechanism[J]. Power System Technology, 2019, 43(9): 3119-3134.
[23]	李正茂, 张峰, 梁军, 等. 含电热联合系统的微电网运行优化[J]. 中国电机工程学报, 2015, 35(14): 3569-3576.
	LI Zhengmao, ZHANG Feng, LIANG Jun, et al. Optimization on microgrid with combined heat and power system[J]. Proceedings of the CSEE, 2015, 35(14): 3569-3576.
[24]	李彦君, 裴玮, 肖浩, 等. 基于深度学习的微网需求响应特性封装与配电网优化运行[J]. 电力系统自动化, 2021, 45(10): 157-165.
	LI Yanjun, PEI Wei, XIAO Hao, et al. Deep learning based characteristic packaging of demand response for microgrids and optimal operation of distribution network[J]. Automation of Electric Power Systems, 2021, 45(10): 157-165.
[25]	MNIH V, KAVUKCUOGLU K, SILVER D, et al. Human-level control through deep reinforcement learning[J]. Nature, 2015, 518(7540): 529-533. doi: 10.1038/nature14236 URL