Energy Management Approach for Integrated Electricity-Heat Energy System Based on Deep Q-Learning Network

doi:10.12204/j.issn.1000-7229.2021.03.002

Abstract

Abstract:

Energy management plays an important role in the operation optimization of integrated electricity-heat energy systems. However, the fluctuation of renewable energy power generation and the randomness of energy loads in the system make the energy management problem full of challenges. In order to solve this problem, this paper proposes an optimal energy management approach for integrated energy system considering the uncertainties of renewable energy and load demands. In this paper, the energy management problem of the system is expressed as a Markov decision process with unknown transition probability, and the state space, action space and reward function of the process are defined. In order to solve the Markov decision process, an optimal energy management approach based on deep Q-learning network is proposed. Simulation results show that the proposed method can adaptively respond to the random fluctuations of source and loads and realize the optimal energy management.

Key words: energy management, integrated energy system, reinforcement learning, deep Q-learning network (DQN)

CLC Number:

TM73

WANG Xinying, ZHAO Qi, ZHAO Liyuan, YANG Ting. Energy Management Approach for Integrated Electricity-Heat Energy System Based on Deep Q-Learning Network[J]. ELECTRIC POWER CONSTRUCTION, 2021, 42(3): 10-18.

Figures/Tables 9

Fig.1 The training process of DQN

Fig.2 Schematic diagram of the integrated electricity-heat energy system

Fig.3 Flowchart of the DQN-based IES energy management method

Table 1 Operational parameters of the IES

Table 2 Other parameters of the IES

Fig.4 Electric power management results based on DQN method

Fig.5 Heat power management results based on DQN method

Table 3 Statistics of the daily operation costs under different methods 元

Fig.6 Comparison of system operation costs under different methods

References 27

[1]	WU J, YAN J Y, JIA H J, et al. Integrated energy systems[J]. Applied Energy, 2016,167:155-157. doi: 10.1016/j.apenergy.2016.02.075 URL
[2]	WANG Y L, WANG Y D, HUANG Y J, et al. Optimal scheduling of the regional integrated energy system considering economy and environment[J]. IEEE Transactions on Sustainable Energy, 2019,10(4):1939-1949. doi: 10.1109/TSTE.5165391 URL
[3]	朱承治, 陆帅, 周金辉, 等. 基于电-热分时间尺度平衡的综合能源系统日前经济调度[J]. 电力自动化设备, 2018,38(6):138-143.
	ZHU Chengzhi, LU Shuai, ZHOU Jinhui, et al. Day-ahead economic dispatch of integrated energy system based on electricity and heat balance in different time scales[J]. Electric Power Automation Equipment, 2018,38(6):138-143.
[4]	李正茂, 张峰, 梁军, 等. 含电热联合系统的微电网运行优化[J]. 中国电机工程学报, 2015,35(14):3569-3576. doi: 10.13334/j.0258-8013.pcsee.2015.14.011 URL
	LI Zhengmao, ZHANG Feng, LIANG Jun, et al. Optimization on microgrid with combined heat and power system[J]. Proceedings of the CSEE, 2015,35(14):3569-3576.
[5]	黄宗宏, 曹雨晨, 胡志冰, 等. 基于改进遗传算法的工业园区综合能源系统能量管理方法[J]. 电力建设, 2019,40(8):51-58.
	HUANG Zonghong, CAO Yuchen, HU Zhibing, et al. Energy management of industrial park integrated energy system based on improved genetic algorithm[J]. Electric Power Construction, 2019,40(8):51-58.
[6]	刘洪, 陈星屹, 李吉峰, 等. 基于改进CPSO算法的区域电热综合能源系统经济调度[J]. 电力自动化设备, 2017,37(6):193-200.
	LIU Hong, CHEN Xingyi, LI Jifeng, et al. Economic dispatch based on improved CPSO algorithm for regional power-heat integrated energy system[J]. Electric Power Automation Equipment, 2017,37(6):193-200.
[7]	ZHENG J H, KOU Y N, LI M S, et al. Stochastic optimization of cost-risk for integrated energy system considering wind and solar power correlated[J]. Journal of Modern Power Systems and Clean Energy, 2019,7(6):1472-1483. doi: 10.1007/s40565-019-0519-4 URL
[8]	李文博, 李华东, 张鹏飞, 等. 考虑不确定性的区域能源互联网源-荷-储协调优化[J]. 现代电力, 2019,36(3):11-18.
	LI Wenbo, LI Huadong, ZHANG Pengfei, et al. Generation-load-storage coordinated optimization for regional energy Internet considering uncertainties[J]. Modern Electric Power, 2019,36(3):11-18.
[9]	郭帅, 刘鲁嘉, 杨萌, 等. 计及可控负荷的微网自治调度模型[J]. 电网技术, 2017,41(1):48-56.
	GUO Shuai, LIU Lujia, YANG Meng, et al. Autonomous scheduling model for microgrid considering controllable loads[J]. Power System Technology, 2017,41(1):48-56.
[10]	LUO Z, GU W, WU Z, et al. A robust optimization method for energy management of CCHP microgrid[J]. Journal of Modern Power Systems and Clean Energy, 2018,6(1):132-144. doi: 10.1007/s40565-017-0290-3 URL
[11]	税月, 刘俊勇, 高红均, 等. 考虑风电不确定性的电气能源系统两阶段分布鲁棒协同调度[J]. 电力系统自动化, 2018,42(13):43-50.
	SHUI Yue, LIU Junyong, GAO Hongjun, et al. Two-stage distributed robust cooperative dispatch for integrated electricity and natural gas energy systems considering uncertainty of wind power[J]. Automation of Electric Power Systems, 2018,42(13):43-50.
[12]	YANG T, ZHAO L Y, LI W, et al. Reinforcement learning in sustainable energy and electric systems: A survey[J]. Annual Reviews in Control, 2020,49:145-163. doi: 10.1016/j.arcontrol.2020.03.001 URL
[13]	SUTTON R S, BARTO A G. Reinforcement learning: An introduction[J]. IEEE Transactions on Neural Networks, 1998,9(5):1054.
[14]	BUI V H, HUSSAIN A, KIM H M. Q-learning-based operation strategy for community battery energy storage system (CBESS) in microgrid system[J]. Energies, 2019,12(9):1789. doi: 10.3390/en12091789 URL
[15]	CHENG Z J, ZHAO Q C, WANG F L, et al. Satisfaction based Q-learning for integrated lighting and blind control[J]. Energy and Buildings, 2016,127:43-55. doi: 10.1016/j.enbuild.2016.05.067 URL
[16]	刘洪, 李吉峰, 葛少云, 等. 基于多主体博弈与强化学习的并网型综合能源微网协调调度[J]. 电力系统自动化, 2019,43(1):40-48.
	LIU Hong, LI Jifeng, GE Shaoyun, et al. Coordinated scheduling of grid-connected integrated energy microgrid based on multi-agent game and reinforcement learning[J]. Automation of Electric Power Systems, 2019,43(1):40-48.
[17]	李怡瑾, 唐昊, 吕凯, 等. 源荷不确定冷热电联供微网能量调度的建模与学习优化[J]. 控制理论与应用, 2018,35(1):56-64.
	LI Yijin, TANG Hao, LÜ Kai, et al. Modeling and learning-based optimization of the energy dispatch for a combined cooling, heat and power microgrid system with uncertain sources and loads[J]. Control Theory & Applications, 2018,35(1):56-64.
[18]	RAYATI M, SHEIKHI A, RANJBAR A M. Optimising operational cost of a smart energy hub, the reinforcement learning approach[J]. International Journal of Parallel, Emergent and Distributed Systems, 2015,30(4):325-341. doi: 10.1080/17445760.2014.974600 URL
[19]	LIANG X Y, DU X S, WANG G L, et al. A deep reinforcement learning network for traffic light cycle control[J]. IEEE Transactions on Vehicular Technology, 2019,68(2):1243-1253. doi: 10.1109/TVT.2018.2890726 URL
[20]	LECUN Y, BENGIO Y, HINTON G. Deep learning[J]. Nature, 2015,521(7553):436-444. doi: 10.1038/nature14539 URL pmid: 26017442
[21]	杨挺, 赵黎媛, 王成山. 人工智能在电力系统及综合能源系统中的应用综述[J]. 电力系统自动化, 2019,43(1):2-14.
	YANG Ting, ZHAO Liyuan, WANG Chengshan. Review on application of artificial intelligence in power system and integrated energy system[J]. Automation of Electric Power Systems, 2019,43(1):2-14.
[22]	MNIH V, KAVUKCUOGLU K, SILVER D, et al. Human-level control through deep reinforcement learning[J]. Nature, 2015,518(7540):529-533. doi: 10.1038/nature14236 URL pmid: 25719670
[23]	YU L, XIE W W, XIE D, et al. Deep reinforcement learning for smart home energy management[J]. IEEE Internet of Things Journal, 2020,7(4):2751-2762. doi: 10.1109/JIoT.6488907 URL
[24]	MCKENNA E, THOMSON M. High-resolution stochastic integrated thermal-electrical domestic demand model[J]. Applied Energy, 2016,165:445-461. doi: 10.1016/j.apenergy.2015.12.089 URL
[25]	AYYADI S, BILIL H, MAAROUFI M. Optimal charging of electric vehicles in residential area[J]. Sustainable Energy, Grids and Networks, 2019,19:100240. doi: 10.1016/j.segan.2019.100240 URL
[26]	GAN L K, HUSSAIN A, HOWEY D A, et al. Limitations in energy management systems: A case study for resilient interconnected microgrids[J]. IEEE Transactions on Smart Grid, 2019,10(5):5675-5685. doi: 10.1109/TSG.5165411 URL
[27]	GOODFELLOW I, BENGIO Y, COURVILLE A. Deep learning[M]. MIT Press, 2016.