Demand-Side Energy Management Method for Building Clusters Applying Reinforcement Learning

doi:10.12204/j.issn.1000-7229.2021.05.003

Abstract

Abstract:

In view of the current need to further explore the feasibility of the application of reinforcement learning in demand-side energy management and user-side demand response, this paper proposes a demand-side energy management method for building clusters on the basis of reinforcement learning. Firstly, the building clusters are used as the terminal energy load carrier to construct the demand-side energy management framework of the building clusters. Secondly, according to the virtual energy storage characteristics of the intelligent buildings, a novel intelligent building thermal resistance-capacity (R-C) thermal balance model and user flexibility load model are constructed, and a demand-side energy management model based on reinforcement learning is constructed by combining Q-learning algorithm. Finally, the effectiveness and practicability of the proposed theoretical method are verified by comparing the results of demand-side energy management and the performance of the algorithm through actual simulation cases.

Key words: building cluster, R-C thermal balance model, reinforcement learning, demand-side energy management

CLC Number:

TM734

AN Jiakun, HE Chunguang, LIU Hong, LING Yunpeng, QI Xiaoguang, LI Weiyu, SUN Pengfei, TAN Xiaolin. Demand-Side Energy Management Method for Building Clusters Applying Reinforcement Learning[J]. ELECTRIC POWER CONSTRUCTION, 2021, 42(5): 16-26.

Figures/Tables 16

Fig.1 Overall framework of demand-side energy management of building clusters

Fig.2 Building R-C network model

Fig.3 Structural model of HVAC system

Fig.4 Structural model of water heater

Table 1 Classification of indoor thermal comfort level

Fig.5 Demand-side energy management process based on reinforcement learning

Fig.6 Architecture of the test system

Table 2 Building cluster information

Table 3 Building parameters

Table 4 HVAC system parameters

Fig.7 Typical load curves of different building clusters

Fig.8 Regional time-sharing electricity price

Fig.9 Results of building cluster demand-side energy management

Table 5 Comparison of daily energy cost of different building clusters元

Table 6 Comparison of energy consumption of HVAC system with different control methods kW·h

Fig.10 Comparison of iterative process between different algorithms

References 36

[1]	IEA Publications. Repowering markets[EB/OL]. [2018-03-05]. http://www.iea.org/publications/freepublications/publication/REPOWERINGMARKETS.PDF.
[2]	孙毅, 刘迪, 李彬, 等. 深度强化学习在需求响应中的应用[J]. 电力系统自动化, 2019,43(5):183-191.
	SUN Yi, LIU Di, LI Bin, et al. Application of deep reinforcement learning in demand response[J]. Automation of Electric Power Systems, 2019,43(5):183-191.
[3]	徐彪, 尹项根, 汪旸, 等. 基于模糊时间Petri网的电网故障诊断方法[J]. 电力系统自动化, 2018,42(2):70-76.
	XU Biao, YIN Xianggen, WANG Yang, et al. Fault diagnosis method of power system based on fuzzy time petri net[J]. Automation of Electric Power Systems, 2018,42(2):70-76.
[4]	孔祥玉, 郑锋, 鄂志君, 等. 基于深度信念网络的短期负荷预测方法[J]. 电力系统自动化, 2018,42(5):133-139.
	KONG Xiangyu, ZHENG Feng, E Zhijun, et al. Short-term load forecasting based on deep belief network[J]. Automation of Electric Power Systems, 2018,42(5):133-139.
[5]	徐龙博, 王伟, 张滔, 等. 基于神经网络平均影响值的超短期风电功率预测[J]. 电力系统自动化, 2017,41(21):40-45.
	XU Longbo, WANG Wei, ZHANG Tao, et al. Ultra-short-term wind power prediction based on neural network and mean impact value[J]. Automation of Electric Power Systems, 2017,41(21):40-45.
[6]	HAN M J, MAY R, ZHANG X X, et al. A review of reinforcement learning methodologies for controlling occupant comfort in buildings[J]. Sustainable Cities and Society, 2019,51:101748. doi: 10.1016/j.scs.2019.101748 URL
[7]	NGARAMBE J, YUN G Y, SANTAMOURIS M. The use of artificial intelligence (AI) methods in the prediction of thermal comfort in buildings: Energy implications of AI-based thermal comfort controls[J]. Energy and Buildings, 2020,211:109807. doi: 10.1016/j.enbuild.2020.109807 URL
[8]	MEHMOOD M U, CHUN D Y, ZEESHAN, et al. A review of the applications of artificial intelligence and big data to buildings for energy-efficiency and a comfortable indoor living environment[J]. Energy and Buildings, 2019,202:109383. doi: 10.1016/j.enbuild.2019.109383 URL
[9]	程乐峰, 余涛, 张孝顺, 等. 信息-物理-社会融合的智慧能源调度机器人及其知识自动化: 框架、技术与挑战[J]. 中国电机工程学报, 2018,38(1):25-40,340.
	CHENG Lefeng, YU Tao, ZHANG Xiaoshun, et al. Cyber-physical-social systems based smart energy robotic dispatcher and its knowledge automation: Framework, techniques and challenges[J]. Proceedings of the CSEE, 2018,38(1):25-40,340.
[10]	刘洪, 李吉峰, 葛少云, 等. 基于多主体博弈与强化学习的并网型综合能源微网协调调度[J]. 电力系统自动化, 2019,43(1):40-48.
	LIU Hong, LI Jifeng, GE Shaoyun, et al. Coordinated scheduling of grid-connected integrated energy microgrid based on multi-agent game and reinforcement learning[J]. Automation of Electric Power Systems, 2019,43(1):40-48.
[11]	LI F D, WU M, HE Y, et al. Optimal control in microgrid using multi-agent reinforcement learning[J]. ISA Transactions, 2012,51(6):743-751. doi: 10.1016/j.isatra.2012.06.010 URL
[12]	LI D, JAYAWEERA S K. Machine-learning aided optimal customer decisions for an interactive smart grid[J]. IEEE Systems Journal, 2015,9(4):1529-1540. doi: 10.1109/JSYST.2014.2334637 URL
[13]	KOFINAS P, DOUNIS A I, VOUROS G A. Fuzzy Q-Learning for multi-agent decentralized energy management in microgrids[J]. Applied Energy, 2018,219:53-67. doi: 10.1016/j.apenergy.2018.03.017 URL
[14]	PÉREZ-LOMBARD L, ORTIZ J, POUT C. A review on buildings energy consumption information[J]. Energy and Buildings, 2008,40(3):394-398. doi: 10.1016/j.enbuild.2007.03.007 URL
[15]	Science and Technology Development Promotion Center of the Ministry of Housing and Urban-Rural Development . China building energy efficiency development report (2016): Building energy efficiency operation management[M]. Beijing: China Building Industry Press, 2016.
[16]	Buildings energy data book [EB/OL]. http://buildingsdatabook.eren.doe.gov/ChapterIntro1.aspx.
[17]	刘向军, 周行洁, 刘迪, 等. 面向新能源消纳的热水器集群精细化均衡控制[J]. 电力自动化设备, 2020,40(1):52-58.
	LIU Xiangjun, ZHOU Xingjie, LIU Di, et al. Refined equilibrium control of electric water heater clusters for accommodation of renewable energy sources[J]. Electric Power Automation Equipment, 2020,40(1):52-58.
[18]	栾文鹏, 赵磊, 王兵, 等. 智能电表数据分析元及示例[J]. 南方电网技术, 2016,10(1):1-5.
	LUAN Wenpeng, ZHAO Lei, WANG Bing, et al. Smart meter data analytics and application examples[J]. Southern Power System Technology, 2016,10(1):1-5.
[19]	JIN X L, MU Y F, JIA H J, et al. Dynamic economic dispatch of a hybrid energy microgrid considering building based virtual energy storage system[J]. Applied Energy, 2017,194:386-398. doi: 10.1016/j.apenergy.2016.07.080 URL
[20]	HANIF S, MASSIER T, GOOI H B, et al. Cost optimal integration of flexible buildings in congested distribution grids[J]. IEEE Transactions on Power Systems, 2017,32(3):2254-2266. doi: 10.1109/TPWRS.2016.2605921 URL
[21]	JIANG T, LI Z N, JIN X L, et al. Flexible operation of active distribution network using integrated smart buildings with heating, ventilation and air-conditioning systems[J]. Applied Energy, 2018,226:181-196. doi: 10.1016/j.apenergy.2018.05.091 URL
[22]	WANG D, HU Q E, JIA H J, et al. Integrated demand response in district electricity-heating network considering double auction retail energy market based on demand-side energy stations[J]. Applied Energy, 2019,248:656-678. doi: 10.1016/j.apenergy.2019.04.050 URL
[23]	刘洪, 王亦然, 李积逊, 等. 考虑建筑热平衡与柔性舒适度的乡村微能源网电热联合调度[J]. 电力系统自动化, 2019,43(9):50-58.
	LIU Hong, WANG Yiran, LI Jixun, et al. Coordinated heat and power dispatch of micro-energy network of countryside considering heat balance model of building and flexible indoor comfort constraint[J]. Automation of Electric Power Systems, 2019,43(9):50-58.
[24]	葛少云, 李吉峰, 刘洪, 等. 考虑物理特征与行为因素的家庭用能特性建模[J]. 电力自动化设备, 2019,39(3):36-44.
	GE Shaoyun, LI Jifeng, LIU Hong, et al. Modelling of household energy consumption characteristics considering physical features and behavior factors[J]. Electric Power Automation Equipment, 2019,39(3):36-44.
[25]	MURATORI M, ROBERTS M C, SIOSHANSI R, et al. A highly resolved modeling technique to simulate residential power demand[J]. Applied Energy, 2013,107:465-473. doi: 10.1016/j.apenergy.2013.02.057 URL
[26]	陈厚合, 李泽宁, 靳小龙, 等. 集成智能楼宇的主动配电网建模及优化方法[J]. 中国电机工程学报, 2018,38(22):6550-6563.
	CHEN Houhe, LI Zening, JIN Xiaolong, et al. Modeling and optimization of active distribution network with integrated smart buildings[J]. Proceedings of the CSEE, 2018,38(22):6550-6563.
[27]	ZENG L K, SUN Y, YE Q Z, et al. A centralized demand response control strategy for domestic electric water heater group based on appliance cloud platform[J]. IEEJ Transactions on Electrical and Electronic Engineering, 2017,12:S16-S22. doi: 10.1002/tee.2017.12.issue-S2 URL
[28]	FARZAN F, JAFARI M A, GONG J, et al. A multi-scale adaptive model of residential energy demand[J]. Applied Energy, 2015,150:258-273. doi: 10.1016/j.apenergy.2015.04.008 URL
[29]	KORKAS C D, BALDI S, MICHAILIDIS I, et al. Occupancy-based demand response and thermal comfort optimization in microgrids with renewable energy sources and energy storage[J]. Applied Energy, 2016,163:93-104. doi: 10.1016/j.apenergy.2015.10.140 URL
[30]	LI C J, YU X H, YU W W, et al. Efficient computation for sparse load shifting in demand side management[J]. IEEE Transactions on Smart Grid, 2017,8(1):250-261. doi: 10.1109/TSG.2016.2521377 URL
[31]	余涛, 张水平. 在策略SARSA算法在互联电网CPS最优控制中的应用[J]. 电力系统保护与控制, 2013,41(1):211-216. doi: 10.7667/j.issn.1674-3415.2013.01.032 URL
	YU Tao, ZHANG Shuiping. Optimal CPS control for interconnected power systems based on SARSA on-policy learning algorithm[J]. Power System Protection and Control, 2013,41(1):211-216. doi: 10.7667/j.issn.1674-3415.2013.01.032 URL
[32]	刘国静, 韩学山, 王尚, 等. 基于强化学习方法的风储合作决策[J]. 电网技术, 2016,40(9):2729-2736.
	LIU Guojing, HAN Xueshan, WANG Shang, et al. Optimal decision-making in the cooperation of wind power and energy storage based on reinforcement learning algorithm[J]. Power System Technology, 2016,40(9):2729-2736.
[33]	瞿凯平, 张孝顺, 余涛, 等. 基于知识迁移Q学习算法的多能源系统联合优化调度[J]. 电力系统自动化, 2017,41(15):18-25.
	QU Kaiping, ZHANG Xiaoshun, YU Tao, et al. Knowledge transfer based Q-learning algorithm for optimal dispatch of multi-energy system[J]. Automation of Electric Power Systems, 2017,41(15):18-25.
[34]	RAZMARA M, BHARATI G R, SHAHBAKHTI M, et al. Bilevel optimization framework for smart building-to-grid systems[J]. IEEE Transactions on Smart Grid, 2018,9(2):582-593. doi: 10.1109/TSG.2016.2557334 URL
[35]	RAZMARA M, MAASOUMY M, SHAHBAKHTI M, et al. Optimal exergy control of building HVAC system[J]. Applied Energy, 2015,156:555-565. doi: 10.1016/j.apenergy.2015.07.051 URL
[36]	王诚良, 刘洪, 宫建锋, 等. 面向风电消纳的综合能源系统多类型储能联合调度[J]. 电力建设, 2018,39(4):35-44.
	WANG Chengliang, LIU Hong, GONG Jianfeng, et al. Joint scheduling of different energy storage for improving wind power accommodation ability in integrated community energy system[J]. Electric Power Construction, 2018,39(4):35-44.