Research on Two-Layer Configuration and Operation Optimization Based on Proximal Policy Optimization for Electrochemical/Hydrogen Hybrid Energy Storage System

doi:10.12204/j.issn.1000-7229.2022.08.003

Abstract

Abstract:

According to the complementary characteristics of electrochemical energy storage and hydrogen storage, an integrated optimization model for the configuration and operation of a hybrid energy storage system is given, including electrochemical energy storage, hydrogen storage proposed and an intelligent algorithm. The model is based on a two-layer decision optimization problem, in which two different time dimensions of the hybrid energy storage system configuration and operation are solved in upper and lower layers, and the interaction between them is considered. A reinforcement learning proximal policy optimization (PPO) algorithm is used to solve the two-layer optimization model. By comparing the results of applying various traditional algorithms to solve the scenery data of a region in Gansu Province, it is verified that the used algorithm has the highest adaptability and the fastest convergence speed in a complex environment. The results show that the application of this model can reduce the abandoning rate of wind and solar power by 24% and effectively improve the comprehensive benefit of the system, and that hydrogen storage as a capacity-based energy storage configuration is not limited by topographical factors and is suitable for diverse application scenarios, thus providing an application demonstration for the widespread deployment of hydrogen storage, a new form of energy storage, in the whole country.

Key words: wind-solar consumption, energy storage configuration, two-level optimization, hydrogen energy storage, proximal policy optimization (PPO) algorithm

CLC Number:

TM73

YAN Qingyou, SHI Chaofan, QIN Guangyu, XU Chuanbo. Research on Two-Layer Configuration and Operation Optimization Based on Proximal Policy Optimization for Electrochemical/Hydrogen Hybrid Energy Storage System[J]. ELECTRIC POWER CONSTRUCTION, 2022, 43(8): 22-32.

Figures/Tables 19

Fig.1 Hybrid energy storage system with hydrogen energy storage

Table 1 Comparison between electrochemical energy storage and hydrogen energy storage

Table 2 Parameter description value of hydrogen storage field constraint

Table 3 Parameter Description Value Low-level optimization

Fig.2 Flowchart of policy gradient algorithm

Fig.3 Flow chart of PPO algorithm using deep neural network

Fig.4 Forecast curves of wind and solar power and user load

Table A1 Parameter description value of electrochemical energy storage

Table A2 Parameter description value of hydrogen storage system

Table A3 Parameter description value of hydrogen storage system cost

Table A4 Hybrid energy storage element service life

Fig.5 Capacity distribution of electrochemical energy storage

Fig.6 Capacity distribution of hydrogen energy storage under

Fig.7 Hybrid energy storage capacity distribution under assumed condition 1

Fig.8 Hybrid energy storage capacity distribution under assumed condition 2

Table A5 Time of day-tariff

Fig.9 Curve of daily comprehensive benefit changing with storage capacity

Table 4 Comparison of optimization results of energy storage system

Table 5 Comparison of algorithm results

References 27

[1]	孙玉树, 杨敏, 师长立, 等. 储能的应用现状和发展趋势分析[J]. 高电压技术, 2020, 46(1): 80-89.
	SUN Yushu, YANG Min, SHI Changli, et al. Analysis of application status and development trend of energy storage[J]. High Voltage Engineering, 2020, 46(1): 80-89.
[2]	王辉, 梁登香, 韩晓娟. 储能参与泛在电力物联网辅助服务应用综述[J]. 发电技术, 2021, 42(2): 171-179.
	WANG Hui, LIANG Dengxiang, HAN Xiaojuan. Reviews of energy storage participating in auxiliary services under ubiquitous internet of things[J]. Power Generation Technology, 2021, 42(2): 171-179.
[3]	叶季蕾, 李斌, 张宇, 等. 基于全球能源互联网典型特征的储能需求及配置分析[J]. 发电技术, 2021, 42(1): 20-30.
	YE Jilei, LI Bin, ZHANG Yu, et al. Energy storage requirements and configuration analysis based on typical characteristics of global energy internet[J]. Power Generation Technology, 2021, 42(1): 20-30.
[4]	国家电网公司“电网新技术前景研究”项目咨询组. 大规模储能技术在电力系统中的应用前景分析[J]. 电力系统自动化, 2013, 37(1): 3-8.
	Consulting Group of State Grid Corporation of China to Prospects of New Technologies in Power Systems. An analysis of prospects for application of large-scale energy storage technology in power systems[J]. Automation of Electric Power Systems, 2013, 37(1): 3-8.
[5]	胡娟, 杨水丽, 侯朝勇, 等. 规模化储能技术典型示范应用的现状分析与启示[J]. 电网技术, 2015, 39(4): 879-885.
	HU Juan, YANG Shuili, HOU Chaoyong, et al. Present condition analysis on typical demonstration application of large-scale energy storage technology and its enlightenment[J]. Power System Technology, 2015, 39(4): 879-885.
[6]	严晓辉, 徐玉杰, 纪律, 等. 我国大规模储能技术发展预测及分析[J]. 中国电力, 2013, 46(8): 22-29.
	YAN Xiaohui, XU Yujie, JI Lü, et al. Forecasting and analysis on large-scale energy storage technologies in China[J]. Electric Power, 2013, 46(8): 22-29.
[7]	郑乐, 胡伟, 陆秋瑜, 等. 储能系统用于提高风电接入的规划和运行综合优化模型[J]. 中国电机工程学报, 2014, 34(16): 2533-2543.
	ZHENG Le, HU Wei, LU Qiuyu, et al. Research on planning and operation model for energy storage system to optimize wind power integration[J]. Proceedings of the CSEE, 2014, 34(16): 2533-2543.
[8]	许健, 刘念, 于雷, 等. 计及重要负荷的工业光伏微电网储能优化配置[J]. 电力系统保护与控制, 2016, 44(9): 29-37.
	XU Jian, LIU Nian, YU Lei, et al. Optimal allocation of energy storage system of PV microgrid for industries considering important load[J]. Power System Protection and Control, 2016, 44(9): 29-37.
[9]	陶琼, 王德顺, 叶季蕾, 等. 考虑储能配置模式的多数据源融合分布式光伏发电并网接纳分析方法[J]. 高电压技术, 2018, 44(4): 1093-1098.
	TAO Qiong, WANG Deshun, YE Jilei, et al. Capacity analysis of distributed photovoltaic generation integrated into power grid considering energy storage configuration mode based on fusion of multiple data sources[J]. High Voltage Engineering, 2018, 44(4): 1093-1098.
[10]	徐飞, 陈磊, 金和平, 等. 抽水蓄能电站与风电的联合优化运行建模及应用分析[J]. 电力系统自动化, 2013, 37(1): 149-154.
	XU Fei, CHEN Lei, JIN Heping, et al. Modeling and application analysis of optimal joint operation of pumped storage power station and wind power[J]. Automation of Electric Power Systems, 2013, 37(1): 149-154.
[11]	戴远航, 陈磊, 闵勇, 等. 风电场与含储热的热电联产联合运行的优化调度[J]. 中国电机工程学报, 2017, 37(12): 3470-3479, 3675.
	DAI Yuanhang, CHEN Lei, MIN Yong, et al. Optimal dispatch for joint operation of wind farm and combined heat and power plant with thermal energy storage[J]. Proceedings of the CSEE, 2017, 37(12): 3470-3479, 3675.
[12]	夏沛, 邓长虹, 龙志君, 等. 含抽水蓄能机组的风电消纳鲁棒机组组合[J]. 电力系统自动化, 2018, 42(19): 41-49.
	XIA Pei, DENG Changhong, LONG Zhijun, et al. Robust unit commitment with pumped storage units for wind power accommodation[J]. Automation of Electric Power Systems, 2018, 42(19): 41-49.
[13]	张文建, 崔青汝, 李志强, 等. 电化学储能在发电侧的应用[J]. 储能科学与技术, 2020, 9(1): 287-295.
	ZHANG Wenjian, CUI Qingru, LI Zhiqiang, et al. Application of electrochemical energy storage in power generation[J]. Energy Storage Science and Technology, 2020, 9(1): 287-295.
[14]	蔡霁霖, 徐青山, 袁晓冬, 等. 基于风电消纳时序场景的电池储能系统配置策略[J]. 高电压技术, 2019, 45(3): 993-1001.
	CAI Jilin, XU Qingshan, YUAN Xiaodong, et al. Configuration strategy of large-scale battery storage system orienting wind power consumption based on temporal scenarios[J]. High Voltage Engineering, 2019, 45(3): 993-1001.
[15]	刘闯, 孙同, 蔡国伟, 等. 基于同步机三阶模型的电池储能电站主动支撑控制及其一次调频贡献力分析[J]. 中国电机工程学报, 2020, 40(15): 4854-4866.
	LIU Chuang, SUN Tong, CAI Guowei, et al. Third-order synchronous machine model based active support control of battery storage power plant and its contribution analysis for primary frequency response[J]. Proceedings of the CSEE, 2020, 40(15): 4854-4866.
[16]	周荣昱. 基于小生境多目标粒子群优化算法的配电网混合储能配置研究[J]. 电气开关, 2021, 59(6): 35-41.
	ZHOU Rongyu. Optimal allocation of hybrid energy storage in distribution networks based on niche multi-objective particle swarm optimization algorithm[J]. Electric Switchgear, 2021, 59(6): 35-41.
[17]	王玥娇, 张兴友, 郭俊山. 储能技术在高比例可再生能源电力系统中的应用[J]. 山东电力技术, 2021, 48(7): 19-25.
	WANG Yuejiao, ZHANG Xingyou, GUO Junshan. Review on the application of energy storage technology in power system with high proportion of renewable energy[J]. Shandong Electric Power, 2021, 48(7): 19-25.
[18]	何颖源, 陈永翀, 刘勇, 等. 储能的度电成本和里程成本分析[J]. 电工电能新技术, 2019, 38(9): 1-10.
	HE Yingyuan, CHEN Yongchong, LIU Yong, et al. Analysis of cost per kilowatt-hour and cost per mileage for energy storage technologies[J]. Advanced Technology of Electrical Engineering and Energy, 2019, 38(9): 1-10.
[19]	彭春华, 余愿, 孙惠娟. 基于源网荷协同优化的配电网光储联合系统规划[J]. 电网技术, 2019, 43(11): 3944-3951.
	PENG Chunhua, YU Yuan, SUN Huijuan. Planning of combined PV-ESS system for distribution network based on source-network-load collaborative optimization[J]. Power System Technology, 2019, 43(11): 3944-3951.
[20]	刘冠群, 袁越, 王敏, 等. 考虑经济成本的光伏电站储能容量配置[J]. 可再生能源, 2014, 32(1): 1-5.
	LIU Guanqun, YUAN Yue, WANG Min, et al. Energy storage capacity determining of PV plant considering economic cost[J]. Renewable Energy Resources, 2014, 32(1): 1-5.
[21]	蒋伟, 陈照光, 颜浩. 基于改进人工蜂群算法的家庭储能容量优化配置[J/OL]. 电测与仪表: 1-7 (2021-12-20)[2022-04-02]. http://kns.cnki.net/kcms/detail/23.1202.TH.20211218.0938.002.html.
	JANG Wei, CHEN Zhaoguang, YAN Hao. Optimal allocation of household energy storage capacity based on improved artificial bee colony algorithm[J/OL]. Electrical Measurement & Instrumentation: 1-7(2021-12-20)[2022-04-02]. http://kns.cnki.net/kcms/detail/23.1202.TH.20211218.0938.002.html.
[22]	王佳蕊, 李德鑫, 张家郡, 等. 微电网储能系统控制对电能质量的保障[J]. 哈尔滨理工大学学报, 2021, 26(6): 87-93.
	WANG Jiarui, LI Dexin, ZHANG Jiajun, et al. The control of microgrid energy storage system guarantees the power quality[J]. Journal of Harbin University of Science and Technology, 2021, 26(6): 87-93.
[23]	杨晓雷, 丁磊明, 李逸鸿, 等. 计及风电不确定性的混合储能系统优化配置研究[J]. 电力需求侧管理, 2021, 23(6): 69-74.
	YANG Xiaolei, DING Leiming, LI Yihong, et al. Study on optimal allocation of hybrid energy storage system considering wind power uncertainty[J]. Power Demand Side Management, 2021, 23(6): 69-74.
[24]	许传博, 赵云灏, 王晓晨, 等. 碳中和愿景下考虑电氢耦合的风光场站氢储能优化配置[J]. 电力建设, 2022, 43(1): 10-18.
	XU Chuanbo, ZHAO Yunhao, WANG Xiaochen, et al. Optimal configuration of hydrogen energy storage for wind and solar power stations considering electricity-hydrogen coupling under carbon neutrality vision[J]. Electric Power Construction, 2022, 43(1): 10-18.
[25]	李红, 朱立位, 郭朝辉. 基于深度强化学习的混合储能系统最优控制[J]. 电器与能效管理技术, 2021(11): 42-49.
	LI Hong, ZHU Liwei, GUO Zhaohui. Optimal control of hybrid energy storage system based on deep reinforcement learning[J]. Electrical & Energy Management Technology, 2021(11): 42-49.
[26]	聂欢欢, 张家琦, 陈颖, 等. 基于双层强化学习方法的多能园区实时经济调度[J]. 电网技术, 2021, 45(4): 1330-1336.
	NIE Huanhuan, ZHANG Jiaqi, CHEN Ying, et al. Real-time economic dispatch of community integrated energy system based on a double-layer reinforcement learning method[J]. Power System Technology, 2021, 45(4): 1330-1336.
[27]	XU C B, KE Y M, LI Y B, et al. Data-driven configuration optimization of an off-grid wind/PV/hydrogen system based on modified NSGA-II and CRITIC-TOPSIS[J]. Energy Conversion and Management, 2020, 215: 112892. doi: 10.1016/j.enconman.2020.112892 URL