基于条件风险强化学习的梯级水光蓄联合优化调度

陈实; 唐国登; 刘艺洪; 许刘超; 朱钰杰; 周毅; 李华强; 臧天磊

PDF(1216 KB)

电力建设 ›› 0

基于条件风险强化学习的梯级水光蓄联合优化调度

陈实, 唐国登, 刘艺洪, 许刘超, 朱钰杰, 周毅, 李华强, 臧天磊

作者信息 +

Condition-Based Risk Reinforcement Learning for Joint Optimal Scheduling of Cascade Hydropower and Solar-Powered Reservoirs

CHEN Shi, TANG Guodeng, LIU Yihong, XU Liuchao, ZHU Yujie, ZHOU Yi, LI Huaqiang, ZANG Tianlei

Author information +

文章历史 +

摘要

【目的】随着“双碳”目标的推进,水电开发商积极推进所在流域水力发电抽蓄化改造的同时,加大区域光伏等新能源的建设力度,逐渐形成高比例新能源与大规模抽水蓄能并存的局面。但在新能源出力与区间来水的双重不确定性影响下,系统运行配合更加复杂,尤其是考虑区间来水不确定后含蓄梯级水电调度方法鲜有研究。【方法】提出一种基于条件风险强化学习的梯级水光蓄联合优化调度方法,通过Informer深度神经网络预测流域区间来水,进一步将区间来水不确定性转化为灵活性供给指标;再结合风险理论,用条件风险价值（conditional value at risk, CVaR）量化灵活性缺额;最后提出考虑条件风险价值的近端策略优化（risk-managing twin delayed deep deterministic policy gradient, RM-PPO）强化学习算法进行求解以获得优化调度策略。【结果】以中国某实际梯级水光蓄基地为例进行验证,所提预测方法相较仅考虑时序的传统方法,均方误差（mean-square error,MSE）降低18.9%,平均绝对误差(mean absolute error, MAE)降低58.8%,尖峰事件捕获率为78.5%;所提RM-PPO调度算法实现了各站对系统灵活性的调控,抽蓄电站与光伏联动消纳,传统水电站与光伏协同源荷匹配。【结论】所提RM-PPO调度策略能够降低成本、控制风险,并保持系统运行灵活性,促进新能源消纳的同时提高水能利用率。

Abstract

[Objective] With the advancement of the dual-carbon strategy, hydropower developers are actively promoting pumped-storage retrofitting of hydroelectric facilities while accelerating regional construction of photovoltaic (PV) and other renewable energy sources, gradually forming a scenario with high-penetration renewables and large-scale pumped storage. However, under the dual uncertainties of renewable energy output and interval water inflow, system operational coordination becomes more complex, particularly as scheduling methods for cascade hydropower systems considering interval inflow uncertainty have rarely been studied. [Methods] This paper proposes a conditional risk-aware reinforcement learning-based optimal scheduling method for cascade hydro-PV-pumped storage systems. First, the "Informer" deep neural network is employed to predict basin interval water inflow, transforming inflow uncertainty into flexibility supply indicators. Subsequently, risk theory is integrated to quantify flexibility deficits using Conditional Value at Risk (CVaR). Finally, an improved Risk-Managing Proximal Policy Optimization (RM-PPO) reinforcement learning algorithm is developed to derive optimized scheduling strategies. [Results] Validation using an actual cascade hydro-PV-pumped storage base in China demonstrates that the proposed forecasting method reduces MSE by 18.9% and MAE by 58.8% compared to traditional time-series approaches, with a 78.5% capture rate for peak events. The RM-PPO scheduling algorithm achieves flexible system regulation through pumped-storage plants coordinating with PV for surplus energy absorption, while conventional hydropower stations synergize with PV for source-load dynamic matching. [Conclusions] The RM-PPO-based scheduling strategy effectively reduces costs, controls operational risks while maintaining system flexibility, promotes renewable energy accommodation, and enhances hydropower utilization efficiency.

导出引用

陈实, 唐国登, 刘艺洪, 许刘超, 朱钰杰, 周毅, 李华强, 臧天磊. 基于条件风险强化学习的梯级水光蓄联合优化调度[J]. 电力建设. 0

CHEN Shi, TANG Guodeng, LIU Yihong, XU Liuchao, ZHU Yujie, ZHOU Yi, LI Huaqiang, ZANG Tianlei. Condition-Based Risk Reinforcement Learning for Joint Optimal Scheduling of Cascade Hydropower and Solar-Powered Reservoirs[J]. Electric Power Construction. 0

中图分类号： TM732

参考文献

[1] 齐郑, 徐希茜, 熊巍, 等. 基于变时段设计改进多目标差分进化算法的风/光/火/储日前优化调度[J]. 电力系统保护与控制, 2024, 52(16): 62-71.
QI Zheng, XU Xixi, XIONG Wei, et al.An improved multi objective differential evolution algorithm based on variable time period design for a wind/photovoltaic/thermal/storage system day ahead optimization scheduling[J]. Power System Protection and Control, 2024, 52(16): 62-71.
[2] 孙志媛, 彭博雅, 孙艳. 考虑多能互补的电力电量平衡优化调度策略[J]. 中国电力, 2024, 57(1): 115-122.
SUN Zhiyuan, PENG Boya, SUN Yan.Optimal dispatch strategy of power and electricity balance based on multi-energy complementation[J]. Electric Power, 2024, 57(1): 115-122.
[3] 任大伟, 侯金鸣, 肖晋宇, 等. 支撑双碳目标的新型储能发展潜力及路径研究[J]. 中国电力, 2023, 56(8): 17-25.
REN Dawei, HOU Jinming, XIAO Jinyu, et al.Research on development potential and path of new energy storage supporting carbon peak and carbon neutrality[J]. Electric Power, 2023, 56(8): 17-25.
[4] 吴智泉, 贾纯超, 陈磊, 等. 新型电力系统中储能创新方向研究[J]. 太阳能学报, 2021, 42(10): 444-451.
WU Zhiquan, JIA Chunchao, CHEN Lei, et al.Research on innovative direction of energy storage in new power system construction[J]. Acta Energiae Solaris Sinica, 2021, 42(10): 444-451.
[5] 彭煜民, 王雪林, 刘德旭, 等. 考虑混蓄与纯蓄配置方式的水风光蓄互补系统调度运行对比研究[J]. 电力系统保护与控制, 2024, 52(10): 179-187.
PENG Yumin, WANG Xuelin, LIU Dexu, et al.Comparative research on the dispatching operation of a hydro-wind-solar-storage complementary system considering configurations with hybrid and pure pumped-storage hydropower plants[J]. Power System Protection and Control, 2024, 52(10): 179-187.
[6] 申建建, 王月, 程春田, 等. 水风光多能互补发电调度问题研究现状及展望[J]. 中国电机工程学报, 2022, 42(11): 3871-3885.
SHEN Jianjian, WANG Yue, CHENG Chuntian, et al.Research status and prospect of generation scheduling for hydropower-wind-solar energy complementary system[J]. Proceedings of the CSEE, 2022, 42(11): 3871-3885.
[7] 赵明哲. 雅砻江流域清洁能源基地多尺度风光水互补调度与灵活性研究[D]. 西安: 西安理工大学, 2022.
ZHAO Mingzhe.Study on multi-scale complementary scheduling and flexibility of wind, solar and water for clean energy base in yalong river basin[D]. Xi’an: Xi’an University of Technology, 2022.
[8] TAN Q F, NIE Z, WEN X, et al.Complementary scheduling rules for hybrid pumped storage hydropower-photovoltaic power system reconstructing from conventional cascade hydropower stations[J]. Applied Energy, 2024, 355: 122250.
[9] 程孟增, 唐一金, 商文颖, 等. 风-光-火-抽蓄联合系统中抽水蓄能电站最佳容量配置[J]. 电力建设, 2021, 42(11): 72-81.
CHENG Mengzeng, TANG Yijin, SHANG Wenying, et al.Optimal capacity configuration of pumped-storage power station in wind-PV-fire-pump storage system[J]. Electric Power Construction, 2021, 42(11): 72-81.
[10] 陈雨潇, 龚锦霞, 赵文彬, 等. 面向新型电力系统的变速抽蓄电站调频策略研究[J]. 智慧电力, 2023, 51(3): 104-110.
CHEN Yuxiao, GONG Jinxia, ZHAO Wenbin, et al.Frequency modulation strategy for variable-speed pumped-storage power station for new power system[J]. Smart Power, 2023, 51(3): 104-110.
[11] 周云海, 张智颖, 徐飞, 等. 含多座抽水蓄能电站的省级电网日前计划分层递进优化算法[J]. 中国电力, 2023, 56(5): 41-50.
ZHOU Yunhai, ZHANG Zhiying, XU Fei, et al.Hierarchical progressive optimization algorithm for day-ahead planning of provincial power grid with multiple pumped storage power stations[J]. Electric Power, 2023, 56(5): 41-50.
[12] 刘德旭, 杨迎, 黄宏旭, 等. 新型电力系统大规模抽水蓄能调度运行与控制综述及展望[J]. 中国电机工程学报, 2025, 45(1): 80-98.
LIU Dexu, YANG Ying, HUANG Hongxu, et al.An overview and outlook of the operation and control of large-scale pumped hydro storages in modern power systems[J]. Proceedings of the CSEE, 2025, 45(1): 80-98.
[13] 罗彬, 陈永灿, 刘昭伟, 等. 梯级水光互补系统最大化可消纳电量期望短期优化调度模型[J]. 电力系统自动化, 2023, 47(10): 66-75.
LUO Bin, CHEN Yongcan, LIU Zhaowei, et al.Short-term optimal dispatch model for maximizing expectation of accommodation power of cascade hydro-photovoltaic complementary system[J]. Automation of Electric Power Systems, 2023, 47(10): 66-75.
[14] 李咸善, 杨拯, 李飞, 等. 基于梯级水电调节的风光水联盟与区域电网联合运行优化调度策略[J]. 中国电机工程学报, 2023, 43(6): 2234-2248.
LI Xianshan, YANG Zheng, LI Fei, et al.Optimization scheduling strategy for joint operation of wind-solar-water power alliance and regional power grid based on cascade hydropower regulation[J]. Proceedings of the CSEE, 2023, 43(6): 2234-2248.
[15] 曹敏健, 胡泽春, 孟颖, 等. 含抽蓄电站与新能源发电的柔性直流系统日前优化调度方法[J]. 电力系统自动化, 2021, 45(15): 36-44.
CAO Minjian, HU Zechun, MENG Ying, et al.Day-ahead optimal dispatch method for flexible DC system with pumped storage hydropower plant and new energy power generation[J]. Automation of Electric Power Systems, 2021, 45(15): 36-44.
[16] 夏依莎, 刘俊勇, 刘继春, 等. 基于电量共享的梯级水光蓄联合发电系统优化调度策略[J]. 电力自动化设备, 2021, 41(9): 118-125.
XIA Yisha, LIU Junyong, LIU Jichun, et al.Optimal scheduling strategy of cascaded hydro-photovoltaic-pumped storage hybrid generation system based on electric energy sharing[J]. Electric Power Automation Equipment, 2021, 41(9): 118-125.
[17] 葛晓琳, 钟俊玲, 符杨. 考虑来水多区间不确定集的跨年梯级水电鲁棒优化调度[J]. 中国电机工程学报, 2018, 38(17): 5097-5106, 5306.
GE Xiaolin, ZHONG Junling, FU Yang.Robust biennial scheduling of cascade hydropower system considering with the multi-band uncertainty set of the inflow[J]. Proceedings of the CSEE, 2018, 38(17): 5097-5106, 5306.
[18] 赵志鹏, 于志辉, 程春田, 等. 水风光综合基地多风险量化及长期多目标协调优化调度方法[J]. 电力系统自动化, 2024, 48(22): 118-130.
ZHAO Zhipeng, YU Zhihui, CHENG Chuntian, et al.Multi-risk quantification and long-term multi-objective coordinative optimal dispatch method for hydro-wind-solar integrated energy base[J]. Automation of Electric Power Systems, 2024, 48(22): 118-130.
[19] 孙惠娟, 巩磊, 彭春华, 等. 考虑风光水多重不确定性置信风险的多目标动态分解优化调度[J]. 电网技术, 2022, 46(9): 3416-3428.
SUN Huijuan, GONG Lei, PENG Chunhua, et al.Multi-objective dynamic decomposition optimal dispatch considering wind-photovoltaic-hydro multiple uncertainty confidence risk[J]. Power System Technology, 2022, 46(9): 3416-3428.
[20] 马宇航, 黄媛, 刘俊勇, 等. 考虑日内来水不确定和电网断面约束的梯级水电日前调度[J]. 电力建设, 2020, 41(9): 39-49.
MA Yuhang, HUANG Yuan, LIU Junyong, et al.Day-ahead dispatching of cascade hydropower stations considering daily interval flow uncertainty and the section constraint of power grid[J]. Electric Power Construction, 2020, 41(9): 39-49.
[21] JOHNATHON C, AGALGAONKAR A P, PLANIDEN C, et al.A proposed hedge-based energy market model to manage renewable intermittency[J]. Renewable Energy, 2023, 207: 376-384.
[22] 王辉, 王军杰, 王腾飞, 等. 出力不确定下基于CVaR的风电供应链优化决策[J]. 电力科学与技术学报, 2023, 38(4): 134-142.
WANG Hui, WANG Junjie, WANG Tengfei, et al.Optimal decision of wind power supply chain based on CVaR under uncertainty output[J]. Journal of Electric Power Science and Technology, 2023, 38(4): 134-142.
[23] 周新颖. 数据驱动的电力批发市场竞价策略研究[D]. 北京: 华北电力大学, 2023.
ZHOU Xinying.Research on bidding strategy of data-driven power wholesale market[D]. Beijing: North China Electric Power University, 2023.
[24] 高万胜, 蔺红. 考虑配电网灵活性不足风险的分布鲁棒低碳优化调度[J]. 电力系统保护与控制, 2024, 52(16): 49-61.
GAO Wansheng, LIN Hong.Distributionally robust low-carbon optimal scheduling considering flexibility deficiency risk in a distribution network[J]. Power System Protection and Control, 2024, 52(16): 49-61.
[25] 曾小青, 唐超雯. 分时电价环境下计及新能源消纳的虚拟电厂优化调度研究[J]. 电力科学与技术学报, 2023, 38(3): 24-34.
ZENG Xiaoqing, TANG Chaowen.Research on optimization of virtual power plants dispatch by considering the consumption of new energy under time-of-use electricity price environment[J]. Journal of Electric Power Science and Technology, 2023, 38(3): 24-34.
[26] 周耀辉, 司渭滨, 代立君, 等. 基于随机规划的新型电力系统优化调度方法[J/OL]. 自动化技术与应用, 2024: 1-6. (2024-11-26). https://kns.cnki.net/KCMS/detail/detail.aspx?filename=ZDHJ20241122006&dbname=CJFD&dbcode=CJFQ.
ZHOU Yaohui, SI Weibin, DAI Lijun, et al. A new power system optimal dispatching method based on stochastic programming[J/OL]. China Industrial Economics, 2024: 1-6. (2024-11-26). https://kns.cnki.net/KCMS/detail/detail.aspx?filename=ZDHJ20241122006&dbname=CJFD&dbcode=CJFQ.
[27] GAO Y, TAHIR M, SIANO P, et al.Optimization of renewable energy-based integrated energy systems: a three-stage stochastic robust model[J]. Applied Energy, 2025, 377: 124635.
[28] YUAN W L, WANG X Q, SU C G, et al.Stochastic optimization model for the short-term joint operation of photovoltaic power and hydropower plants based on chance-constrained programming[J]. Energy, 2021, 222: 119996.
[29] 闻昕, 刘凡骞, 谭乔凤, 等. 复杂水利-电力约束下梯级水电站短期多层级多目标优化调度研究[J]. 水利学报, 2023, 54(4): 392-404.
WEN Xin, LIU Fanqian, TAN Qiaofeng, et al.Research on short-term multi-level and multi-objective optimal scheduling of cascade hydropower stations under complex water resources and power constraints[J]. Journal of Hydraulic Engineering, 2023, 54(4): 392-404.
[30] 陈实, 朱亚斌, 刘艺洪, 等. 基于世界模型深度强化学习的含风电电力系统低碳经济调度[J]. 电网技术, 2024, 48(8): 3143-3154.
CHEN Shi, ZHU Yabin, LIU Yihong, et al.Low-carbon economic dispatch of wind-containing power systems based on world model deep reinforcement learning[J]. Power System Technology, 2024, 48(8): 3143-3154.
[31] YANG Z X, REN Z Y, LI H, et al.A multi-stage stochastic dispatching method for electricity-hydrogen integrated energy systems driven by model and data[J]. Applied Energy, 2024, 371: 123668.
[32] 刘云飞, 张俊然. 深度神经网络学习率策略研究进展[J]. 控制与决策, 2023, 38(9): 2444-2460.
LIU Yunfei, ZHANG Junran.Research advances in deep neural networks learning rate strategies[J]. Control and Decision, 2023, 38(9): 2444-2460.
[33] CHOW Y, GHAVAMZADEH M, JANSON L, et al.Risk-constrained reinforcement learning with percentile risk criteria[J]. Journal of Machine Learning Research, 2017, 18(1):6070-6120
[34] 马帅, 夏俐. 风险敏感马氏决策过程与状态扩充变换[J]. 中山大学学报(自然科学版)(中英文), 2023, 62(1): 181-191.
MA Shuai, XIA Li.Risk-sensitive Markov decision processes and state augmentation transformation[J]. Acta Scientiarum Naturalium Universitatis Sunyatseni, 2023, 62(1): 181-191.
[35] 李静轩, 周明, 朱凌志, 等. 可再生能源电力系统运行灵活性需求量化及优化调度方法[J]. 电网技术, 2021, 45(7): 2647-2656.
LI Jingxuan, ZHOU Ming, ZHU Lingzhi, et al.Flexibility requirement quantifying and optimal dispatching for renewable integrated power systems[J]. Power System Technology, 2021, 45(7): 2647-2656.
[36] ZHOU H Y, ZHANG S H, PENG J Q, et al.Informer: beyond efficient transformer for long sequence time-series forecasting[J]. ArXiv e-Prints, 2020: arXiv: 2012.07436.
[37] 雷嘉明, 姜爱华, 吴新飞, 等. 计及源荷不确定性的综合能源系统近端策略优化调度[J]. 电力科学与技术学报, 2023, 38(5): 1-11.
LEI Jiaming, JIANG Aihua, WU Xinfei, et al.Optimal scheduling of near-end strategy of integrated energy system considering source-load uncertainty[J]. Journal of Electric Power Science and Technology, 2023, 38(5): 1-11.
[38] 潘超, 王锦鹏, 包钰婷, 等. 考虑源荷侧灵活性资源的风光消纳互动调控[J]. 智慧电力, 2023, 51(1): 1-8.
PAN Chao, WANG Jinpeng, BAO Yuting, et al.Interactive control of wind and photovoltaic power consumption considering source-load side flexible resources[J]. Smart Power, 2023, 51(1): 1-8.
[39] 陈治铭, 刘建华, 柯添赐, 等. 一种对抗性的权重注意力机制Seq2Seq模型估算SOC[J/OL]. 电工技术学报, 2023: 1-12. (2023-11-20). https://kns.cnki.net/KCMS/detail/detail.aspx?filename=DGJS20231113001&dbname=CJFD&dbcode=CJFQ.
CHEN Zhiming, LIU Jianhua, KE Tianci, et al. An antagonistic weight attention mechanism Seq2Seq model to estimate SOC[J/OL]. China Industrial Economics, 2023: 1-12. (2023-11-20). https://kns.cnki.net/KCMS/detail/detail.aspx?filename=DGJS20231113001&dbname=CJFD&dbcode=CJFQ.
[40] 国家能源局. 关于印发《水能利用率计算导则》的通知:国能发新能[2017]4号[EB/OL]. (2017-05-27)[2024-11-12]. https://zfxxgk.nea.gov.cn/auto87/201706/t20170608_2802.htm