Condition-Based Risk Reinforcement Learning for Joint Optimal Scheduling of Cascade Hydropower and Solar-Powered Reservoirs

CHEN Shi; TANG Guodeng; LIU Yihong; XU Liuchao; ZHU Yujie; ZHOU Yi; LI Huaqiang; ZANG Tianlei

PDF(1216 KB)

Electric Power Construction ›› 0

Condition-Based Risk Reinforcement Learning for Joint Optimal Scheduling of Cascade Hydropower and Solar-Powered Reservoirs

CHEN Shi, TANG Guodeng, LIU Yihong, XU Liuchao, ZHU Yujie, ZHOU Yi, LI Huaqiang, ZANG Tianlei

Author information +

History +

Abstract

[Objective] With the advancement of the dual-carbon strategy, hydropower developers are actively promoting pumped-storage retrofitting of hydroelectric facilities while accelerating regional construction of photovoltaic (PV) and other renewable energy sources, gradually forming a scenario with high-penetration renewables and large-scale pumped storage. However, under the dual uncertainties of renewable energy output and interval water inflow, system operational coordination becomes more complex, particularly as scheduling methods for cascade hydropower systems considering interval inflow uncertainty have rarely been studied. [Methods] This paper proposes a conditional risk-aware reinforcement learning-based optimal scheduling method for cascade hydro-PV-pumped storage systems. First, the "Informer" deep neural network is employed to predict basin interval water inflow, transforming inflow uncertainty into flexibility supply indicators. Subsequently, risk theory is integrated to quantify flexibility deficits using Conditional Value at Risk (CVaR). Finally, an improved Risk-Managing Proximal Policy Optimization (RM-PPO) reinforcement learning algorithm is developed to derive optimized scheduling strategies. [Results] Validation using an actual cascade hydro-PV-pumped storage base in China demonstrates that the proposed forecasting method reduces MSE by 18.9% and MAE by 58.8% compared to traditional time-series approaches, with a 78.5% capture rate for peak events. The RM-PPO scheduling algorithm achieves flexible system regulation through pumped-storage plants coordinating with PV for surplus energy absorption, while conventional hydropower stations synergize with PV for source-load dynamic matching. [Conclusions] The RM-PPO-based scheduling strategy effectively reduces costs, controls operational risks while maintaining system flexibility, promotes renewable energy accommodation, and enhances hydropower utilization efficiency.

Key words

Cascade Hydropower-Photovoltaic Storage / Conditional Risk / Reinforcement Learning / Scheduling Optimization

Cite this article

EndNote

Ris (Procite)

Bibtex

Download Citations

CHEN Shi, TANG Guodeng, LIU Yihong, XU Liuchao, ZHU Yujie, ZHOU Yi, LI Huaqiang, ZANG Tianlei. Condition-Based Risk Reinforcement Learning for Joint Optimal Scheduling of Cascade Hydropower and Solar-Powered Reservoirs[J]. Electric Power Construction. 0

References

[1] 齐郑, 徐希茜, 熊巍, 等. 基于变时段设计改进多目标差分进化算法的风/光/火/储日前优化调度[J]. 电力系统保护与控制, 2024, 52(16): 62-71.
QI Zheng, XU Xixi, XIONG Wei, et al.An improved multi objective differential evolution algorithm based on variable time period design for a wind/photovoltaic/thermal/storage system day ahead optimization scheduling[J]. Power System Protection and Control, 2024, 52(16): 62-71.
[2] 孙志媛, 彭博雅, 孙艳. 考虑多能互补的电力电量平衡优化调度策略[J]. 中国电力, 2024, 57(1): 115-122.
SUN Zhiyuan, PENG Boya, SUN Yan.Optimal dispatch strategy of power and electricity balance based on multi-energy complementation[J]. Electric Power, 2024, 57(1): 115-122.
[3] 任大伟, 侯金鸣, 肖晋宇, 等. 支撑双碳目标的新型储能发展潜力及路径研究[J]. 中国电力, 2023, 56(8): 17-25.
REN Dawei, HOU Jinming, XIAO Jinyu, et al.Research on development potential and path of new energy storage supporting carbon peak and carbon neutrality[J]. Electric Power, 2023, 56(8): 17-25.
[4] 吴智泉, 贾纯超, 陈磊, 等. 新型电力系统中储能创新方向研究[J]. 太阳能学报, 2021, 42(10): 444-451.
WU Zhiquan, JIA Chunchao, CHEN Lei, et al.Research on innovative direction of energy storage in new power system construction[J]. Acta Energiae Solaris Sinica, 2021, 42(10): 444-451.
[5] 彭煜民, 王雪林, 刘德旭, 等. 考虑混蓄与纯蓄配置方式的水风光蓄互补系统调度运行对比研究[J]. 电力系统保护与控制, 2024, 52(10): 179-187.
PENG Yumin, WANG Xuelin, LIU Dexu, et al.Comparative research on the dispatching operation of a hydro-wind-solar-storage complementary system considering configurations with hybrid and pure pumped-storage hydropower plants[J]. Power System Protection and Control, 2024, 52(10): 179-187.
[6] 申建建, 王月, 程春田, 等. 水风光多能互补发电调度问题研究现状及展望[J]. 中国电机工程学报, 2022, 42(11): 3871-3885.
SHEN Jianjian, WANG Yue, CHENG Chuntian, et al.Research status and prospect of generation scheduling for hydropower-wind-solar energy complementary system[J]. Proceedings of the CSEE, 2022, 42(11): 3871-3885.
[7] 赵明哲. 雅砻江流域清洁能源基地多尺度风光水互补调度与灵活性研究[D]. 西安: 西安理工大学, 2022.
ZHAO Mingzhe.Study on multi-scale complementary scheduling and flexibility of wind, solar and water for clean energy base in yalong river basin[D]. Xi’an: Xi’an University of Technology, 2022.
[8] TAN Q F, NIE Z, WEN X, et al.Complementary scheduling rules for hybrid pumped storage hydropower-photovoltaic power system reconstructing from conventional cascade hydropower stations[J]. Applied Energy, 2024, 355: 122250.
[9] 程孟增, 唐一金, 商文颖, 等. 风-光-火-抽蓄联合系统中抽水蓄能电站最佳容量配置[J]. 电力建设, 2021, 42(11): 72-81.
CHENG Mengzeng, TANG Yijin, SHANG Wenying, et al.Optimal capacity configuration of pumped-storage power station in wind-PV-fire-pump storage system[J]. Electric Power Construction, 2021, 42(11): 72-81.
[10] 陈雨潇, 龚锦霞, 赵文彬, 等. 面向新型电力系统的变速抽蓄电站调频策略研究[J]. 智慧电力, 2023, 51(3): 104-110.
CHEN Yuxiao, GONG Jinxia, ZHAO Wenbin, et al.Frequency modulation strategy for variable-speed pumped-storage power station for new power system[J]. Smart Power, 2023, 51(3): 104-110.
[11] 周云海, 张智颖, 徐飞, 等. 含多座抽水蓄能电站的省级电网日前计划分层递进优化算法[J]. 中国电力, 2023, 56(5): 41-50.
ZHOU Yunhai, ZHANG Zhiying, XU Fei, et al.Hierarchical progressive optimization algorithm for day-ahead planning of provincial power grid with multiple pumped storage power stations[J]. Electric Power, 2023, 56(5): 41-50.
[12] 刘德旭, 杨迎, 黄宏旭, 等. 新型电力系统大规模抽水蓄能调度运行与控制综述及展望[J]. 中国电机工程学报, 2025, 45(1): 80-98.
LIU Dexu, YANG Ying, HUANG Hongxu, et al.An overview and outlook of the operation and control of large-scale pumped hydro storages in modern power systems[J]. Proceedings of the CSEE, 2025, 45(1): 80-98.
[13] 罗彬, 陈永灿, 刘昭伟, 等. 梯级水光互补系统最大化可消纳电量期望短期优化调度模型[J]. 电力系统自动化, 2023, 47(10): 66-75.
LUO Bin, CHEN Yongcan, LIU Zhaowei, et al.Short-term optimal dispatch model for maximizing expectation of accommodation power of cascade hydro-photovoltaic complementary system[J]. Automation of Electric Power Systems, 2023, 47(10): 66-75.
[14] 李咸善, 杨拯, 李飞, 等. 基于梯级水电调节的风光水联盟与区域电网联合运行优化调度策略[J]. 中国电机工程学报, 2023, 43(6): 2234-2248.
LI Xianshan, YANG Zheng, LI Fei, et al.Optimization scheduling strategy for joint operation of wind-solar-water power alliance and regional power grid based on cascade hydropower regulation[J]. Proceedings of the CSEE, 2023, 43(6): 2234-2248.
[15] 曹敏健, 胡泽春, 孟颖, 等. 含抽蓄电站与新能源发电的柔性直流系统日前优化调度方法[J]. 电力系统自动化, 2021, 45(15): 36-44.
CAO Minjian, HU Zechun, MENG Ying, et al.Day-ahead optimal dispatch method for flexible DC system with pumped storage hydropower plant and new energy power generation[J]. Automation of Electric Power Systems, 2021, 45(15): 36-44.
[16] 夏依莎, 刘俊勇, 刘继春, 等. 基于电量共享的梯级水光蓄联合发电系统优化调度策略[J]. 电力自动化设备, 2021, 41(9): 118-125.
XIA Yisha, LIU Junyong, LIU Jichun, et al.Optimal scheduling strategy of cascaded hydro-photovoltaic-pumped storage hybrid generation system based on electric energy sharing[J]. Electric Power Automation Equipment, 2021, 41(9): 118-125.
[17] 葛晓琳, 钟俊玲, 符杨. 考虑来水多区间不确定集的跨年梯级水电鲁棒优化调度[J]. 中国电机工程学报, 2018, 38(17): 5097-5106, 5306.
GE Xiaolin, ZHONG Junling, FU Yang.Robust biennial scheduling of cascade hydropower system considering with the multi-band uncertainty set of the inflow[J]. Proceedings of the CSEE, 2018, 38(17): 5097-5106, 5306.
[18] 赵志鹏, 于志辉, 程春田, 等. 水风光综合基地多风险量化及长期多目标协调优化调度方法[J]. 电力系统自动化, 2024, 48(22): 118-130.
ZHAO Zhipeng, YU Zhihui, CHENG Chuntian, et al.Multi-risk quantification and long-term multi-objective coordinative optimal dispatch method for hydro-wind-solar integrated energy base[J]. Automation of Electric Power Systems, 2024, 48(22): 118-130.
[19] 孙惠娟, 巩磊, 彭春华, 等. 考虑风光水多重不确定性置信风险的多目标动态分解优化调度[J]. 电网技术, 2022, 46(9): 3416-3428.
SUN Huijuan, GONG Lei, PENG Chunhua, et al.Multi-objective dynamic decomposition optimal dispatch considering wind-photovoltaic-hydro multiple uncertainty confidence risk[J]. Power System Technology, 2022, 46(9): 3416-3428.
[20] 马宇航, 黄媛, 刘俊勇, 等. 考虑日内来水不确定和电网断面约束的梯级水电日前调度[J]. 电力建设, 2020, 41(9): 39-49.
MA Yuhang, HUANG Yuan, LIU Junyong, et al.Day-ahead dispatching of cascade hydropower stations considering daily interval flow uncertainty and the section constraint of power grid[J]. Electric Power Construction, 2020, 41(9): 39-49.
[21] JOHNATHON C, AGALGAONKAR A P, PLANIDEN C, et al.A proposed hedge-based energy market model to manage renewable intermittency[J]. Renewable Energy, 2023, 207: 376-384.
[22] 王辉, 王军杰, 王腾飞, 等. 出力不确定下基于CVaR的风电供应链优化决策[J]. 电力科学与技术学报, 2023, 38(4): 134-142.
WANG Hui, WANG Junjie, WANG Tengfei, et al.Optimal decision of wind power supply chain based on CVaR under uncertainty output[J]. Journal of Electric Power Science and Technology, 2023, 38(4): 134-142.
[23] 周新颖. 数据驱动的电力批发市场竞价策略研究[D]. 北京: 华北电力大学, 2023.
ZHOU Xinying.Research on bidding strategy of data-driven power wholesale market[D]. Beijing: North China Electric Power University, 2023.
[24] 高万胜, 蔺红. 考虑配电网灵活性不足风险的分布鲁棒低碳优化调度[J]. 电力系统保护与控制, 2024, 52(16): 49-61.
GAO Wansheng, LIN Hong.Distributionally robust low-carbon optimal scheduling considering flexibility deficiency risk in a distribution network[J]. Power System Protection and Control, 2024, 52(16): 49-61.
[25] 曾小青, 唐超雯. 分时电价环境下计及新能源消纳的虚拟电厂优化调度研究[J]. 电力科学与技术学报, 2023, 38(3): 24-34.
ZENG Xiaoqing, TANG Chaowen.Research on optimization of virtual power plants dispatch by considering the consumption of new energy under time-of-use electricity price environment[J]. Journal of Electric Power Science and Technology, 2023, 38(3): 24-34.
[26] 周耀辉, 司渭滨, 代立君, 等. 基于随机规划的新型电力系统优化调度方法[J/OL]. 自动化技术与应用, 2024: 1-6. (2024-11-26). https://kns.cnki.net/KCMS/detail/detail.aspx?filename=ZDHJ20241122006&dbname=CJFD&dbcode=CJFQ.
ZHOU Yaohui, SI Weibin, DAI Lijun, et al. A new power system optimal dispatching method based on stochastic programming[J/OL]. China Industrial Economics, 2024: 1-6. (2024-11-26). https://kns.cnki.net/KCMS/detail/detail.aspx?filename=ZDHJ20241122006&dbname=CJFD&dbcode=CJFQ.
[27] GAO Y, TAHIR M, SIANO P, et al.Optimization of renewable energy-based integrated energy systems: a three-stage stochastic robust model[J]. Applied Energy, 2025, 377: 124635.
[28] YUAN W L, WANG X Q, SU C G, et al.Stochastic optimization model for the short-term joint operation of photovoltaic power and hydropower plants based on chance-constrained programming[J]. Energy, 2021, 222: 119996.
[29] 闻昕, 刘凡骞, 谭乔凤, 等. 复杂水利-电力约束下梯级水电站短期多层级多目标优化调度研究[J]. 水利学报, 2023, 54(4): 392-404.
WEN Xin, LIU Fanqian, TAN Qiaofeng, et al.Research on short-term multi-level and multi-objective optimal scheduling of cascade hydropower stations under complex water resources and power constraints[J]. Journal of Hydraulic Engineering, 2023, 54(4): 392-404.
[30] 陈实, 朱亚斌, 刘艺洪, 等. 基于世界模型深度强化学习的含风电电力系统低碳经济调度[J]. 电网技术, 2024, 48(8): 3143-3154.
CHEN Shi, ZHU Yabin, LIU Yihong, et al.Low-carbon economic dispatch of wind-containing power systems based on world model deep reinforcement learning[J]. Power System Technology, 2024, 48(8): 3143-3154.
[31] YANG Z X, REN Z Y, LI H, et al.A multi-stage stochastic dispatching method for electricity-hydrogen integrated energy systems driven by model and data[J]. Applied Energy, 2024, 371: 123668.
[32] 刘云飞, 张俊然. 深度神经网络学习率策略研究进展[J]. 控制与决策, 2023, 38(9): 2444-2460.
LIU Yunfei, ZHANG Junran.Research advances in deep neural networks learning rate strategies[J]. Control and Decision, 2023, 38(9): 2444-2460.
[33] CHOW Y, GHAVAMZADEH M, JANSON L, et al.Risk-constrained reinforcement learning with percentile risk criteria[J]. Journal of Machine Learning Research, 2017, 18(1):6070-6120
[34] 马帅, 夏俐. 风险敏感马氏决策过程与状态扩充变换[J]. 中山大学学报(自然科学版)(中英文), 2023, 62(1): 181-191.
MA Shuai, XIA Li.Risk-sensitive Markov decision processes and state augmentation transformation[J]. Acta Scientiarum Naturalium Universitatis Sunyatseni, 2023, 62(1): 181-191.
[35] 李静轩, 周明, 朱凌志, 等. 可再生能源电力系统运行灵活性需求量化及优化调度方法[J]. 电网技术, 2021, 45(7): 2647-2656.
LI Jingxuan, ZHOU Ming, ZHU Lingzhi, et al.Flexibility requirement quantifying and optimal dispatching for renewable integrated power systems[J]. Power System Technology, 2021, 45(7): 2647-2656.
[36] ZHOU H Y, ZHANG S H, PENG J Q, et al.Informer: beyond efficient transformer for long sequence time-series forecasting[J]. ArXiv e-Prints, 2020: arXiv: 2012.07436.
[37] 雷嘉明, 姜爱华, 吴新飞, 等. 计及源荷不确定性的综合能源系统近端策略优化调度[J]. 电力科学与技术学报, 2023, 38(5): 1-11.
LEI Jiaming, JIANG Aihua, WU Xinfei, et al.Optimal scheduling of near-end strategy of integrated energy system considering source-load uncertainty[J]. Journal of Electric Power Science and Technology, 2023, 38(5): 1-11.
[38] 潘超, 王锦鹏, 包钰婷, 等. 考虑源荷侧灵活性资源的风光消纳互动调控[J]. 智慧电力, 2023, 51(1): 1-8.
PAN Chao, WANG Jinpeng, BAO Yuting, et al.Interactive control of wind and photovoltaic power consumption considering source-load side flexible resources[J]. Smart Power, 2023, 51(1): 1-8.
[39] 陈治铭, 刘建华, 柯添赐, 等. 一种对抗性的权重注意力机制Seq2Seq模型估算SOC[J/OL]. 电工技术学报, 2023: 1-12. (2023-11-20). https://kns.cnki.net/KCMS/detail/detail.aspx?filename=DGJS20231113001&dbname=CJFD&dbcode=CJFQ.
CHEN Zhiming, LIU Jianhua, KE Tianci, et al. An antagonistic weight attention mechanism Seq2Seq model to estimate SOC[J/OL]. China Industrial Economics, 2023: 1-12. (2023-11-20). https://kns.cnki.net/KCMS/detail/detail.aspx?filename=DGJS20231113001&dbname=CJFD&dbcode=CJFQ.
[40] 国家能源局. 关于印发《水能利用率计算导则》的通知:国能发新能[2017]4号[EB/OL]. (2017-05-27)[2024-11-12]. https://zfxxgk.nea.gov.cn/auto87/201706/t20170608_2802.htm