Collaborative Optimization of Electricity-Hydrogen Coupled Multi-Virtual Power Plants Based on Multi-Agent Reinforcement Learning

ZHU Chunxu, YANG Shuxia

Electric Power Construction ›› 2026, Vol. 47 ›› Issue (4) : 93-107.

PDF(2068 KB)
PDF(2068 KB)
Electric Power Construction ›› 2026, Vol. 47 ›› Issue (4) : 93-107. DOI: 10.12204/j.issn.1000-7229.2026.04.008
Research and Application of AI Technology in the Market Mechanism and Operation Optimization of New-type Power System·Hosted by LI Yanbin, ZHANG Shuo, DONG Fugui, ZENG Bo·

Collaborative Optimization of Electricity-Hydrogen Coupled Multi-Virtual Power Plants Based on Multi-Agent Reinforcement Learning

Author information +
History +

Abstract

[Objective] In the context of large-scale grid integration of renewable energy and deepening reforms in the electricity market, a new collaborative optimization framework consisting of multi-virtual power plants (MVPP) and an aggregate-level virtual power plant operator (VPPO) is established to address the challenges posed by the uncertainty of wind and solar power output, the complexity of electricity-hydrogen energy complementarity, and multi-level market coordination. [Methods] First, the Wasserstein distance is employed to construct a fuzzy set for wind and solar power output, and a two-stage distribution robust optimization model is combined to quantify the risk of prediction errors. Subsequently, a Stackelberg-Nash bi-level game framework is established, where VPPO, as the leader, dynamically formulates strategies for electricity/hydrogen energy trading prices and resource allocation, while MVPP, as the follower, optimizes electricity purchase and sale plans, hydrogen scheduling, and flexible load response. Finally, the multi-agent twin delayed deep deterministic policy gradient (MATD3) algorithm is adopted to efficiently solve high-dimensional non-convex problems through the utilization of a centralized training and decentralized execution mechanism and a dual Critic network to mitigate strategy bias. [Results] The numerical results demonstrate that the proposed strategy successfully optimizes the cross-market arbitrage profits of VPPO and the operational costs of MVPP, significantly enhancing the convergence efficiency and strategy stability of the algorithm. It also bolsters the system's robustness against fluctuations in wind and solar power and the flexibility of electricity-hydrogen coordination, while optimizing the spatiotemporal distribution of resources through flexible peak load shaving services. [Conclusions] By integrating robust optimization and game theory, this study effectively coordinates electricity-hydrogen coupling with multi-market interactions, providing an innovative solution to the economic and robust operation of MVPP under high penetration of integrated renewable energy.

Key words

multi-virtual power plants (MVPP) / electricity-hydrogen coupling / multi-agent reinforcement learning / distributionally robust optimization / Stackelberg-Nash game / wind and solar power uncertainty

Cite this article

Download Citations
ZHU Chunxu , YANG Shuxia. Collaborative Optimization of Electricity-Hydrogen Coupled Multi-Virtual Power Plants Based on Multi-Agent Reinforcement Learning[J]. Electric Power Construction. 2026, 47(4): 93-107 https://doi.org/10.12204/j.issn.1000-7229.2026.04.008

References

[1]
张智刚, 康重庆. 碳中和目标下构建新型电力系统的挑战与展望[J]. 中国电机工程学报, 2022, 42(8): 2806-2818.
ZHANG Zhigang, KANG Chongqing. Challenges and prospects for constructing the new-type power system towards a carbon neutrality future[J]. Proceedings of the CSEE, 2022, 42(8): 2806-2818.
[2]
钟海旺, 张宁, 杜尔顺, 等. 新型电力系统中的规划运营与电力市场: 研究进展与科研实践[J]. 中国电机工程学报, 2024, 44(18): 7084-7104.
ZHONG Haiwang, ZHANG Ning, DU Ershun, et al. Planning, operation and market of new power system: research progress and practice[J]. Proceedings of the CSEE, 2024, 44(18): 7084-7104.
[3]
李振坤, 张兆柯, 李景岳, 等. 面向电能量与调频联合市场的虚拟电厂集群投标策略[J]. 电力系统自动化, 2025, 49(20): 94-102.
LI Zhengkun, ZHANG Zhaoke, LI Jingyue, et al. Bidding strategy of virtual power plant clusters for joint market of electric power energy and frequency regulation[J]. Automation of Electric Power Systems, 2025, 49(20): 94-102.
[4]
肖白, 于海洋, 焦明曦, 等. 基于演变虚拟净负荷的新型电力系统日前优化调度[J]. 电力建设, 2025, 46(9): 98-110.
XIAO Bai, YU Haiyang, JIAO Mingxi, et al. Day-ahead optimal dispatch of new power system based on evolving virtual net load[J]. Electric Power Construction, 2025, 46(9): 98-110.
[5]
王秋杰, 刘国安, 谭洪, 等. 考虑电氢耦合的虚拟电厂鲁棒可行域模型与求解[J]. 电网技术, 2025, 49(3): 889-898.
WANG Qiujie, LIU Guoan, TAN Hong, et al. Models and solutions of robust feasible region for virtual power plants with electric hydrogen coupling[J]. Power System Technology, 2025, 49(3): 889-898.
[6]
王家怡, 贺帅佳. 计及新型分布式资源与电碳交易的虚拟电厂分布鲁棒低碳调度模型[J]. 电力建设, 2025, 46(7): 13-26.
WANG Jiayi, HE Shuaijia. Distributionally robust low-carbon scheduling model for virtual power plants considering emerging distributed resources and electricity carbon trading[J]. Electric Power Construction, 2025, 46(7): 13-26.
[7]
汪岩佳, 王宇川, 王西田, 等. 考虑虚拟电厂分布式资源联动不确定性的优化调度策略[J]. 电力系统自动化, 2025, 49(20): 103-112.
WANG Yanjia, WANG Yuchuan, WANG Xitian, et al. Optimal scheduling strategy considering linkage uncertainty of distributed resources in virtual power plant[J]. Automation of Electric Power Systems, 2025, 49(20): 103-112.
[8]
陈景文, 叶鹏程, 刘耀先, 等. 基于阶梯碳交易的含碳捕集与高耗能负荷的虚拟电厂优化调度[J/OL]. 电网技术, 2025: 1-17. (2025-04-18) [2025-07-22]. https://doi.org/10.13335/j.1000-3673.pst.2025.0155.
CHEN Jingwen, YE Pengcheng, LIU Yaoxian, et al. Optimization dispatch of virtual power plant with carbon capture and high-energy-consumption loads based on stepped carbon trading[J/OL]. Power System Technology, 2025: 1-17. (2025-04-18) [2025-07-22]. https://doi.org/10.13335/j.1000-3673.pst.2025.0155.
[9]
陈文颖, 郑修诺, 闫倩文, 等. 绿证-碳交易交互机制下的虚拟电厂优化调度[J/OL]. 南方电网技术, 2025: 1-12. (2025-03-25) [2025-07-22]. https://link.cnki.net/urlid/44.1643.tk.20250324.1500.015.
CHEN Wenying, ZHENG Xiunuo, YAN Qianwen, et al. Optimal scheduling of virtual power plant under the green certificate carbon trading interaction mechanism[J/OL]. Southern Power System Technology, 2025: 1-12. (2025-03-25) [2025-07-22]. https://link.cnki.net/urlid/44.1643.tk.20250324.1500.015.
[10]
刘汶瑜, 陈中, 杜璞良, 等. 基于联盟博弈的多虚拟电厂参与日前电力市场竞标模型[J]. 电力自动化设备, 2024, 44(5): 135-142, 150.
LIU Wenyu, CHEN Zhong, DU Puliang, et al. Bidding model for multiple virtual power plants participating in day-ahead electricity market based on coalition game[J]. Electric Power Automation Equipment, 2024, 44(5): 135-142, 150.
[11]
LIU X O. Bi-layer game method for scheduling of virtual power plant with multiple regional integrated energy systems[J]. International Journal of Electrical Power & Energy Systems, 2023, 149: 109063.
[12]
JIA D Q, LI X M, GONG X, et al. Bi-level strategic bidding model of novel virtual power plant aggregating waste gasification in integrated electricity and hydrogen markets[J]. Applied Energy, 2024, 357: 122468.
[13]
ZHOU K L, PENG N, YIN H, et al. Urban virtual power plant operation optimization with incentive-based demand response[J]. Energy, 2023, 282: 128700.
[14]
汤晨阳, 王磊, 江伟建. 计及不确定性风险与电能贡献度的多虚拟电厂协同优化策略[J]. 电力建设, 2025, 46(7): 27-41.
TANG Chenyang, WANG Lei, JIANG Weijian. Collaborative optimization strategy for multiple virtual power plants considering uncertainty risk and energy contribution[J]. Electric Power Construction, 2025, 46(7): 27-41.
[15]
栗然, 王炳乾, 彭湘泽, 等. 基于主从博弈的多虚拟电厂动态定价与优化调度[J]. 可再生能源, 2024, 42(7): 986-994.
LI Ran, WANG Bingqian, PENG Xiangze, et al. Dynamic pricing and optimal scheduling of multi-virtual power plants based on master-slave game[J]. Renewable Energy Resources, 2024, 42(7): 986-994.
[16]
周步祥, 张越, 臧天磊, 等. 基于区块链的多虚拟电厂主从博弈优化运行[J]. 电力系统自动化, 2022, 46(1): 155-163.
ZHOU Buxiang, ZHANG Yue, ZANG Tianlei, et al. Blockchain-based Stackelberg game optimal operation of multiple virtual power plants[J]. Automation of Electric Power Systems, 2022, 46(1): 155-163.
[17]
沈思辰, 韩海腾, 周亦洲, 等. 基于条件风险价值的多虚拟电厂电-碳-备用P2P交易模型[J]. 电力系统自动化, 2022, 46(18): 147-157.
SHEN Sichen, HAN Haiteng, ZHOU Yizhou, et al. Electricity-carbon-reserve peer-to-peer trading model for multiple virtual power plants based on conditional value-at-risk[J]. Automation of Electric Power Systems, 2022, 46(18): 147-157.
[18]
程雪婷, 王金浩, 金玉龙, 等. 计及配电网运行约束的多虚拟电厂合作博弈策略[J]. 南方电网技术, 2023, 17(4): 119-131.
Abstract
虚拟电厂(virtual power plant,VPP)作为一种有效的可再生能源聚合利用手段,近年来得到迅速发展,随着VPP并网规模的扩大,多个互联VPP交易博弈问题日益凸显。针对多个VPP间的交易博弈问题,考虑物理网络特性,提出了计及配电网运行约束的多VPP合作博弈策略。首先,考虑配电网运行约束,对VPP内部资源进行整合和建模,建立了VPP 能量管理模型。其次,通过引入点对点(peer to peer,P2P)能量交易,实现多VPP系统自主能量管理与协同定价,能够在不损害各方利益的情况下达成P2P能量交易。同时考虑到用电负荷和可再生能源出力的不确定性,利用典型场景生成算法构造了不确定性变量的概率分布模糊集。针对多主体交易产生的隐私性问题,采用列和约束生成算法联合交替方向乘子法对模型进行求解。最后,在IEEE 123节点测试系统上进行算例仿真,仿真结果验证了所提模型和算法的有效性。
CHENG Xueting, WANG Jinhao, JIN Yulong, et al. Cooperative game strategy of multiple virtual power plants considering the operational constraints of distribution network[J]. Southern Power System Technology, 2023, 17(4): 119-131.

In recent years, virtual power plant (VPP) has been developing rapidly as an effective means of aggregate utilization of renewable energy. With the expansion of VPP grid-connected scale, the problem of multiple interconnected VPP transactions has become increasingly prominent. Aiming at the transaction game between multiple VPPs, considering the characteristics of the physical network, cooperative game strategy of multiple VPPs considering the operational constraints of distribution network is proposed in this paper. Firstly, considering the operation constraints of the distribution network, the internal resources of the VPP are integrated and modeled, and then the VPP energy management model is established. Secondly, through the introduction of peer-to-peer (P2P) energy transactions, the autonomous energy management and collaborative pricing of multiple VPPs systems can be realized, and P2P energy transactions can be achieved without harming the interests of all parties. At the same time, considering the uncertainty of electricity load and renewable energy output, a typical scenario generation algorithm is used to construct a fuzzy set of probability distribution of uncertain variables. Aiming at the privacy problem arising from multi-agent transactions, column constraint generation algorithm combined with alternating direction multiplier method are adopted to solve the model in this paper. Finally, a numerical example is simulated on the IEEE 123 nodes test system, and the simulation results verify the effectiveness of the proposed model and algorithm.

[19]
葛晓琳, 曹旭丹, 李佾玲, 等. 考虑风险与碳流动的多虚拟电厂优化运行方法[J]. 电力系统及其自动化学报, 2023, 35(8): 126-135.
GE Xiaolin, CAO Xudan, LI Yiling, et al. Optimal operation method for multiple virtual power plants considering risk and carbon flow[J]. Proceedings of the CSU-EPSA, 2023, 35(8): 126-135.
[20]
LI Q, DONG F X, ZHOU G W, et al. Co-optimization of virtual power plants and distribution grids: emphasizing flexible resource aggregation and battery capacity degradation[J]. Applied Energy, 2025, 377: 124519.
[21]
樊伟, 范英, 谭忠富, 等. 基于多层利益共享的虚拟电厂参与电碳市场分布鲁棒优化模型[J]. 系统工程理论与实践, 2024, 44(2): 661-683.
Abstract
风电和光伏的高渗透率增加了新型电力系统对灵活性资源需求.虚拟电厂作为一个特殊电厂聚合了可控分布式电源、新能源、储能、碳处理、负荷等各类资源,“对内协同”可以实现内部资源协同调控,“对外统一”可以参与外部电碳市场获利.基于此,本文创新地提出了虚拟电厂参与电碳多类市场分布鲁棒优化模型.为了刻画风电和光伏的不确定性,构造了基于Wasserstein距离的分布模糊集和基于数据驱动的误差不确定集.为了兼顾经济性和鲁棒性,考虑内部运行成本以及外部参与多类市场收益,构建了最坏分布下期望收益最大的两阶段鲁棒优化模型,并提出模型求解方法.为了保证联盟动态平衡,提出了多层利益分配方法.最后,算例分析表明:在“对内协同,对外统一”的经营模式下,有效激发虚拟电厂内各类资源的潜力,参与多个市场后获取共享利益,实现了多方互利共赢.所提模型具有数据驱动、快速求解、灵活可控、经济实用等优越性.多层利益分配方法能够简便、有效地将共享效益传导至各主体.
FAN Wei, FAN Ying, TAN Zhongfu, et al. Distributionally robust optimization model for virtual power plant participation in electricity carbon market based on multi-layer benefit sharing[J]. Systems Engineering - Theory & Practice, 2024, 44(2): 661-683.
[22]
XUE L, ZHANG Y, WANG J X, et al. Privacy-preserving multi-level co-regulation of VPPs via hierarchical safe deep reinforcement learning[J]. Applied Energy, 2024, 371: 123654.
[23]
钟永洁, 汤成俊, 王紫东, 等. 我国虚拟电厂的发展演进和关键技术及难点分析[J]. 浙江电力, 2025, 44(2): 13-31.
ZHONG Yongjie, TANG Chengjun, WANG Zidong, et al. Analysis of the evolution, key technologies, and challenges of virtual power plants in China[J]. Zhejiang Electric Power, 2025, 44(2): 13-31.
[24]
KONG X Y, XIAO J, WANG C S, et al. Bi-level multi-time scale scheduling method based on bidding for multi-operator virtual power plant[J]. Applied Energy, 2019, 249: 178-189.
[25]
GUO W S, LIU P K, SHU X L. Optimal dispatching of electric-thermal interconnected virtual power plant considering market trading mechanism[J]. Journal of Cleaner Production, 2021, 279: 123446.
[26]
宋铎洋, 薛田良, 李艺瀑, 等. 考虑风光不确定性的虚拟电厂合作博弈调度及收益分配策略[J]. 电力工程技术, 2025, 44(1): 193-206.
SONG Duoyang, XUE Tianliang, LI Yipu, et al. Cooperative game scheduling and revenue sharing strategy for virtual power plants considering scenery uncertainty[J]. Electric Power Engineering Technology, 2025, 44(1): 193-206.
[27]
高明, 曾平良, 冯永朝. 新型电力系统中的虚拟电厂研究综述[J]. 电力工程技术, 2025, 44(6): 143-154.
GAO Ming, ZENG Pingliang, FENG Yongchao. Review of virtual power plant in new power system[J]. Electric Power Engineering Technology, 2025, 44(6): 143-154.
[28]
DUAN C, FANG W L, JIANG L, et al. Distributionally robust chance-constrained approximate AC-OPF with Wasserstein metric[J]. IEEE Transactions on Power Systems, 2018, 33(5): 4924-4936.
[29]
赵宇轩, 宋伟峰, 李伟康, 等. 考虑共享储能容量配置的多虚拟电厂优化运行方法[J]. 电网与清洁能源, 2024, 40(1): 92-101.
ZHAO Yuxuan, SONG Weifeng, LI Weikang, et al. Optimal operation method of multiple virtual power plants considering shared energy storage capacity allocation[J]. Power System and Clean Energy, 2024, 40(1): 92-101.
[30]
陈胜, 张景淳, 卫志农, 等. 面向能源转型的电-气-氢综合能源系统规划与运行[J]. 电力系统自动化, 2023, 47(19): 16-30.
CHEN Sheng, ZHANG Jingchun, WEI Zhinong, et al. Energy transition oriented planning and operation of electricity-gas-hydrogen integrated energy system[J]. Automation of Electric Power Systems, 2023, 47(19): 16-30.
[31]
彭生江, 杨德州, 孙传帅, 等. 基于氢负荷需求的氢能系统容量规划[J]. 中国电力, 2023, 56(7): 13-20, 32.
PENG Shengjiang, YANG Dezhou, SUN Chuanshuai, et al. Capacity planning of hydrogen production and storage system based on hydrogen load demand[J]. Electric Power, 2023, 56(7): 13-20, 32.
[32]
陈永权, 方瑜. 多组合虚拟电厂中氢储能低碳经济配置与优化[J]. 电网与清洁能源, 2024, 40(3): 107-118.
CHEN Yongquan, FANG Yu. Low carbon economic configuration and optimization of hydrogen storage in multi-portfolio virtual power plants[J]. Power System and Clean Energy, 2024, 40(3): 107-118.
[33]
林顺富, 高一焱, 周波, 等. 计及能量共享的多虚拟电厂参与电能量-FRP市场优化运行策略[J]. 浙江电力, 2025, 44(10): 139-151.
LIN Shunfu, GAO Yiyan, ZHOU Bo, et al. An optimal operation strategy for multiple virtual power plants participating in energy-FRP markets with energy sharing[J]. Zhejiang Electric Power, 2025, 44(10): 139-151.
[34]
ZHANG J C, SANG L W, XU Y L, et al. Networked multiagent-based safe reinforcement learning for low-carbon demand management in distribution networks[J]. IEEE Transactions on Sustainable Energy, 2024, 15(3): 1528-1545.
[35]
周健树, 向月, 张新, 等. 基于深度强化学习的高速公路服务区新能源充电站两阶段优化调控策略[J]. 中国电机工程学报, 2025, 45(11): 4130-4144.
ZHOU Jianshu, XIANG Yue, ZHANG Xin, et al. Two-stage optimal dispatch strategy of new energy charging station in highway service area based on deep reinforcement learning[J]. Proceedings of the CSEE, 2025, 45(11): 4130-4144.

Footnotes

利益冲突声明(Conflict of Interests) 所有作者声明不存在利益冲突。

Funding

National Natural Science Foundation of China(72074074)
PDF(2068 KB)

Accesses

Citation

Detail

Sections
Recommended

/