PDF(4274 KB)
Expert Experience-Guided Hierarchical Reinforcement Learning for Coordinated Post-Disaster Restoration of Distribution Networks
XUE Zhang, LI Xiaolu, LIU Jinsong, LIN Shunfu
Electric Power Construction ›› 2026, Vol. 47 ›› Issue (10) : 1-14.
PDF(4274 KB)
PDF(4274 KB)
Expert Experience-Guided Hierarchical Reinforcement Learning for Coordinated Post-Disaster Restoration of Distribution Networks
[Objective] Extreme disasters are prone to trigger multi-point faults in distribution networks, and the lack of coordination between network reconfiguration and fault emergency repair will degrade the efficiency of fault restoration. This paper proposes a coordinated post-disaster restoration strategy for distribution networks based on expert experience-guided hierarchical reinforcement learning (HRL). [Methods] Firstly, a decision-making model for post-disaster coordinated restoration of distribution networks is constructed. Taking the minimization of load loss, switch operation cost and maintenance travel cost as the optimization objectives, the model comprehensively considers the operation constraints, network topology constraints and fault repair constraints of distribution networks, as well as the time-varying traffic accessibility constraints of transportation networks. Secondly, targeting the sequential decision-making characteristics of post-disaster restoration, a hierarchical reinforcement learning decision framework consisting of high-level task selection and low-level action execution is established. The high-level layer selects network reconfiguration or repair scheduling tasks according to real-time system states, while the low-level layer generates topology reconfiguration operations, repair task allocation schemes and route adjustment strategies under corresponding constraint conditions. Finally, expert trajectories are adopted to pre-train the policy network, and the parameters of the hierarchical policy are continuously updated through online interactive training. [Results] Random disaster scenario tests on a coupled system of the modified IEEE 33-bus distribution network and the Sioux-Falls traffic network demonstrate that the average cumulative load loss of the proposed method is 5.45 MWh, which is reduced by 22.43% and 29.57% compared with the mixed-integer second-order cone programming (MISOCP) and binary particle swarm optimization (BPSO) algorithms, respectively. The average online solution time of the proposed method is 0.032 7 s, which is significantly lower than that of the comparative methods. [Conclusions] The proposed method realizes the coordinated optimization of post-disaster network reconfiguration and repair scheduling for distribution networks. It exhibits superior performance in reducing cumulative load loss and improving online decision-making efficiency, and can provide auxiliary decision support for the rapid restoration of distribution networks under extreme disaster conditions.
distribution network restoration / hierarchical reinforcement learning / behavior cloning / fault emergency repair / network reconfiguration
一个区域的交通网可以由有限个节点集合与边集合构成的图表示,即GTN=(NTN, BTN),在DSR问题中,根据RC所处的初始节点及故障点,可形成由Nrd集合所构成的交通调度子网模型。RC的交通调度网示例如图A1所示,其中RC的交通调度网由节点集Nrd={N0,…,N8,N10}及其所构成的路段组成,RC驻点N0与故障点N10空间位置是固定的,但灾后路面极可能受到破坏影响RC移动速度,甚至部分路段无法通行,在灾后抢修的急迫情况下,等待交通恢复势必延长供电恢复过程,因此,在规划RC调度方案时需要考虑道路通行风险因素,本文将道路广义距离与路阻函数相结合,对任意路段两节点的转移时间
式中:dmn为路段实际距离;Dmn为灾后道路广义距离;η为可安全通行的最高阈值;θmn为路段的损毁率,取值范围均为[0,1],当损毁率大于安全通行阈值时,道路中断;
在多个路由可达情况下,选择总行程时间T(l,t)最短者:
式中:Ro→k为起点o至终点k可行路由集合,在图A1的调度网络中,RC可以通过三条路由集合到达抢修点,即l1,l2,l3∈R0→10,但距离最短的路由l1部分道路损毁(图A1中N6—N7路段)无法安全通行,同时,路由l2上所需要的转移时间相较l3要长。在这种情况下,RC将选择路由l3作抢修路线。
表B1 基于PPO两阶段分层训练伪代码Table B1 Two-stage hierarchical training pseudocode with PPO |
| Input: 预训练参数 Output:训练完成的策略网参数 |
|---|
| 1: 加载预训练保存的策略网络参数 2: 初始化critic networks及PPO相关参数 3: For each training episode do 4: 初始化环境状态s0 /* ----------阶段一:低层执行策略训练---------- */ 5: For each 6: 初始化低层经验池 7: Repeat 8: 选择动作 9: 10: 11: Until 经验池达到预设长度 12: PPO算法按照式(27)计算更新低层参数 13: End for 14: 保存训练后最优低层策略参数 /* ----------阶段二:高层选项策略训练 ---------- */ 15: 初始化高层经验池 16: 加载训练保存的低层策略网络参数 17: Repeat 18: 选择选项 19: 在低层策略网络参数 20: 获得高层状态 21: 22: 更新环境状态 23: Until 经验池达到预设长度 24: 按照式(30)计算更新高层策略参数 25: End for 26: 保存训练完成的策略网参数 |
表B2 IEEE 33系统DG参数Table B2 IEEE 33 system DG parameters |
| 编号 | 接入点 | 有功限制/kW | 无功限制/kvar |
|---|---|---|---|
| DG1 | 4 | [0,170] | [-83,83] |
| DG2 | 8 | [0,410] | [-198,198] |
| DG3 | 13 | [0,600] | [-290,290] |
| DG4 | 20 | [0,260] | [-126,126] |
| DG5 | 23 | [0,220] | [-106,106] |
| DG6 | 32 | [0,350] | [-169,169] |
表B3 超参数设置Table B3 Hyperparameter settings |
| 超参数 | 值 |
|---|---|
| 学习率(learning rate) | 0.000 3 |
| 折扣因子(discount factor) | 0.99 |
| 批量大小(batch size) | 128 |
| 最大回合步长(max episode steps) | 50 |
| 训练周期(training episodes) | 300 000 |
| 每轮更新步数(rollout) | 4096 |
表B4 故障候选点参数Table B4 Fault candidate point parameters |
| 编号 | DN位置 | TN位置 | 修复时间/min |
|---|---|---|---|
| M1 | 线路23-24 | 9 | 75 |
| M2 | 线路27-28 | 4 | 55 |
| M3 | 线路16-17 | 18 | 50 |
| M4 | 线路6-26 | 3 | 60 |
| M5 | 线路6-7 | 12 | 80 |
| M6 | 线路2-3 | 10 | 50 |
| M7 | 线路19-20 | 20 | 30 |
| [1] |
孙为民, 孙华东, 何剑, 等. 面向严重自然灾害的电力系统韧性评估技术综述[J]. 电网技术, 2024, 48(1): 129-139.
|
| [2] |
|
| [3] |
王乐, 唐捷, 黄远飞, 等. 考虑微地形的配电系统雨涝淹没停电风险区划分方法[J]. 电力科学与技术学报, 2024, 39(4): 84-92.
|
| [4] |
宋梦, 周佳妮, 高赐威, 等. CPSS视角下城市建筑与配电网高韧性协调运行: 研究述评与展望[J]. 电力系统自动化, 2023, 47(23): 105-121.
|
| [5] |
侯慧, 徐海峰, 王少华, 等. 含降雨量修正的台风灾害下输电杆塔数据机理联合故障概率预测[J]. 高电压技术, 2025, 51(4): 1654-1662.
|
| [6] |
舒印彪, 汤涌, 张正陵, 等. 欧洲伊比利亚半岛电网“4·28”大停电事故分析及启示[J]. 中国电机工程学报, 2025, 45(17): 6603-6612.
|
| [7] |
田书欣, 苏鹏斌, 赵昊星, 等. 冰灾场景下考虑无人机应用的综合能源系统韧性提升方法[J]. 浙江电力, 2025, 44(11): 1-13.
|
| [8] |
耿宁, 覃智君, 黄镇. 台风灾害下含多元储能的电-气-热综合能源系统韧性提升策略[J]. 电网与清洁能源, 2025, 41(9): 64-73.
|
| [9] |
袁家海, 张凯, 张健, 等. 极端高温天气下电力系统韧性提升策略[J]. 发电技术, 2025, 46(4): 694-704.
目的 为应对极端高温天气对电力系统供需平衡造成的冲击,提出一种面向中长期的电力系统韧性提升方法。 方法 首先,模拟持续性极端高温场景下“源-荷”耦合出力特性,并利用时序运行模拟模型对系统韧性表现进行初评估;其次,基于运行模拟结果,在电力供需平衡的基础上以供电成本与负荷削减成本之和最小为目标,建立面向韧性提升的电力资源规划模型,得到韧性提升组合措施;最后,对韧性提升策略场景进行运行模拟,以验证其有效性。 结果 以2025年广东省为案例场景,与未实施电力系统韧性提升策略的场景相比,实施策略后极端高温天气下系统累积缺供电量、最大失负荷规模和因电力短缺而造成的经济损失分别降低了86.71%、60.72%和91.55%,系统最小供电水平提升了11.07%。 结论 通过协同供电资源扩容与负荷资源部署,电力系统韧性提升策略有效地提升了极端高温天气下系统的供电能力,同时降低了因电力短缺而造成的经济损失和社会代价。
Objectives To address the impact of extreme high-temperature weather on the supply and demand balance of power systems, a medium- to long-term method for enhancing power system resilience is proposed. Methods First, the coupled “source-load” output characteristics under sustained extreme high-temperature scenarios are simulated, and an initial evaluation of system resilience is performed using a chronological operation simulation model. Second, based on the operation simulation results, a resilience enhancement-oriented power resource planning model is established, with the objective of minimizing the combined cost of power supply and load shedding under the constraint of supply and demand balance, thereby deriving combined measures for resilience enhancement. Finally, operation simulations of the resilience enhancement strategy scenarios are conducted to verify their effectiveness. Results Taking Guangdong Province in 2025 as a case study, compared with the scenario without the implementation of the resilience enhancement strategies for power systems, the application of such strategies under extreme high-temperature weather decreases the cumulative power shortage, maximum lost load scale, and economic losses caused by power shortage by 86.71%, 60.72%, and 91.55%, respectively. The system’s minimum power supply level increases by 11.07%. Conclusions By coordinating the expansion of power supply resources and deployment of load resources, the resilience enhancement strategies effectively improve the power system’s supply capacity under extreme high-temperature weather, while reducing the economic loss and social cost caused by power shortages. |
| [10] |
谭静, 王东, 张英华, 等. 台风灾害下电网多维韧性评估研究[J]. 山东电力技术, 2024, 51(3): 27-35.
|
| [11] |
陈磊, 邓欣怡, 陈红坤, 等. 电力系统韧性评估与提升研究综述[J]. 电力系统保护与控制, 2022, 50(13): 11-22.
|
| [12] |
吴桂联, 朱乃璇, 陈浩, 等. 基于多尺度注意力机制与多资源协同优化的台风灾害下配电网韧性提升方法[J]. 电力科学与技术学报, 2025, 40(6): 90-100.
|
| [13] |
刘自发, 薛文博. 考虑智能软开关及多种应急资源协同的配电网韧性提升策略[J]. 电力科学与技术学报, 2025, 40(3): 1-16.
|
| [14] |
麻灿皓, 陈丽娟, 吴志. 考虑多元灵活性资源协同的配电网韧性提升策略[J]. 电力工程技术, 2025, 44(1): 115-125.
|
| [15] |
许寅, 王颖, 和敬涵, 等. 多源协同的配电网多时段负荷恢复优化决策方法[J]. 电力系统自动化, 2020, 44(2): 123-131.
|
| [16] |
|
| [17] |
高兆丽, 胥明凯, 丁素英, 等. 基于改进人工蜂群算法的配电网多点故障应急抢修优化调度[J]. 电力系统保护与控制, 2019, 47(13): 107-114.
|
| [18] |
郁琛, 李尚轩, 谢云云, 等. 考虑交通网与配电网信息融合的台风后配电网抢修策略优化[J]. 电力系统自动化, 2022, 46(4): 15-24.
|
| [19] |
|
| [20] |
黄婧杰, 聂焕, 张荐, 等. 主动配电网灾后韧性提升的孤岛划分与故障抢修协同方法[J]. 电力自动化设备, 2025, 45(10): 128-134.
|
| [21] |
王涵, 孙磊, 文福拴. 考虑孤岛动态划分与融合的配电系统故障恢复策略[J]. 电力自动化设备, 2025, 45(10): 92-99.
|
| [22] |
徐岩, 郭佳睿, 马天祥. 考虑韧性提升的配电网故障恢复与抢修协调优化[J]. 高电压技术, 2024, 50(12): 5516-5528.
|
| [23] |
徐岩, 张荟, 孙易洲. 基于变异粒子群算法的主动配电网故障恢复策略[J]. 电力自动化设备, 2021, 41(12): 45-53.
|
| [24] |
萧文聪, 陈俊斌, 余涛, 等. 基于专家知识嵌入强化学习的配电系统灾后恢复决策方法[J]. 电力系统自动化, 2025, 49(12): 91-100.
|
| [25] |
江昌旭, 林骏驰, 郭辰, 等. 图强化学习在配电网优化中的应用综述[J]. 电网技术, 2026, 50(3): 922-936.
|
| [26] |
胡维昊, 曹迪, 黄琦, 等. 深度强化学习在配电网优化运行中的应用[J]. 电力系统自动化, 2023, 47(14): 174-191.
|
| [27] |
张沛, 陈玉鑫, 王光华, 等. 基于图强化学习的配电网故障恢复决策[J]. 电力系统自动化, 2024, 48(2): 151-158.
|
| [28] |
何小龙, 高红均, 王仁浚, 等. 基于图深度强化学习的有源配电网故障恢复方法[J]. 电网技术, 2025, 49(10): 4342-4352.
|
| [29] |
江昌旭, 周龙灿, 庄鹏威, 等. 基于数据-模型混合驱动方法的多类型移动应急资源优化调度策略[J]. 电网技术, 2026, 50(2): 858-868.
|
| [30] |
陈逸飞, 郑子萱, 肖先勇, 等. 数据-物理混合驱动的配电网运行韧性评估方法与提升策略[J]. 电力系统保护与控制, 2025, 53(10): 13-22.
|
| [31] |
田启东, 张家琦, 陈颖, 等. 基于强化学习的灾后配电网应急抢修决策方法[J]. 电工电能新技术, 2023, 42(3): 66-75.
灾后故障配电网的应急抢修及重构需求,面临着灾情的不确定性、多抢修队调配特性、抢修与重构耦合等多个挑战。本文基于强化学习构建了灾后多抢修队的抢修调配模型,设计了状态粗筛机制以固定强化学习状态及动作空间大小,搭建了配电网抢修恢复测试环境。在不同恢复模式、多个故障场景下的测试结果表明:强化学习在提出的多个评价指标下综合表现稳定,可以作为灾后动态抢修的可选方案。
The post-disaster emergency repair and reconfiguration plan of the faulty distribution system are confronted with the challenges from the uncertainty of the disaster, the deployment of multiple emergency repair teams and the coupling of emergency repair and reconfiguration. This paper builds a model for emergency repair deployment of multiple repair teams based on reinforcement learning. A state coarse screening mechanism is designed to fix the size of the state and action space of reinforcement learning. An environment based on OpenAI Gym is constructed for the repair and reconfiguration of the distribution system. The experimental results show that reinforcement learning has a stable comprehensive performance under the proposed multiple evaluation indicators, and it can be used as an optional solution for dynamic repair after a disaster.
|
| [32] |
|
| [33] |
赖俊, 魏竞毅, 陈希亮. 分层强化学习综述[J]. 计算机工程与应用, 2021, 57(3): 72-79.
近年来强化学习愈发体现其强大的学习能力,2017年AlphaGo在围棋上击败世界冠军,同时在复杂竞技游戏星际争霸2和DOTA2中人类的顶尖战队也败于AI之手,但其自身又存在着自身的弱点,在不断的发展中瓶颈逐渐出现。分层强化学习因为能够解决其维数灾难问题,使得其在环境更为复杂,动作空间更大的环境中表现出更加优异的处理能力,对其的研究在近几年不断升温。对强化学习的基本理论进行简要介绍,对Option、HAMs、MAXQ这3种经典分层强化学习算法进行介绍,之后对近几年在分层的思想下提出的分层强化学习算法从3个方面进行综述,并对其进行分析,讨论了分层强化学习的发展前景和挑战。
In recent years, reinforcement learning has increasingly reflected its strong learning ability. In 2017, AlphaGo beat the world champion in go. Meanwhile, in the complex competitive games StarCraft 2 and dota2, the top human teams are also defeated by AI. However, it has its own weaknesses, and the bottleneck gradually appears in the continuous development. Hierarchical reinforcement learning can solve the problem of dimension disaster, which makes it show more excellent processing ability in the environment with more complex environment and larger action space. This paper briefly introduces the basic theory of reinforcement learning. It introduces three classical hierarchical reinforcement learning algorithms, option, hams and MAXQ. It summarizes and analyzes the hierarchical reinforcement learning algorithm proposed in recent years under the idea of stratification from three aspects. It discusses the development prospects and challenges of hierarchical reinforcement learning. |
| [34] |
彭佳盛, 文云峰, 梁晓锐, 等. 基于弹性系数的配电网重构与故障抢修协同优化方法[J]. 电力系统自动化, 2025, 49(6): 88-95.
|
| [35] |
|
| [36] |
陈潇瑞, 李晓露, 林顺富. 考虑交通网延误的配电网灾后滚动抢修策略优化[J]. 现代电力, 2023, 40(5): 807-816.
|
| [37] |
|
| [38] |
王宏威, 阮晓波, 李雨龙, 等. 基于日变交通流量降雨损伤LSTM模型的城市路网韧性分析[J]. 中国安全科学学报, 2025, 35(12): 213-220.
为探究常规降雨作用下城市路网性能时变性,建立合理的路网韧性评价体系,首先,构建城市道路路段交通流量降雨损伤的长短期记忆网络(LSTM)模型;其次,定义具有时变特性的路网性能函数;然后,基于韧性理念推导出路网时变韧性计算模型;最后,基于某城市降雨信息与路网交通数据及Sioux Falls网络,分析降雨对路段与路网时变特性的作用效果。结果表明:当降雨持续时间或降雨量增加时,路段工作日交通流量损伤均有增加趋势;路网韧性是各相关路段协同作用的综合体现,因而其对降雨作用呈现出较交通流量损伤更为迟钝的响应,且其值在常规降雨条件下通常不小于0.9;少雨水季节明显强于多雨水季节的路网韧性,而路网年韧性在未受到降雨灾害影响时一般会维持在一个较高水平上,且其在相近年份之间的差异并不显然。
To explore the time-varying performance of urban road networks under regular rainfall and establish a reasonable evaluation system for road network resilience, an LSTM model for the rainfall-induced traffic flow degradation in urban road segments was first constructed, and a network performance function with time-varying characteristics was second defined. Then, based on the resilience concept, a time-varying resilience calculation model for road networks was derived. Finally, the effect of rainfall on the time-varying characteristics of road segments and networks was investigated based on rainfall information and traffic data of a certain city and the Sioux Falls network. The results show that when the duration or amount of rainfall increases, the traffic flow degradation of road segments increases on weekdays. The road network resilience is a comprehensive reflection of the synergistic effect of various related road segments, and therefore its response to rainfall is slower than that of traffic flow degradation, and its value is generally not less than 0.9 under regular rainfall conditions. The road network resilience during low rainfall seasons is significantly stronger than that during high rainfall seasons, and the annual resilience of road networks is usually maintained at a higher level when not affected by rainfall disasters, and the difference between adjacent years is not significant. |
利益冲突声明(Conflict of Interests): 所有作者声明不存在利益冲突。
作者贡献声明(Authors' Contributions): 薛章负责研究设计、资料收集及分析,撰写论文;李晓露负责研究指导,修订论文,审核论文;柳劲松负责研究指导,审核论文;林顺富负责审核论文。所有作者均阅读并同意了论文终稿内容。
/
| 〈 |
|
〉 |