基于MADDPG和智能合约的微电网交易决策优化

doi:10.12204/j.issn.1000-7229.2022.11.014

摘要/Abstract

摘要：

为解决微电网在传统集中化交易模式下面临的决策耗时长、信任成本高和隐私安全等问题,提出了基于多智能体深度确定性策略梯度(multi-agent deep deterministic policy gradient, MADDPG)算法与智能合约的微电网去中心化市场交易体系。首先,对微电网市场中多智能体进行划分后设计了适用于各主体参与分布式交易的微电网去中心化交易机制,以保障市场主体利益。其次,为实现交易确认阶段微电网市场主体的交易策略优化,采用MADDPG算法对各主体追求利益最大的竞价模型进行求解。最后,通过算例仿真验证了MADDPG算法在智能合约下微电网市场主体交易策略优化过程中的可行性和经济性。

关键词: 智能合约, 微电网, 多智能体, 深度强化学习, 博弈论

Abstract:

In order to solve the problems of long decision-making time, high cost of trust and privacy security faced by microgrid in the traditional centralized trading, a decentralized market trading system based on multi-agent deep deterministic policy gradient (MADDPG) algorithm and smart contract is proposed for microgrid. Firstly, after dividing the multiple agents in the microgrid market, a decentralized transaction mechanism for microgrids that is suitable for all entities to participate in distributed transactions is designed to protect the interests of market entities. Secondly, in order to realize the optimization of the transaction strategy of the microgrid market entities in the transaction confirmation stage, the multi-agent deep deterministic policy gradient algorithm is used to solve the bidding model that each entity pursues the most benefits. Finally, the feasibility and economy of the MADDPG algorithm in the optimization process of the transaction strategy of the microgrid market entities under the smart contract is verified by example simulation.

Key words: smart contract, microgrid, multi-agent, deep reinforcement learning, game theory

中图分类号:

TM73

谢昕怡, 应黎明, 田书圣, 朱贵琪. 基于MADDPG和智能合约的微电网交易决策优化[J]. 电力建设, 2022, 43(11): 142-150.

XIE Xinyi, YING Liming, TIAN Shusheng, ZHU Guiqi. Optimization of Microgrid Trading Strategy Based on MADDPG and Smart Contracts[J]. ELECTRIC POWER CONSTRUCTION, 2022, 43(11): 142-150.

图/表 9

图1 微电网市场的多智能体结构

Fig.1 The multi-agent structure of the microgrid market

图2 智能合约下的微电网交易流程

Fig.2 The process of microgrid transaction under smart contract

表1 微电网相关参数

Table 1 Related parameters of the microgrid

表2 MADDPG 算法中的网络结构参数

Table 2 Parameters of network structure in MADDPG algorithm

表3 不同场景下各微网制定的价格策略及收益

Table 3 The price strategy and revenue of each microgrid in different scenarios

图3 场景3中训练累计奖励值变化曲线

Fig.3 The curve of training cumulative reward value in scenario 3

图4 场景4中训练累计奖励值变化曲线

Fig.4 The curve of training cumulative reward value in scenario 4

图5 微网1平均累计奖励值变化情况

Fig.5 Changes in the average cumulative reward value of microgrid 1

图6 DDPG算法下训练累计奖励值变化曲线

Fig.6 The curve of training cumulative reward value under DDPG algorithm

参考文献 21

[1]	MENGELKAMP E, STAUDT P, GARTTNER J, et al. Trading on local energy markets: a comparison of market designs and bidding strategies[C]// 2017 14th International Conference on the European Energy Market (EEM). Dresden, Germany: IEEE, 2017: 1-6.
[2]	王蓓蓓, 李雅超, 赵盛楠, 等. 基于区块链的分布式能源交易关键技术[J]. 电力系统自动化, 2019, 43(14): 53-64.
	WANG Beibei, LI Yachao, ZHAO Shengnan, et al. Key technologies on blockchain based distributed energy transaction[J]. Automation of Electric Power Systems, 2019, 43(14): 53-64.
[3]	沈俭荣, 文云峰, 郭创新, 等. 基于产消方式的互联微网协同自治运行策略[J]. 电力系统自动化, 2016, 40(9): 40-47.
	SHEN Jianrong, WEN Yunfeng, GUO Chuangxin, et al. Prosumer-based autonomous-synergetic operation strategy for interconnected microgrids[J]. Automation of Electric Power Systems, 2016, 40(9): 40-47.
[4]	王丹, 苏朋飞, 桂勋, 等. 售电侧市场开放环境下微网端对端电能交易关键技术综述及展望![J]. 电力建设, 2019, 40(1): 112-122. doi: 10.3969/j.issn.1000-7229.2019.01.014
	WANG Dan, SU Pengfei, GUI Xun, et al. Overview and prospect of key technologies of peer-to-peer energy trading in micro-grid under power-sales-side market liberalization[J]. Electric Power Construction, 2019, 40(1): 112-122. doi: 10.3969/j.issn.1000-7229.2019.01.014
[5]	林俐, 许冰倩, 王皓怀. 典型分布式发电市场化交易机制分析与建议[J]. 电力系统自动化, 2019, 43(4): 1-8.
	LIN Li, XU Bingqian, WANG Haohuai. Analysis and recommendations of typical market-based distributed generation trading mechanisms[J]. Automation of Electric Power Systems, 2019, 43(4): 1-8.
[6]	秦金磊, 孙文强, 朱有产, 等. 微电网中基于区块链的电能交易方法[J]. 电力自动化设备, 2020, 40(11): 130-138.
	QIN Jinlei, SUN Wenqiang, ZHU Youchan, et al. Energy transaction method of microgrid based on blockchain[J]. Electric Power Automation Equipment, 2020, 40(11): 130-138.
[7]	平健, 陈思捷, 张宁, 等. 基于智能合约的配电网去中心化交易机制[J]. 中国电机工程学报, 2017, 37(13): 3682-3690.
	PING Jian, CHEN Sijie, ZHANG Ning, et al. Decentralized transactive mechanism in distribution network based on smart contract[J]. Proceedings of the CSEE, 2017, 37(13): 3682-3690.
[8]	刘杨, 刘天羽. 基于区块链和动态定价模型的微电网P2P能源交易[J]. 智慧电力, 2022, 50(3): 30-36.
	LIU Yang, LIU Tianyu. P2P energy trading in microgrid based on blockchain and dynamic pricing model[J]. Smart Power, 2022, 50(3): 30-36.
[9]	于韶源, 杨胜春, 李亚平, 等. 基于区块链智能合约的分布式发电市场化交易机制研究[J]. 智慧电力, 2018, 46(10): 43-48.
	YU Shaoyuan, YANG Shengchun, LI Yaping, et al. Research on distributed generation marketization-oriented electricity transaction mechanism based on blockchain smart contract[J]. Smart Power, 2018, 46(10): 43-48.
[10]	单俊嘉, 董子明, 胡俊杰, 等. 基于区块链技术的产消者P2P电能智能交易合约[J]. 电网技术, 2021, 45(10): 3830-3839.
	SHAN Junjia, DONG Ziming, HU Junjie, et al. P2P smart power trading contract based on blockchain technology[J]. Power System Technology, 2021, 45(10): 3830-3839.
[11]	刘维扬, 王冰, 王敏, 等. 智能合约技术下电动汽车入网竞价机制研究[J]. 电网技术, 2019, 43(12): 4344-4352.
	LIU Weiyang, WANG Bing, WANG Min, et al. Research on electric vehicles participating in bidding mechanism of power grid based on smart contract technology[J]. Power System Technology, 2019, 43(12): 4344-4352.
[12]	王毅, 赵辉辉, 侯兴哲, 等. 基于合约订单的微电网分布式电能交易模型[J]. 电网技术, 2020, 44(10): 3900-3907.
	WANG Yi, ZHAO Huihui, HOU Xingzhe, et al. Distributed electricity trading model in microgrid based on contract orders[J]. Power System Technology, 2020, 44(10): 3900-3907.
[13]	WU Q, XIE Z, LI Q F, et al. Economic optimization method of multi-stakeholder in a multi-microgrid system based on Stackelberg game theory[J]. Energy Reports, 2022, 8: 345-351.
[14]	金涛, 朱莉, 陈博文, 等. 基于合作博弈的微电网群运行优化方法研究[J]. 智慧电力, 2022, 50(3): 15-21, 29.
	JIN Tao, ZHU Li, CHEN Bowen, et al. Operation optimization method of microgrid group based on cooperative game[J]. Smart Power, 2022, 50(3): 15-21, 29.
[15]	HUANG Q H, HUANG R K, HAO W T, et al. Adaptive power system emergency control using deep reinforcement learning[J]. IEEE Transactions on Smart Grid, 2020, 11(2): 1171-1182. doi: 10.1109/TSG.2019.2933191 URL
[16]	LILLICRAP T P, HUNT J J, PRITZEL A, et al. Continuous control with deep reinforcement learning[C]// Proceedings of the 4th International Conference on Learning Representations, 2016.
[17]	WANG L, WANG K Z, PAN C H, et al. Multi-agent deep reinforcement learning-based trajectory planning for multi-UAV assisted mobile edge computing[J]. IEEE Transactions on Cognitive Communications and Networking, 2021, 7(1): 73-84. doi: 10.1109/TCCN.2020.3027695 URL
[18]	李军祥, 王宇倩, 何建佳, 等. 基于区块链的微电网电力市场电价与电量动态博弈[J]. 电力系统自动化, 2021, 45(17): 11-19.
	LI Junxiang, WANG Yuqian, HE Jianjia, et al. Blockchain-based dynamic game of electricity price and power for microgrid electricity market[J]. Automation of Electric Power Systems, 2021, 45(17): 11-19.
[19]	窦春霞, 罗维, 岳东, 等. 基于多智能体的微网群内电力市场交易策略[J]. 电网技术, 2019, 43(5): 1735-1744.
	DOU Chunxia, LUO Wei, YUE Dong, et al. Multi-agent system based electricity market trading strategy within microgrid groups[J]. Power System Technology, 2019, 43(5): 1735-1744.
[20]	王林, 李晨, 刘嘉佳, 等. 基于复式竞价撮合的电力市场交易模式设计与实践[J]. 电力系统自动化, 2018, 42(24): 188-195.
	WANG Lin, LI Chen, LIU Jiajia, et al. Design and practice of electricity market trading mode based on compound bidding matchmaking[J]. Automation of Electric Power Systems, 2018, 42(24): 188-195.
[21]	郝翠甜. 电力市场中不同竞价撮合机制的优化模型研究[D]. 北京: 华北电力大学, 2019.
	HAO Cuitian. Optimization model research on different matchmaking trade off competition mechanism in electricity power market[D]. Beijing: North China Electric Power University, 2019.