基于Q强化学习的综合能源服务商现货市场申报策略研究

郝旭东; 孙伟; 程定一; 张国强; 匡洪辉

doi:10.12204/j.issn.1000-7229.2020.09.015

PDF(1786 KB)

电力建设 ›› 2020, Vol. 41 ›› Issue (9) : 132-138. DOI: 10.12204/j.issn.1000-7229.2020.09.015

智能电网

基于Q强化学习的综合能源服务商现货市场申报策略研究

郝旭东 ¹ ,
孙伟 ² ,
程定一 ¹ ,
张国强 ³ ,
匡洪辉 ⁴

作者信息 +

A Novel Declaration Strategy for Integrated Energy Servicer Based on Q-Learning Algorithm in Power Spot Market

Author information +

文章历史 +

摘要

随着综合能源系统建设和电力市场改革推进,综合能源服务商有望成为新的市场交易成员。为解决申报阶段有限的决策参考信息制约申报策略制定的问题,文章提出了一种基于Q强化学习的综合能源服务商现货市场申报策略,以提升申报策略的理想度。该方法的主要特点在于充分利用庞大的历史运行信息,通过人工智能算法训练申报策略智能体,建立综合能源服务商所掌握的有限参考信息与最优申报策略之间的内在关系。智能体以市场公开信息、社会公共信息及服务商私有信息为环境变量,能够实现申报策略的自动生成和智能改进。最后,基于某省电网实际数据构造算例表明,该方法能较好地拟合合作博弈下的申报策略,具有收敛速度快、理想度高、计算效率高等特点,更符合综合能源服务商决策需求。

Abstract

With the development of integrated energy system and power market reform, integrated energy servicer is expected to become a new market member in power market transaction. In order to solve the problem that the limited reference information in the declaration stage restricts the formulation of the declaration strategy, a declaration strategy based on Q-learning for integrated energy servicer is proposed to improve the ideal degree of the declaration strategy. The core idea of the proposed strategy is to make full use of the huge historical operation information and train the declaration strategy agent by artificial intelligence algorithms to establish the inherent relationship between the limited reference information grasped by integrated energy servicer during the market bidding process and its optimal declaration strategy. The declaration agent can realize automatic generation and intelligent improvement of declaration policies, which takes energy market public information, social public information and enterprise private information as environment variables. Finally, a case study based on the actual data of a provincial power grid shows that the proposed method can better match the declaration strategy under the cooperative game and has the characteristics of fast convergence, high ideal degree and high computational efficiency, which is more suitable for the actual needs of integrated energy servicer.

导出引用

郝旭东, 孙伟, 程定一, 等. 基于Q强化学习的综合能源服务商现货市场申报策略研究[J]. 电力建设. 2020, 41(9): 132-138 https://doi.org/10.12204/j.issn.1000-7229.2020.09.015

Xudong HAO, Wei SUN, Dingyi CHENG, et al. A Novel Declaration Strategy for Integrated Energy Servicer Based on Q-Learning Algorithm in Power Spot Market[J]. Electric Power Construction. 2020, 41(9): 132-138 https://doi.org/10.12204/j.issn.1000-7229.2020.09.015

中图分类号： TM743

参考文献

列表( 原文顺序 | 文献年度倒序 | 文中引用次数倒序 ) 可视化分析

[1]	韩宇, 彭克, 王敬华 , 等. 多能协同综合能源系统协调控制关键技术研究现状与展望[J]. 电力建设, 2018,39(12):81-87. HAN Yu, PENG Ke, WANG Jinghua , et al. Research status and prospect of key technologies for coordinated control of multi-energy synergic integrated energy systems[J]. Electric Power Construction, 2018,39(12):81-87. 本文引用 [1]

[2]	葛睿, 陈龙翔, 汤俊 , 等. 跨区域省间可再生能源增量现货市场设计与实践[J]. 电力建设, 2019,40(1):11-18. 本文引用 [1]

3	GE Rui, CHEN Longxiang, TANG Jun , et al. Design and practice of cross-regional incremental spot market for renewable energy[J]. Electric Power Construction, 2019,40(1):11-18. 本文引用 [1]

[3]	谈金晶, 李扬 . 多能源协同的交易模式研究综述[J]. 中国电机工程学报, 2019,39(22):6483-6497. TAN Jinjing, LI Yang . Review on transaction mode in multi-energy collaborative market[J]. Proceedings of the CSEE, 2019,39(22):6483-6497. 本文引用 [1]

[4]	戚艳, 刘敦楠, 徐尔丰 , 等. 面向园区能源互联网的综合能源服务关键问题及展望[J]. 电力建设, 2019,40(1):123-132. QI Yan, LIU Dunnan, XU Erfeng , et al. Key issues and prospects of integrated energy service for energy Internet in park[J]. Electric Power Construction, 2019,40(1):123-132. 本文引用 [1]

[5]	WANG D, UNIVERSITY T, LIU L , et al. Review of key problems related to integrated energy distribution systems[J]. CSEE Journal of Power and Energy Systems, 2018,4(2):130-145. 本文引用 [1]

[6]	黄伟, 葛良军, 华亮亮 , 等. 参与双重市场的区域综合能源系统日前优化调度[J]. 电力系统自动化, 2019,43(12):68-82. HUANG Wei, GE Liangjun, HUA Liangliang , et al. Day-ahead optimal scheduling of regional integrated energy system participating in dual market[J]. Automation of Electric Power Systems, 2019,43(12):68-82. 本文引用 [3]

[7]	吴嘉豪, 曾成碧, 苗虹 . 计及子区域间能量交换的多区域综合能源系统协调经济调度[J]. 电力建设, 2019,40(11):39-47. WU Jiahao, ZENG Chengbi, MIAO Hong . Coordinated economic scheduling for multi-regional integrated energy system considering energy exchange between sub-regions[J]. Electric Power Construction, 2019,40(11):39-47. 本文引用 [3]

[8]	刘珮云, 丁涛, 贺元康 , 等. 基于综合需求响应的负荷聚合商最优市场交易策略[J]. 电力自动化设备, 2019,39(8):224-231. LIU Peiyun, DING Tao, HE Yuankang , et al. Optimal trading strategy for load aggregator based on integrated demand response[J]. Electric Power Automation Equipment, 2019,39(8):224-231. 本文引用 [2]

[9]	TAHIR M F, CHEN H Y, MEHMOOD K , et al. Integrated energy system modeling of China for 2020 by incorporating demand response, heat pump and thermal storage[J]. IEEE Access, 2019(7):40095-40108. 本文引用 [1]

[10]	武赓, 武庆国, 王昊婧 , 等. 考虑风险规避的综合能源服务商能源购置策略[J]. 电力自动化设备, 2019,39(6):12-20. WU Geng, WU Qingguo, WANG Haojing , et al. Energy purchasing strategy of multi-energy service provider considering risk aversion[J]. Electric Power Automation Equipment, 2019,39(6):12-20. 本文引用 [1]

[11]	尹龙, 刘继春, 高红均 , 等. 考虑多种用户价格机制下的综合型能源售电公司购电竞价策略[J]. 电网技术, 2018,42(1):88-97. YIN Long, LIU Jichun, GAO Hongjun , et al. Study on bidding strategy of comprehensive power retailer under multiple user-price mechanisms[J]. Power System Technology, 2018,42(1):88-97. 本文引用 [1]

[12]	耿琪, 胡炎, 何建宗 , 等. 基于纳什谈判的区域综合能源系统运行优化[J]. 电力建设, 2020,41(1):114-125. GENG Qi, HU Yan, HE Jianzong , et al. Optimal operation of integrated community energy system based on Nash bargaining[J]. Electric Power Construction, 2020,41(1):114-125. 本文引用 [3]

[13]	练依情, 郭祚刚, 喻磊 , 等. 市场出清机制下综合能源服务商竞争策略[J]. 南方电网技术, 2019,13(7):10-16, 42. LIAN Yiqing, GUO Zuogang, YU Lei , et al. Competition strategy of integrated energy service provider under pool-based market mechanism[J]. Southern Power System Technology, 2019,13(7):10-16, 42. 本文引用 [3]

[14]	郭祚刚, 喻磊, 胡洋 , 等. 基于合作博弈的综合能源服务商现货市场风险规避策略[J]. 中国电力, 2019,52(11):28-34. GUO Zuogang, YU Lei, HU Yang , et al. Research on competition strategy of integrated energy service provider under the pool-based market mechanism[J]. Electric Power, 2019,52(11):28-34. 本文引用 [2]

[15]	LIU N, WANG J, WANG L F . Hybrid energy sharing for multiple microgrids in an integrated heat-electricity energy system[J]. IEEE Transactions on Sustainable Energy, 2019,10(3):1139-1151. 本文引用 [1]

[16]	赵晓东, 杜小泽, 曾鸣 . 商业综合能源系统经济性分析[J]. 电力需求侧管理, 2019,21(4):87-92. ZHAO Xiaodong, DU Xiaoze, ZENG Ming . The influence of market electricity price signal on the economy of integrated energy system[J]. Power Demand Side Management, 2019,21(4):87-92. 本文引用 [1]

[17]	马辉, 陈雨果, 陈晔 , 等. 南方(以广东起步)电力现货市场机制设计[J]. 南方电网技术, 2018,12(12):42-48. MA Hui, CHEN Yuguo, CHEN Ye , et al. Mechanism design of Southern China(starting from Guangdong Province)electric spot market[J]. Southern Power System Technology, 2018,12(12):42-48. 本文引用 [1]

[18]	HWANG K S, JIANG W C, CHEN Y J , et al. Model learning for multistep backward prediction in dyna- mic Q-learning[J]. IEEE Transactions on Systems, Man, and Cybernetics: Systems, 2018,48(9):1470-1481. 本文引用 [2]

[19]	SPANO S, CARDARILLI G C, DI NUNZIO L , et al. An efficient hardware implementation of reinforcement learning: The Q-learning algorithm[J]. IEEE Access, 2019(7):186340-186351. 本文引用 [1]

[20]	DENG Z, GUAN H J, HUANG R , et al. Combining model-based Q-learning with structural knowledge transfer for robot skill learning[J]. IEEE Transactions on Cognitive and Developmental Systems, 2019,11(1):26-35. 本文引用 [1]