售电侧市场放开是我国新一轮电力体制改革的重点任务之一。文章考虑售电侧放开对需求侧资源参与电力市场的影响，提出了售电侧放开环境下计及需求响应的虚拟电厂（virtual power plant, VPP）竞价策略，建立了含虚拟电厂的电力市场竞价模型，在对虚拟电厂内部进行优化调度的基础上采用自适应学习方法对竞价模型进行求解，并以社会效益最大化为目标进行市场出清，最后选取算例对竞价策略的有效性进行验证。结论表明，售电侧放开环境下发电商与售电商的经济性有所提高且市场整体的用电需求得到调整，同时售电侧放开有利于需求侧资源参与市场竞争。

The opening of the retail market is one of the key tasks of the power system reform in China. In this paper, under retail power market deregulation, we consider the influence on demand-side resource participating in electric market, propose a bidding strategy of virtual power plant considering demand response, and set up a bidding model for the whole market including virtual power plant （VPP）. On the basis of the optimization of the virtual power plant, we use the adaptive learning method to solve the model, and clear the market with the goal of maximizing the social benefit. Finally, an example is selected to verify the validity of the bidding strategy, and the results show that the economy of the generation companies and retailing suppliers is improved and the demand for electricity in the whole market is adjusted. What's more, retail power market deregulation is benefit to demand-side resources participating in market competition.

多园区综合能源系统可通过多能互补互济显著提升运行经济性,然而园区之间的复杂互动、多能耦合决策会给多园区综合能源系统的能量管理带来决策空间庞大、算法难以收敛等挑战性问题。为解决上述问题,提出了一种基于改进深度Q网络(modified deep Q network,MDQN)算法的多园区综合能源系统能量管理方法。首先,采用独立于园区的外部气象数据、历史互动功率数据,构建了基于长短期记忆(long short-term memory,LSTM)深度网络的各园区综合能源系统外部互动环境等值模型,降低了强化学习奖励函数的计算复杂度;其次,提出一种基于k优先采样策略的MDQN算法,用k-优先采样策略来代替ε贪心策略,克服了大规模动作空间中探索效率低下的问题;最后,在含3个园区综合能源系统的算例中进行验证,结果表明MDQN算法相比原DQN算法具有更好的收敛性与稳定性,同时可以提升园区经济效益达29.16%。

Multi-park integrated energy system can significantly improve the operation economy by complementing each other with multiple energy sources. However, the complex interactions between parks and multi-energy coupling decisions can bring challenging problems such as large decision space and difficult convergence of algorithms to the energy management of multi-park integrated energy system. To solve the above problems, an energy management method based on modified deep Q network (MDQN) algorithm for multi-park integrated energy systems is proposed. Firstly, the external meteorological data and historical interactive power data independent of the park are used to construct a long short-term memory (LSTM) deep network-based external interactive environmental equivalence model for each park integrated energy system, which reduces the computational complexity of the reinforcement learning reward function. Secondly, an improved DQN algorithm based on k-first sampling strategy is proposed to replace the greedy strategy with k-first sampling strategy to overcome the inefficiency of exploration in large-scale action spaces. Finally, the results are validated in an algorithm containing three integrated energy systems in the park, and show that the MDQN algorithm has better convergence and stability compared with the original DQN algorithm, while it can improve the economic efficiency of the park by 29.16%.

智能电网是人工智能 （artificial intelligence, AI） 的重要应用领域之一, 以高级机器学习理论、大数据、云计算为主要代表的新一代人工智能 （new generation artificial intelligence, NGAI） 技术的进步和突破, 将会促进智能电网的发展。首先概述AI的主要方法, 并对NGAI的内涵、特点与技术体系进行论述。之后, 对NGAI在能源供应、电力系统安全与控制、运维与故障诊断、电力需求和电力市场等领域中的最新应用研究情况进行比较系统的综述。最后, 总结NGAI在智能电网中应用的关键问题, 提出人工智能在智能电网中的应用可分为三阶段实施的建议。

Smart grid is an important application field of artificial intelligence （AI）. The progress and breakthrough of new generation artificial intelligence （NGAI） technologies, mainly represented by advanced machine learning, big data, and cloud computing, will provide incentives for the development of smart grids. Dominated methods of AI are first introduced, and the connotation, characteristics and technical system of NGAI briefly described. Next, the recent applications of NGAI in some fields including energy supply, power system security and control, operation maintenance and fault diagnosis, power demand and electricity market are surveyed in detail. Finally, some key issues concerning the applications of NGAI in smart grids are summarized, and a proposal is suggested that the applications of AI in smart grids be implemented through three stages.

为解决微电网在传统集中化交易模式下面临的决策耗时长、信任成本高和隐私安全等问题,提出了基于多智能体深度确定性策略梯度(multi-agent deep deterministic policy gradient, MADDPG)算法与智能合约的微电网去中心化市场交易体系。首先,对微电网市场中多智能体进行划分后设计了适用于各主体参与分布式交易的微电网去中心化交易机制,以保障市场主体利益。其次,为实现交易确认阶段微电网市场主体的交易策略优化,采用MADDPG算法对各主体追求利益最大的竞价模型进行求解。最后,通过算例仿真验证了MADDPG算法在智能合约下微电网市场主体交易策略优化过程中的可行性和经济性。

In order to solve the problems of long decision-making time, high cost of trust and privacy security faced by microgrid in the traditional centralized trading, a decentralized market trading system based on multi-agent deep deterministic policy gradient (MADDPG) algorithm and smart contract is proposed for microgrid. Firstly, after dividing the multiple agents in the microgrid market, a decentralized transaction mechanism for microgrids that is suitable for all entities to participate in distributed transactions is designed to protect the interests of market entities. Secondly, in order to realize the optimization of the transaction strategy of the microgrid market entities in the transaction confirmation stage, the multi-agent deep deterministic policy gradient algorithm is used to solve the bidding model that each entity pursues the most benefits. Finally, the feasibility and economy of the MADDPG algorithm in the optimization process of the transaction strategy of the microgrid market entities under the smart contract is verified by example simulation.

Reinforcement learning (RL) algorithms have been around for decades and employed to solve various sequential decision-making problems. These algorithms, however, have faced great challenges when dealing with high-dimensional environments. The recent development of deep learning has enabled RL methods to drive optimal policies for sophisticated and capable agents, which can perform efficiently in these challenging environments. This article addresses an important aspect of deep RL related to situations that require multiple agents to communicate and cooperate to solve complex tasks. A survey of different approaches to problems related to multiagent deep RL (MADRL) is presented, including nonstationarity, partial observability, continuous state and action spaces, multiagent training schemes, and multiagent transfer learning. The merits and demerits of the reviewed methods will be analyzed and discussed with their corresponding applications explored. It is envisaged that this review provides insights about various MADRL methods and can lead to the future development of more robust and highly useful multiagent learning methods for solving real-world problems.