Comprehensive Evaluation of Big Data Quality in Power Systems with Entropy Weight and Grey System Theory

doi:10.3969/j.issn.1000－7229.2016.12.003

Abstract

Abstract: With the continuous expansion of power systems, as well as ever-developing technology and reduced costs of measurement devices, the recorded data in power systems have been increasing significantly and progressively exhibit the feature of big data. Much attention has been paid to the full use of big data for improving the planning, operation and control of power system, and hence how to evaluate the quality of big data is becoming an important problem to be examined. Some research publications are available on data quality improvement, such as data cleaning, data integration, and the detection of similar records, but the existing research work is still preliminary in data quality evaluations. Given this background, considering the characteristics of power systems and associated big data, this paper proposes a comprehensive method for evaluating the quality of big data in power systems. Firstly, we construct an index system for big data quality evaluations. Then aiming at the timeliness of big data, we adopt the K-means clustering algorithm in parallel with MapReduce for fast preprocessing of the big data sample set. Secondly, we use entropy weight method to calculate the objective weight of each dataset and grey evaluation method to determine the data quality level. On this basis, the comprehensive evaluation of the sample data set is carried out. Finally, the recorded electric load historical data in a city power company are employed to demonstrate the proposed method.

Key words: power system, big data, data quality, evaluation index, K-means clustering, entropy weight method, grey theory

CLC Number:

TM 72

LI Gang, JIAO Yafei, LIU Fuyan, YU Min, SONG Yu, WEN Fushuan.

Comprehensive Evaluation of Big Data Quality in Power Systems with Entropy Weight and Grey System Theory

[J]. Electric Power Construction, 2016, 37(12): 24-.

References

［1］宋亚奇，周国亮，朱永利.智能电网大数据处理技术现状与挑战［J］.电网技术，2013，37(4)：927-935.

SONG Yaqi，ZHOU Guoliang，ZHU Yongli.Present status and challenges of big data processing in smart grid ［J］.Power System Technology，2013，37(4)：927-935.

［2］高峰，刘广一，SAUNDERS Chris，等.智能电网大数据的分析与应用［J］.电力建设，2015，36(10)：11-19.

GAO Feng，LIU Guangyi，SAUNDERS Chris，et al.Applications of smart grid big data analytics ［J］.Electric Power Construction，2015，36(10)：11-19.

［3］李刚，杨立业，刘福炎，等.能源互联网关联数据融合的互信息方法［J］.电力建设，2016，37(9)：22-29.

LI Gang，YANG Liye，LIU Fuyan，et al.A mutual information method for associated data fusion in energy internet［J］.Electric Power Construction，2016，37(9)：22-29.

［4］章元德，史亮，陆巍，等.线损信息化统计中数据质量管控机制及实现［J］.电力系统自动化，2016，40(7)：128-133.

ZHANG Yuande，SHI Liang，LU Wei，et al.Realization of data management and control mechanism of line loss statistical information ［J］.Automation of Electric Power Systems，2016，40(7)：128-133.

［5］刘世成，张东霞，朱朝阳，等.能源互联网中大数据技术思考［J］.电力系统自动化，2016，40(8)：14-23.

LIU Shicheng，ZHANG Dongxia，ZHU Zhaoyang，et al.A view on big data in energy internet ［J］.Automation of Electric Power Systems，2016，40(8)：14-23.

［6］孙世国，黄志敏，叶尚兴，等.基于分布式光纤的电力电缆检测数据质量优化技术［J］.电力建设，2014，35(9)：53-58.

SUN Shiguo，HUANG Zhimin，YE Shangxing，et al.Optimization technology of detection data quality for power cable based on distributed optical fiber ［J］.Electric Power Construction，2014，35(9)：53-58.

［7］KANDEL S，PARIKH R，PAEPCKE A，et al.Profiler：integrated statistical analysis and visualization for data quality assessment［C］// Proceedings of the International Working Conference on Advanced Visual Interfaces，Capri Island，Italy，2012：547-554.

［8］MAMATJAN Y，GRYCHTOL B，GAGGERO P，et al.Evaluation and real-time monitoring of data quality in electrical impedance tomography［J］.IEEE Transactions on Medical Imaging，2013，32(11)：1997-2005.

［9］张亮.电力调度数据中心数据质量问题研究［J］.华东电力，2009，37(3)：403-406.

ZHANG Liang.Research on data quality of electric power dispatching data center ［J］.East China Electric Power，2009，37(3)：403-406.

［10］南方电网报.南网公司研发完成营销系统数据质量核查工具［EB/OL］.（2012-09-03）［2016-11-15］. http://www.cec.org.cn/zdlhuiyuandongtai/dianwang/2012-09-03/90195.html.

［11］陈建明，韩建民.面向微聚集技术的k-匿名数据质量评估模型［J］.计算机应用研究，2010，27(6)：2344-2347.

CHEN Jianming，HAN Jianmin.Evaluation model for quality of k-anonymity data oriented to micro aggregation ［J］.Application Research of Computers，2010，27(6)：2344-2347.

［12］许涤龙，叶少波.统计数据质量评估方法研究述评［J］.统计与信息论坛，2011，26(7)：3-14.

XU Dilong，YE Shaobo.Reviews on assessment methods quality of statistics ［J］.Statistics and Information Forum，2011，26(7)：3-14.

［13］成邦文，石林芬，杨宏进.统计数据质量检查与异常点识别的模型与方法［J］.系统工程，2001，19(3)：85-89.

CHENG Bangwen，SHI Linfen，YANG Hongjin.A model and a method for evaluating the quality of statistical data and identifying outliers from the data ［J］.Systems Engineering，2001，19(3)：85-89.

［14］胡桔州，兰秋军.金融时间序列的数据挖掘技术与经典统计模型的比较［J］.系统工程，2005，23(6)：95-98.

HU Juzhou，LAN Qiujun.Comparing between data mining and classic statistical model in financial time series ［J］.Systems Engineering，2005，23(6)：95-98.

［15］CABALLERO I，SERRANO M，PIATTINI M.A data quality in use model for big data ［J］.Future Generation Computer Systems，2015，51(1)：65-74.

［16］彭小圣，邓迪元，程时杰，等.面向智能电网应用的电力大数据关键技术［J］.中国电机工程学报，2015，35(3)：503-511.

PENG Xiaosheng，DENG Diyuan，CHENG Shijie，et al.Key technologies of electric power big data and its application prospects in smart grid ［J］.Proceedings of the CSEE，2015，35(3)：503-511.

［17］张沛，吴潇雨，和敬涵.大数据技术在主动配电网中的应用综述［J］.电力建设，2015，36(1)：52-59.

ZHANG Pei，WU Xiaoyu，HE Jinghan.Review on big data technology applied in active distribution network ［J］.Electric Power Construction，2015，36(1)：52-59.［18］江小平，李成华，向文，等.K-means聚类算法的MapReduce并行化实现［J］.华中科技大学学报(自然科学版)，2011，39(S1)：120-124.

JIANG Xiaoping，LI Chunghwa，XIANG Wen，et al.Parallel implementation K-means clustering algorithm using MapReduce programming mode ［J］.Journal of Huazhong University of Science and Technology (Natural Science Edition)，2011，39(S1)：120-124.

[1]	YUAN Lüzerui, GU Jie,JIN Zhijian. User-Side Data Application Framework Based on Cloud-Edge-User Collaboration in Power Internet of Things [J]. Electric Power Construction, 2020, 41(7): 1-8.
[2]	FU Zhixin， LI Xiaoyi， YUAN Yue. Research on Key Technologies of Ubiquitous Power Internet of Things [J]. Electric Power Construction, 2019, 40(5): 1-12.
[3]	YIN Qukai， MI Zengqiang， JIA Yulong， FAN Hui. Economy Regulation Method for Distributed Energy Storage in Distribution Network According to K-means Clustering [J]. Electric Power Construction, 2019, 40(5): 20-27.
[4]	CAI Pengfei，YANG Xiu，LI Taijie，FANG Chen，ZHANG Yong. Electricity Consumption Behavior Analysis of a Large Office Building Based on Improved Cluster Ensemble Algorithm [J]. Electric Power Construction, 2019, 40(1): 60-67.
[5]	SHENG Meng，ZHANG Shuya. Study on the Optimal Replacement Time of Physical Assets in Power Grid on the Basis of Grey Theory [J]. Electric Power Construction, 2018, 39(6): 131-.
[6]	DAI Yan, WANG Liuwang, LI Yuan, YAN Yong, HAN Jiajia, WEN Fushuan. A Brief Survey on Applications of New Generation Artificial Intelligence in Smart Grids [J]. Electric Power Construction, 2018, 39(10): 1-11.
[7]	WENG Yixuan, ZHAO Jie, CHENG Renli, LIU Dichen, XU Junru, ZHANG Shengfeng, HE Xiaofeng. Network-Reconfiguration Scheme Based on Node Search [J]. Electric Power Construction, 2018, 39(1): 76-.
[8]	LIU Dunnan, LI Qi, QIN Lijuan, ZHAO Jiawei. Evaluation of Grid Accepting Renewable Energy in Multi-Time Scale [J]. Electric Power Construction, 2017, 38(7): 44-.
[9]	ZHANG Pei，HE Yi，ZHANG Dahai，SUN Yixin，CHENG Jiaxu. Judgment Rules of Big Data Application in Power Industry [J]. Electric Power Construction, 2017, 38(5): 85-.
[10]	LI Li, ZHU Yongli, SONG Yaqi . Stream Computing and Dynamic Visualization for Electric Power Equipment Monitoring Data [J]. Electric Power Construction, 2017, 38(5): 91-.
[11]	DONG Zhaoyang, CHEN Yingying, LUO Fengji. Innovative Data-Driven Applications in Future Active Distribution Network: Technologies, Prospect and Challenges [J]. ELECTRIC POWER CONSTRUCTION, 2017, 38(5): 2-.
[12]	ZHAO Dongyuan, GAO Feng. Impact of Energy Internet on Power Demand Side Management Evolution [J]. Electric Power Construction, 2017, 38(5): 11-.
[13]	YANG Yinguo,　TAN Yan, LIN Zhenzhi, WEN Fushuan. Black-Start Zoning Strategy Based Semi-Supervised Spectral Clustering Algorithm [J]. Electric Power Construction, 2017, 38(4): 9-.
[14]	QIAN Feng, FENG Changsen, WEN Fushuan, LI Li, TAN Yan, LIANG Yongqing. Robust Optimization Based Security-Constrained Unit Commitment [J]. Electric Power Construction, 2017, 38(4): 18-.
[15]	LAN Xiaoming, WANG Ying, ZHAO Hongshan, MI Zengqiang. Nonlinear Voltage Prediction Control of Power System Based on Dynamic Reduced Model [J]. Electric Power Construction, 2017, 38(3): 34-.