The clustering and fuzzy logic methods complex for Big Data processing

Abstract

Currently, telecom operators are facing a problem that is conditionally called "Big Data". The telecom industry is growing rapidly and dynamically, new technologies are emerging (IoT, M2M, D2D, P2P), new companies are using them, new information and communication services are being introduced to automate production processes, and so on. Methods of statistical analysis, A\B testing, data fusion and integration, Data Mining, machine learning, data visualization are used in the Big Data processing and analysis, but due to the fact that large amounts of Big Data are not structured, come in real-time with various delays related to bandwidth and network congestion, in each case the processes of processing and analysis of Big Data are extremely costly in terms of time and resources. As a result, telecom operators need not only to process large amounts of data but also to extract knowledge from them. However, the analytical processing of large data is characterized by blurred boundaries, which determine certain logical relationships between data. This study proposes the flexible complex of clustering and fuzzy logic methods for big data processing, which increases the speed and reliability of their processing in network nodes, as well as an architectural solution for analysis and processing Big Data realization using micro-services, which increases system scalability and reduces the load on the servers that process them. Experimental studies have confirmed the effectiveness of the proposed modifications. Studies of the K-means algorithm when processing 1500 rows in 3 columns showed decreasing in execution time by 2 seconds. Studies of the Fuzzy C-means algorithm have shown a reduction in execution time by almost 2 times. The validity of the developed fuzzy knowledge base for the K-means and fuzzy C-means algorithms increased by 9% and 4%, respectively.