一种K-means改进算法的并行化实现与应用

The Parallel Implementation and Application of an Improved K-means Algorithm

  • 摘要: 随着数据的爆炸式增长,聚类研究作为大数据的核心问题之一,正面临计算复杂度高和计算能力不足等诸多问题。提出了一种基于Hadoop的分布式改进K-means算法,该算法通过引入Canopy算法初始化K-means算法的聚类中心,克服传统K-means算法因初始中心点的不确定性,易陷入局部最优解的问题。本算法在Canopy(罩盖)中完成K-means聚类,并在Canopy间完成簇的合并,聚类效果稳定,迭代次数少。同时,结合MapReduce分布式计算模型,给出改进后算法的并行化设计方法和策略,进一步通过改进相似度度量方法,将该方法用于文本聚类中。实验结果证明该算法具有良好的准确率和扩展性。

     

    Abstract: Following with the growth of massive data, clustering research, one of the core problems of big dataisfaced with more and more problems such as high computing complexity and lack of resource. It has proposed an improved parallel K-means algorithm based on Hadoop. To overcomethe problem that the traditional K-means algorithm often has local optimal solution due to the randomness choice of initial center, we introduce Canopy algorithm to initialize clustering center andapply K-means algorithm on canopy. Meanwhile, clusters are merged among canopies. The result is stable and iteration number is less. In addition, the parallel implementation methods and strategies of the improved algorithm are presented, combining with the distributed computing model of MapReduce. And a new method of text clustering is introduced by improving the similarity of measurement. The experiment results indicate the validity and scalability of our method.

     

/

返回文章
返回