Skip to the content

A new unsupervised feature selection method for text clustering based on genetic algorithms

Shamsinejadbabki, P and Saraee, M 2011, 'A new unsupervised feature selection method for text clustering based on genetic algorithms' , Journal of Intelligent Information Systems .

[img]
Preview
PDF - Accepted Version
Download (1585kB) | Preview

    Abstract

    Nowadays a vast amount of textual information is collected and stored in various databases around the world, including the Internet as the largest database of all. This rapidly increasing growth of published text means that even the most avid reader cannot hope to keep up with all the reading in a field and consequently the nuggets of insight or new knowledge are at risk of languishing undiscovered in the literature. Text mining offers a solution to this problem by replacing or supplementing the human reader with automatic systems undeterred by the text explosion. It involves analyzing a large collection of documents to discover previously unknown information. Text clustering is one of the most important areas in text mining, which includes text preprocessing, dimension reduction by selecting some terms (features) and finally clustering using selected terms. Feature selection appears to be the most important step in the process. Conventional unsupervised feature selection methods define a measure of the discriminating power of terms to select proper terms from corpus. However up to now the valuation of terms in groups has not been investigated in reported works. In this paper a new and robust unsupervised feature selection approach is proposed that evaluates terms in groups. In addition a new Modified Term Variance measuring method is proposed for evaluating groups of terms. Furthermore a genetic based algorithm is designed and implemented for finding the most valuable groups of terms based on the new measure. These terms then will be utilized to generate the final feature vector for the clustering process . In order to evaluate and justify our approach the proposed method and also a conventional term variance method are implemented and tested using corpus collection Reuters-21578. For a more accurate comparison, methods have been tested on three corpuses and for each corpus clustering task has been done ten times and results are averaged. Results of comparing these two methods are very promising and show that our method produces better average accuracy and F1-measure than the conventional term variance method.

    Item Type: Article
    Uncontrolled Keywords: Text clustering, unsupervised feature selection, genetic algorithm
    Themes: Media, Digital Technology and the Creative Economy
    Memory, Text and Place
    Schools: Colleges and Schools > College of Science & Technology > School of Computing, Science and Engineering > Data Mining and Pattern Recognition Research Centre
    Journal or Publication Title: Journal of Intelligent Information Systems
    Publisher: Kluwer
    Refereed: Yes
    ISSN: 0925-9902
    Depositing User: Dr Mo Saraee
    Date Deposited: 19 Oct 2011 11:34
    Last Modified: 31 Jul 2014 10:39
    URI: http://usir.salford.ac.uk/id/eprint/18510

    Actions (login required)

    Edit record (repository staff only)

    Downloads per month over past year

    View more statistics