SVM Aggregating Intelligence SVM, SVM Ensemble, SVM Classification Tree, and Evolving SVM C

SVM Aggregating Intelligence: SVM, SVM Ensemble, SVM Classification Tree, and Evolving

SVM Classification Tree

Shaoning Pang

Knowledge Engineering & Discovery Research Institute

Auckland University of Technology

Private Bag 92006, Auckland

1020, New Zealand

E-mail: Shaoning.Pang@https://www.360docs.net/doc/463287053.html,

Abstract—This article scopes a concept of SVM aggregating intelligence as 3 levels research: aggregating for a better machine learning performance, aggregating for an adaptive/dynamic intelligent system, and aggregating for multitask and life-long continuous machine learning, and reviews existing SVM aggregating methods including SVM ensemble, SVM classification tree, and evolving SVM classification tree.

I. INTRODUCTION

Support Vector Machine (SVM), since first proposed by Vapnik and his group at AT&T laboratory [1-3] has been extensively studied and discussed to develop its working principle of classification and regression. As a result, different types of SVM and SVM extensions [4] have been proposed. Suykens introduced the quadratic cost function in SVM and proposed LSSVM (Least Squares Support Vector Machine) [5]. Mangasarian et al. used an implicit Lagrangian reformulation in SVM, and proposed LSVM (Lagrangian Support Vector Machine) [6] and NSVM (Newton Support Vector Machine) [7]. Later, Lee and Mangasarian used a smooth unconstrained optimization in SVM, and had SSVM (Smooth Support Vector Machine)[8]. Recently, new interesting SVM models were published, such as, Chun-fu Lin's FSVM (Fuzzy Support Vector Machine) [9]. Zhang et al proposed HSSVMs (Hidden Space Support Vector Machines) [10]. Shilton et al. proposed an incremental version of SVM [11]. All these SVM types have significantly enhanced the original SVM performance. Most importantly, they have applied the original SVM to suit different real application needs..

SVM aggregating Intelligence, is an alternative aspect of SVM study, specializes on combining a family of concurrent SVMs for advanced artificial intelligence. Specifically, It includes three level aggregating researches,

1.Aggregating for Improved Machine Learning Performance.

2.Aggregating for Adaptive/Dynamic Machine Learning.

3.Aggregating for Multitask and life-long continuous Machine Learning.

At the first level research, the well known SVM aggregation methods are the One-against-all and One-against-one methods. The purpose of such aggregations is to expand SVM binary classification to multi-class classification. A typical procedure of SVM aggregation can be summarized as three steps, SVM model selection, convex aggregation, and aggregation training.

Over the last 5 years, I have been working on SVM aggregation, and have developed the original single SVM classification in our previous work, to SVM ensemble for classification, SVM classification tree (including 2-class SVM tree (2-SVMT), and Multi-class SVMT tree (m-SVMT)). Evolving SVM classification tree is an ongoing research topic of adapting SVMT to the incremental learning of data stream by evolving SVM and SVM tree structure.

II. METHDOLOGIES

A. SVM Ensemble

In SVM ensemble, individual SVMs are aggregated to make a collective decision in several ways such as the majority

voting, least-squares estimation-based weighting, and the double layer hierarchical combing. The training SVM ensemble can be conducted in the way of bagging or boosting. In bagging, each individual SVM is trained independently using the randomly chosen training samples via a boostrap technique. In boosting, each individual SVM is trained using the training samples chosen according to the sample’s probability distribution that is updated in proportion to the error in the sample. SVM ensemble is essentially a type of cross-validation optimization of single SVM, having a more stable classification performance than other models. The details on SVM ensemble construction and application are described in [12,13].

B. 2-Class SVM Tree

The principle of SVMT is to encapsulate a number of binary SVMs into a multi-layer hierarchy by adapting a "divide and conquer" strategy. The benefits of SVMT model can be summarized as: (1) SVMT is capable of effectively reducing classification difficulties from class mixture and overlap through a supervised LLE data partitioning. (2) Importantly, SVMT outperforms single SVM and SVM ensemble on the robustness to class imbalance.

A 2-class SVM tree can be modeled under the ‘depth first’ policy. The employed partitioning function for depth first is a binary data splitting whose targeting function is to partition all samples of class 1 into one cluster and all samples of class 2 into the other cluster. 2-SVMT of this type is particularly useful for the 2-class task with serious class overlap. Fig 1 shows an example of 2-class SVM binary tree over a face membership authentication [13,14] case with 30 of 271 persons as membership group.

Fig. 1.Example of 2-class SVM binary tree

Fig. 2.Example of 2-class SVM multiple tree.

Alternatively, a 2-class SVM tree also can be modeled under the ‘width first’ policy [15], where the employed partitioning function is a multiple data splitting, and the targeting function for partitioning here is to steer data samples in the same cluster with the same class label. A multiple data splitting is capable of controlling the size of the tree to a limited size, which is very optimal for decision making in such a tree structural model. Fig 3 gives an example of 2-class SVM multiple tree over the same case of face membership authentication as Fig. 2.

B. multi-class SVM Tree (m-SVMT)

The above SVMTs are merely two-class SVM classification tree (2-SVMT) model, which are not sustainable for normal multi-class classification tasks. However in real application, class imbalance of multi-class problem is also a critical challenge for most classifiers, thus it is desirable to develop a multi-class SVM tree with the above properties of 2-SVMT The construction of m-SVMT [17] is to decompose an m-class task into a certain number of 1-m classes regional tasks, under the criterion of minimizing SVM tree loss function. The proposed m-SVMT is demonstrated very competitive in discriminability as compared to other typical classifiers such as single SVMs, C4.5, K-NN, and MLP neural network, and particularly has a superior robustness to class imbalance, which other classifiers can not match. Fig.3 gives an example of m-SVMT for a 3-class task.

B. Evolving SVMT, an ongoing research topic

Learning over datasets in real-world application, we often confront difficult situations where a complete set of training samples is not given in advance. Actually in most of cases, data is being presented as a data stream where we can not know what kind of data, even what class of data, is coming in the future. Obviously, one-pass incremental learning gives a method to deal with such data streams [16,18,19].

For the needs of incremental learning over data stream, I am working to realize a concept of evolving SVM classification tree (eSVMT) for the classification of streaming data, where chunks of data is being presented at different time. The constructed eSVMT is supposed to be capable of accommodating new data by adjusting SVM classification tree as in the simulation shown in Fig. 4.

Fig. 3.Example m-SVMT over a 3 class task

Fig.4. A simulation of evolving SVM classification tree

The difficulty for eSVMT modelling is, (1) eSVMT needs to acquire knowledge with a single presentation of training data, and retaining the knowledge acquired in the past without keeping a large number of training samples in memory. (2) eSVMT needs to accommodate new data continuously, while always keeping a good size of SVM tree structure, and a good classification in real time.

II. CONCLUSIONS AND FUTURE WORK

On SVM aggregating for improved machine learning performance, methods of SVM aggregating such as SVM ensemble and SVM classification has significantly improved SVM classification on system stability, robustness to class imbalance and class overlap, which has lead SVM aggregating intelligence to various real-world applications on pattern recognition, data mining and bioinformatics.

For the next generation model of SVM aggregating intelligence, it is intended to build an adaptive/dynamic intelligent SVM composite system capable of always adapting itself to a new environment. eSVMT is a starting work at this level aggregating SVM by evolving tree structure. eSVMT with incremental SVM is a future work, which will empower eSVMT with an adaptability of both the evolution learning of tree structure, and the incremental learning of tree nodes.

Going to the third level, the concept of SVM aggregating intelligence in terms of multitask and continuous learning can be explained as, a structural model like a SVM Tree, or a SVM Forest system capable of evolving and growing itself to accommodate new tasks continuously in a life-long time.

ACKNOWLEDGMENT

The research reported in the article was partially funded by the ministry of Education, South Korea under the program of BK21, and the New Zealand Foundation for Research, Science and Technology under the grant: NERF/AUTX02-01. Also

the author would like to present the thanks to Prof. Nik Kasabov of Auckland University of Technology, Prof. S. Y.Bang, and Prof. Dajin Kim of Pohang University of Science and Technology, for their support and supervision during 2001 to 2005, when most of the reported research in this article was carried out.

REFERENCES

[1] V. Vapnik,, Estimation of dependences based on empirical data .Springer-Verlag, 1982.

[2] V.Vapnik, The nature of statistical learning theory, New York:Spinger-Verlag, 1995.

[3] C. Cortes and V. Vapnik, "Support vector network," Machine learning, vol. 20, pp. 273-297, 1995.

[4] https://www.360docs.net/doc/463287053.html,;https://www.360docs.net/doc/463287053.html,; https://www.360docs.net/doc/463287053.html,/dmi/

[5] J.A.K. Suykens and J. Vandewalle, “Least Squares Support Vector Machine Classifiers,” Neural Processing Letters, vol. 9, no. 2, pp. 293-300, 1999.

[6] O. L. Mangasarian and D. R. Musicant, “Lagrangian Support Vector Machines,” Journal of Machine Learning Research, vol. 1, pp. 161-177, 2001.

[7] G. Fung and O. L. Mangasarian, “Finite Newton method for Lagrangian support vector machine classification,” Data Mining Institute Technical Report, vol. 02, 2002.

[8] Y.-J. Lee and O. L. Mangasarian, “SSVM: A smooth support vector machine,” Computational Optimization and Applications, vol. 20, pp. 5-22, 2001.

[9] C.F. Lin and S.D. Wang, “Fuzzy Support Vector Machines,” IEEE Trans on Neural Networks, vol. 13, no. 2, pp. 464-471, 2002

[10] Li Zhang, Weida Zhou, and Licheng Jiao,“Hidden space support vector machines,” IEEE Transactions on Neural Networks, vol. 15, no. 6, pp.

1424-1434.

[11] A. Shilton, M. Palaniswami, D. Ralph and Ah Chung Tsoi, “Incremental training of support vector machines" IEEE Trans on Neural Networks, vol.

16, no. 1, pp. 114-131, 2005.

[12] Hyun-Chul Kim, Shaoning Pang, Hong-Mo Je, Daijin Kim, Sung Yang Bang: Constructing support vector machine ensemble. Pattern Recognition

vol. 36, no. 12, pp. 2757-2767, 2003

[13] Shaoning Pang, D. Kim, S. Y. Bang, Membership authentication in the dynamic group by face classification using SVM ensemble. Pattern

Recognition Letters, vol. 24, no. (1-3), pp. 215-225, 2003.

[14] Shaoning Pang, D. Kim, S. Y. Bang, Face Membership Authentication Using SVM Classification Tree Generated by Membership-based LLE Data

Partition, IEEE Trans. on Neural Network, vol. 16 no. 2, pp. 436-446, 2005.

[15]Shaoning Pang, Constructing SVM Multiple Tree for Face Membership Authentication. ICBA 2004, Lecture Notes in Computer Science 3072, pp.

37-43, Springer 2004.

[16] Shaoning Pang, Seiichi Ozawa, Nikola Kasabov, One-pass Incremental Membership Authentication by Face Classification. ICBA 2004, Lecture

Notes in Computer Science 3072, pp. 155-161, Springer 2004.

[17] Shaoning Pang, and Nikola Kasabov, Multi-Class SVM Classification Tree, (submitted), 2005.

[18] Shaoning Pang, Seiichi Ozawa and Nik Kasabov, Incremental Linear Discriminant Analysis for Classification of Data Streams ,IEEE Trans. on

System, Man, and Cybernetics-Part B, vol. 35, no. 5, pp. 905 – 914, 2005

[19] Seiichi Ozawa, Soon Toh, Shigeo Abe, Shaoning Pang and Nikola Kasabov, Incremental Learning for Online Face Recognition, Neural Network,

vol.18, no. (5-6), pp. 575-584, 2005.