半监督支持向量机的研究

Relax S3VMs as convex Semi-Definite Programming (SDP) SDP typically scales O(n6.5) where n is the sample size [Zhang
et al., TNN2011].
Pro: promising performance Con: poor scalability (i.e., could not handle with more than several
supervised learning Semi-supervised learning: the learner tries to exploit the unlabeled
examples by itself. Active learning: the learner actively selects some unlabeled examples to query from an oracle
Ivor Tsang
Nanyang Technological University
Acknowledge
Teng Zhang, Linli Xu and Kai Zhang
Supervised Learning

Labeled Data
Train
Learning Methods
Research on Semi-Supervised SVMs (半监督支持向量机的研究)
Yu-Feng Li
National Key Laboratory for Novel Software Technology, Nanjing University, China

Bioinformatics [Kasabov & Pang, 2004] Named Entity Recognition [Goutte et al., 2002] …
Outline

Scalability of S3VMs WellSVM [Li et al., JMLR13] Efficiency of S3VMs
S3VMs: Applications

Text Categorization [Joachims 1999; Joachims, 2002] Email Classification [Kockelkorn et al., 2003]
Image Retrieval [Wang et al., 2003]
Semi-Supervised Learning

SLearner
SSLearner
Several Surveys and Books O. Chapelle et al. Semi-supervised learning. MIT Press Cambridge, 2006. X. Zhu and A. Goldberg. Introduction to semi-supervised learning. Morgan & Claypool Publishers, 2009. Z.-H. Zhou and M. Li. Semi-supervised learning by disagreement. Knowledge and Information Systems, 24(3):415–439, 2010. 周志华. 基于分歧的半监督学习, 特邀综述. 自动化学报. 2013年11月.
The seminal work [Zhu et al., 2003] has won the ‘10-year best paper’ award in the 30th International Conference on Machine Learning (ICML’13).
Semi-supervised support vector machines (S3VMs) [Vapnik, 1998;
thousand examples)
Can we have a scalable and convex S3VM?
WellSVM [Li et al., JMLR13]
Observation

Labeled Data Is Expensive

However, labeling process is expensive in many real tasks
Disease diagnosis Drug detection Image classification Text categorization …
URL: /liyf/
Email: liyf@
MLA’13, Shanghai
Joint work with
Zhi-Hua Zhou
Nanjing University
James Kwok
Hong Kong University of Science and Technology
Four Major Paradigms of SSL
Generative methods [Miller & Uyar, 1997; Nigam et al., 2000; Cozman & Cohen, 2002] Co-training/Disagreement-based methods [Blum & Mitchell, 1998; Balcan et al., 2005; Zhou & Li, 2010]
Pro: good performance on very small data sets Con: poor scalability (i.e., could not handle with more than several
hundred examples)
Related Works

al., JMLR2006]
Pro: good scalability
Con: suffer from local optima, suboptimal performance
Related Works

SDP convex relaxation [Xu et al., 2005; De Bie and Cristianini, 2006]
Predict
Unseen Data
In order to have a good generalization performance, supervised learning methods often assumes that a large amount of labeled data are available.
Local optimization
Local Conbinatorial Search [Joachims, ICML1999] Alternating Optimization [Zhang et al., ICML2009] Constrained Convex-Concave Procedure (CCCP) [Collobert et
S3VMs: Formulation

Control model complexity
Losses on labeled and unlabeled data
The label of unlabeled data are unknown, and need to be optimized
Human efforts and material resources
Exploiting Unlabeled Data

Collection of unlabeled data is usually cheaper
Two popular schemes for exploiting unlabeled data to help

Scalability of S3VMs WellSVM [Li et al., JMLR13] Efficiency of S3VMs
“多”
MeanS3VM [Li et al., ICML09]
Safeness of S3VMs S4VM [Li and Zhou, ICML11] Cost sensitivity of S3VMs CS4VM [Li et al., AAAI10]
Bennett & Demiriz, 1999; Joachims, 1999; Chapelle & Zien, 2005]
The seminal work [Joachims, 1999] has won the ‘10-year best paper’ award in the 26th International Conference on Machine Learning (ICML’09).
S3VMs

Unlabeled Data
Large-margin separator (or, low-density separator)
Labeled Data
In [Vapnik, SLT’98], it is shown that large margin could help improve the generalization learning bound.
“多”
MeanS3VM [Li et al., ICML09]
Safeness of S3VMs S4VM [Li and Zhou, ICML11] Cost sensitivity of S3VMs CS4VM [Li et al., AAAI10]
“快”
“好”
“省”
Outline