半监督学习

et al., NIPS’04; etc.]
Disagreement-based methods
/zhouzh/
An example of S3VM TSVM (Transductive SVM)

Using unlabeled data to help identify the maximum margin hyperplane which goes through low-density region while keeping correct classification on labeled examples
SSL: Why unlabeled data can be helpful? (con’t) Intuitively, blue or red? Blue !

/zhouzh/
SSL: Representative approaches
X1 view
learner1
learner2
X2 view
labeled training examples [A. Blum & T. Mitchell, COLT’98]
/zhouzh/
Co-training (con’t)

A brief introduction to semi-supervised learning Advances in Disagreement-based SSL Some Probunlabeled data can be helpful?
0.15 0.71 0.63 0.76 0.71 0.26 0.22 0.70 0. 47
0. 60
0. 50 0.83 0. 20 0. 38 0. 55 0. 60 0. 32
/zhouzh/
SSL: Representative approaches Generative methods S3VMs Graph-based methods Disagreement-based methods
? = yes
label unknown
/zhouzh/
Labeled vs. Unlabeled

In many practical applications, unlabeled training examples are readily available but labeled ones are fairly expensive to obtain because labeling the unlabeled examples
requires human effort
(almost) infinite number of web pages on the Internet
class = “war”
?
/zhouzh/
Brief Outline

the edge weights correspond to some relationship (such as similarity/distance) between the examples
•
Propagate the labels on the graph
the regularization term usually specifies the local smoothness and/or global harmonic property
/zhouzh/
Co-training (con’t)

unlabeled training examples
labeled unlabeled examples
labeled unlabeled examples
α j f xi θ j
(
)
∑α f (x
l =1 l
L
i
θl )
unlabeled examples can be used to help estimate this term
/zhouzh/
[D.J. Miller & H.S. Uyar, NIPS’96]
label
training
Tenured no yes yes yes no no
trained model decision trees, neural networks, support vector machines, etc.
unseen data (Jeff, Professor, 7, ?)

Advances in Semi-Supervised Learning
(半监督学习研究进展)
Zhi-Hua Zhou
/zhouzh/ Email: zhouzh@
LAMDA Group National Key Laboratory for Novel Software Technology, Nanjing University, China
e.g. two views for web page classification: 1) the text appearing on the page itself, and 2) the anchor text attached to hyperlinks pointing to this page, from other pages
Zien, AISTATS’05; Collobert et al., ICML’06; etc.]
Graph-based methods
Using unlabeled data to regularize the learning process via graph regularization [Blum & Chawla, ICML’01; Belkin & Niyogi, MLJ04; Zhou
unlabeled training examples
Generative methods

& Uyar, NIPS’96; Nigam et al., MLJ00; Fujino et al., AAAI’05; etc.]
Using a generative model for the classifier and employing EM to model the label estimation or parameter estimation process [Miller
/zhouzh/
Co-training

In some applications, there are two sufficient and redundant views, i.e. two attribute sets each of which is sufficient for learning and conditionally independent to the other given the class label
Suppose the data is well-modeled by a mixture density:

f ( x θ ) = ∑ α l f ( x θl ) where and θ = {θl } ∑ l =1 α l = 1
Figure reprinted from [T. Joachims, ICML99]
/zhouzh/
An example of SSL with graph regularization
•

Construct a weighted graph on the labeled and unlabeled training examples
Thus, the optimal classification rule for this model is the MAP rule:
S ( x ) = arg max ∑ j P ⎡ci = k mi = j , xi ⎦ P ⎣ mi = j xi ⎤ ⎤ ⎡ ⎣ ⎦ k
where P ⎡ mi = j xi ⎦ = ⎤ ⎣

multiple learners are trained for the task and the disagreements among the learners are exploited during the SSL process [Blum &
S3VMs (Semi-Supervised SVMs)
Using unlabeled data to adjust the decision boundary such that it goes through the less dense region [Joachims, ICML’99; Chapelle &
/zhouzh/
Co-training (con’t)

unlabeled training examples
X1 view
learner1
learner2
X2 view
labeled training examples [A. Blum & T. Mitchell, COLT’98]
L
L
l =1
The class labels are viewed as random quantities and are assumed chosen conditioned on the selected mixture component mi ∈ {1,2,…,L} and possibly on the feature value, i.e. according to the probabilities P[ci|xi,mi]
Mitchell, COLT’98; Goldman & Zhou, ICML’00; Zhou & Li, TKDE05; etc.]