Text Classification

c1=2
c1=8
...
Milk Bread
cd=D
Pet
Dog
Milk
Cat
Eat
Food
Dry
Michal Rosen-Zvi, UCI 2004
...
Simple model for topics
Given the topic words are independent
C
W
The probability for a word, w, given a topic, z, is θwz
Michal Rosen-Zvi, UCI 2004
W
Nd D
Inferring model parameters
One can find the distribution of θ by sampling
P(θ | c, w, α ) =
∫ dθP(w | c,θ ,α )P(c )
P(w | c, θ , α )P(c )
A need to define conceptual closeness
Michal Rosen-Zvi, UCI 2004
Feature Vector representation
From: Modeling the Internet and the Web: Probabilistic methods and Algorithms, Pierre Baldi, Paolo Frasconi, Padhraic Smyth
Michal Rosen-Zvi, UCI 2004
Michal Rosen-Zvi, UCI 2004
Classification: assigning words to topics
Different models for data:
Prediction of Categorical output e.g., SVM Discrete Classifier, modeling the boundaries between different classes of the data
Michal Rosen-Zvi, UCI 2004
Nave Bayes, multinomial:
α θ C
P({w,C}) = ∫d θ ∏dP(Cd)∏ndP(wnd|Cd,θ)P(θ) w
Generative parameters θwj = P(ω|c=j) ω Must satisfy ∑wθwj = 1, therefore the integration is over the simplex, (space of vectors with non-negative elements that sum up to 1) Might have Dirichlet prior, α
Michal Rosen-Zvi, UCI 2004
Making use of the topics model in cognitive science…
The need for dimensionality reduction Classification methods
Nave Bayes The LDA model
Document/Term count matrix
Doc1
LOVE SOUL RESEAR CH SCIENCE …
High dimensional space, not as high as |V|
Doc2 0 0 19 16 …
Doc3 … 3 2 6 1 …
34 12 0 0 …
SVD
LOVE
Michal Rosen-Zvi, UCI 2004
Nave Bayes classifier: words and topics
A set of labeled documents is given: {Cd,wd: d=1,…,D} w Note: classes are mutually exclusive
Model assumptions Inference by Gibbs sampling Results: applying the model to massive datasets
Michal Rosen-Zvi, UCI 2004
The need for dimensionality reduction
Michal Rosen-Zvi, UCI 2004
The Nave Bayes classifier
Assumes that each of the data points is distributed independently: Results in a trivial learning algorithm Usually does not suffer from overfitting
Nave Bayes
The LDA model
Topics model and semantic representation The Author Topic Model
Model assumptions Inference by Gibbs sampling Results: applying the model to massive datasets
This is a point estimation of the PDF, provides the mean of the posterior PDF under some conditions provides the full PDF
Michal Rosen-Zvi, UCI 2004
Where are we?
Making use of the MAP:
P(w, c | θ , α ) = P ( w | c, θ , α ) P ( c ) ∝ P ( w | c, θ , α ) P ( c ) P(w | θ , α )
θ w, j =
α + n (jw) αV + ∑=1 n (jl )
|V |
Michal Rosen-Zvi, UCI 2004
Sampling in the LDA model
The update rule for fixed α,β and integrating out θ β
Provides point estimates to θ and distributions of the latent variables, z.
∏ P(wi | θ , cd = j )
d =1... D
∏ P(cd = j ) = θ jw
is the
n (j w )
Here θ jw is the probability for word w given topic j and number of times the word w is assigned to topic j
Density Estimator: modeling the distribution of the data points themselves
Generative Models e.g. NB
Michal Rosen-Zvi, UCI 2004
A Spatial Representation: Latent Semantic Analysis (Landauer & Dumais, 1997)
Text Classification
Michal Rosen-Zvi University of California, Irvine
Outline
The need for dimensionality reduction Classification methods Nave Bayes The LDA model Topics model and semantic representation The Author Topic Model
Content-Based Ranking:
Ranking matching documents in a search engine according to their relevance to the user Presenting documents as vectors in the words space - 'bag of words' representation It is a sparse representation, V>>|D|
SOUL
RESEARCH SCIENCE
EACH WORD IS A SINGLE POINT IN A SEMANTIC SPACE
Michal Rosen-Zvi, UCI 2004
Where are we?
The need for dimensionality reduction Classification methods
Michal Rosen-Zvi, UCI 2004
Nd
D
P({w,C}| θ) = ∏dP(Cd)∏ndP(wnd|Cd,θ) w
Learning model parameters
Estimating θ from the probability: P({w, c}| θ ) =
i =1... N D
Michal Rosen-Zvi, UCI 2004
LDA: A generative model for topics
A model that assigns Dirichlet priors to multinomial distributions: Latent Dirichlet Allocation Assumes that a document is a mixture of topics