智能信息检索课程第13讲

Assumption 3: Discriminative OneDirectionally and Locally Dependent Model
Maximum Entropy Markov Model (MEMM)
Assumption 4: Discriminative Two-Directionally and Locally Dependent Model
Example: POS Tagging Problem
Verb Verb Verb Article
Verb
Noun
Noun
Preposition
Noun
Time
flies
like
an
arrow
Technologies for Information Extraction (Tagging)
• • • • • • Hidden Markov Model (HMM) Maximum Entropy (ME) Maximum Entropy Markov Model (MEMM) Conditional Random Fields (CRF) Perceptron Support Vector Machines (SVM)
s1 ...st 1
t 1 ( j ) max t (i)aij b jo
i
i
t 1
t 1 ( j ) arg max t (i)aij b jo
t 1
Supervised Learning
• Maximum Likelihood
s'
s o
f ( s, s ' ) ˆ P( s | s' ) f ( s' ) f (o, s) ˆ P (o | s ) f (s)
• An efficient algorithm using dynamic programming.
t (i) P(o1...ot , st i)
t (i) P(ot ...oT | st i)
Forward Probability
s1=1 s1 =2 s1 = N st-1 =1 st-1=2 st-1=N st =1 st =2 st = N st+1 =1 st+1=2 st+1 =N sT =1 sT=2 sT=N
Information Extraction
Hang Li Information Retrieval and Mining Group Microsoft Research Asia
Outline of Talk
• • • • • • • What is Information Extraction General Framework Hidden Markov Model Maximum Entropy Model Maximum Entropy Markov Model Conditional Random Fields Information Extraction Tasks
Conditional Random Fields (CRF) Perceptron Support Vector Machines
Hidden Markov Model
HMM Formalism
s1 o1
st 1 ot 1
st
st 1 ot 1
sT oT
ot
• s : {1,2,…,N} are values of hidden states • o : {1,2,…,M} are values of observations
P(O) T (i )
i 1
N
N
Forward Procedure Backward Procedure Combination
P(O) p i 1 (i )
i 1
P(O) t (i ) t (i)
i 1
N
Tagging
o1
ot-1
ot
ot+1
oT
• Viterbi algorithm
• Smoothing (Laplace Estimation)
Unsupervised Learning
o1
ot-1
ot
ot+1
oT
• No analytic method • EM Algorithm
Unsupervised :Baum-Welch Algorithm
Unstructured Data IE System
Structured Data Information Analysis
IM System
Unstructured Data
Information Extraction = Tagging or Parsing
Tagging Problem
• Green nodes are states • Purple nodes are observations
States generates observations
Observations (features) determine states Learning = learning a model mapping observation sequence to state sequence Tagging = finding most likely state sequence mapped from given observation sequence
What is Information Extraction?
Task: Filling tables in database from segments of document
October 14, 2002, 4:00 a.m. PT For years, Microsoft Corporation CEO Bill Gates railed against the economic philosophy of open-source software with Orwellian fervor, denouncing its communal licensing as a "cancer" that stifled technological innovation. Today, Microsoft claims to "love" the opensource concept, by which software code is made public to encourage improvement and development by outside programmers. Gates himself says Microsoft will gladly disclose its crown jewels--the coveted code behind the Windows operating system--to select customers. "We can be open source. We love the concept of shared source," said Bill Veghte, a Microsoft VP. "That's a super-important shift for us in terms of code access.“ Richard Stallman, founder of the Free Software Foundation, countered saying…
Structured Data
Unstructured Data IE System
Structured Data Information Access
IR System
Unstructured Data
Information Extraction vs Information Mining
Structured Data
t (i) P(o1...ot , st i)
1 (i) p i bio
t 1 ( j )
1
i 1...N
(i)a b
t ij
jot 1
Backward Probability
s1
=1
st-1 =1
st-1 =2 st-1=N
st =1
st =2 st = N
P(O, S ) P( s1 ) P(o1 | s1 ) P( st | st 1 ) P(ot | st )
t 2
T
HMM Formalism
s'
s o
P( s | s' ) P (o | s )
Probability Calculation
• • • • {P, A, B} P {pi} are initial state probabilities A = {aij} are state transition probabilities B = {bik} are observation generation probabilities
O (o1...oT ) Compute P(O)
Probability Calculation
P(O)
{ s1 ...sT }
p
s1 s1Hale Waihona Puke 1bPat 1
T 1
st st 1 st 1ot 1
b
Not tractable !
Forward Backward Algorithm
arg max P(S | O) arg max P(S , O)