In Neural Computation (to appear). Practical identifiability of finite mixtures of multivar

In:
Neural Computation (to appear).
Practical identi ability of nite mixtures of multivariate Bernoulli distributions
Miguel A. Carreira-Perpi~an n Steve Renals Dept. of Computer Science, University of She eld, She eld S1 4DP, UK
1
which p(tj ) = p(tj 0 ) 8t 2 f0; 1gD . For example, it can be readily veri ed that the four mixtures given by the following parameter tuples represent the same distribution (here D m = 1; : : : ; M d = 1; : : : ; D
Q
(2) (3)
N X @l = 1 @pmd pmd (1 ? pmd) n=1 p(mjtn ; ; P)(tnd ? pmd )
where
D ptnd ? 1?tnd p(mjtn ; ; P) = PM p(tn jm; ;0P)p(m) 0 = PM m d=1 Dmd (1nd pmd) 1?tnd (4) Q t (1 ? p ) m =1 p(tn jm ; ; P)p(m ) m =1 m d=1 pm d md are the posterior probabilities (or responsibilities) that component m generated data point tn . The term \?N " P
M = 1; M = 2; M = 2; M = 2;
1 1 1 1
1 1 = 1; p1 = ( 2 1 2 )T 2 1 ; p1 = ( 1 0 1 )T ; =2 2 2 1 1 = 4 ; p1 = ( 1 0 2 )T ; 2 1 ; p1 = (1 1 1 )T ; =4 2 2
2= 2= 2=
p(tjm) =
M X m=1
ptd (1 ? pmd)1?td : md
These parameters must obey the following constraints:
m = 1; m 2 (0; 1)
8m = 1; : : : ; M ;
pmd 2 0; 1] 8m = 1; : : : ; M; d = 1; : : : ; D:
Abstract
1 Introduction
Finite mixtures of multivariate Bernoulli distributions have been extensively used in diverse elds (such as bacterial taxonomy) to model a population of binary, multivariate measurements in terms of a few latent classes; see (Everitt and Hand, 1981; Gyllenberg et al., 1994) and references therein. However, it has been recently proven that this class of mixture models is nonidenti able (Gyllenberg et al., 1994), which potentially undermines the interpretation of sample estimates, since the same sample could be equally attributed to a number of di erent estimates. We consider a nite mixture distribution (Everitt and Hand, 1981) de ned on the D-dimensional binary space f0; 1gD :
0: 00 : 000 :
:
However, it does not mean that for every parameter tuple there must exist at least one di erent 0 representing the same distribution. Identi ability is a property of the class of mixtures, rather than of a particular parameter tuple. Hence, in principle there are many tuples in parameter space that are completely equivalent but that would give rise to di erent interpretations. This may seem an insurmountable di culty for parameter estimation, but our practical studies have produced promising results: we will show that, given a sample from a mixture of multivariate Bernoulli distributions, maximum likelihood estimates of the parameters (obtained by an EM algorithm) can be still interpretable. Before giving some experimental results to support this claim, we give some properties of the log-likelihood surface of mixtures of multivariate Bernoulli distributions and introduce an EM algorithm for them.
fM.Carreira,S.Renalsg@
June 4, 1998
The class of nite mixtures of multivariate Bernoulli distributions is known to be nonidenti able, i.e., di erent values of the mixture parameters can correspond to exactly the same probability distribution. In principle, this would mean that sample estimates using this model would give rise to di erent interpretations. We give empirical support to the fact that estimation of this class of mixtures can still produce meaningful results in practice, thus lessening the importance of the identi ability problem. We also show that the EM algorithm is guaranteed to converge to a proper maximum likelihood estimate, owing to a property of the log-likelihood surface. Experiments with synthetic data sets show that an original generating distribution can be estimated from a sample. Experiments with an electropalatography (EPG) data set show important structure in the data.
l( ; P) =
N X n=1
log p(tn ; ; P) =
N X n=1
log
M X m=1
m
D Y d=1
ptnd (1 ? pmd )1?tnd md
!
(1)
and its gradient is easily seen to be
N @l = 1 X p(mjt ; ; P) ? N n @ m m n=1
m p(tjm) m=1 is the number of components, t = (t1 : : : tD )T is a binary D-dimensional vector, the m = p(m) are D Y d=1
p(t) =
M X
where M the mixture proportions and p(tjm) is a multivariate Bernoulli distribution with parameters (called prototypes ) pm = (pm1; : : : ; pmD )T , that is,
As opposed to trivial nonidenti ability, which is given by permutations of the mixture components or by coincident component distributions p(tjm) for several components.