机器学习算法介绍及相关参考文献

Linear Discriminant Analysis

Linear Discriminant Analysis (LDA) is a linear inherently multi-class classification method. It was originally introduced by Fisher for two classes [9], but was later extended for multiple classes by Rao [26]. In particular, LDA computes a classification function

()g x W x T =，where W is selected as the linear projection that maximizes the Fisher-criterion

||

arg max ||T B opt

T

W W

W S W W W S W =，where W S and B S are the within-class and the between class scatter matrices (see, e.g., [7]). The

corresponding optimal solution for this optimization problem is given by the solution of the generalized eigenproblem B W S w S w λ= or directly by computing the eigenvectors for 1

W B S S -. Since the rank

of

1

w b S S - is bounded by the rank of b S there are 1c - non-zero eigenvalues resulting in a

(1c -)-dimensional subspace (1)T

c n

L W X R

-?=∈, which preserves the most discriminant

information. For classification of a new sample m

x ∈

the class label

{1,...,}c ω∈ is assigned

according to the result of a nearest neighbor classification. For that purpose, the Euclidean distances

d of th

e projected sample ()g x and the class centers T i i v W μ= in the LDA space are compared:

1arg min ((),)i i c

d g x v ω≤≤=.

Loog et al. [19] showed that for more than two classes maximizing the Fisher criterion in Eq. (7) provides only a suboptimal solution! In particular, optimizing the Fisher criterion provides an optimal solution with respect to the Bayes error for two classes, but this can not be generalized for multiple classes. Nevertheless LDA can be applied for many practical multi-class problems. This was also confirmed by theoretical considerations by Mart′?nez and Zhu [20]. However, they showed that increasing the number of classes decreases the separability.

[9] R. A. Fisher. The use of multiple measurements in taxonomic problems. Annals of Eugenics, 7:179–188, 1936. [26] C. R. Rao. The utilization of multiple measurements in problems of biological classification. Journal of the Royal Statistical Society – Series B, 10(2):159–203, 1948.

[7] R. O. Duda, P. E. Hart, and D. G. Stork. Pattern Classification.John Wiley & Sons, 2000.

[19] M. Loog, R. P. W. Duin, and R. Haeb-Umbach. Multiclass linear dimension reduction by weighted pairwise fisher criteria. IEEE Trans. PAMI, 23(7):762 – 766, 2001.

[20] A. M. Mart′?nez and M. Zhu. Where are linear feature extraction methods applicable? IEEE Trans. PAMI, 27(12):1934 –1944, 2005.

Multi Kernel Learning

Recently, Multiple Kernel Learning (MKL) [25, 16, 29] has become a quite popular method to combine data from multiple information sources. The main idea is to create a weighted linear combination of the kernels obtained from each information source. Moreover, in Rakotomamonjy etal. [25] it was shown that by using multiple kernels instead of one a more effective decision function can be obtained. In particular, the kernel (,)x x 'K can be considered a convex combination of M

basis kernels

(,)j x x 'K :1

(,)(,)M

j j j K x x d K x x =''=∑, where 0j d ≥ are the weights of the kernels

j K and

1

1M j j d ==∑

. Thus, the decision function ()g x of an SVM with multiple kernels can be

represented as

1

()(,)N

i i i i g x y K x x b α==-∑

1

1

(,)N M

i i j j i i j y d K x x b α=='=-∑∑,

where

i x are the training samples and {1,1}i y ∈-+ are the corresponding class labels. Hence,

when training an MKL model the goal is to learn both, the coefficients i α and the weights m d , in

parallel.

[25] A. Rakotomamonjy, F. R. Bach, S. Canu, and Y. Grandvalet. SimpleMKL. J. of Machine Learning Research, 9:2491–2521, 2008.

[16] G. R. G. Lanckriet, T. d. Bie, N. Cristianini, M. I. Jordan, and W. S. Noble. A statistical framework for genomic data fusion. Bioinformatics, 20(16):2626–2635, 2004.

[29] S. Sonnenburg, B. S. Bernhard, P. Bennett, and E. ParradoHern′andez. Large scale multiple kernel learning. J. of Machine Learning Research, 7, 2006.

AdaBoost Learning

AdaBoost [6] is a popular machine learning method combining properties of an efficient classifier and feature selection. The discrete version of AdaBoost defines a strong binary classifier H

1

()sgn(())T t t t H z h z α==∑1

()sgn(())T

t t t H z h z α==∑

using a weighted combination of T weak learners h t with weights a t . At each new round t, AdaBoost selects a new hypothesis

t h that best classifies training samples with high classification error in the

previous rounds. Each weak learner

may explore any feature f of the data z. In the context of visual object recognition it is attractive to define f in terms of local image properties over image regions r and then use AdaBoost for selecting features maximizing the classification performance. This idea was first explored by Viola and Jones

Adaboost 算法是1995年由Yoav Freund 和Robert E ．Schapire 提出的，是目前最具实用价值的机器学习方法之一，它是一种迭代的学习方法，可以将一组弱学习算法提升为一个强学习算法。Adaboost 算法本身是通过改变数据分布来实现的，它根据每次训练集中每个样本的分类是否正确，以及上次的总体分类准确率，来确定每个样本的权值，将每次训练得到的弱分类器最后融合起来，作为最后的决策分类器。这里所说的弱分类器是指它的分类结果应该比随机猜想的准确性稍高，即准确率大于50， 随着弱分类器个数的增加，最终集成的强分类器的错误率将趋近于零[6].

行人检测问题实质上是一个两类分类问题，其检测结果只有人和非人两种。本文采用了一种改进的Adaboost 算法，该算法的特点是使每一个弱分类器对应于一个特征，根据错误率最小的原则选择弱分类器，即在。Freund 提出的Adaboost 算法的迭代过程中计算针对样本集的分类最小错误率选择弱分类器。该算法的简单描述如下：

定义X 表示样本空间，Y 表示样本类别标识集合，此处限定{1,0}Y =，分别对应样本的正和负。假定共有K 个特征。已知1122(,),(,),(,)N N x y x y x y ???构成训练样本集，其中

,i i x X y Y ∈∈，假设样本集中共有p N 个正样本，n N 个负样本，p n N N N +=。

1. 初始化样本权重

1,1,1

12112i N

p

i i i i n

x N x N ωω=???==?

???∑为正样本，为负样本

2. for t=1,2,…,T(T 为最大训练轮数) （1）归一化样本权重分布：

j v ,,,1

,

1,2,,t i

t i N

t i

i i N ωωω

==

=???∑

(2)对于每个特征，在给定的样本权重分布下训练弱分类器

,1()0t j t j

t j p v p f x otherwise

θ

? 其中j v 为第j 个特征值，,,,1

|()|,1,2,,N

t j

t i t j i i i e f x y j K ω==-=???∑为阈值，{1,1}i p ∈-表示

不等号的偏置方向，1,2,,j K =???。

（3）计算每个弱分类器对样本集的分类错误率：

,,,1

|()|,1,2,,N

t j t i t j i i i e f x y j K ω==-=???∑

（4）选择第t 轮训练的最优弱分类器()t h x ： 令,arg min t j j

k e =，则,()()t t k h x f x =

，并且对样本集的分类错误率,t t k e ζ=

（5）更新样本权重：

11,,,

1,2,,t

t i t i t i N εωωβ-+==??? （4）

若i x 被t h 正确分类，则0t ε=，否则1t ε=，其中1t t t

ζβζ=

-

3. 最终强分类器

最终的强分类器由T 个弱分类器线性叠加而成：

1

ln

t t

αβ

=1

2

1

1

1

()()0T T t t t

t t r h x H x otherwise

αα==?≥?=?

??∑

∑

(5)

其中1r r ?+

单个强分类器训练过程

由于特征数目巨大，若按照传统的Adaboost 算法， 分类器的训练过程是非常耗时的，故此对训练过程引入以下3点改进： (1) 特征筛选：传统的Adaboost 算法在每一轮的训练过程中遍历所有弱分类器，然后挑出一个最优的弱分类器。由于矩形特征数目庞大，且性能参差不齐，故此在训练之前在初始权重情况下先进行一次特征筛选，挑选一批性能较高的弱分类器。选取的原则为0.4t e <，然后进行Adaboost 的训练过程。特征筛选用在层状分类器的前几层是尤为重要的，因为前几层强分类器的设计原则就是集合少量性能较高的弱分类器，提高检测速度，剔除非行人样本。 (2)阈值选取：传统的Adaboost 算法在弱分类器的最优阈值j θ选取上也是采用大范围数值遍历的方法。对单个特征()j v x 而言，所有的训练样本对于此特征的特征值可表示为

123(),(),(),()j j j j n v x v x v x v x ???，因此只要求得min ()j i i

v x 和max ()j i i

v x ，1,2,,i n =???，最

优阈值j θ只需在区间[min ()j i i

v x ,max ()j i i

v x ]内遍历寻找即可，从而缩小遍历空间，减少

训练时间。

(3)权重更新：传统的Adaboost 算法在每轮训练过程后都需要进行样本权重更新，以加重对错误分类的样本的关注度。但是随着训练轮数不断增加， 由式（4）可知，已经被正确分类的样本的权重会不断减小，从而导致分类器对其关注度逐渐减小，很可能逐渐破坏已经生成的较准确的分类规则，因此限制部分困难样本的权重扩张是非常必要的。假设每轮训练后的 正负样本权重之和分别为,|0

n

i N n t i

i y S ω

==

∑，,|1

p

i N p t i

i y S ω

==

∑，限定:1n p S S =。式(4) 中的样本

权重更新将在各自的目标类内进行。这样可以保持已经生成的较为准确的分类规则，减少分类器训练的轮数，从而提高检测速度。 层状分类器设计

层状分类器设计的原则是保证正样本检测率（99.5%）不变的情况下，逐层减小负样本的错检率。训练过程中正样本不变，在该层被正确分类的负样本将被剔除，重新加入新的负样本，作为下一层分类器训练集中的负样本。前端强分类器设计的目的就是尽可能的排除非人样本，因此要求该分类器结构简单，检测速度快， 由于Adaboost 算法中每层的强分类器的阈值是最优选定的(式(5))，为了保证以最少的弱分类器构造强分类器，可能需要手动调整每层强分类器的阈值，以达到对正样本预定的检测率。越往后的分类器结构越复杂，就需要大量的弱分类器来构造，以提高样本整体的检测性能。 由算法的训练过程可知，当已经获得的分类器对于某些样本分类正确，那么减小这些样本的

权重；当分类错误，增加这些样本的权重 J 。这样，后面训练提取的简单分类器就会更加强化对这些分类错误样本的训练。最终，通过权重组合所有的弱分类器以形成强分类器.

多检测窗口合并

由于检测器对尺度变化和位置平移较为敏感，可能导致对同一目标产生多个检测结果的现象发生，因此需要引入检测窗口的合并以将这些重叠的窗口合并为一个。合并的原则较为简单，如果两个矩形1R 和2R 相交的面积S 与两个矩形面积的比值均大于0.6，那么就将这两个矩形求其平均窗口合并为一个矩形，否则视其为两个不同的目标。

[18] who used AdaBoost to train an efficient face detector by selecting a discriminative set of local Haar features.

[6] Y. Freund and R. E. Schapire. A decision-theoretic generalization of on-line learning and an application to boosting. J. of Comp. and Sys. Sc., 55(1):119–139, 1997.

[18] P. Viola and M. Jones. Rapid object detection using a boosted cascade of simple features. In Proc. CVPR, pages I:511-518, 2001.