在SVM中String Kernel的应用

KEDRI-NICT Project Report - APPENDIX:DString Kernel Based SVM for InternetSecurity ImplementationZbynek Michlovsk´y1,Shaoning Pang1,Nikola Kasabov1,Tao Ban2,and Youki Kadobayashi21Knowledge Engineering&Discover Research InstituteAuckland University of Technology,Private Bag92006,Auckland1020,New Zealand{spang,nkasabov}@2Information Security Research Center,National Institute of Information andCommunications Technology,Tokyo,184-8795Japanbantao@nict.go.jp,youki-k@is.aist-nara.ac.jpAbstract.For network intrusion and virus detection,ordinary meth-ods detect malicious network traffic and viruses by examining packets,flow logs or content of memory for any signatures of the attack.Thisimplies that if no signature is known/created in advance,attack detec-tion will be problematical.Addressing unknown attacks detection,wedevelop in this paper a network traffic and spam analyzer using a stringkernel based SVM(support vector machine)supervised machine learn-ing.The proposed method is capable of detecting network attack with-out known/earlier determined attack signatures,as SVM automaticallylearning attack signatures from traffic data.For application to internetsecurity,we have implemented the proposed method for spam email de-tection over the SpamAssasin and E.M.Canada datasets,and networkapplication authentication via real connection data analysis.The ob-tained above99%accuracies have demonstrated the usefulness of stringkernel SVMs on network security for either detecting‘abnormal’or pro-tecting‘normal’traffic.1IntroductionUpon computers and Internet being more and more integrated into our com-mon life,higher security requirements have been imposed on our computer andnetwork system.One of many ways that we can take for increasing the securityis to use the Intrusion Detection System(IDS).Intrusion detection(system)isa process of monitoring events occurring in a computer system or network andanalyzing them on signs of possible incidents.IDS,as described in[10],can begrouped into two categories:statistical anomaly based IDS and signature basedIDS.The idea of statistical anomaly IDS is to detect intrusions by comparingtraffic with normal traffic model,looking for deviations.Due to the diversity ofnetwork traffic,it is difficult to model normal traffic,as we know that a normalemail relaying or peer-to-peer queries may also show like with some intrusiontraffic characteristics.Moreover,even for abnormal traffic,it does not in factC.S.Leung,M.Lee,and J.H.Chan(Eds.):ICONIP2009,Part II,LNCS5864,pp.530–539,2009.c Springer-Verlag Berlin Heidelberg2009String Kernel Based SVM for Internet Security Implementation531 constitute an intrusion/attack.Hence,anomaly detection often has a high false alarm rate,thus is seldom used in practice.For signature based IDS,network traffic is examined for predetermined attack patterns known as signatures.A signature consists of a string of characters(or bytes).Nowadays many intrusion detection systems also support regular expressions and even behavioralfinger-prints[13].The difficulty of signature based IDS system is that only intrusions whose signatures are known can be detected and it is necessary to constantly update a collection of these signatures to mitigate emerging threats[14].Another approach for enhancing network security is to authenticate and pro-tect legitimate network traffic.Traditional traffic authentication method based on network protocol(e.g.the port number)is becoming more inaccurate and not appropriate for the identification of P2P and other new types of network traf-fic.Other methods are mostly based on protocol anomaly detection,which are highly limited because that Internet legitimate traffic does not strictly conform to any specific traffic models.Thus those methods often involve the difficulty of high false positives error in real applications.For example,legitimate Smtp traffic could be identified as malicious traffic,if there is misconfiguration in MTA server adding suspiciousfields to the header of email message.Thus,it is likely to mis-authenticate valid Smtp traffics following just the standard Smtp network protocol.For either attack detection or legitimate network traffic authentication,most signatures/rules are created by security experts who analyze network traffic and host logs after intrusions have occurred,whereas sifting through thousands lines of logfiles and looking for characteristics that uniquely identify as an intrusion is a vast and error prone undertaking.To overcome this shortcoming and de-tect unknown attack(i.e.signature is not determined),we researched machine learning on string content recognition techniques.The motivation is to train a classifier to distinguish between malicious and harmlessflows or memory dump and utilize the trained classifier to classify real networkflow and memory dump.Support vector machine(SVM)is known as one of the most successful classi-fication algorithms for data mining,in this work we address string,rather than numerical,data analysis,and implemented string kernel SVM for network secu-rity in the way of intrusion detection and network application authentication. In spite of the limitation of the SVM on training efficiency,the advantage of string kernel SVMs is in needless of complete knowledge about attack signa-ture/normal application features,as string kernel SVM is able to automatically learn the problem knowledge during the training procedure.In this work,we develop SVM based string kernel method according to differ-ent mathematical similarity expressions of two strings/substrings.For network security,we derive string kernel SVM for automatical attack(i.e.spam emails) signature analysis,conducting spamfiltering without early determined spam signature.Moreover,we have used string kernel SVM to authenticate legitimate network applications,learning SVM from connection differences against normal connections.532Z.Michlovsk´y et al.2SVM and Kernels TheorySupport vector machines(SVM)are groups of supervised learning methods ap-plicable to classification or regression.SVM maps the data points into a high dimensional feature space,where a linear learning machine is used tofind a maximal margin separation[7,8].One of the main statistical properties of the maximal margin solution is that its performance does not depend on the dimen-sionality of the space where the separation takes place.In this way,it is possible to work in very high dimensional spaces,such as those induced by kernels,with-out overfitting.Kernels provide support vector machines with the capability of implicitly mapping non-linearly separable data points into a different higher dimensional space,where they are more separable than the original space.This method is also called Kernel trick[7].Kernel function K(x,y)can be expressed as a dot product in a high dimen-sional space.If the arguments to the kernel are in a measurable space X,and if the kernel is positive semi-definite for anyfinite subset{x1,...,x n}of X and subset{c1,...,c n}of objectsK(x i,x j)c i c j≥0,(1)i,jthen there must exist a functionφ(x)whose range is in an inner product space of possibly high dimension,such that K(x,y)=φ(x)φ(y).The kernel method allows for a linear algorithm to be transformed into a non-linear algorithm.This non-linear algorithm is equivalent to the linear algorithm operating in the range space ofφ.However,because kernels are used,theφfunction is never explicitly computed.The kernel representation of data amounts to a nonlinear projection of data into a high-dimensional space where it is easier to separate into classes [12].Most popular kernels suitable for SVM are e.g.Polynomial Kernel,Gaussian Radial Basis Kernel,Hyberbolic Tangent Kernel[11].All of these kernels operate with numerical data.For our purpose is necessary to use string kernels which are described in following section.3String Kernels Used in SVMRegular kernels for SVM work merely on numerical data,which is unsuitable for internet security where huge amount of string data is presented.Towards extending SVM for string data processing,we implemented the following string kernels algorithms in our experiments.3.1Gap-Weighted Subsequence KernelThe theory of subsequence kernel is described in the book Kernel Methods for Pattern Analysis[2].The main idea behind the gap-weighted subsequence kernelString Kernel Based SVM for Internet Security Implementation 533is to compare strings by means of the subsequences they contain -the more subsequences and less gaps they contain the more similar they are.For reducing dimensionality of the feature space we consider non-contiguous substrings that have fixed length p .The feature space of gap-weighted subsequence kernel is defined as φp u (s )=i :u =s (i )λl (i ),u ∈Σp ,(2)where λ∈(0,1)is decay factor,i is index the occurrence of subsequence u =s (i )in string s and l (i )is length of the string in s .We weight the occurrence of u with the exponentially decaying factor λl (i ).The associated kernel is defined as κ(s,t )= φp (s ),φp (t ) = u ∈Σpφp u (s )φp u (t ).(3)In Eq.(3),it is required to conduct an intermediate dynamic programming table DP p whose entries are:DP p (k,l )=ki =1l j =1λk −i +l −j κS p −1(s (1:i ),t (1:j )).(4)Then,the computational complexity is evaluated as κS p (sa,tb )= λ2DP p (|s |,|t |)if a =b ;0otherwise (5)which follows that for a single value of p ,the complexity of computing κS p is O (|s ||t |).Thus,the overall computational complexity of κp (s,t )is O (p |s ||t |).3.2Levenshtein DistanceLevenshtein (or edit)distance [4]counts differences between two strings.The distance is the number of substitutions,deletions or insertions required to trans-form string s with length n to string t with length m .The formal definition of Levenshtein distance [17]is given as follows:Given a string s ,let s (i )stand for its i th character.For two characters a and b ,definer (a,b )=0ifa =b.Let r (a,b )=1(6)Assuming two strings s and t with the length of n and m ,respectively,then a (n +1)(m +1)array d furnishes the required values of the Levenshtein distance L (s,t ).The calculation of d is a recursive procedure.First set d (i,0)=i ,i =0,1,...,n and d (0,j )=j ,j =0,1,...,m ;Then,for other pairs i ,j ,we haved (i,j )=min (d (i −1,j )+1,d (i,j −1)+1,d (i −1,j −1)+r (s (i ),t (j ))).(7)In our implementation,we use D =e −λ·d (i,j )for getting better results.Analog to the above substring kernel,the computational complexity of Levenshtein Dis-tance is O (|s ||t |).In the case that s and t have the same length,the complexity is O (n 2).534Z.Michlovsk´y et al.3.3Bag of Words KernelThe Bag of words kernel is represented as an unordered collection of words, disregarding grammar and word order.Words are any sequences of letters from the basic alphabet separated by punctuation or spaces.We represent a bag as a vector in a space in which each dimension is associated with one term from the dictionaryφ:d→φ(d)=(tf(t1,d),tf(t2,d),....tf(t N,d))∈R N,(8) where tf(t i,d)is the frequency of the term t i in the document d.Hence,a docu-ment is mapped into a space of dimensionality N being the size of the dictionary, typically a very large number[2].3.4N-Gram KernelN-grams transform documents into high dimensional feature vectors where each feature corresponds to a contiguous substring[5].The feature space associated with n-gram kernel is defined asφn u(s)φn u(t)(9)κ(s,t)= φn(s),φn(t) =u∈Σnwhereφn u(s)=|{(v1,v2):s=v1uv2}|,u∈Σn.We have used for computing n-gram kernel naive approach therefore the time complexity is O(n|s||t|).4Experiments and DiscussionsFor internet security,one useful approach is to actively detect andfilter spam/ attack by building Bayesianfilters.In addition,another practical approach is to protect legitimate network communication by authenticating every type of legitimate network application.In this section,we implemented string kernel SVMs on the spamAssasin public mail corpus for email spam detection,and experimented the authentication of12categories standard network application.For multi-class classification,we used support vector machine software-lib-SVM[1].In our experiments,we used precomputed kernel matrices from pre-pared training and testing datasets.All values in precomputed kernel matrices have been scaled to interval[-1,1].Optimal parameters of string kernel functions have been determined by cross validation tests on training datasets.Kernel ma-trices has been applied as input to libSVM[1].String Kernel Based SVM for Internet Security Implementation535 4.1Spam DetectionFor spam detection experiment,we used5500ham messages from SpamAssasin public mail corpus[18]and24038spam messages from E.M.Canada[19].The ham emails dataset consists of two categories:EasyHam emails,5000non-spam messages without any spam signatures and Hard Ham emails,500non-spam mes-sages similar in many aspect to spam messages-using unusual HTML markup, colored text,spam-sounding phrases,etc.Each email message has a header,a body and some potentially attachments.Note that for the convenience of com-parison,we employed the exact same data setup as in[20],training dataset23630 (19230spam vs.4400ham)messages and testing dataset5918(4808spam vs. 1110ham)messages.Analog to[20],we intended to determine which part of email message have critical influence on the classification results.To this end,we prepared four subsets:Subject,Body,Header and All subsets.The Subject subset uses only the subjectfield of the email message,and all Subject data are normalized to the length of100characters;The body subset is the body part of the email message normalized to the length of1000characters;The header subset is the header section of the email message normalized to the length of100characters; The All subset concludes the Fromfield and the Subjectfield of the header section plus the whole body of the email massage.Also,every instance of All subset is normalized to the length of1200characters.Table1.The results from email classification using each kernel function with an comparison to[20]Features String Kernel Function Ref.Acc.[20]N-gram Subsequence Edit Distance Bag of wordSubject96.3896.6495.7881.1692.64Body99.2899.2397.1981.0484.41Header99.8099.8099.7582.6892.12All99.3499.3898.0281.2490.13 Table1presents the percentage accuracy of correctly classified email messages for each type of string kernel and email subset,where percentage accuracy of [20]is presented in the Ref.Acc.column for comparison.As seen from the table,the classification results reached by string kernel SVM are exceeding the percentage accuracy from the the reference paper[20].The results for subset Header demonstrate that thefirst100characters of a message header is enough for correct spam classification.Results for other subsets are consistently good, however they are seemed to more susceptible to spammers tricks.Among4string kernels,the outstanding performance of N-Gram and Subse-quence kernel functions proves that the classification with substring/subsequence kernels is more suitable for spam detection than the kernels using whole string, such as Edit Distance.The Bag of word kernel is seen problematical on spam536Z.Michlovsk´y et al.detection,this could be explained that these spammers normally use the same words for both spam and ham similarity evaluation.4.2Network Application AuthenticationIn this experiment,we used data from tcp network traffic produced by com-mon network applications using different communication protocols like http, https,imap,pop3,ssh,ftp and some applications using dynamic ports like Bit-torent.All network data was captured,and sorted by program Wireshark[15] into separatedfiles according to protocols and then split into individualflows (connections)by program tcpflow[16]see Fig.1.Fig.1.Schema of data preparation for network application work traffic data are sorted into separatedfiles according to protocol with program Wireshark [15]and then split into individualflows using program tcpflow[16].At preprocessing stage,we removed the traffic data in unreliable protocols like UDP,and reorder traffic data by the type of connections using program tcpflow[16].All connections have been shorten to the length of450bytes and750 bytes respectively,where connections shorter than450/750bytes are normalized by repeating its content.Every connection is labelled as its connection port number within the interval of 1,49151 .For connections using dynamic ports or the same port as other applications,we labelled the connections with some unoccupied ports.For example,Http,Msnms and Ocsp connections are noticed using the same port80.To avoid repeat,we label Msnms and Ocsp as port6 and8,respectively.The summary of all connections is given in Table2.Table3presents the best percentage accuracies obtained by4types of string kernel.Percentage Gen.Acc.is recorded as a ratio of the correctly classified connection to total connection.Parameters Lambda and C have been gained from previous5-cross validation tests.Parameter Substring length represents the length of contiguous(non-contiguous)substring for N-gram(Subsequence) kernel function.String Kernel Based SVM for Internet Security Implementation537Table2.Number of connections(flows)for each protocol for training and testingProtocol name(port number)Class label Training set Testing setHttp(80)80675235BitTorrent(dynamic)429999Https(443)44329290Imap(993)9934214Pop3(110)9933210Aol(5190)5190217Ftp(21)21124Msnms(80)694Microsoft-ds(445)44593Ocsp(80)883Ssh(22)2262Pptp(1723)172321Sum1407472Table3.The best results of network application classification for each kernel function Kernel Function Para m eters Gen.AccSubstring length Connection length C of C-SVC La m bda Subsequence kernel4750655360.599.58% N-gra m kernel445040960.599.58% Edit distance kernel-450160.00199.15% Bag of word kernel-75010.2549.75% As seen,Subsequence and N-gram kernel functions give the best99.58%gen-eral accuracy,which follows that only2of total472connections are misclassified (two Msnms connections are misclassified as Http connections).It is worth not-ing that N-gram kernel function performs extremely well for distinguishing those network applications with shorter(450chars)connection instance.However,the Bag of word kernel function is unable to recognize network applications because this kernel distinguishes network applications based on word similarity,when the word(continuous sequence of characters between any two gaps)represents a whole network connection,it is often too long for the kernel to differentiate applications.Fig.2discloses the relationship between the substring length and general accuracy over networkflows(connections)with length450and750chars,re-spectively.As seen,the general accuracy decreases over the length of substring for both N-gram and Subsequence kernels.This suggests that an longer substring normally leads to a decreased classification performance from the presented two string kernel functions.In general,results in Table3and Fig.2have proved the effectiveness of ap-plying Edit Distance,Subsequence and N-gram kernel functions for recognizing hidden network traffic including encrypted connections.Edit Distance kernel538Z.Michlovsk´y et al.Fig.2.Relation between the substring length and general accuracy for networkflows (connections)with length450and750chars,respectivelyidentifies a global similarity of the string(connection),but it surprisingly gives a99.15%authentication accuracy.This implies that functions based on complete string comparison are also suitable for network application recognition to some extent.5Conclusions and Future WorkIn this paper,we propose a new network security technique for spam and hidden network application detection.Our technique is based on known string kernel functions with precomputed kernel matrices.In our implementation of the pro-posed technique,we used support vector machines with optimal configuration associated for each kernel function.This paper makes two major contributions.First,a study of Spam detec-tion.Our email classification results presents excellent ability of string kernels SVM to detect spam from relevant emails.Kernel functions using substrings/ subsequences are evidently less susceptible to spammers tricks than function which comparing the whole strings.As seen,the best classification accuracy has been reached for Header email subset and thisfinding proves the email header has critical influence on the spam classification.Second,a detection of hidden network application traffic.Our experimental analysis shows that sizes offirst 450B of TCP connection are capable for accurate distinguishing network appli-cations.From results is evident the suitable functions for application recognition from TCP connections are N-Gram,Subsequence and Edit Distance function. Our experiments prove the best results in network application recognition was gained with short length of substrings in function parameter.For future work,we will continue to develop new string kernels,and address specially next network security tasks including testing memory dump and net-work intrusion detection.String Kernel Based SVM for Internet Security Implementation539 References1.Chang,C.-C.,Lin,C.-J.:LIBSVM:a library for support vector machines(2001),.tw/~cjlin/libsvm2.Shawe-Taylor,J.,Cristianini,N.:Kernel Methods for Pattern Analysis.CambridgeUniversity Press,New York(2004)3.Duda,R.O.,Hart,P.E.,Stork, D.G.:Pattern Classification,2nd edn.Wiley-Interscience,Hoboken(2000)4.Charras,C.,Lecroqk,T.:Sequence comparison(1998),http://www-igm.univ-mlv.fr/~lecroq/seqcomp/index.html5.Lodhi,H.,Saunders,C.,Shawe-Taylor,J.,Cristianini,N.,Watkins,C.:Text clas-sification using string kernels.J.Mach.Learn.Res.2,419–4446.Fisk,M.,Varghese,G.:Applying Fast String Matching to Intrusion Detection(September2002)7.Aizerman,A.,Braverman,E.M.,Rozoner,L.I.:Theoretical foundations of the po-tential function method in pattern recognition learning.Automation and Remote Control25,821–837(1964)8.Boser,B.E.,Guyon,I.M.,Vapnik,V.N.:A training algorithm for optimal marginclassifiers.In:COLT1992:Proceedings of thefifth annual workshop on Computa-tional learning theory,pp.144–152.ACM,New York(1992)9.Yuan,G.-X.,Chang,C.-C.,Lin,C.-J.:LIBSVM:libsvm experimental code forstring inputs,http://140.112.30.28/~cjlin/libsvmtools/string/libsvm-2.88-string.zip 10.Scarfone,K.,Mell,P.:Guide to intrusion detection and prevention systems(idps).In:NIST:National Institute of Standards and Technology(2007), /publications/nistpubs/800-94/SP800-94.pdf11.Vapnik,V.N.:The nature of statistical learning.Springer,New York(1995)12.Cristianini,N.,Shawe-Taylor,J.:An Introduction to Support Vector Machines andOther Kernel-based Learning Methods.Cambridge University Press,Cambridge (2000)13.Caswell,B.,Beale,J.,Foster,J.C.,Faircloth,J.:Snort2.0Intrusion Detection.Syngress(2003),http://www.amazon.ca/exec/obidos/redirect?tag=citeulike09-20&path=ASIN/193183674414.Whitman,M.E.,Mattord,H.J.:Principles of Information Security.Course Tech-nology Press,Boston(2004)bs,G.,et al.:Wireshark:network protocol analyzer,/16.Elson,J.:tcpflow:tcpflow reconstructs the actual data streams and stores eachflow in a separatefile for later analysis,/jelson/software/tcpflow/17.Bogomolny,A.:Distance Between Strings,/doyouknow/Strings.shtml18.SpamAssassin public mail corpus,/publiccorpus/19.Spam dataset,http://www.em.ca/7Ebruceg/spam/i,C.-C.:An empirical study of three machine learning methods for spamfiltering.Knowledge-Based Systems20,249–254(2007)。