Entropies of complex networks with hierarchically constrained topologies

a r X i v :0803.1247v 1 [c o n d -m a t .d i s -n n ] 8 M a r 2008Entropies of complex networks with hierarchically constrainedtopologiesGinestra Bianconi 1,Anthony C.C.Coolen 2,3,Conrad J.Perez Vicente 41Abdus Salam International Center for Theoretical Physics,Strada Costiera 11,34014Trieste,Italy2Department of Mathematics,King’s College London,The Strand,London WC2R 2LS,United Kingdom3Randall Division of Cell and Molecular Biophysics,Kings College London,New Hunts House,London SE11UL,United Kingdom4Departament de F´ısica Fonamental,Facultat de F´ısica,Universitat de Barcelona,08028Barcelona,SpainAbstractThe entropy of a hierarchical network topology in an ensemble of sparse random networks,with ”hidden variables”associated to its nodes,is the log-likelihood that a given network topology is present in the chosen ensemble.We obtain a general formula for this entropy,which has a clear interpretation in some simple limiting cases.The results provide new keys with which to solve the general problem of “fitting”a given network with an appropriate ensemble of random networks.PACS numbers:89.75.Hc,89.75.Fb,75.10.NrI.INTRODUCTIONThe entropy is a key concept in information theory[1]and in the theory of dynamical systems[2].In information theory,the problem of inference of a probability distribution on the basis offinite number of independent observations is usually addressed using the maximum likelihood principle or via the minimization of the Kullback-Leibler distance be-tween the given(empirical)distribution and the inferred one.Recently,several studies have extended the tools of information theory along these lines in order to measure the perfor-mance offiltering procedures of correlation matrices in the case of multivariate data[3,4]. In the framework of graph theory the large deviations of the ensemble of random Erd¨o s and Renyi graphs where derived by studying the free energies of statistical mechanics models defined on them[5,6].There is now increased interest,in the community of complex net-works[7,8,9],in the definition of entropy measures that are related with the networks’topological structure[10]or with diffusion processes defined on them[11].The inference problem applied to complex networks can be formulated as the identification of the ensemble of networks which retains the essential structural characteristics and complexity of a given real network realization.The identification of this ensemble,is an activefield of research. One aims tofit a given specific network with a suitable network ensemble that retains some information on its structure.Newman has proposed this approach tofind the community structure in a given network[12].Later,this method has been extended to define ensembles of networks that have other topological characteristics in common with the real network, such as the degree sequence and/or the degree correlations.As we add further features that a desired ensemble is to have in common with a given real network,we effectively consider ensembles with decreasing cardinality.The cardinality of an ensemble of networks with a given topology has attracted the attention of the graph theory community[13,14,15],and more recently also of the statistical mechanics community[10].In this paper we evaluate the entropy of a given hierarchical topology in a”canonical”or”hidden variable”ensemble,i.e.we calculate the normalized logarithm of the probability that a given topology appears in this ensemble.By hierarchical topology we will mean the set of the generalized degrees of the nodes,defined as the sequence k i=(k1i,k2i,...,k L i)of the number of nodes at distance1,2,...,L from the node i.The”canonical”or“hidden variable”[16,17,18,19,20]ensembles are generalization of the G(N,p)ensemble for het-erogeneous nodes.The hetereogeneity of the nodes is described in terms of some”hidden variables”x i,defined on each node i of the network,and the probability p ij of a link between a node i and a node j is not p as in G(N,p)but it is a general function Q(x i,x j)of the hidden variables at i and j nodes.These ensembles correspond to networks which satisfy soft constraints,for example the degree of a node is notfixed,but only the average degree of each node isfixed,allowing for Poissonianfluctuations.We derive a general formula for the entropy of a given topology in a”canonical”ensemble using ideas and methods from the study of diluted combinatorial optimization problems and statistical mechanical systems on sparse networks[21,22,23,24,25,26,27,28,29,30,31, 32,33,34,35,36,37,38,39,40,41].In the simple case where we study the likelihood of a degree distribution of a network belonging to the chosen ensemble the entropy is found to be the Kullback-Leibler distance between the probability distribution of the degrees and the expected probability of the typical topology of the network.The paper is structured as follows:in section II we introduce the definition of the problem, in section III we provide the asymptotic entropy expression of the network topology in a given ensemble,in section IV we study the form that the entropy takes in special and relevant cases,and the conclusions are presented in section V.II.FORMULATION OF THE PROBLEM AND DEFINITIONSTo model the essential properties of a real network it is useful to think of it as an instance of an ensemble of networks.The ensemble can be either”microcanonical”or”canonical”depending on whether the networks in the ensemble are subject to hard or soft constraints. The main example of what we call a”microcanonical”ensemble is G(N,M)where the number of links isfixed to be exactly M,and the main example of”canonical”ensemble is G(N,p)in which only the average number of links M =pN(N−1)/2isfixed.These ensembles can be generalized to ensembles of random graphs with a given degree sequence and with a given hidden variable distribution.In this paper we will calculate the entropy of a given network topology(defined in terms of its hierarchical structure)in a general ”canonical”ensemble.This entropy is defined as the probability that the given network topology is found in the”canonical”network ensemble under consideration.A.“Canonical”ensemblesWe consider networks characterized by N nodes(or‘sites’)labeled i=1,...,N,and a symmetric matrix c with entries c ij∈{0,1}that specify whether(c ij=1)or not(c ij=0)nodes i and j are connected.We choose c ii=0for all i.We write the set of all such undirected networks as G={0,1}1N Q(x i,x j)δcij,1+(1−cthemselves,or by giving the collective generalized degree statistics,conditioned on the values of the hidden variables,i.e.P(k|x,c)=P(k1,...,k L|c)=1log c∈G W(c) iδk i,k i(c)(5)N1constrain statistics:ΩL[P]=log k1...k N k,xδ P(k|x)− iδk,k iδ[x−x i]N(6)but absent from (5).We evaluate (5,6)by writing each Kronecker δand each δ-functionin integral form.Upon defining the short-hands k 0i =1for all i ,and ωi ·k i = L ℓ=1ωℓi k ℓi ,expression (3)allows us to simplify the term δk ,k i (c )toδk i ,k i (c )=π−πd ωie i ωi ·k iN logπ−π id ωie i ωi ·k iNlogk ,xd ˆP (k |x )e iN ˆP (k |x )P (k |x )N lim ∆→0log k ,xd ˆP (k |x )e iN ∆ˆP (k |x )P (k |x )(2π)LD [{ω,k }](10)The core of the problem is apparently to calculate the function D [{ω,k }]in (8),whichinvolves the introduction of a measure W (ω,k ,x |{ω,k })=N −1i δk ,k i δ[x −x i ]δ[ω−ωi ]:D [{ω,k }]= i<j1+c2cNdxdx ′Q (x,x ′)π−πd ωd ω′kk ′W (ω,k ,x |...)×W (ω′,k ′,x ′|...)e−i P L ℓ=1[ωℓk ′ℓ−1+ω′ℓk ℓ−1]−1N/2π],resulting inD [{ω,k }]={dW d ˆW}e iNR π−πd ωdxPk ˆW (ω,k ,x )W (ω,k ,x )+O (N0)×e12cN −iP i ˆW (ωi ,k i ,x i )(12)Now only the last line contains microscopic variables,and it factorizes fully over the nodes of the network.Upon inserting(12)into(9)and(10)this allows us to evaluate both expressions for N→∞via steepest descent integration over the distributions W(ω,k,x),leading toΩL[{k}]=extr{W,ˆW}Ψ1[{W,ˆW}](13)ΩL[P]=extr{W,ˆW,ˆP}Ψ2[{W,ˆW,ˆP}](14)with the functionsΨ1[{W,ˆW}]=i π−πdωdx kˆW(ω,k,x)W(ω,k,x)+Φ[{W}]+ dx p(x) k P(k|x)log π−πdω(2π)L k e i[ω·k−ˆP(k|x)/p(x)−ˆW(ω,k,x)](16) whereΦ[{W}]=−12c dxdx′Q(x,x′) π−πdωdω′ kk′W(ω,k,x)W(ω′,k′,x′)×e−i P Lℓ=1[ωℓk′ℓ−1+ω′ℓkℓ−1](17)It will be convenient to introduce new functions Q(k|x)=exp[−iˆP(k|x)/p(x)]and V(ω,k,x)=exp[−iˆW(ω,k,x)]so that our saddle-point equations simplify toΩL[{k}]=extr{V,W}˜Ψ1[{V,W}](18)ΩL[P]=extr{Q,V,W}˜Ψ2[{Q,V,W}](19)with the functions˜Ψ1[{V,W}]=Φ[{W}]− π−πdωdx k W(ω,k,x)log V(ω,k,x)+ dx p(x) k P(k|x)log π−πdω˜Ψ[{Q,V,W}]=Φ[{W}]− π−πdωdx k W(ω,k,x)log V(ω,k,x)2− dx p(x) k P(k|x)log Q(k|x)+ dx p(x)log k Q(k|x) π−πdωπ−πdω′V(ω′,k,x)e iω′·k(23) For˜Ψ2[...](referring to ensembles with constrained distributions of generalized degrees) these are found to be the following:log V(ω,k,x)=c dx′Q(x,x′) π−πdω′ k′W(ω′,k′,x′)e−i P Lℓ=1[ωℓk′ℓ−1+ω′ℓkℓ−1](24)p(x)Q(k|x)V(ω,k,x)e iω·kW(ω,k,x)=k′Q(k′|x) π−πdωV(ω,k′,x)e iω·k′(26) The last equation is easily solved,viz.P(k|x)Q(k|x)=×π−πd ω′V (ω′,k ′,x ′)e i ω′·k ′−iP Lℓ=1[ωℓk ′ℓ−1+ω′ℓk ℓ−1]π−πd ωexp i ω·k ′+c ξ′∈I N L γ(k ′,ξ′,x ′)e −i ω·ξ′ The two integrals over ωin the latter fraction can be done.Both are of the formI (k ,k ′,x )= π−πd ωexp i ω·k +c ξ′∈I N Lγ(k ′,ξ′,x ′)e −i ω·ξ′ =(2π)Lm ≥0c m m ! ξ1...ξm ∈I NL m n =1γ(k ′,ξn ,x ′) L ℓ=1δk ′ℓ,k ℓ−1+P n ≤m ξn ℓm ! ξ1...ξm ∈I N Lm n =1γ(k ′,ξn ,x ′) δk ′,P n ≤m ξn=δξ1,1 dx ′p (x ′)Q (x,x ′)k ′k ′1 ξ1...ξk ′1 k ′1n =1γ(k ′,ξn ,x ′) δk ′,Pn ≤k ′1ξn(33)where we use the conventions that [ m n =1u n ]m =0≡1,[ mn =1u n ]m =0≡0,and [ξ1...ξm u (ξ1,...,ξm )]m =0≡1.If L =1we have k =→k and ξ→1,so γ(k ,ξ,x )→γ(k,x ).This describes the situation where the degrees are not generalized,but measure as usual onlythe number of direct links per node.Here our equation forγ(...)simplifies drastically to L=1:γ(k,x)= dx′p(x′)Q(x,x′) k′k′P(k′|x′)cγ(x′) k kP(k|x′)(35) If L>1we can manipulate at most some further Kroneckerδs,and thefinal form is thereforeγ(k,ξ,x)=ξ2δξ1,1ξ1...ξξ2 ξ2n=1γ((ξ2,...,ξL,k′),ξn,x′) δk′,P n≤ξ2ξn L L−1ℓ=1δξℓ+1,P n≤ξ2ξnℓC.Simplification of the asymptotic entropy formulasAt this stage we insert our previous results for the kernels{V,W,Q}into(18,19,20,21) to arrive at more explicit expressions for the asymptotic entropies,which will only involve the functionγ(k,ξ,x)of(36).Thefirst step is to substitute expression(27)into(21).This leads,in combination with the fact that at the relevant saddle-points the kernels{V,W} obey identical equations for the two cases(constrained generalized degrees versus constrained statistics of generalized degrees),to a simple and natural relation between our two entropies:lim N→∞ΩL[P]=limN→∞ΩL[{k}]− dx p(x) k P(k|x)log P(k|x)(37)The extra freedom to construct microscopic network realizations in the case where we only constrain the generalized degree distribution,as opposed to constraining the actual values of the generalized degrees,is measured by the Shannon entropy of the imposed distribution P(k|x).So we only need to express lim N→∞ΩL[{k}]in terms of the functionγ(k,ξ,x).We first note that at the relevant saddle-point the functionΦ[{W}](17)takes the valueΦ[{W}]=−12 dxπ−πdωkW(ω,k,x)log V(ω,k,x)(38)Insertion into(20),followed by elimination of W(ω,k,x)via(23),leads us to limN→∞ΩL[{k}]= dx p(x) k P(k|x)log π−πdω2c−1π−πd ωV(ω,k ,x )e i ω·k(39)The final step is the elimination of V (ω,k ,x )via (29),followed by integration over ω,using the property that γ(k ,ξ,x )=0unless ξ1=1:π−πd ωm !ξ1...ξm n ≤mγ(k ,ξn,x ) δk ,Pn ≤mξn =δk ,0+c k 1θ[k 1−1k 1! ξ1...ξk 1 n ≤k 1 γ(k ,ξn ,x ) δk ,Pn ≤k 1ξn (40)π−πd ω(m −1)! ξ1...ξm n ≤mγ(k ,ξn ,x ) δk ,Pn ≤m ξn =c k 1θ[k 1−1(k 1−1)! ξ1...ξk 1n ≤k1γ(k ,ξn ,x ) δk ,P n ≤k 1ξn (41)So one arrives at the compact result,where we have used the fact that if k 1=0then k ℓ=0for all ℓ(which follows from the definition of the generalized degrees):lim N →∞ΩL [{k }]=k 1P (k 1)log πc (k 1)(42)+dx p (x )kP (k |x )logξ1...ξk 1n ≤k 1γ(k ,ξn,x )δk ,Pn ≤k 1ξn with the average-c Poissonian degree distribution πc (k )=c k e −c /k !.If we introduce the Fourier transforms of γ(k ,ω,x )defined asγ(k ,ω,x )=ξγ(k ,ξ,x )e i ω·ξ(43)we can write the entropy (42)equivalently aslim N →∞ΩL [{k }]= dx p (x )kP (k |x )log d ωe −i ω·ξ+cγ(k ,ω,x )πcγ(k ,ω,x )(k 1)IV.APPLICATIONS OF THE GENERAL THEORYA.Regular random graphsOurfirst application domain is that of r-regular degree distribution P(k|x)=δk,k(r), with k(r)=(r,r2,...,r L).Here one can solve(36)explicitly:γ(k,ξ,x)=ξ2δξ1,1ξ1...ξk1(x′) r n=1γ(k(r),ξn,x′) Lℓ=1δrℓ,P n≤rξnℓThe solution is seen to be of the formγ(k,ξ,x)=γ(k,x)δξ,(1,r,r2,...,r L−1),and independent of k L.Insertion of this form into the above equation then givesγ(k,x)=rγ(k(r),x′)(45)We conclude thatγ(k,ξ,x)=γ(x) L−1 ℓ=1δkℓ,rℓ L ℓ=1δξℓ,rℓ−1 (46) whereγ(x)is the solution ofγ(x)=rγ(x′)(47)For the entropies(37,42)one thenfindslim N→∞ΩL[P]=limN→∞ΩL[{k}]=logπc(r)+r dx p(x)logγ(x)(48)As expected,the two entropies are identical(since for regular graphs there is no entropy contribution from degree permutations)and independent of L(since upon specifying that the degrees are r-regular,the full distributions P(k|x)are uniquely specified for any L).In the special case Q(x,x′)=1of uncorrelated degrees the above solution simplifies further. Nowγ(x)=2r log(r/c)(49)B.The case L =1For L =1we have already simplified our formula for the function γ(k ,ξ,x )to relation (35)forasimplefunction γ(x ).We can do the same for expression (42)for the entropy,which giveslim N →∞Ω1[{k }]=kP (k )log πc (k )+dx p (x )k P (k |x )log γk (x )(50)lim N →∞Ω1[P ]=kP (k )log πc (k )−dx p (x )kP (k |x )log[P (k |x )/γk (x )](51)with πc (k )=c k e −c /k !,and where γ(x )is to be solved fromγ(x )= dx ′p (x ′)Q (x,x ′)k =kkP (k ).Expression (50)now becomes Q (x,x ′)=1:lim N →∞Ω1[{k }]=kP (k )log πc (k )+1k log(2k/c )(53)So,if one also choosesp (x )k (x ),withk (x )=kkP (k |x )thendx ′Q (x,x ′)f (x ′)=λf (x ),f (x )=λ2dx p (x )k (x )log[λp (x )k (x )/c ]= k P(k)logπc(k)+12k= dx p(x)k(x)).Let us next discuss some example kernels Q(x,x′)for which γ(x)can be solved explicitly,either directly,or via the above procedure based on using eigenfunctions of Q(.,.):•First example:Here we assume Q(x,x′)to be such that the conditional connectivities k(x)= k kP(k|x)are the typical ones for the ensemble(1),which implies thatk(x)=c dx′Q(x,x′)p(x′)(56) In this case(52)has the solutionγ(x)=k(x)/c,which leads to the following simple expression for the entropies:Ω1[{k}]= dx p(x) k P(k|x)logπk(x)(57) limN→∞limΩ1[P]=− dx p(x) k P(k|x)log[P(k|x)/πk(x)](58) N→∞This indicates that in this case the entropy lim N→∞Ω1[P]takes the form of an integral over p(x)of the Kullback-Leibler distance between the probabilities P(k|x)and the Possion distributionπk(x).We note that for the hidden variable model the typical degree distribution of the nodes with hidden variable x is indeedπk(x)[20].•Second example:1Q(x,x′)=a0+a1δ(x−x′),k(x)=1−a1 dx p2(x)+1c log 1−a1 1−1dx p2(x)+12•Third example:Q (x,x ′)=g (x )+g (x ′)g 2 0+g (x )22 1−1dx φ(x ),and with g (x )≥0for allx ∈[−1,1].Here one finds the solutionγ(x )=1cg 2 0 g 2 0+g (x ),λ= g 0+dx p (x )g (x )(63)C.The case L =2Here we have to find first the solution of (36),which now reduces to γ((k 1,k 2),(1,ξ),x )=ξξ1...ξξξn =1γ((ξ,k ′),(1,ξn ),x ′)δk ′,P n ≤ξξn(64)We observe that the right-hand side is independent of k 2,so the solution of our equation must have the form γ((k 1,k 2),(1,ξ),x )=γ(k 1,ξ,x ),whereγ(k,ξ,x )=ξξ1...ξξξn =1γ(ξ,ξn ,x ′)δk ′,P n ≤ξξn.(65)The entropy would becomelim N →∞Ω2[{k }]=k 1P (k 1)log πc (k 1)+dx p (x )k 1k 2P (k 1,k 2|x )logξ1...ξk 1n ≤k 1γ(k 1,ξn ,x )δk 2,P n ≤k1ξn(66)Let us limit ourselves to the simplest scenario where there are no degree correlations,i.e.Q (x,x ′)=1.Here we have γ(k,ξ,x )=γ(k,ξ),and we need only the generalized degreestatistics P (k 1,k 2)=dx p (x )P (k 1,k 2|x ).Our formulae thereby reduce toγ(k,ξ)=ξξ1...ξξξn =1γ(ξ,ξn )δk ′,P n ≤ξξn(67)limΩ2[{k}]= k1P(k1)logπc(k1)(68) N→∞+ k1k2P(k1,k2)log ξ1...ξk1 n≤k1γ(k1,ξn) δk2,P n≤k1ξnHere one observes the validity of the following simple relation:k2γ(k1,k2)γ(k2,k1)=k1[3]M.Tummiello,F.Lillo and R.N.Mantegna,Phys.Rev.E76,031123(2007).[4]G.Biroli,J.-P.Bouchaud and M.Potters,arXiv:0710.0802[cond-mat](2007).[5] A.Engel,R.Monasson and A.K.Hartmann,Jour.Stat.Phys.117,387(2004).[6]O.Rivoire,Jour.Stat.Phys.117,453(2004).[7]R.Albert and A.-L.Barab´a si,Rev.Mod.Phys.74,47(2002).[8]S.N.Dorogovtsev,A.Goltsev and J.F.F.Mendes,arXiv:0705.0010[cond-mat](2007).[9]S.Boccaletti,tora,Y.Moreno,M.Chavez and D.U.Hwang,Phys.Rep.424,175(2006).[10]G.Bianconi,Europhys.Lett.81,28005(2008).[11]J.Gomez-Gardenes and tora,arXiv:0712.0278[cond-mat].[12]M.E.J.Newman and E.A.Leicht,PNAS104,9564(2007).[13] E.Bender and E.Rodney Canfield,bin.Theory Ser.A24,296(1978).[14] B.D.McKay,Ars Combin.19A,15(1985).[15] B.Pittel and N.Wormald,bin.Theory Ser.A101,249(2003).[16]J.Park and M.E.J.Newman,Phys.Rev.E70,066146(2004).[17] B.Sodeberg,Phys.Rev.E66,066121(2002).[18] F.Chung and L.Lu,PNAS99,15879(2002).[19]G.Caldarelli,A.Capocci,P.De Los Rios and M.A.Mu˜n oz,Phys.Rev.Lett.89,258702(2002).[20]M.Bogu˜n´a and R.Pastor-Satorras,Phys.Rev.E68,036112(2003).[21]R.Monasson,J.Phys.A31,513(1998).[22]R.Monasson and R.Zecchina,Phys.Rev.E56,1357(1997).[23]R.Monasson and R.Zecchina,J.Phys.A:Math.Gen.31,9209(1998).[24]R.Monasson,R.Zecchina,S.Kirkpatrick,B.Selman and L.Troyansky Nature400,133(1999).[25]M.Weigt and A.K.Hartmann,Phys.Rev.Lett.84,6118(2000).[26]R.Mulet,A.Pagnani,M.Weigt and R.Zecchina,Phys.Rev.Lett.89,268701(2002).[27]M.Leone,A.V`a zquez,A.Vespignani and R.Zecchina,Eur.Phys.Jour.B28,191(2002).[28] B.Wemmenhove and A.C.C.Coolen,J.Phys.A:Math.Gen.36,9617(2003).[29]I.P´e rez-Castillo and N.S.Skantzos,J.Phys.A:Math.Gen.37,9087(2004).[30]I.P´e rez-Castillo,B.Wemmenhove,J.P.L.Hatchett,A.C.C.Coolen,N.S.Skantzos and T.Nikoletopoulos,J.Phys.A:Math.Gen.37,8789(2004).[31]T.Nikoletopoulos T,A.C.C.Coolen,I.P´e rez Castillo,N.S.Skantzos,J.P.L.Hatchett andB.Wemmenhove J.Phys.A:Math.Gen.37,6455(2004).[32]T.Nikoletopoulos and A.C.C.Coolen,J.Phys.A:Math.Gen.37,8433(2004).[33]J.P.L.Hatchett,I.P´e rez-Castillo,A.C.C.Coolen and N.S.Skantzos,Phys.Rev.Lett.95,117204(2005).[34]H.Hansen-Goos and M.Weigt,J.Stat.Mech.Theory Exp.P08001(2005).[35] A.C.C.Coolen,N.S.Skantzos,I.P´e rez Castillo,C.J.P´e rez Vicente,J.P.L.Hatchett,B.Wemmenhove and T.Nikoletopoulos,J.Phys.A:Math.Gen.38,8289(2005).[36]N.S.Skantzos,I.P´e rez Castillo and J.P.L.Hatchett,Phys.Rev.E72,066127(2005).[37]N.S.Skantzos and J.P.L.Hatchett,Physica A381,202(2007).[38] A.Mozeika and A.C.C.Coolen,preprint arXiv:0712.1139(2007).[39]L.Zdeborova and F.Krzakala,Phys.Rev.E76,031131(2007).[40]J.Raymond and D.Saad,J.Phys.A:Math.Gen.40,12315(2007).[41]R.C.Alamino and D.Saad,J.Phys.A:Math.Gen.40,12259(2007).。