ScopeCast A Service for Consistent Peer-to-Peer Multicasts

ScopeCast:A Service for Consistent Peer-to-Peer MulticastsX.B.Shen Y.C.TayNational University of SingaporeOctober26,2003Keywords:distributed systems,peer-to-peer multi-cast,causal ordering.AbstractPeer-to-peer systems can benefit from a service that delivers multicasts in a consistent manner to appli-cations and other services.ScopeCast offers such a service.It does not assume membership is static or known,nor require logical or physical infrastructure for support.It allows users and applications to force (manually or via timeouts,etc.)delivery of received messages when congestion or failures are detected. The protocol resumes consistent delivery thereafter. 1IntroductionWhen a message arrives at a node in a multicast group,its delivery to the application may need to be delayed to ensure that all nodes have a consistent view of the communication[AN].The delays slow down communication,and other costs include mem-ory(to store messages and metadata)and bandwidth (to carry control data).Nonetheless,such multicast ordering provides an abstraction that facilitates the design of distributed systems(e.g.fault tolerance[B],virtual memory [AHJ],application monitor[RST]),and was used in several systems(e.g.Psync[MPS],ISIS[BSS]).How consistency is defined determines the order-ing for the multicast delivery(Atomic,FIFO,Causal, etc.[HT]),and various techniques have been used todesign the delivery protocol(logical clocks[EMS], causality graphs[DKM],message logging[KD],to-kens[RM],vectors[BSS]and matrices[RST]).Like most distributed systems,peer-to-peer ap-plications need consistent views of the network state (e.g.replicatedfiles in Freenet[C],replicated root in Bayeux[Z]or location and routing information in Pastry[RD]).There is,however,one important difference: Multicast membership changes frequently as peers continually join and quit.Peer-to-peer multicasts therefore cannot assume membership is static or known globally(e vector clocks).The problem is exacerbated for many peer-to-peer systems,since the underlying wide-area,multi-hop networks make out-of-order message reception and concurrent join or quit events more likely[AN]. Some techniques that are suitable for local-area net-works(e.g.tokens)also become impractical.For this setting,we propose ScopeCast:a ser-vice for consistent peer-to-peer multicast delivery. ScopeCast does not require support from some logi-cal or physical infrastructure[AN,YHH],and a mul-ticast may be viaflooding or through some mecha-nism(e.g.multicast tree).ScopeCast’s consistency definition includes uni-casts and is,essentially,causal multicast[BSS]after taking dynamic group membership into account.The protocol inserts a peer’s view of the multicast frontier [BM]in an outgoing message’s header;the protocol uses this information to decide whether a received message is ready for delivery to the application.Failures or congestion in the network can cause 1an unacceptable delivery delay or buffer overflow. When this happens,the user or application can force the(inconsistent)delivery of received messages. Such a disruption does not affect consistent delivery at other peers;moreover,ScopeCast resumes consis-tent delivery for the disrupted peer.We state the consistency definition in Section2. Section3then presents the protocol and describes how it allows for a temporary break from consis-tency.We prove the protocol is correct in Section 4,and conclude with a summary in Section5.2Consistency DefinitionWefirst use events and messages as abstractions to describe the system,then define consistency.2.1Events and MessagesWe are interested in these events: Join()Quit()Send(,)Receive(,)Deliver(,)Discard(,)the-th message sent bythe Msg()received byIn particular,Join()is announced by the multicast event Send(,Msg(,0)).After joins a multicast group,it may receive some old messages that it can discard.The ones that matter are those that arrive at from senders whom knows have joined,or who know has joined,i.e. Scope()(Deliver(Msg(.Sender,0),) Deliver(Msg(),.Sender)Send(.Sender,)) (Receive()Quit()).Scope is thus a messagefilter that ScopeCast uses to take dynamic membership into account.2.2ConsistencyFigs.1and2illustrate the motivation for our consis-tency definition.The following is illustrated in Fig. 3,where may be a unicast or multicast.2BCD"(a) File f is copied from C to D , then deleted at C after thetransfer is acknowledged. If the acknowledgement is delivered at C before the multicast, the search may fail to find f .C andD have f .(b) An object f is passed (not copied) from peer to peer.f If send( ) is delivered at D before the multicast,B may conclude that both BCDFigure 2:Search problem:inconsistent interleav-ing of unicast with multicast.Consistency Definition Consider message sent by peer .Deliver()is consistent if and only if for any multicast in Scope(),Deliver()Send()implies Deliver()Deliver().Thus,the immediate delivery of received messages will be inconsistent in Figs.1and 2.The offendingmessage is from to in Fig.1,the acknowledge-ment in Fig.2(a)and send in Fig.2(b).Since Send()Deliver()and there are no intermediate events at be-tween them,this definition is the same as the common definition for causal multicasts [PRS],i.e.Send()Send()implies Deliver()Deliver(),except our defini-mcast .PDeliver(M ,P ) Deliver( )M,P Figure 3:Consistency definition.tion orders delivery only if is a multicast thatis in Scope .In particular,we allow non-FIFO delivery if and are both unicasts.One can easily use sequence numbers to impose FIFO unicast delivery.The following result justifies the definition:PropositionIf a request is multicast and delivered consistently (according to the above definition),and each peer replies with its state at delivery time,then the repliesconstitute a global snapshot that is consistent (by Chandy and Lamport’s definition [CL]).Since ScopeCast delivers a peer ’s join an-nouncement consistently (see Theorem),this result implies that the join acknowledgements yield a con-sistent time cut [BM]that defines scope().3ScopeCast:Description and Costs Each peer has a unique identifier PeerID .Each mes-sage carries a sequence number .SeqNum ,which is 0for join announcements,incremented for every multicast but unchanged for unicasts.For each peer ,ScopeCast maintains a mes-sage queue OnHold ,and two lists:History contains [PeerID,SeqNum]records for the latest messages from PeerID that are delivered at ;Frontier con-3tains those History records for multicasts that are not causally preceded by other multicast delivered at.3.1Protocol DescriptionThe pseudocode for the main ScopeCast routines is shown in Fig.4(a),and the utilities in Fig.4(b). 3.2Costs and ControlPeer’s History keeps just one[,SeqNum]pair in local storage for each that sent to—this is cheap.The size of Frontier depends on the com-munication pattern;each delivered message adds one pair,but may also remove others;moreover,Frontier is reset(line9)whenever sends a multicast.ScopeCast can be tuned in various ways.If’s Frontier reaches a(say,user-specified)threshold in size,ScopeCast canflush it by multicasting a NULL message.Similarly,if the message queue overflows, or the time between message arrival and delivery reaches a threshold,ScopeCast can notify the user andflush(i.e.deliver)all or part of the queue.A lengthy queue or long delay may be caused by an overload or some failure in the network,and aban-doning consistency is reasonable in such cases.When the user or application intervenes byflush-ing,we view them as taking the responsibility to re-solve any resulting inconsistency.ScopeCast there-fore considers theflushed messages—and the miss-ing messages they were waiting for—as delivered consistently.With this assumption,subsequent de-livery by ScopeCast is consistent.A new peer relies on a joining acknowledge-ment(line23)or Frontier records carried by arriv-ing messages to decide if an arrived message is in Scope().If this information is slow in coming,’sOnHold list may grow indefinitely(as if there is a network failure).A new peer may therefore need to flush and accept some initial inconsistency.Flushing at a peer does not affect consistent de-livery at other peers.In fact,a peer can choose to flush every arriving message,i.e.ignore consistency.Such options may help put aside reservations about ordering message delivery.Birman’s review of how real users use multicasts in deployed systems cautions against“excessive transparency”and advo-cates some visibility,so knowledgeable users can make trade-offs[B].4Correctness ProofThis sectionfirst discusses the underlying assump-tions,before proving that ScopeCast is correct.4.1AssumptionsMessage duplication in lower layers is not an issue for ScopeCast,since multicast duplicates are dis-carded(line19and line29),while unicast duplicates do not affect ScopeCast’s consistency and state.Network failures and buffer overflows may cause message loss,but that does not affect ScopeCast’s consistency per se.Failures can prevent messages from being received but(by our definition)there is no correctness issue since the messages are not de-livered.The effect of message loss on ScopeCast is to de-lay delivery;if delivery is forced byflushing,con-sistency is lost for the forced delivery but resumes thereafter(Section3.2).4InitializePeer():SeqNum=;line1 Frontier=NULL;line2 ScopeMulticast(NULL,“ALL”,“JOIN”);//null message to announce JOIN to all peers line3 Quit():ScopeMulticast(NULL,“ALL”,“QUIT”);line4 ScopeMulticast(data,dest,type)://type is UNICAST,MULTICAST,JOIN,ACK or QUIT SeqNum=SeqNum+1;line5 msg=new ScopeCastMsg(myPeerID,dest,SeqNum,Frontier,data,type);//dest is multicast group line6 multicast(msg,dest);//now messages in Frontier precede msg...line7 UpdateHistory(myPeerID,SeqNum);line8 Frontier=[myPeerID,SeqNum];//...so Frontier contains msg alone;line9 ScopeUnicast(data,dest,type)://unicast:SeqNum,History,(multicast)Frontier unchangedmsg=new ScopeCastMsg(myPeerID,dest,SeqNum,Frontier,data,type);line10 unicast(msg,dest);line11 ScopeReceive(msg):if(ProcessMsg(msg))//if processing msg changes History...line12 for(i=onhold.size();i;i)//...then process the messages line13 m=onhold.dequeue();//...that are on hold line14 if(ProcessMsg(m))i=onhold.size()+1;//i is reset,in case ProcessMsg changes History line15ProcessMsg(msg):if(msg.type==“UNICAST”||msg.type==“ACK”)changed=ProcessUnicast(msg)line16else changed=ProcessMulticast(msg)line17 return changed;line18 ProcessMulticast(msg):if(InHistory(msg.Sender,msg.SeqNum))Discard(msg);return FALSE;//old multicast line19 else if(ToHold(msg))onhold.enqueue(msg);return FALSE;line20 else UpdateHistory(msg.Sender,msg.SeqNum);line21 UpdateFrontier(msg);line22 if(msg.type==“JOIN”)Ack(msg);//this ack helps the new peer initialize its History line23 if(msg.type==“MULTICAST”)Deliver(msg.data);//do not deliver QUIT messages line24 return TRUE;line25ProcessUnicast(msg)://sender knows dest,so sender must know dest has joinedif(msg.type==“ACK”)if(History.contains(msg.sender,*))Discard(msg);return FALSE;//History already initializedelse UpdateHistory(msg.Sender,msg.SeqNum);return TRUE;line26 else if(ToHold(msg))onhold.enqueue(msg);return FALSE;line27 else Deliver(msg.data);return FALSE;line28Fig.4(a)Main ScopeCast routines.5InHistory(PeerID,SeqNum):return(History.contains(PeerID,SeqNum)&&(SeqNum SeqNum));line29 ToHold(msg):for(each[PeerID,SeqNum]in msg.Frontier)//have we seen the multicasts that preceded msg?line30 if(!InHistory(PeerID,SeqNum))return TRUE;line31 return FALSE;line32 UpdateHistory(PeerID,SeqNum):record=History.lookup(PeerID);line33 if(record==NULL)History.add([PeerID,SeqNum]);line34 else if(record.SeqNum SeqNum)record.SeqNum=SeqNum;line35 UpdateFrontier(msg):for(each[PeerID,SeqNum]in msg.Frontier)//remove records for messages preceding msg...line36 Frontier.remove([PeerID,SeqNum]);line37 Frontier.add([msg.Sender,msg.SeqNum]);//...because msg is now on the Frontier line38Fig.4(b)Utilities for ScopeCast.4.2ProofsLet and be local times at peer,and Delivered(multicastDeliver.time.Then the con-sistency definition has the following property,whichcan be used for induction:Induction Lemma(See Fig.5)Let Msg,Msg,Send.time and Send.time.Suppose Deliver Deliver and Deliver Deliver for allDelivered Scope.If Deliver is consistent,then Deliver is consistent.Proof As per the consistency definition,con-sider any multicast Scope such that Deliver Send.We must show that Deliver Deliver.Send Deliver:Then(see Fig.5)Delivered. Since Scope as well,we get Deliver Deliver.Consider now a peer’s view of the frontier: Frontier LemmaLet be local time at and let.Frontier be the set of messages for records in.Frontier at time.(i)If has not sent any messages before,then.Frontier.6PQ M"Fig.6Frontier Lemma (ii):.Frontier.(ii)If is the last message sent byat time(see Fig.6),letDelivered DeliveredDeliver .Sender Send .Sender .Then .Frontier Delivered.(iii)If .Frontier has a record for ,then Deliver Deliver for any peer .Proof If has not even sent its join announce-ment,then (i)follows from line2.Otherwise,for (ii),consider when a record [PeerID,SeqNum]for is added to and removed from .Frontier.Addition happens only when is delivered (line22and line38).Removal happens when sends amulticast (line9)or [PeerID,SeqNum]appears in .Frontier for some delivered at (line37).A removed record will not be added to .Frontier again because of the filtering by SeqNum (line19).It follows that .Frontier Delivered for (ii).For (iii),would be on hold (lines 20,27,30and 31)at anyuntil is delivered at .TheoremMessage delivery in ScopeCast is consistent.Proof Consistency for a multicast does not de-pend on preceding unicasts (Fig.3),so we may firstignore the unicasts and consider just the multicasts.Consider the first multicastMsg that is within Scope().If ,then is ’s join announcement;has no preceding multicasts,so Deliver is consistent (line23and line24).Otherwise,is the first multicast by after it acknowledged ’s join announcement.ScopeCast checks every message in .Frontier (line20)to see if it is within scope and already delivered,soDeliveris again consistent.We thus have the base case for an induction on ,sonow consider message Msg .In thefollowing,note (line19)that we only need considerdelivery of messages that are in Scope.Let ,,and be as in the Induction Lemma (Fig.5).ScopeCast imposes first-in-first-out delivery for multicasts fromto (line30),so Deliver Deliver .We firstshow that Deliver Deliver for allDeliver Scope .If .Frontier has a record for ,we getDeliver Deliver from FrontierLemma (iii).If not,by Fron-tier Lemma (ii),and’s record was removed from .Frontier by some Delivered .Let ,,be in Delivered ,such that delivery ofat removes the record forfrom .Frontier,but .Frontier has a record for,so .Frontier carries that record.The record removal by means (line37)that .Frontier has a record for ,soDeliverDeliver by Fron-tier Lemma (iii).Similarly,Deliver Deliver .Since ,we getDeliver Deliver as desired.Now an induction on using the InductionLemma proves that Deliver is consistent,so ScopeCast delivers all the multicasts consistently.7For a unicast,consider any preceding multicast delivered at before Send.time. If.Frontier has a record for,then Frontier Lemma(iii)gives DeliverDeliver.If not,there are again multi-casts(possibly from)etc.delivered at that induce the ordering Deliver Deliver.5SummaryEngineering a distributed system is easier if there is a service for consistent multicast delivery.For peer-to-peer systems,inconsistency is more likely because membership is dynamic,and wide-area net-work delays make message reordering and concur-rent join/quit events more likely.We propose ScopeCast,a protocol for ensuring consistent delivery of peer-to-peer multicasts,and proved its correctness.It does not assume group membership is known orfixed.Furthermore,if there is network failure or traffic congestion,delivery of received messages can be forced;consistent delivery resumes after this interruption.Simulations show that the delay between mes-sage reception and delivery grows less quickly than end-to-end network delay[S].On the other hand, they also show that,because ScopeCast does not use any supporting logical or physical infrastructure,its overheads limit it to small peer groups.We present elsewhere another multicast protocol[S]that uses a spanning tree supported by a distributed hash table, and is thus more scalable.References[AHJ]M.Ahamad,P.Hutto,and R.John,Im-plementing and Programming Causal Dis-tributed Memory.Proc.ICDCS,Arlington,TX(May1991),271–281.[AN]N.Adly and M.Nagi,Maintaining Causal Order in Large Scale Distributed Systems Us-ing a Logical Hierarchy,Proc.IASTED Int.Conf.on Applied Informatics,Innsbruck,Austria(Feb.1995),214–219.[B]K.P.Birman,A review of experiences withreliable multicasts,Software—Practice andExperience29,9(July1999),741–774. [BSS]K.Birman,A.Schiper,and P.Stephenson, Lightweight Causal and Atomic Group Mul-ticast,ACM puter Systems9,3(Aug.1991),272–314.[BM]Ö.Babao˘g lu and K.Marzullo,Consistent global states,in Distributed Systems,S.Mul-lender(ed.),Addison Wesley(1993),55–96.[C]I.Clarke,O.Sandberg,B.Wiley and T.W.Hong,Freenet:A distributed anonymousinformation storage and retrieval system,LNCS2009,Springer-Verlag(2001),46–66. [CL]K.M.Chandy and mport.Distributed snapshots:determining global states of dis-tributed systems.ACM puter Sys-tems3,1(Feb.1985),63–75.[DKM]D.Dolev,S.Kramer and D.Malki,Early delivery totally ordered multicast in asyn-chronous environments,Proc.FTCS,Toulouse,France(June1993),544–553. [EMS]P.D.Ezhilchelvan,R.A.Macêdo and S.K.Shrivastava,Newtop:A fault-tolerant groupcommunication protocol,Proc.ICDCS,Van-couver,Canada(May1995),296–306. [HT]V.Hadzilacos and S.Toueg,Fault-tolerant broadcasts and related problems,in Dis-tributed Systems,S.Mullender(ed.),Addi-son Wesley(1993),97–145.8[KD]I.Keidar and D.Dolev,Efficient message or-dering in dynamic networks,Proc.PODC,Philadelphia,PA(May1996),68–76. [MPS]S.Mishra,L.L.Peterson,R.D.Schlicht-ing,Implementing Fault-Tolerant ReplicatedOjects Using Psync,Proc.SRDS,Seattle,WA,(Oct.1989),42–52.[RD] A.Rowstron and P.Druschel,Pastry:scal-able,distributed object location and routingfor large-scale peer-to-peer systems,Proc.IFIP/ACM Middleware2001,Heidelberg,Germany(Nov.2001).[RM] B.Rajagopalan and P.K.McKinley,A token-based protocol for reliable,ordered multicastcommunication,Proc.SRDS,Seattle,WA(Oct.1989),84–93.[RST]M.Raynal,A.Schiper and S.Toueg,The causal ordering abstraction and a simple wayto implement it,Information Processing Let-ters39,6(1991),343–350.[S]X.B.Shen,On concurrent states and peer-to-peer multicasts,MSc Thesis,National Uni-versity of Singapore(2003).[YHH]L.-H.Yen,T.-L.Huang,S.-Y.Hwang,A pro-tocol for causally ordered message deliveryin mobile computing systems,MONET2,4(Jan.1998),365–372.[Z]S.Q.Zhuang,B.Y.Zhao,A.D.Joseph,R.H.Katz,J.Kubiatowicz,Bayeux:An Archi-tecture for Scalable and Fault-tolerant Wide-Area Data Dissemination,Proc.NOSSDA V,Port Jefferson,NY,(June2001),11–20.9。