An Adaptive Deadlock and Livelock Free Routing Algorithm

An Adaptive Deadlock and Livelock Free Routing AlgorithmM.Coli, P.PalazzariDipartimento di Ingegneria ElettronicaUniversità "La Sapienza"Via Eudossiana 18, 00184 RomaE-mail coli@die.ing.uniroma1.it palazzari@casaccia.enea.itAbstractThis paper is concerned with Store and Forward deadlocks (DL) arising in interprocessor network systems with buffered packet switched communications.Algorithms which implement DL free routing use adaptive or not adaptive routing modality. Not adaptive algorithms underuse the interconnection network bandwidth because they impose restrictions to the routing paths; adaptive algorithms are DL free only if certain hypothesis on the communication topology occur. In order to override these drawbacks, we have implemented an adaptive DL free routing which fully exploits the connectivity of the network and which is unrelated to its topology. DL is avoided by adopting a recovery policy: whenever DL arises, our algorithm is able to remove it within a finite time. We demonstrate 'deadlock' and 'livelock' avoidance by ensuring the presence of a hole in the network buffers; the hole is subjected to casual movement.Performance tests, executed on a transputer based parallel machine, show the effectiveness of the algorithm and demonstrate its fault tolerance capabilities.Keywords: adaptive, routing, deadlock, livelock, fault tolerance.1 IntroductionIn parallel systems the deadlock (DL) phenomenon can take place when there is a share of resources [8]. In this work we refer to DLs caused by the filling of buffers present in the nodes of a packet switched communication network (Store and Forward DL [13]). In order to avoid deadlocks in packet switched communication networks, several techniques have been proposed. These techniques can be divided into two categories:- static routing. Only minimal length paths are used; a path between two nodes is uniquely determined by the source and destination nodes ([4], [5], [7]);- adaptive routing. All the minimal length paths are used [14]; in order to avoid DL, also the not minimal length paths are used ([1]-[3], [6], [9], [10], [12], [13]).Static routing algorithms avoid DL by choosing communication paths so that the communication graph results acyclic. As a consequence, they underuse the communication network by imposing restrictions to the number of paths available to transmit a message between two processors.Adaptive routing algorithms can use all the minimal length paths available. DL is avoided either by ensuring that communication queue graph is acyclic [14] or by performing a misrouting (i.e. a communication which increases the distance of the message from its destination) whenever a channel which approaches a message to its destination cannot be found (deflection or hot potato routing; chaotic routing waits for the internal queue to become full before performing a misrouting). These algorithms ensure DL free routing only if each communication node has the number of input channels equal to the number of output channels. Deflection routing algorithms perform a misrouting as soon as a message cannot be transmitted on a direction which approaches it to the destination; as a consequence, a message can be deflected many times before reaching its destination, so that the communication latency can be considerably enlarged.We have developed a DL free adaptive routing algorithm whichA- does not require communication nodes to have input degree equal to output degree;B- waits for a fixed time T before executing a misrouting, so that it is performed only when the possibility that DL occurs is high .A fault on one or more communication (input or output) channels does not stop the transmission of messages, because they are routed around the fault channels. Our algorithm ensures that routing modality still remains DL free when one or more faulty communication channels are present (point A). Other algorithms do not ensure that routing remains DL free in presence of faulty channels because the fault causes the input degree of a communication node not to be equal to the output degree.The algorithm we developed is based on a policy of DL recovery and implements adaptive routing because, ateach step of retransmission, a free communication channel is chosen among those channels approaching the message to its destination. The algorithm is based on the concept of the hole in the network buffers [11]; we have demonstrated that adaptive deadlock and livelock free routing modality can be implemented through the assumption of not restrictive hypothesis .Parallel machine PM can be described as PM=<P,Ls>,in which P = {P 1,P 2,...,P N } is the processors set and Ls ÍPxP is the communication channel set. A channel c ij which connects Pi to Pj is specified by the ordered couple (P i ,P j ).Each processor P i has a set of communication channelsC(P i ) = Cin(P i ) È Cout(P i )in whichCin(P i ) = {(P x ,P i ) | P x ÎP, (P x ,P i )ÎLS}Cout(P i ) = {(P i ,P y ) | P y ÎP, (P i ,P y )ÎLS}.Ls is a network which makes PM connected: thus,given any two processors P i and P j , in Ls they are connected by a path. No other constraint on Ls exists; we may have tooCin(P i ) ¹ Cout(P i ) for some i C(P i ) ¹ C(P j ) for some i,j.The messages transmission among processors is packet switched.Each processor P i has (fig. 1) a set of buffers given bySBUF(P i ) = Bin È {B i } È Boutin whichB i is an internal buffer;Bin = {Bin 1,Bin 2,...,Bin Ni } is the set of the input buffers;Bin k is the buffer on the k th input channel; its output goes toward B i ;Bout = {Bout 1,Bout 2,...,Bout No } is the set of the output buffers; Bout k is the buffer on k th output channel; its input comes from B i ;we suppose that each buffer can hold only one message;whenever ambiguity arises, we indicate explicitly theprocessor P which buffer B belongs to, i.e. B(P).fig. 1 Connections among the Pi internal buffers2. DL in communicationsLet us define a processor P i to be in a transmission deadlock state when it tries to execute a transmission without being ever successful.We suppose that a message, as soon as it arrives to the target processor P i , is absorbed by a receiving process without being transmitted to B i .In order to describe the routing algorithm, we premise some definitions:- Tx i (P k ): (routing function) it is the channel which must be used by processor P i (i=1,2,...,N) to transmit a message directed to processor P k ; the transmission of the message through the channel Tx i (P k ) approaches the message to its destination.-Dest(P k ): it gives the processor to which the message,contained in the internal buffer B i of processor P k , is directed;-Free(B,P k ) is the number of free positions in buffer B;-Bin(P i ,P j ) is the input buffer associated with channel (P i ,P j );-Bout(P i ,P j ) is the output buffer associated with channel (P i ,P j );- Neigh(P k )={P i | P i ÎP, P i ¹P k , C(P i ) Ç C(P k )¹{} }: it is the set of processors adjacent to P k ;-Next(P k ,P i ): it is the next processor to which a message must be transmitted; the message is in processor P k and is directed to processor P i ;Next(P k ,P i )ÎNeigh(P k ) andTx k (P i )=Cout(P k )ÇCin(Next(P k ,P i )).We demonstrate that our algorithm is DL free by finding a necessary condition for DL existence and,subsequently, by determining a routing modality which ensures that the necessary condition does not occur.In order to establish a theorem which supplies a necessary and sufficient condition for DL setting up in a processor network, we define in Ls a cyclic path of length n asp n = (P k 0,P k 1),(P k 1,P k 2),...,(P k n-1,P k n =P k 0),(P k i ,P k (i+1))ÎLs, i=0,1,...,n-1P k j Î P, j=0,1,...,n-1.The processors set involved in p n is:PC = {P k 0,P k 1,...,P k n-1}.theorem 2.1: NSC because DL takes place in PM is that$ p n , n•2 :1) P k (i+1)=Next(P k i ,Dest(P k i))2) Free(Bin(P k (i-1),P k i ),P k i )=0 i=1,2,...,n3) Free(B i ,P k i )=0 "P k i ÎPC4) Free(Bout(P k i,P k(i+1)),P k i)=0 i=0,1,...,n-1The proof of theorem 2.1 is reported in appendix A.Theorem 2.1 shows that DL can exist if and only if:- a cyclic path p n exists in Ls; internal buffer B i of each processor belonging to p n contains a message that must be transmitted toward the next processor of the cycle (1) - the cyclic path p n involves the existence of a cycle of full buffers; this cycle is constituted by 3n buffers (2),(3),(4).In order to avoid DLs, it is sufficient to ensure that one of 1), 2), 3) and 4) of theorem 2.1 is not verified.3 A deadlock recovery policy for adaptive routingWe say that buffer B in processor Pi contains a hole when it is empty [11]; the number of holes in B is given by Free(B,P i).When a message is transmitted from B i buffer to B j buffer, we say that a hole has moved from B j to B i.In order to implement adaptive routing modality, routing functions Tx i (i=1,2,...,N) must not be single valued.As the adaptive routing management is not well suited for a DL prevention policy, we adopted a DL recovery policy. Our algorithm performs adaptive routing by transmitting the messages toward the first available channel which approaches them to their destination; if, after a T interval of time is elapsed, no free channel has been found to approach the message to its destination, we suppose DL to be occurred and a misrouting is performed.The p n time dependence, with obvious notation, is introduced:- p n(t) is a cyclic path on Ls which, at instant t, satisfies the 1) of theorem 2.1,- PC(t) is the set of processors belonging to p n(t).In order to avoid DL we must ensure that, if a t0 exists for which a cyclic path p n(t0) satisfies 1), 2), 3) and 4) of theorem 2.1, a t1 (t0<t1<¥) exists for which p n(t1)=p n(t0) and a buffer B, with Free(B,P k i(t1))>0, is found among those referred in conditions 2), 3) and 4) of theorem 2.1. In other words we want a hole to break, in a finite time, a cycle causing DL. Particularly it is sufficient that, in a given P k iÎPC, the transmission from B i toward Bout(P k i,Next(P k i,Dest(P k i))) is aborted and directed toward a different Bout i with Free(Bout i,P k i)>0. After the transmission from B i toward Bout i is executed, Free(B i,P k i)=1 results. Condition 3) of theorem 2.1 is not more verified and DL is removed.In order to remove DL it is so sufficient to ensure that,if a p n(t) exists which satisfies all the conditions of theorem 2.1, it is always possible to find a P k iÎPC(t) and an instant t'>t for which condition 3) of theorem 2.1 is not verified.Afterwards we shall suppose holes to be subjected to casual movement, according to the followingdefinition 3.1: a hole is subjected to casual movement if, being Free(B i,P)=1, the input buffer Bin j from which to receive a message is chosen in a random and equiprobable way (Bin jÎ{B | Free(B,P)=0, BÎBin(P)}).Since to avoid DL it is sufficient that a hole breaks the cycle of full buffers reaching a locked processor P k iÎPC in a finite time, let us demonstrate the following theorem 3.1: Sufficient conditions for which a hole reaches a P i within a time t<¥ are that1) Ls is connected,2) at least a hole is present,3) holes are subjected to casual movement according to definition 3.1.The proof of theorem 3.1 is reported in appendix A.When the hypothesis of theorem 3.1 are verified, we are certain that, within a finite time, a hole will reach any processor of PM.We define a DL configuration in PM as a set PC of processors which satisfies the conditions of theorem 2.1.Now we demonstrate a theorem which, on the basis of theorem 3.1, ensures that each deadlock configuration can be removed within a finite time.theorem 3.2: in order to remove any DL configuration, a sufficient condition is that1) at least a hole is present in PM,2) the hole is subjected to casual movement accordingto definition 3.1,3) if a communication, in a processor P, from B itoward Bout j is locked, it can be redirected toward any Bout k, with Free(Bout k,P)=1, after a T time interval is elapsed.4) Ls is connected.proof: let us suppose a DL configuration is present (hypothesis of theorem 2.1 are verified); we show that it is removed if 1), 2), 3) and 4) are verified. Two situations can happen:a) in a P k jÎPC, a Bout k exists for which Free(Bout k,P k j)=1; on the basis of the 3) hypothesis, when a T time interval is elapsed, the communication coming out from B i is directed toward Bout k and, after the transmission, Free(B i,P k j)=1; as 3) of theorem 2.1 is not more verified, the DL configuration has been removed.b) no P k jÎPC has a Bout k for which Free(Bout k,P k j)=1. As the hypothesis of theorem 3.1 are verified, we are sure that by a t<¥ a hole reaches a P k jÎPC through a Bout k (ahole comes into a processor through its output buffers). We are again in situation a), so DL is removed.4 The livelockWhen a routing algorithm allows the messages to be misrouted, the livelock phenomenon can arise.If a DL free routing algorithm is based on theorem 3.2, 3) of this theorem brings the possibility of livelock, because a message can be misrouted. With the following theorem 4.1 we demonstrate that, if the hypothesis of theorem 3.2 are verified, livelock cannot take place.theorem 4.1: sufficient condition for which livelock does not arise is that all the hypothesis of theorem 3.2 are verified.The proof of theorem 4.1 is reported in appendix A.corollary to theorems 4.1 and 3.2: if the hypothesis of theorem 3.2 are satisfied, it is impossible that either deadlock or livelock situations exist.5 DL and livelock free adaptive routingIn order to implement an adaptive, deadlock and livelock free routing algorithm it is sufficient to build the internal buffer process B i in a proper manner. The input buffer processes Bin i will simply receive messages and transmit them to B i or to the node processes; similarly the output buffer processes Bout i must receive messages from B i or from the node processes and retransmit them through the output channels which they are connected to.Process B i(P k) must be such thatA- hypothesis 2) and 3) of theorem 3.2 are satisfied;B- (adaptivity condition) a message is transmitted to the first free output buffer which is placed on a c ij channel beingc ij = Tx k(Dest(P k));The algorithm which implements B i(P k) is shown in the following:algorithm which implements B i:while true do<receive a message m from a process Bin i; Bin i israndomly chosen in the set BusyBin ={B | Free(B,P k)=0, BÎBin(P k)}>{hypothesis 2) of theorem 3.2 is satisfied}repeat<try to transmit message m, which is directed tothe processor P j, toward a Bout i placed on thechannel c kiÎNextCh={c ki | c ki=Tx k(P j)}>{the adaptivity condition is satisfied}until <T sec have elapsed or the communicationwas successfully done>if <the communication has failed>then repeat<try to transmit m toward any Bout i placed ona channel c kjÎCout(P k)>{hypothesis 3) of theorem 3.2 is satisfied}until <the communication was done> endwhileIn order to implement a deadlock and livelock free routing algorithm, we must yet satisfy 1) and 4) of theorem 3.2. 4) is verified because, as the only constraint on Ls, we suppose it to be connected. In order to prove a sufficient condition to satisfy 1) of theorem 3.2, let us define the number NL(P i) of holes in P iÎP asNL(P i) = Free(B i,P i)+Free(Bin j,P i)åj=1Cin(P i)+Free(Bout j,P i)åj=1Cout(P i);the number NL of holes in PM is so given byNL =NL(P i)åP iÎP;- when a message reaches its destination (delivery phase), NL is incremented;- when a process executed in P iÎP introduces into the router a message directed to a process executed in P jÎP (injection phase), NL is decreased;- all the transmission between buffers (forwarding phase) leave NL unchanged.theorem 5.1: sufficient condition to be always NL³1 is that a process executed in P iÎP can inject a message m into a buffer BÎSBUF(P i) only if NL(P i)³2.proof: let us suppose initially NL>01; we demonstrate that, if NL>0, NL=0 cannot ever exist. The hypothesis that a message m is introduced into BÎSBUF(P i) only if NL(P i)³2 means that m can be introduced into B only if1After reset it is reasonable to suppose all the buffers being empty.NL³2, because NL(P i)³2 implies NL³2; after a message injection NL decreases of one unity. If the message injection causes NL=1, we have that$P i : NL(P i)=1andNL(P j)=0 "P jÎP, P j¹P i;thus no more messages can be injected becauseØ$P iÎP : NL(P i) ³ 2.As NL is decreased only when a new message is injected and when NL=1 new messages cannot be introduced, NL=0 cannot ever be verified.In order to satisfy also 1) of theorem 3.2, a sufficient condition is to allow the injection of new messages from P iÎP only if at least two holes are present in the buffers of P i.6 Performances evaluationWe have implemented our adaptive routing algorithm and the not adaptive one presented in [5] on a 3x3 mesh connected transputer machine (fig. 2). The algorithm presented in [5] achieves immunity from DL by transmitting messages on the horizontal direction in the first phase and on the vertical direction in the second phase.The data reported in the following are relative to the performances of the two algorithms submitted to the same traffic.The first test compares the two routing algorithms in traffic situations which are completely at random and of growing intensity; we have fixed a time T0 (two minutes) and we have computed the average number of messages which, during T0, each processor sends to the other randomly selected processors. In order to cause growing random traffic intensity, a process as the following is executed in each processor:t p:=10 (seconds)while < (t p>64 m s) > dowhile <T0 seconds are not elapsed> do<wait for t p seconds> {simulates a computing part}<send 10 messages (each of them is 16Kbyte) to 10randomly chosen processors>endwhilet p:=t p/2endwhilePrevious process behaves like a program where the communication part becomes ever more predominant than the computing part (t p is halved; the pause of t p seconds simulates the program computing phase).adaptive), figure 3 reports the average number of messages transmitted by a processor for each t p value; in abscissa the iteration number n i is reported: t p is related to n i according to the followingt p = 10 sec.2n i15105200040006000Iteration numberAverageNTxfig.3 : average number of messages transmitted by a processor for the two A and NA routing algorithms.Our algorithm performs a misrouting if, after T seconds are elapsed, no free channel is found to approach the message to its destination; when T=0, our algorithm is similar to hot potato routing. In figure 3 we report the average number of messages transmitted by a processor when our algorithm is adopted with T=200 ut (unit of time ut=64m s) or with T=0 ut.Fig. 3 underlines that, if the traffic has random behaviour, NA algorithm and A algorithm with T = 200 ut are equivalent. A algorithm with T=0 ut gives worst performances because, when the traffic load is heavy and equally distributed, many misroutings are performed and the latency of each message is enlarged.Usually in a parallel machine the traffic shows a proper regularity and it is not correctly modelled by an equiprobable random traffic. In order to characterize better A and NA features, let us consider some traffic situations.case 1) Processor 4 sends 10 messages (each of them is 16 Kbytes) to any other processor in the network (broadcast operation).case 2) Each processor sends 10 messages (16Kbytes)to all the processors.case 3) Transmission of 100 messages (16Kbytes)between two opposite processors (processor 2 sends 100messages to processor 6).case 4) Transmission of 100 messages (16Kbytes)between the opposite processors of the main diagonal and of the underdiagonal (processor 0 sends 100 messages to processor 8 and processor 5 sends 100 messages to processor 7).Figure 4 reports the normalized times spent by A and NA (T=200 ut) algorithms in the execution of thecaso 1)caso 2)caso 3)caso 4)100200300T c o m m n o r m a l i z e dfigure 4: normalized communication times spent by A and NA algorithms to execute the communications of cases 1)-4)In order to show the influence of T parameter (condition 3 of theorem 3.2) on the performances of our routing algorithm, we have considered its behaviour for many T values and for different kinds of traffic. Figure 5reports, as a function of log 10(T), the communication time necessary to exchange 1000 messages (16Kbyte)between processors 0 and 8 and between processors 2 and 6; the two plots concern to the communication network without faults (solid line) and with one communication channel fault (dotted line).Figure 5 shows that communication time is minimum when T=200 ut. When T decreases the communication time grows because messages are frequently misrouted;when T increases the communication time grows because,once a DL situation has occured, more time is requiredbefore the DL situation is removed. The figure shows also that the routing algorithm is still DL free when one link fault occurs: all the messages are correctly delivered and the DL situations are correctly removed. The only consequence of the channel fault is a diminishing of the network bandwidth.543210e+01e+62e+6log Tt c o m m (u t )figure 5: communication time vs log(T) a)with no faults and b)with C05 fault7 ConclusionsThe analysis of DL free routing algorithm presented in literature shows that adaptive routing algorithms are dependent on the network topology; DL free behaviour can be achieved only by using communication nodes with input degree equal to output degree. This is a strong limitation, because the immunity from DL cannot be ensured when a link fault occurs. We have presented an adaptive routing algorithm which does not suffer from the previous drawback and which gives communication time responses better than the ones given by the already known adaptive and not adaptive routing algorithms. This better behaviour is achieved by performing a misrouting only when this is effectively required.We have demonstrated that our routing algorithm is deadlock and livelock free.The actual execution of the algorithm on a 9 transputer parallel machine has validated the theoretical results, i.e.:- the algorithm showed better time responses with respect to adaptive or not adaptive hot potato routing algorithms;- the algorithm removed all the DLs;- even when there was a communication link fault, the algorithm correctly performed all the communications and all DLs were removed.Appendix A: proofs of theoremsIn order to demonstrate theorem 2.1 we premise definition 2.1 and we demonstrate the auxiliary theorem 2.2.B 1 and B 2 are two buffers connected through a channel from B 1 to B 2 and TX(B 1,B 2) is a transmission directed from B 1 to B 2;definition A.1: the transmission TX(B 1,B 2) directed from B 1 to B 2 is said to be locked if B 1 attempts totransmit the message to B2 without being ever successful.Since buffers contain one message, TX(B1,B2) locked means buffer B1 is full and, consequently, cannot receive new data.theorem 2.2: let us consider the two transmissions TX(B1,B2) and TX(B2,B3), being B1,B2 and B3 three buffers sequentially connected; necessary and sufficient condition because TX(B1,B2) is locked, is that TX(B2,B3) is locked.proof NC: if TX(B1,B2) is locked, B2 buffer is never available to receive data coming from B1; this happens only if B2 does never become empty, that is only if B2 does never succeed to transmit its own content toward B3; but this means that TX(B2,B3) is locked.SC: if TX(B2,B3) is locked, B2 is never free because it does never succeed to transmit its own content to B3;as a consequence B1 does never succeed to transmit its content to B2, so TX(B1,B2) is locked.proof of theorem 2.1:proof NC: deadlock presence involves that $B1 for which TX (B1,B2) is locked; in virtue of theorem 2.2 also TX (B2,B3) is locked (B2 is a buffer connected with B1 output and B3 is a buffer connected with B2 output). As a locked transmission involves the presence of another locked transmission, we can consider the infinite buffer sequenceSB = B1,B2, ....,B n,B n+1,....in which- B n is a buffer which transmits toward the buffer B n+1; - TX (B n,B n+1) is locked.By analyzing SB starting from B1, as the buffer number in PM is limited, we surely meet a buffer B k already appeared in SB in a previous position p (p³1). SB can beso written asSB = B1,B2, ....,B p-1,B p,B p+1,...,B k-1,B k=B p,B p+1,... which underlines in SB the existence of a periodic partSBP = B p,B p+1,...,B k-1.In order to show the periodic nature of SB, it is sufficient to note that TX(B p,B p+1) is locked and, when we meet TX(B k-1,B k=B p), this is again locked by TX(B k=B p,B p+1).If we remember the structure of the internal buffers of a processor, we see that SBP is a sequence of buffer terns, that is- internal buffer B i of a processor P k i (indicated with B i(P k i)),- output buffer of the channel which is the destination of the message contained in B i, that is Bout(P k i,Next(P k i,Dest(P k i))),- input buffer of the same channel, that is Bin(P k i,Next(P k i,Dest(P k i))).Without loss of generality, we substitute explicitly thebuffers in SBP by supposing that B p is the internal buffer of processor P k0; so we haveSBP = B i(P k0), Bout(P k0,P k1=Next(P k0,Dest(P k0))), Bin(P k0,P k1), B i(P k1), Bout(P k1, P k2 = Next(P k1,Dest(P k1))), Bin(P k1,P k2),..., B i(P k n-1), Bout(P k n-1,P k0 = Next(P k n-1,Dest(P k n-1))), Bin(P k n-1, P k0).Our hypothesis about DL existence brings to select aset of processors PC which satisfies 1) of the theorem; asall the transmissions among consecutive buffers in SBPare locked, all the buffers must be full, so that also 2), 3)and 4) of the theorem are verified. As the connection ofan output channel with an input channel of the same processor must be excluded, n³2 is justified because cyclic dependence cannot exist among buffers in the sameprocessor.SC: let us consider internal buffer B i(P k i) of any P k iÎPC; for 1) the transmission of the message in B i(P k i) is directed to processor P k i+1= Next(P k i,Dest(P k i)) and, for 4), this transmission cannot be executed because the Bout(P k i,P k i+1) is full. Similarly the transmission from Bout(P k i,P k i+1) to Bin(P k i,P k i+1) cannot be executed for 2) and the transmission from Bin(P k i,P k i+1) to B i(P k i+1) cannot be executed for 3). As the previous reasonment subsists "P k iÎPC and, on the basis of 1), it does not exist a P k i such as Next(P k i,Dest(P k i))ÏPC, each B i(P k i) is full and is connected with another full B i(P k j) through two full buffers; because any transmission is impossible in this situation, DL subsists.proof of theorem 3.1:we demonstrate the theorem by considering the worstcase, i.e.a) only one hole is present,b) in its movement the hole does not generate new holes.Under these hypothesis the hole movement in the networkis a stationary Markoff process of order 1; infact, being L j the event for which the hole is in P j, we havePr(L j|L i,L k,..,L m; t k|t k-1,t k-2,..,t k-n)=Pr(L j|L i; t k|t k-1) 2 because the hole position at a certain instant depends only on the hole position at the previous instant (time is discreet and is incremented at each transmission). Afterwards we shall not explicitly show the time2 Pr(E2|E1; t2|t1) is the probability that, at the instant t2, the event E2 takes place having supposed the event E1 verified at the instant t1.。