Optimal multicast communication in wormhole-routed torus networks

Optimal Multicast Communication in Wormhole-Routed Torus Networks
David F. RobinsonPhilip. K. McKinley and Betty. H. C. Cheng
Technical Report MSU-CPS-93-28 November 1993 Revised December 1994.
1 Introduction
The recent trend in supercomputer design has been towards scalable parallel computers, which are designed to o er corresponding gains in performance as the number of processors is increased. Many such systems, known as massively parallel computers (MPCs), are characterized by the distribution of memory among an ensemble of processing nodes. Each node has its own processor, local memory, and other supporting devices. In parallel scienti c computing, data must be redistributed periodically in such a way that all processors can be kept busy performing useful tasks. Because they do not physically share memory, nodes in MPCs must communicate by passing messages through a communications network. Some communication operations are point-to-point , that is, they involve only a single source and a single destination. Other operations are collective , in that they involve more than two nodes. Examples of collective communication include multicast , reduction , and barrier synchronization . The growing interest in the use of such routines is evidenced by their inclusion in many commercial communication libraries and in the Message Passing Interface (MPI) 1], an emerging standard for communication routines used by message-passing programs. Besides message-passing, collective communication operations are also important in supporting data-parallel languages, such as High Performance Fortran 2], on distributed-memory systems. In a multicast operation, a source node must deliver copies of a single message to each node in a speci ed set of destinations. A special case of multicast is broadcast, in which the destination set contains every node in the network. In practice, parallel algorithms are rarely allocated all nodes of a machine, thus, multicast communication is needed even in cases where the algorithm calls for one process to send a message to all other processes in the application. Multicast is important in many parallel numerical algorithms, including matrix multiplication 3], matrix transpose 4], tridiagonalization 5], eigenvalue computation 6], Gaussian elimination 7], and LU factorization 8]. E cient implementation of multicast is also useful in many other aspects of parallel computing, including support for barrier synchronization 9], memory updates and invalidation in distributed shared-memory systems 10], and global noti cation of events in parallel simulation. Collective communication operations may be implemented in either hardware or software. However, many existing MPCs support only point-to-point, or unicast communication in hardware. In these environments, collective communication operations must be implemented in software (typically in a communications library) by sending one or more unicast messages; such 1
This paper presents e cient algorithms that implement one-to-many, or multicast, communication in wormhole-routed torus networks. By exploiting the properties of the switching technology and the use of virtual channels, a minimum-time multicast algorithm is presented for n-dimensional torus networks that use deterministic, dimension-ordered routing of unicast messages. The algorithm can deliver a multicast message to m ? 1 destinations in dlog2 me message-passing steps, while avoiding contention among the constituent unicast messages. Performance results of a simulation study on torus networks with up to 4096 nodes are also given.
Optimal Multicast Communication in Wormhole-Routed Torus Networks
David F. Robinson
Division of Business and Computer Science Quincy University Quincy, Illinois 62301
implementations are called unicast-based 11]. A multicast operation may be implemented using separate addressing, in which a separate copy of the message is sent directly from the source to every destination. An alternative is to use a multicast tree 11] of unicast messages. The tree can be considered as a sequence of communication (message-passing) steps. In the rst step, the source node actually sends the message to only a subset of the destinations. In the next step, each node holding a copy of the message forwards it to some subset of the destinations that have not yet received it. The sequence of message-passing steps continues until all destinations have received the message. Using this approach, the time required for the operation can be greatly reduced 11]. The design and performance of tree-based multicast operations depends on several characteristics of the network architecture, including the switching strategy and network topology 12]. The predominant switching technique is wormhole routing 13], in which a message is divided into a number of its that are pipelined through the network. The network latency equals the elapsed time after the head of a message has entered the network at the source until the tail of the packet emerges at the destination. For long messages, the pipelining e ect of wormhole routing reduces the e ect of path length on network latency 12]. The startup latency is the time required for the system to handle the packet at both the source and destination nodes. For small messages, the startup latency often dominates unicast latency 11]. Therefore, in the absence of contention among messages for network resources, the latency of wormhole-routed messages is nearly distance-insensitive 12]. Regarding network topology, early systems that used store-and-forward switching often adopted a hypercube topology because the relatively dense interconnection network resulted in shorter message paths. However, in systems with wormhole routing, the distance between communicating nodes is less important; hence, the more easily constructed lower-dimension meshes and tori are better suited topologies for current MPCs 14]. Figure 1 shows a two-dimensional (2D), 4 4 mesh and a 2D, 4 4 torus. An n-dimensional torus is equivalent to an n-dimensional mesh in which \wraparound" channels have been added to connect each edge node to the corresponding node on the opposite edge. The symmetry of torus networks leads to a more balanced utilization of communication links, under random tra c, than in mesh topologies 15]. Examples of wormholerouted torus architectures include the Torus Routing Chip 13] and the Cray T3D 15]. In wormhole-routed MPCs, communication among nodes is handled by a separate router . Several external channels connect the router to neighboring routers; the pattern in which the external channels are connected de nes the network topology. Usually, the router can relay multiple messages simultaneously, provided that each incoming message requires a unique outgoing channel. 2