PVM-SCI A PVM Implementation for Heterogeneous SCI-Clusters

I. Introduction
Scienti c computing on clusters of workstations or PC's has become a very viable option in the recent years. LAN interconnects like ATM, Fast-Ethernet, and GigabitEthernet o er a high bandwidth, which however is often slowed down by a time-consuming TCP IP stack on top of the network layer. Some recent implementations access the network directly and thereby try to avoid costly system calls by by-passing the kernel. This results in a much lower latency as well as a higher bandwidth. Some work related to ours has been done in providing an ATM port for PVM 3.3 18 . In that project, only few modi cations were needed because the TCP IP stack is used on top of the ATM layer. As a consequence, the performance was moderate due to the overhead caused by the protocol. PULC 1 is another PVM compliant version using the ParaStation. In this case however, the original source has been modi ed thoroughly to achieve performance and is designed to run in homogeneous environments only. In our work, we are mainly interested in plugging in SCI as a promising technology to see the overhead of the PVM message passing library as compared to the raw performance of SCI. An experience curve evaluating the optimal fragment size for various message sizes and a one-receivets-all strategy have been examined. The standard PVM functionality has been kept, while heterogeneous interconnects can inter-operate in PVM-SCI, too. This paper is organized as follows: First we compare some contemporary interconnection technologies and their limitations. We then describe the architecture of our PVMSCI implementation which runs directly on top of SCI. In the nal section, we give a brief summary and outlook.
Abstract | We present PVM-SCI, an implementation of the PVM 3.4 message passing standard for workstation clusters with scalable coherent interface SCI. PVM-SCI provides two complementary communication schemes: the usual UDP IP protocol for messages sent via PVM daemons, and a fast communication protocol that has been implemented directly on top of SCI for messages sent with the PvmRouteDirect directive. No time-consuming protocol stack is used, thereby allowing for a low communication latency and a high message throughput. PVM-SCI is fault-tolerant and it also supports multiple interconnects, giving a convenient and e cient means for utilizing heterogeneous compute clusters. When SCI is available, PVM-SCI makes use of it, otherwise it automatically defaults to the slower Ethernet communication. PVM-SCI runs on Solaris, Linux and Windows NT.
Hale Waihona Puke A. Overview Ethernet, Fast-Ethernet, Gigabit-Ethernet and ATM have been devised for the use in Local Area Networks LANs where high data rates are to be supported over a distance of some hundred meters via unreliable media. In the faster System Area Networks SANs, message passing networks must be able to cope with even higher data rates, while the error rate is typically quite low and ow control is provided on every communication link. Myrinet 2 is one such example of a modern SAN. It connects the compute nodes through full duplex 2 x 1.28 GBit s links and low-latency switches. It provides ow control, packet framing and error control. Packets can be of any size, with long packets being automatically fragmented by the adapter cards. There is an 80 bytes slack bu er on every link. Myrinet uses a blocking cut-through wormhole routing strategy. It allows to run multiple protocols e.g., TCP IP, UDP IP where the IP packets are encapsulated without the overhead of an additional adaptation layer. SCI Scalable Coherent Interface is an IEEE interconnection standard 10 that provides high-speed bus like services to compute nodes in a distributed environment. Based on unidirectional point-to-point links and switches, SCI scales up to a maximum of 64k nodes. It supports distributed shared memory DSM and message passing for loosely and tightly coupled systems. The SCI standard de nes cache coherent transactions for CC-NUMA cache coherent non-uniform memory access architectures. Practical systems exist, for example Sequent's NUMA-Q multiprocessor 14 with the IQ-Link architecture, and the HP Convex Exemplar with the CTI network 4 . SCI is originally designed as a bus architecture. It provides hardware support for read, write and atomic operations. SCI is packet based. A cyclic redundancy check is done for each packet, with an automatic retry if an error occurs. Cache coherency is maintained by a directory scheme with doubly linked lists. Unfortunately, cache coherency works only for the more costly architectures with memory bus snooping. The cheaper PCI-based systems do not provide cache coherency, because the SCI adapter card is only indirectly connected to the memory bus via PCI, and hence cannot snoop the memory bus Fig. 1. With the availability of cheap, non cache coherent adapter cards for PCI, SBUS, and VME buses, SCI has gained much of its popularity in the recent years. Such adapter cards have been built by Dolphin Interconnect Solutions 5 , at CERN 11 , and by the University of Munich 9 .