CASS An Efficient Task Management System for Distributed Memory Architectures

CASS: An Efficient Task Management System for Distributed Memory Architectures *

Jing-Chiou Liou Michael A. Palis AT&T Laboratories

Middletown, NJ 07748-4801, USA

jing@j olt .mt .at t .com

Department of Computer Science

Rutgers University Camden, NJ 08102, USA

palis~crab.rutgers.edu

Abstract The thesis of this research is that the task of exposing the parallelism in a given application should be left to the algorithm designer, who has intimate knowledge of the application characteristics. On the other hand, the task of limiting the parallelism in

a chosen parallel algorithm is best handled by the

compiler or operating system for the target MPP ma-

chine. Toward this end, we have developed CASS (for Clustering And Scheduling System), a task man- agement system that provides facilities for automatic

granularity optimization and task scheduling of parallel

programs on distributed memory parallel architectures. Our tool environment, CASS, consists of a two- phase method of compiler-time scheduling, in which task clustering is performed prior to the actual schedul- ing process. The clustering module identifies the opti- mal number of processing nodes that the program will require to obtain maximum performance on the tar- get parallel machine. The scheduling module maps the

clusters onto a fixed number of processors and deter- mines the order of execution of tasks in each proces- sor.

1 INTRODUCTION

In the last decade, massively parallel process- ing (MPP) has become consensus approach to high- performance computing. MPP vendors have leveraged the small size, low cost, and high performance of com- modity microprocessors to build large-scale parallel machines with hundreds or even thousands of nodes. These powerful machines are now capable of perform- ing billions of floating-point operations per second (gi- gaflops and at least one machine, the ”Ultra” com- puter 1) uilt by Intel, has reached the teraflop level (1000 gigaflops) in 1996.

Although the peak performance of MPP machines are impressive, they are rarely achieved in practice. A typical application program running on an MPP machine distributes its tasks and data among the pro-

cessing nodes and relies on message-passing to transfer

‘This research was supported in part by NSF Grant IRI- 9296249.

data between tasks or to synchronize the tasks oper- ations. At the physical level, the resultant inter-node communication causels some nodes to sit idle waiting for data. In existing ldPP machines, this communica- tion overhead can be large, typically in excess of 500

instruction cycles [2]. As a result, the actual perfor- mance of an application often falls short of its theo- retical performance, except for a few “embarrassingly parallel” applications that do not require inter-task communication.

A parallel program can be viewed abstractly as a

collection of tasks, where each task consists of a se- quence of instructions and input and output parame- ters. A task starts execution only after all of its in- put parameters are available; output parameters are sent out to other tasks only after task completion. This notion of a task is called the “macro-dataflow model” by Sarkar [13] and is used by other researchers 12, 4, 11, 15, 171. Loosely speaking, the granularity t

or gram szze) of a task is the ratio of its execution time vs. the overhead incurred when communicating with other tasks. Thie granularity of a parallel pro- gram is the minimurn granularity of its constituent tasks. (A more precise definition of granularity will be given later.)

The high communication overhead in existing MPP machines imposes a minimum threshold on task granu- larity below which performance degrades significantly. Consequently, to obi,ain maximum performance, a fine-grain parallel program (i.e., a program with small granularity) may have to be restructured to pro- duce an equivalent coarse-grain program by coalesc- ing many fine-grain tssks into a single task. Manual “fine-tuning” of a parallel program is, unfortunately, too excessive a burden to place on the shoulders of

an algorithm designer, be s)he novice or expert. Not

of exposing the parallelism in a given application, but

(s)he also needs to worry about the equally difficult problem of limiting the parallelism in the algorithm to minimize communiication overhead. Moreover, the latter problem requires of the designer deep knowl- edge of the characterisitics of the target MPP machine, e.g., the number of processing nodes, CPU speed, lo-