ALEWIFE SYSTEMS MEMO #11 The LimitLESS Cache Coherence Protocol

Dir1NB
Dir2NB
Dir4NB
Full-Map
| 0.00
| 0.40
| 0.80
| | 1.20 1.60 Execution Time (Mcycles)
Figure 1: Weather, 64 Processors, Perfect optimization. shows several coherence schemes on the vertical axis, and the horizontal axis shows the total execution time (in millions of cycles) required to complete a xed portion of the application. These measurements are from ASIM, the Alewife system simulator. Both the two-pointer limited directory (Dir2NB ) and the four-pointer limited directory (Dir4NB ) perform about as well as the full-map directory. How is it possible for these limited directory protocols to be successful? If the multiprocessor exhibits processor locality in the sense that in any given interval of time only a small subset of all the processors access a given memory word, then a limited directory is sucient to capture this small worker-set of processors. The results in Figure 1 show that the two-pointer and the four-pointer directories capture the worker-sets in Weather. The performance of the one-pointer directory (Dir1NB ) su ers slightly, because it does not capture the worker-set of all of Weather's variables. But, what happens if a multiprocessor's software is not properly optimized for the limited directory protocol? Figure 2 gives the execution times for the same application as above, except that one variable in the Weather application has not been properly optimized. The Weather application initializes this variable at the beginning of the program, allows all processors to read the variable, but never writes to the variable after initialization. In fact, the cache coherence protocol implemented in ASIM provides a special data fetch mechanism to allow the system's software to optimize for this type of \write-once" data. The optimization, which bypasses the normal coherence scheme, was used by the software system to achieve the performance results presented in Figure 1, but was not used for the results in Figure 2. The dramatic di erence between the performance of the protocols shown in Figures 1 and 2 is entirely due to neglecting the optimization of just one word of shared memory! That is, when the worker-set of one location in memory is much larger than the size of a limited directory, the whole system may su er from the hot-spot access to this location. So, limited directory protocols are extremely sensitive to the size of an application's worker-set. If a multiprocessor's software is not perfectly optimized, then limited directory thrashing may negate the bene ts of caching shared data. The LimitLESS protocol avoids the sensitivity displayed by limited directories by taking advantage of the fact that each processing node in Alewife contains both a processor and a shared-memory module. (See Figure 3.) While the shared-memory module can implement a limited directory in its cache controller, the local processor can emulate a full-map directory 2
The LimitLESS Cache Coherence Protocol
David Chaiken (chaiken@) Laboratory for Computer Science Room NE43-633
ALEWIFE SYSTEMS MEMO #11
|
August 10,calable cache coherence protocols led to the study of two cache coherence protocols that use directories to store locations of cached data: a full-map directory protocol and a limited directory protocol. Full-map directories [1] store enough state associated with each block in global memory so that every cache in the system can simultaneously store a copy of any block of data. That is, each directory entry contains N pointers, where N is the number of processors in the system. Limited directories [2] di er from full-map directories in that they have a xed number of pointers per entry, regardless of the number of processors in the system. See [3] for a detailed tutorial on the di erences between these two protocols, in terms of both implementation and scalability. For the purposes of understanding the LimitLESS protocol, it is sucient to understand two important properties of these protocols: First, while the full-map protocol provides a useful upper bound for the performance of directory-based cache coherence, it is not scalable with respect to memory overhead. Second, the limited directory protocol is scalable, and it can provide performance similar to the full-map protocol when multiprocessor software is optimized for the protocol. The LimitLESS protocol uses a combination of hardware and software techniques to realize the performance of the full-map directory protocol and the scalability of the limited directory protocol, without the limited directory's sensitivity to software optimization. This new coherence scheme takes advantage of the fact that, in the Alewife architecture, shared-memory modules are distributed to the nodes that contain the processors. The LimitLESS scheme implements a small set of pointers in the memory modules (as does the limited directory). But when necessary, the scheme uses the Alewife processors to emulate a full-map directory by storing pointers that over ow the hardware directory in local blocks of shared memory. Figure 1 presents results for a Weather Forecasting program that illustrate the fact that | assuming the software optimizes for synchronization, read-only, and rarely written data types | a limited directory protocol can perform as well as a full-map directory protocol. The graph