for high-speed

Modelling and simulation of o?-chip communication architectures

for high-speed packet processors

Jacob Engel *,Daniel Lacks,Taskin Kocak

Department of Electrical and Computer Engineering,University of Central Florida,4000Central Florida Blvd.,Orlando,FL 32816,United States

Received 19September 2005;received in revised form 27January 2006;accepted 31January 2006

Available online 6May 2006

Abstract

In this work,we propose a visual,custom-designed,event-driven interconnect simulation framework to evaluate the performance of o?-chip multi-processor/memory communications architectures for line cards.The simulator uses the state-of-the-art software design techniques to provide the user with a ?exible,robust and comprehensive tool that can evaluate k -ary n -cube based network topologies under non-uniform tra?c patterns.The simulator provides full control over essential network parameters and ?ow control mechanisms such as virtual channels and sub-channeling.We compare three low-dimensional k -ary n -cube based interconnects that can ?t into the physical limitations on line cards,where each one of these interconnects has multiple processor-memory con?gurations.Performance results show that k -ary n -cube architectures perform better than existing interconnects,and they can sustain current line rates and higher.In addition,we provide performance tradeo?s between multiple ?ow control mechanisms and performance metrics such as throughput,routing accuracy,failure rate and interconnect utilization.ó2006Elsevier Inc.All rights reserved.

Keywords:Network processors;Memory management;Linecards;Interconnect systems;k -Ary n -cube networks;Shared-bus

1.Introduction

Increasing line rates and deep packet processing opera-tions place heavy strain on the memory bandwidth require-ments between the line card network processing elements (PE)and memory modules (M).In order to support new services,line cards are required to perform multiple func-tions simultaneously.Moreover,as the network expands,lookup table entries and parameters consume more memory space to store data.As a result,the memory band-width requirements,which are greatly limited by the inter-connection mechanism used to communicate between PEs and memories,are raised.Although new router architec-tures and packet processing techniques improve the perfor-mance,they still cannot keep up with network capacity growth rates in order to avoid a major tra?c bottleneck.

In the heart of every line card there is a network proces-sor unit (NPU)which performs multiple processes in order to analyze the ?ow of incoming packets.The nature of packet processing requires frequent read/write operations to memories distributed around the NPU.The simulator described in this work replicates the physical and func-tional environments by imitating di?erent con?gurations in which the PEs and memories are physically located on the line card.The simulator generates random messages with explicitly random parameters such as source/destina-tion addresses,size of messages,and arrival/departure times from PEs to memory modules and vice versa.

There are many candidates in the area of interconnects that can be used to provide a communication link between processors and https://www.360docs.net/doc/ef12620454.html,works such as k -ary n -cubes include hypercubes,mesh and torus networks.But the uniqueness of the interconnect architecture we seek is con-tained by the physical constraints characterizing the line card board.Area and I/O pins are limited on the line card.Hence,the number of alternative designs that can physically

0164-1212/$-see front matter ó2006Elsevier Inc.All rights reserved.doi:10.1016/j.jss.2006.01.033

*

Corresponding author.Tel.:+14078234758;fax:+14078235835.E-mail addresses:jengel@https://www.360docs.net/doc/ef12620454.html, (J.Engel),dlacks@https://www.360docs.net/doc/ef12620454.html, (https://www.360docs.net/doc/ef12620454.html,cks),tkocak@https://www.360docs.net/doc/ef12620454.html, (T.Kocak).

https://www.360docs.net/doc/ef12620454.html,/locate/jss

The Journal of Systems and Software 79(2006)

1701–1714

and functionally?t,given those constraints,is limited.Each embedded chip has?xed and limited number of I/O pins. Therefore,a low-dimensional,packet-switched network may be a good solution.

Our objective is to explore k-ary n-cube networks in the context of line card designs.One of our goals is to?nd which k-ary n-cube based interconnect architecture can be the best candidate to replace existing line card commu-nication mechanisms,such as shared-bus or crossbars. Both shared-bus and the crossbar cannot scale well as the number of modules(PEs or memories)connected to it increases.In addition,the shared-bus requires a distributed arbitration mechanism,as the number of modules con-nected to it grows,thus,adding latency and space to the overall system.Pin constraints bound the bus size that can be interfaced with the NPU(Kumar et al.,2005). Hence,only a packet-based network-on-board can provide the required performance improvement between the NPU and o?-chip memory modules.

Our simulation model includes statistical data such as IP length distribution(Test Procedures,2001)and physical measures of PCB placement and spacing,as well as network properties such as IP packet size,in order to increase the accuracy of our calculations.In addition,we apply true IP network properties such as switching,propa-gation and routing latencies.The simulator provides real time performance analysis with detailed metrics on packets processed at each simulation cycle and overall detailed results at the end of each simulation.The user inputs param-eters using an easy to use GUI or command line interface.

1.1.Related work

There are many discrete event network simulation and modelling tools available that contain some of the architec-tural features and functionalities we incorporate in our model.However,to the best of our knowledge,none of these simulation frameworks are capable of delivering the physical and functional attributes required to emulate o?-chip communications on line cards.We would like to con-sider three of these simulators(NS-2,Qualnet and OPNET)and make the distinction between their applica-tions and ours.We chose NS-2,Qualnet and OPNET since they are well known network simulators currently deployed by universities and network design companies.

NS-2is an object-oriented,discrete,event-driven net-work simulator developed at UC Berkeley,written in C++and OTcl.NS2is primarily useful for simulating local and wide area networks and it supports simulation of TCP, UDP,routing,and multicast protocols over wired and wireless networks(Jung et al.,2003;Kornblit et al., 2004).The Qualnet is a real-time simulation framework, developed by Scalable Network Technologies(SNT),to emulate the communications of multiple network models. Qualnet includes a rich3D-visualization interface to pro-vide the user with control over data packets,network topology and performance evaluation.Qualnet supports wireless and ad hoc networks as well as parallel and distrib-uted architectures(Hsu et al.,2003).In addition,it sup-ports multiple routing protocols such as BGP,SIP,RIP, ARP,and BRP.Some related applications that can bene?t by using this network simulator include:microwave tech-nologies,high frequency radio communications or satellite communications.OPNET’s network modelling and simu-lation environment delivers a scalable simulation engine that can emulate wireless,point-to-point and multi-point network links.It has the capability to support routing pro-tocols such as voice,HTTP,TCP,IP,Ethernet,frame relay and more(Wu et al.,2001).Some of the application best suit for this simulator are mobile,cellular,ad hoc,wireless LAN,and satellite networks.The OPNET simulator allows the user to custom design tra?c models since it sup-ports?nite state machines and object-oriented modelling (Chang,1999).

These network simulators are not designed to emulate o?-chip communication environment required for our application based on the following di?erentiations:?Physical attributes:none of these simulators include spe-ci?c PCB physical properties which have a great e?ect on the interconnect performance.Physical properties are crucial to meet the stringent area restrictions on line cards.

?Applications:all three simulators?t better for LAN, WAN,mobile and ad hoc communications,not small scale interconnects which require di?erent routing algo-rithms and?ow control mechanisms.The line card sim-ulator must include message?ow enhancement features such as virtual channels and sub-channeling.?Message control:our interconnect simulator provides control of how to deliver messages,perform statistics, gather data,route the packets through the network and run auto test cases.Furthermore,the user has more control of how to save and re-run data using the simu-lator options menus,rather than learning OTcl or Parsec.

?Participants:while our simulator models communica-tion among PEs and memories,the other simulators include other participants such as PCs,satellite commu-nication,routers or other moving objects.?Communication medium:most of communication medi-ums used in these simulators have di?erent signal prop-agation characteristics and performance.Our o?-chip interconnect model is a small scale network in which packets propagate from point-to-point via PCB buses no longer than1inch in length.

2.Preliminaries

2.1.k-Ary n-cube networks

A k-ary n-cube network consists of N=k n nodes,where n represents the dimension of the network and k represents

1702J.Engel et al./The Journal of Systems and Software79(2006)1701–1714

the number of nodes in each dimension.Fig.1presents 8-ary2-cube and4-ary3-cube networks as captured from the interconnect simulator.Each node in k-ary n-cube interconnect is uniquely labeled and elements of the same plane are connected together.We also introduce a modi?-cation of2-ary3-cube interconnect architecture by extend-ing it in the x-direction(Fig.2).PEs and memories are distributed throughout the interconnect in di?erent con?g-urations and allow each PE to use multiple memories as storage as well as data sharing with other processing elements.

Each node is connected to all of its nearest neighbors via bi-directional channels.The address/location of a node can be represented as a vector consists of two bit-vector?elds (Dally,1990).Fig.2represents the3D-mesh interconnect architecture,which is based on a2-ary3-cube network, that is extended in the x-direction.The3D-mesh intercon-nect is a packet-based multiple path interconnect that allows network packets to be shared by di?erent processing elements(PE)and memory modules(M)on the network line card.Memories are distributed around processing elements,such as tra?c manager,QoS co-processor or classi?cation processor,to allow data sharing among mod-ules and direct processor memory storage.If a link goes down,not only should the fault be limited to the link, the additional links from the intermediate nodes should ensure the connectivity continues.

Processors and memories communicate by using mes-sage passing mechanisms.Each message is transmitted independently.Each message is partitioned into smaller data segments,also called?its,which contain the maxi-mum amount of data(in bits)that can be transmitted in one cycle from one node to another.Each cycle another?it of the same message is transmitted.Flits of the same mes-sage follow one another in a pipeline manner.Therefore,a message is also referred to as a worm since the movement of the message within the interconnect resembles a worm movement.Virtual channels(VCs)allow worms to be stored within a node if all of the output ports of that node are busy transferring other messages.This technique pre-vents worm transmission failures by holding a worm within a node until one of its ports becomes

available.

Fig.1.(a)4-Ary3-cube and(b)8-ary

2-cube.

Fig.2.3D-mesh interconnect architecture.

J.Engel et al./The Journal of Systems and Software79(2006)1701–17141703

Channels can change their con?guration by dividing their width into two or four sub-channels (shown in Fig.3).Sub-channeling (SC)permits worms to share the same channel simultaneously.Although per-worm the channel has smaller capacity when sub-divided,it provides worms with extra ?exibility in routing through the inter-connect instead of being bu?ered or retransmitted.

The message passing algorithm adaptively routes worms according to prede?ned guidelines and by incorporating interconnect tra?c conditions.The ?rst guideline ensures that a worm will always attempt to take the shortest path possible to its destination.If the port required to be taken by the shortest path rule is occupied as a result of high traf-?c load,it will test the availability of other ports.The movement of each worm is recorded by the worm manager.Therefore,the second guideline utilizes past moves to determine the next node that a worm will take towards its destination and avoids certain consecutive moves to inhibit deadlock/livelock situations.The last guideline pre-serves the worm relative movement from its source node towards its destination.That is it will never reverse its direction towards its source.

PEs and memories can be physically located in many di?erent con?gurations depending on the number of PEs and memories required to complete packet processing tasks.The location and ratio between the number of PEs to memory modules will determine the average distance that a message has to pass in order to reach destination.Average distance has a direct e?ect on the interconnect per-formance.Intuitively,as network dimensions increase more con?gurations can be formed.During performance evaluation we used the con?guration that result in the shortest average path to get the best performance.

One objective,which can be gained by utilizing the sim-ulator,is to ?nd the optimal value of k and n to achieve best performance.The optimal con?guration depends on many design constraints as well,such as channel width/density,number of elements connected to the network,and cost.In general,when node delays are neglected and constant bisection width is assumed,a network with lower dimensions has lower latency than higher dimensional net-works (Agarwal,1991).

3.k -Ary n -cube interconnect simulator architecture The simulator architecture,shown in Fig.4,depicts the interconnect interaction with the control modules which adjust,collect and modify the interconnect settings,data ?ow,and performance metrics.These architectural attri-butes are built in the functionalities of the modules.The simulator con?guration manager sets the interconnect type,its properties (wire propagation delay,switching delay or routing delay)and enable/disable enhanced fea-tures such as channel width,VC on/o?,and bidirectional channel.

The interconnect properties are set by the user interface and are recorded to allow the con?guration manager to be updated via the worm manager.The worm manager uti-lizes interconnect properties and con?guration parameters in order to set other modules accordingly in the system that participate in the simulation.The tra?c sampler continu-ously records performance data such as throughput,latency,routing accuracy,interconnect bandwidth utiliza-tion and interconnect resources utilization.This informa-tion is feedback to the worm manager which adjusts worm generation rate and load balances the tra?c.The routing algorithm receives each individual worm location and its destination node from the worm manager.Then,it determines the shortest route possible for each worm by avoiding spots of heavy tra?c.

The worm jar is a storage module that contains worms.In the simulator we use two instances of the worm jar:one jar is for worms waiting to enter the interconnect and the other jar contains worms that were processed.The total number of worms during simulation are initially deter-mined by the user.The scheduler is responsible to inject worms into the interconnect taking into account the total network capacity and tra?c load.Since the worm manager knows the total number of worms that are modelled throughout the simulation,it must inform the scheduler the end of the simulation (when there are no more worms to model).

The simulator accounts for all practical parameters characterizing o?-chip interconnect architectures such as switching delays (T s ),routing delays (T r )and propagation delays (T w )as well as the complete functionality of each system components (nodes,links,PE/Memory,interfaces,virtual channels,and channel partitioning)(Zhang,2003).The user has the option to change each of these parameters in case new technology introduces higher standards.Simu-lation time is based on a unit cycle which is equal to one clock cycle (T w +T r ).All other delays are calculated as multiples of it.That provides the advantage of having sin-gle uniform simulation

clock.

Fig.3.Four sub-channels containing four worms simultaneously.

1704J.Engel et al./The Journal of Systems and Software 79(2006)1701–1714

Message size in bytes and message generation-time are obtained by using pseudo-random number generator,which is utilized to resemble the randomness of packet transmission by both processors and memories.Each worm is linked to performance-bookkeeping function which records its latency,throughput,simulation cycles,failures,and route-taken from the moment the worm enters the interconnect until it completely reaches its https://www.360docs.net/doc/ef12620454.html,prehensive performance results are provided at the end of each simulation in a spreadsheet.3.1.Simulator modelling approach

The high-level design of the simulator is comprised of four sets of C++classes (Fig.5a)supporting:the intercon-nect topology and con?guration (Interconnect),user inter-face (Interface),worm controller and administrator (WormManager),and worm structure and characteristics class (Worm).The worm contains a header ?eld and data payload.

The cyclic diagram (Fig.5b)depicts the relationship between the interconnect architecture,the simulator and the physical properties of the available resources.The interconnect architecture represents the physical structure and includes all the hardware required to implement it.The properties represent two types of parameters:physical parameters of electrical components comprising the inter-connect (such as wire delays,switching delays,routing delays),and parameters of additional features,which enhance the interconnect performance (for example,channel partitioning,virtual channels,interconnect con?guration).The simulator emulates the interconnect functionality in order to evaluate and compare di?erent con?gurations and settings.

Interconnect properties a?ect worm routing ?exibility and resources it can use while propagating through the interconnect,such as VCs and SC.The port class contains VCs and SCs which are modelled as logical topologies on top of the physical network architecture.VCs as well as SCs have a great e?ect on the worms transmission success/failure rates and deadlock/livelock avoidance.Although VCs improve routing accuracy and reduce worm transmission failure rate,they also increase the worm latency and interconnect implementation costs.The worm manager class records worm data,arrival and depar-ture time stamps of worms,and controls the worm genera-tion rate in order to load balance the number of worms processed simultaneously within the interconnect.The worm class encapsulates the properties of a worm such as the header with source/destination ?elds and the route that the worm takes through the interconnect.

The worm routes itself through the interconnect while continuously being monitored by the worm manager.The adaptive routing algorithm is used by the worm to deter-mine the best available path that it can take to reach desti-nation.The routing algorithm is derived and based on Chiu (2000),Dally (1992),and Lysne (1999).The worm updates its shortest path coordinates with each movement to ensure its optimal path even when it is required to take a detour as a result of hot-spot node.Fig.6shows a UML

class

Fig.4.Simulation control modules.

J.Engel et al./The Journal of Systems and Software 79(2006)1701–17141705

diagram of the interconnect architecture (Cope,2005;Nak-ata et al.,2002).A single type of interconnect is a set of faces which each comprise of multiple nodes.Within each node there are multiple ports.A node can be modelled as either a memory or a PE.Hence,the node still possess the same structure and functionality as any node,but it reserves one port as an I/O port to the device.

The simulation setup shown in Fig.7is an abstract view of the high level system components and their interaction in order to initialize and execute the simulation.

First,the simulation properties are set by the user.These properties are crucial for worm generation,timing delays,

and other simulation aspects.Then,the messages (worms)are created and are placed in a data structure (jar).Since the interconnect con?gurations can be changed,PE and memory locations will be changed accordingly.Therefore,source/destination addresses must be correctly set before the worms can be generated.When the user chooses to run the simulation,the properties and the data of the worms in the jar are recorded in separate ?les.The inter-connect receives worms from the jar of generated worms according to a probability called worm generation rate (GR)which is controlled by the user.In addition,the user can determine how many worms can occupy the intercon-nect at all simulation cycles by changing the value of the max worms in interconnect variable (MWII).If no value is set for this variable,the default value is unlimited num-ber of worms.The worms that enter the interconnect are modelled until they reach their destination.All runtime worm data is collected in a separate output ?le which provides individual details about each worm.After the complete simulation is modelled,several spreadsheet ?les are generated recording the performance of the simulation.4.Software components

Fig.8portrays a dynamic model (action oriented)of the routing algorithm class and its subclasses with interconnect system components and the worm manager class.

This model determines the actions performed by the routing algorithm in order to maneuver each worm

within

Fig.5.(a)Class relationship diagram and (b)cyclical relationship

diagram.

Fig.6.UML class diagram of interconnect

architecture.

Fig.7.Simulation setup.

1706J.Engel et al./The Journal of Systems and Software 79(2006)1701–1714

the interconnect with respect to its current position,its des-tination and tra?c conditions (Schach,1996).The routing algorithm is closely coupled with the worm manager since the worm manager controls worms entering and leaving the interconnect while the routing algorithm controls the worms within the interconnect.First,the routing algorithm analyzes the source node type (where the worm is gener-ated)and the enabled interconnect features such as virtual channels,bidirectional channels and PE–M con?guration.Then,it checks the preferred (shortest path)direction in which the worm needs to move.The routing algorithm scans each node’s port and dictate the movement of the worm giving priority to ports which are pointing in direc-tion towards its destination.If none of the ports are avail-able,the routing algorithm will check the availability of virtual channels.If enabled,the worm will be queued into one of the virtual channels until one of the ports clears.If virtual channels are not available then the routing algo-rithm noti?es the worm manager of a worm failure.This will result in a retransmission of the same worm.

Fig.9depicts a data ?ow diagram (DFD)of the user interface module.DFD charts assisted us in determining what to automate in the simulator design and which data must be inputted exclusively by the user (Fritz and Sargent,1995;Schach,1996).The user has two choices:using default settings or changing settings/properties in order to simulate the interconnect with di?erent con?guration.Once the interconnect type and con?guration are de?ned,the user must complete the following steps before the sim-ulation execution:

?Select if new worms will be generated or worms should be restored from an existing ?le.

?Determine the number of worms to simulate.

?Decide if worms are generated randomly or manually.?Input the number of sampled throughput points (include the initial sampling point and the number of simulation cycles between samples).

?Select if the newly generated worms will be saved or not.

5.Optimization strategies

Object-oriented software design optimization strategies such as STL,singleton and pure virtual functions

have

Fig.8.(a)Dynamic model of routing algorithm and (b)data ?ow diagram of the user interface

module.

Fig.9.Sequence diagram of user–simulator interaction.

J.Engel et al./The Journal of Systems and Software 79(2006)1701–17141707

been employed to provide a?exible,extensible,and robust means to establish network hardware structures.These optimization strategies are vital implementation methodol-ogies to our network benchmark model in order to obtain higher modularity and lower integration complexity.A systematic usage of these functionalities throughout the simulator design lead to a better model which improves system performance and supports future upgrades such as additional types of networks,protocols and/or?ow con-trol mechanisms.

5.1.The Singleton class

The singleton classes(Townsend,2002)such as Worm-Manager and Interconnect shown in Fig.10,guarantee that only one class instantiation is created.The single instance is held as a static variable as a private member of the class.These singleton classes are not automatically initialized.Instead,initialization occurs the?rst time that singleton class’create method is called by the client.The create method also allows the callers to access methods of that singleton class.In a similar manner it can destroy the object by calling destroy.The interconnect is a single-ton class,that is only one interconnect is created per simu-lation.The WormManager creates a new interconnect for each simulation and destroys it when done.The reason for this is that there might be di?erent con?gurations which require construction of the object in di?erent ways.Fig.10 shows all the objects and functions(public and private) included in each of these singleton classes.

5.2.Pure virtual functions

The SaveRestoreInterface class provides save and restore functions which are pure virtual functions which forces derived classes to override(Pure Virtual Functions and Abstract Classes,2005).The following is an example of pure virtual function signatures:

class SaveRestoreInterface{

public:

virtual File&save(File&?le)=0;

virtual File&restore(File&?le)=0};

In the save/restore functionality,we utilize a sentinel as a safeguard to assure that the correct version of code is used. The sentinel is recorded in the saved?le.Upon restore,it is veri?ed that the saved?le matches the current software version.

const int save_restore_sentinel=3;

File&WormManager::save(File&?le){

?le.save(save_restore_sentinel);

Properties::save(?le);

return?le.save(worms_to_process);}

File&WormManager::restore(File&?le){

int tmp_save_restore_sentinel;

?le.restore(tmp_save_restore_sentinel);

if(tmp_save_restore_sentinel!=save_restore_senti-nel){

std::cerr ’’cannot restore?le

tmp_save_restore_sentinel std::endl;

restored=false;

else{Properties::restore(?le);

Interconnect::create;()

?le.restore(worms_to_process);

restored=true;

return?le;}

5.3.System design with Standard Template Library (STL)functions

The interconnect is modelled using a map data structure from the Standard Template Library(STL).The STL is a general purpose library of algorithms and data structures. The STL enables generic programming where reusable functions,data structures and algorithms are

available

Fig.10.(a)WormManager singleton and(b)interconnect singleton.

1708J.Engel et al./The Journal of Systems and Software79(2006)1701–1714

for the programmer(Introduction to the Standard Template Library,2003).The interconnect is basically con-structed of three main components:a face,a node,and a port(Fig.11).For the3D-mesh interconnect,each face has four nodes at the corners.Each node has four ports. Therefore,a map is created for each component to orga-nize the connectivity and construct the interconnect struc-ture.Each component has an ID and an integer variable which represents the number of instances available (Meyers,2001).

Face#ID to face map:

typedef std::map h int,Face i FaceMap;

Node#ID to node map:

typedef std::map h int,Node i NodeMap;

Port#ID to port map:

typedef std::map h int,Port i PortMap;

VC#ID to VC map:

typedef std::map h int,VirtualChannel i MemoryManager;6.Evaluation and results

During execution,the network simulator provides two windows to control simulation pace and collect simulation data.The runtime data window(bottom right side of Fig.12)shows performance metrics which are updated on-the-?y.In addition,runtime data is also recorded in the output spreadsheet?les.The pacing window(on the bottom left side of Fig.12)allows the user to control the pace of simulation and if required to pause it completely.

https://www.360docs.net/doc/ef12620454.html,tency and throughput analysis

Latency represents the time it takes for a worm to reach its destination.Depending on the worm movement,latency sums wire transfer,switching and routing delays at each cycle.The resulting latency is an average of latencies col-lected from all worms modelled at the end of the simulation.

We chose three representative k-ary n-cube intercon-nects for our simulations:8-ary2-cube,4-ary3-cube and 3D-mesh(all three interconnects have64nodes).Fig.13 shows a comparison among all three interconnects with VC and channel partitioning enabled.The results shown are an average of10di?erent simulations with both short (128B–1KB)and long(1KB–8KB)worms and identical interconnect settings.The lowest latency was recorded for the3D-mesh,while the4-ary3-cube network has slightly higher latency than the3D-mesh.Throughput is measured by taking samples of the total bits processed within the interconnect at each cycle.Throughput signi?cantly increases when VC are enabled since VC allow more worms to occupy the interconnect without transmission

failures. Fig.11.Structural hierarchy of classes:face,node,

port.

Fig.12.Simulation execution with runtime data and pacing windows.

J.Engel et al./The Journal of Systems and Software79(2006)1701–17141709

Fig.14shows that the highest throughput was reached by the 3D-mesh interconnect for both short and long messages.

6.2.Worm allocation and distribution

Worm allocation and distribution measurements,depicted in Fig.15,show three groups of worms:worms that are currently propagating in the interconnect,worms that are waiting in jar to be modelled and worms that are ?nished and reached their destinations.Fig.15provides a good indication of the worm manager functionality.The ?gure shows that the number of currently modelled worms (worms in the interconnect)increases as the number of worms waiting in the jar and the number of already mod-elled worms (?nished)decreases.

When VCs are enabled,more worms occupy the inter-connect with faster rate than presented in Fig.15.Fig.16shows that as more worms are modelled,the number of worms waiting to be modelled diminishes.It is also notice-able that when VCs are enabled more simulation cycles are required.

6.3.Routing accuracy

Routing accuracy measures how close the actual path of each worm is to its shortest path.Routing accuracy is cal-culated by taking the ratio between the shortest path possible to the actual path taken,which signi?es the worm’s deviation from its shortest path.Fig.17shows a simulation of 100worms using 3D-mesh interconnect with VC disabled and no sub-channelling.At the top of the ?g-ure the line portrays the number of additional links

passed

https://www.360docs.net/doc/ef12620454.html,tency

comparison.Fig.14.Throughput

comparison.

Fig.15.Worm allocation and

distribution.

Fig.16.Worm distribution with VC =8KB.

1710J.Engel et al./The Journal of Systems and Software 79(2006)1701–1714

by each worm until it reaches its destination.On the bot-tom part we see the deviation of each worm (top line)from its shortest path (bottom line).If both lines overlap,then the worm has taken its shortest path.The path worm takes depends on the tra?c load at certain nodes of the intercon-nect.As the load increases,most worms deviate from their shortest path and adaptively propagate to their destination avoiding areas of hot-spots (Lysne,1999).

Fig.17shows the number of channels passed for each worm modelled using 3D-mesh interconnect.The top line shows the percentage of deviation from the shortest path.For example,if the top line points for a certain worm #ID to 100,that means the worm has taken the shortest path possible.If the value of the line is equal to 20,the worm deviated from its shortest path by 80%(more chan-nel links).At the bottom of the ?gure there are two over-lapping lines.The thin line represents the actual path taken (measured in channel links).The thick line represents the shortest path.Therefore,when both line completely overlap each other for a certain worm,that worm has taken the shortest path.As the number of channel links passed increases with respect to the shortest path possible,the thin line becomes further apart from the thick line.In Fig.17neither VC nor SC were enabled.Therefore,more worms deviated from their shortest path due to high tra?c load at certain interconnect nodes.6.4.Interconnect and bandwidth utilization

Interconnect bandwidth utilization measures the num-ber of occupied channels (or sub-channels)with respect to the total number of channels available in the intercon-nect.Fig.18portrays that the highest bandwidth utiliza-tion is achieved by using the 4-ary 3-cube network,while the 8-ary 2-cube has the lowest utilization rate.Sub-

channelling improves bandwidth utilization as the channel is partitioned into more sub-channels.The combination of VC and SC brings all interconnects close to their full capacity.

Interconnect utilization counts the number of busy ports within each tra?c controller per simulation cycle.At the end of the simulation it provides the average number of ports that were set to busy status out of the total number of ports available in the interconnect throughout simula-tion.The results of interconnect utilization show very close relationship to bandwidth utilization (Fig.19).Again,4-ary 3-cube ports are set to busy status more often than the 3D-mesh or 8-ary 2-cube.Although interconnect utili-zation seems an equivalent measure to bandwidth utiliza-tion,it is a little di?erent since the port status is not directly related to the channel usage.An output port can stay in the not-busy state if a worm that intends to use it is bu?ered into virtual channels.Since each

tra?c

Fig.17.3D-mesh worm deviation from shortest

path.

Fig.18.Bandwidth utilization rate.

J.Engel et al./The Journal of Systems and Software 79(2006)1701–17141711

controller has four ports,the channel connected to the non-busy port can be utilized by a worm entering from a di?er-ent direction.6.5.Failure rate

Failure rate is a measure of the number of worms,out of the total worms generated,that were retransmitted during simulation.Retransmission takes place when a worm is blocked and it cannot obtain the resources it required to maintain an active status within the interconnect.For example,when VCs are disabled,then a worm will require retransmission if it cannot be routed to any output port within a certain node for more than one simulation cycle.Fig.20depicts a failure rate comparison for all intercon-nect types with VC switched to enabled/disabled.This ?g-ure show that using VCs signi?cantly reduces failure rate.Moreover,the size of the VC has a major e?ect on failure rate as well (Fig.21).As the size of the VC increases more worms can be bu?ered for longer periods of time within each node instead of failing and being retransmitted (Dally,1992).

At each cycle there is a random number of worms gen-erated in which only a fraction of those worms enter the interconnect.The software interface provides the user with a variable called generation rate (in units of %)which determines what percentage of worms generated each cycle will enter the interconnect.The rest of the worms are left in the jar.For example,if at a certain simulation cycle 10worms were generated and the parameter g is set to 70%then each worm has a 70%chance to enter the interconnect until the maximum occupancy of the interconnect is reached.The value of g is shown in Figs.20and 21in the x -axis.The combination of VCs and SCs reduces fail-ure rate dramatically,on average,to less than 10%for all interconnects.This is a 75%reduction since VCs allow worms to be bu?ered within a node until one of the ports becomes available.SCs provide worms with additional alternative paths to surpass areas of tra?c congestion.6.6.Routing accuracy vs.hot-spot nodes

In this simulation the paths taken by all worms,using 3D-mesh,8-ary 2-cube and 4-ary 3-cube interconnects,were recorded.Then,the paths were analyzed to collect the nodes which were most frequently used and as a result caused other worms to deviate from their shortest path to avoid transmission failure.

Results given in Figs.22and 23for all interconnects show that some hot-spot nodes caused approaching worms to deviate from their shortest path by 50–60%more channel links.For example,Fig.22depicts that hot-spot in face 11node 3(F[11],n[3])caused six approaching worms to devi-ate from their shortest path by 62.5%.Hot-spots patterns are not repeated in the same locations.Tra?c is randomly generated with random message lengths and from random nodes.Since the adaptive routing algorithm changes the path the worms take in each simulation,every simulation creates hot-spots in di?erent locations and in di?erent fre-quencies.Fig.23shows a hot-spot which occurred in face 3node 6(F[3],n[6]),that caused approaching worms

to

Fig.19.Interconnect utilization

rate.

Fig.20.Failure rate with VC switched

on/o?.

Fig.21.Failure rate vs.VC size.

1712J.Engel et al./The Journal of Systems and Software 79(2006)1701–1714

deviate from their shortest path by an average of 85%.Although only few hot-spots occur per simulation,their

e?ects on performance were signi?cant.As the rate of hot-spot increases (a function of tra?c load),worms tend to deviate from their shortest path more frequently and,as a result,the overall interconnect latency increases.6.7.K-Ary n-cube interconnects performance comparison with common interconnects

In this section,we compare 3D-mesh,8-ary 2-cube,and 4-ary 3-cube interconnects with other currently used high-performance interconnect technologies such as Hypertransport (HyperTransport Consortium,2005),In?niband (In?niband Trade Association,2000)and PCI-Express (PCI Special Interest Group,2003;Sassone,2003).

We used reported results provided by each individual vendor to compare with our results.In addition,the perfor-mance properties of these technologies take into account a constant channel size of 32-bits and a single communica-tion link.For the 3D-mesh interconnect the settings are:channel width is 32bits,interconnect size is 16cubes,num-ber of worms generated is 10,each worm is 1KB in size.Virtual channels as well as channel partitions were enabled.The throughput comparison results are shown in Fig.24.The throughput values of the 3D-mesh,8-ary 2-cube and 4-ary 3-cube interconnects represent the average through-put of each interconnect.3D-mesh shows superior results compared to all of its competitors reaching a peak through-put of 452Gbps (about twice the throughput of the best interconnect available not including the other types of k -ary n -cubes tested).7.Conclusions

We presented an event-driven,custom-designed interconnect simulation environment to evaluate

the

Fig.22.3D-mesh routing accuracy vs.hot-spot

nodes.

Fig.23.4-Ary 3-cube routing accuracy vs.hot-spot

nodes.

Fig.24.Throughput comparison:k -ary n -cube interconnects https://www.360docs.net/doc/ef12620454.html,mon interconnect technologies.

J.Engel et al./The Journal of Systems and Software 79(2006)1701–17141713

performance of o?-chip k-ary n-cube interconnect architec-tures for line cards.The interconnects are examined using our network simulator in order to?nd which of the inter-connects can provide the highest performance and memory bandwidth to replace the existing shared-bus systems.The simulator uses the state-of-the-art software design tech-niques to provide the user with a?exible yet robust tool that can emulate multiple interconnect architectures under non-uniform tra?c patterns.The simulator o?ers the user with extensive control over network parameters,perfor-mance enhancing features and simulation time frames that make the platform as close as possible to the physical line card features.Performance results show that k-ary n-cube topologies can sustain higher tra?c load than the currently used interconnects.We demonstrated that the?ow control mechanisms such as virtual channels(VC)and sub-chan-neling(SC)have an important impact on the interconnect performance.VC and SC mechanisms,together,reduce the transmission failure rate signi?cantly by75%and increase the interconnects bandwidth utilization in the range of15–25%depending on the topology.We included a variation of2-ary3-cube,called3D-mesh that provides a better processor-memory distribution under non-uniform tra?c.The combination of the3D-mesh interconnect and our adaptive routing algorithm facilitate to reach the highest throughput of452Gbps,this is better than twice the throughput of the leading solution in the marketplace. 3D-mesh meets both the stringent performance requirements and the physical constraints on the line card while enabling future scalability to adopt higher line rates.

References

Agarwal, A.,1991.Limits on interconnection network performance.

IEEE Transactions on Parallel and Distributed Systems2(4),398–412.

Chang,X.,https://www.360docs.net/doc/ef12620454.html,work simulations with OPNET.In:Proceedings of the1999Winter Simulation Conference,pp.307–314.

Chiu,G.M.,2000.The odd–even turn model for adaptive routing.IEEE Transactions on Parallel and Distributed Systems11(7),729–738. Cope,M.C.,2005.Object Oriented Analysis and Design Using UML, white paper,Ratio group.Available from:.

Dally,W.J.,1990.Performance analysis of k-ary n-cube interconnection networks.IEEE Transactions on Computers39(6),775–785. Dally,W.J.,1992.Virtual-channel?ow control.IEEE Transactions on Parallel and Distributed Systems3(2),194–199.Fritz,D.G.,Sargent,R.G.,1995.An overview of hierarchical control?ow graph models.In:Proceedings of the IEEE Simulation Conference,pp.

1347–1355.

HyperTransport Consortium,HyperTransport Technology Speci?ca-tions,2005.Available from:.

Hsu,J.,Bhatia,S.,Takai,M.,Bagrodia,R.,Acriche,M.J.,2003.

Performance of mobile ad hoc networking routing protocols in realistic scenarios.In:Proceedings of the Military Communications Confer-ence,vol.2,pp.1268–1273.

In?niband Trade Association,2000.In?niband architecture speci?cation, rev. 1.0,October2000.Available from:

org>.

Introduction to the Standard Template Library,2003.SGI,white paper.

Available from:. Jung,J.W.,Mudumbai,R.,Montgomery, D.,Kahng,H.K.,2003.

Performance evaluation of two layered mobility management using mobile IP and session initiation protocol.In:Proceedings of the Global Telecommunications Conference,vol.3,pp.1190–1194. Kornblit,R.,Schwartzmann,E.,2004.Multicast Protocols Evaluation in Wireless Domains,Project report,Technion,Israel.

Kumar,R.,Zyuban,V.,Tullsen,D.M.,2005.Interconnections in multi-core architectures:understanding mechanisms,overheads and scaling.

In:Proceedings of the IEEE32nd International Symposium on Computer Architecture,June2005.

Lysne,O.,1999.Deadlock avoidance for switches based on wormhole networks.In:Proceedings of the Annual International Conference of Parallel Processing,pp.68–74.

Meyers,S.,2001.E?ective STL:50Speci?c Ways to Improve Your Use of the Standard Template Library.Addison-Wesley,Boston,Mass. Nakata,T.,Kuwamura,S.,Zhu,Q.,Matsuda,A.,Shoji,M.,2002.An object-oriented design process for system-on-chip using UML.In: Proceedings of the15th international symposium on System Synthesis, pp.249–254.

PCI Special Interest Group,2003.PCI express base speci?cation rev.1.0a, April2003.

Pure Virtual Functions and Abstract Classes,Microsoft corp.,MSDN library,2005.Available from:.

Sassone,P.,https://www.360docs.net/doc/ef12620454.html,mercial trends in o?-chip communication,Tech-nical Report,Georgia Institute of Technology,May2003. Schach,S.R.,1996.Classical and Object-Oriented Software Engineering, third ed.Irwin group.

Test Procedures,March5,2001.Available from:

com>.

Townsend,M.,2002.The Singleton Design Pattern,Microsoft corp., MSDN library,February2002.Available from:https://www.360docs.net/doc/ef12620454.html,/library/default.asp?url=/library/en-us/dnbda/html/singleton-despatt.asp.

Wu,H.,Fujimoto,R.M.,Riley,G.,2001.Experiences parallelizing a commercial network simulator.In:Proceedings of the2001Winter Simulation Conference,pp.1353–1360.

Zhang,Y.,2003.Microstrip-multilayer delay line on printed-circuit board, Technical Report,University of Nebraska,Lincoln,April2003.

1714J.Engel et al./The Journal of Systems and Software79(2006)1701–1714