PerfView- A Tool for Source-Level Performance Analysis on the Itanium Processor Family

PerfView - A Tool for Source-Level Performance Analysis on the Itanium Processor FamilyAnanth LingamneniMaster of Science, Computer & Information SciencePlan B ProjectUniversity of MinnesotaAdvisor : Prof. Wei-Chung HsuAbstractApplication performance analysis in modern microprocessors has become extremely complex due to substantial instruction level parallelism, complex processor pipelines and deep memory hierarchies. Performance analysts need to have a thorough understanding of the dynamic behavior of programs in order to identify and fix performance bottlenecks. In order to help in the performance analysis process, modern day processors are providing hardware support in the form of performance registers that help to capture microarchitectural events that occur during the running of a program. However, the data provided by the hardware registers is at a very low level and an extensive effort has to be made by performance analysts to make sense of the data. Therefore, it is extremely beneficial to make use of performance analysis tools that can assemble various types of complex performance related data available from the performance registers and provide a high level summary of the data. In this paper, I discuss PerfView, a performance analysis tool for the Itanium Processor family, that can be used to correlate low level program profile data with corresponding source code, and provide an execution trace of the statements leading to the regions of performance bottlenecks.1IntroductionMicroprocessor speeds have been increasing at a dramatic rate leading to very high peak performances. However, other components of the computer system such as the memory system have not been able to keep up with these enormous speed improvements, and have consequently proved to be a bottleneck in achieving processor peak performance. The complex memory systems with deep hierarchies result in high latency cache misses, and this significantly affects the performance of the processor. With complex pipelines, branch mispredictions too have a significant effect on the performance of the processor. It is therefore becoming increasingly necessary to analyze the dynamic behavior of applications to identify and fix performance bottlenecks such as cache misses and branch mispredictions in order to achieve a major fraction of the processor peak performance. However, the performance analysis of applications has also become extremely difficult due to the substantial instruction level parallelism, complex processor pipelines and deep memory hierarchies.In order to help in performance analysis, modern microprocessors are providing special hardware support in the form of performance registers that capture data related to events affecting performance. Specifically, the IA-64 architecture specifies a single hardware component called the Performance Monitoring Unit(PMU) which provides detailed low-level profiling information, and captures information related to microarchitectural events that occur during the running of a program. The PMU can be programmed to capture information about specific events at specific sampling periods. Though the information provided by the hardware registers is very useful, it requires a substantial effort on the part of the performance analyst to analyze the low-level data, to identify the key bottlenecks and the causes for these bottlenecks. In light of this, I propose PerfView - a tool to provide a high-level view of the performance data obtained from the PMU of the Itanium Processor family. PerfView processes the data captured by the performance registers and identifies the regions of the source program that are responsible for a high number of performance related events. The information is provided at the source code and assembly code level. After highlighting lines in the source codehaving high event activity, PerfView also provides for a sorted set of execution traces leading to the specific source line. This helps to identify the cause for specific events.2PerfView Architectural Framework2.1 Itanium-2 Performance Monitor Unit(PMU)The IA-64 architecture specifies special hardware support in the form of the PMU to help in characterizing application performance. The interface to the PMU consists of a set of dedicated registers that can be programmed to count occurrences of certain micro-architecture events such as number of cycles elapsed or the number of cache misses in the L1 cache. The register file is composed of the performance monitor data(pmd) registers which collect the data, and the performance monitor configuration(pmc) registers which are used to configure the events to be monitored.The Itanium-2 PMU provides four 48-bit performance counters that help in counting up to 4 micro-architectural events simultaneously. The 4 counting monitors use 4 pairs of registers(pmc4-pmc7 and pmd4-pmd7). The configuration registers pmc4-pmc7 are used to configure the events that are to be counted by the data registers pmd4-pmd7. Each of these 4 data registers contain a counter that is incremented each time an event specified by the configuration register occurs. When a counter eventually overflows and wraps around to 0, this condition is detected and stored in the pmc0-pmc3 registers, the overflow status registers. The PMU can be programmed to generate an interrupt whenever a pmd register overflows. This interrupt facility can be used by the operating system to maintain a 64 bit software counter for every hardware counter. The pmd register that overflowed can be determined by inspecting the overflow status registers, pmc0-pmc3.The PMU also provides filters such as opcode matchers and address range matchers that help to provide a finer control over the events being monitored. With the help of these counters and filters, the PMU can be used for cycle accounting to help in breaking down the number of cycles that are lost due to various kinds of micro-architectural events. By classifying the stall cycles, it is possible to identify performanceproblems and possible solutions. For instance, if there a large number of stalls due to instruction accesses, prefetching the instructions may help in improving performance.Cycle accounting is very helpful in characterizing the overall behavior of applications, but does not help in identifying the specific locations in the program that are responsible for performance related events. In order to address this problem, the Itanium-2 processor provides a set of Event Address Registers(EARs) that can be programmed to capture micro-architectural event information along with the instruction pointer(IP) related to that event. In the absence of Event Address Registers, the IP address captured on the occurrence of an event can be many instructions beyond the instruction that actually triggered the event. When such a “skid” occurs, programmers and performance tools must perform additional analysis to "guess" the possible IP location associated with the performance event. The provision of EARs greatly simplifies the use of the reported IP locations. Some limitations with the use of EARs will be discussed in section 5.The EARs can be used for data, instruction and branch events, and can capture information such as the latency of a cache miss or the level of the TLB in which a miss is resolved, in addition to the instruction pointer. The cache miss events can be captured at programmable frequencies. For example, we can program the PMU to generate an interrupt for every 500 data cache misses having a miss latency greater than 8 clock cycles.In addition to the EARs, the PMU provides an eight entry Branch Trace Buffer(BTB) that can record a trace of the branch instructions executed. It is possible to configure the BTB to capture only a specific subset of branches such as taken branches, not taken branches, correctly predicted and incorrectly predicted branches. The BTB can record up to four branches in the buffer and for each, the source and target addresses are captured.2.2 Kernel support for configuration of the PMUThe Linux kernel includes a subsystem called perfmon to provide access to the IA-64 PMU. The perfmon interface consists of a single system call, perfmonctl() which supports a set of requests to configure, measure, and collect performance monitoring information by providing read and write access to the pmd and pmc registers.Setting up the PMU for monitoring involves programming the pmc registers with information such as the events to be monitored, the privilege level at which monitoring is to occur and interrupts to be generated on counter overflow. For event-based sampling, at least one counter has to be programmed to generate interrupts at the end of each sampling period. The sampling period defines the distance between two samples as an event count.At the end of each sampling period, the perfmon overflow interrupt handler records the contents of a set of specified pmd registers in a sampling buffer, the size of which can be configured by the monitoring task. When the sampling buffer becomes full, the perfmon overflow handler sends a signal to the monitoring task. The monitoring task can now process the contents of the filled sampling buffer through its signal handler. In order to avoid copying of large amounts of data from the kernel area to the application, perfmon automatically maps the buffer read-only into the user address space of the monitoring task.2.3 Pfmon - A user level tool for performance data collectionTo simplify the task of setting up the PMU for monitoring applications, a user level tool called pfmon can be used. The tasks performed by Pfmon for monitoring a single application are as follows :1.Pfmon takes in parameters such as the events to be monitored, the samplingperiods and the file in which the collected data is to be stored, and then uses the perfmon kernel interface to set up the PMU accordingly. This sets up a performance context for the pfmon process which is inherited by any child process, and the performance registers are saved on context switches.2.Pfmon registers a signal handler to process the contents of the sampling bufferwhen it receives a signal from the perfmon subsystem. The contents of the sampling buffer are generally stored in a file on secondary storage for further analysis.3.Pfmon then forks the application to be monitored. The forked process inherits theperformance context from pfmon and is monitored as it runs.The overhead of gathering the performance samples is not very high. The table below shows the overhead of collecting samples when using the SPEC CPU2000 integer benchmarks as the workload and measuring at different sampling rates.Monitored Events SamplingPeriod = 1KSamplingPeriod = 5KSamplingPeriod = 50KSamplingPeriod = 100KSamplingPeriod = 500KICache ~ 4.6% ~ 0.7 % < 0.1%Icache + BTB ~ 10% ~ 2% <0.1%CPUCycles ~1.4%<0.1%As can be seen from the above table, the overhead when sampling 1 in 50K I-Cachemisses is negligible, and this sampling period seemed as successful in identifying the topmisses as a higher sampling frequency.2.4 Perfview - Performance data processingThe data collected by pfmon consists of all the samples taken during a monitoringsession. In order to understand the performance behavior of the monitored application, itis necessary to extract the relevant information from the collected data, and summarize itto provide a high level view of the data.Perfview has been designed to sort out and group the samples according to IP addresses collected from the Event Address registers, in order to help in identification ofthe most frequently occurring IP addresses. This enables identification of hot spots in theprogram with respect to performance bottlenecks. For instance, if the monitoring wasbased on Instruction cache miss sampling, the IP addresses occurring most often in theEvent Address registers correspond to instructions that missed most frequently in thecache. Identification of hot spots in an application is an important step when trying totune for performance since 80 percent of the execution time falls in 20 percent of thecode.Perfview also associates the collected samples with corresponding entries from the Branch Trace Buffer, enabling the construction of an execution trace leading to theevent of interest.Finally, Perfview maps the IP addresses back to the source code using the debugging symbol table information present in the executable. This helps in identifying the code and data structures that are the causes for various bottlenecks, and possible reasons for the bottleneck.Fig 1. Architectural framework for PerfView3Using PerfViewPerfview provides a graphical user interface for setting up a performance monitoring session and correlating data from the Performance Monitoring Unit with program source code. A performance monitoring session consists of programming the pmc and pmd registers of the PMU to set up a performance context for the process to be monitored.Fig 2. Dialog box for setting up a monitoring sessionIn order to help the user in setting up a monitoring session, Perfview provides a dialog box as shown in figure 2, for setting up values for each of the following parameters :Sampling Event : This specifies the event on which the counting monitors of the PMU will be activated. i.e. Whenever the specified sampling event is detected by the processor, the performance counter is incremented by 1. For instance, if the sampling event is set to be all Taken Branches, the PMU is configured such that a performance counter is incremented by one for each branch that is taken.Sampling Period : The sampling period specifies the number of occurrences of an event that needs to occur before a sample is recorded in the sampling buffer. For instance, if a sampling period of 50000 was selected with Taken Branches as the sampling event, then a single sample of the performance registers from the PMUwould be recorded in the sampling buffer for every 50000 taken branches that occur.Other events to be captured : This is used to specify the event information that is to be captured by the PMU and stored in the sampling buffer. By default, the PMU only stores information about the event specified as the sampling event. To make the PMU store information about other events, the user selects the required events from the list of options provided by Perfview. Each sample recorded in the sampling buffer consists of the contents of the performance data registers that contain information relevant to the events selected by the user. For instance, if the user has opted to collect information on Data Cache misses and Instruction cache misses, each sampling buffer entry on an Itanium machine will contain the contents of performance registers pmd2(memory address responsible for the data cache miss), pmd3(latency of the cache miss), pmd17(IP address of instruction causing the data cache miss), pmd0(IP address of instruction causing the Instruction cache miss) and pmd1(latency of Icache miss).Latency of misses to be captured : In the case of capturing cache miss events, it is possible to filter out events which are responsible for latencies that are lower thana selected value. This enables a finer control over the type of cache miss eventsthat are captured. By specifying that all cache misses causing latencies greater than 4 cycles are to be captured, the PMU can be programmed to capture all L1 cache misses.Based on the above description, if the user wishes to capture branch traces for every 5000 instruction cache misses with a latency greater than 4, the following options have to be selected :Sampling event : Instruction Cache MissSampling Period : 5000Other events to be captured : Taken BranchesMiss Latency : >=4Once the configuration for the PMU has been performed as specified above, PerfView is ready to monitor an application. The application which is to be analyzed has to be compiled with debug information, so as to include the debugging symbol table with the object file. The addition of debugging information to the executables does not affect compiler optimizations and there is no notable difference in the execution time of the executable. In the case of the gcc compiler, the –g flag is specified to include debugging information in the object file. PerfView makes use of the debugging symbol table to map IP addresses obtained from the PMU to the source code statements. This makes it necessary that the source files be present in the same location that they were at the time of compilation.To start running an application with performance monitoring turned on, the user needs to specify the command used to start the application along with any necessary command line arguments in the text box present at the top right corner of the PerfView application window. This is similar to typing in the command to start the application at the shell prompt. It is necessary that any filenames specified in the command should have a pathname relative to the directory from which the PerfView tool is being run, or the pathname should be absolute. Once the command has been typed, the run button is clicked to start running an instance of the application.When the run button is clicked, an instance of the pfmon process is started which in turn initiates the application to be analyzed with a performance context attached to it. As the application runs, information about the required events is captured by the PMU. At the end of each sampling period, the contents of the performance registers are copied into the sampling buffer. When the sampling buffer overflows, the signal handler registered by pfmon is used to copy the samples in the buffer to a file. This process continues till the monitored application terminates.Once all the performance samples about the behavior of the application are stored in a file, PerfView goes about analyzing the data, and finally presents a high level view of the collected performance data by highlighting regions of performance bottlenecks in the source code.The PerfView interface consists of a single application window with multiple frames. A screenshot of the interface displaying some data from the gcc benchmark program is shown in figure 3.The top right frame of the application window displays a single file at a time. The source lines that were responsible for a significant fraction of the monitored events are highlighted. The source lines are augmented with line numbers, and a count of the number of events that occurred at each highlighted line is displayed in the top left frame.Fig 3. User Interface provided by PerfViewThe top left frame consists of links to source files for which there is performance information. Clicking on a file name in the left frame causes the file to be displayed in the source file frame on the right. In addition to filenames, there is also a listing of line numbers and the event counts associated with each of these line numbers. Clicking on a line number displays the corresponding line of the file to which the line number belongs, in the source file frame.The bottom frame displays any errors that may have occurred during the running of the application, and any output that is generated by pfmon.A single source line statement may be performing a large number of machine level operations, and this may make it difficult for a performance analyst to exactlyidentify the reason for a performance bottleneck at a particular line. For this purpose, it was felt that the performance information should also be provided at a finer granularity, namely the assembly code level. Clicking on any highlighted line with the left mouse button will open up a new window displaying the assembly code corresponding to the clicked source line and a few immediately surrounding lines. The exact assembly code statement whose IP address was initially captured as the source of the event by the performance registers is highlighted as shown in figure 4. If multiple assembly statements contributed to the high amount of event activity at the selected source line, then each assembly statement is augmented with a number indicating the percentage contribution of that statement to the total events occurring at the source line. This information at the assembly level should give the user a clearer picture about the nature and cause of the bottleneck in a piece of code.Fig 4. Assembly Code WindowIf the monitoring session included capturing the contents of the Branch Trace Buffer, then there is a provision to display a set of execution traces sorted on the number of occurrences of the trace. The execution traces are associated with captured events sothat it enables the user to view the top execution traces leading to an event of interest. This would help in identifying the reason for the occurrence of the event.The execution traces can be viewed by clicking on any the highlighted source lines with the right mouse button. When the line is clicked, the bottom frame of the application window displays a list of hyperlinks linked to the top execution traces leading to the clicked line. Clicking on each hyperlink opens up a new execution trace window as shown in figure 5, which consists of three blocks of code, that were executed before the selected source line was executed. The three blocks of code can be displayed either in source code or assembly code. The assembly code view is provided since compiler optimizations rearrange statements such that the execution sequence does not match the sequence of source code statements.Fig 5. Execution Trace Window4Case Study4.1 Cycle AccountingIn general, the first step in the performance tuning of an application is to identify the nature of performance bottlenecks affecting the application. This could be accomplished by looking at basic cycle accounting to understand the efficiency with which the processor is executing the instructions. Cycle accounting involves accounting for each cycle as either an inherent execution cycle or a stall cycle and this helps in giving a high level view of how the processor is spending its time. Studying the stall cycles is useful for identifying performance problems and possible solutions.By performing cycle accounting, it is possible to narrow down the performance bottlenecks that need to be studied. If a particular bottleneck is responsible for a majority of the stall cycles, then it is most beneficial to study the program with respect to that particular bottleneck. For instance, by performing cycle accounting it can be found that the gcc SPEC 2000 benchmark program has an L1 I-Cache hit ratio of just about 60%. Consequently, maximum benefit can be gained by studying the I-Cache behavior while running the gcc program. Similarly, the mcf benchmark program can be seen to have an L1 D-cache hit ratio of less than 10%.4.2 Using PerfViewThe use of PerfView for the analysis of the GCC SPEC 2000 benchmark program is now discussed.The gcc program is first compiled with debug information. Next, the PMU is configured for a monitoring session by using the ‘Setting’ menubar option at the top of the PerfView application window. The sampling event is set to Instruction cache misses because it has been determined through cycle accounting that gcc suffers from a large number of I-cache misses. In order to help in identifying the reason for the I-cache misses, the taken branches are also captured. This helps in constructing an execution trace leading to the I-cache miss. The sampling period is set to 5000 and the miss latency is set to >=4 to capture all L1 I-cache misses.The gcc program is now started by typing in the command in the text box in the upper right corner and then clicking on the Run button. At this time, PerfView forks a pfmon process which in turn forks the gcc process with a performance context attached to it. The PMU now captures information about the I-cache misses and taken branches as the gcc process continues to run. When the gcc process terminates, all the performance samples are placed in a file. PerfView now processes this performance data and presents it by highlighting source lines responsible for high event activity.From figure 6, it can be seen that the source line number 7290 in the file cse.c was attributed with the occurrence of 926 I-cache miss events. The source statement on this line corresponds to a For statement in the C language. In this case, the I-cache miss events could have occurred either at the beginning of the For loop(test for i<10) or at the end of the loop(i++). In order to identify the exact location of the occurrence of the event in the source statement, it is necessary to look at the assembly code statements corresponding to the source line statement. This is done by clicking on the highlighted line with the left mouse button. This opens up a new window displaying the assembly code corresponding to the clicked line(7290) and immediately surrounding lines, as shown in figure 7.Fig 6. Source Line highlighting windowFig 7. Assembly code windowIn the assembly code window, each assembly statement that contributes to the event count of the source statement is highlighted and its percentage contribution is augmented at the end of the assembly statement. In figure 7, it can be seen that the assembly statement at IP address 0x4000000000183940 was responsible for all the event activity occurring at the source line 7290. From this, it can be inferred that the I-cache miss events actually occurred at the end of the For loop.In order to identify the cause for the I-cache miss events, it is useful to look at the execution traces leading to the events. To do this, right click on the highlighted line. This immediately opens up a set of links in the bottom frame of the window. These are links to execution traces sorted according to the number of occurrences. When any of the linksare clicked, an execution trace window is displayed as shown in figure 8.Fig 8. Execution Trace WindowFigure 8 shows the top execution trace leading to the I-cache misses at line 7290. To get a more detailed view of the execution trace, there is a link provided to view the traces in assembly code as shown in figure 9. From the execution traces, it can be inferred that the I-cache miss event occurs most frequently when the value of the variable n_sets is 1, and the for loop statement at line 7283 exits after just one iteration without gettinginto the body of the If statement at line 7284.Fig 9. Execution Trace in Assembly WindowOnce the regions of performance bottlenecks and the possible reasons are analyzed as just described, it is possible to apply optimization techniques such as instruction prefetching, and evaluate their effect on performance. In the same way, the other highlighted lines are evaluated.The effect of D-cache miss events on the program can be analyzed by programming the PMU to sample on those events, and then carrying out the detailed analysis as explained above.5Implementation issuesThere are some limitations in providing a high level view of the performance datacollected by the PMU. Some of the information that is present at the IP address level maynot be apparent to the user when given a source code level view. Listed below are a few of the limitations and design decisions taken during the implementation of PerfView :The Instruction Event Address registers which capture the IP address responsible for cache misses triggered by instruction fetches, record the instruction cache line that triggered the miss, and not the address of the exact bundle responsible for the miss. On Itanium-2, an L1 cache line is 64 bytes long(4 bundles), and this means that in cases when execution jumps directly to the middle of a cache line, the miss address does not correspond to the bundle that was about to be executed. This could cause the wrong source statement to be attributed for the miss.Due to compiler optimizations, the attributed statement may actually be more than a few source statements away from the actual statement responsible for the miss. For this reason, we provide for the identification of instructions attributed to performance events at the assembly code level too. At the assembly level, the attributed bundle can be at most three bundles away from the actual bundle responsible for the event, and in most cases, the user can clearly identify the assembly instruction most likely to have been responsible for the event.In the case of C macro statements which are expanded by the preprocessor, all events associated with any of the expanded statements will be attributed to the single macro source statement. This makes it impossible to identify the actual cause for the performance bottleneck. As above, in this case too, the information provided at the assembly code level can be useful to identify the individual instruction responsible for the concerned event.Source code statements will not always give a clear idea about the location or cause for a performance bottleneck. For instance, in the case of a For loop statement in the C language, an event could occur either at the beginning of the loop or at the end of the loop. The exact location cannot be determined by just looking at the source code statement as described below :。