TotalView Part 3:
Debugging Parallel Programs

Part 3 Contents

1. Process/Thread Groups
2. Debugging Threaded Codes
   1. Overview
   2. Finding Thread Information
   3. Selecting a Thread
   4. Execution Control for Threaded Programs
   5. Viewing and Modifying Thread Data
3. Debugging OpenMP Codes
   1. Overview
   2. Debugging OpenMP Programs
4. Debugging MPI Codes
   1. Overview
   2. Starting an MPI Debug Session
   3. Selecting an MPI Process
   4. Controlling MPI Process Execution
   5. Viewing and Modifying Multi-process Data
   6. Displaying Message Queue State
5. Debugging Hybrid Codes
   1. Overview
   2. Debugging Hybrid Programs
6. Batch System Debugging
7. Topics Not Covered
8. References and More Information

• Back to Part 1: The Very Basics
• Back to Part 2: Common Functions

Preface

• TotalView supports most HPC parallel programming models:
  • MPI
  • Threads
  • OpenMP
  • PVM
  • SHMEM
  • Fork/exec
  • Hybrid

• This tutorial will only cover the parallel models of threads, MPI, OpenMP, and hybrids of these three models.

• Most examples, commands and images shown are for the IBM AIX platform. However, TotalView's appearance and behavior are fairly consistent across all platforms.

• Please consult the TotalView Documentation for platform specific details.

TotalView P/T Groups:

• Process/Thread (P/T) groups are a TotalView fabrication. Their purpose is to organize processes and threads into associations that a user can operate on.

• Dynamic membership: TotalView automatically creates these P/T groups and places processes and threads in them as they are created.

• Motivation: TotalView commands typically act upon a specific P/T group. It is important for parallel program users to know which P/T group is being acted upon!

• User-defined P/T Groups:
  • In most cases, the default TotalView P/T groups are sufficent - however...
  • TotalView provides a way for users to create their own P/T groups.
  • Non-trivial and not covered here.

Types of P/T Groups:

• Control Group:
  • Contains all processes and threads created by the program across all processors

• Share Group:
  • Contains all of the processes and their threads, that are running the same executable
  • A program may have multiple Share Groups. For example all processes executing a.out would be in one Share Group, and all processes executing b.out would be in another Share Group

• Workers Group:
  • Contains all threads that are executing user code
  • May span multiple process Share Groups
  • Does not contain kernel-level manager threads

• Lockstep Group:
  • Includes all threads in a Share Group that are at the same PC (program counter) address
  • A subset of the Workers Group
  • Only valid for stopped threads - meaningless otherwise

Selecting P/T Groups:

• When you select a P/T group, you are telling TotalView which set of processes and threads to act upon.

• You can select any of the available predefined P/T groups. The default is Control Group.

• Always relative to the Thread-of-Interest (TOI) and the Process-of-Interest (POI), which are the thread and process being viewed in the current Process Window.

• P/T groups can be selected from the Process Window's P/T Selection menu as shown below.

  

• The table below describes what happens when a particular P/T group is selected.

  P/T Selection	What is affected by any execution Command
  Group (Control)	Default. All processes and their threads.
  Group (Share)	All processes and their threads that are in the same share group as the POI (process-of-interest)
  Group (Workers)	All threads that are executing user code
  Group (Lockstep)	All user threads that are stopped at the same PC
  Rank 0	Only the POI and its threads. In the above example, the POI happens to have an MPI rank of 0
  Process (Workers)	User threads in the POI
  Process (Lockstep)	User threads stopped at the same PC in the POI
  Thread 2.1	Only the TOI (thread-of-interest). In the above example, the TOI happens to be 2.1

• P/T groups can also be selected from other locations, such as the Evaluate Dialog Box:

  

Important:

• For most users (especially new users), just accepting the TotalView default Control P/T group does the trick.

• There is quite a bit more to TotalView's P/T groups than what is described above. See the TotalView documentation for details.

Overview

• Most operating systems support programs that have multiple threads of execution. Although implementations differ, they usually possess the following common characteristics:
  • Shared address space - threads can read/write the same variables and execute the same code.
  • Private execution context - every thread has its own set of registers
  • Private execution stack - every thread has address space reserved for its stack
  • Thread - process association - threads exist within and use the resources of a process. They cannot exist outside of a process.

• The diagram below depicts the general threads model. TotalView follows this general model.

  

Supported Platforms:

• TotalView supports debugging threaded applications on all of its supported platforms.

Important Differences:

• Threads are implemented differently by different operating systems. Also, different versions of the same operating system may differ in the way threads are handled.

• Because of this, some thread behavior within TotalView is both architecture and software version dependent:
  • Not all features are implemented, or implemented identically on all platforms
  • Patches and/or upgrades to the OS and other software may be required
  • Hardware requirements vary between platforms (minimum disk, memory, etc.)
  • Restrictions and known problems vary between platforms

• Please consult the TotalView documentation for important details.

Finding Thread Information

• In order to view thread information in the Root Window, you must first click on the "expand toggle" (triangle icon) that appears next to a process.

• A list of threads associated with that process will then appear. (below), and show one line of information for each thread.

  

• Note that the list of threads may contain both user threads and system threads. User threads are created and managed by your program, whereas system threads are not. The latter should be ignored (don't try to debug them).

Process Window:

• Most of what TotalView knows about a thread is able to be found in the Process Window's panes.
  1. Status Bars: Show status information for the selected thread and its associated process.
  2. Stack Trace Pane: Displays the call stack of routines that the selected thread is executing.
  3. Stack Frame Pane: Shows a selected thread's stack variables, registers, etc.
  4. Source Pane: Shows the source code for the selected thread.
  5. Threads Pane: Shows threads associated with the selected process.

  

Selecting a Thread

• After selecting a thread in either the Root Window or the Process Window Threads Pane, you can dive on it by three different methods:
  • Double left clicking
  • Right clicking and then selecting Dive from the pop-up menu
  • Selecting Dive from the Root Window's View Menu.

• That thread's information will then be displayed in the current Process Window.

• To force a new Process Window for a thread, use Dive in New Window from the View Menu or pop-up menu. Multiple Process Windows, one for each thread, can be created this way.

By Thread Navigation Buttons: Use the thread navigation control buttons (at right) located in the bottom right corner of the Process Window. "Cycle-through" the threads until the desired thread's information fills the Process Window.

Differentiating Threads:

• Debugging multi-threaded programs can be confusing - especially if you've opened multiple Process Windows for the different threads. TotalView provides two easy ways for you to differentiate threads from each other:

• Every thread has a unique "Thread ID" number assigned by TotalView. The TID appears in several locations, such as the Root Window, Process Window Threads Pane and Process Window Status Bar.

• Different threads are given different window and pane "trim". The same trim will be consistent from window to window for a given thread.

• The examples below demonstrate how threads are differentiated from each other as just described.

  

  

Execution Control for Threaded Programs

• Depending upon the type of parallel application, TotalView can provide up to three different levels of control for thread execution commands. The table below describes these.

  Scope	Description
  Group	Typically used for multi-process, multi-threaded codes Execution commands apply to all threads in all processes Process Window >  Group Menu
  Process	Typically used for a multi-threaded process Applies to all threads in a single process Process Window >  Process Menu
  Thread	Applies to a single thread within a single process Process Window >  Thread Menu  Note that the thread specific execution control commands are not available on all platforms. They will appear to be dimmed in the menu if they are not available on the platform you are using.

• Note that command scope is constrained to the selected TotalView P/T group (Control, Share, Workers, Lockstep) as discussed in the Process/Thread Groups section.

Synchronous vs. Asynchronous:

• Synchronous: if one thread in a process runs/stops, all threads must do likewise.

• Asynchronous: threads within a process can run/stop independently of each other.

• Platforms may differ in the way individual threads can be stopped and made to run.

For asynchronous thread control, unexpected program behavior (like hanging) can occur if some threads step or run while others are stopped - particularly in library routines. CTRL-C may be able to be used to cancel the command that caused the hang.

Thread-specific Breakpoints:

• Normally, all threads in a process stop when any one of them encounters a breakpoint.

• Thread-specific breakpoints are implemented through evaluation points and the use of TotalView expressions that include intrinsic variables and built-in statements.

• For example, the following expression will cause the process to stop only when thread 3 encounters it as part of an evaluation point:

  if ($tid == 3) $stop

Viewing and Modifying Thread Data

• Most of the basics of viewing and modifying data as covered in Part I hold true for threads.

• Beyond the basics, TotalView allows you to go a bit further with threads. You can display/modify "Laminated" variables and on some platforms you can display Thread Object data.

• Often times in a parallel program, the same variable will have multiple instances across threads and/or processes. In such cases, it is frequently desirable to view all occurrences simultaneously.

• TotalView provides a way for you to do this by "laminating" the variable. Laminating a variable means to display all occurrences simultaneously in a Variable Window.

• Laminated variables can include scalars, arrays, structures and pointers.

• TotalView also enables you to edit laminated variables - either collectively (same value applies to all instances) or individually.

• To laminate a thread variable, simply dive on it so that it appears in a new Variable Window. Then:

  Variable Window >  View Menu  > Show Across > Thread

• After laminating a variable, you can return to the non-laminated view by:

  Variable Window >  View Menu  > Show Across > None

• An example is shown below. Note that when laminating a variable, not all threads may be at a point in the program yet where the variable has a value. In such cases, the "Has no matching call frame" message will appear.

  

In the Kernel:

• The Process Window below shows what can happen when a thread calls a system kernel routine. The debugger may not have full access to thread state information when it executes within the kernel. There's not much you can do at this point, debugging wise.

  

Overview

• The OpenMP programming model is intrinsically based on threads.

• All OpenMP programs begin with a single master thread (usually the original executable) that executes serially until a PARALLEL region in the program is encountered.

• When a PARALLEL region is encountered, the master thread forks a team of worker threads to execute that region in parallel.

• At the end of the PARALLEL region, the team joins/disbands and serial execution resumes by the master thread (see diagram below).

  

• OpenMP threads behave very similarly to other types of threads within TotalView.

Supported Platforms:

• TotalView provides support for OpenMP on most of its supported platforms.

• However, the actual implementation supported varies. For example, it may be KAI C and Fortran, native Fortran compiler, native C compiler, or a combination.

• Please consult the TotalView documentation for important platform / compiler specific requirements and limitations.

Supported Features:

• Source level debugging of the original OpenMP code

• Ability to place breakpoints throughout the OpenMP code, including lines that are executed in parallel.

• Visibility of worker threads

• Access to PRIVATE and SHARED variables in PARALLEL regions - for both master and worker threads.

• Worker threads' Stack Trace Pane provides a convenient "stack back link token" to the master thread's stack.

• Access to THREADPRIVATE data on some platforms (currently IBM and COMPAQ).

Debugging OpenMP Programs

• Because OpenMP is fundamentally thread-based, debugging OpenMP codes with TotalView is very similar to debugging threaded codes. In fact, Pthreads are often used by vendors to implement OpenMP.

• Most of what was covered previously in the Debugging Threaded Codes section holds true for OpenMP:
  • Finding Thread Information
  • Selecting a Thread
  • Execution Control for Threaded Programs
  • Viewing and Modifying Thread Data

• Because of this, only the important differences between OpenMP and threaded debugging will be covered here.

Setting the Number of Threads:

• Setting the number of threads to use during a debug session is handled exactly as specified by the OpenMP standard. In order of precedence (lowest to highest):
  1. Default: usually equal to the number of cpus on the machine
  2. OMP_NUM_THREADS environment variable at run time
  3. OMP_SET_NUM_THREADS routine within the source code

Code Transformation:

• Probably the most obvious difference between OpenMP codes and other threaded codes is the compiler's creation of outlined routines.

• Outlined routines are created when the compiler replicates the body of a PARALLEL region into a new, compiler created routine. This process is called outlining because it is the inverse of inlining a subroutine into its call site.

• In place of the parallel region, the compiler inserts a call to a run-time library routine. As the master thread creates worker threads, it dispatches them to the outlined routine, and then actually calls the outlined routine itself.

• Outlined Routine Name: This varies by compiler/platform. In most cases, the name of the outlined routine is based on, but not identical to, the original routine's name. For example:

  

• The OpenMP master thread always has a logical thread id of 1, and the worker threads greater than 1. Note that these are TotalView assigned ids. They do not match the actual OpenMP thread number. For example, in OpenMP, the master thread’s id is zero.

Master Thread vs. Worker Threads:

• For the most part, the master thread and worker threads behave similarly and can be debugged similarly. The most important difference is how shared variables and private variables are treated.

• Most compilers allocate the program's shared variables with the master thread's original (spawning) routine. Private variables are allocated on the worker thread outlined routine's stack. An exception to this occurs with KAI-based compilers (such as icc and ifort) which pass the shared variables to the worker threads.

• The most important difference between the master thread and the worker threads is that the master thread is the only thread where TotalView displays shared variables in the Stack Frame. Worker threads only show private variables in their Stack Frames.

• The two Stack Frames below show the difference between the master thread and a worker thread.

  

Stack Parent Token Line:

Note: This feature seems to "come and go" with each release of TotalView. It is not clear why, but if you don't see it when you are debugging, then ignore this information.
• TotalView inserts a special "Stack Parent Token Line" in the Stack Trace Pane of worker threads whenever they are stopped inside an outlined routine.

• The image at right shows the Stack Trace Pane of a worker thread. The worker thread's current frame is the outlined routine (highlighted line). The Stack Parent Token Line is denoted by the black arrow.

• When you select or dive on the Stack Parent Token Line, the Process Window will display the master thread's information instead of the worker thread.

• The primary purpose of this feature is to allow you to easily view the original routine's stack frame for the master thread. This stack frame is the only one which displays the program's shared variables.

Example OpenMP Session:

1. Master thread Stack Trace Pane showing original routine
2. Process/thread status bars differentiating threads
3. Master thread Stack Frame Pane showing shared variables
4. Worker thread Stack Trace Pane showing outlined routine. Stack Parent Token Line is missing in this case.
5. Worker thread Stack Frame Pane showing private variables
6. Root Window showing all threads
7. Threads Pane showing all threads plus selected thread

   

Execution Control:

• Similar to threads as discussed previously.

• Stepping: you can not step into or out of a PARALLEL region. Instead, set a breakpoint within the parallel region and allow the process to run to it. From there you can single step within the parallel region.

Asynchronous execution: single stepping or running one OpenMP thread while others are stopped can lead to unexpected program behavior (like hanging). CTRL-C may be able to be used to cancel the command that caused the hang.

Viewing and Modifying Data:

• With the exception of SHARED and THREADPRIVATE variables/common blocks, (covered later) viewing and displaying data behaves the same as for other threaded codes.

• As with threaded codes, TotalView supports laminated variable displays for OpenMP also.

Manager Threads:

• Some platforms create additional threads for management purposes. Manager threads are given a negative thread id by TotalView.

• Manager threads should be ignored - do not try to debug them.

• Example showing manager threads in addition to OpenMP threads. The Process Window Threads Pane is shown.

  

THREADPRIVATE Data:

• The OpenMP THREADPRIVATE directive allows each thread to maintain its own local copies of globally defined common blocks and variables - which are persistent to a thread through the execution of multiple parallel regions.

• The easiest way to view THREADPRIVATE data is to find the master thread, and then select the original routine in the Stack Trace Pane. The Process Window will then refresh with the master thread's routine and show THREADPRIVATE (and all shared variables) in the Stack Frame Pane. Example below:

  

• After finding a THREADPRIVATE variable, you can dive on it as with any ordinary variable. Laminated variables are supported for THREADPRIVATE variables also.

Overview

• MPI programs behave as multiple processes within TotalView:
  • Each MPI task comprises its own process.
  • Every MPI task can run/stop and be debugged independently from other MPI tasks.
  • MPI tasks can also be debugged collectively with related MPI tasks.

• As discussed in the Process/Thread Groups section , TotalView assigns processes into Share groups. In most cases, if all of your MPI tasks are running the same executable, (SPMD Model) they will all be in the same Share Group. Otherwise, MPI tasks running different executables (MPMD Model) will be in different Share groups.

• Most of the usual TotalView commands/features behave as would be expected with an individual MPI process. However, there are several important considerations and unique features associated with multi-process MPI debugging.

• MPI codes can be combined with threads and OpenMP (covered later) to create multi-threaded, multi-process programs.

Supported Platforms:

• TotalView supports the native vendor MPI implementation and also the MPICH implementation. For platform specifics, see the TotalView User Guide.

Starting an MPI Debug Session

• MPI manager process:
  • Typically, MPI programs run under a "manager" process, such as poe, srun, prun, mpirun, dmpirun, etc.
  • Because of this, you must start TotalView with the manager process, NOT the name of your MPI executable.

• Automatic process acquisition:
  • Most MPI programs run on multiple hosts, however when you start TotalView it is on a single host.
  • TotalView is able to automatically acquire all parallel processes at start-up.
  • TotalView is also able to attach to an already running parallel program and automatically acquire all of its processes.
  • This is accomplished by TotalView starting a tvdsvr process on each machine where it must acquire and manage a parallel task.

• Configuration Details:
  • There are several issues involved in configuring TotalView to run multi-process jobs, most of which should normally be transparent to the user. See the TotalView User Guide for details if problems arise with starting MPI sessions under TotalView.

Example:

1. Start TotalView with the parallel task manager process. Note that the order of arguments and executables is important, and differs between platforms.

   Examples:

   IBM (at LC)	totalview poe -a myprog -procs 4 -rmpool 0
   QUADRICS Intel Linux under SLURM	totalview srun -a -n 16 -p pdebug myprog
   MVAPICH Opteron Linux under SLURM	totalview srun -a -n 16 -p pdebug myprog
   SGI	totalview mpirun -a myprog -np 16
   Sun	totalview mprun -a myprog -np 16
   MPICH	mpirun -np 16 -tv myprog

2. The Root Window and Process Window will appear as usual, however it will be the manager process that will be loaded, not your program. Start the manager process by typing g in the Process Window or by:

   Process Window >  Process Menu  >  Go

3. A dialog window will then appear notifying you that it is a parallel job and asking whether or not you wish to stop the job now. Click on "Yes" (see below)

4. TotalView will then acquire the MPI tasks which are running under the manager process. When this is done, the Process Window will default to displaying the state information and source for MPI task 0. You are now ready to begin debugging your program.

   

Selecting an MPI Process

• After selecting a process in the Root Window, you can dive on it by three different methods:
  • Double left clicking
  • Right clicking and then selecting Dive from the pop-up menu
  • Selecting Dive from the Root Window's View Menu.

• That process's information will then be displayed in the current Process Window.

• To force a new Process Window for a process, use Dive In New Window from the View Menu or right click pop-up menu. Multiple Process Windows, one for each MPI task, can be created this way.

By Process Navigation Buttons:

• Use the process navigation control buttons (at right) located in the bottom right corner of the Process Window.

• "Cycle-through" the processes until the desired task's information fills the Process Window.

Example:

• The example below demonstrates an MPI debug session. Some items of interest:
  1. Process Windows differentiated by pane trim and status bars.
  2. Multiple process windows - one for MPI task 0 and one for MPI task 3
  3. Root Window showing manager task and multiple MPI processes
  4. Tasks are in different states
  5. Navigation buttons enabled for processes

  

Controlling MPI Process Execution

• MPI task execution can be controlled at the individual process level, or collectively as a "group".

• TotalView provides two different levels of control for MPI process execution commands. The table below describes these.

  Scope	Description
  Group	Execution commands apply to all MPI processes Process Window >  Group Menu
  Process	Applies to a single MPI process Process Window >  Process Menu

• Note that command scope is constrained to the selected TotalView P/T group (Control, Share, Workers, Lockstep) as discussed in the Process/Thread Groups section.

Starting and Stopping Processes:

• As seen previously, TotalView will ask you whether or not you wish to stop your parallel job before it starts to execute. Saying "Yes" to this allows you to set breakpoints and do other things before your tasks actually start running.

• Starting your program and controlling its execution is then up to you, using either the Group Menu or the Process Menu from the Process Window.

If you use accelerator keys to control execution, be sure to type the right key! It is a fairly common accident to use a process level command instead of group level command (and vice-versa). For example, typing g instead of G.

Holding and Releasing Processes:

• When a process is held, it is unresponsive to commands that would cause it to run, such as Go, Step, Next...

• Processes are automatically placed in a hold state when they encounter a barrier point. They can also be placed on hold manually by either method below, depending upon whether you want to hold all processes or just one:

  	Process Window >  Group Menu  >  Hold
  	Process Window >  Process Menu  >  Hold

• Held processes will display an H state code in the Root Window (below).

  

• Processes are released automatically whenever all processes have reached the same barrier point. They can also be released manually:

  	Process Window >  Group Menu  >  Release
  	Process Window >  Process Menu  >  Hold (toggle)

• Note that releasing a process does not make it "Go". It only allows it to respond again to run type commands.

Breakpoints and Barrier Points:

• TotalView provides two options that control the behavior of breakpoints and barrier points:
  • Sharing: Should the action point be "planted" in all processes of the group? Planting means that if you set the action point in one MPI task, TotalView will automatically replicate it in all MPI tasks. The default behavior for both breakpoints and barrier points is to automatically plant the action point in all processes.
  • Scoping: Should the action point affect the group, the process or the thread(s)? The default behavior for both breakpoints and barrier points is to stop the process.

• Individual breakpoint and barrier point behavior can be customized via the Action Point Properties Dialog Box. To open this window, first select a source line with a breakpoint or barrier point. Then do either:
  • Dive (right-mouse) click on the source code line and then select Properties from the resulting pop-up menu.

    Process Window >  Action Point Menu >  Properties

• Action Point Properties Dialog Boxes for both breakpoints and barrier points are shown below.

  

• You can also customize the default behavior for all breakpoints and barrier points:
  1. Use either:

     	Root Window  >  File Menu  >  Preferences
     	Process Window >  File Menu  >  Preferences

  2. Select the Action Points Page and then choose your desired options.

• See the previous discussion on Action Points for more information on using Breakpoints and Barrier Points.

Warning About Single Process Commands:

• If you use a process-level single stepping command in a multi-process MPI program, it is possible that TotalView will appear to hang. This happens when you step over a statement that can not complete because the process it depends upon is stopped (as in communications).

• Using CTRL-C may be able to be used to cancel the step command that caused the hang.

Viewing and Modifying Multi-process Data

• Most of the basics of the viewing and modifying data as covered in Part I hold true for multi-process MPI programs.

Laminated Variables:

• Often times in a parallel program, the same variable will have multiple instances across threads and/or processes. In such cases, it is frequently desirable to view all occurrences simultaneously.

• TotalView provides a way for you to this by "laminating" the variable. Laminating a variable means to display all occurrences simultaneously in a Variable Window.

• Laminated variables can include scalars, arrays, structures and pointers.

• TotalView also enables you to edit laminated variables - either collectively (same value applies to all instances) or individually.

• To laminate a multi-process variable, simply dive on it so that it appears in a new Variable Window. Then:

  Variable Window >  View Menu  > Show Across >  Process

• Two examples are shown below - the first is a laminated scalar variable and the second is a laminated array variable.

  

• The laminated variable view is a toggle display. After laminating a variable, you can return to the non-laminated view by:

  Variable Window >  View Menu  > Show Across >  None

Displaying Message Queue State

• TotalView allows you to examine the run-time state of your MPI program's message passing. This can be helpful when debugging deadlocked programs.

• To view the message queue state for a selected MPI process, first stop execution, then:

  Process Window >  Tools Menu  >  Message Queue

• The Message Queue Window will then appear - an example is shown below.

  

Types of Messages Displayed:

• Pending receives - non-blocking and blocking.

• Pending sends - non-blocking and blocking.

• Unexpected messages - messages sent to this process which do not yet have a matching receive operation.

• Normally completed messages are not saved or viewable.

Actions:

• Because the Message Queue Window information is actually derived from the MPI library, the data is view only - no modification is permitted.

• Diving on the "Source" field will refocus the current Process Window with that task's information or else open a new window for the source task.

• Diving on the "Buffer" field will allow you to see the message's contents in a Variable Window. This data can then be treated as normal data - modify values, type, laminate, etc.

Message Queue Graph:

• TotalView also provides a graphical representation of your program's message queue state at a given instant.

• To view your program's message queue state graph, first stop execution. Then select:

  Process Window >  Tools Menu  >  Message Queue Graph

• The Message Queue Graph Window will then appear - an example is shown below.

  

• Some usage notes:
  • Processes are indicated by yellow boxes in the graph, and as blocks in the communicator box on the right side. Task ranks are the numbers that appear in both locations.
  • Select/deselect types of messages to display then click on Update button
  • Red = Unexpected, Blue = Pending Receive, Green = Pending Send
  • Numbers next to arrow points indicate the message tag
  • Diving on a box causes that task's information to appear in a Process Window
  • Diving on an arc/arrow point will open the detailed Message Queue Window for that task
  • Boxes and arcs can be repositioned by dragging them with the mouse, however clicking on the Update button will reset the view back to the original object positions
  • See the built-in Help for additional information

Notes:

• For the IBM POE environment, you will need to set the MP_EUIDEVELOP environment variable to "deb" (debug) in order to see pending blocking receives and sends.

• The information displayed in the Message Queue Window may vary slightly between platforms and MPI implementations.

• There are several important platform and implementation prerequisites and limitations. See the TotalView documentation for details.

Overview

• Hybrid codes are programs that use more than one type of parallelism. This programming model is becoming increasingly popular as systems comprised of clusters of SMPs are now very common.

• Probably the most frequently used type of hybrid programming is MPI with Pthreads or MPI with OpenMP. One scenario (there are certainly others):
  • A large problem is decomposed for execution on a cluster of SMP machines.
  • A single MPI process is started on each SMP machine.
  • Each MPI process divides up its work between multiple threads.
  • Threads execute on the CPUs of a single SMP machine, using shared memory parallelism.
  • When data needs to be exchanged between machines, one of the threads uses MPI to communicate with the MPI tasks on other machines.

Nothing New (Just More of It):

• TotalView includes no new "features" or special functions to handle hybrid codes. There is nothing new to learn.

• Everything that applies to MPI, threads and OpenMP holds true essentially unaltered for hybrid codes.

• The real challenge is managing and understanding the increased complexity that arises from combining two different types of parallelism.

Supported Platforms:

• Basically, whatever is supported / restricted for MPI, threads and OpenMP on any given platform will hold true for hybrid programs on that platform.

• See the TotalView documentation for details.

Debugging Hybrid Programs

• If your hybrid code is a combination of MPI with either OpenMP or Pthreads, then you will most likely start your debug session as you would for MPI. See Starting an MPI Debug Session for examples.

• OpenMP programs will typically follow the usual convention for setting the number of threads as defined by the OpenMP standard: In order of precedence (lowest to highest):
  1. Default: usually equal to the number of cpus on the machine
  2. OMP_NUM_THREADS environment variable at run time
  3. OMP_SET_NUM_THREADS routine within the source code

Tying it All Together:

• Debugging hybrid programs combines everything previously discussed in Debugging Threaded Codes, Debugging MPI Codes and Debugging OpenMP Codes.

• MPI tasks behave individually as processes and collectively as a group

• Threads exist within an MPI process

• Execution control can be specified at the thread, process or group level within the selected P/T group

• Action points can be shared across a group or remain local to a process

• Every thread and process can have its own Process Window here.

• Selection and navigation between threads and processes works as usual

Example:

• An example debug session with a hybrid MPI / Pthreads program is shown below. Some details of interest:
  1. Each MPI task / thread can have its own Process Window - two are shown here
  2. Processes and threads are differentiated by pane trim and status bars
  3. Root Window showing multiple MPI processes, each with multiple threads. Manager process (POE in this case) also appears.
  4. Process barrier point in effect across multiple processes
  5. MPI task identifiers and thread identifiers are the same as usual
  6. Both process and thread navigation buttons are active

  

Why Would You Want To?

• For the most part, debugging applications is best done in an interactive environment. There are several practical reasons for this:
  • You may not have access to jobs running in a batch system
  • It may be impossible to predict when a batch job will start
  • There may be problems with sending X displays back from a batch environment
  • Determining which machine(s) your batch job is running on may be difficult
  • Initiating a debugging session on an already running batch job may be difficult

• However, it is often the case that the number of interactive machines that are available for interactive debugging is small, making it impossible to debug "real size" parallel applications.

• It is common for large parallel problems to encounter bugs that are not seen with small interactive parallel runs. Debugging the application while it is running in the larger batch system may be the only means of identifying the problem.

• Debugging programs (especially parallel ones) while they are executing in a batch system can range from "similar to interactive" to "next to impossible". Fortunately, at LC, it is relatively easy to debug running batch jobs.

A General Approach:

• The primary goal is to have interactive access to your job while it is running in batch. LC makes life easy in this respect by allowing users to login to batch nodes when their job is running there.

• Two possible methods for debugging an MPI parallel batch job are listed below. There are more ways.

  From Batch Command Script

  Job Already Running

  Modify your batch command script so that it sets up the correct X display environment and then launches your executable through TotalView - as you would in an interactive session. Stand by your terminal and wait (and wait?) for your job to begin execution and for the TotalView GUI to appear on your screen. Then debug as usual. Alternately, you could have your job command script launch an xterm. When the xterm appears on your screen, you can start TotalView from within it (as well as do other things).

  Login to where your job is running. For parallel jobs this should be the machine that is running the parallel manager process (poe, srun, prun, etc.). Set up your X display environment. Start TotalView alone. When the New Program dialog box appears, attach to the parallel manager process. Halt your program and then debug as normal.

A Couple LC Specific Notes:

• For non-MPI jobs on LC's IBMs, you will need to put at least one poe command in your batch script if you plan to login to a batch node where your job is running. Something as simple as poe date or poe hostname will do the trick. Otherwise you will be prompted for a password, which will never be recognized.

• For Tri-lab cross-cell authentication users: instead of using ssh for connecting to a batch node, use rsh. The above note for non-MPI jobs also applies.

Example Batch Debug Session:

Use the squeue command to see if your job is running. Note the NODELIST column. The first node in the list is where the manager process is running. This will be the machine you login to.

up041% squeue
  JOBID PARTITION     NAME     USER  ST       TIME  NODES NODELIST(REASON)
  33228    pbatch  RhJJf01    4rtrt   R    1:47:18      1 up089
  34144    pbatch    hchem     augh   R    1:42:11      4 up[099-102]
  34032    pbatch    Mo27e     niel   R    1:41:47     15 up[072-084,103-104]
  34041    pbatch     nMod     riel   R    1:41:35      4 up[033-036]
  34635    pbatch   r.uP16     nuel   R    1:19:55      8 up[009-016]
  43742    pbatch   t1.cmd  joeuser   R       0:20      2 up[097-098]

Login to the machine where the manager process is running.

up041% ssh up097 NOTE: rsh may also be used, especially for cross-cell authentication accounts

No mail.
                              NOTICE TO USERS

This computer is a federal computer system and is the property of the
United States government. It is for authorized use only. Users (authorized
or unauthorized) have no explicit or implicit expectation of privacy.
 
[ remainder of lengthy login banner deleted ]

up097%

Verify that the parallel manager process (poe in this case) is running on this machine.

up097% ps -Af|grep poe
  joeuser 106694  70122   0 18:04:51      -  0:00 poe 
  joeuser 111206 106694   0 18:04:51      -  0:00 poe 
  joeuser  66960  99546   0 18:19:15  pts/0  0:00 grep poe 
up097%

Set up your X windows display variable. Change to the directory where your executable's source is located. Then, start totalview by itself.

up097% setenv DISPLAY 64.75.147.115:0
up097% cd totalview/
up097% totalview &
[1] 123392
up097% AIX TotalView 7.1.0-1
Copyright 1999-2005 by Etnus, LLC. ALL RIGHTS RESERVED.
Copyright 1999 by Etnus, Inc.
Copyright 1996-1998 by Dolphin Interconnect Solutions, Inc.
Copyright 1989-1996 by BBN Inc.

Eventually, you should see the Root Window and a New Program dialog box appear on your local workstation, sent to you from the batch machine where your job is running. Shown below.

In the New Program dialog box, select Attach to an existing process. The dialog box should then present you with a list of running processes. Find and select the manager process. In this case, there are two instances of the poe process. You should select the parent poe process, as shown below.

A Process Window for the manager task will appear. Halt the job. You can now select any MPI task and debug as usual (shown below).

TotalView includes a number of other features and functions not covered in this tutorial. A partial list of these appears below. Please consult the TotalView Documentation for more information.

• Most of the CLI is not covered
• Setting up remote debugging sessions
• Setting environment variables within TotalView
• Most platform specific information
• Debugging PVM / DPVM applications
• Debugging MPICH applications
• Debugging SHMEM applications
• Debugging UPC applications
• Debugging PGI HPF applications
• Shared and dynamic library information
• Process state graphs
• Heap and Memory debugging
• Performance Monitor display
• Architecture specific register address usages
• Most X resources
• Full set of command line options
• Operating system features
• Visualizer Details

The most useful documentation and reference material is from Etnus. You can download this from the "Documentation" section of their web site at www.etnus.com. If you already have TotalView installed, the same documentation comes with the installation and is available from the install directory and TotalView's "Help" menu.

This concludes TotalView Part 3

Please complete the online evaluation form - unless you are doing the exercise, in which case please complete it at the end of the exercise.

Where would you like to go now?

• Go to Exercise 3
• Back to the beginning of Part 3
• Back to Part 1: The Very Basics
• Back to Part 2: Common Functions
• Workshop Agenda