View figure.
This section describes how to collect MPI and user event traces or hardware and operating system profiles for a particular serial or POE program's run. It describes how you can use the PCT's graphical user interface or command-line interface to:
For information on the tool's graphical user interface, refer to Using the Performance Collection Tool's Graphical User Interface. For information on the tool's command-line interface, refer to Using the Performance Collection Tool's Command-Line Interface.
This section describes how you can use the PCT's graphical user interface to collect either MPI and user event traces, or hardware and operating system profiles. This section begins with a brief overview of the tasks you can perform using the PCT's graphical user interface, and then describes each of these tasks in more detail. You can also operate the PCT using its command-line interface. For information on the tool's command-line interface, refer to Using the Performance Collection Tool's Command-Line Interface.
Here's an overview of the steps you'll follow when using the PCT's graphical user interface to collect either MPI and user event traces, or hardware and operating system profiles. More detailed instructions on each of the tasks summarized are provided later in this chapter. To use the PCT, you:
| If you are collecting: | Then: |
|---|---|
| MPI and user event traces. | Use the Probe Selection Panel of the PCT's Main Window to specify which MPI events you want to collect data for. For example, you can select "All MPI events", "Collective communication", "Point-to-point communication", and so on. In addition to specifying MPI trace data to be collected, you can also add user markers to processes to mark events or states of interest. Marking these states or events of interest gives you a frame of reference when analyzing the trace record in a graphical visualization tool like Jumpshot. You can also use user markers to mark locations where tracing should be stopped or started. Since you can add MPI probes only at a program, file, or function level (meaning that the entire program, file, or function will be traced), this gives you more control over which part of your program is traced. |
| hardware and operating system profiles. | Use the Probe Selection Panel of the PCT's Main Window to specify the hardware and operating system information you want to collect for later analysis within the PVT. |
In addition to the tasks summarized above, you can also:
You can start the PCT in either graphical-user-interface mode or command-line mode. For instructions on starting the PCT in command-line mode, refer to Using the Performance Collection Tool's Command-Line Interface. To start the PCT in graphical-user-interface mode:
$ pct
Doing this starts the PCT in graphical-user-interface mode and opens its first window -- the Welcome Dialog.
| If: | Then: |
|---|---|
| You want to examine an application that is not already running. | Select the Load a new application option button and click the
OK command button.
Doing this closes the Welcome Dialog, and opens the Load Application Dialog. The Load Application Dialog will enable you to specify the serial or POE program you wish to run. |
| You want to examine an application that is already running. | Select the Connect to a running application option button and
click the OK command button.
Doing this closes the Welcome Dialog, and opens the Connect Application Dialog. The Connect Application Dialog will enable you to specify the serial or POE program to which you want to connect. |
| You do not want to make the decision between whether to load a new, or connect to an existing, application at this time. | Click on the Cancel command button.
Doing this closes the Welcome Dialog and opens the PCT's Main Window. Since you have neither loaded a new application, nor connected to an existing application, the Main Window will not provide any application information at this time. |
The PCT's graphical user interface has been designed to be intuitive and easy to use. If you do have any trouble using it to accomplish the tasks outlined in Performance Collection Tool (Graphical User Interface) Overview, refer to the PCT's online help system. To access the tool's online help, select Help > Contents off the main window's menu bar, or else press the Help button that appears on many of the PCT's dialogs. Doing this opens the PCT help window.
If you open the help from one of the PCT's dialogs, a help topic describing that dialog is displayed. If you open the help from the main window, a task overview topic is displayed.
The PCT help contains topics for each of the major tasks you can perform with the PCT. The left hand pane of the window enables you to navigate the help system to display the needed help topic in the right hand pane. There are three ways to navigate the help system -- using the contents tab, using the index tab, or using the search tab:
This section describes how you can use the PCT in command-line mode to collect either MPI and user event traces or hardware and operating system profiles. The purpose of this section is to illustrate how the various subcommands of the pct command can be used to instrument serial or POE programs. Note, however, that this section does not necessarily describe all the options of all the pct subcommands. For complete reference information on any of the subcommands described in this section, refer to the pct command's man page in Appendix A, Parallel environment tools commands.
This section begins with a brief overview of the tasks you can perform using the PCT's command-line interface, and then describes each of these tasks in more detail. You can also operate the PCT using its graphical user interface. For information on how to do this, refer to Using the Performance Collection Tool's Graphical User Interface.
To use the PCT's command-line interface to collect either MPI and user event traces or hardware and operating system profiles:
When you load or connect to a serial or POE application, two task groups are created. A task group is simply a named set of tasks -- in this case, the task groups are named "all" and "connected". Task groups are intended for when you are working with POE applications as opposed to serial applications. The all task group represents all the tasks in the POE application, while the connected task group represents the POE application's connected tasks only. You can also create your own named task groups. Task groups enable you to more easily manipulate the tasks of a POE application, since many of the PCT's subcommands are designed to operate upon one or more tasks. By default, the tasks operated upon are those in a "current task group" that you specify. By default, the current task group is the automatically-created task group connected. If you are instrumenting a serial application, you naturally do not need to concern yourself with task groups. You should be aware, however, that the all and connected groups are still created by the PCT. For more information on task groups, refer to Grouping Tasks of a POE Application.
To specify which type of data you'll be collecting, use the select subcommand. For more information, refer to Selecting the Type of Probe Data To Be Collected.
| If: | Then: |
|---|---|
| you are collecting MPI and user event traces. |
|
| you are collecting hardware and operating system profile information. |
|
In addition to the tasks summarized above, you can also:
To start the PCT in command-line mode, enter, at the AIX command prompt, the pct command with its -c option:
pct -c
The PCT displays the pct> command prompt. You can now enter PCT subcommands at this prompt.
When starting the PCT in command-line mode, you can optionally specify the -s option to instruct the PCT to read subcommands from a particular script file of PCT subcommands. For example, to have the PCT read the subcommands in the script file myscript.cmd:
pct -c -s myscript.cmd
For more information on PCT script files, refer to Creating and Running PCT Script Files.
The first thing you'll want to do after starting the PCT is either connect to a running application, or load and connect to a new application. If the application you wish to examine is already running, you can connect to it; refer to Connecting to a Running Application. If the application you wish to examine is not already running, you can load it; refer to Loading and Starting a New Application. If you are going to connect or load a POE application, you need to understand the concept of task groups; refer to Grouping Tasks of a POE Application.
To get a listing of all of the PCT's subcommands, enter the help subcommand at the pct> prompt.
pct> help
To get the syntax of a particular subcommand, enter the help subcommand followed by the name of the subcommand whose syntax you want displayed. For example, to get the syntax of the load subcommand.
pct> help load
In the Parallel Operating Environment, the multiple cooperating processes of your program are referred to as "tasks". Many of the PCT subcommands are designed to operate on one or more tasks of a POE application. By default, the tasks operated upon are those in a "current task group" that you can specify. A task group is simply a named set of tasks. Two such task groups -- all and connected -- are created automatically when you either connect to a running application (using the connect subcommand), or load a new application (using the load subcommand). The all task group represents all the tasks in the POE application. The connected task group is the current task group by default -- it represents the POE application's connected tasks only. You can also create your own task groups.
By default, the current task group will be connected; the subcommands you issue will act upon all connected tasks in the POE application. You can change the current task group to be the automatically created group all, or a task group that you have created. You can also, for all of the subcommands that act upon task groups, specify a set of tasks or a task group when issuing the subcommand. If you do this, the subcommand will operate on the tasks specified rather than the current task group. For example, consider the suspend subcommand for suspending execution of one or more tasks. If you issue this subcommand without options as in:
pct> suspend
The tasks in the current task group are suspended. However, if you specify a task list using the task clause, you suspend execution for the tasks specified -- in this next example tasks 0 through 5:
pct> suspend task 0:5
You can also specify a named task group (other than the current task group) using the group clause:
pct> suspend group workers
To understand why you might want to specify a task group, consider the following example. Say that the application you're examining follows the master/workers model in which one task (the "master") coordinates the activities of all the other tasks -- the "workers". You could create two task groups -- one containing just the master task, and the other containing all the other tasks. To do this, you would use the group subcommand with its add clause. To create a task group master containing just task 0:
pct> group add master 0
To create a task group workers containing the tasks 1 through 10:
pct> group add workers 1:10
Once these groups are created, you can make either one the current task group. To do this, you would use the group subcommand with its default clause. For example, the following subcommand sets the current task group to be the task group master:
pct> group default master
While master is the current task group, any subcommands that operate upon tasks will operate only upon task 0 -- the only task in the group master. To make the group workers the current task group:
pct> group default workers
While you cannot modify or delete the two groups that the PCT automatically creates (all and connected), you can modify and delete the groups that you have created. To add tasks 11 though 20 to the task group workers:
pct> group add workers 11:20
To delete task 11 from the task group workers:
pct> group delete workers 11
To delete the entire task group workers:
pct> group delete workers
Notes:
View figure.
View figure.
View figure.pct> show groups
Default Group Name
------- ----------
all
@ connected
pct>
The @ symbol indicates which group is the current task group.
To list the tasks in the task group all:
pct> show group all Tid Program Name Host Cpu Type State --- ------------------------ ---------------- -------- ------ 0 /home/strofino/prod_cons pe04.pok.ibm.com Unknown Loaded 1 /home/strofino/prod_cons pe04.pok.ibm.com Unknown Loaded 2 /home/strofino/prod_cons pe04.pok.ibm.com Unknown Loaded 3 /home/strofino/prod_cons pe04.pok.ibm.com Unknown Loaded pct>
If the serial or POE application you wish to examine is not already running, you can load it onto one or more nodes. When you load an application using the load subcommand, it is loaded in a stopped state with execution suspended at the first executable instruction. You can then start its execution using the start subcommand.
To load a serial application, you simply supply the load subcommand with the absolute path to the executable. The exec clause indicates the absolute path to the executable. If the application takes arguments, you can specify them using the args clause. For example:
pct> load exec /u/example/bin/foo args "a b c"
If loading a POE application, you specify the poe clause, and can also supply any POE arguments using the poeargs clause. For information on the POE command-line flags available to you, refer to the manual IBM Parallel Environment for AIX: Operation and Use, Volume 1, Using the Parallel Operating Environment.
The procedure for loading a POE application differs depending on whether the application follows the Single Program Multiple Data (SPMD) or Multiple Program Multiple Data (MPMD) model. If your program follows the SPMD model, you specify the absolute path to the executable using the exec clause:
pct> load poe exec /u/example/bin/parallel_foo poeargs "-procs 4 -hfile /tmp/host.list"
If your program follows the MPMD model, you supply the absolute path to a POE commands file (which lists the individual programs to load) using the mpmdcmd clause:
pct> load poe mpmdcmd /u/example/bin/foo.cmds poeargs "-procs 3 -hfile /tmp/host.list"
For information on creating a POE commands file for loading multiple programs, refer to the manual IBM Parallel Environment for AIX: Operation and Use, Volume 1, Using the Parallel Operating Environment.
The load subcommand also enables you to specify that standard input, standard output, or standard error should be redirected. To read standard input from a file, use the stdin clause:
pct> load exec /u/example/bin/foo args "a b c" stdin input_file
To redirect standard output to a file, use the stdout clause:
pct> load exec /u/example/bin/foo args "a b c" stdout output_file
To redirect standard error to a file, use the stderr clause:
pct> load exec /u/example/bin/foo args "a b c" stderr error_file
When you load an application, two task groups -- all and connected -- are automatically created, and connected is made the current task group. Task groups are important to know about only if you are working with a POE application and are described in Grouping Tasks of a POE Application. Also note that the application is loaded in a stopped state with execution suspended at the first executable instruction. To start execution of the application, use the start subcommand:
pct> start
If the serial or POE application you wish to examine is already running, you can connect to it using the connect subcommand. To list the processes to which you can connect, use the show subcommand with its ps clause:
pct> show ps Pid Command ----- --------------------------------------------------------------- 10652 /home/strofino/dpcltest/WORK/prod_cons 13256 /etc/dpcld /tmp/dpclsd 13316 /home/strofino/dpcltest/WORK/prod_cons 14302 /usr/lpp/ppe.dpcl/dpcl_beta/bin/poe 18108 /home/strofino/dpcltest/WORK/prod_cons 20614 /u/alfeng/public/perf/seqsleep 21996 /u/alfeng/bin/sesmgr 22644 /home/strofino/dpcltest/WORK/prod_cons 22802 java com/ibm/ppe/perf/main/Startup -l /u/alfeng/bin/sesmgr -cmd 23236 -ksh 24894 /etc/dpcld /tmp/dpclsd 27632 -ksh pct>
If you are connecting to a serial application, you simply supply the process ID of the process you wish to connect to using the pid clause of the connect subcommand.
pct> connect pid 12345
If you are connecting to a POE application, you connect to the processes in two steps. First, you issue the connect subcommand to connect to the controlling, home node, POE process. Once connected to the controlling POE process, you can then reissue the connect subcommand to connect to any of its processes. For example, to connect to the application whose AIX process ID is 12345:
pct> connect poe pid 12345
When you connect to the POE home node process, the PCT creates two task groups -- all and connected. The all task group refers to all of the tasks in the application, while the connected task group refers only to connected tasks. The connected task group will initially be empty since no tasks are connected. You can list the existing task groups by issuing the show subcommand with its groups clause:
pct> show groups
Default Group Name
------- ----------
all
@ connected
pct>
To connect to all tasks in the POE application:
pct> connect group all
To connect to select tasks in the POE application, use the task clause:
pct> connect task 2,3
The PCT enables you to suspend and resume execution of connected processes by issuing the suspend and resume subcommands. You might, for example, wish to suspend execution of your target application prior to instrumenting it as described in Collecting MPI Trace and Custom User Marker Information. Once your performance collection probes have been added to the application, you could resume the application's execution. By default, the suspend and resume subcommands act upon the current task group. Unless you have specified another task group to be the current task group, the current task group will be the task group connected. The task group connected is created automatically by the PCT when you either connect to or load an application (as described in Connecting to a Running Application and Loading and Starting a New Application). The task group connected consists of all connected tasks in a POE application. If you are instrumenting a serial application, you do not need to concern yourself with task groups. If you are instrumenting a POE application, however, it is useful to understand the concept of task groups as described in Grouping Tasks of a POE Application.
To suspend execution of the tasks in the current task group:
pct> suspend
To suspend execution of tasks in a specific task group (in this case, the task group connected), use the group clause on the suspend subcommand:
pct> suspend group connected
To suspend a specific set of tasks in a POE application, use the task clause on the suspend subcommand. To determine how many tasks are available, you can use the show group subcommand to list the tasks in the task group all:
pct> show group all Tid Program Name Host Cpu Type State --- ------------------------ ---------------- -------- ------ 0 /home/strofino/prod_cons pe04.pok.ibm.com Unknown Loaded 1 /home/strofino/prod_cons pe04.pok.ibm.com Unknown Loaded 2 /home/strofino/prod_cons pe04.pok.ibm.com Unknown Loaded 3 /home/strofino/prod_cons pe04.pok.ibm.com Unknown Loaded pct> suspend task 1,3
The resume subcommand works in the same way. By default, it operates on the current task group:
pct> resume
But you can override this by specifying a task group:
pct> resume group connected
or supplying a task list:
pct> resume task 1,5
If you have loaded an application (as described in Loading and Starting a New Application), you can use the stdin subcommand to send standard input text to your application. However, if you have instead merely connected to an application (as described in Connecting to a Running Application), you cannot send standard input text to the application using the stdin subcommand.
If you are instrumenting a serial application, the standard input text will be sent to that application process. If you are instrumenting a POE application, the standard input text will be sent to the controlling, "home node", POE process. As described in Loading and Starting a New Application, you can, when loading an application using the load subcommand, specify that standard input should be read from a file. If you are reading standard input from a file, you cannot use the stdin subcommand.
To send a standard input string to the application, specify the string on the stdin subcommand. The string must be enclosed in double quotes:
> stdin "Now is the time for all good men"
If desired, you can use embedded formatting characters (such as \n) in your standard input string:
> stdin "Now is the time \nfor all good men"
To send a newline character to the input stream reading this input data, issue the stdin command without any text string:
stdin
To send an end-of-file character to the input stream reading this input data, use the eof clause on the stdin subcommand:
> stdin eof
Using the list subcommand, you can display the contents of source files. Unless you are certain of the file name of the source file you want to examine, you may want to list the available source files using the file subcommand. The file subcommand lists, for one or more connected tasks, the associated source file names that match a regular expression you supply. By default, the file subcommand acts upon the current task group. Unless you have specified another task group to be the current task group (as described in Grouping Tasks of a POE Application), the current task group will be the task group connected. The task group connected is created automatically by the PCT when you either connect to or load an application (as described in Connecting to a Running Application and Loading and Starting a New Application). The task group connected consists of all connected tasks in a POE application. If you are instrumenting a serial application, you do not need to concern yourself with task groups. If you are instrumenting a POE application, however, it is useful to understand the concept of task groups as described in Grouping Tasks of a POE Application.
You supply the file subcommand with an AIX regular expression file-matching pattern (enclosed in double quotation marks) to match the source files you want to list. For example, to list all the available source files in the current task group:
pct> file "*" Tid File Id File Name Path --- ------- --------- ------------- 0 0 bar.c ../../lib/src 0 1 foo1.c ../../lib/src 0 2 foo2.c ../src pct>
Although this subcommand, by default, acts upon the current task group, you can specify that it should instead act upon a different task group, or all the tasks in a task list that you supply. This is done by using the task or group clause on the file subcommand. For more information on the task and group clauses, refer to Grouping Tasks of a POE Application.
After issuing the file subcommand, you'll have both the file name and the file identifier of the source file(s) you want to examine. Now you can use the list subcommand to display the contents of one or more files. Like the file subcommand, the list subcommand will, by default, act upon the current task group. Using either the file or fileid clause of the list subcommand, you indicate the file(s) whose contents you want listed.
When listing the contents of files using the list subcommand, the PCT uses a special source path to locate the source files. This source path is, by default, the directory in which the PCT was started, and can be displayed using the sourcepath clause on the show subcommand as in:
pct> show sourcepath Path ---- ./ pct>
To modify the source path so that the PCT can locate source files that are not located in the directory in which the tool was started, use the set subcommand. As with setting your AIX PATH environment variable, you separate the various directories in your source path using colons. For example:
pct> set sourcepath "/afs/aix/u/jbrady:/afs/aix/u/dlecker"
Using the file clause, you supply the list subcommand with an AIX regular expression file-matching pattern (enclosed in double quotation marks) to match the source file(s) whose contents you want to list. If desired, you can supply additional regular expressions separated by commas (file "f*","b*"). For example, the following subcommand lists the contents of the file bar.c:
pct> list file "bar.c"
While this subcommand lists the contents of the first file found in the application that begins with the letter "f":
pct> list file "f*"
Using the fileid clause, you identify the file whose contents you want to list using the process identifier(s) returned by the file subcommand. For example, the following subcommand lists the contents of the file bar.c (whose file identifier is 0):
pct> list fileid 0
You can also use the line clause of the list subcommand to list only a portion of the file's contents. Use a colon to separate the ends of the line number range. For example, the following subcommand lists lines 1 through 20 of the file bar.c.
pct> list file "bar.c" line 1:20
To list the next few lines in bar.c, simply specify the next clause on the list subcommand.
pct> list next
The PCT is capable of collecting two different types of information. It can collect:
Be aware that, before you can collect either type of information, you must
specify, using the select subcommand, which type you are interested
in:
| If you want to collect: | Then: |
|---|---|
| MPI and user event traces. | Specify the trace clause on the select
subcommand:
select trace |
| Hardware and operating system profiles. | Specify the profile clause on the select
subcommand:
select profile |
Using the PCT, you can collect MPI and user event traces for:
The trace information collected is stored as an AIX trace file on each node running instrumented processes. After you have generated these AIX trace files, you can convert them into the Unified Trace Environment (UTE) format (using the uteconvert utility) for analysis using the utestats utility. You can then also convert the UTE files into the SLOG format (using the slogmerge utility) for analysis within Jumpshot. For more information on the utilities for converting the AIX trace files output by the PCT into formats readable by the utestats utility and Jumpshot, refer to Creating, Converting, and Viewing Information Contained In, UTE Interval Files.
In order to collect MPI trace information, the application to be traced must be linked with the libute_a library. To cause this UTE library to be added to the link step, set the MP_UTE environment variable to yes.
Before you can use any of the MPI trace collection subcommands described in this section, you must first specify that you are collecting MPI trace information rather than hardware profile information. Refer to Selecting the Type of Probe Data To Be Collected for more information. Once you have indicated that you'll be collecting MPI and/or user event traces, you can select the output location for the trace files generated by the PCT. To do this, you simply supply an output directory and "base name" (file prefix) for the trace files. Refer to Setting the Output Location and Other Preferences for the AIX Trace Files Generated for more information. You can collect information about:
The trace information collected by the PCT is stored as a separate AIX
trace file on each node running instrumented processes. You can select
the output location and other preferences for the trace files using the
trace set subcommand.
| To specify: | Use this clause of the trace set subcommand: | For example: |
|---|---|---|
| The output location and a "base name" prefix for the generated files. | path |
pct> trace set path "/home/timf/trace files/mytrace" Specifies /home/timf/tracefiles as the location for the generated files. The basename prefix is mytrace. |
| The AIX trace buffer size in Kilobytes. This value can be at most 1024, which is the default. | bufsize |
pct> trace set bufsize 1000 |
| The type of events (MPI events, process dispatch events, and CPU idle events) that are traced. By default, MPI and process dispatch events are traced. Tracing process dispatch events and CPU idle events can result in larger trace files, but the additional information can provide useful context for the MPI information collected. | event |
pct> trace set event mpi pct> trace set event process pct> trace set event idle |
| The maximum trace file size in Megabytes. This can be any value between 2 and 2048 inclusive. The default is 20. | logsize |
pct> trace set logsize 25 |
By adding MPI trace probes to processes, you can trace such MPI events as collective communication, point-to-point communication, and one-sided communication.
To add MPI trace probes, you'll need to know the specific MPI probe type identifier or name as returned by the trace show subcommand. To list the available MPI probe type identifiers and names, specify the probetypes clause on the trace show subcommand:
pct> trace show probetypes MPI Id MPI Name Description ------ ----------- -------------------------------------- 0 all all MPI events 1 blkcollcomm blocking collective communication 2 pttopt point-to-point communication 3 onesided one-sided communication 4 commgroup communication groups 5 topo topologies 6 collcomm non-blocking collective communications 7 env environmental 8 data data type 9 file file 10 info information 11 comm communicators 12 wait wait calls 13 test test calls pct>
Once you have the probe type information, you can use the trace add subcommand to add one or more probe types to one or more processes. You can add the probes at the file level, in which case the MPI events for the entire file will be traced, or at the function level. If that granularity is not small enough, and you want to trace only a portion of a function, you can use special markers to force tracing on and off at particular points.
By default, the trace add subcommand acts upon the current task group. Unless you have specified another task group to be the current task group (as described in Grouping Tasks of a POE Application), the current task group will be the task group connected. The task group connected is created automatically by the PCT when you either connect to or load an application (as described in Connecting to a Running Application and Loading and Starting a New Application). The task group connected consists of all connected tasks in a POE application. If you are instrumenting a serial application, you do not need to concern yourself with task groups. If you are instrumenting a POE application, however, it is useful to understand the concept of task groups as described in Grouping Tasks of a POE Application.
If you are tracing at the file level, you'll need to specify the files using either the file or fileid clause on the trace add subcommand. To do this, you'll need the file identifier or file name information as returned by the file subcommand. To list all available source files in the current task group:
pct> file "*" Tid File Id File Name Path --- ------- --------- ------------- 0 0 bar.c ../../lib/src 0 1 foo1.c ../../lib/src 0 2 foo2.c ../src pct>
To add a certain type of MPI probe, you supply the trace add subcommand with the MPI probe type and file information. You can specify the MPI probe type by supplying the:
Similarly, you can specify the file information by supplying the:
For example:
pct> trace add mpiid 0 to fileid 0 pct> trace add mpiname all to file "bar.c"
You can also specify multiple MPI probe types or multiple files:
pct> trace add mpiid 1,2 to fileid 0,1 pct> trace add mpiname collcom,pttopt to file "bar.c","f*"
If you would like to trace at a function level rather than tracing an entire file, you need to specify the function(s) using either the function or funcid clause. You'll need the function identifier or function name information as returned by the function subcommand. To list all functions in the file bar.c:
pct> function file "bar.c" "*" Tid File Id Function Id File Name Function Name --- ------- ----------- --------- ------------- 0 1 0 bar.c func0 0 1 1 bar.c func1 pct>
If you wish to instrument a particular function, but do not know which file the function is located in, you can use the find subcommand. For example, to search all files in task 0 for functions that match the regular expression comp*:
pct> find task 0 function "comp*" Tid File Id File Name Function Name --- ------- --------- ------------- 0 23 main.c compute 0 23 main.c compare 0 25 sort.c compare2 pct>
You can then specify the function on the trace add subcommand:
pct> trace add mpiid 0 to file "bar.c" function "func0"
You can also specify multiple functions:
pct> trace add mpiid 0 to file "bar.c" function "*" pct> trace add mpiid 0 to file "bar.c" function "func0","func1"
When you issue the trace add subcommand to install MPI trace probes, the probes are given a unique probe identifier. You can use the probe identifier on the trace remove subcommand to remove the probes. To ascertain the probe identifier, use the trace show subcommand with its probes clause as in:
pct> trace show probes Probe Id Command -------- ---------------------------------------------------------------- 0 trace add mpiid 0 to file "prod_cons.c" function "alarm_handler" 1 trace add mpiid 0 to file "prod_cons.c" function "consume" pct>
To remove the probe set whose probe identifier is 0:
pct> trace remove probe 0
User markers are special types of probes that you can install at specific instrumentation points in your application code. You can:
To install a user marker, you'll need to identify not only the file and function, but also the instrumentation point at which you want the probe installed. To list instrumentation points, issue the point subcommand.
pct> point task 0 file "bar.c" Tid File Id Function Id Point Id Point Type Callee Name --- ------- ----------- -------- ---------- ------------ 0 54 0 0 0 0 54 0 1 2 printf 0 54 0 2 3 printf 0 54 0 3 2 MPI_Abort 0 54 0 4 3 MPI_Abort 0 54 0 5 1 0 54 1 0 0 0 54 1 1 2 printf 0 54 1 2 3 printf 0 54 1 3 2 printf 0 54 1 4 3 printf 0 54 1 5 2 MPI_Recv 0 54 1 6 3 MPI_Recv 0 54 1 7 2 consume_data 0 54 1 8 3 consume_data 0 54 1 9 2 printf 0 54 1 10 3 printf 0 54 1 11 1 pct>
To understand the point type number returned by the point command, issue the show points command.
pct> show points Point Type Point Name ---------- ---------- 0 function entry 1 function exit 2 before callsite 3 after callsite pct>
| To: | Use: | For example: |
|---|---|---|
| mark a state of interest. | the simplemarker clause on the trace add subcommand. |
pct> trace add simplemarker "simple" to file "bar.c" funcid 0 pointid 0 |
| mark a region | the beginmarker and endmarker clauses on the
trace add subcommand. You must mark the beginning and end of
the range with the same "marker name" (a string that will be used to identify
the user state in the trace record). You can only use a particular name
for one begin marker/end marker pair. The state will appear in the
trace record as a region.
You should place all markers after the target application's call to MPI_init (which initializes MPI), and before the call to MPI_Finalize (which terminates MPI processing). For more information in the MPI_init and MPI_Finalize calls, refer to the IBM Parallel Environment for AIX: MPI Programming Guide or the IBM Parallel Environment for AIX: MPI Subroutine Reference. When marking a region, you must ensure that the begin and end state markers are placed so that if either marker is reached during execution, the other marker will also be reached. If you nest region markers, you must also ensure that the regions are properly nested. In other words, the inner region should be fully enclosed by the outer region. If you do not follow these guidelines, and the begin and end state markers are not correctly nested, you will get an error when you run the uteconvert utility. For more information on the uteconvert utility, refer to Creating, Converting, and Viewing Information Contained In, UTE Interval Files. |
pct> trace add beginmarker "green" to file "bar.c" funcid 1 pointid 0 pct> trace add endmarker "green" to file "bar.c" funcid 1 pointid 1 |
| force tracing on or off | the traceon or traceoff clause on the trace add subcommand. |
pct> trace add traceoff to file "bar.c" funcid 0 pointid 0 pct> trace add traceon to file "bar.c" funcid 0 pointid 1 |
When you issue the trace add subcommand to install a custom user marker, the marker is given a unique marker identifier. You can use this marker identifier on the trace remove subcommand to remove the markers. To ascertain the marker identifier, use the trace show subcommand with its markers clause as in:
pct> trace show markers Marker Id Command --------- ------- 0 trace add simplemarker "simple" to file "bar.c" funcid 0 pointid 0 1 trace add beginmarker "green" to file "bar.c" funcid 1 pointid 0 2 trace add endmarker "green" to file "bar.c" funcid 1 pointid 1 pct>
To remove the marker whose identifier is 2:
> trace remove marker 2
Using the PCT, you can collect hardware and operating system profiles for analysis within the PVT.
The profile information collected is stored in netCDF (network Common Data Form) format on each node running instrumented processes. The PVT can read netCDF files and summarize the profile information in reports. For more information on using the PVT to read netCDF files output by the PCT, refer to Using the Profile Visualization Tool.
Before you can use any of the profile collection subcommands described in this section, you must first specify that you are collecting hardware profile information rather than MPI and user event traces. Refer to Selecting the Type of Probe Data To Be Collected for more information. Once you have indicated that you'll be collecting hardware profile information, you can select the output location for the netCDF files generated by the PCT. To do this, you simply supply an output directory and "base name" (file prefix) for the netCDF files. Refer to Setting the Output Location for the netCDF Files Generated for more information.
The hardware profile information is saved as a separate netCDF file on each node running instrumented processes. Using the profile set path subcommand, you can specify the output location and "base name" file prefix for these files. For example:
pct> profile set path "profile/output"
By adding hardware profile probes to processes, you can collect hardware and operating system information such as elapsed wall-clock time, process resource usage, and hardware counters. To add hardware profile probes, you need to know the specific probe type identifier or name as returned by the profile show subcommand. To list available probe type identifiers and names, specify the probetypes clause on the profile show subcommand.
For example:
pct> profile show probetypes Prof Id Prof Name Description ------- --------- ---------------- 0 wclock wall clock 1 rusage resource usage 2 hwcount hardware counter pct>
For hardware counters, you can also display a list of the specific hardware counter information you can collect. The list of available hardware counter groups will differ depending on whether the current or supplied task group:
If the current or supplied task group has tasks running on mixed CPUs, then no hardware counters are available, and so none will be listed.
To list available hardware counter groups, specify the probetype hwcount clauses on the profile show subcommand:
pct> profile show probetype hwcount Prof Type Name Description --------- ------- --------------------------------- 0 FPU FPU, FXU, and LSU operations 1 Branch Branch operations 2 L1_TLB L1 cache and TLB operations 3 L2 Prefetch and L2 cache operations 4 Fpop Floating-point operations 5 xFPU FPU, FXU, LSU, and BPU operations pct>
The hardware counter groups you see listed are, by default, the hardware counter groups we have created. For more information on the counters included in each group, and for information on creating your own counter groups, refer to Appendix E, Understanding and Creating PCT Hardware Counter Groups.
Once you have the probe type and hardware counter information, you can use the profile add subcommand to add one or more probe types to one or more processes. You can add the probes at the file level, in which case profile information for the entire file will be produced, or at the function level.
By default, the profile add subcommand acts upon the current task group. Unless you have specified another task group to be the current task group (as described in Grouping Tasks of a POE Application), the current task group will be the task group connected. The task group connected is created automatically by the PCT when you either connect to or load an application (as described in Connecting to a Running Application and Loading and Starting a New Application). The task group connected consists of all connected tasks in a POE application. If you are instrumenting a serial application, you do not need to concern yourself with task groups. If you are instrumenting a POE application, however, it is useful to understand the concept of task groups as described in Grouping Tasks of a POE Application.
If you are collecting profile information at the file level, you'll need to specify the files using either the file or fileid clause on the profile add subcommand. To do this, you'll need the file identifier or file name information as returned by the file subcommand. To list all available source files in the current task group:
pct> file "*" Tid File Id File Name Path --- ------- --------- ------------- 0 0 bar.c ../../lib/src 0 1 foo1.c ../../lib/src 0 2 foo2.c ../src pct>
To add a certain type of profile probe, you can supply the profile add subcommand with the profile probe type and option information, as well as the file information. You can specify:
For example:
pct> profile add profname wclock to fileid 0 pct> profile add profid 0 to file "bar.c" pct> profile add profname hwcount groupid 2 to fileid 3
You can also specify multiple profile probe types or multiple files:
pct> profile add profname wclock profname hwcount groupid 2 to fileid 3,4
If you would like to collect profile information at the function level (instead of collecting profile information for an entire file), you'll need to specify the function(s) using either the function or funcid clause. You'll need the function identifier or function name information as returned by the function subcommand. To list all the functions in the file bar.c:
pct> function file "bar.c" "*" Tid File Id Function Id File Name Function Name --- ------- ----------- --------- ------------- 0 1 0 bar.c func0 0 1 1 bar.c func1 pct>
You can specify the function on the profile add subcommand using its identifier or name:
pct> profile add profname wclock to file "bar.c" function "func0" pct> profile add profname wclock to file "bar.c" funcid 0
You can also specify multiple functions:
pct> profile add profname wclock to file "bar.c" funcid 0,1 pct> profile add profname wclock to file "bar.c" function "*" pct> profile add profname wclock to file "bar.c" function "func0","func1"
When you issue the profile add subcommand to install profile probes, the probes are given a unique probe identifier. You can use this probe identifier on the profile remove subcommand to remove the probes. To ascertain the probe identifier, use the profile show subcommand with its probes clause as in:
pct> profile show probes Probe Id Command -------- ------------------------------------------------------------------- 0 profile add profid 0 to file "prod_cons.c" function "alarm_handler" 1 profile add profid 0 to file "prod_cons.c" function "consume" pct>
To remove the probe set whose identifier is 0:
pct> profile remove probe 0
The PCT enables you to terminate execution of connected processes by issuing the destroy subcommand. You might, for example, wish to terminate execution of your target application after you have finished examining it. By default, the destroy subcommand acts upon the current task group. Unless you have specified another task group to be the current task group (as described in Grouping Tasks of a POE Application), the current task group will be the task group connected. The task group connected is created automatically by the PCT when you either connect to or load an application (as described in Connecting to a Running Application and Loading and Starting a New Application). The task group connected consists of all connected tasks in a POE application. If you are instrumenting a serial application, you do not need to concern yourself with task groups. If you are instrumenting a POE application, however, it is useful to understand the concept of task groups as described in Grouping Tasks of a POE Application.
To terminate execution of all tasks in the current task group:
pct> destroy
To terminate execution of tasks in a specific task group (in this case, the task group connected), use the group clause on the destroy subcommand.
pct> destroy group connected
To terminate a specific set of tasks in a POE application, use the task clause on the destroy subcommand. To determine how many tasks are available, you can use the show group subcommand to list the tasks in the task group all:
pct> show group all Tid Program Name Host Cpu Type State --- ------------------------ ---------------- -------- ------ 0 /home/strofino/prod_cons pe04.pok.ibm.com Unknown Loaded 1 /home/strofino/prod_cons pe04.pok.ibm.com Unknown Loaded 2 /home/strofino/prod_cons pe04.pok.ibm.com Unknown Loaded 3 /home/strofino/prod_cons pe04.pok.ibm.com Unknown Loaded . . . pct> destroy task 1,3
You can also, optionally, terminate execution of all connected tasks when exiting the PCT. To do this, use the exit command with its destroy clause (as described in Exiting the Performance Collection Tool).
Once you are through examining a particular application, or particular tasks in an application, you can disconnect from the application or application tasks by issuing the disconnect subcommand. Once a process is disconnected, the PCT will no longer be able to control execution of, or instrument, the process unless it reconnects to the process. By default, the disconnect subcommand acts upon the current task group. Unless you have specified another task group to be the current task group (as described in Grouping Tasks of a POE Application), the current task group will be the task group connected. The task group connected is created automatically by the PCT when you either connect to or load an application (as described in Connecting to a Running Application and Loading and Starting a New Application). The task group connected consists of all connected tasks in a POE application. If you are instrumenting a serial application, you do not need to concern yourself with task groups. If you are instrumenting a POE application, however, it is useful to understand the concept of task groups as described in Grouping Tasks of a POE Application.
To disconnect all tasks in the current task group:
pct> disconnect
To disconnect tasks in a specific task group (in this case, the task group connected), use the group clause on the disconnect subcommand.
pct> disconnect group connected
To disconnect a specific set of tasks in a POE application, use the task clause on the disconnect subcommand. To determine how many tasks are available, you can use the show group subcommand to list the tasks in the task group all:
pct> show group all Tid Program Name Host Cpu Type State --- ------------------------ ---------------- -------- ------ 0 /home/strofino/prod_cons pe04.pok.ibm.com Unknown Loaded 1 /home/strofino/prod_cons pe04.pok.ibm.com Unknown Loaded 2 /home/strofino/prod_cons pe04.pok.ibm.com Unknown Loaded 3 /home/strofino/prod_cons pe04.pok.ibm.com Unknown Loaded. . . pct> disconnect task 1,3
To exit the PCT and return to your AIX command prompt, issue the exit subcommand:
pct> exit
If you loaded the target application, it will be terminated when PCT exits. If you merely connected to the target application, you must explicitly instruct the PCT to terminate processes. To terminate execution of all connected processes as you exit the PCT, include the destroy clause on the exit subcommand.
pct> exit destroy
Using the command-line interface of the PCT, you are able to run a series of commands that are stored in a file. This file, called a "PCT script file" is a simple text file that lists a sequence of PCT commands that you want to run. Because PCT script files are reusable, they are ideal for situations where you have a set of commands you want to run during multiple PCT sessions. For example, you might want to create a PCT script file that loads and prepares an application so that you can then perform a variety of tasks on the prepared application.
To create a PCT script file, use any ASCII text editor. In the file, place one PCT command per line. You can add comment lines to the file using the # (pound sign) character. For example, here is a simple PCT script file.
# This example uses the 'chaotic' application from the DPCL samples. # The script loads a four-way chaotic application, inserts probes, # starts the application, and then waits for the application to complete load poe exec /home/user/chaotic poeargs "-procs 4" select trace trace set path "/scratch/trace_out" trace add mpiid 0 to file "chaotic.f" start wait
In the sample PCT script file shown above, note the use of the wait subcommand. You need to use the wait subcommand in PCT script files to prevent the PCT from exiting before it has collected probe data. The wait subcommand blocks the PCT's execution so that it can wait for asynchronous events (such as a task terminating) to occur. When one of these asynchronous events occurs, the PCT resumes execution and returns the event that occurred. Be aware that the wait subcommand is intended for use only within PCT script files; it is not intended for interactive command-line sessions.
To run the script file, you can either use the -s option of the pct command when starting the tool (as described in Starting the Performance Collection Tool In Command-Line Mode), or you can use the run subcommand of the pct command. For example, to run the PCT script file myscript.cmd when starting the tool, you would enter the following at the AIX command prompt:
pct -c -s myscript.cmd
Alternatively, you could run the myscript.cmd script file using the run subcommand. For example:
pct> run "myscript.cmd"