Chapter 4. Obtaining profiling results

Table of Contents

1. Profile specifications
1.1. Examples
1.2. Profile specification parameters
1.3. Locating and managing binary images
1.4. What to do when you don't get any results
2. Image summaries and symbol summaries (opreport)
2.1. Merging separate profiles
2.2. Side-by-side multiple results
2.3. Callgraph output
2.4. Differential profiles with opreport
2.5. Anonymous executable mappings
2.6. XML formatted output
2.7. Options for opreport
3. Outputting annotated source (opannotate)
3.1. Locating source files
3.2. Usage of opannotate
4. OProfile results with JIT samples
5. gprof-compatible output (opgprof)
5.1. Usage of opgprof
6. Analyzing profile data on another system (oparchive)
6.1. Usage of oparchive
7. Converting sample database files (opimport)
7.1. Usage of opimport

After collecting profile data, the raw data must undergo special processing in order for you to perform your analysis. The analysis tools that perform this special processing are opreport, opannotate, and opgprof. Additionally, the oparchive is used to gather together profile data, sampled binary files, etc. for the purpose of off-line analysis. While not really an analysis tool, oparchive is put in that category for convenience since it takes many of the same options as the other analysis tools.

1. Profile specifications

All of the analysis tools take a profile specification as an input argument. This is a set of definitions that describes the specific profile data that should be examined. The simplest profile specification is empty: this will match all the available profile files for the current session.

Specification parameters are of the form name:value[,value]. For example, if I wanted to get a combined symbol summary for /bin/myprog and /bin/myprog2, I could do opreport -l image:/bin/myprog,/bin/myprog2. As a special case, you don't actually need to specify the image: part of the specification. Anything left on the command line after all other opreport options have been processed is assumed to be an image: name. Similarly, if no session: is specified, then session:current is assumed ("current" is a special name of the current (i.e., most recent) profiling session).

In addition to the comma-separated list shown above, some of the specification parameters can take glob-style values. For example, if I want to see image summaries for all binaries profiled in /usr/bin/, I could do opreport image:/usr/bin/\*. Note the necessity to escape the special character from the shell.

For opreport, profile specifications can be used to define two profiles, giving differential output. This is done by enclosing each of the two specifications within curly braces, as shown in the examples below. Any specifications outside of curly braces are shared across both.

1.1. Examples

Image summaries for all profiles with DATA_MEM_REFS samples in the saved session called "stresstest" :

# opreport session:stresstest event:DATA_MEM_REFS

Symbol summary for the application called "test_sym53c8xx,9xx". Note the escaping is necessary as image: takes a comma-separated list.

# opreport -l ./test/test_sym53c8xx\,9xx

Image summaries for all binaries in the test directory, excepting boring-test :

# opreport image:./test/\* image-exclude:./test/boring-test

Differential profile of a binary stored in two archives :

# opreport -l /bin/bash { archive:./orig } { archive:./new }

Differential profile of an archived binary with the current session :

# opreport -l /bin/bash { archive:./orig } { }

1.2. Profile specification parameters

archive: archivepath

A path to an archive made with oparchive. Absence of this tag, unlike others, means "the current system", equivalent to specifying "archive:".

session: sessionlist

A comma-separated list of session names to resolve in. Absence of this tag, unlike others, means "the current session", equivalent to specifying "session:current".

session-exclude: sessionlist

A comma-separated list of sessions to exclude.

image: imagelist

A comma-separated list of image names to resolve. Each entry may be relative path, glob-style name, or full path, e.g.

opreport 'image:/usr/bin/oprofiled,*op*,./opreport'
image-exclude: imagelist

Same as image:, but the matching images are excluded.

lib-image: imagelist

Same as image:, but only for images that are for a particular primary binary image (namely, an application).

lib-image-exclude: imagelist

Same as lib-image:, but the matching images are excluded.

event: eventlist

The symbolic event name to match on, e.g. event:DATA_MEM_REFS. You can pass a list of events for side-by-side comparison with opreport.

count: eventcountlist

The event count to match on, e.g. event:DATA_MEM_REFS count:30000. Note that this value refers to the count value in the event spec you passed to operf when setting up to do a profile run. It has nothing to do with the sample counts in the profile data itself. You can pass a list of events for side-by-side comparison with opreport.

unit-mask: masklist

The unit mask value of the event to match on, e.g. unit-mask:1. You can pass a list of events for side-by-side comparison with opreport.

cpu: cpulist

Only consider profiles for the given numbered CPU (starting from zero). This is only useful when using CPU profile separation.

tgid: pidlist

Only consider profiles for the given task groups. Unless some program is using threads, the task group ID of a process is the same as its process ID. This option corresponds to the POSIX notion of a thread group. This is only useful when using per-process profile separation.

tid: tidlist

Only consider profiles for the given threads. When using recent thread libraries, all threads in a process share the same task group ID, but have different thread IDs. You can use this option in combination with tgid: to restrict the results to particular threads within a process. This is only useful when using per-process profile separation.

1.3. Locating and managing binary images

Each session's sample files can be found in the $SESSION_DIR/samples/ directory (default for operf is <cur_dir>/oprofile_data/samples/). These are used, along with the binary image files, to produce human-readable data. In some circumstances (e.g., kernel modules), OProfile will not be able to find the binary images. All the tools have an --image-path option to which you can pass a comma-separated list of alternate paths to search. For example, I can let OProfile find my 2.6 modules by using --image-path /lib/modules/2.6.0/kernel/. It is your responsibility to ensure that the correct images are found when using this option.

Note that if a binary image changes after the sample file was created, you won't be able to get useful symbol-based data out. This situation is detected for you. If you replace a binary, you should make sure to save the old binary if you need to do comparative profiles.

1.4. What to do when you don't get any results

When attempting to get output, you may see the error :

error: no sample files found: profile specification too strict ?

What this is saying is that the profile specification you passed in, when matched against the available sample files, resulted in no matches. There are a number of reasons this might happen:

spelling

You specified a binary name, but spelt it wrongly. Check your spelling !

profiler wasn't running

Make very sure that OProfile was actually up and running when you ran the application you wish to profile.

application didn't run long enough

Remember OProfile is a statistical profiler - you're not guaranteed to get samples for short-running programs. You can help this by using a lower count for the performance counter, so there are a lot more samples taken per second.

application spent most of its time in libraries

Similarly, if the application spends little time in the main binary image itself, with most of it spent in shared libraries it uses, you might not see any samples for the binary image (i.e., executable) itself.

specification was really too strict

For example, you specified something like tgid:3433, but no task with that group ID ever ran the code.

application didn't generate any events

If you're profiling a particular event, for example counting MMX operations, the code might simply have not generated any events in the first place. Verify the code you're profiling does what you expect it to.

you didn't specify kernel module name correctly

If you're trying to get reports for a kernel module, make sure to use the -p option, and specify the module name with the .ko extension. Check if the module is one loaded from initrd.