Chapter 4. Obtaining profiling results

Table of Contents

1. Profile specifications
1.1. Examples
1.2. Profile specification parameters
1.3. Locating and managing binary images
1.4. What to do when you don't get any results
2. Image summaries and symbol summaries (opreport)
2.1. Merging separate profiles
2.2. Side-by-side multiple results
2.3. Callgraph output
2.4. Differential profiles with opreport
2.5. Anonymous executable mappings
2.6. XML formatted output
2.7. Options for opreport
3. Outputting annotated source (opannotate)
3.1. Locating source files
3.2. Usage of opannotate
4. OProfile results with JIT samples
5. gprof-compatible output (opgprof)
5.1. Usage of opgprof
6. Analyzing profile data on another system (oparchive)
6.1. Usage of oparchive
7. Converting sample database files (opimport)
7.1. Usage of opimport

OK, so the profiler has been running, but it's not much use unless we can get some data out. Sometimes, OProfile does a little too good a job of keeping overhead low, and no data reaches the profiler. This can happen on lightly-loaded machines. If you're using OPorifle legacy mode, you can force a dump at any time with :

opcontrol --dump

This ensures that any profile data collected by the oprofiled daemon has been flusehd to disk. Remember to do a dump, stop, shutdown, or deinit before complaining there is no profiling data!

Now that we've got some data, it has to be processed. That's the job of opreport, opannotate, or opgprof.

1. Profile specifications

All of the analysis tools take a profile specification. This is a set of definitions that describe which actual profiles should be examined. The simplest profile specification is empty: this will match all the available profile files for the current session (this is what happens when you do opreport).

Specification parameters are of the form name:value[,value]. For example, if I wanted to get a combined symbol summary for /bin/myprog and /bin/myprog2, I could do opreport -l image:/bin/myprog,/bin/myprog2. As a special case, you don't actually need to specify the image: part here: anything left on the command line is assumed to be an image: name. Similarly, if no session: is specified, then session:current is assumed ("current" is a special name of the current / last profiling session).

In addition to the comma-separated list shown above, some of the specification parameters can take glob-style values. For example, if I want to see image summaries for all binaries profiled in /usr/bin/, I could do opreport image:/usr/bin/\*. Note the necessity to escape the special character from the shell.

For opreport, profile specifications can be used to define two profiles, giving differential output. This is done by enclosing each of the two specifications within curly braces, as shown in the examples below. Any specifications outside of curly braces are shared across both.

1.1. Examples

Image summaries for all profiles with DATA_MEM_REFS samples in the saved session called "stresstest" :

# opreport session:stresstest event:DATA_MEM_REFS

Symbol summary for the application called "test_sym53c8xx,9xx". Note the escaping is necessary as image: takes a comma-separated list.

# opreport -l ./test/test_sym53c8xx\,9xx

Image summaries for all binaries in the test directory, excepting boring-test :

# opreport image:./test/\* image-exclude:./test/boring-test

Differential profile of a binary stored in two archives :

# opreport -l /bin/bash { archive:./orig } { archive:./new }

Differential profile of an archived binary with the current session :

# opreport -l /bin/bash { archive:./orig } { }

1.2. Profile specification parameters

archive: archivepath

A path to an archive made with oparchive. Absence of this tag, unlike others, means "the current system", equivalent to specifying "archive:".

session: sessionlist

A comma-separated list of session names to resolve in. Absence of this tag, unlike others, means "the current session", equivalent to specifying "session:current".

session-exclude: sessionlist

A comma-separated list of sessions to exclude.

image: imagelist

A comma-separated list of image names to resolve. Each entry may be relative path, glob-style name, or full path, e.g.

opreport 'image:/usr/bin/oprofiled,*op*,./opreport'
image-exclude: imagelist

Same as image:, but the matching images are excluded.

lib-image: imagelist

Same as image:, but only for images that are for a particular primary binary image (namely, an application). This only makes sense to use if you're using --separate. This includes kernel modules and the kernel when using --separate=kernel.

lib-image-exclude: imagelist

Same as lib-image:, but the matching images are excluded.

event: eventlist

The symbolic event name to match on, e.g. event:DATA_MEM_REFS. You can pass a list of events for side-by-side comparison with opreport. When using the timer interrupt, the event is always "TIMER".

count: eventcountlist

The event count to match on, e.g. event:DATA_MEM_REFS count:30000. Note that this value refers to the count value in the event spec you passed to opcontrol or operf when setting up to do a profile run. It has nothing to do with the sample counts in the profile data itself. You can pass a list of events for side-by-side comparison with opreport. When using the timer interrupt, the count is always 0 (indicating it cannot be set).

unit-mask: masklist

The unit mask value of the event to match on, e.g. unit-mask:1. You can pass a list of events for side-by-side comparison with opreport.

cpu: cpulist

Only consider profiles for the given numbered CPU (starting from zero). This is only useful when using CPU profile separation.

tgid: pidlist

Only consider profiles for the given task groups. Unless some program is using threads, the task group ID of a process is the same as its process ID. This option corresponds to the POSIX notion of a thread group. This is only useful when using per-process profile separation.

tid: tidlist

Only consider profiles for the given threads. When using recent thread libraries, all threads in a process share the same task group ID, but have different thread IDs. You can use this option in combination with tgid: to restrict the results to particular threads within a process. This is only useful when using per-process profile separation.

1.3. Locating and managing binary images

Each session's sample files can be found in the $SESSION_DIR/samples/ directory (default when using legacy mode: /var/lib/oprofile/samples/; default when using operf: <cur_dir>/oprofile_data/samples/). These are used, along with the binary image files, to produce human-readable data. In some circumstances (e.g., kernel modules), OProfile will not be able to find the binary images. All the tools have an --image-path option to which you can pass a comma-separated list of alternate paths to search. For example, I can let OProfile find my 2.6 modules by using --image-path /lib/modules/2.6.0/kernel/. It is your responsibility to ensure that the correct images are found when using this option.

Note that if a binary image changes after the sample file was created, you won't be able to get useful symbol-based data out. This situation is detected for you. If you replace a binary, you should make sure to save the old binary if you need to do comparative profiles.

1.4. What to do when you don't get any results

When attempting to get output, you may see the error :

error: no sample files found: profile specification too strict ?

What this is saying is that the profile specification you passed in, when matched against the available sample files, resulted in no matches. There are a number of reasons this might happen:

spelling

You specified a binary name, but spelt it wrongly. Check your spelling !

profiler wasn't running

Make very sure that OProfile was actually up and running when you ran the application you wish to profile.

application didn't run long enough

Remember OProfile is a statistical profiler - you're not guaranteed to get samples for short-running programs. You can help this by using a lower count for the performance counter, so there are a lot more samples taken per second.

application spent most of its time in libraries

Similarly, if the application spends little time in the main binary image itself, with most of it spent in shared libraries it uses, you might not see any samples for the binary image (i.e., executable) itself. If you're using OProfile legacy mode profiling, then we recommend using opcontrol --separate=lib before the profiling session so that opreport and friends show the library profiles on a per-application basis. This is done automatically when profiling with operf, so no special setup is necessary.

specification was really too strict

For example, you specified something like tgid:3433, but no task with that group ID ever ran the code.

application didn't generate any events

If you're using a particular event counter, for example counting MMX operations, the code might simply have not generated any events in the first place. Verify the code you're profiling does what you expect it to.

you didn't specify kernel module name correctly

If you're trying to get reports for a kernel module, make sure to use the -p option, and specify the module name with the .ko extension. Check if the module is one loaded from initrd.