Page doesn't render properly ?

How do I build OProfile ?

For 2.4 kernels, you must have the kernel source available for the kernel you want to run OProfile under. Do ./configure --with-linux=/path/to/kernel/source then make install. This option is no longer supported with OProfile 0.9.8 and higher.

For 2.6 or higher kernels, ensure that the kernel has been configured to include OProfile, either built into the kernel or as a module. Next, do ./configure --with-kernel-support. As of OProfile 0.9.8, kernel support is assumed, and the --with-kernel-support option is no longer needed nor available.

Where can I get the required libraries ?

Most required runtime libraries should already be installed on your system. For building OProfile, you may need to install additional devel packages -- for example, the binutils development package, which provides libiberty and others. On Debian, this package is not installed by default, and is named "binutils-dev".

How do I get started ?

Read Getting started.

How do I reset the profiling data ?

For legacy OProfile (i.e., opcontrol-based profiling), use opcontrol --reset, or you can save the session under a name with opcontrol --save <sessionname>. When using operf for profiling, a new profiling session will cause the current profile data (stored in <cur-dir>/oprofile_data/samples/current) to be renamed from current to previous, so there is usually no need to manually delete old profile data. However, a manual deletion can be accomplished by simply doing rm -rf <cur-dir>/oprofile_data/samples/current.

I get "error: no sample files found: profile specification too strict ?"

If using operf, make sure you run OProfile post-processing tools (e.g., opreport, opannotate) from the directory where you collected the profile. Alternatively, you can specify the location of sample data using the --session-dir option.

Incorrect specification of the image name (i.e., app name) when invoking post-processing tools is a common cause for this error message. If you used operf to profile a single application, there is no need to append the image name at the end of your invocation since samples are collected only for the given application.

For other possible causes for this error message, read What to do when you don't get any results.

Can OProfile profile the kernel too ?

Yes, if you pass the --vmlinux option when profiling, OProfile will profile the kernel and all kernel modules, and the post-processing tools will show kernel symbols for which samples were taken. In order for post-processing tools to show symbol info for kernel modules, you must pass the --image-path option; for example, opreport --symbols /lib/modules/`uname -r`.

I don't have a vmlinux file, but I want to profile the kernel anyway !

If using operf, all kernel samples are automatically attributed to a "no-vmlinux" pseudo-symbol, unless the --vmlinux option is used. If profiling with legacy opcontrol, use opcontrol --no-vmlinux if no vmlinux file is available. Then use this script to list the samples.

Why do the profile tools fail to open the vmlinux kernel image ?

Probably because you have accidentally specified the vmlinuz not vmlinux file. If you don't have a vmlinux file, most Linux distributions provide a kernel debuginfo package that includes it. Otherwise, you need to recompile your kernel from source. If you're not interested in kernel samples, then don't use the --vmlinux option (and for legacy profiling, use opcontrol --no-vmlinux).

Why is OProfile ignoring me when I try to change the events to profile ?

When using legacy opcontrol, the oprofiled daemon must be restarted in order to use newly-specified events and other new profiling parameters. To restart the daemon, you must use --shutdown, not --stop, as the latter does not restart the daemon. Note that old profiling data is still kept; only --reset or --save will clear the default sample data directory.

But Red Hat doesn't ship a vmlinux

Actually they do, but it's not installed by default; it's included the kernel-debuginfo package.

Why is OProfile in timer mode on x86? I have performance counters...

Some 2.6+ kernels disable the local APIC by default. You can try enabling the local APIC (if you do indeed have one) by booting with the lapic boot option.

Why does OProfile fail with "can't get RTC I/O ports" ?

On some systems running with pre-2.6 kernels, only RTC mode profiling is available, as described in the documentation. If RTC mode is used, you must not use a kernel with the RTC driver built-in (that is, CONFIG_RTC must not be set to y). Otherwise, OProfile cannot acquire the RTC hardware for its own use, and you will get this error message. If you have compiled the kernel's RTC driver as a module, you must unload it before using OProfile.

Can I get a summary of the whole system ?

Assuming you collected a system-wide profile with either opcontrol or operf --system-wide, try

opreport
or
opreport -l
.

Can I get top-like output from OProfile ?

Sure ! Try something like this :

watch --interval=1 "opcontrol --dump && opreport -l 2>/dev/null | head -30 ; opcontrol --reset"

for a symbol-based system-wide summary, or

watch --interval=1 "opcontrol --dump && opreport -l /boot/2.6.0/vmlinux | head -30 ; opcontrol --reset"

for an image-specific summary.

Oprofile and laptops

Users have reported a few problems using OProfile on laptops.

  1. Many laptops do not have performance counter hardware; in this case OProfile falls back to RTC/timer mode (read the documentation for more details).
  2. Users report that power management is not compatible with OProfile on some laptops. Enabling power management is likely to hang your box if you are using the performance counters. This does not apply if you're using a 2.6 or higher kernel.

What sort of overhead does OProfile have ?

It's dependent upon how you set up the counters, but it's typically lost in the noise (1-3%). This graph gives typical examples.

What architectures are supported by OProfile ?

OProfile ports for IA-64, AMD64, ARM, Alpha, PA-RISC, sparc64, s390, and ppc64 are all in various stages of support, mostly only on 2.6 or higher kernels. Older processor models or defunct architectures may only be supported with the legacy (i.e., opcontrol) profiler, while newer processors/architectures may only be supported with operf.

Can OProfile collect call graphs like gprof?

Yes, starting with OProfile 0.8 and kernel version 2.6, callgraph support is available on many (but not all) architectures, such as x86, ARM, and ppc/ppc64.

What about the "security hole" CVE-2006-0576?

This "problem" only occurs if you actively, and mistakenly, configure access to OProfile via sudo. OProfile uses shell scripts which have not been audited (nor is it likely to happen) for use through the broken sudo facility (anything that lets you alter root's path arbitrarily counts as horribly broken). Do not use sudo!

I get an error from the post-profiling tools referring here?

Some binutils versions (at least 2.12) are mis-compiled by gcc 2.95.3, which causes a crash or hang in opreport and the other tools. See the OProfile bug report for a patch to work around the problem. This is reported to be a problem with Slackware 8.1, and probably some other older Linux distributions.

Using the 2.4 kernel module, I get "Failed to open hash map device: Operation not permitted"?

This is an intermittent problem that was never completely resolved. The device file /var/lib/oprofile/hashmapdev/ is failing the capable(CAP_SYS_PTRACE) test. The simplest workaround is to remove both of these tests in the source (oprofile.c), recompile and reinstall OProfile, then run opcontrol --deinit and start again.

SUSE kernel, I get: "May 8 11:05:14 mynode kernel: Unable to handle kernel paging request at virtual address ffffe030"

This occurs with older SUSE UP kernel when you compile your kernel with CONFIG_X86_LOCAL_APIC. By default, these kernels do not up the local apic. You must force local apic initialization by booting with the kernel command line parameter apic.

VMware

OProfile can't work with VMware when using performance counter interface. A workaround is to use RTC mode (2.4 kernel) or timer interrupt mode (2.6+ kernel)

What is the meaning of "cpu_type 'unset' is not valid"

This error can occur when using legacy opcontrol for profiling. Generally this means your kernel supports your hardware, but user space tools are not up to date. You should try to upgrade. You can check if your hardware is supported by looking at /dev/oprofile/cpu_type.

What are the [vdso] messages I sometimes see from opreport?

Occasionally, opreport displays messages like the following:

warning: [vdso] (tgid:1664 range:0xf732a000-0xf732b000) could not be found
If you get this error message, you can safely ignore it. It just indicates temporary memory allocations from the kernel where it loaded some executable code into memory. The fact you are seeing those messages indicates some samples were taken when that code was running, but now, at post-processing time, 'opreport' cannot find the VDSO anymore. VDSO is virtual dynamic shared object. From http://kernelnewbies.org/KernelGlossary, it's "a kernel-provided shared library that helps userspace perform a few kernel actions without the overhead of a system call, as well as automatically choosing the most efficient syscall mechanism. Also called the "vsyscall page".

What do I do when configure fails with "syntax error near unexpeced token QT"?

Under certain conditions, you may see the following oprofile configure error:

./configure: line 20300: syntax error near unexpected token `QT,' ./configure: line 20300: ` PKG_CHECK_MODULES(QT, Qt3Support QtGui QtCore ,,'

In general, there are multiple possible causes for this error, including the pkgconfig package not being installed. But you can also get this error if you have installed aclocal in a location other than /usr. Here's a scenario that can get you into trouble.

You have installed the automake package (which includes aclocal) from your distro, along with other packages that have m4 files (such as pkgconfig). Then, for one reason or another, you install another version of aclocal via a local build/install. Unless you specify differently with the '--prefix' option, this other version of aclocal will be installed at /usr/local/bin/aclocal, which typically will be found in your PATH ahead of the /usr/bin/aclocal. Then, when running oprofile's autogen.sh, aclocal is called and it looks for a dirlist file in <aclocal-install-dir>/share/aclocal. The distro-supplied aclocal includes a dirlist file, which tells aclocal of locations to look for m4 files (such as the pkg.m4 file). But your version of aclocal in /usr/local does not have this dirlist file automatically, so your aclocal is not able to find pkg.m4 and fails with the above syntax error.

There are two ways to eliminate this error: 1) copy the /usr/share/aclocal/dirlist file to your other aclocal installation, and edit the file to point back to '/usr/share/aclocal' (RECOMMENDED); or 2) add "-I /usr/share/aclocal" to the invocation of aclocal in oprofile's autogen.sh either by editing the autogen.s file or by invoking autogen.sh as follows:

ACLOCAL='aclocal -I /usr/share/aclocal' ./autogen.sh

operf issues on older kernels

The operf tool depends on a kernel feature called Linux Kernel Performance Events Subsystem (aka "perf_events"). This feature was initially included with the 2.6.31 kernel, but over the next several versions, it evolved rapidly to fix bugs and add new capabilities. Several Linux distributions were released using kernels based on 2.6.31 or 2.6.32 which were heavily patched with perf_events fixes. However, not all critical fixes made their way into these early perf_events-enabled Linux distributions. For example, the operf tool from the official 0.9.8 release is completely broken when used on SLES 11 SP1, where the following error message occurs:

Unexpected error running operf: Permission denied Please use the opcontrol command instead of operf.
A fix for this error is available upstream and will be available in 0.9.9. But the following permanent restriction is in place when running on such older kernels:
When running operf as the root user using the syntax operf <command>, the profiling phase seems to succeed, but opreport shows no samples collected. The workaround is to use a non-root user account for profiling a single application, and use the root user account only for --system-wide profiling.

Why doesn't the summary sample count for some binary images match the number of individual symbols' samples for the same binary?

Usually, the total of all per-symbols samples for a given binary image equals the summary count for the binary image (shown by running opreport with no options). However, it's possible for some sample addresses to fall outside the range of any symbols for a given binary image. In such cases, the total number of per-symbols samples for the binary image may be less than the summary count for the image. Running opreport with the --verbose=debug option will display an informational message when this difference is detected. This difference is typically very small and can be ignored.

It is a capital mistake to theorise before one has data. Insensibly one begins to twist facts to suit theories instead of theories to suit facts. - Sherlock Holmes
2013/07/29