Revision as of 22:48, 21 November 2010

Introduction

LTTng (Linux Tracing Toolkit, next generation) is a highly efficient tracing tool for Linux that can be used to track down performance issues and debug problems involving multiple concurrent processes and threads. It consists of a set of kernel patches and a daemon, to collect the raw tracing data, as well as a set of tools to control, visualize and analyze the generated data.

The LTTng plug-in for Eclipse aims at replacing LTTv (LTTng visualizer), the component used to control, fetch, analyze and visualize traces produced by the LTTng tracer. A present, the LTTng Eclipse plug-in for Eclipse supports the following analyses: Control Flow, Resources and Statistics.

For more information about LTTng, refer to the project site

About Tracing

Tracing is a technique used to understand what is going on in a system in order to debug or monitor it. A tracer is the software used for tracing. Tracing can be used to debug a wide range of bugs that are otherwise extremely challenging. These include, for example, performance problems in complex parallel systems or real-time systems.

Tracing is similar to logging: it consists in recording events that happen in a system. However, compared to logging, it usually records much lower-level events that occur much more frequently. Tracers must therefore be optimized to handle a lot of data while having a small impact on the system. Tracers typically generate thousands of events per second. They generated trace frequently can easily contain millions of events and have sizes from many megabytes to tens of gigabytes.

Traces may include events from the operating system kernel (IRQ handler entry/exit, system call entry/exit, scheduling activity, network activity, etc). It can also consists of application events (a.k.a UST - User Space Tracing) or a mix of the two.

For the maximum level of detail, a tracing events may be viewed like a log file. However, trace analyzers and viewers are available to produce useful information from the raw data. These programs must be specially designed to handle quickly the enormous amount of data traces contain.

In the case of LTTng, low tracing overhead is achieved by instrumenting the Linux kernel with a set of custom patches. The same set of patches can be used for both Linux kernel as well as UST.

Traces and Experiments

In the scope of the LTTng plug-in for Eclipse, a trace is essentially a (very) large set of time-ordered LTTng events. The LTTng application can read these traces and provide a number of standard views to analyze their contents.

An experiment consists in an arbitrary number of aggregated LTTng traces for purpose of correlation. In the degenerate case, an experiment can consist of a single trace.

The experiment provides a unified, time-ordered stream of the individual LTTng trace events.

Installation

• LTTv comes with a very efficient parsing library (in C) which was not re-implemented in Java for this project. Instead, the library is simply integrated using JNI. As a consequence, you need to download, compile and install the parsing library. See #Installing the LTTng parsing library for detailed information.

• Recording of an LTTng trace directly from Eclipse is currently a work in progress. To record your own trace, you will need to install the LTTng kernel patches and the LTTv application (refer to the LTTng site for more information on how to proceed). To verify that your installation is functional, you can download sample traces from the Eclipse repository. See #Sample LTTng Traces for detailed information.

Installing the LTTng parsing library

This trace parsing library is not delivered as part of Eclipse (licensing stuff) and has to be downloaded from the LTTng site, compiled and then installed on your Linux system.

Currently, the Eclipse plug-in supports the following version of the LTTng trace format: 2.3, 2.5 and 2.6. Depending on which trace version(s) you wish to analyze, you will need to download the appropriate snapshot from the corresponding git repository (pick the "snapshot" link next to the latest commit):

Trace format 2.3
Trace format 2.5
Trace format 2.6

Build and install the snapshot(s) you downloaded. If you wish to install more than one version of the library, you will have to build them in format version sequence (e.g. build 2.6 last). For a plain library installation (in /usr/local/lib), you can execute the following commands for each version you wish to install:

$ cd liblttngtrace-2.n
$ ./autogen.sh --with-jni-interface --without-lttv && make
# make install

You will end up with something like this (the loader acts as a factory to pre-analyze the trace and load the correct parser library):

$ ls -l1 /usr/local/lib
total 1048
-rwxr-xr-x. 1 root root 210300 2010-06-07 11:24 liblttvtraceread-2.5.so
-rwxr-xr-x. 1 root root 211052 2010-06-07 11:25 liblttvtraceread-2.6.so
-rw-r--r--. 1 root root 316040 2010-06-07 11:25 liblttvtraceread.a
-rwxr-xr-x. 1 root root   1175 2010-06-07 11:25 liblttvtraceread.la
-rwxr-xr-x. 1 root root  93963 2010-06-07 11:24 liblttvtraceread_loader-2.5.so
-rwxr-xr-x. 1 root root  93963 2010-06-07 11:25 liblttvtraceread_loader-2.6.so
-rw-r--r--. 1 root root 125442 2010-06-07 11:25 liblttvtraceread_loader.a
-rwxr-xr-x. 1 root root   1224 2010-06-07 11:25 liblttvtraceread_loader.la
lrwxrwxrwx. 1 root root     30 2010-06-07 11:25 liblttvtraceread_loader.so -> liblttvtraceread_loader-2.6.so
lrwxrwxrwx. 1 root root     23 2010-06-07 11:25 liblttvtraceread.so -> liblttvtraceread-2.6.so

A number of configuration options are available to fine tune the library builds:

$ ./configure -- help
Usage: ./configure [OPTION]... [VAR=VALUE]...

To assign environment variables (e.g., CC, CFLAGS...), specify them as
VAR=VALUE.  See below for descriptions of some of the useful variables.

Defaults for the options are specified in brackets.

Configuration:
  -h, --help              display this help and exit
      --help=short        display options specific to this package
      --help=recursive    display the short help of all the included packages
  -V, --version           display version information and exit
  -q, --quiet, --silent   do not print `checking ...' messages
...
  <more options>

Finally, you have to set your LD_LIBRARY_PATH environment variable so Eclipse can find the parsing libraries. Otherwise, you have to specify the path when you start Eclipse:

$ LD_LIBRARY_PATH="/usr/local/lib:$LD_LIBRARY_PATH" ./eclipse

Sample LTTng Traces

The following LTTng traces are provided as samples.

Each trace as its own characteristics and complexity. The size given refers to the size of the trace once "untarred", the actual tar size should be ~10 times smaller.

• trace-15316 (456K) Very small trace (format 2.5 - 15316 events) made from a single LTTng marker. Recorded with a VirtualBox system with 1 processor.

• trace-15471 (460K) Very small trace (format 2.5 - 15471 events) made from a single LTTng marker. Recorded with a VirtualBox system with 1 processor.

(More sample traces to come)

LTTng Perspective

Project View

Events View

Statistics View

Control Flow View

Resources View

Histogram View

Limitations

References