396 lines · plain
1====================2XRay Instrumentation3====================4 5:Version: 1 as of 2016-11-086 7.. contents::8 :local:9 10 11Introduction12============13 14XRay is a function call tracing system which combines compiler-inserted15instrumentation points and a runtime library that can dynamically enable and16disable the instrumentation.17 18More high level information about XRay can be found in the `XRay whitepaper`_.19 20This document describes how to use XRay as implemented in LLVM.21 22XRay in LLVM23============24 25XRay consists of three main parts:26 27- Compiler-inserted instrumentation points.28- A runtime library for enabling/disabling tracing at runtime.29- A suite of tools for analysing the traces.30 31 **NOTE:** As of July 25, 2018 , XRay is only available for the following32 architectures running Linux: x86_64, arm7 (no thumb), aarch64, powerpc64le,33 mips, mipsel, mips64, mips64el, NetBSD: x86_64, FreeBSD: x86_64 and34 OpenBSD: x86_64.35 36The compiler-inserted instrumentation points come in the form of nop-sleds in37the final generated binary, and an ELF section named ``xray_instr_map`` which38contains entries pointing to these instrumentation points. The runtime library39relies on being able to access the entries of the ``xray_instr_map``, and40overwrite the instrumentation points at runtime.41 42Using XRay43==========44 45You can use XRay in a couple of ways:46 47- Instrumenting your C/C++/Objective-C/Objective-C++ application.48- Generating LLVM IR with the correct function attributes.49 50The rest of this section covers these main ways and later on how to customize51what XRay does in an XRay-instrumented binary.52 53Instrumenting your C/C++/Objective-C Application54------------------------------------------------55 56The easiest way of getting XRay instrumentation for your application is by57enabling the ``-fxray-instrument`` flag in your clang invocation.58 59For example:60 61::62 63 clang -fxray-instrument ...64 65By default, functions that have at least 200 instructions (or contain a loop) will66get XRay instrumentation points. You can tweak that number through the67``-fxray-instruction-threshold=`` flag:68 69::70 71 clang -fxray-instrument -fxray-instruction-threshold=1 ...72 73The loop detection can be disabled with ``-fxray-ignore-loops`` to use only the74instruction threshold. You can also specifically instrument functions in your75binary to either always or never be instrumented using source-level attributes.76You can do it using the GCC-style attributes or C++11-style attributes.77 78.. code-block:: c++79 80 [[clang::xray_always_instrument]] void always_instrumented();81 82 [[clang::xray_never_instrument]] void never_instrumented();83 84 void alt_always_instrumented() __attribute__((xray_always_instrument));85 86 void alt_never_instrumented() __attribute__((xray_never_instrument));87 88When linking a binary, you can either manually link in the `XRay Runtime89Library`_ or use ``clang`` to link it in automatically with the90``-fxray-instrument`` flag. Alternatively, you can statically link-in the XRay91runtime library from compiler-rt -- those archive files will take the name of92`libclang_rt.xray-{arch}` where `{arch}` is the mnemonic supported by clang93(x86_64, arm7, etc.).94 95LLVM Function Attribute96-----------------------97 98If you're using LLVM IR directly, you can add the ``function-instrument``99string attribute to your functions, to get the similar effect that the100C/C++/Objective-C source-level attributes would get:101 102.. code-block:: llvm103 104 define i32 @always_instrument() uwtable "function-instrument"="xray-always" {105 ; ...106 }107 108 define i32 @never_instrument() uwtable "function-instrument"="xray-never" {109 ; ...110 }111 112You can also set the ``xray-instruction-threshold`` attribute and provide a113numeric string value for how many instructions should be in the function before114it gets instrumented.115 116.. code-block:: llvm117 118 define i32 @maybe_instrument() uwtable "xray-instruction-threshold"="2" {119 ; ...120 }121 122Special Case File123-----------------124 125Attributes can be imbued through the use of special case files instead of126adding them to the original source files. You can use this to mark certain127functions and classes to be never, always, or instrumented with first-argument128logging from a file. The file's format is described below:129 130.. code-block:: bash131 132 # Comments are supported133 [always]134 fun:always_instrument135 fun:log_arg1=arg1 # Log the first argument for the function136 137 [never]138 fun:never_instrument139 140These files can be provided through the ``-fxray-attr-list=`` flag to clang.141You may have multiple files loaded through multiple instances of the flag.142 143XRay Runtime Library144--------------------145 146The XRay Runtime Library is part of the compiler-rt project, which implements147the runtime components that perform the patching and unpatching of inserted148instrumentation points. When you use ``clang`` to link your binaries and the149``-fxray-instrument`` flag, it will automatically link in the XRay runtime.150 151The default implementation of the XRay runtime will enable XRay instrumentation152before ``main`` starts, which works for applications that have a short153lifetime. This implementation also records all function entry and exit events154which may result in a lot of records in the resulting trace.155 156Also by default the filename of the XRay trace is ``xray-log.XXXXXX`` where the157``XXXXXX`` part is randomly generated.158 159These options can be controlled through the ``XRAY_OPTIONS`` environment160variable during program run-time, where we list down the options and their161defaults below.162 163+-------------------+-----------------+---------------+------------------------+164| Option | Type | Default | Description |165+===================+=================+===============+========================+166| patch_premain | ``bool`` | ``false`` | Whether to patch |167| | | | instrumentation points |168| | | | before main. |169+-------------------+-----------------+---------------+------------------------+170| xray_mode | ``const char*`` | ``""`` | Default mode to |171| | | | install and initialize |172| | | | before ``main``. |173+-------------------+-----------------+---------------+------------------------+174| xray_logfile_base | ``const char*`` | ``xray-log.`` | Filename base for the |175| | | | XRay logfile. |176+-------------------+-----------------+---------------+------------------------+177| verbosity | ``int`` | ``0`` | Runtime verbosity |178| | | | level. |179+-------------------+-----------------+---------------+------------------------+180 181 182In addition to environment variable, you can also provide your own definition of183``const char *__xray_default_options(void)`` function, which returns the option184strings. This method effectively provides default options during program build185time. For example, you can create an additional source file (e.g. ``xray-opt.c``186) with the following ``__xray_default_options`` definition:187 188.. code-block:: c189 190 __attribute__((xray_never_instrument))191 const char *__xray_default_options() {192 return "patch_premain=true,xray_mode=xray-basic";193 }194 195And link it with the program you want to instrument:196 197::198 199 clang -fxray-instrument prog.c xray-opt.c ...200 201The instrumented binary will use 'patch_premain=true,xray_mode=xray-basic' by202default even without setting ``XRAY_OPTIONS``.203 204Note that you still can override options designated by ``__xray_default_options``205using ``XRAY_OPTIONS`` during run-time.206 207If you choose to not use the default logging implementation that comes with the208XRay runtime and/or control when/how the XRay instrumentation runs, you may use209the XRay APIs directly for doing so. To do this, you'll need to include the210``xray_log_interface.h`` from the compiler-rt ``xray`` directory. The important API211functions we list below:212 213- ``__xray_log_register_mode(...)``: Register a logging implementation against214 a string Mode identifier. The implementation is an instance of215 ``XRayLogImpl`` defined in ``xray/xray_log_interface.h``.216- ``__xray_log_select_mode(...)``: Select the mode to install, associated with217 a string Mode identifier. Only implementations registered with218 ``__xray_log_register_mode(...)`` can be chosen with this function.219- ``__xray_log_init_mode(...)``: This function allows for initializing and220 re-initializing an installed logging implementation. See221 ``xray/xray_log_interface.h`` for details, part of the XRay compiler-rt222 installation.223 224Once a logging implementation has been initialized, it can be "stopped" by225finalizing the implementation through the ``__xray_log_finalize()`` function.226The finalization routine is the opposite of the initialization. When finalized,227an implementation's data can be cleared out through the228``__xray_log_flushLog()`` function. For implementations that support in-memory229processing, these should register an iterator function to provide access to the230data via the ``__xray_log_set_buffer_iterator(...)`` which allows code calling231the ``__xray_log_process_buffers(...)`` function to deal with the data in232memory.233 234All of this is better explained in the ``xray/xray_log_interface.h`` header.235 236Basic Mode237----------238 239XRay supports a basic logging mode which will trace the application's240execution, and periodically append to a single log. This mode can be241installed/enabled by setting ``xray_mode=xray-basic`` in the ``XRAY_OPTIONS``242environment variable. Combined with ``patch_premain=true`` this can allow for243tracing applications from start to end.244 245Like all the other modes installed through ``__xray_log_select_mode(...)``, the246implementation can be configured through the ``__xray_log_init_mode(...)``247function, providing the mode string and the flag options. Basic-mode specific248defaults can be provided in the ``XRAY_BASIC_OPTIONS`` environment variable.249 250Flight Data Recorder Mode251-------------------------252 253XRay supports a logging mode which allows the application to only capture a254fixed amount of memory's worth of events. Flight Data Recorder (FDR) mode works255very much like a plane's "black box" which keeps recording data to memory in a256fixed-size circular queue of buffers, and have the data available257programmatically until the buffers are finalized and flushed. To use FDR mode258on your application, you may set the ``xray_mode`` variable to ``xray-fdr`` in259the ``XRAY_OPTIONS`` environment variable. Additional options to the FDR mode260implementation can be provided in the ``XRAY_FDR_OPTIONS`` environment261variable. Programmatic configuration can be done by calling262``__xray_log_init_mode("xray-fdr", <configuration string>)`` once it has been263selected/installed.264 265When the buffers are flushed to disk, the result is a binary trace format266described by `XRay FDR format <XRayFDRFormat.html>`_267 268When FDR mode is on, it will keep writing and recycling memory buffers until269the logging implementation is finalized -- at which point it can be flushed and270re-initialised later. To do this programmatically, we follow the workflow271provided below:272 273.. code-block:: c++274 275 // Patch the sleds, if we haven't yet.276 auto patch_status = __xray_patch();277 278 // Maybe handle the patch_status errors.279 280 // When we want to flush the log, we need to finalize it first, to give281 // threads a chance to return buffers to the queue.282 auto finalize_status = __xray_log_finalize();283 if (finalize_status != XRAY_LOG_FINALIZED) {284 // maybe retry, or bail out.285 }286 287 // At this point, we are sure that the log is finalized, so we may try288 // flushing the log.289 auto flush_status = __xray_log_flushLog();290 if (flush_status != XRAY_LOG_FLUSHED) {291 // maybe retry, or bail out.292 }293 294The default settings for the FDR mode implementation will create logs named295similarly to the basic log implementation, but will have a different log296format. All the trace analysis tools (and the trace reading library) will297support all versions of the FDR mode format as we add more functionality and298record types in the future.299 300 **NOTE:** We do not promise perpetual support for when we update the log301 versions we support going forward. Deprecation of the formats will be302 announced and discussed on the developers mailing list.303 304Trace Analysis Tools305--------------------306 307We currently have the beginnings of a trace analysis tool in LLVM, which can be308found in the ``tools/llvm-xray`` directory. The ``llvm-xray`` tool currently309supports the following subcommands:310 311- ``extract``: Extract the instrumentation map from a binary, and return it as312 YAML.313- ``account``: Performs basic function call accounting statistics with various314 options for sorting, and output formats (supports CSV, YAML, and315 console-friendly TEXT).316- ``convert``: Converts an XRay log file from one format to another. We can317 convert from binary XRay traces (both basic and FDR mode) to YAML,318 `flame-graph <https://github.com/brendangregg/FlameGraph>`_ friendly text319 formats, as well as `Chrome Trace Viewer (catapult)320 <https://github.com/catapult-project/catapult>` formats.321- ``graph``: Generates a DOT graph of the function call relationships between322 functions found in an XRay trace.323- ``stack``: Reconstructs function call stacks from a timeline of function324 calls in an XRay trace.325 326These subcommands use various library components found as part of the XRay327libraries, distributed with the LLVM distribution. These are:328 329- ``llvm/XRay/Trace.h`` : A trace reading library for conveniently loading330 an XRay trace of supported forms, into a convenient in-memory representation.331 All the analysis tools that deal with traces use this implementation.332- ``llvm/XRay/Graph.h`` : A semi-generic graph type used by the graph333 subcommand to conveniently represent a function call graph with statistics334 associated with edges and vertices.335- ``llvm/XRay/InstrumentationMap.h``: A convenient tool for analyzing the336 instrumentation map in XRay-instrumented object files and binaries. The337 ``extract`` and ``stack`` subcommands uses this particular library.338 339 340Minimizing Binary Size341----------------------342 343XRay supports several different instrumentation points including ``function-entry``,344``function-exit``, ``custom``, and ``typed`` points. These can be enabled individually345using the ``-fxray-instrumentation-bundle=`` flag. For example if you only wanted to346instrument function entry and custom points you could specify:347 348::349 350 clang -fxray-instrument -fxray-instrumentation-bundle=function-entry,custom ...351 352This will omit the other sled types entirely, reducing the binary size. You can also353instrument just a sampled subset of functions using instrumentation groups.354For example, to instrument only a quarter of available functions invoke:355 356::357 358 clang -fxray-instrument -fxray-function-groups=4359 360A subset will be chosen arbitrarily based on a hash of the function name. To sample a361different subset you can specify ``-fxray-selected-function-group=`` with a group number362in the range of 0 to ``xray-function-groups`` - 1. Together these options could be used363to produce multiple binaries with different instrumented subsets. If all you need is364runtime control over which functions are being traced at any given time it is better365to selectively patch and unpatch the individual functions you need using the XRay366Runtime Library's ``__xray_patch_function()`` method.367 368Future Work369===========370 371There are a number of ongoing efforts for expanding the toolset building around372the XRay instrumentation system.373 374Trace Analysis Tools375--------------------376 377- Work is in progress to integrate with or develop tools to visualize findings378 from an XRay trace. Particularly, the ``stack`` tool is being expanded to379 output formats that allow graphing and exploring the duration of time in each380 call stack.381- With a large instrumented binary, the size of generated XRay traces can382 quickly become unwieldy. We are working on integrating pruning techniques and383 heuristics for the analysis tools to sift through the traces and surface only384 relevant information.385 386More Platforms387--------------388 389We're looking forward to contributions to port XRay to more architectures and390operating systems.391 392.. References...393 394.. _`XRay whitepaper`: http://research.google.com/pubs/pub45287.html395 396