250 lines · plain
1Tracing with Intel Processor Trace2==================================3 4Intel PT is a technology available in modern Intel CPUs that allows efficient5tracing of all the instructions executed by a process.6LLDB can collect traces and dump them using its symbolication stack.7You can read more here8https://easyperf.net/blog/2019/08/23/Intel-Processor-Trace.9 10Prerequisites11-------------12 13Confirm that your CPU supports Intel PT14(see https://www.intel.com/content/www/us/en/support/articles/000056730/processors.html)15and that your operating system is Linux.16 17Check for the existence of this particular file on your Linux system18 19::20 21 $ cat /sys/bus/event_source/devices/intel_pt/type22 23The output should be a number. Otherwise, try upgrading your kernel.24 25 26Build Instructions27------------------28 29Clone and build the low level Intel PT30decoder library `LibIPT library <https://github.com/intel/libipt>`_.31::32 33 $ git clone git@github.com:intel/libipt.git34 $ mkdir libipt-build35 $ cmake -S libipt -B libipt-build36 $ cd libipt-build37 $ make38 39This will generate a few files in the ``<libipt-build>/lib``40and ``<libipt-build>/libipt/include`` directories.41 42Configure and build LLDB with Intel PT support43 44::45 46 $ cmake \47 -DLLDB_BUILD_INTEL_PT=ON \48 -DLIBIPT_INCLUDE_PATH="<libipt-build>/libipt/include" \49 -DLIBIPT_LIBRARY_PATH="<libipt-build>/lib" \50 ... other common configuration parameters51 52::53 54 $ cd <lldb-build> && ninja lldb lldb-server # if using Ninja55 56 57How to Use58----------59 60When you are debugging a process, you can turn on intel-pt tracing,61which will “record” all the instructions that the process will execute.62After turning it on, you can continue debugging, and at any breakpoint,63you can inspect the instruction list.64 65For example:66 67::68 69 lldb <target>70 > b main71 > run72 > process trace start # start tracing on all threads, including future ones73 # keep debugging until you hit a breakpoint74 75 > thread trace dump instructions76 # this should output something like77 78 thread #2: tid = 2861133, total instructions = 530567379 libc.so.6`__GI___libc_read + 45 at read.c:25:180 [4962255] 0x00007fffeb64c63d subq $0x10, %rsp81 [4962256] 0x00007fffeb64c641 movq %rdi, -0x18(%rbp)82 libc.so.6`__GI___libc_read + 53 [inlined] __libc_read at read.c:26:1083 [4962257] 0x00007fffeb64c645 callq 0x7fffeb66b640 ; __libc_enable_asynccancel84 libc.so.6`__libc_enable_asynccancel85 [4962258] 0x00007fffeb66b640 movl %fs:0x308, %eax86 libc.so.6`__libc_enable_asynccancel + 887 [4962259] 0x00007fffeb66b648 movl %eax, %r11d88 89 # you can keep pressing ENTER to see more and more instructions90 91The number between brackets is the instruction index,92and by default the current thread will be picked.93 94Configuring the trace size95--------------------------96 97The CPU stores the instruction list in a compressed format in a ring buffer,98which keeps the latest information.99By default, LLDB uses a buffer of 4KB per thread,100but you can change it by running.101The size must be a power of 2 and at least 4KB.102 103::104 105 thread trace start all -s <size_in_bytes>106 107For reference, a 1MB trace buffer can easily store around 5M instructions.108 109Printing more instructions110--------------------------111 112If you want to dump more instructions at a time, you can run113 114::115 116 thread trace dump instructions -c <count>117 118Printing the instructions of another thread119-------------------------------------------120 121By default the current thread will be picked when dumping instructions,122but you can do123 124::125 126 thread trace dump instructions <#thread index>127 #e.g.128 thread trace dump instructions 8129 130to select another thread.131 132Crash Analysis133--------------134 135What if you are debugging + tracing a process that crashes?136Then you can just do137 138::139 140 thread trace dump instructions141 142To inspect how it crashed! There's nothing special that you need to do.143For example144 145::146 147 * thread #1, name = 'a.out', stop reason = signal SIGFPE: integer divide by zero148 frame #0: 0x00000000004009f1 a.out`main at main.cpp:8:14149 6 int x;150 7 cin >> x;151 -> 8 cout << 12 / x << endl;152 9 return 0;153 10 }154 (lldb) thread trace dump instructions -c 5155 thread #1: tid = 604302, total instructions = 8388156 libstdc++.so.6`std::istream::operator>>(int&) + 181157 [8383] 0x00007ffff7b41665 popq %rbp158 [8384] 0x00007ffff7b41666 retq159 a.out`main + 66 at main.cpp:8:14160 [8385] 0x00000000004009e8 movl -0x4(%rbp), %ecx161 [8386] 0x00000000004009eb movl $0xc, %eax162 [8387] 0x00000000004009f0 cltd163 164.. note::165 At this moment, we are not including the failed instruction in the trace,166 but in the future we might do it for readability.167 168 169Offline Trace Analysis170----------------------171 172It's also possible to record a trace using a custom Intel PT collector173and decode + symbolicate the trace using LLDB.174For that, the command trace load is useful.175In order to use trace load, you need to first create a JSON file with176the definition of the trace session.177For example178 179::180 181 {182 "type": "intel-pt",183 "cpuInfo": {184 "vendor": "GenuineIntel",185 "family": 6,186 "model": 79,187 "stepping": 1188 },189 "processes": [190 {191 "pid": 815455,192 "triple": "x86_64-*-linux",193 "threads": [194 {195 "tid": 815455,196 "iptTrace": "trace.file" # raw thread-specific trace from the AUX buffer197 }198 ],199 "modules": [ # this are all the shared libraries + the main executable200 {201 "file": "a.out", # optional if it's the same as systemPath202 "systemPath": "a.out",203 "loadAddress": 4194304,204 },205 {206 "file": "libfoo.so",207 "systemPath": "/usr/lib/libfoo.so",208 "loadAddress": "0x00007ffff7bd9000",209 },210 {211 "systemPath": "libbar.so",212 "loadAddress": "0x00007ffff79d7000",213 }214 ]215 }216 ]217 }218 219You can see the full schema by typing220 221::222 223 trace schema intel-pt224 225The JSON file mainly contains all the shared libraries that226were part of the traced process, along with their memory load address.227If the analysis is done on the same computer where the traces were obtained,228it's enough to use the “systemPath” field.229If the analysis is done on a different machines, these files need to be230copied over and the “file” field should point to the231location of the file relative to the JSON file.232Once you have the JSON file and the module files in place, you can simple run233 234::235 236 lldb237 > trace load /path/to/json238 > thread trace dump instructions <optional thread index>239 240Then it's like in the live session case241 242References243----------244 245- Original RFC document_ for this feature.246- Some details about how Meta is using Intel Processor Trace can be found in this blog_ post.247 248.. _document: https://docs.google.com/document/d/1cOVTGp1sL_HBXjP9eB7qjVtDNr5xnuZvUUtv43G5eVI249.. _blog: https://engineering.fb.com/2021/04/27/developer-tools/reverse-debugging/250