442 lines · plain
1==========================2Symbolizer Markup Format3==========================4 5.. contents::6 :local:7 8Overview9========10 11This document defines a text format for log messages that can be processed by a12symbolizing filter. The basic idea is that logging code emits text that contains13raw address values and so forth, without the logging code doing any real work to14convert those values to human-readable form. Instead, logging text uses the15markup format defined here to identify pieces of information that should be16converted to human-readable form after the fact. As with other markup formats,17the expectation is that most of the text will be displayed as is, while the18markup elements will be replaced with expanded text, or converted into active UI19elements, that present more details in symbolic form.20 21This means there is no need for symbol tables, DWARF debugging sections, or22similar information to be directly accessible at runtime. There is also no need23at runtime for any logic intended to compute human-readable presentation of24information, such as C++ symbol demangling. Instead, logging must include markup25elements that give the contextual information necessary to make sense of the raw26data, such as memory layout details.27 28This format identifies markup elements with a syntax that is both simple and29distinctive. It's simple enough to be matched and parsed with straightforward30code. It's distinctive enough that character sequences that look like the start31or end of a markup element should rarely if ever appear incidentally in logging32text. It's specifically intended not to require sanitizing plain text, such as33the HTML/XML requirement to replace ``<`` with ``<`` and the like.34 35:doc:`llvm-symbolizer <CommandGuide/llvm-symbolizer>` includes a symbolizing36filter via its ``--filter-markup`` option. Also, LLVM utilites emit stack37traces as markup when the ``LLVM_ENABLE_SYMBOLIZER_MARKUP`` environment38variable is set.39 40Scope and assumptions41=====================42 43A symbolizing filter implementation will be independent both of the target44operating system and machine architecture where the logs are generated and of45the host operating system and machine architecture where the filter runs.46 47This format assumes that the symbolizing filter processes intact whole lines. If48long lines might be split during some stage of a logging pipeline, they must be49reassembled to restore the original line breaks before feeding lines into the50symbolizing filter. Most markup elements must appear entirely on a single line51(often with other text before and/or after the markup element). There are some52markup elements that are specified to span lines, with line breaks in the middle53of the element. Even in those cases, the filter is not expected to handle line54breaks in arbitrary places inside a markup element, but only inside certain55fields.56 57This format assumes that the symbolizing filter processes a coherent stream of58log lines from a single process address space context. If a logging stream59interleaves log lines from more than one process, these must be collated into60separate per-process log streams and each stream processed by a separate61instance of the symbolizing filter. Because the kernel and user processes use62disjoint address regions in most operating systems, a single user process63address space plus the kernel address space can be treated as a single address64space for symbolization purposes if desired.65 66Dependence on Build IDs67=======================68 69The symbolizer markup scheme relies on contextual information about runtime70memory address layout to make it possible to convert markup elements into useful71symbolic form. This relies on having an unmistakable identification of which72binary was loaded at each address.73 74An ELF Build ID is the payload of an ELF note with name ``"GNU"`` and type75``NT_GNU_BUILD_ID``, a unique byte sequence that identifies a particular binary76(executable, shared library, loadable module, or driver module). The linker77generates this automatically based on a hash that includes the complete symbol78table and debugging information, even if this is later stripped from the binary.79 80This specification uses the ELF Build ID as the sole means of identifying81binaries. Each binary relevant to the log must have been linked with a unique82Build ID. The symbolizing filter must have some means of mapping a Build ID back83to the original ELF binary (either the whole unstripped binary, or a stripped84binary paired with a separate debug file).85 86Colorization87============88 89The markup format supports a restricted subset of ANSI X3.64 SGR (Select Graphic90Rendition) control sequences. These are unlike other markup elements:91 92* They specify presentation details (bold or colors) rather than semantic93 information. The association of semantic meaning with color (e.g. red for94 errors) is chosen by the code doing the logging, rather than by the UI95 presentation of the symbolizing filter. This is a concession to existing code96 (e.g. LLVM sanitizer runtimes) that use specific colors and would require97 substantial changes to generate semantic markup instead.98 99* A single control sequence changes "the state", rather than being an100 hierarchical structure that surrounds affected text.101 102The filter processes ANSI SGR control sequences only within a single line. If a103control sequence to enter a bold or color state is encountered, it's expected104that the control sequence to reset to default state will be encountered before105the end of that line. If a "dangling" state is left at the end of a line, the106filter may reset to default state for the next line.107 108An SGR control sequence is not interpreted inside any other markup element.109However, other markup elements may appear between SGR control sequences and the110color/bold state is expected to apply to the symbolic output that replaces the111markup element in the filter's output.112 113The accepted SGR control sequences all have the form ``"\033[%um"`` (expressed here114using C string syntax), where ``%u`` is one of these:115 116==== ============================ ===============================================117Code Effect Notes118==== ============================ ===============================================1190 Reset to default formatting.1201 Bold text Combines with color states, doesn't reset them.12130 Black foreground12231 Red foreground12332 Green foreground12433 Yellow foreground12534 Blue foreground12635 Magenta foreground12736 Cyan foreground12837 White foreground129==== ============================ ===============================================130 131Common markup element syntax132============================133 134All the markup elements share a common syntactic structure to facilitate simple135matching and parsing code. Each element has the form::136 137 {{{tag:fields}}}138 139``tag`` identifies one of the element types described below, and is always a140short alphabetic string that must be in lower case. The rest of the element141consists of one or more fields. Fields are separated by ``:`` and cannot contain142any ``:`` or ``}`` characters. How many fields must be or may be present and143what they contain is specified for each element type.144 145No markup elements or ANSI SGR control sequences are interpreted inside the146contents of a field.147 148Implementations must ignore markup fields after those expected; this allows149adding new fields to backwards-compatibly extend elements. Implementations need150not ignore them silently, but the element should behave otherwise as if the151fields were removed.152 153In the descriptions of each element type, ``printf``-style placeholders indicate154field contents:155 156``%s``157 A string of printable characters, not including ``:`` or ``}``.158 159``%p``160 An address value represented by ``0x`` followed by an even number of161 hexadecimal digits (using either lower-case or upper-case for ``A``–``F``).162 If the digits are all ``0`` then the ``0x`` prefix may be omitted. No more163 than 16 hexadecimal digits are expected to appear in a single value (64 bits).164 165``%u``166 A nonnegative decimal integer.167 168``%i``169 A nonnegative integer. The digits are hexadecimal if prefixed by ``0x``, octal170 if prefixed by ``0``, or decimal otherwise.171 172``%x``173 A sequence of an even number of hexadecimal digits (using either lower-case or174 upper-case for ``A``–``F``), with no ``0x`` prefix. This represents an175 arbitrary sequence of bytes, such as an ELF Build ID.176 177Presentation elements178=====================179 180These are elements that convey a specific program entity to be displayed in181human-readable symbolic form.182 183``{{{symbol:%s}}}``184 Here ``%s`` is the linkage name for a symbol or type. It may require185 demangling according to language ABI rules. Even for unmangled names, it's186 recommended that this markup element be used to identify a symbol name so that187 it can be presented distinctively.188 189 Examples::190 191 {{{symbol:_ZN7Mangled4NameEv}}}192 {{{symbol:foobar}}}193 194``{{{pc:%p}}}``, ``{{{pc:%p:ra}}}``, ``{{{pc:%p:pc}}}``195 196 Here ``%p`` is the memory address of a code location. It might be presented as a197 function name and source location. The second two forms distinguish the kind of198 code location, as described in detail for bt elements below.199 200 Examples::201 202 {{{pc:0x12345678}}}203 {{{pc:0xffffffff9abcdef0}}}204 205``{{{data:%p}}}``206 207 Here ``%p`` is the memory address of a data location. It might be presented as208 the name of a global variable at that location.209 210 Examples::211 212 {{{data:0x12345678}}}213 {{{data:0xffffffff9abcdef0}}}214 215``{{{bt:%u:%p}}}``, ``{{{bt:%u:%p:ra}}}``, ``{{{bt:%u:%p:pc}}}``216 217 This represents one frame in a backtrace. It usually appears on a line by218 itself (surrounded only by whitespace), in a sequence of such lines with219 ascending frame numbers. So the human-readable output might be formatted220 assuming that, such that it looks good for a sequence of bt elements each221 alone on its line with uniform indentation of each line. But it can appear222 anywhere, so the filter should not remove any non-whitespace text surrounding223 the element.224 225 Here ``%u`` is the frame number, which starts at zero for the location of the226 fault being identified, increments to one for the caller of frame zero's call227 frame, to two for the caller of frame one, etc. ``%p`` is the memory address228 of a code location.229 230 Code locations in a backtrace come from two distinct sources. Most backtrace231 frames describe a return address code location, i.e. the instruction232 immediately after a call instruction. This is the location of code that has233 yet to run, since the function called there has not yet returned. Hence the234 code location of actual interest is usually the call site itself rather than235 the return address, i.e. one instruction earlier. When presenting the source236 location for a return address frame, the symbolizing filter will subtract one237 byte or one instruction length from the actual return address for the call238 site, with the intent that the address logged can be translated directly to a239 source location for the call site and not for the apparent return site240 thereafter (which can be confusing). When inlined functions are involved, the241 call site and the return site can appear to be in different functions at242 entirely unrelated source locations rather than just a line away, making the243 confusion of showing the return site rather the call site quite severe.244 245 Often the first frame in a backtrace ("frame zero") identifies the precise246 code location of a fault, trap, or asynchronous interrupt rather than a return247 address. At other times, even the first frame is actually a return address248 (for example, backtraces collected at the time of an object allocation and249 reported later when the allocated object is used or misused). When a system250 supports in-thread trap handling, there may also be frames after the first251 that represent a precise interrupted code location rather than a return252 address, presented as the "caller" of a trap handler function (for example,253 signal handlers in POSIX systems).254 255 Return address frames are identified by the ``:ra`` suffix. Precise code256 location frames are identified by the ``:pc`` suffix.257 258 Traditional practice has often been to collect backtraces as simple address259 lists, losing the distinction between return address code locations and260 precise code locations. Some such code applies the "subtract one" adjustment261 described above to the address values before reporting them, and it's not262 always clear or consistent whether this adjustment has been applied or not.263 These ambiguous cases are supported by the ``bt`` and ``pc`` forms with no264 ``:ra`` or ``:pc`` suffix, which indicate it's unclear which sort of code265 location this is. However, it's highly recommended that all emitters use the266 suffixed forms and deliver address values with no adjustments applied. When267 traditional practice has been ambiguous, the majority of cases seem to have268 been of printing addresses that are return address code locations and printing269 them without adjustment. So the symbolizing filter will usually apply the270 "subtract one byte" adjustment to an address printed without a disambiguating271 suffix. Assuming that a call instruction is longer than one byte on all272 supported machines, applying the "subtract one byte" adjustment a second time273 still results in an address somewhere in the call instruction, so a little274 sloppiness here often does little or no harm.275 276 Examples::277 278 {{{bt:0:0x12345678:pc}}}279 {{{bt:1:0xffffffff9abcdef0:ra}}}280 281``{{{hexdict:...}}}`` [#not_yet_implemented]_282 283 This element can span multiple lines. Here ``...`` is a sequence of key-value284 pairs where a single ``:`` separates each key from its value, and arbitrary285 whitespace separates the pairs. The value (right-hand side) of each pair286 either is one or more ``0`` digits, or is ``0x`` followed by hexadecimal287 digits. Each value might be a memory address or might be some other integer288 (including an integer that looks like a likely memory address but actually has289 an unrelated purpose). When the contextual information about the memory layout290 suggests that a given value could be a code location or a global variable data291 address, it might be presented as a source location or variable name or with292 active UI that makes such interpretation optionally visible.293 294 The intended use is for things like register dumps, where the emitter doesn't295 know which values might have a symbolic interpretation but a presentation that296 makes plausible symbolic interpretations available might be very useful to297 someone reading the log. At the same time, a flat text presentation should298 usually avoid interfering too much with the original contents and formatting299 of the dump. For example, it might use footnotes with source locations for300 values that appear to be code locations. An active UI presentation might show301 the dump text as is, but highlight values with symbolic information available302 and pop up a presentation of symbolic details when a value is selected.303 304 Example::305 306 {{{hexdict:307 CS: 0 RIP: 0x6ee17076fb80 EFL: 0x10246 CR2: 0308 RAX: 0xc53d0acbcf0 RBX: 0x1e659ea7e0d0 RCX: 0 RDX: 0x6ee1708300cc309 RSI: 0 RDI: 0x6ee170830040 RBP: 0x3b13734898e0 RSP: 0x3b13734898d8310 R8: 0x3b1373489860 R9: 0x2776ff4f R10: 0x2749d3e9a940 R11: 0x246311 R12: 0x1e659ea7e0f0 R13: 0xd7231230fd6ff2e7 R14: 0x1e659ea7e108 R15: 0xc53d0acbcf0312 }}}313 314Trigger elements315================316 317These elements cause an external action and will be presented to the user in a318human-readable form. Generally they trigger an external action to occur that319results in a linkable page. The link or some other informative information about320the external action can then be presented to the user.321 322``{{{dumpfile:%s:%s}}}`` [#not_yet_implemented]_323 324 Here the first ``%s`` is an identifier for a type of dump and the second325 ``%s`` is an identifier for a particular dump that's just been published. The326 types of dumps, the exact meaning of "published", and the nature of the327 identifier are outside the scope of the markup format per se. In general it328 might correspond to writing a file by that name or something similar.329 330 This element may trigger additional post-processing work beyond symbolizing331 the markup. It indicates that a dump file of some sort has been published.332 Some logic attached to the symbolizing filter may understand certain types of333 dump file and trigger additional post-processing of the dump file upon334 encountering this element (e.g. generating visualizations, symbolization). The335 expectation is that the information collected from contextual elements336 (described below) in the logging stream may be necessary to decode the content337 of the dump. So if the symbolizing filter triggers other processing, it may338 need to feed some distilled form of the contextual information to those339 processes.340 341 An example of a type identifier is ``sancov``, for dumps from LLVM342 `SanitizerCoverage <https://clang.llvm.org/docs/SanitizerCoverage.html>`_.343 344 Example::345 346 {{{dumpfile:sancov:sancov.8675}}}347 348Contextual elements349===================350 351These are elements that supply information necessary to convert presentation352elements to symbolic form. Unlike presentation elements, they are not directly353related to the surrounding text. Contextual elements should appear alone on354lines with no other non-whitespace text, so that the symbolizing filter might355elide the whole line from its output without hiding any other log text.356 357The contextual elements themselves do not necessarily need to be presented in358human-readable output. However, the information they impart may be essential to359understanding the logging text even after symbolization. So it's recommended360that this information be preserved in some form when the original raw log with361markup may no longer be readily accessible for whatever reason.362 363Contextual elements should appear in the logging stream before they are needed.364That is, if some piece of context may affect how the symbolizing filter would365interpret or present a later presentation element, the necessary contextual366elements should have appeared somewhere earlier in the logging stream. It should367always be possible for the symbolizing filter to be implemented as a single pass368over the raw logging stream, accumulating context and massaging text as it goes.369 370``{{{reset}}}``371 372 This should be output before any other contextual element. The need for this373 contextual element is to support implementations that handle logs coming from374 multiple processes. Such implementations might not know when a new process375 starts or ends. Because some identifying information (like process IDs) might376 be the same between old and new processes, a way is needed to distinguish two377 processes with such identical identifying information. This element informs378 such implementations to reset the state of a filter so that information from a379 previous process's contextual elements is not assumed for new process that380 just happens have the same identifying information.381 382``{{{module:%i:%s:%s:...}}}``383 384 This element represents a so-called "module". A "module" is a single linked385 binary, such as a loaded ELF file. Usually each module occupies a contiguous386 range of memory.387 388 Here ``%i`` is the module ID which is used by other contextual elements to389 refer to this module. The first ``%s`` is a human-readable identifier for the390 module, such as an ELF ``DT_SONAME`` string or a file name; but it might be391 empty. It's only for casual information. Only the module ID is used to refer392 to this module in other contextual elements, never the ``%s`` string. The393 ``module`` element defining a module ID must always be emitted before any394 other elements that refer to that module ID, so that a filter never needs to395 keep track of dangling references. The second ``%s`` is the module type and it396 determines what the remaining fields are. The following module types are397 supported:398 399 * ``elf:%x``400 401 Here ``%x`` encodes an ELF Build ID. The Build ID should refer to a single402 linked binary. The Build ID string is the sole way to identify the binary from403 which this module was loaded.404 405 Example::406 407 {{{module:1:libc.so:elf:83238ab56ba10497}}}408 409``{{{mmap:%p:%i:...}}}``410 411 This contextual element is used to give information about a particular region412 in memory. ``%p`` is the starting address and ``%i`` gives the size in hex of the413 region of memory. The ``...`` part can take different forms to give different414 information about the specified region of memory. The allowed forms are the415 following:416 417 * ``load:%i:%s:%p``418 419 This subelement informs the filter that a segment was loaded from a module.420 The module is identified by its module ID ``%i``. The ``%s`` is one or more of421 the letters 'r', 'w', and 'x' (in that order and in either upper or lower422 case) to indicate this segment of memory is readable, writable, and/or423 executable. The symbolizing filter can use this information to guess whether424 an address is a likely code address or a likely data address in the given425 module. The remaining ``%p`` gives the module relative address. For ELF files426 the module relative address will be the ``p_vaddr`` of the associated program427 header. For example if your module's executable segment has428 ``p_vaddr=0x1000``, ``p_memsz=0x1234``, and was loaded at ``0x7acba69d5000``429 then you need to subtract ``0x7acba69d4000`` from any address between430 ``0x7acba69d5000`` and ``0x7acba69d6234`` to get the module relative address.431 The starting address will usually have been rounded down to the active page432 size, and the size rounded up.433 434 Example::435 436 {{{mmap:0x7acba69d5000:0x5a000:load:1:rx:0x1000}}}437 438.. rubric:: Footnotes439 440.. [#not_yet_implemented] This markup element is not yet implemented in441 :doc:`llvm-symbolizer <CommandGuide/llvm-symbolizer>`.442