1100 lines · plain
1=============2Event Tracing3=============4 5:Author: Theodore Ts'o6:Updated: Li Zefan and Tom Zanussi7 81. Introduction9===============10 11Tracepoints (see Documentation/trace/tracepoints.rst) can be used12without creating custom kernel modules to register probe functions13using the event tracing infrastructure.14 15Not all tracepoints can be traced using the event tracing system;16the kernel developer must provide code snippets which define how the17tracing information is saved into the tracing buffer, and how the18tracing information should be printed.19 202. Using Event Tracing21======================22 232.1 Via the 'set_event' interface24---------------------------------25 26The events which are available for tracing can be found in the file27/sys/kernel/tracing/available_events.28 29To enable a particular event, such as 'sched_wakeup', simply echo it30to /sys/kernel/tracing/set_event. For example::31 32 # echo sched_wakeup >> /sys/kernel/tracing/set_event33 34.. Note:: '>>' is necessary, otherwise it will firstly disable all the events.35 36To disable an event, echo the event name to the set_event file prefixed37with an exclamation point::38 39 # echo '!sched_wakeup' >> /sys/kernel/tracing/set_event40 41To disable all events, echo an empty line to the set_event file::42 43 # echo > /sys/kernel/tracing/set_event44 45To enable all events, echo ``*:*`` or ``*:`` to the set_event file::46 47 # echo *:* > /sys/kernel/tracing/set_event48 49The events are organized into subsystems, such as ext4, irq, sched,50etc., and a full event name looks like this: <subsystem>:<event>. The51subsystem name is optional, but it is displayed in the available_events52file. All of the events in a subsystem can be specified via the syntax53``<subsystem>:*``; for example, to enable all irq events, you can use the54command::55 56 # echo 'irq:*' > /sys/kernel/tracing/set_event57 582.2 Via the 'enable' toggle59---------------------------60 61The events available are also listed in /sys/kernel/tracing/events/ hierarchy62of directories.63 64To enable event 'sched_wakeup'::65 66 # echo 1 > /sys/kernel/tracing/events/sched/sched_wakeup/enable67 68To disable it::69 70 # echo 0 > /sys/kernel/tracing/events/sched/sched_wakeup/enable71 72To enable all events in sched subsystem::73 74 # echo 1 > /sys/kernel/tracing/events/sched/enable75 76To enable all events::77 78 # echo 1 > /sys/kernel/tracing/events/enable79 80When reading one of these enable files, there are four results:81 82 - 0 - all events this file affects are disabled83 - 1 - all events this file affects are enabled84 - X - there is a mixture of events enabled and disabled85 - ? - this file does not affect any event86 872.3 Boot option88---------------89 90In order to facilitate early boot debugging, use boot option::91 92 trace_event=[event-list]93 94event-list is a comma separated list of events. See section 2.1 for event95format.96 973. Defining an event-enabled tracepoint98=======================================99 100See The example provided in samples/trace_events101 1024. Event formats103================104 105Each trace event has a 'format' file associated with it that contains106a description of each field in a logged event. This information can107be used to parse the binary trace stream, and is also the place to108find the field names that can be used in event filters (see section 5).109 110It also displays the format string that will be used to print the111event in text mode, along with the event name and ID used for112profiling.113 114Every event has a set of ``common`` fields associated with it; these are115the fields prefixed with ``common_``. The other fields vary between116events and correspond to the fields defined in the TRACE_EVENT117definition for that event.118 119Each field in the format has the form::120 121 field:field-type field-name; offset:N; size:N;122 123where offset is the offset of the field in the trace record and size124is the size of the data item, in bytes.125 126For example, here's the information displayed for the 'sched_wakeup'127event::128 129 # cat /sys/kernel/tracing/events/sched/sched_wakeup/format130 131 name: sched_wakeup132 ID: 60133 format:134 field:unsigned short common_type; offset:0; size:2;135 field:unsigned char common_flags; offset:2; size:1;136 field:unsigned char common_preempt_count; offset:3; size:1;137 field:int common_pid; offset:4; size:4;138 field:int common_tgid; offset:8; size:4;139 140 field:char comm[TASK_COMM_LEN]; offset:12; size:16;141 field:pid_t pid; offset:28; size:4;142 field:int prio; offset:32; size:4;143 field:int success; offset:36; size:4;144 field:int cpu; offset:40; size:4;145 146 print fmt: "task %s:%d [%d] success=%d [%03d]", REC->comm, REC->pid,147 REC->prio, REC->success, REC->cpu148 149This event contains 10 fields, the first 5 common and the remaining 5150event-specific. All the fields for this event are numeric, except for151'comm' which is a string, a distinction important for event filtering.152 1535. Event filtering154==================155 156Trace events can be filtered in the kernel by associating boolean157'filter expressions' with them. As soon as an event is logged into158the trace buffer, its fields are checked against the filter expression159associated with that event type. An event with field values that160'match' the filter will appear in the trace output, and an event whose161values don't match will be discarded. An event with no filter162associated with it matches everything, and is the default when no163filter has been set for an event.164 1655.1 Expression syntax166---------------------167 168A filter expression consists of one or more 'predicates' that can be169combined using the logical operators '&&' and '||'. A predicate is170simply a clause that compares the value of a field contained within a171logged event with a constant value and returns either 0 or 1 depending172on whether the field value matched (1) or didn't match (0)::173 174 field-name relational-operator value175 176Parentheses can be used to provide arbitrary logical groupings and177double-quotes can be used to prevent the shell from interpreting178operators as shell metacharacters.179 180The field-names available for use in filters can be found in the181'format' files for trace events (see section 4).182 183The relational-operators depend on the type of the field being tested:184 185The operators available for numeric fields are:186 187==, !=, <, <=, >, >=, &188 189And for string fields they are:190 191==, !=, ~192 193The glob (~) accepts a wild card character (\*,?) and character classes194([). For example::195 196 prev_comm ~ "*sh"197 prev_comm ~ "sh*"198 prev_comm ~ "*sh*"199 prev_comm ~ "ba*sh"200 201If the field is a pointer that points into user space (for example202"filename" from sys_enter_openat), then you have to append ".ustring" to the203field name::204 205 filename.ustring ~ "password"206 207As the kernel will have to know how to retrieve the memory that the pointer208is at from user space.209 210You can convert any long type to a function address and search by function name::211 212 call_site.function == security_prepare_creds213 214The above will filter when the field "call_site" falls on the address within215"security_prepare_creds". That is, it will compare the value of "call_site" and216the filter will return true if it is greater than or equal to the start of217the function "security_prepare_creds" and less than the end of that function.218 219The ".function" postfix can only be attached to values of size long, and can only220be compared with "==" or "!=".221 222Cpumask fields or scalar fields that encode a CPU number can be filtered using223a user-provided cpumask in cpulist format. The format is as follows::224 225 CPUS{$cpulist}226 227Operators available to cpumask filtering are:228 229& (intersection), ==, !=230 231For example, this will filter events that have their .target_cpu field present232in the given cpumask::233 234 target_cpu & CPUS{17-42}235 2365.2 Setting filters237-------------------238 239A filter for an individual event is set by writing a filter expression240to the 'filter' file for the given event.241 242For example::243 244 # cd /sys/kernel/tracing/events/sched/sched_wakeup245 # echo "common_preempt_count > 4" > filter246 247A slightly more involved example::248 249 # cd /sys/kernel/tracing/events/signal/signal_generate250 # echo "((sig >= 10 && sig < 15) || sig == 17) && comm != bash" > filter251 252If there is an error in the expression, you'll get an 'Invalid253argument' error when setting it, and the erroneous string along with254an error message can be seen by looking at the filter e.g.::255 256 # cd /sys/kernel/tracing/events/signal/signal_generate257 # echo "((sig >= 10 && sig < 15) || dsig == 17) && comm != bash" > filter258 -bash: echo: write error: Invalid argument259 # cat filter260 ((sig >= 10 && sig < 15) || dsig == 17) && comm != bash261 ^262 parse_error: Field not found263 264Currently the caret ('^') for an error always appears at the beginning of265the filter string; the error message should still be useful though266even without more accurate position info.267 2685.2.1 Filter limitations269------------------------270 271If a filter is placed on a string pointer ``(char *)`` that does not point272to a string on the ring buffer, but instead points to kernel or user space273memory, then, for safety reasons, at most 1024 bytes of the content is274copied onto a temporary buffer to do the compare. If the copy of the memory275faults (the pointer points to memory that should not be accessed), then the276string compare will be treated as not matching.277 2785.3 Clearing filters279--------------------280 281To clear the filter for an event, write a '0' to the event's filter282file.283 284To clear the filters for all events in a subsystem, write a '0' to the285subsystem's filter file.286 2875.4 Subsystem filters288---------------------289 290For convenience, filters for every event in a subsystem can be set or291cleared as a group by writing a filter expression into the filter file292at the root of the subsystem. Note however, that if a filter for any293event within the subsystem lacks a field specified in the subsystem294filter, or if the filter can't be applied for any other reason, the295filter for that event will retain its previous setting. This can296result in an unintended mixture of filters which could lead to297confusing (to the user who might think different filters are in298effect) trace output. Only filters that reference just the common299fields can be guaranteed to propagate successfully to all events.300 301Here are a few subsystem filter examples that also illustrate the302above points:303 304Clear the filters on all events in the sched subsystem::305 306 # cd /sys/kernel/tracing/events/sched307 # echo 0 > filter308 # cat sched_switch/filter309 none310 # cat sched_wakeup/filter311 none312 313Set a filter using only common fields for all events in the sched314subsystem (all events end up with the same filter)::315 316 # cd /sys/kernel/tracing/events/sched317 # echo common_pid == 0 > filter318 # cat sched_switch/filter319 common_pid == 0320 # cat sched_wakeup/filter321 common_pid == 0322 323Attempt to set a filter using a non-common field for all events in the324sched subsystem (all events but those that have a prev_pid field retain325their old filters)::326 327 # cd /sys/kernel/tracing/events/sched328 # echo prev_pid == 0 > filter329 # cat sched_switch/filter330 prev_pid == 0331 # cat sched_wakeup/filter332 common_pid == 0333 3345.5 PID filtering335-----------------336 337The set_event_pid file in the same directory as the top events directory338exists, will filter all events from tracing any task that does not have the339PID listed in the set_event_pid file.340::341 342 # cd /sys/kernel/tracing343 # echo $$ > set_event_pid344 # echo 1 > events/enable345 346Will only trace events for the current task.347 348To add more PIDs without losing the PIDs already included, use '>>'.349::350 351 # echo 123 244 1 >> set_event_pid352 353 3546. Event triggers355=================356 357Trace events can be made to conditionally invoke trigger 'commands'358which can take various forms and are described in detail below;359examples would be enabling or disabling other trace events or invoking360a stack trace whenever the trace event is hit. Whenever a trace event361with attached triggers is invoked, the set of trigger commands362associated with that event is invoked. Any given trigger can363additionally have an event filter of the same form as described in364section 5 (Event filtering) associated with it - the command will only365be invoked if the event being invoked passes the associated filter.366If no filter is associated with the trigger, it always passes.367 368Triggers are added to and removed from a particular event by writing369trigger expressions to the 'trigger' file for the given event.370 371A given event can have any number of triggers associated with it,372subject to any restrictions that individual commands may have in that373regard.374 375Event triggers are implemented on top of "soft" mode, which means that376whenever a trace event has one or more triggers associated with it,377the event is activated even if it isn't actually enabled, but is378disabled in a "soft" mode. That is, the tracepoint will be called,379but just will not be traced, unless of course it's actually enabled.380This scheme allows triggers to be invoked even for events that aren't381enabled, and also allows the current event filter implementation to be382used for conditionally invoking triggers.383 384The syntax for event triggers is roughly based on the syntax for385set_ftrace_filter 'ftrace filter commands' (see the 'Filter commands'386section of Documentation/trace/ftrace.rst), but there are major387differences and the implementation isn't currently tied to it in any388way, so beware about making generalizations between the two.389 390.. Note::391 Writing into trace_marker (See Documentation/trace/ftrace.rst)392 can also enable triggers that are written into393 /sys/kernel/tracing/events/ftrace/print/trigger394 3956.1 Expression syntax396---------------------397 398Triggers are added by echoing the command to the 'trigger' file::399 400 # echo 'command[:count] [if filter]' > trigger401 402Triggers are removed by echoing the same command but starting with '!'403to the 'trigger' file::404 405 # echo '!command[:count] [if filter]' > trigger406 407The [if filter] part isn't used in matching commands when removing, so408leaving that off in a '!' command will accomplish the same thing as409having it in.410 411The filter syntax is the same as that described in the 'Event412filtering' section above.413 414For ease of use, writing to the trigger file using '>' currently just415adds or removes a single trigger and there's no explicit '>>' support416('>' actually behaves like '>>') or truncation support to remove all417triggers (you have to use '!' for each one added.)418 4196.2 Supported trigger commands420------------------------------421 422The following commands are supported:423 424- enable_event/disable_event425 426 These commands can enable or disable another trace event whenever427 the triggering event is hit. When these commands are registered,428 the other trace event is activated, but disabled in a "soft" mode.429 That is, the tracepoint will be called, but just will not be traced.430 The event tracepoint stays in this mode as long as there's a trigger431 in effect that can trigger it.432 433 For example, the following trigger causes kmalloc events to be434 traced when a read system call is entered, and the :1 at the end435 specifies that this enablement happens only once::436 437 # echo 'enable_event:kmem:kmalloc:1' > \438 /sys/kernel/tracing/events/syscalls/sys_enter_read/trigger439 440 The following trigger causes kmalloc events to stop being traced441 when a read system call exits. This disablement happens on every442 read system call exit::443 444 # echo 'disable_event:kmem:kmalloc' > \445 /sys/kernel/tracing/events/syscalls/sys_exit_read/trigger446 447 The format is::448 449 enable_event:<system>:<event>[:count]450 disable_event:<system>:<event>[:count]451 452 To remove the above commands::453 454 # echo '!enable_event:kmem:kmalloc:1' > \455 /sys/kernel/tracing/events/syscalls/sys_enter_read/trigger456 457 # echo '!disable_event:kmem:kmalloc' > \458 /sys/kernel/tracing/events/syscalls/sys_exit_read/trigger459 460 Note that there can be any number of enable/disable_event triggers461 per triggering event, but there can only be one trigger per462 triggered event. e.g. sys_enter_read can have triggers enabling both463 kmem:kmalloc and sched:sched_switch, but can't have two kmem:kmalloc464 versions such as kmem:kmalloc and kmem:kmalloc:1 or 'kmem:kmalloc if465 bytes_req == 256' and 'kmem:kmalloc if bytes_alloc == 256' (they466 could be combined into a single filter on kmem:kmalloc though).467 468- stacktrace469 470 This command dumps a stacktrace in the trace buffer whenever the471 triggering event occurs.472 473 For example, the following trigger dumps a stacktrace every time the474 kmalloc tracepoint is hit::475 476 # echo 'stacktrace' > \477 /sys/kernel/tracing/events/kmem/kmalloc/trigger478 479 The following trigger dumps a stacktrace the first 5 times a kmalloc480 request happens with a size >= 64K::481 482 # echo 'stacktrace:5 if bytes_req >= 65536' > \483 /sys/kernel/tracing/events/kmem/kmalloc/trigger484 485 The format is::486 487 stacktrace[:count]488 489 To remove the above commands::490 491 # echo '!stacktrace' > \492 /sys/kernel/tracing/events/kmem/kmalloc/trigger493 494 # echo '!stacktrace:5 if bytes_req >= 65536' > \495 /sys/kernel/tracing/events/kmem/kmalloc/trigger496 497 The latter can also be removed more simply by the following (without498 the filter)::499 500 # echo '!stacktrace:5' > \501 /sys/kernel/tracing/events/kmem/kmalloc/trigger502 503 Note that there can be only one stacktrace trigger per triggering504 event.505 506- snapshot507 508 This command causes a snapshot to be triggered whenever the509 triggering event occurs.510 511 The following command creates a snapshot every time a block request512 queue is unplugged with a depth > 1. If you were tracing a set of513 events or functions at the time, the snapshot trace buffer would514 capture those events when the trigger event occurred::515 516 # echo 'snapshot if nr_rq > 1' > \517 /sys/kernel/tracing/events/block/block_unplug/trigger518 519 To only snapshot once::520 521 # echo 'snapshot:1 if nr_rq > 1' > \522 /sys/kernel/tracing/events/block/block_unplug/trigger523 524 To remove the above commands::525 526 # echo '!snapshot if nr_rq > 1' > \527 /sys/kernel/tracing/events/block/block_unplug/trigger528 529 # echo '!snapshot:1 if nr_rq > 1' > \530 /sys/kernel/tracing/events/block/block_unplug/trigger531 532 Note that there can be only one snapshot trigger per triggering533 event.534 535- traceon/traceoff536 537 These commands turn tracing on and off when the specified events are538 hit. The parameter determines how many times the tracing system is539 turned on and off. If unspecified, there is no limit.540 541 The following command turns tracing off the first time a block542 request queue is unplugged with a depth > 1. If you were tracing a543 set of events or functions at the time, you could then examine the544 trace buffer to see the sequence of events that led up to the545 trigger event::546 547 # echo 'traceoff:1 if nr_rq > 1' > \548 /sys/kernel/tracing/events/block/block_unplug/trigger549 550 To always disable tracing when nr_rq > 1::551 552 # echo 'traceoff if nr_rq > 1' > \553 /sys/kernel/tracing/events/block/block_unplug/trigger554 555 To remove the above commands::556 557 # echo '!traceoff:1 if nr_rq > 1' > \558 /sys/kernel/tracing/events/block/block_unplug/trigger559 560 # echo '!traceoff if nr_rq > 1' > \561 /sys/kernel/tracing/events/block/block_unplug/trigger562 563 Note that there can be only one traceon or traceoff trigger per564 triggering event.565 566- hist567 568 This command aggregates event hits into a hash table keyed on one or569 more trace event format fields (or stacktrace) and a set of running570 totals derived from one or more trace event format fields and/or571 event counts (hitcount).572 573 See Documentation/trace/histogram.rst for details and examples.574 5757. In-kernel trace event API576============================577 578In most cases, the command-line interface to trace events is more than579sufficient. Sometimes, however, applications might find the need for580more complex relationships than can be expressed through a simple581series of linked command-line expressions, or putting together sets of582commands may be simply too cumbersome. An example might be an583application that needs to 'listen' to the trace stream in order to584maintain an in-kernel state machine detecting, for instance, when an585illegal kernel state occurs in the scheduler.586 587The trace event subsystem provides an in-kernel API allowing modules588or other kernel code to generate user-defined 'synthetic' events at589will, which can be used to either augment the existing trace stream590and/or signal that a particular important state has occurred.591 592A similar in-kernel API is also available for creating kprobe and593kretprobe events.594 595Both the synthetic event and k/ret/probe event APIs are built on top596of a lower-level "dynevent_cmd" event command API, which is also597available for more specialized applications, or as the basis of other598higher-level trace event APIs.599 600The API provided for these purposes is describe below and allows the601following:602 603 - dynamically creating synthetic event definitions604 - dynamically creating kprobe and kretprobe event definitions605 - tracing synthetic events from in-kernel code606 - the low-level "dynevent_cmd" API607 6087.1 Dyamically creating synthetic event definitions609---------------------------------------------------610 611There are a couple ways to create a new synthetic event from a kernel612module or other kernel code.613 614The first creates the event in one step, using synth_event_create().615In this method, the name of the event to create and an array defining616the fields is supplied to synth_event_create(). If successful, a617synthetic event with that name and fields will exist following that618call. For example, to create a new "schedtest" synthetic event::619 620 ret = synth_event_create("schedtest", sched_fields,621 ARRAY_SIZE(sched_fields), THIS_MODULE);622 623The sched_fields param in this example points to an array of struct624synth_field_desc, each of which describes an event field by type and625name::626 627 static struct synth_field_desc sched_fields[] = {628 { .type = "pid_t", .name = "next_pid_field" },629 { .type = "char[16]", .name = "next_comm_field" },630 { .type = "u64", .name = "ts_ns" },631 { .type = "u64", .name = "ts_ms" },632 { .type = "unsigned int", .name = "cpu" },633 { .type = "char[64]", .name = "my_string_field" },634 { .type = "int", .name = "my_int_field" },635 };636 637See synth_field_size() for available types.638 639If field_name contains [n], the field is considered to be a static array.640 641If field_names contains[] (no subscript), the field is considered to642be a dynamic array, which will only take as much space in the event as643is required to hold the array.644 645Because space for an event is reserved before assigning field values646to the event, using dynamic arrays implies that the piecewise647in-kernel API described below can't be used with dynamic arrays. The648other non-piecewise in-kernel APIs can, however, be used with dynamic649arrays.650 651If the event is created from within a module, a pointer to the module652must be passed to synth_event_create(). This will ensure that the653trace buffer won't contain unreadable events when the module is654removed.655 656At this point, the event object is ready to be used for generating new657events.658 659In the second method, the event is created in several steps. This660allows events to be created dynamically and without the need to create661and populate an array of fields beforehand.662 663To use this method, an empty or partially empty synthetic event should664first be created using synth_event_gen_cmd_start() or665synth_event_gen_cmd_array_start(). For synth_event_gen_cmd_start(),666the name of the event along with one or more pairs of args each pair667representing a 'type field_name;' field specification should be668supplied. For synth_event_gen_cmd_array_start(), the name of the669event along with an array of struct synth_field_desc should be670supplied. Before calling synth_event_gen_cmd_start() or671synth_event_gen_cmd_array_start(), the user should create and672initialize a dynevent_cmd object using synth_event_cmd_init().673 674For example, to create a new "schedtest" synthetic event with two675fields::676 677 struct dynevent_cmd cmd;678 char *buf;679 680 /* Create a buffer to hold the generated command */681 buf = kzalloc(MAX_DYNEVENT_CMD_LEN, GFP_KERNEL);682 683 /* Before generating the command, initialize the cmd object */684 synth_event_cmd_init(&cmd, buf, MAX_DYNEVENT_CMD_LEN);685 686 ret = synth_event_gen_cmd_start(&cmd, "schedtest", THIS_MODULE,687 "pid_t", "next_pid_field",688 "u64", "ts_ns");689 690Alternatively, using an array of struct synth_field_desc fields691containing the same information::692 693 ret = synth_event_gen_cmd_array_start(&cmd, "schedtest", THIS_MODULE,694 fields, n_fields);695 696Once the synthetic event object has been created, it can then be697populated with more fields. Fields are added one by one using698synth_event_add_field(), supplying the dynevent_cmd object, a field699type, and a field name. For example, to add a new int field named700"intfield", the following call should be made::701 702 ret = synth_event_add_field(&cmd, "int", "intfield");703 704See synth_field_size() for available types. If field_name contains [n]705the field is considered to be an array.706 707A group of fields can also be added all at once using an array of708synth_field_desc with add_synth_fields(). For example, this would add709just the first four sched_fields::710 711 ret = synth_event_add_fields(&cmd, sched_fields, 4);712 713If you already have a string of the form 'type field_name',714synth_event_add_field_str() can be used to add it as-is; it will715also automatically append a ';' to the string.716 717Once all the fields have been added, the event should be finalized and718registered by calling the synth_event_gen_cmd_end() function::719 720 ret = synth_event_gen_cmd_end(&cmd);721 722At this point, the event object is ready to be used for tracing new723events.724 7257.2 Tracing synthetic events from in-kernel code726------------------------------------------------727 728To trace a synthetic event, there are several options. The first729option is to trace the event in one call, using synth_event_trace()730with a variable number of values, or synth_event_trace_array() with an731array of values to be set. A second option can be used to avoid the732need for a pre-formed array of values or list of arguments, via733synth_event_trace_start() and synth_event_trace_end() along with734synth_event_add_next_val() or synth_event_add_val() to add the values735piecewise.736 7377.2.1 Tracing a synthetic event all at once738-------------------------------------------739 740To trace a synthetic event all at once, the synth_event_trace() or741synth_event_trace_array() functions can be used.742 743The synth_event_trace() function is passed the trace_event_file744representing the synthetic event (which can be retrieved using745trace_get_event_file() using the synthetic event name, "synthetic" as746the system name, and the trace instance name (NULL if using the global747trace array)), along with an variable number of u64 args, one for each748synthetic event field, and the number of values being passed.749 750So, to trace an event corresponding to the synthetic event definition751above, code like the following could be used::752 753 ret = synth_event_trace(create_synth_test, 7, /* number of values */754 444, /* next_pid_field */755 (u64)"clackers", /* next_comm_field */756 1000000, /* ts_ns */757 1000, /* ts_ms */758 smp_processor_id(),/* cpu */759 (u64)"Thneed", /* my_string_field */760 999); /* my_int_field */761 762All vals should be cast to u64, and string vals are just pointers to763strings, cast to u64. Strings will be copied into space reserved in764the event for the string, using these pointers.765 766Alternatively, the synth_event_trace_array() function can be used to767accomplish the same thing. It is passed the trace_event_file768representing the synthetic event (which can be retrieved using769trace_get_event_file() using the synthetic event name, "synthetic" as770the system name, and the trace instance name (NULL if using the global771trace array)), along with an array of u64, one for each synthetic772event field.773 774To trace an event corresponding to the synthetic event definition775above, code like the following could be used::776 777 u64 vals[7];778 779 vals[0] = 777; /* next_pid_field */780 vals[1] = (u64)"tiddlywinks"; /* next_comm_field */781 vals[2] = 1000000; /* ts_ns */782 vals[3] = 1000; /* ts_ms */783 vals[4] = smp_processor_id(); /* cpu */784 vals[5] = (u64)"thneed"; /* my_string_field */785 vals[6] = 398; /* my_int_field */786 787The 'vals' array is just an array of u64, the number of which must788match the number of field in the synthetic event, and which must be in789the same order as the synthetic event fields.790 791All vals should be cast to u64, and string vals are just pointers to792strings, cast to u64. Strings will be copied into space reserved in793the event for the string, using these pointers.794 795In order to trace a synthetic event, a pointer to the trace event file796is needed. The trace_get_event_file() function can be used to get797it - it will find the file in the given trace instance (in this case798NULL since the top trace array is being used) while at the same time799preventing the instance containing it from going away::800 801 schedtest_event_file = trace_get_event_file(NULL, "synthetic",802 "schedtest");803 804Before tracing the event, it should be enabled in some way, otherwise805the synthetic event won't actually show up in the trace buffer.806 807To enable a synthetic event from the kernel, trace_array_set_clr_event()808can be used (which is not specific to synthetic events, so does need809the "synthetic" system name to be specified explicitly).810 811To enable the event, pass 'true' to it::812 813 trace_array_set_clr_event(schedtest_event_file->tr,814 "synthetic", "schedtest", true);815 816To disable it pass false::817 818 trace_array_set_clr_event(schedtest_event_file->tr,819 "synthetic", "schedtest", false);820 821Finally, synth_event_trace_array() can be used to actually trace the822event, which should be visible in the trace buffer afterwards::823 824 ret = synth_event_trace_array(schedtest_event_file, vals,825 ARRAY_SIZE(vals));826 827To remove the synthetic event, the event should be disabled, and the828trace instance should be 'put' back using trace_put_event_file()::829 830 trace_array_set_clr_event(schedtest_event_file->tr,831 "synthetic", "schedtest", false);832 trace_put_event_file(schedtest_event_file);833 834If those have been successful, synth_event_delete() can be called to835remove the event::836 837 ret = synth_event_delete("schedtest");838 8397.2.2 Tracing a synthetic event piecewise840-----------------------------------------841 842To trace a synthetic using the piecewise method described above, the843synth_event_trace_start() function is used to 'open' the synthetic844event trace::845 846 struct synth_event_trace_state trace_state;847 848 ret = synth_event_trace_start(schedtest_event_file, &trace_state);849 850It's passed the trace_event_file representing the synthetic event851using the same methods as described above, along with a pointer to a852struct synth_event_trace_state object, which will be zeroed before use and853used to maintain state between this and following calls.854 855Once the event has been opened, which means space for it has been856reserved in the trace buffer, the individual fields can be set. There857are two ways to do that, either one after another for each field in858the event, which requires no lookups, or by name, which does. The859tradeoff is flexibility in doing the assignments vs the cost of a860lookup per field.861 862To assign the values one after the other without lookups,863synth_event_add_next_val() should be used. Each call is passed the864same synth_event_trace_state object used in the synth_event_trace_start(),865along with the value to set the next field in the event. After each866field is set, the 'cursor' points to the next field, which will be set867by the subsequent call, continuing until all the fields have been set868in order. The same sequence of calls as in the above examples using869this method would be (without error-handling code)::870 871 /* next_pid_field */872 ret = synth_event_add_next_val(777, &trace_state);873 874 /* next_comm_field */875 ret = synth_event_add_next_val((u64)"slinky", &trace_state);876 877 /* ts_ns */878 ret = synth_event_add_next_val(1000000, &trace_state);879 880 /* ts_ms */881 ret = synth_event_add_next_val(1000, &trace_state);882 883 /* cpu */884 ret = synth_event_add_next_val(smp_processor_id(), &trace_state);885 886 /* my_string_field */887 ret = synth_event_add_next_val((u64)"thneed_2.01", &trace_state);888 889 /* my_int_field */890 ret = synth_event_add_next_val(395, &trace_state);891 892To assign the values in any order, synth_event_add_val() should be893used. Each call is passed the same synth_event_trace_state object used in894the synth_event_trace_start(), along with the field name of the field895to set and the value to set it to. The same sequence of calls as in896the above examples using this method would be (without error-handling897code)::898 899 ret = synth_event_add_val("next_pid_field", 777, &trace_state);900 ret = synth_event_add_val("next_comm_field", (u64)"silly putty",901 &trace_state);902 ret = synth_event_add_val("ts_ns", 1000000, &trace_state);903 ret = synth_event_add_val("ts_ms", 1000, &trace_state);904 ret = synth_event_add_val("cpu", smp_processor_id(), &trace_state);905 ret = synth_event_add_val("my_string_field", (u64)"thneed_9",906 &trace_state);907 ret = synth_event_add_val("my_int_field", 3999, &trace_state);908 909Note that synth_event_add_next_val() and synth_event_add_val() are910incompatible if used within the same trace of an event - either one911can be used but not both at the same time.912 913Finally, the event won't be actually traced until it's 'closed',914which is done using synth_event_trace_end(), which takes only the915struct synth_event_trace_state object used in the previous calls::916 917 ret = synth_event_trace_end(&trace_state);918 919Note that synth_event_trace_end() must be called at the end regardless920of whether any of the add calls failed (say due to a bad field name921being passed in).922 9237.3 Dyamically creating kprobe and kretprobe event definitions924--------------------------------------------------------------925 926To create a kprobe or kretprobe trace event from kernel code, the927kprobe_event_gen_cmd_start() or kretprobe_event_gen_cmd_start()928functions can be used.929 930To create a kprobe event, an empty or partially empty kprobe event931should first be created using kprobe_event_gen_cmd_start(). The name932of the event and the probe location should be specified along with one933or args each representing a probe field should be supplied to this934function. Before calling kprobe_event_gen_cmd_start(), the user935should create and initialize a dynevent_cmd object using936kprobe_event_cmd_init().937 938For example, to create a new "schedtest" kprobe event with two fields::939 940 struct dynevent_cmd cmd;941 char *buf;942 943 /* Create a buffer to hold the generated command */944 buf = kzalloc(MAX_DYNEVENT_CMD_LEN, GFP_KERNEL);945 946 /* Before generating the command, initialize the cmd object */947 kprobe_event_cmd_init(&cmd, buf, MAX_DYNEVENT_CMD_LEN);948 949 /*950 * Define the gen_kprobe_test event with the first 2 kprobe951 * fields.952 */953 ret = kprobe_event_gen_cmd_start(&cmd, "gen_kprobe_test", "do_sys_open",954 "dfd=%ax", "filename=%dx");955 956Once the kprobe event object has been created, it can then be957populated with more fields. Fields can be added using958kprobe_event_add_fields(), supplying the dynevent_cmd object along959with a variable arg list of probe fields. For example, to add a960couple additional fields, the following call could be made::961 962 ret = kprobe_event_add_fields(&cmd, "flags=%cx", "mode=+4($stack)");963 964Once all the fields have been added, the event should be finalized and965registered by calling the kprobe_event_gen_cmd_end() or966kretprobe_event_gen_cmd_end() functions, depending on whether a kprobe967or kretprobe command was started::968 969 ret = kprobe_event_gen_cmd_end(&cmd);970 971or::972 973 ret = kretprobe_event_gen_cmd_end(&cmd);974 975At this point, the event object is ready to be used for tracing new976events.977 978Similarly, a kretprobe event can be created using979kretprobe_event_gen_cmd_start() with a probe name and location and980additional params such as $retval::981 982 ret = kretprobe_event_gen_cmd_start(&cmd, "gen_kretprobe_test",983 "do_sys_open", "$retval");984 985Similar to the synthetic event case, code like the following can be986used to enable the newly created kprobe event::987 988 gen_kprobe_test = trace_get_event_file(NULL, "kprobes", "gen_kprobe_test");989 990 ret = trace_array_set_clr_event(gen_kprobe_test->tr,991 "kprobes", "gen_kprobe_test", true);992 993Finally, also similar to synthetic events, the following code can be994used to give the kprobe event file back and delete the event::995 996 trace_put_event_file(gen_kprobe_test);997 998 ret = kprobe_event_delete("gen_kprobe_test");999 10007.4 The "dynevent_cmd" low-level API1001------------------------------------1002 1003Both the in-kernel synthetic event and kprobe interfaces are built on1004top of a lower-level "dynevent_cmd" interface. This interface is1005meant to provide the basis for higher-level interfaces such as the1006synthetic and kprobe interfaces, which can be used as examples.1007 1008The basic idea is simple and amounts to providing a general-purpose1009layer that can be used to generate trace event commands. The1010generated command strings can then be passed to the command-parsing1011and event creation code that already exists in the trace event1012subsystem for creating the corresponding trace events.1013 1014In a nutshell, the way it works is that the higher-level interface1015code creates a struct dynevent_cmd object, then uses a couple1016functions, dynevent_arg_add() and dynevent_arg_pair_add() to build up1017a command string, which finally causes the command to be executed1018using the dynevent_create() function. The details of the interface1019are described below.1020 1021The first step in building a new command string is to create and1022initialize an instance of a dynevent_cmd. Here, for instance, we1023create a dynevent_cmd on the stack and initialize it::1024 1025 struct dynevent_cmd cmd;1026 char *buf;1027 int ret;1028 1029 buf = kzalloc(MAX_DYNEVENT_CMD_LEN, GFP_KERNEL);1030 1031 dynevent_cmd_init(cmd, buf, maxlen, DYNEVENT_TYPE_FOO,1032 foo_event_run_command);1033 1034The dynevent_cmd initialization needs to be given a user-specified1035buffer and the length of the buffer (MAX_DYNEVENT_CMD_LEN can be used1036for this purpose - at 2k it's generally too big to be comfortably put1037on the stack, so is dynamically allocated), a dynevent type id, which1038is meant to be used to check that further API calls are for the1039correct command type, and a pointer to an event-specific run_command()1040callback that will be called to actually execute the event-specific1041command function.1042 1043Once that's done, the command string can by built up by successive1044calls to argument-adding functions.1045 1046To add a single argument, define and initialize a struct dynevent_arg1047or struct dynevent_arg_pair object. Here's an example of the simplest1048possible arg addition, which is simply to append the given string as1049a whitespace-separated argument to the command::1050 1051 struct dynevent_arg arg;1052 1053 dynevent_arg_init(&arg, NULL, 0);1054 1055 arg.str = name;1056 1057 ret = dynevent_arg_add(cmd, &arg);1058 1059The arg object is first initialized using dynevent_arg_init() and in1060this case the parameters are NULL or 0, which means there's no1061optional sanity-checking function or separator appended to the end of1062the arg.1063 1064Here's another more complicated example using an 'arg pair', which is1065used to create an argument that consists of a couple components added1066together as a unit, for example, a 'type field_name;' arg or a simple1067expression arg e.g. 'flags=%cx'::1068 1069 struct dynevent_arg_pair arg_pair;1070 1071 dynevent_arg_pair_init(&arg_pair, dynevent_foo_check_arg_fn, 0, ';');1072 1073 arg_pair.lhs = type;1074 arg_pair.rhs = name;1075 1076 ret = dynevent_arg_pair_add(cmd, &arg_pair);1077 1078Again, the arg_pair is first initialized, in this case with a callback1079function used to check the sanity of the args (for example, that1080neither part of the pair is NULL), along with a character to be used1081to add an operator between the pair (here none) and a separator to be1082appended onto the end of the arg pair (here ';').1083 1084There's also a dynevent_str_add() function that can be used to simply1085add a string as-is, with no spaces, delimiters, or arg check.1086 1087Any number of dynevent_*_add() calls can be made to build up the string1088(until its length surpasses cmd->maxlen). When all the arguments have1089been added and the command string is complete, the only thing left to1090do is run the command, which happens by simply calling1091dynevent_create()::1092 1093 ret = dynevent_create(&cmd);1094 1095At that point, if the return value is 0, the dynamic event has been1096created and is ready to use.1097 1098See the dynevent_cmd function definitions themselves for the details1099of the API.1100