607 lines · plain
1.. SPDX-License-Identifier: (GPL-2.0+ OR CC-BY-4.0)2 3======================================================4Discovering Linux kernel subsystems used by a workload5======================================================6 7:Authors: - Shuah Khan <skhan@linuxfoundation.org>8 - Shefali Sharma <sshefali021@gmail.com>9:maintained-by: Shuah Khan <skhan@linuxfoundation.org>10 11Key Points12==========13 14 * Understanding system resources necessary to build and run a workload15 is important.16 * Linux tracing and strace can be used to discover the system resources17 in use by a workload. The completeness of the system usage information18 depends on the completeness of coverage of a workload.19 * Performance and security of the operating system can be analyzed with20 the help of tools such as:21 `perf <https://man7.org/linux/man-pages/man1/perf.1.html>`_,22 `stress-ng <https://www.mankier.com/1/stress-ng>`_,23 `paxtest <https://github.com/opntr/paxtest-freebsd>`_.24 * Once we discover and understand the workload needs, we can focus on them25 to avoid regressions and use it to evaluate safety considerations.26 27Methodology28===========29 30`strace <https://man7.org/linux/man-pages/man1/strace.1.html>`_ is a31diagnostic, instructional, and debugging tool and can be used to discover32the system resources in use by a workload. Once we discover and understand33the workload needs, we can focus on them to avoid regressions and use it34to evaluate safety considerations. We use strace tool to trace workloads.35 36This method of tracing using strace tells us the system calls invoked by37the workload and doesn't include all the system calls that can be invoked38by it. In addition, this tracing method tells us just the code paths within39these system calls that are invoked. As an example, if a workload opens a40file and reads from it successfully, then the success path is the one that41is traced. Any error paths in that system call will not be traced. If there42is a workload that provides full coverage of a workload then the method43outlined here will trace and find all possible code paths. The completeness44of the system usage information depends on the completeness of coverage of a45workload.46 47The goal is tracing a workload on a system running a default kernel without48requiring custom kernel installs.49 50How do we gather fine-grained system information?51=================================================52 53strace tool can be used to trace system calls made by a process and signals54it receives. System calls are the fundamental interface between an55application and the operating system kernel. They enable a program to56request services from the kernel. For instance, the open() system call in57Linux is used to provide access to a file in the file system. strace enables58us to track all the system calls made by an application. It lists all the59system calls made by a process and their resulting output.60 61You can generate profiling data combining strace and perf record tools to62record the events and information associated with a process. This provides63insight into the process. "perf annotate" tool generates the statistics of64each instruction of the program. This document goes over the details of how65to gather fine-grained information on a workload's usage of system resources.66 67We used strace to trace the perf, stress-ng, paxtest workloads to illustrate68our methodology to discover resources used by a workload. This process can69be applied to trace other workloads.70 71Getting the system ready for tracing72====================================73 74Before we can get started we will show you how to get your system ready.75We assume that you have a Linux distribution running on a physical system76or a virtual machine. Most distributions will include strace command. Let’s77install other tools that aren’t usually included to build Linux kernel.78Please note that the following works on Debian based distributions. You79might have to find equivalent packages on other Linux distributions.80 81Install tools to build Linux kernel and tools in kernel repository.82scripts/ver_linux is a good way to check if your system already has83the necessary tools::84 85 sudo apt-get build-essentials flex bison yacc86 sudo apt install libelf-dev systemtap-sdt-dev libaudit-dev libslang2-dev libperl-dev libdw-dev87 88cscope is a good tool to browse kernel sources. Let's install it now::89 90 sudo apt-get install cscope91 92Install stress-ng and paxtest::93 94 apt-get install stress-ng95 apt-get install paxtest96 97Workload overview98=================99 100As mentioned earlier, we used strace to trace perf bench, stress-ng and101paxtest workloads to show how to analyze a workload and identify Linux102subsystems used by these workloads. Let's start with an overview of these103three workloads to get a better understanding of what they do and how to104use them.105 106perf bench (all) workload107-------------------------108 109The perf bench command contains multiple multi-threaded microkernel110benchmarks for executing different subsystems in the Linux kernel and111system calls. This allows us to easily measure the impact of changes,112which can help mitigate performance regressions. It also acts as a common113benchmarking framework, enabling developers to easily create test cases,114integrate transparently, and use performance-rich tooling subsystems.115 116Stress-ng netdev stressor workload117----------------------------------118 119stress-ng is used for performing stress testing on the kernel. It allows120you to exercise various physical subsystems of the computer, as well as121interfaces of the OS kernel, using "stressor-s". They are available for122CPU, CPU cache, devices, I/O, interrupts, file system, memory, network,123operating system, pipelines, schedulers, and virtual machines. Please refer124to the `stress-ng man-page <https://www.mankier.com/1/stress-ng>`_ to125find the description of all the available stressor-s. The netdev stressor126starts specified number (N) of workers that exercise various netdevice127ioctl commands across all the available network devices.128 129paxtest kiddie workload130-----------------------131 132paxtest is a program that tests buffer overflows in the kernel. It tests133kernel enforcements over memory usage. Generally, execution in some memory134segments makes buffer overflows possible. It runs a set of programs that135attempt to subvert memory usage. It is used as a regression test suite for136PaX, but might be useful to test other memory protection patches for the137kernel. We used paxtest kiddie mode which looks for simple vulnerabilities.138 139What is strace and how do we use it?140====================================141 142As mentioned earlier, strace which is a useful diagnostic, instructional,143and debugging tool and can be used to discover the system resources in use144by a workload. It can be used:145 146 * To see how a process interacts with the kernel.147 * To see why a process is failing or hanging.148 * For reverse engineering a process.149 * To find the files on which a program depends.150 * For analyzing the performance of an application.151 * For troubleshooting various problems related to the operating system.152 153In addition, strace can generate run-time statistics on times, calls, and154errors for each system call and report a summary when program exits,155suppressing the regular output. This attempts to show system time (CPU time156spent running in the kernel) independent of wall clock time. We plan to use157these features to get information on workload system usage.158 159strace command supports basic, verbose, and stats modes. strace command when160run in verbose mode gives more detailed information about the system calls161invoked by a process.162 163Running strace -c generates a report of the percentage of time spent in each164system call, the total time in seconds, the microseconds per call, the total165number of calls, the count of each system call that has failed with an error166and the type of system call made.167 168 * Usage: strace <command we want to trace>169 * Verbose mode usage: strace -v <command>170 * Gather statistics: strace -c <command>171 172We used the “-c” option to gather fine-grained run-time statistics in use173by three workloads we have chose for this analysis.174 175 * perf176 * stress-ng177 * paxtest178 179What is cscope and how do we use it?180====================================181 182Now let’s look at `cscope <https://cscope.sourceforge.net/>`_, a command183line tool for browsing C, C++ or Java code-bases. We can use it to find184all the references to a symbol, global definitions, functions called by a185function, functions calling a function, text strings, regular expression186patterns, files including a file.187 188We can use cscope to find which system call belongs to which subsystem.189This way we can find the kernel subsystems used by a process when it is190executed.191 192Let’s checkout the latest Linux repository and build cscope database::193 194 git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git linux195 cd linux196 cscope -R -p10 # builds cscope.out database before starting browse session197 cscope -d -p10 # starts browse session on cscope.out database198 199Note: Run "cscope -R -p10" to build the database and c"scope -d -p10" to200enter into the browsing session. cscope by default cscope.out database.201To get out of this mode press ctrl+d. -p option is used to specify the202number of file path components to display. -p10 is optimal for browsing203kernel sources.204 205What is perf and how do we use it?206==================================207 208Perf is an analysis tool based on Linux 2.6+ systems, which abstracts the209CPU hardware difference in performance measurement in Linux, and provides210a simple command line interface. Perf is based on the perf_events interface211exported by the kernel. It is very useful for profiling the system and212finding performance bottlenecks in an application.213 214If you haven't already checked out the Linux mainline repository, you can do215so and then build kernel and perf tool::216 217 git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git linux218 cd linux219 make -j3 all220 cd tools/perf221 make222 223Note: The perf command can be built without building the kernel in the224repository and can be run on older kernels. However matching the kernel225and perf revisions gives more accurate information on the subsystem usage.226 227We used "perf stat" and "perf bench" options. For a detailed information on228the perf tool, run "perf -h".229 230perf stat231---------232The perf stat command generates a report of various hardware and software233events. It does so with the help of hardware counter registers found in234modern CPUs that keep the count of these activities. "perf stat cal" shows235stats for cal command.236 237Perf bench238----------239The perf bench command contains multiple multi-threaded microkernel240benchmarks for executing different subsystems in the Linux kernel and241system calls. This allows us to easily measure the impact of changes,242which can help mitigate performance regressions. It also acts as a common243benchmarking framework, enabling developers to easily create test cases,244integrate transparently, and use performance-rich tooling.245 246"perf bench all" command runs the following benchmarks:247 248 * sched/messaging249 * sched/pipe250 * syscall/basic251 * mem/memcpy252 * mem/memset253 254What is stress-ng and how do we use it?255=======================================256 257As mentioned earlier, stress-ng is used for performing stress testing on258the kernel. It allows you to exercise various physical subsystems of the259computer, as well as interfaces of the OS kernel, using stressor-s. They260are available for CPU, CPU cache, devices, I/O, interrupts, file system,261memory, network, operating system, pipelines, schedulers, and virtual262machines.263 264The netdev stressor starts N workers that exercise various netdevice ioctl265commands across all the available network devices. The following ioctls are266exercised:267 268 * SIOCGIFCONF, SIOCGIFINDEX, SIOCGIFNAME, SIOCGIFFLAGS269 * SIOCGIFADDR, SIOCGIFNETMASK, SIOCGIFMETRIC, SIOCGIFMTU270 * SIOCGIFHWADDR, SIOCGIFMAP, SIOCGIFTXQLEN271 272The following command runs the stressor::273 274 stress-ng --netdev 1 -t 60 --metrics command.275 276We can use the perf record command to record the events and information277associated with a process. This command records the profiling data in the278perf.data file in the same directory.279 280Using the following commands you can record the events associated with the281netdev stressor, view the generated report perf.data and annotate the to282view the statistics of each instruction of the program::283 284 perf record stress-ng --netdev 1 -t 60 --metrics command.285 perf report286 perf annotate287 288What is paxtest and how do we use it?289=====================================290 291paxtest is a program that tests buffer overflows in the kernel. It tests292kernel enforcements over memory usage. Generally, execution in some memory293segments makes buffer overflows possible. It runs a set of programs that294attempt to subvert memory usage. It is used as a regression test suite for295PaX, and will be useful to test other memory protection patches for the296kernel.297 298paxtest provides kiddie and blackhat modes. The paxtest kiddie mode runs299in normal mode, whereas the blackhat mode tries to get around the protection300of the kernel testing for vulnerabilities. We focus on the kiddie mode here301and combine "paxtest kiddie" run with "perf record" to collect CPU stack302traces for the paxtest kiddie run to see which function is calling other303functions in the performance profile. Then the "dwarf" (DWARF's Call Frame304Information) mode can be used to unwind the stack.305 306The following command can be used to view resulting report in call-graph307format::308 309 perf record --call-graph dwarf paxtest kiddie310 perf report --stdio311 312Tracing workloads313=================314 315Now that we understand the workloads, let's start tracing them.316 317Tracing perf bench all workload318-------------------------------319 320Run the following command to trace perf bench all workload::321 322 strace -c perf bench all323 324**System Calls made by the workload**325 326The below table shows the system calls invoked by the workload, number of327times each system call is invoked, and the corresponding Linux subsystem.328 329+-------------------+-----------+-----------------+-------------------------+330| System Call | # calls | Linux Subsystem | System Call (API) |331+===================+===========+=================+=========================+332| getppid | 10000001 | Process Mgmt | sys_getpid() |333+-------------------+-----------+-----------------+-------------------------+334| clone | 1077 | Process Mgmt. | sys_clone() |335+-------------------+-----------+-----------------+-------------------------+336| prctl | 23 | Process Mgmt. | sys_prctl() |337+-------------------+-----------+-----------------+-------------------------+338| prlimit64 | 7 | Process Mgmt. | sys_prlimit64() |339+-------------------+-----------+-----------------+-------------------------+340| getpid | 10 | Process Mgmt. | sys_getpid() |341+-------------------+-----------+-----------------+-------------------------+342| uname | 3 | Process Mgmt. | sys_uname() |343+-------------------+-----------+-----------------+-------------------------+344| sysinfo | 1 | Process Mgmt. | sys_sysinfo() |345+-------------------+-----------+-----------------+-------------------------+346| getuid | 1 | Process Mgmt. | sys_getuid() |347+-------------------+-----------+-----------------+-------------------------+348| getgid | 1 | Process Mgmt. | sys_getgid() |349+-------------------+-----------+-----------------+-------------------------+350| geteuid | 1 | Process Mgmt. | sys_geteuid() |351+-------------------+-----------+-----------------+-------------------------+352| getegid | 1 | Process Mgmt. | sys_getegid |353+-------------------+-----------+-----------------+-------------------------+354| close | 49951 | Filesystem | sys_close() |355+-------------------+-----------+-----------------+-------------------------+356| pipe | 604 | Filesystem | sys_pipe() |357+-------------------+-----------+-----------------+-------------------------+358| openat | 48560 | Filesystem | sys_opennat() |359+-------------------+-----------+-----------------+-------------------------+360| fstat | 8338 | Filesystem | sys_fstat() |361+-------------------+-----------+-----------------+-------------------------+362| stat | 1573 | Filesystem | sys_stat() |363+-------------------+-----------+-----------------+-------------------------+364| pread64 | 9646 | Filesystem | sys_pread64() |365+-------------------+-----------+-----------------+-------------------------+366| getdents64 | 1873 | Filesystem | sys_getdents64() |367+-------------------+-----------+-----------------+-------------------------+368| access | 3 | Filesystem | sys_access() |369+-------------------+-----------+-----------------+-------------------------+370| lstat | 1880 | Filesystem | sys_lstat() |371+-------------------+-----------+-----------------+-------------------------+372| lseek | 6 | Filesystem | sys_lseek() |373+-------------------+-----------+-----------------+-------------------------+374| ioctl | 3 | Filesystem | sys_ioctl() |375+-------------------+-----------+-----------------+-------------------------+376| dup2 | 1 | Filesystem | sys_dup2() |377+-------------------+-----------+-----------------+-------------------------+378| execve | 2 | Filesystem | sys_execve() |379+-------------------+-----------+-----------------+-------------------------+380| fcntl | 8779 | Filesystem | sys_fcntl() |381+-------------------+-----------+-----------------+-------------------------+382| statfs | 1 | Filesystem | sys_statfs() |383+-------------------+-----------+-----------------+-------------------------+384| epoll_create | 2 | Filesystem | sys_epoll_create() |385+-------------------+-----------+-----------------+-------------------------+386| epoll_ctl | 64 | Filesystem | sys_epoll_ctl() |387+-------------------+-----------+-----------------+-------------------------+388| newfstatat | 8318 | Filesystem | sys_newfstatat() |389+-------------------+-----------+-----------------+-------------------------+390| eventfd2 | 192 | Filesystem | sys_eventfd2() |391+-------------------+-----------+-----------------+-------------------------+392| mmap | 243 | Memory Mgmt. | sys_mmap() |393+-------------------+-----------+-----------------+-------------------------+394| mprotect | 32 | Memory Mgmt. | sys_mprotect() |395+-------------------+-----------+-----------------+-------------------------+396| brk | 21 | Memory Mgmt. | sys_brk() |397+-------------------+-----------+-----------------+-------------------------+398| munmap | 128 | Memory Mgmt. | sys_munmap() |399+-------------------+-----------+-----------------+-------------------------+400| set_mempolicy | 156 | Memory Mgmt. | sys_set_mempolicy() |401+-------------------+-----------+-----------------+-------------------------+402| set_tid_address | 1 | Process Mgmt. | sys_set_tid_address() |403+-------------------+-----------+-----------------+-------------------------+404| set_robust_list | 1 | Futex | sys_set_robust_list() |405+-------------------+-----------+-----------------+-------------------------+406| futex | 341 | Futex | sys_futex() |407+-------------------+-----------+-----------------+-------------------------+408| sched_getaffinity | 79 | Scheduler | sys_sched_getaffinity() |409+-------------------+-----------+-----------------+-------------------------+410| sched_setaffinity | 223 | Scheduler | sys_sched_setaffinity() |411+-------------------+-----------+-----------------+-------------------------+412| socketpair | 202 | Network | sys_socketpair() |413+-------------------+-----------+-----------------+-------------------------+414| rt_sigprocmask | 21 | Signal | sys_rt_sigprocmask() |415+-------------------+-----------+-----------------+-------------------------+416| rt_sigaction | 36 | Signal | sys_rt_sigaction() |417+-------------------+-----------+-----------------+-------------------------+418| rt_sigreturn | 2 | Signal | sys_rt_sigreturn() |419+-------------------+-----------+-----------------+-------------------------+420| wait4 | 889 | Time | sys_wait4() |421+-------------------+-----------+-----------------+-------------------------+422| clock_nanosleep | 37 | Time | sys_clock_nanosleep() |423+-------------------+-----------+-----------------+-------------------------+424| capget | 4 | Capability | sys_capget() |425+-------------------+-----------+-----------------+-------------------------+426 427Tracing stress-ng netdev stressor workload428------------------------------------------429 430Run the following command to trace stress-ng netdev stressor workload::431 432 strace -c stress-ng --netdev 1 -t 60 --metrics433 434**System Calls made by the workload**435 436The below table shows the system calls invoked by the workload, number of437times each system call is invoked, and the corresponding Linux subsystem.438 439+-------------------+-----------+-----------------+-------------------------+440| System Call | # calls | Linux Subsystem | System Call (API) |441+===================+===========+=================+=========================+442| openat | 74 | Filesystem | sys_openat() |443+-------------------+-----------+-----------------+-------------------------+444| close | 75 | Filesystem | sys_close() |445+-------------------+-----------+-----------------+-------------------------+446| read | 58 | Filesystem | sys_read() |447+-------------------+-----------+-----------------+-------------------------+448| fstat | 20 | Filesystem | sys_fstat() |449+-------------------+-----------+-----------------+-------------------------+450| flock | 10 | Filesystem | sys_flock() |451+-------------------+-----------+-----------------+-------------------------+452| write | 7 | Filesystem | sys_write() |453+-------------------+-----------+-----------------+-------------------------+454| getdents64 | 8 | Filesystem | sys_getdents64() |455+-------------------+-----------+-----------------+-------------------------+456| pread64 | 8 | Filesystem | sys_pread64() |457+-------------------+-----------+-----------------+-------------------------+458| lseek | 1 | Filesystem | sys_lseek() |459+-------------------+-----------+-----------------+-------------------------+460| access | 2 | Filesystem | sys_access() |461+-------------------+-----------+-----------------+-------------------------+462| getcwd | 1 | Filesystem | sys_getcwd() |463+-------------------+-----------+-----------------+-------------------------+464| execve | 1 | Filesystem | sys_execve() |465+-------------------+-----------+-----------------+-------------------------+466| mmap | 61 | Memory Mgmt. | sys_mmap() |467+-------------------+-----------+-----------------+-------------------------+468| munmap | 3 | Memory Mgmt. | sys_munmap() |469+-------------------+-----------+-----------------+-------------------------+470| mprotect | 20 | Memory Mgmt. | sys_mprotect() |471+-------------------+-----------+-----------------+-------------------------+472| mlock | 2 | Memory Mgmt. | sys_mlock() |473+-------------------+-----------+-----------------+-------------------------+474| brk | 3 | Memory Mgmt. | sys_brk() |475+-------------------+-----------+-----------------+-------------------------+476| rt_sigaction | 21 | Signal | sys_rt_sigaction() |477+-------------------+-----------+-----------------+-------------------------+478| rt_sigprocmask | 1 | Signal | sys_rt_sigprocmask() |479+-------------------+-----------+-----------------+-------------------------+480| sigaltstack | 1 | Signal | sys_sigaltstack() |481+-------------------+-----------+-----------------+-------------------------+482| rt_sigreturn | 1 | Signal | sys_rt_sigreturn() |483+-------------------+-----------+-----------------+-------------------------+484| getpid | 8 | Process Mgmt. | sys_getpid() |485+-------------------+-----------+-----------------+-------------------------+486| prlimit64 | 5 | Process Mgmt. | sys_prlimit64() |487+-------------------+-----------+-----------------+-------------------------+488| arch_prctl | 2 | Process Mgmt. | sys_arch_prctl() |489+-------------------+-----------+-----------------+-------------------------+490| sysinfo | 2 | Process Mgmt. | sys_sysinfo() |491+-------------------+-----------+-----------------+-------------------------+492| getuid | 2 | Process Mgmt. | sys_getuid() |493+-------------------+-----------+-----------------+-------------------------+494| uname | 1 | Process Mgmt. | sys_uname() |495+-------------------+-----------+-----------------+-------------------------+496| setpgid | 1 | Process Mgmt. | sys_setpgid() |497+-------------------+-----------+-----------------+-------------------------+498| getrusage | 1 | Process Mgmt. | sys_getrusage() |499+-------------------+-----------+-----------------+-------------------------+500| geteuid | 1 | Process Mgmt. | sys_geteuid() |501+-------------------+-----------+-----------------+-------------------------+502| getppid | 1 | Process Mgmt. | sys_getppid() |503+-------------------+-----------+-----------------+-------------------------+504| sendto | 3 | Network | sys_sendto() |505+-------------------+-----------+-----------------+-------------------------+506| connect | 1 | Network | sys_connect() |507+-------------------+-----------+-----------------+-------------------------+508| socket | 1 | Network | sys_socket() |509+-------------------+-----------+-----------------+-------------------------+510| clone | 1 | Process Mgmt. | sys_clone() |511+-------------------+-----------+-----------------+-------------------------+512| set_tid_address | 1 | Process Mgmt. | sys_set_tid_address() |513+-------------------+-----------+-----------------+-------------------------+514| wait4 | 2 | Time | sys_wait4() |515+-------------------+-----------+-----------------+-------------------------+516| alarm | 1 | Time | sys_alarm() |517+-------------------+-----------+-----------------+-------------------------+518| set_robust_list | 1 | Futex | sys_set_robust_list() |519+-------------------+-----------+-----------------+-------------------------+520 521Tracing paxtest kiddie workload522-------------------------------523 524Run the following command to trace paxtest kiddie workload::525 526 strace -c paxtest kiddie527 528**System Calls made by the workload**529 530The below table shows the system calls invoked by the workload, number of531times each system call is invoked, and the corresponding Linux subsystem.532 533+-------------------+-----------+-----------------+----------------------+534| System Call | # calls | Linux Subsystem | System Call (API) |535+===================+===========+=================+======================+536| read | 3 | Filesystem | sys_read() |537+-------------------+-----------+-----------------+----------------------+538| write | 11 | Filesystem | sys_write() |539+-------------------+-----------+-----------------+----------------------+540| close | 41 | Filesystem | sys_close() |541+-------------------+-----------+-----------------+----------------------+542| stat | 24 | Filesystem | sys_stat() |543+-------------------+-----------+-----------------+----------------------+544| fstat | 2 | Filesystem | sys_fstat() |545+-------------------+-----------+-----------------+----------------------+546| pread64 | 6 | Filesystem | sys_pread64() |547+-------------------+-----------+-----------------+----------------------+548| access | 1 | Filesystem | sys_access() |549+-------------------+-----------+-----------------+----------------------+550| pipe | 1 | Filesystem | sys_pipe() |551+-------------------+-----------+-----------------+----------------------+552| dup2 | 24 | Filesystem | sys_dup2() |553+-------------------+-----------+-----------------+----------------------+554| execve | 1 | Filesystem | sys_execve() |555+-------------------+-----------+-----------------+----------------------+556| fcntl | 26 | Filesystem | sys_fcntl() |557+-------------------+-----------+-----------------+----------------------+558| openat | 14 | Filesystem | sys_openat() |559+-------------------+-----------+-----------------+----------------------+560| rt_sigaction | 7 | Signal | sys_rt_sigaction() |561+-------------------+-----------+-----------------+----------------------+562| rt_sigreturn | 38 | Signal | sys_rt_sigreturn() |563+-------------------+-----------+-----------------+----------------------+564| clone | 38 | Process Mgmt. | sys_clone() |565+-------------------+-----------+-----------------+----------------------+566| wait4 | 44 | Time | sys_wait4() |567+-------------------+-----------+-----------------+----------------------+568| mmap | 7 | Memory Mgmt. | sys_mmap() |569+-------------------+-----------+-----------------+----------------------+570| mprotect | 3 | Memory Mgmt. | sys_mprotect() |571+-------------------+-----------+-----------------+----------------------+572| munmap | 1 | Memory Mgmt. | sys_munmap() |573+-------------------+-----------+-----------------+----------------------+574| brk | 3 | Memory Mgmt. | sys_brk() |575+-------------------+-----------+-----------------+----------------------+576| getpid | 1 | Process Mgmt. | sys_getpid() |577+-------------------+-----------+-----------------+----------------------+578| getuid | 1 | Process Mgmt. | sys_getuid() |579+-------------------+-----------+-----------------+----------------------+580| getgid | 1 | Process Mgmt. | sys_getgid() |581+-------------------+-----------+-----------------+----------------------+582| geteuid | 2 | Process Mgmt. | sys_geteuid() |583+-------------------+-----------+-----------------+----------------------+584| getegid | 1 | Process Mgmt. | sys_getegid() |585+-------------------+-----------+-----------------+----------------------+586| getppid | 1 | Process Mgmt. | sys_getppid() |587+-------------------+-----------+-----------------+----------------------+588| arch_prctl | 2 | Process Mgmt. | sys_arch_prctl() |589+-------------------+-----------+-----------------+----------------------+590 591Conclusion592==========593 594This document is intended to be used as a guide on how to gather fine-grained595information on the resources in use by workloads using strace.596 597References598==========599 600 * `Discovery Linux Kernel Subsystems used by OpenAPS <https://elisa.tech/blog/2022/02/02/discovery-linux-kernel-subsystems-used-by-openaps>`_601 * `ELISA-White-Papers-Discovering Linux kernel subsystems used by a workload <https://github.com/elisa-tech/ELISA-White-Papers/blob/master/Processes/Discovering_Linux_kernel_subsystems_used_by_a_workload.md>`_602 * `strace <https://man7.org/linux/man-pages/man1/strace.1.html>`_603 * `perf <https://man7.org/linux/man-pages/man1/perf.1.html>`_604 * `paxtest README <https://github.com/opntr/paxtest-freebsd/blob/hardenedbsd/0.9.14-hbsd/README>`_605 * `stress-ng <https://www.mankier.com/1/stress-ng>`_606 * `Monitoring and managing system status and performance <https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/monitoring_and_managing_system_status_and_performance/index>`_607