180 lines · plain
1.. highlight:: none2 3Debugging kernel and modules via gdb4====================================5 6The kernel debugger kgdb, hypervisors like QEMU or JTAG-based hardware7interfaces allow to debug the Linux kernel and its modules during runtime8using gdb. Gdb comes with a powerful scripting interface for python. The9kernel provides a collection of helper scripts that can simplify typical10kernel debugging steps. This is a short tutorial about how to enable and use11them. It focuses on QEMU/KVM virtual machines as target, but the examples can12be transferred to the other gdb stubs as well.13 14 15Requirements16------------17 18- gdb 7.2+ (recommended: 7.4+) with python support enabled (typically true19 for distributions)20 21 22Setup23-----24 25- Create a virtual Linux machine for QEMU/KVM (see www.linux-kvm.org and26 www.qemu.org for more details). For cross-development,27 https://landley.net/aboriginal/bin keeps a pool of machine images and28 toolchains that can be helpful to start from.29 30- Build the kernel with CONFIG_GDB_SCRIPTS enabled, but leave31 CONFIG_DEBUG_INFO_REDUCED off. If your architecture supports32 CONFIG_FRAME_POINTER, keep it enabled.33 34- Install that kernel on the guest, turn off KASLR if necessary by adding35 "nokaslr" to the kernel command line.36 Alternatively, QEMU allows to boot the kernel directly using -kernel,37 -append, -initrd command line switches. This is generally only useful if38 you do not depend on modules. See QEMU documentation for more details on39 this mode. In this case, you should build the kernel with40 CONFIG_RANDOMIZE_BASE disabled if the architecture supports KASLR.41 42- Build the gdb scripts (required on kernels v5.1 and above)::43 44 make scripts_gdb45 46- Enable the gdb stub of QEMU/KVM, either47 48 - at VM startup time by appending "-s" to the QEMU command line49 50 or51 52 - during runtime by issuing "gdbserver" from the QEMU monitor53 console54 55- cd /path/to/linux-build56 57- Start gdb: gdb vmlinux58 59 Note: Some distros may restrict auto-loading of gdb scripts to known safe60 directories. In case gdb reports to refuse loading vmlinux-gdb.py, add::61 62 add-auto-load-safe-path /path/to/linux-build63 64 to ~/.gdbinit. See gdb help for more details.65 66- Attach to the booted guest::67 68 (gdb) target remote :123469 70 71Examples of using the Linux-provided gdb helpers72------------------------------------------------73 74- Load module (and main kernel) symbols::75 76 (gdb) lx-symbols77 loading vmlinux78 scanning for modules in /home/user/linux/build79 loading @0xffffffffa0020000: /home/user/linux/build/net/netfilter/xt_tcpudp.ko80 loading @0xffffffffa0016000: /home/user/linux/build/net/netfilter/xt_pkttype.ko81 loading @0xffffffffa0002000: /home/user/linux/build/net/netfilter/xt_limit.ko82 loading @0xffffffffa00ca000: /home/user/linux/build/net/packet/af_packet.ko83 loading @0xffffffffa003c000: /home/user/linux/build/fs/fuse/fuse.ko84 ...85 loading @0xffffffffa0000000: /home/user/linux/build/drivers/ata/ata_generic.ko86 87- Set a breakpoint on some not yet loaded module function, e.g.::88 89 (gdb) b btrfs_init_sysfs90 Function "btrfs_init_sysfs" not defined.91 Make breakpoint pending on future shared library load? (y or [n]) y92 Breakpoint 1 (btrfs_init_sysfs) pending.93 94- Continue the target::95 96 (gdb) c97 98- Load the module on the target and watch the symbols being loaded as well as99 the breakpoint hit::100 101 loading @0xffffffffa0034000: /home/user/linux/build/lib/libcrc32c.ko102 loading @0xffffffffa0050000: /home/user/linux/build/lib/lzo/lzo_compress.ko103 loading @0xffffffffa006e000: /home/user/linux/build/lib/zlib_deflate/zlib_deflate.ko104 loading @0xffffffffa01b1000: /home/user/linux/build/fs/btrfs/btrfs.ko105 106 Breakpoint 1, btrfs_init_sysfs () at /home/user/linux/fs/btrfs/sysfs.c:36107 36 btrfs_kset = kset_create_and_add("btrfs", NULL, fs_kobj);108 109- Dump the log buffer of the target kernel::110 111 (gdb) lx-dmesg112 [ 0.000000] Initializing cgroup subsys cpuset113 [ 0.000000] Initializing cgroup subsys cpu114 [ 0.000000] Linux version 3.8.0-rc4-dbg+ (...115 [ 0.000000] Command line: root=/dev/sda2 resume=/dev/sda1 vga=0x314116 [ 0.000000] e820: BIOS-provided physical RAM map:117 [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable118 [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved119 ....120 121- Examine fields of the current task struct(supported by x86 and arm64 only)::122 123 (gdb) p $lx_current().pid124 $1 = 4998125 (gdb) p $lx_current().comm126 $2 = "modprobe\000\000\000\000\000\000\000"127 128- Make use of the per-cpu function for the current or a specified CPU::129 130 (gdb) p $lx_per_cpu("runqueues").nr_running131 $3 = 1132 (gdb) p $lx_per_cpu("runqueues", 2).nr_running133 $4 = 0134 135- Dig into hrtimers using the container_of helper::136 137 (gdb) set $next = $lx_per_cpu("hrtimer_bases").clock_base[0].active.next138 (gdb) p *$container_of($next, "struct hrtimer", "node")139 $5 = {140 node = {141 node = {142 __rb_parent_color = 18446612133355256072,143 rb_right = 0x0 <irq_stack_union>,144 rb_left = 0x0 <irq_stack_union>145 },146 expires = {147 tv64 = 1835268000000148 }149 },150 _softexpires = {151 tv64 = 1835268000000152 },153 function = 0xffffffff81078232 <tick_sched_timer>,154 base = 0xffff88003fd0d6f0,155 state = 1,156 start_pid = 0,157 start_site = 0xffffffff81055c1f <hrtimer_start_range_ns+20>,158 start_comm = "swapper/2\000\000\000\000\000\000"159 }160 161 162List of commands and functions163------------------------------164 165The number of commands and convenience functions may evolve over the time,166this is just a snapshot of the initial version::167 168 (gdb) apropos lx169 function lx_current -- Return current task170 function lx_module -- Find module by name and return the module variable171 function lx_per_cpu -- Return per-cpu variable172 function lx_task_by_pid -- Find Linux task by PID and return the task_struct variable173 function lx_thread_info -- Calculate Linux thread_info from task variable174 lx-dmesg -- Print Linux kernel log buffer175 lx-lsmod -- List currently loaded modules176 lx-symbols -- (Re-)load symbols of Linux kernel and currently loaded modules177 178Detailed help can be obtained via "help <command-name>" for commands and "help179function <function-name>" for convenience functions.180