On 2016/10/31 at 15:15, Xunlei Pang wrote:
The current method for kdump memory debug is to use dracut
"rd.memdebug=[0-3]",
it is not enough for debugging kernel modules. For example, when we want to find
out which kernel module consumes a large amount of memory, "rd.memdebug"
won't
help too much.
A better way is needed to achieve this requirement, this is very useful for kdump
OOM debugging.
The principle of this patch series is to use kernel trace to track slab and buddy
allocation calls during kernel module loading(module_init), thus we can analyze
all the trace data and get the total memory consumption. As for large slab allocation,
it will probably fall into buddy allocation, thus tracing "mm_page_alloc" alone
should
be enough for the purpose.
The trace events include memory calls under "tracing/events/":
kmem/mm_page_alloc
We also inpect the following events to detect the module loading:
module/module_load
module/module_put
We can get the module name and task pid from "module_load" event which
also mark the beginning of the loading, and module_put called by the
same task pid implies the end of the loading. So the memory events
recorded in between by the same task pid are consumed by this module
during loading(i.e. modprobe or module_init()).
With these information, we can record the total memory(the larger, the more
precise) consumption involved by each kernel module loading.
One major flaw of this method is that the trace ring buffer consumes a lot
of memory. If it is too small, old records maybe be overwritten by subsequent
records. The trace ring buffer is set to be 5MB by default, but it can be
overridden by users via the standard kernel boot parameter "trace_buf_size".
Users should increase the crash kernel memory reservation as needed after
setting large trace ring buffer size, in case oom happens during debugging.
Usage:
1)Pass "rd.kodebug" to kdump kernel cmdline using
"KDUMP_COMMANDLINE_APPEND" in /etc/sysconfig/kdump.
Dave once mentioned that to integrate this feature into dracut "rd.memdebug",
thinking more that
this is implemented in a completely different way, and there is one major flaw that it
consumes
lots of memory and may probably cause OOM on large systems, also we need to find a call
site
where all the modules are supposed to be loaded. I'd like to consider it on
dracut's side if we can
find some way to reduce the trace memory one day.
Thus I added a new "rd.kodebug" on kdump side for this feature, after all kdump
is the known user
that longs for it until now.
Regards,
Xunlei
2)Pass the extra "trace_buf_size=nn[KMG]" to specify trace
ring buffer size(per cpu) as needed.
As an example, it prints out something below on my kvm machine:
== debug_mem for kernel modules during loading begin ==
4 pages consumed by "dm_mod" [load finished]
0 pages consumed by "dm_log" [load finished]
0 pages consumed by "dm_region_hash" [load finished]
0 pages consumed by "dm_mirror" [load finished]
9 pages consumed by "sunrpc" [load finished]
24 pages consumed by "floppy" [load finished]
0 pages consumed by "libata" [load finished]
0 pages consumed by "i2c_core" [load finished]
27 pages consumed by "ata_piix" [load finished]
0 pages consumed by "drm" [load finished]
1 pages consumed by "ttm" [load finished]
0 pages consumed by "drm_kms_helper" [load finished]
1604 pages consumed by "qxl" [load finished]
0 pages consumed by "virtio" [load finished]
0 pages consumed by "virtio_ring" [load finished]
10 pages consumed by "virtio_pci" [load finished]
1 pages consumed by "pata_acpi" [load finished]
0 pages consumed by "ata_generic" [load finished]
0 pages consumed by "serio_raw" [load finished]
0 pages consumed by "crc32c_intel" [load finished]
0 pages consumed by "crct10dif_common" [load finished]
0 pages consumed by "crct10dif_pclmul" [load finished]
278 pages consumed by "virtio_net" [load finished]
198 pages consumed by "virtio_console" [load finished]
0 pages consumed by "cdrom" [load finished]
6 pages consumed by "sr_mod" [load finished]
162 pages consumed by "virtio_blk" [load finished]
1 pages consumed by "fscache" [load finished]
0 pages consumed by "lockd" [load finished]
17 pages consumed by "nfs" [load finished]
0 pages consumed by "libcrc32c" [load finished]
0 pages consumed by "dns_resolver" [load finished]
8 pages consumed by "xfs" [load finished]
0 pages consumed by "nfsv4" [load finished]
== debug_mem for kernel modules during loading end ==
We can clearly see that "qxl" loading consumed more than 6MB memory.
Xunlei Pang (3):
memdebug-ko: add dracut-memdebug-ko.sh to debug kernel module memory
consumption
module-setup: apply kernel module memory debug support
kexec-kdump-howto: add the debugging tip for rd.kodebug
dracut-kdump.sh | 9 ++++
dracut-memdebug-ko.sh | 117 +++++++++++++++++++++++++++++++++++++++++++++++++
dracut-module-setup.sh | 9 ++++
kdumpctl | 14 ++++++
kexec-kdump-howto.txt | 37 ++++++++++++++++
kexec-tools.spec | 2 +
6 files changed, 188 insertions(+)
create mode 100755 dracut-memdebug-ko.sh