On 2016/10/10 at 15:43, Xunlei Pang wrote:
The current method for kdump memory debug is to use dracut
"rd.memdebug=[0-3]",
it is not enough for debugging kernel modules. For example, when we want to find
out which kernel module consumes a large amount of memory, "rd.memdebug"
won't
help too much.
A better way is needed to achieve this requirement, this is very useful for kdump
OOM debugging.
The principle of this patch series is to use kernel trace to track slab and buddy
allocation calls during kernel module loading(module_init), thus we can analyze
all the trace data and get the total memory consumption.
The trace events include memory calls under /sys/kernel/debug/tracing/events:
kmem/mm_page_alloc
kmem/mm_page_free
kmem/kmalloc
kmem/kmalloc_node
kmem/kmem_cache_alloc
kmem/kmem_cache_alloc_node
We also inpect the following events to detect the module loading
module/module_load
module/module_put
We can get the module name and task pid from "module_load" event which
also mark the beginning of the loading, and module_put called by the
same task pid implies the end of the loading. So the memory events
recorded in between by the same task pid are consumed by this module
during loading(i.e. modprobe or module_init()).
With these information, we can record approximately the total memory
consumption involved by each kernel module loading.
One major flaw of this method is that the trace ring buffer consumes a lot
of memory. If it is too small, old records maybe be overwritten by subsequent
For some large systems, the amount of memory consumption is appreciable.
Does anyone have a good idea to reduce the memory size consumed by tracing?
I previously tried to use trace filter to track specific process doing modprobe, but
according to the test results, normally the process name with "systemd-udevd"
and "modprobe" will insert the ko, but sometimes I found some unknown process
will also insert the ko. So it's unreliable to do so.
I am currently thinking to monitor buddy event only, as all the large slab allocation
will actually fall back to buddy to request pages. This will considerably reduce the
requirement of trace buffer size.
Regards,
Xunlei
records. The trace ring buffer is set to be 10MB by default, but it
can be
overridden by users via the standard kernel boot parameter "trace_buf_size".
Users should increase the crash kernel memory reservation as needed after
setting large trace ring buffer size, in case oom happens during debugging.
Usage:
1)Pass "rd.memdebug" to kdump kernel cmdline using
"KDUMP_COMMANDLINE_APPEND" in /etc/sysconfig/kdump.
2)Pass the extra "trace_buf_size=nn[KMG]" to specify trace ring buffer size(per
cpu) as needed.
Xunlei Pang (2):
memdebug-ko: add dracut-memdebug-ko.sh to debug kernel module memory
consumption
module-setup: apply kernel module memory debug support
dracut-kdump.sh | 11 ++++
dracut-memdebug-ko.sh | 144 +++++++++++++++++++++++++++++++++++++++++++++++++
dracut-module-setup.sh | 12 +++++
kdumpctl | 14 +++++
kexec-tools.spec | 2 +
5 files changed, 183 insertions(+)
create mode 100755 dracut-memdebug-ko.sh