On Tue, Oct 20, 2020 at 10:46 AM Lianbo Jiang lijiang@redhat.com wrote:

Currently, all messages are directly printed to the console, sometimes, we also need to output these messages to the journal log according to the log level.

In view of this, introduce the kdump logger from the dracut module.

Signed-off-by: Lianbo Jiang lijiang@redhat.com

kdump-logger.sh | 344 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 344 insertions(+) create mode 100755 kdump-logger.sh

diff --git a/kdump-logger.sh b/kdump-logger.sh new file mode 100755 index 000000000000..c2c5c35820e1 --- /dev/null +++ b/kdump-logger.sh @@ -0,0 +1,344 @@ +#!/bin/bash +# +# This comes from the dracut-logger.sh +# +# The logger defined 6 logging levels: +# - dtrace() (6) +# The TRACE Level designates finer-grained informational events than the +# DEBUG. +# - ddebug (5) +# The DEBUG Level designates fine-grained informational events that are most +# useful to debug an application. +# - dinfo (4) +# The INFO level designates informational messages that highlight the +# progress of the application at coarse-grained level. +# - dwarn (3) +# The WARN level designates potentially harmful situations. +# - derror (2) +# The ERROR level designates error events that might still allow the +# application to continue running. +# - dfatal (1) +# The FATAL level designates very severe error events that will presumably +# lead the application to abort. +# +# Logging is controlled by following global variables: +# - @var kdump_stdloglvl - logging level to standard error (console output) +# - @var kdump_sysloglvl - logging level to syslog (by logger command) +# - @var kdump_kmsgloglvl - logging level to /dev/kmsg (only for boot-time) +# +# If any of the variables is not set, this function set it to default: +# - @var kdump_stdloglvl = 4 (info) +# - @var kdump_sysloglvl = 4 (info) +# - @var kdump_kmsgloglvl = 0 (no logging) +# +# First of all you have to start with dlog_init() function which initializes +# required variables. Don't call any other logging function before that one! +# +# @brief Initializes Logger. +# @retval 1 if something has gone wrong +# @retval 0 on success. +# +dlog_init() {

• local ret=0; local errmsg
• [ -z "$kdump_stdloglvl" ] && kdump_stdloglvl=4
• [ -z "$kdump_sysloglvl" ] && kdump_sysloglvl=4
• [ -z "$kdump_kmsgloglvl" ] && kdump_kmsgloglvl=0
• # Skip initialization if it's already done.
• [ -n "$kdump_maxloglvl" ] && return 0
• if [[ $UID -ne 0 ]]; then

•    kdump_kmsgloglvl=0
  

•    kdump_sysloglvl=0
  

• fi
• if [[ $kdump_sysloglvl -gt 0 ]]; then

•    if [[ -d /run/systemd/journal ]] \
  

•        && type -P systemd-cat &>/dev/null \
  

•        && systemctl --quiet is-active systemd-journald.socket &>/dev/null; then
  

•        readonly _systemdcatfile="/var/tmp/systemd-cat"
  

•        mkfifo "$_systemdcatfile" &>/dev/null
  

•        readonly _dlogfd=15
  

•        systemd-cat -t 'kdump' --level-prefix=true <"$_systemdcatfile" &
  

•        exec 15>"$_systemdcatfile"
  

•    elif ! [ -S /dev/log -a -w /dev/log ] || ! command -v logger >/dev/null; then
  

•        # We cannot log to syslog, so turn this facility off.
  

•        kdump_kmsgloglvl=$kdump_sysloglvl
  

•        kdump_sysloglvl=0
  

•        ret=1
  

•        errmsg="No '/dev/log' or 'logger' included for syslog logging"
  

•    fi
  

• fi
• if [[ $kdump_sysloglvl -gt 0 ]] || [[ $kdump_kmsgloglvl -gt 0 ]]; then

•    if [ -n "$dracutbasedir" ]; then
  

Hi Lianbo, I think this dracut related code should be dropped.

From dracut comment, syslogfacility is set either to 'user' (when building initramfs) or 'daemon' (when booting). I thin we can either just drop it, as usually we already know where the log comes from, or maybe we can use /proc/vmcore to judge if it's the kdump kernel.

•        readonly kdump_syslogfacility=user
  

•    else
  

•        readonly kdump_syslogfacility=daemon
  

•    fi
  

•    export kdump_syslogfacility
  

• fi
• local lvl; local maxloglvl_l=0
• for lvl in $kdump_stdloglvl $kdump_sysloglvl $kdump_kmsgloglvl; do

•    [[ $lvl -gt $maxloglvl_l ]] && maxloglvl_l=$lvl
  

• done
• readonly kdump_maxloglvl=$maxloglvl_l
• export kdump_maxloglvl
• if [[ $kdump_stdloglvl -lt 6 ]] && [[ $kdump_kmsgloglvl -lt 6 ]] && [[ $kdump_sysloglvl -lt 6 ]]; then

•    unset dtrace
  

•    dtrace() { :; };
  

• fi
• if [[ $kdump_stdloglvl -lt 5 ]] && [[ $kdump_kmsgloglvl -lt 5 ]] && [[ $kdump_sysloglvl -lt 5 ]]; then

•    unset ddebug
  

•    ddebug() { :; };
  

• fi
• if [[ $kdump_stdloglvl -lt 4 ]] && [[ $kdump_kmsgloglvl -lt 4 ]] && [[ $kdump_sysloglvl -lt 4 ]]; then

•    unset dinfo
  

•    dinfo() { :; };
  

• fi
• if [[ $kdump_stdloglvl -lt 3 ]] && [[ $kdump_kmsgloglvl -lt 3 ]] && [[ $kdump_sysloglvl -lt 3 ]]; then

•    unset dwarn
  

•    dwarn() { :; };
  

•    unset dwarning
  

•    dwarning() { :; };
  

• fi
• if [[ $kdump_stdloglvl -lt 2 ]] && [[ $kdump_kmsgloglvl -lt 2 ]] && [[ $kdump_sysloglvl -lt 2 ]]; then

•    unset derror
  

•    derror() { :; };
  

• fi
• if [[ $kdump_stdloglvl -lt 1 ]] && [[ $kdump_kmsgloglvl -lt 1 ]] && [[ $kdump_sysloglvl -lt 1 ]]; then

•    unset dfatal
  

•    dfatal() { :; };
  

• fi
• [ -n "$errmsg" ] && derror "$errmsg"
• return $ret

+}

+## @brief Converts numeric logging level to the first letter of level name. +# +# @param lvl Numeric logging level in range from 1 to 6. +# @retval 1 if @a lvl is out of range. +# @retval 0 if @a lvl is correct. +# @result Echoes first letter of level name. +_lvl2char() {

• case "$1" in

•    1) echo F;;
  

•    2) echo E;;
  

•    3) echo W;;
  

•    4) echo I;;
  

•    5) echo D;;
  

•    6) echo T;;
  

•    *) return 1;;
  

• esac

+}

+## @brief Converts numeric level to logger priority defined by POSIX.2. +# +# @param lvl Numeric logging level in range from 1 to 6. +# @retval 1 if @a lvl is out of range. +# @retval 0 if @a lvl is correct. +# @result Echoes logger priority. +_lvl2syspri() {

• printf $kdump_syslogfacility.
• case "$1" in

•    1) echo crit;;
  

•    2) echo error;;
  

•    3) echo warning;;
  

•    4) echo info;;
  

•    5) echo debug;;
  

•    6) echo debug;;
  

•    *) return 1;;
  

• esac

+}

+## @brief Converts logger numeric level to syslog log level +# +# @param lvl Numeric logging level in range from 1 to 6. +# @retval 1 if @a lvl is out of range. +# @retval 0 if @a lvl is correct. +# @result Echoes kernel console numeric log level +# +# Conversion is done as follows: +# +# <tt> +# none -> LOG_EMERG (0) +# none -> LOG_ALERT (1) +# FATAL(1) -> LOG_CRIT (2) +# ERROR(2) -> LOG_ERR (3) +# WARN(3) -> LOG_WARNING (4) +# none -> LOG_NOTICE (5) +# INFO(4) -> LOG_INFO (6) +# DEBUG(5) -> LOG_DEBUG (7) +# TRACE(6) / +# </tt> +# +# @see /usr/include/sys/syslog.h +_dlvl2syslvl() {

• local lvl
• case "$1" in

•    1) lvl=2;;
  

•    2) lvl=3;;
  

•    3) lvl=4;;
  

•    4) lvl=6;;
  

•    5) lvl=7;;
  

•    6) lvl=7;;
  

•    *) return 1;;
  

• esac
• [ "$kdump_syslogfacility" = user ] && echo $((8+$lvl)) || echo $((24+$lvl))

+}

+## @brief Prints to stderr and/or writes to file, to syslog and/or /dev/kmsg +# given message with given level (priority). +# +# @param lvl Numeric logging level. +# @param msg Message. +# @retval 0 It's always returned, even if logging failed. +# +# @note This function is not supposed to be called manually. Please use +# dtrace(), ddebug(), or others instead which wrap this one. +# +# This is core logging function which logs given message to standard error, file +# and/or syslog (with POSIX shell command <tt>logger</tt>) and/or to /dev/kmsg. +# The format is following: +# +# <tt>X: some message</tt> +# +# where @c X is the first letter of logging level. See module description for +# details on that. +# +# Message to syslog is sent with tag @c kdump. Priorities are mapped as +# following: +# - @c FATAL to @c crit +# - @c ERROR to @c error +# - @c WARN to @c warning +# - @c INFO to @c info +# - @c DEBUG and @c TRACE both to @c debug +_do_dlog() {

• local lvl="$1"; shift
• local lvlc=$(_lvl2char "$lvl") || return 0
• local msg="$*"
• local lmsg="$lvlc: $*"
• [[ $lvl -le $kdump_stdloglvl ]] && printf -- 'kdump: %s\n' "$msg" >&2
• if [[ $lvl -le $kdump_sysloglvl ]]; then

•    if [[ "$_dlogfd" ]]; then
  

•        printf -- "<%s>%s\n" "$(($(_dlvl2syslvl $lvl) & 7))" "$msg" >&$_dlogfd
  

•    else
  

•        logger -t "kdump[$$]" -p $(_lvl2syspri $lvl) -- "$msg"
  

•    fi
  

• fi
• [[ $lvl -le $kdump_kmsgloglvl ]] && \

•    echo "<$(_dlvl2syslvl $lvl)>kdump[$$] $msg" >/dev/kmsg
  

+}

+## @brief Internal helper function for _do_dlog() +# +# @param lvl Numeric logging level. +# @param msg Message. +# @retval 0 It's always returned, even if logging failed. +# +# @note This function is not supposed to be called manually. Please use +# dtrace(), ddebug(), or others instead which wrap this one. +# +# This function calls _do_dlog() either with parameter msg, or if +# none is given, it will read standard input and will use every line as +# a message. +# +# This enables: +# dwarn "This is a warning" +# echo "This is a warning" | dwarn +dlog() {

• [ -z "$kdump_maxloglvl" ] && return 0
• [[ $1 -le $kdump_maxloglvl ]] || return 0
• if [[ $# -gt 1 ]]; then

•    _do_dlog "$@"
  

• else

•    while read line || [ -n "$line" ]; do
  

•        _do_dlog "$1" "$line"
  

•    done
  

• fi

+}

+## @brief Logs message at TRACE level (6) +# +# @param msg Message. +# @retval 0 It's always returned, even if logging failed. +dtrace() {

• set +x
• dlog 6 "$@"
• [ -n "$debug" ] && set -x || :

+}

+## @brief Logs message at DEBUG level (5) +# +# @param msg Message. +# @retval 0 It's always returned, even if logging failed. +ddebug() {

• set +x
• dlog 5 "$@"
• [ -n "$debug" ] && set -x || :

+}

+## @brief Logs message at INFO level (4) +# +# @param msg Message. +# @retval 0 It's always returned, even if logging failed. +dinfo() {

• set +x
• dlog 4 "$@"
• [ -n "$debug" ] && set -x || :

+}

+## @brief Logs message at WARN level (3) +# +# @param msg Message. +# @retval 0 It's always returned, even if logging failed. +dwarn() {

• set +x
• dlog 3 "$@"
• [ -n "$debug" ] && set -x || :

+}

+## @brief It's an alias to dwarn() function. +# +# @param msg Message. +# @retval 0 It's always returned, even if logging failed. +dwarning() {

• set +x
• dwarn "$@"
• [ -n "$debug" ] && set -x || :

+}

+## @brief Logs message at ERROR level (2) +# +# @param msg Message. +# @retval 0 It's always returned, even if logging failed. +derror() {

• set +x
• dlog 2 "$@"
• [ -n "$debug" ] && set -x || :

+}

+## @brief Logs message at FATAL level (1) +# +# @param msg Message. +# @retval 0 It's always returned, even if logging failed. +dfatal() {

• set +x
• dlog 1 "$@"
• [ -n "$debug" ] && set -x || :

+}

2.17.1

-- Best Regards, Kairui Song

在 2020年10月26日 16:05, Kairui Song 写道:

Hi Lianbo, thanks for the patch.

On Tue, Oct 20, 2020 at 10:46 AM Lianbo Jiang lijiang@redhat.com wrote:

...

Since the logger was introduced into kdump, let's enable it for kdump so that we can output kdump messages according the log level and save these messages for debugging.

Signed-off-by: Lianbo Jiang lijiang@redhat.com

dracut-early-kdump-module-setup.sh | 3 + dracut-early-kdump.sh | 24 +++- kdump-lib.sh | 23 +-- kdumpctl | 224 ++++++++++++++++------------- kexec-tools.spec | 2 + mkdumprd | 12 +- 6 files changed, 165 insertions(+), 123 deletions(-)

diff --git a/dracut-early-kdump-module-setup.sh b/dracut-early-kdump-module-setup.sh index f30bd6790fda..3b233839e24b 100755 --- a/dracut-early-kdump-module-setup.sh +++ b/dracut-early-kdump-module-setup.sh @@ -48,7 +48,10 @@ install() { inst_simple "/etc/sysconfig/kdump" inst_binary "/usr/sbin/kexec" inst_binary "/usr/bin/gawk" "/usr/bin/awk"

• inst_binary "/usr/bin/logger" "/usr/bin/logger"
• inst_binary "/usr/bin/printf" "/usr/bin/printf" inst_script "/lib/kdump/kdump-lib.sh" "/lib/kdump-lib.sh"
• inst_script "/lib/kdump/kdump-logger.sh" "/lib/kdump-logger.sh" inst_hook cmdline 00 "$moddir/early-kdump.sh" inst_binary "$KDUMP_KERNEL" inst_binary "$KDUMP_INITRD"

diff --git a/dracut-early-kdump.sh b/dracut-early-kdump.sh index 92913fbab116..add9cc1bfacd 100755 --- a/dracut-early-kdump.sh +++ b/dracut-early-kdump.sh @@ -12,6 +12,14 @@ EARLY_KEXEC_ARGS="" . /etc/sysconfig/kdump . /lib/dracut-lib.sh . /lib/kdump-lib.sh +. /lib/kdump-logger.sh

There should be no need to include /lib/kdump-logger.sh, it's already included in kdump-lib.sh. Same in other files.

Good findings.

...

+#initiate the kdump logger +dlog_init +if [ $? -ne 0 ]; then

•    echo "failed to initiate the kdump logger."
  

•    exit 1
  

+fi

prepare_parameters() { @@ -28,7 +36,7 @@ early_kdump_load() fi

 if is_fadump_capable; then

•    echo "WARNING: early kdump doesn't support fadump."
  

•    dwarn "WARNING: early kdump doesn't support fadump."
     return 1
  

  fi

@@ -42,18 +50,22 @@ early_kdump_load() EARLY_KEXEC_ARGS=$(prepare_kexec_args "${KEXEC_ARGS}")

 if is_secure_boot_enforced; then

•    echo "Secure Boot is enabled. Using kexec file based syscall."
  

•    dinfo "Secure Boot is enabled. Using kexec file based syscall."
     EARLY_KEXEC_ARGS="$EARLY_KEXEC_ARGS -s"
  

  fi

• ddebug "earlykdump: $KEXEC ${EARLY_KEXEC_ARGS} $standard_kexec_args \

•   --command-line=$EARLY_KDUMP_CMDLINE --initrd=$EARLY_KDUMP_INITRD \
  

•   $EARLY_KDUMP_KERNEL"
  

• $KEXEC ${EARLY_KEXEC_ARGS} $standard_kexec_args \ --command-line="$EARLY_KDUMP_CMDLINE" \ --initrd=$EARLY_KDUMP_INITRD $EARLY_KDUMP_KERNEL if [ $? == 0 ]; then

•    echo "kexec: loaded early-kdump kernel"
  

•    dinfo "kexec: loaded early-kdump kernel"
     return 0
  

  else

•    echo "kexec: failed to load early-kdump kernel"
  

•    derror "kexec: failed to load early-kdump kernel"
     return 1
  

  fi

} @@ -61,10 +73,10 @@ early_kdump_load() set_early_kdump() { if getargbool 0 rd.earlykdump; then

•    echo "early-kdump is enabled."
  

•    dinfo "early-kdump is enabled."
     early_kdump_load
  

  else

•    echo "early-kdump is disabled."
  

•    dinfo "early-kdump is disabled."
  

  fi

  return 0

diff --git a/kdump-lib.sh b/kdump-lib.sh index e18c47333b17..abbb1ab5e3f4 100755 --- a/kdump-lib.sh +++ b/kdump-lib.sh @@ -8,6 +8,12 @@ FENCE_KDUMP_CONFIG_FILE="/etc/sysconfig/fence_kdump" FENCE_KDUMP_SEND="/usr/libexec/fence_kdump_send" FADUMP_ENABLED_SYS_NODE="/sys/kernel/fadump_enabled"

+if [ -f /lib/kdump/kdump-logger.sh ]; then

• . /lib/kdump/kdump-logger.sh

+elif [ -f /lib/kdump-logger.sh ]; then

• . /lib/kdump-logger.sh

+fi

is_fadump_capable() { # Check if firmware-assisted dump is enabled @@ -20,14 +26,10 @@ is_fadump_capable() }

perror_exit() {

• echo $@ >&2

• derror "$@" exit 1

}

-perror() {

• echo $@ >&2

-}

is_fs_type_nfs() { [ "$1" = "nfs" ] || [ "$1" = "nfs4" ] @@ -503,7 +505,7 @@ check_crash_mem_reserved()

 mem_reserved=$(cat /sys/kernel/kexec_crash_size)
 if [ $mem_reserved -eq 0 ]; then

•    echo "No memory reserved for crash kernel"
  

•    derror "No memory reserved for crash kernel"
     return 1
  

  fi

@@ -513,7 +515,7 @@ check_crash_mem_reserved() check_kdump_feasibility() { if [ ! -e /sys/kernel/kexec_crash_loaded ]; then

•    echo "Kdump is not supported on this kernel"
  

•    derror "Kdump is not supported on this kernel"
     return 1
  

  fi check_crash_mem_reserved

@@ -523,7 +525,7 @@ check_kdump_feasibility() check_current_kdump_status() { if [ ! -f /sys/kernel/kexec_crash_loaded ];then

•    echo "Perhaps CONFIG_CRASH_DUMP is not enabled in kernel"
  

•    derror "Perhaps CONFIG_CRASH_DUMP is not enabled in kernel"
     return 1
  

  fi

@@ -645,8 +647,7 @@ prepare_kexec_args() found_elf_args=`echo $kexec_args | grep elf32-core-headers` if [ -n "$found_elf_args" ] then

•            echo -n "Warning: elf32-core-headers overrides correct elf64 setting"
  

•            echo
  

•            dwarn "Warning: elf32-core-headers overrides correct elf64 setting"
         else
             kexec_args="$kexec_args --elf64-core-headers"
         fi
  

@@ -697,7 +698,7 @@ prepare_kdump_bootinfo() done

 if ! [ -e "$KDUMP_KERNEL" ]; then

•    echo "Failed to detect kdump kernel location"
  

•    derror "Failed to detect kdump kernel location"
     return 1
  

  fi

diff --git a/kdumpctl b/kdumpctl index 2248da48da2b..c61774364709 100755 --- a/kdumpctl +++ b/kdumpctl @@ -21,10 +21,6 @@ FADUMP_REGISTER_SYS_NODE="/sys/kernel/fadump_registered" DEFAULT_DUMP_MODE="kdump" image_time=0

-[[ $dracutbasedir ]] || dracutbasedir=/usr/lib/dracut -. $dracutbasedir/dracut-functions.sh -. /lib/kdump/kdump-lib.sh

standard_kexec_args="-p"

# Some default values in case /etc/sysconfig/kdump doesn't include @@ -34,13 +30,25 @@ if [ -f /etc/sysconfig/kdump ]; then . /etc/sysconfig/kdump fi

+[[ $dracutbasedir ]] || dracutbasedir=/usr/lib/dracut +. $dracutbasedir/dracut-functions.sh +. /lib/kdump/kdump-lib.sh +. /lib/kdump/kdump-logger.sh

+#initiate the kdump logger +dlog_init +if [ $? -ne 0 ]; then

•   echo "failed to initiate the kdump logger."
  

•   exit 1
  

+fi

single_instance_lock() { local rc timeout=5

    exec 9>/var/lock/kdump
    if [ $? -ne 0 ]; then

•           echo "Create file lock failed"
  

•           derror "Create file lock failed"
            exit 1
    fi
  
  

@@ -48,7 +56,7 @@ single_instance_lock() rc=$?

    while [ $rc -ne 0 ]; do

•           echo "Another app is currently holding the kdump lock; waiting for it to exit..."
  

•           dinfo "Another app is currently holding the kdump lock; waiting for it to exit..."
            flock -w $timeout 9
            rc=$?
    done
  

@@ -59,9 +67,10 @@ determine_dump_mode() # Check if firmware-assisted dump is enabled # if yes, set the dump mode as fadump if is_fadump_capable; then

•           echo "Dump mode is fadump"
  

•           dinfo "Dump mode is fadump"
            DEFAULT_DUMP_MODE="fadump"
    fi
  

•   ddebug "DEFAULT_DUMP_MODE=$DEFAULT_DUMP_MODE"
  

}

save_core() @@ -69,22 +78,25 @@ save_core() coredir="/var/crash/`date +"%Y-%m-%d-%H:%M"`"

    mkdir -p $coredir

•   ddebug "cp --sparse=always /proc/vmcore $coredir/vmcore-incomplete"
    cp --sparse=always /proc/vmcore $coredir/vmcore-incomplete
    if [ $? == 0 ]; then
            mv $coredir/vmcore-incomplete $coredir/vmcore
  

•           echo "saved a vmcore to $coredir"
  

•           dinfo "saved a vmcore to $coredir"
    else
  

•           echo "failed to save a vmcore to $coredir" >&2
  

•           derror "failed to save a vmcore to $coredir"
    fi
  
    # pass the dmesg to Abrt tool if exists, in order
    # to collect the kernel oops message.
    # https://fedorahosted.org/abrt/
    if [ -x /usr/bin/dumpoops ]; then
  

•           ddebug "makedumpfile --dump-dmesg $coredir/vmcore $coredir/dmesg"
            makedumpfile --dump-dmesg $coredir/vmcore $coredir/dmesg >/dev/null 2>&1
  

•           ddebug "dumpoops -d $coredir/dmesg"
            dumpoops -d $coredir/dmesg >/dev/null 2>&1
            if [ $? == 0 ]; then
  

•                   echo "kernel oops has been collected by abrt tool"
  

•                   dinfo "kernel oops has been collected by abrt tool"
            fi
    fi
  

} @@ -96,16 +108,18 @@ rebuild_fadump_initrd() # this file tells the initrd is fadump enabled touch /tmp/fadump.initramfs target_initrd_tmp="$TARGET_INITRD.tmp"

•   ddebug "rebuild fadump initrd: $target_initrd_tmp $DEFAULT_INITRD_BAK $KDUMP_KERNELVER"
    $MKDUMPRD $target_initrd_tmp --rebuild $DEFAULT_INITRD_BAK --kver $KDUMP_KERNELVER \
            -i /tmp/fadump.initramfs /etc/fadump.initramfs
    if [ $? != 0 ]; then
  

•           echo "mkdumprd: failed to rebuild initrd with fadump support" >&2
  

•           derror "mkdumprd: failed to rebuild initrd with fadump support"
            rm -f /tmp/fadump.initramfs
            return 1
    fi
    rm -f /tmp/fadump.initramfs
  
    # updating fadump initrd
  

•   ddebug "updating fadump initrd: $target_initrd_tmp $TARGET_INITRD"
    mv $target_initrd_tmp $TARGET_INITRD
    sync
  
  

@@ -120,14 +134,15 @@ check_earlykdump_is_enabled()

rebuild_kdump_initrd() {

•   ddebug "rebuild kdump initrd: $MKDUMPRD $TARGET_INITRD $KDUMP_KERNELVER"
    $MKDUMPRD $TARGET_INITRD $KDUMP_KERNELVER
    if [ $? != 0 ]; then
  

•           echo "mkdumprd: failed to make kdump initrd" >&2
  

•           derror "mkdumprd: failed to make kdump initrd"
            return 1
    fi
  
    if check_earlykdump_is_enabled; then
  

•           echo "Tips: If early kdump is enabled, also require rebuilding the system initramfs to make the changes take effect for early kdump."
  

•           dwarn "Tips: If early kdump is enabled, also require rebuilding the system initramfs to make the changes take effect for early kdump."
    fi
  
    return 0
  

@@ -136,7 +151,7 @@ rebuild_kdump_initrd() rebuild_initrd() { if [[ ! -w "$KDUMP_BOOTDIR" ]];then

•           echo "$KDUMP_BOOTDIR does not have write permission. Can not rebuild $TARGET_INITRD"
  

•           derror "$KDUMP_BOOTDIR does not have write permission. Can not rebuild $TARGET_INITRD"
            return 1
    fi
  
  

@@ -154,7 +169,7 @@ check_exist() { for file in $1; do if [ ! -e "$file" ]; then

•                   echo -n "Error: $file not found."; echo
  

•                   derror "Error: $file not found."
                    return 1
            fi
    done
  

@@ -165,7 +180,7 @@ check_executable() { for file in $1; do if [ ! -x "$file" ]; then

•                   echo -n "Error: $file is not executable."; echo
  

•                   derror "Error: $file is not executable."
                    return 1
            fi
    done
  

@@ -173,17 +188,19 @@ check_executable()

backup_default_initrd() {

•   ddebug "backup default initrd: $DEFAULT_INITRD"
  

•   if [ ! -f "$DEFAULT_INITRD" ]; then
            return
    fi
  
    if [ ! -e $DEFAULT_INITRD_BAK ]; then
  

•           echo "Backing up $DEFAULT_INITRD before rebuild."
  

•           dinfo "Backing up $DEFAULT_INITRD before rebuild."
            # save checksum to verify before restoring
            sha1sum $DEFAULT_INITRD > $INITRD_CHECKSUM_LOCATION
            cp $DEFAULT_INITRD $DEFAULT_INITRD_BAK
            if [ $? -ne 0 ]; then
  

•                   echo "WARNING: failed to backup $DEFAULT_INITRD."
  

•                   dwarn "WARNING: failed to backup $DEFAULT_INITRD."
                    rm -f $DEFAULT_INITRD_BAK
            fi
    fi
  

@@ -191,6 +208,8 @@ backup_default_initrd()

restore_default_initrd() {

•   ddebug "restore default initrd: $DEFAULT_INITRD"
  

•   if [ ! -f "$DEFAULT_INITRD" ]; then
            return
    fi
  

@@ -202,13 +221,12 @@ restore_default_initrd() backup_checksum=`sha1sum $DEFAULT_INITRD_BAK | awk '{ print $1 }'` default_checksum=`cat $INITRD_CHECKSUM_LOCATION | awk '{ print $1 }'` if [ "$default_checksum" != "$backup_checksum" ]; then

•                   echo "WARNING: checksum mismatch! Can't restore original initrd.."
  

•                   dwarn "WARNING: checksum mismatch! Can't restore original initrd.."
            else
                    rm -f $INITRD_CHECKSUM_LOCATION
                    mv $DEFAULT_INITRD_BAK $DEFAULT_INITRD
                    if [[ $? -eq 0 ]]; then
  

•                           echo -n "Restoring original initrd as fadump mode "
  

•                           echo "is disabled."
  

•                           derror "Restoring original initrd as fadump mode is disabled."
                            sync
                    fi
            fi
  

@@ -221,27 +239,27 @@ check_config()

    nr=$(awk 'BEGIN{cnt=0} /^raw|^ssh[[:blank:]]|^nfs|^ext[234]|^xfs|^btrfs|^minix|^dracut_args .*\-\-mount/{cnt++} END{print cnt}' $KDUMP_CONFIG_FILE)
    [ $nr -gt 1 ] && {

•           echo "More than one dump targets specified."
  

•           derror "More than one dump targets specified."
            return 1
    }
  
    # Check if path option is set more than once.
     nr=$(awk 'BEGIN{cnt=0} /^path /{cnt++} END{print cnt}' $KDUMP_CONFIG_FILE)
     [ $nr -gt 1 ] && {
  

•            echo "Mutiple paths specifed in $KDUMP_CONFIG_FILE"
  

•            derror "Mutiple paths specifed in $KDUMP_CONFIG_FILE"
             return 1
     }
  
    nr=$(grep "^dracut_args .*\-\-mount" $KDUMP_CONFIG_FILE | grep -o "\-\-mount" | wc -l)
    [ $nr -gt 1 ] && {
  

•           echo "Multiple mount targets specified in one \"dracut_args\"."
  

•           derror "Multiple mount targets specified in one \"dracut_args\"."
            return 1
    }
  
    # Check if we have any leading spaces (or tabs) before the
    # variable name in the kdump conf file
    if grep -E -q '^[[:blank:]]+[a-z]' $KDUMP_CONFIG_FILE; then
  

•           echo "No whitespaces are allowed before a kdump option name in $KDUMP_CONFIG_FILE"
  

•           derror "No whitespaces are allowed before a kdump option name in $KDUMP_CONFIG_FILE"
            return 1
    fi
  
  

@@ -252,19 +270,19 @@ check_config() raw|ext2|ext3|ext4|minix|btrfs|xfs|nfs|ssh|sshkey|path|core_collector|kdump_post|kdump_pre|extra_bins|extra_modules|failure_action|default|final_action|force_rebuild|force_no_rebuild|dracut_args|fence_kdump_args|fence_kdump_nodes) # remove inline comments after the end of a directive. [ -z "$config_val" ] && {

•                           echo "Invalid kdump config value for option $config_opt."
  

•                           derror "Invalid kdump config value for option $config_opt."
                            return 1;
                    }
                    if [ -d "/proc/device-tree/ibm,opal/dump" ] && [ "$config_opt" == "raw" ]; then
  

•                           echo "WARNING: Won't capture opalcore when 'raw' dump target is used."
  

•                           dwarn "WARNING: Won't capture opalcore when 'raw' dump target is used."
                    fi
                    ;;
            net|options|link_delay|disk_timeout|debug_mem_level|blacklist)
  

•                   echo "Deprecated kdump config option: $config_opt. Refer to kdump.conf manpage for alternatives."
  

•                   derror "Deprecated kdump config option: $config_opt. Refer to kdump.conf manpage for alternatives."
                    return 1
                    ;;
            *)
  

•                   echo "Invalid kdump config option $config_opt"
  

•                   derror "Invalid kdump config option $config_opt"
                    return 1;
                    ;;
            esac
  

@@ -309,6 +327,7 @@ setup_initrd() { prepare_kdump_bootinfo if [ $? -ne 0 ]; then

•           derror "failed to prepare for kdump bootinfo."
            return 1
    fi
  
  

@@ -379,7 +398,7 @@ check_files_modified() else # If it's not a module nor builtin, give an error if ! ( modprobe --set-version "$KDUMP_KERNELVER" --dry-run "$_module" &>/dev/null ); then

•                                   echo "Module $_module not found"
  

•                                   dwarn "Module $_module not found"
                            fi
                    fi
            done
  

@@ -404,13 +423,12 @@ check_files_modified() fi fi else

•                   echo "$file doesn't exist"
  

•                   dwarn "$file doesn't exist"
            fi
    done
  
    if [ -n "$modified_files" ]; then
  

•           echo "Detected change(s) in the following file(s):"
  

•           echo -n "  "; echo "$modified_files" | sed 's/\s/\n  /g'
  

•           dinfo "Detected change(s) in the following file(s): $modified_files"
            return 1
    fi
  
  

@@ -448,13 +466,16 @@ check_dump_fs_modified() _target=$(to_dev_name $_target) _new_fstype=$(get_fs_type_from_target $_target) if [[ -z "$_target" || -z "$_new_fstype" ]];then

•                   echo "Dump path $_path does not exist"
  

•                   derror "Dump path $_path does not exist"
                    return 2
            fi
    fi
  
  

•   ddebug "_target=$_target _path=$_path _new_fstype=$_new_fstype"
  

•   _record_block_drivers() {
            local _drivers
  

•           if [[ -b /dev/block/$1 ]]; then
                    _drivers=$(udevadm info -a "/dev/block/$1" | sed -n 's/\s*DRIVERS=="\(\S\+\)"/\1/p')
            fi
  

@@ -466,6 +487,7 @@ check_dump_fs_modified() _target_drivers="$_target_drivers $_driver" fi done

•           ddebug "MAJ:MIN=$1 _drivers=$_drivers _target_drivers=$_targer_drivers"
            return 1
    }
  
  

@@ -478,7 +500,7 @@ check_dump_fs_modified() continue fi if ! [[ " $_old_drivers " == *" $_module_name "* ]]; then

•                   echo "Detected change in block device driver, new loaded module: $_module_name"
  

•                   dinfo "Detected change in block device driver, new loaded module: $_module_name"
                    return 1
            fi
    done
  

@@ -488,7 +510,7 @@ check_dump_fs_modified() else _new_dev=$(kdump_get_persistent_dev $_target) if [ -z "$_new_dev" ]; then

•                   echo "Get persistent device name failed"
  

•                   derror "Get persistent device name failed"
                    return 2
            fi
    fi
  

@@ -496,7 +518,7 @@ check_dump_fs_modified() _new_mntpoint="$(get_kdump_mntpoint_from_target $_target)" _dracut_args=$(lsinitrd $TARGET_INITRD -f usr/lib/dracut/build-parameter.txt) if [[ -z "$_dracut_args" ]];then

•           echo "Warning: No dracut arguments found in initrd"
  

•           dwarn "Warning: No dracut arguments found in initrd"
            return 0
    fi
  
  

@@ -514,7 +536,7 @@ check_dump_fs_modified() [[ "$_target" = "$(get_root_fs_device)" ]] && return 0 fi

•   echo "Detected change in File System"
  

•   dinfo "Detected change in File System"
    return 1
  

}

@@ -597,7 +619,7 @@ check_system_modified()

    check_wdt_modified
    if [ $? -ne 0 ]; then

•           echo "Detected change in watchdog state"
  

•           dinfo "Detected change in watchdog state"
            return 1
    fi
  
  

@@ -621,7 +643,7 @@ check_rebuild() if [ $? -eq 0 ]; then force_no_rebuild=`echo $_force_no_rebuild | cut -d' ' -f2` if [ "$force_no_rebuild" != "0" ] && [ "$force_no_rebuild" != "1" ];then

•                   echo "Error: force_no_rebuild value is invalid"
  

•                   derror "Error: force_no_rebuild value is invalid"
                    return 1
            fi
    fi
  

@@ -630,13 +652,13 @@ check_rebuild() if [ $? -eq 0 ]; then force_rebuild=`echo $_force_rebuild | cut -d' ' -f2` if [ "$force_rebuild" != "0" ] && [ "$force_rebuild" != "1" ];then

•                   echo "Error: force_rebuild value is invalid"
  

•                   derror "Error: force_rebuild value is invalid"
                    return 1
            fi
    fi
  
    if [[ "$force_no_rebuild" == "1" && "$force_rebuild" == "1" ]]; then
  

•           echo "Error: force_rebuild and force_no_rebuild are enabled simultaneously in kdump.conf"
  

•           derror "Error: force_rebuild and force_no_rebuild are enabled simultaneously in kdump.conf"
            return 1
    fi
  
  

@@ -666,18 +688,18 @@ check_rebuild() fi

    if [ $image_time -eq 0 ]; then

•           echo  -n "No kdump initial ramdisk found."; echo
  

•           dinfo "No kdump initial ramdisk found."
    elif [ "$capture_capable_initrd" == "0" ]; then
  

•           echo -n "Rebuild $TARGET_INITRD with dump capture support"; echo
  

•           dinfo "Rebuild $TARGET_INITRD with dump capture support"
    elif [ "$force_rebuild" != "0" ]; then
  

•           echo -n "Force rebuild $TARGET_INITRD"; echo
  

•           dinfo "Force rebuild $TARGET_INITRD"
    elif [ "$system_modified" != "0" ]; then
            :
    else
            return 0
    fi
  
  

•   echo "Rebuilding $TARGET_INITRD"
  

•   dinfo "Rebuilding $TARGET_INITRD"
    rebuild_initrd
    return $?
  

} @@ -694,18 +716,20 @@ load_kdump() # Old syscall will always fail as it does not have capability to # to kernel signature verification. if is_secure_boot_enforced; then

•           echo "Secure Boot is enabled. Using kexec file based syscall."
  

•           dinfo "Secure Boot is enabled. Using kexec file based syscall."
            KEXEC_ARGS="$KEXEC_ARGS -s"
    fi
  
  

•   ddebug "$KEXEC $KEXEC_ARGS $standard_kexec_args --command-line=$KDUMP_COMMANDLINE --initrd=$TARGET_INITRD $KDUMP_KERNEL"
  

•   $KEXEC $KEXEC_ARGS $standard_kexec_args \
            --command-line="$KDUMP_COMMANDLINE" \
            --initrd=$TARGET_INITRD $KDUMP_KERNEL
    if [ $? == 0 ]; then
  

•           echo "kexec: loaded kdump kernel"
  

•           dinfo "kexec: loaded kdump kernel"
            return 0
    else
  

•           echo "kexec: failed to load kdump kernel" >&2
  

•           derror "kexec: failed to load kdump kernel"
            return 1
    fi
  

} @@ -720,7 +744,7 @@ check_ssh_config() # canonicalize the path SSH_KEY_LOCATION=$(/usr/bin/readlink -m $config_val) else

•                           echo "WARNING: '$config_val' doesn't exist, using default value '$SSH_KEY_LOCATION'"
  

•                           dwarn "WARNING: '$config_val' doesn't exist, using default value '$SSH_KEY_LOCATION'"
                    fi
                    ;;
            path)
  

@@ -762,22 +786,22 @@ check_and_wait_network_ready() if [ $retval -eq 0 ]; then return 0 elif [ $retval -ne 255 ]; then

•                   echo "Could not create $DUMP_TARGET:$SAVE_PATH, you should check the privilege on server side"  >&2
  

•                   derror "Could not create $DUMP_TARGET:$SAVE_PATH, you should check the privilege on server side"
                    return 1
            fi
  
            # if server removes the authorized_keys or, no /root/.ssh/kdump_id_rsa
  

•           echo $errmsg | grep -q "Permission denied\|No such file or directory\|Host key verification failed"
  

•           ddebug "$errmsg"
  

•           echo $errmsg | grep -q "Permission denied\|No such file or directory\|Host key verification failed" &> /dev/null
            if [ $? -eq 0 ]; then
  

•                   echo "Could not create $DUMP_TARGET:$SAVE_PATH, you probably need to run \"kdumpctl propagate\""  >&2
  

•                   derror "Could not create $DUMP_TARGET:$SAVE_PATH, you probably need to run \"kdumpctl propagate\""
                    return 1
            fi
  
            if [ $warn_once -eq 1 ]; then
  

•                   echo "Network dump target is not usable, waiting for it to be ready"
  

•                   dwarn "Network dump target is not usable, waiting for it to be ready..."
                    warn_once=0
            fi
  

•           echo -n .
  
            cur=$(date +%s)
            let "diff = $cur - $start_time"
  

@@ -788,7 +812,7 @@ check_and_wait_network_ready() sleep 1 done

•   echo "Could not create $DUMP_TARGET:$SAVE_PATH, ipaddr is not ready yet. You should check network connection"  >&2
  

•   dinfo "Could not create $DUMP_TARGET:$SAVE_PATH, ipaddr is not ready yet. You should check network connection"
    return 1
  

}

@@ -805,7 +829,7 @@ propagate_ssh_key() { check_ssh_config if [ $? -ne 0 ]; then

•           echo "No ssh config specified in $KDUMP_CONFIG_FILE.  Can't propagate" >&2
  

•           derror "No ssh config specified in $KDUMP_CONFIG_FILE.  Can't propagate"
            exit 1
    fi
  
  

@@ -814,11 +838,11 @@ propagate_ssh_key()

    #Check to see if we already created key, if not, create it.
    if [ -f $KEYFILE ]; then

•           echo "Using existing keys..."
  

•           dinfo "Using existing keys..."
    else
  

•           echo -n "Generating new ssh keys... "
  

•           dinfo "Generating new ssh keys... "
            /usr/bin/ssh-keygen -t rsa -f $KEYFILE -N "" 2>&1 > /dev/null
  

•           echo "done."
  

•           dinfo "done."
    fi
  
    #now find the target ssh user and server to contact.
  

@@ -829,10 +853,10 @@ propagate_ssh_key() ssh-copy-id -i $KEYFILE $SSH_USER@$SSH_SERVER RET=$? if [ $RET == 0 ]; then

•           echo $KEYFILE has been added to ~$SSH_USER/.ssh/authorized_keys on $SSH_SERVER
  

•           dinfo "$KEYFILE has been added to ~$SSH_USER/.ssh/authorized_keys on $SSH_SERVER"
            return 0
    else
  

•           echo $errmsg, $KEYFILE failed in transfer to $SSH_SERVER  >&2
  

•           derror "$errmsg, $KEYFILE failed in transfer to $SSH_SERVER"
            exit 1
    fi
  

} @@ -842,7 +866,7 @@ show_reserved_mem() local mem=$(cat /sys/kernel/kexec_crash_size) local mem_mb=$(expr $mem / 1024 / 1024)

• echo "Reserved "$mem_mb"MB memory for crash kernel"

• dinfo "Reserved "$mem_mb"MB memory for crash kernel"

}

check_current_fadump_status() @@ -872,12 +896,12 @@ save_raw() raw_target=$(awk '$1 ~ /^raw$/ { print $2; }' $KDUMP_CONFIG_FILE) [ -z "$raw_target" ] && return 0 [ -b "$raw_target" ] || {

•           echo "raw partition $raw_target not found"
  

•           derror "raw partition $raw_target not found"
            return 1
    }
    check_fs=$(lsblk --nodeps -npo FSTYPE $raw_target)
    if [[ $(echo $check_fs | wc -w) -ne 0 ]]; then
  

•           echo "Warning: Detected '$check_fs' signature on $raw_target, data loss is expected."
  

•           dwarn "Warning: Detected '$check_fs' signature on $raw_target, data loss is expected."
            return 0
    fi
    kdump_dir=`grep ^path $KDUMP_CONFIG_FILE | cut -d' '  -f2-`
  

@@ -889,12 +913,12 @@ save_raw()

    mkdir -p "$coredir"
    [ -d "$coredir" ] || {

•           echo "failed to create $coredir"
  

•           derror "failed to create $coredir"
            return 1
    }
    if makedumpfile -R $coredir/vmcore <$raw_target >/dev/null 2>&1; then
            # dump found
  

•           echo "Dump saved to $coredir/vmcore"
  

•           dinfo "Dump saved to $coredir/vmcore"
            # wipe makedumpfile header
            dd if=/dev/zero of=$raw_target bs=1b count=1 2>/dev/null
    else
  

@@ -967,13 +991,13 @@ check_fence_kdump_config()

    for node in $nodes; do
            if [ "$node" = "$hostname" ]; then

•                   echo "Option fence_kdump_nodes cannot contain $hostname"
  

•                   derror "Option fence_kdump_nodes cannot contain $hostname"
                    return 1
            fi
            # node can be ipaddr
            echo $ipaddrs | grep $node > /dev/null
            if [ $? -eq 0 ]; then
  

•                   echo "Option fence_kdump_nodes cannot contain $node"
  

•                   derror "Option fence_kdump_nodes cannot contain $node"
                    return 1
            fi
    done
  

@@ -995,11 +1019,11 @@ start_fadump() { echo 1 > $FADUMP_REGISTER_SYS_NODE if ! check_current_fadump_status; then

•           echo "fadump: failed to register"
  

•           derror "fadump: failed to register"
            return 1
    fi
  
  

•   echo "fadump: registered successfully"
  

•   dinfo "fadump: registered successfully"
    return 0
  

}

@@ -1026,7 +1050,7 @@ check_failure_action_config() if [ -z "$failure_action" -a -z "$default_option" ]; then return 0 elif [ -n "$failure_action" -a -n "$default_option" ]; then

•           echo "Cannot specify 'failure_action' and 'default' option together"
  

•           derror "Cannot specify 'failure_action' and 'default' option together"
            return 1
    fi
  
  

@@ -1040,7 +1064,7 @@ check_failure_action_config() return 0 ;; *)

•           echo $"Usage kdump.conf: $option {reboot|halt|poweroff|shell|dump_to_rootfs}"
  

•           dinfo $"Usage kdump.conf: $option {reboot|halt|poweroff|shell|dump_to_rootfs}"
            return 1
    esac
  

} @@ -1058,7 +1082,7 @@ check_final_action_config() return 0 ;; *)

•                   echo $"Usage kdump.conf: final_action {reboot|halt|poweroff}"
  

•                   dinfo $"Usage kdump.conf: final_action {reboot|halt|poweroff}"
                    return 1
            esac
    fi
  

@@ -1068,13 +1092,13 @@ start() { check_dump_feasibility if [ $? -ne 0 ]; then

•           echo "Starting kdump: [FAILED]"
  

•           derror "Starting kdump: [FAILED]"
            return 1
    fi
  
    check_config
    if [ $? -ne 0 ]; then
  

•           echo "Starting kdump: [FAILED]"
  

•           derror "Starting kdump: [FAILED]"
            return 1
    fi
  
  

@@ -1084,43 +1108,43 @@ start()

    save_raw
    if [ $? -ne 0 ]; then

•           echo "Starting kdump: [FAILED]"
  

•           derror "Starting kdump: [FAILED]"
            return 1
    fi
  
    check_current_status
    if [ $? == 0 ]; then
  

•           echo "Kdump already running: [WARNING]"
  

•           dwarn "Kdump already running: [WARNING]"
            return 0
    fi
  
    if check_ssh_config; then
            if ! check_ssh_target; then
  

•                   echo "Starting kdump: [FAILED]"
  

•                   derror "Starting kdump: [FAILED]"
                    return 1
            fi
    fi
  
    check_rebuild
    if [ $? != 0 ]; then
  

•           echo "Starting kdump: [FAILED]"
  

•           derror "Starting kdump: [FAILED]"
            return 1
    fi
  
    start_dump
    if [ $? != 0 ]; then
  

•           echo "Starting kdump: [FAILED]"
  

•           derror "Starting kdump: [FAILED]"
            return 1
    fi
  
  

•   echo "Starting kdump: [OK]"
  

•   dinfo "Starting kdump: [OK]"
  

}

reload() { check_current_status if [ $? -ne 0 ]; then

•           echo "Kdump was not running: [WARNING]"
  

•           dwarn "Kdump was not running: [WARNING]"
    fi
  
    if [ $DEFAULT_DUMP_MODE == "fadump" ]; then
  

@@ -1131,36 +1155,36 @@ reload() fi

    if [ $? -ne 0 ]; then

•           echo "Stopping kdump: [FAILED]"
  

•           derror "Stopping kdump: [FAILED]"
            return 1
    fi
  
  

•   echo "Stopping kdump: [OK]"
  

•   dinfo "Stopping kdump: [OK]"
  
    setup_initrd
    if [ $? -ne 0 ]; then
  

•           echo "Starting kdump: [FAILED]"
  

•           derror "Starting kdump: [FAILED]"
            return 1
    fi
  
    start_dump
    if [ $? -ne 0 ]; then
  

•           echo "Starting kdump: [FAILED]"
  

•           derror "Starting kdump: [FAILED]"
            return 1
    fi
  
  

•   echo "Starting kdump: [OK]"
  

•   dinfo "Starting kdump: [OK]"
  

}

stop_fadump() { echo 0 > $FADUMP_REGISTER_SYS_NODE if check_current_fadump_status; then

•           echo "fadump: failed to unregister"
  

•           derror "fadump: failed to unregister"
            return 1
    fi
  
  

•   echo "fadump: unregistered successfully"
  

•   dinfo "fadump: unregistered successfully"
    return 0
  

}

@@ -1173,11 +1197,11 @@ stop_kdump() fi

    if [ $? != 0 ]; then

•           echo "kexec: failed to unload kdump kernel"
  

•           derror "kexec: failed to unload kdump kernel"
            return 1
    fi
  
  

•   echo "kexec: unloaded kdump kernel"
  

•   dinfo "kexec: unloaded kdump kernel"
    return 0
  

}

@@ -1185,7 +1209,7 @@ reload_fadump() { echo 1 > $FADUMP_REGISTER_SYS_NODE if [ $? == 0 ]; then

•           echo "fadump: re-registered successfully"
  

•           dinfo "fadump: re-registered successfully"
            return 0
    else
            # FADump could fail on older kernel where re-register
  

@@ -1210,11 +1234,11 @@ stop() fi

    if [ $? != 0 ]; then

•           echo "Stopping kdump: [FAILED]"
  

•           derror "Stopping kdump: [FAILED]"
            return 1
    fi
  
  

•   echo "Stopping kdump: [OK]"
  

•   dinfo "Stopping kdump: [OK]"
    return 0
  

}

@@ -1235,13 +1259,13 @@ rebuild() { return 1 fi

•   echo "Rebuilding $TARGET_INITRD"
  

•   dinfo "Rebuilding $TARGET_INITRD"
    rebuild_initrd
    return $?
  

}

if [ ! -f "$KDUMP_CONFIG_FILE" ]; then

•   echo "Error: No kdump config file found!"  >&2
  

•   derror "Error: No kdump config file found!"
    exit 1
  

fi

@@ -1267,11 +1291,11 @@ main () check_current_status case "$?" in 0)

•                   echo "Kdump is operational"
  

•                   dinfo "Kdump is operational"
                    EXIT_CODE=0
                    ;;
              1)
  

•                   echo "Kdump is not operational"
  

•                   dinfo "Kdump is not operational"
                    EXIT_CODE=3
                    ;;
            esac
  

@@ -1296,7 +1320,7 @@ main () show_reserved_mem ;; *)

•           echo $"Usage: $0 {start|stop|status|restart|reload|rebuild|propagate|showmem}"
  

•           dinfo $"Usage: $0 {start|stop|status|restart|reload|rebuild|propagate|showmem}"
            exit 1
    esac
  

} diff --git a/kexec-tools.spec b/kexec-tools.spec index 669fd6cf4022..f6c012984f9b 100644 --- a/kexec-tools.spec +++ b/kexec-tools.spec @@ -37,6 +37,7 @@ Source27: early-kdump-howto.txt Source28: kdump-udev-throttler Source29: kdump.sysconfig.aarch64 Source30: 60-kdump.install +Source31: kdump-logger.sh

####################################### # These are sources for mkdumpramfs @@ -190,6 +191,7 @@ install -m 644 %{SOURCE12} $RPM_BUILD_ROOT%{_mandir}/man8/mkdumprd.8 install -m 644 %{SOURCE25} $RPM_BUILD_ROOT%{_mandir}/man8/kdumpctl.8 install -m 755 %{SOURCE20} $RPM_BUILD_ROOT%{_prefix}/lib/kdump/kdump-lib.sh install -m 755 %{SOURCE23} $RPM_BUILD_ROOT%{_prefix}/lib/kdump/kdump-lib-initramfs.sh +install -m 755 %{SOURCE31} $RPM_BUILD_ROOT%{_prefix}/lib/kdump/kdump-logger.sh %ifnarch s390x install -m 755 %{SOURCE28} $RPM_BUILD_ROOT%{_udevrulesdir}/../kdump-udev-throttler %endif diff --git a/mkdumprd b/mkdumprd index 36f71b7056c2..5f32a00382ce 100644 --- a/mkdumprd +++ b/mkdumprd

I think we need to call dlog_init again here in mkdumprd, and include the source file too? mkdumprd could be called by user directly.

OK, we also need to source the configuration '/etc/sysconfig/kdump' in mkdumprd like this:

+. /etc/sysconfig/kdump . /lib/kdump/kdump-lib.sh export IN_KDUMP=1

Thanks. Lianbo

...

@@ -180,8 +180,8 @@ check_size() { fi

 if [ $avail -lt $memtotal ]; then

•    echo "Warning: There might not be enough space to save a vmcore."
  

•    echo "         The size of $2 should be greater than $memtotal kilo bytes."
  

•    dwarn "Warning: There might not be enough space to save a vmcore."
  

•    dwarn "         The size of $2 should be greater than $memtotal kilo bytes."
  

  fi

}

@@ -253,7 +253,7 @@ verify_core_collector() {

 if [ "$_cmd" != "makedumpfile" ]; then
     if is_raw_dump_target; then

•        echo "Warning: specifying a non-makedumpfile core collector, you will have to recover the vmcore manually."
  

•        dwarn "Warning: specifying a non-makedumpfile core collector, you will have to recover the vmcore manually."
     fi
     return
  

  fi

@@ -343,7 +343,7 @@ is_unresettable() resettable="$(cat $path)" [ $resettable -eq 0 -a "$OVERRIDE_RESETTABLE" -eq 0 ] && { local device=$(udevadm info --query=all --path=/sys/dev/block/$1 | awk -F= '/DEVNAME/{print $2}')

•        echo "Error: Can not save vmcore because device $device is unresettable"
  

•        derror "Error: Can not save vmcore because device $device is unresettable"
         return 0
     }
  

  fi

@@ -377,7 +377,7 @@ is_crypt() eval "$line" [[ "$ID_FS_TYPE" = "crypto_LUKS" ]] && { dev=$(udevadm info --query=all --path=/sys/dev/block/$majmin | awk -F= '/DEVNAME/{print $2}')

•    echo "Device $dev is encrypted."
  

•    derror "Device $dev is encrypted."
     return 0
  

  } return 1

@@ -400,7 +400,7 @@ if ! check_resettable; then fi

if ! check_crypt; then

• echo "Warning: Encrypted device is in dump path, which is not recommended, see kexec-kdump-howto.txt for more details."

• dwarn "Warning: Encrypted device is in dump path, which is not recommended, see kexec-kdump-howto.txt for more details."

fi

# firstly get right SSH_KEY_LOCATION

2.17.1

-- Best Regards, Kairui Song

在 2020年10月26日 16:06, Kairui Song 写道:

On Tue, Oct 20, 2020 at 10:46 AM Lianbo Jiang lijiang@redhat.com wrote:

...

Currently, the kexec option '--debug/-d' is not enabled by default, which means that users need to set it manually and wait for the next failure to capture the additional information.

Therefore, let's enable the option '-d' for kexec loading by default.

Signed-off-by: Lianbo Jiang lijiang@redhat.com

kdumpctl | 17 +++++++++++++++-- 1 file changed, 15 insertions(+), 2 deletions(-)

diff --git a/kdumpctl b/kdumpctl index c61774364709..7ed64b3d7399 100755 --- a/kdumpctl +++ b/kdumpctl @@ -6,6 +6,7 @@ KDUMP_KERNEL="" KDUMP_COMMANDLINE="" KEXEC_ARGS="" KDUMP_CONFIG_FILE="/etc/kdump.conf" +KDUMP_LOG_PATH="/var/log" MKDUMPRD="/sbin/mkdumprd -f" DRACUT_MODULES_FILE="/usr/lib/dracut/modules.txt" SAVE_PATH=/var/crash @@ -21,7 +22,7 @@ FADUMP_REGISTER_SYS_NODE="/sys/kernel/fadump_registered" DEFAULT_DUMP_MODE="kdump" image_time=0

-standard_kexec_args="-p" +standard_kexec_args="-d -p"

# Some default values in case /etc/sysconfig/kdump doesn't include KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug" @@ -709,6 +710,8 @@ check_rebuild() # as the currently running kernel. load_kdump() {

•   local ret
  

•   KEXEC_ARGS=$(prepare_kexec_args "${KEXEC_ARGS}")
    KDUMP_COMMANDLINE=$(prepare_cmdline "${KDUMP_COMMANDLINE}" "${KDUMP_COMMANDLINE_REMOVE}" "${KDUMP_COMMANDLINE_APPEND}")
  
  

@@ -722,10 +725,20 @@ load_kdump()

    ddebug "$KEXEC $KEXEC_ARGS $standard_kexec_args --command-line=$KDUMP_COMMANDLINE --initrd=$TARGET_INITRD $KDUMP_KERNEL"

•   exec 12>&2
  

•   exec 2>> $KDUMP_LOG_PATH/kdump.log
  

•   PS4='+ $(date "+%Y-%m-%d %H:%M:%S") ${BASH_SOURCE}@${LINENO}: '
  

•   set -x
  

•   $KEXEC $KEXEC_ARGS $standard_kexec_args \
            --command-line="$KDUMP_COMMANDLINE" \
            --initrd=$TARGET_INITRD $KDUMP_KERNEL
  

Can we just reuse the dlog functions here? LIke using process substitution: $KEXEC <args> 2> >(derror) > >(ddebug)

I tried to use the above ways according to your suggestions, but it will output the log to console(terminal), and it doesn't save the log to a file. That is not the result what we expected. Or did I miss anything about this?

Thanks. Lianbo

...

•   if [ $? == 0 ]; then
  

•   ret=$?
  

•   set +x
  

•   exec 2>&12 12>&-
  

•   if [ $ret == 0 ]; then
            dinfo "kexec: loaded kdump kernel"
            return 0
    else
  

-- 2.17.1

-- Best Regards, Kairui Song

在 2020年10月26日 16:04, Kairui Song 写道:

Hi Lianbo, thanks for the work, the log infrastruct will be improved by a log with this patch series.

On Tue, Oct 20, 2020 at 10:45 AM Lianbo Jiang lijiang@redhat.com wrote:

...

Sometimes, debugging the kdump service failure becomes very challenging because there is no complete debugging information, which requires modification of the options or the scripts like kdumpctl, mkdumprd, etc to collect the information for troubleshooting.

That means users have to wait for the next failure so that they can capture the additional information, which could waste valuable time.

This patch series will improve kdump debugging messages and have a chance to save them to journald or a file. It includes the following patches: [1] [PATCH 1/5] introduce the kdump logger from the dracut [2] [PATCH 2/5] enable the logger for kdump [3] [PATCH 3/5] kdump.sysconfig: add the kdump logger configurations [4] [PATCH 4/5] kdumpctl: add the '-d' option to enable the kexec loading debugging messages [5] [PATCH 5/5] Improve debugging in the kdump kernel [6] [PATCH 6/6] Doc: add a documentation for the usage of logger

In addition, this patch series won't cause performance degradation, basically this effect can be ignored. I simply checked the result of executing the loading command as follow:

[1] before # time kdumpctl start Detected change(s) in the following file(s): /etc/kdump.conf Rebuilding /boot/initramfs-4.18.0-232.el8.x86_64kdump.img Tips: If early kdump is enabled, also require rebuilding the system initramfs to make the changes take effect for early kdump. kexec: loaded kdump kernel Starting kdump: [OK]

real 0m9.145s user 0m20.733s sys 0m1.765s

[2] after # time kdumpctl start kdump: Detected change(s) in the following file(s): /etc/kdump.conf kdump: Rebuilding /boot/initramfs-4.18.0-232.el8.x86_64kdump.img kdump: Tips: If early kdump is enabled, also require rebuilding the system initramfs to make the changes take effect for early kdump. kdump: kexec: loaded kdump kernel kdump: Starting kdump: [OK]

real 0m9.177s user 0m20.784s sys 0m1.901s

Changes since v5: [1] Only added a documentation for the usage of logger, there's no code change.

Note: Currently, kexec-tools disable storing logs to journald in order to save memory, please refer to the following changes: +++ b/dracut-module-setup.sh @@ -788,6 +788,8 @@ kdump_install_random_seed() { } ... kdump_install_systemd_conf() { if ! is_fadump_capable && [ "$failure_action" != "shell" ]; then mkdir -p ${initdir}/etc/systemd/journald.conf.d echo "[Journal]" > ${initdir}/etc/systemd/journald.conf.d/kdump.conf echo "Storage=none" >> ${initdir}/etc/systemd/journald.conf.d/kdump.conf ^^^^ If we want to use the journald in kdump kernel, we have to change it to "Storage=auto" or "volatile".

For the log storage in kdump kernel, I'm thinking maybe we have Storage=none and ForwardToKmsg=yes, so kmsg will be used as the sole log storage in kdump kernel, which looks a good balance of keeping the logs and reduce memory usage. How do you think? If you want to keep

We can get more additional logs from journald, but some logs are not in the kmsg. For example:

kmsg(/var/crash/127.0.0.1-2020-10-26-22:38:26/kexec-dmesg.log): -------------------------------------------------------------- ... [ 16.854646] kdump[387] saving to /sysroot/var/crash/127.0.0.1-2020-10-26-22:38:26/ [ 16.904723] kdump[387] saving vmcore-dmesg.txt to /sysroot/var/crash/127.0.0.1-2020-10-26-22:38:26/ [ 16.986037] kdump[387] saving vmcore-dmesg.txt complete [ 16.991214] kdump[387] saving vmcore "/var/crash/127.0.0.1-2020-10-26-22:38:26/kexec-dmesg.log" 772L, 54046B 772,1 Bot

journald(/var/crash/127.0.0.1-2020-10-26-22:44:01/kexec-dmesg.log): ------------------------------------------------------------------ ... Oct 26 22:44:01 dell-pec5125-03.khw.lab.eng.bos.redhat.com systemd[1]: Starting Kdump Vmcore Save Service... Oct 26 22:44:02 dell-pec5125-03.khw.lab.eng.bos.redhat.com kdump.sh[387]: kdump: saving to /sysroot/var/crash/127.0.0.1-2020-10-26-22:44:01/ Oct 26 22:44:02 dell-pec5125-03.khw.lab.eng.bos.redhat.com unknown: kdump[387] saving to /sysroot/var/crash/127.0.0.1-2020-10-26-22:44:01/ Oct 26 22:44:02 dell-pec5125-03.khw.lab.eng.bos.redhat.com kdump.sh[387]: kdump: saving vmcore-dmesg.txt to /sysroot/var/crash/127.0.0.1-2020-10-26-22:44:01/ Oct 26 22:44:02 dell-pec5125-03.khw.lab.eng.bos.redhat.com unknown: kdump[387] saving vmcore-dmesg.txt to /sysroot/var/crash/127.0.0.1-2020-10-26-22:44:01/ Oct 26 22:44:02 dell-pec5125-03.khw.lab.eng.bos.redhat.com kdump.sh[387]: kdump: saving vmcore-dmesg.txt complete Oct 26 22:44:02 dell-pec5125-03.khw.lab.eng.bos.redhat.com unknown: kdump[387] saving vmcore-dmesg.txt complete Oct 26 22:44:02 dell-pec5125-03.khw.lab.eng.bos.redhat.com kdump.sh[387]: kdump: saving vmcore Oct 26 22:44:02 dell-pec5125-03.khw.lab.eng.bos.redhat.com unknown: kdump[387] saving vmcore Oct 26 22:44:04 dell-pec5125-03.khw.lab.eng.bos.redhat.com kdump.sh[445]: ^MChecking for memory holes : [ 0.0 %] / ^MChecking for memory holes : [100.0 %] | ^MExcluding unnecessary pages : [100.0 %] \ ^MCopying data : [ 1.4 %] - eta: 2s^MCopying data : [ 52.9 %] / eta: 0s^MCopying data : [100.0 %] | eta: 0s^MCopying data : [100.0 %] \ eta: 0s Oct 26 22:44:04 dell-pec5125-03.khw.lab.eng.bos.redhat.com kdump.sh[445]: The kernel version is not supported. Oct 26 22:44:04 dell-pec5125-03.khw.lab.eng.bos.redhat.com kdump.sh[445]: The makedumpfile operation may be incomplete. Oct 26 22:44:04 dell-pec5125-03.khw.lab.eng.bos.redhat.com kdump.sh[445]: The dumpfile is saved to /sysroot/var/crash/127.0.0.1-2020-10-26-22:44:01//vmcore-incomplete. Oct 26 22:44:04 dell-pec5125-03.khw.lab.eng.bos.redhat.com kdump.sh[445]: makedumpfile Completed. "/var/crash/127.0.0.1-2020-10-26-22:44:01/kexec-dmesg.log" 858L, 106493B

more log, just increase log_bufsize, simple and easy.

If we manually increase the size of log_buf, which indicates that we have to wait for the next failure to capture the additional information.

In addition, increasing the size of log_buf still increases memory usage, and the log_buf is easy to be overwritten if the size is not enough.

Maybe we can have a is_kdump_kernel helper in kdump-libs.sh, to judge if it's kdump kernel.

Also, how about have a standalong log level like kdump_dumploglvl,

I would not recommend having a standalong log level for kdump. Currently, we can use the following configurations to achieve the same purpose. kdump_sysloglvl=0 kdump_stdloglvl=4 kdump_kmsgloglvl=4

Thanks. Lianbo

aside from "kdump_stdloglvl kdump_sysloglvl kdump_kmsgloglvl". kdump_dumploglvl could be used as the only loglevel in kdump kernel, and all log content just goes to logger (journalctl, it will forward log to console, and store log in kmsg), as I think ideally, there is only one log output/storage in kdump kernel anyway (log is always printed to console, and stored in kmsg). Then to save the log in kdump kernel, just use dmsg.

I've also added some comment in other individual patches.

...

Lianbo Jiang (6): introduce the kdump logger from the dracut enable the logger for kdump kdump.sysconfig: add the kdump logger configurations kdumpctl: add the '-d' option to enable the kexec loading debugging messages Improve debugging in the kdump kernel Doc: add a documentation for the usage of logger

dracut-early-kdump-module-setup.sh | 3 + dracut-early-kdump.sh | 24 +- dracut-kdump.sh | 68 +++--- dracut-module-setup.sh | 4 + kdump-lib-initramfs.sh | 67 ++++-- kdump-lib.sh | 23 +- kdump-logger.sh | 344 +++++++++++++++++++++++++++++ kdump.sysconfig | 16 ++ kdump.sysconfig.aarch64 | 16 ++ kdump.sysconfig.i386 | 16 ++ kdump.sysconfig.ppc64 | 16 ++ kdump.sysconfig.ppc64le | 16 ++ kdump.sysconfig.s390x | 16 ++ kdump.sysconfig.x86_64 | 16 ++ kdumpctl | 241 +++++++++++--------- kexec-kdump-howto.txt | 35 +++ kexec-tools.spec | 2 + mkdumprd | 12 +- 18 files changed, 766 insertions(+), 169 deletions(-) create mode 100755 kdump-logger.sh

-- 2.17.1

-- Best Regards, Kairui Song