On 05/10/2017 at 09:54 AM, Hatayama, Daisuke wrote:
Pang,
Thanks for cc'ing to me.
> -----Original Message-----
> From: Xunlei Pang [mailto:xlpang@redhat.com]
> Sent: Tuesday, May 9, 2017 8:52 PM
> To: kexec(a)lists.fedoraproject.org
> Cc: Xunlei Pang <xlpang(a)redhat.com>
> Subject: [PATCH] kdumpctl: use "apicid" other than "initial
apicid"
>
> We met a problem on AMD machines, when using "nr_cpus=4" for
> kdump, and crash happens on cpus other than cpu0, kdump kernel
> will fail to boot and eventually reset.
>
> After some debugging, we found that it stuck at the kernel path
> do_boot_cpu()-> ... ->wakeup_secondary_cpu_via_init():
> apic_icr_write(APIC_INT_LEVELTRIG|APIC_INT_ASSERT|APIC_DM_INIT,
> phys_apicid);
> that is, it stuck at sending INIT from AP to BP and reset, which
> is actually what "disable_cpu_apicid=X" tries to solve. Printing
> the value of @phys_apicid showed that it was the value of "apicid"
> other that of "initial apicid" showed by /proc/cpuinfo.
>
> As described in x86 specification:
> "In MP systems, the local APIC ID is also used as a processor ID by the
> BIOS and the operating system. Some processors permit software to modify
> the APIC ID. However, the ability of software to modify the APIC ID is
> processor model specific. Because of this, operating system software
> should avoid writing to the local APIC ID register. The value returned by
> bits 31-24 of the EBX register (when the CPUID instruction is executed with
> a
> source operand value of 1 in the EAX register) is always the Initial APIC ID
> (determined by the platform initialization). This is true even if software
> has changed the value in the Local APIC ID register."
>
> From kernel commit 151e0c7de("x86, apic, kexec: Add disable_cpu_apicid
> kernel parameter"), we can see in generic_processor_info(), it uses
> a)read_apic_id() and b)@apicid to compare with @disabled_cpu_apicid.
>
> a)@apicid which is actually @phys_apicid above-mentioned is from the
> following calltrace(on the problematic AMD machine):
> generic_processor_info+0x37/0x300
> acpi_register_lapic+0x30/0x90
> acpi_parse_lapic+0x40/0x50
> acpi_table_parse_entries_array+0x171/0x1de
> acpi_boot_init+0xed/0x50f
> The value of @apicid(from acpi MADT) is equal to the value of "apicid"
> showed by /proc/cpuinfo as proved by our debug printk.
> b)read_apic_id() gets the value from LAPIC ID register which is "apicid"
> as well.
>
> While the value of "initial apicid" is from cpuid instruction.
>
> One example of "apicid" and "initial apicid" of cpu0 from
/proc/cpuinfo
> on AMD machine:
> apicid : 32
> initial apicid : 0
>
> Therefore, we should assign /proc/cpuifo "apicid" to
"disable_cpu_apicid=X".
>
> We've never met such issue before, because we usually tested
"nr_cpus=1",
> and mostly on Intel machines, and "apicid" and "initial apicid"
have the
> same value in most cases on Intel machines.
>
For my understanding, could you show me the following information
on the AMD machines?
- dmesg | grep "ACPI: LAPIC"
- /proc/cpuinfo
# dmesg | grep "ACPI: LAPIC"
[ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x10] enabled)
[ 0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x11] enabled)
[ 0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x12] enabled)
[ 0.000000] ACPI: LAPIC (acpi_id[0x04] lapic_id[0x13] enabled)
[ 0.000000] ACPI: LAPIC (acpi_id[0x05] lapic_id[0x14] enabled)
[ 0.000000] ACPI: LAPIC (acpi_id[0x06] lapic_id[0x15] enabled)
[ 0.000000] ACPI: LAPIC (acpi_id[0x07] lapic_id[0x16] enabled)
[ 0.000000] ACPI: LAPIC (acpi_id[0x08] lapic_id[0x17] enabled)
[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] high edge lint[0x1])
# cat /proc/cpuinfo (there are 8 cpus, paste 4 cpus here)
processor : 0
vendor_id : AuthenticAMD
cpu family : 21
model : 2
model name : AMD FX(tm)-8350 Eight-Core Processor
stepping : 0
microcode : 0x600084f
cpu MHz : 4000.000
cache size : 2048 KB
physical id : 0
siblings : 8
core id : 0
cpu cores : 4
apicid : 16
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36
clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc art
rep_good nopl nonstop_tsc extd_apicid aperfmperf pni pclmulqdq monitor ssse3 fma cx16
sse4_1 sse4_2 popcnt aes xsave avx f16c lahf_lm cmp_legacy svm extapic cr8_legacy abm
sse4a misalignsse 3dnowprefetch osvw ibs xop skinit wdt lwp fma4 tce nodeid_msr tbm
topoext perfctr_core perfctr_nb cpb hw_pstate bmi1 arat npt lbrv svm_lock nrip_save
tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold
bogomips : 7982.77
TLB size : 1536 4K pages
clflush size : 64
cache_alignment : 64
address sizes : 48 bits physical, 48 bits virtual
power management: ts ttp tm 100mhzsteps hwpstate cpb eff_freq_ro
processor : 1
vendor_id : AuthenticAMD
cpu family : 21
model : 2
model name : AMD FX(tm)-8350 Eight-Core Processor
stepping : 0
microcode : 0x600084f
cpu MHz : 4000.000
cache size : 2048 KB
physical id : 0
siblings : 8
core id : 1
cpu cores : 4
apicid : 17
initial apicid : 1
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36
clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc art
rep_good nopl nonstop_tsc extd_apicid aperfmperf pni pclmulqdq monitor ssse3 fma cx16
sse4_1 sse4_2 popcnt aes xsave avx f16c lahf_lm cmp_legacy svm extapic cr8_legacy abm
sse4a misalignsse 3dnowprefetch osvw ibs xop skinit wdt lwp fma4 tce nodeid_msr tbm
topoext perfctr_core perfctr_nb cpb hw_pstate bmi1 arat npt lbrv svm_lock nrip_save
tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold
bogomips : 7982.77
TLB size : 1536 4K pages
clflush size : 64
cache_alignment : 64
address sizes : 48 bits physical, 48 bits virtual
power management: ts ttp tm 100mhzsteps hwpstate cpb eff_freq_ro
processor : 2
vendor_id : AuthenticAMD
cpu family : 21
model : 2
model name : AMD FX(tm)-8350 Eight-Core Processor
stepping : 0
microcode : 0x600084f
cpu MHz : 4000.000
cache size : 2048 KB
physical id : 0
siblings : 8
core id : 2
cpu cores : 4
apicid : 18
initial apicid : 2
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36
clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc art
rep_good nopl nonstop_tsc extd_apicid aperfmperf pni pclmulqdq monitor ssse3 fma cx16
sse4_1 sse4_2 popcnt aes xsave avx f16c lahf_lm cmp_legacy svm extapic cr8_legacy abm
sse4a misalignsse 3dnowprefetch osvw ibs xop skinit wdt lwp fma4 tce nodeid_msr tbm
topoext perfctr_core perfctr_nb cpb hw_pstate bmi1 arat npt lbrv svm_lock nrip_save
tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold
bogomips : 7982.77
TLB size : 1536 4K pages
clflush size : 64
cache_alignment : 64
address sizes : 48 bits physical, 48 bits virtual
power management: ts ttp tm 100mhzsteps hwpstate cpb eff_freq_ro
processor : 3
vendor_id : AuthenticAMD
cpu family : 21
model : 2
model name : AMD FX(tm)-8350 Eight-Core Processor
stepping : 0
microcode : 0x600084f
cpu MHz : 4000.000
cache size : 2048 KB
physical id : 0
siblings : 8
core id : 3
cpu cores : 4
apicid : 19
initial apicid : 3
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36
clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc art
rep_good nopl nonstop_tsc extd_apicid aperfmperf pni pclmulqdq monitor ssse3 fma cx16
sse4_1 sse4_2 popcnt aes xsave avx f16c lahf_lm cmp_legacy svm extapic cr8_legacy abm
sse4a misalignsse 3dnowprefetch osvw ibs xop skinit wdt lwp fma4 tce nodeid_msr tbm
topoext perfctr_core perfctr_nb cpb hw_pstate bmi1 arat npt lbrv svm_lock nrip_save
tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold
bogomips : 7982.77
TLB size : 1536 4K pages
clflush size : 64
cache_alignment : 64
address sizes : 48 bits physical, 48 bits virtual
power management: ts ttp tm 100mhzsteps hwpstate cpb eff_freq_ro
> Signed-off-by: Xunlei Pang <xlpang(a)redhat.com>
> ---
> kdumpctl | 10 +++++-----
> 1 file changed, 5 insertions(+), 5 deletions(-)
>
> diff --git a/kdumpctl b/kdumpctl
> index 4d6b3e8..46b65d2 100755
> --- a/kdumpctl
> +++ b/kdumpctl
> @@ -77,15 +77,15 @@ remove_cmdline_param()
> }
>
> #
> -# This function returns the "initial apicid" of the
> -# boot cpu (cpu 0) if present.
> +# This function returns the "apicid" of the boot
> +# cpu (cpu 0) if present.
> #
> -get_bootcpu_initial_apicid()
> +get_bootcpu_apicid()
> {
> awk ' \
> BEGIN { CPU = "-1"; } \
> $1=="processor" && $2==":" { CPU = $NF; } \
> - CPU=="0" && /initial apicid/ { print $NF; } \
> + CPU=="0" && /^apicid/ { print $NF; } \
> ' \
> /proc/cpuinfo
> }
> @@ -206,7 +206,7 @@ prepare_cmdline()
>
> cmdline="${cmdline} ${KDUMP_COMMANDLINE_APPEND}"
>
> - id=`get_bootcpu_initial_apicid`
> + id=`get_bootcpu_apicid`
> if [ ! -z ${id} ] ; then
> cmdline=`append_cmdline "${cmdline}" disable_cpu_apicid
> ${id}`
> fi
> --
> 1.8.3.1
> _______________________________________________
> kexec mailing list -- kexec(a)lists.fedoraproject.org
> To unsubscribe send an email to kexec-leave(a)lists.fedoraproject.org