Thin provision is a mechanism that you can allocate a lvm volume which has a large virtual size for file systems but actually in a small physical size. The physical size can be autoextended in use if thin pool reached a threshold specified in /etc/lvm/lvm.conf. There are 3 works should be handled when enable lvm2 thinp for kdump: 1) Check if the dump target device or directory is thinp device. 2) Monitor the thin pool and autoextend its size when it reached the threshold during kdump.

3) If thin pool size-autoextend fails, the user space program will not know due to buffered IO. So "sync -f vmcore" is used during kdump in 2nd kernel, to force sync vmcore data into disk.

According to my testing, the memory consumption procedure for lvm2 thinp is the thin pool size-autoextend phase. For fedora and rhel9, the default crashkernel value is enough. But for rhel8, the default crashkernel value 1G-4G:160M is not enough, so it should be handled particularly. v1 -> v2: 1) Modified the usage of lvs cmd when check if target is lvm2 thinp device. 2) Removed the sync flag way of mounting for lvm2 thinp target during kdump, use "sync -f vmcore" to force sync data, and handle the error if fails. Tao Liu (4): Add lvm2 thin provision dump target checker Add lvm2-monitor.service for kdump when lvm2 thinp enabled lvm.conf should be check modified if lvm2 thinp enabled Fix the sync issue for dump_fs dracut-kdump.sh | 10 ++++++++-- dracut-lvm2-monitor.service | 15 +++++++++++++++ dracut-module-setup.sh | 16 ++++++++++++++++ kdump-lib-initramfs.sh | 20 ++++++++++++++++++++ kdumpctl | 1 + kexec-tools.spec | 2 ++ 6 files changed, 62 insertions(+), 2 deletions(-) create mode 100644 dracut-lvm2-monitor.service -- 2.33.1

On Tue, May 24, 2022 at 10:24:32AM -0400, Vivek Goyal wrote: > On Tue, May 24, 2022 at 10:12:40PM +0800, Tao Liu wrote: > > Thin provision is a mechanism that you can allocate a lvm volume which has > > a large virtual size for file systems but actually in a small physical > > size. The physical size can be autoextended in use if thin pool reached a > > threshold specified in /etc/lvm/lvm.conf. > > > > There are 3 works should be handled when enable lvm2 thinp for kdump: > > > > 1) Check if the dump target device or directory is thinp device. > > 2) Monitor the thin pool and autoextend its size when it reached the threshold > > during kdump. > Hi Vivek, > Have you tested that auto-extend logic is working fine? > > Secondly, can you please also test what happens if thin pool gets full > and there is no more space for extension. Is system hanging? If yes, > that's not a good situation. We want to reboot back after saving dump > automatically. > > If it does hang, We need to add logic to configure xfs error handling so > that it does not retry infinitely. I have tested the autoextend logic locally, which works fine. If thinpool gets full, it will first reach "device-mapper: thin: 253:2: switching pool to out-of-data-space (queue IO) mode", then 60s later it will switch to "device-mapper: thin: 253:2: switching pool to out-of-data-space (error IO) mode", then continues. So it will hang for 60s mostly, please see the dmesg log below. As Zdenek suggested, we can use "lvchange --errorwhenfull y|n vgname/thinpoolname" to skip the waiting, but I think it is not harmful for now. [ 3.627063] kdump[506]: saving vmcore-dmesg.txt complete [ 3.635826] kdump[508]: saving vmcore [ 4.248430] device-mapper: thin: 253:2: reached low water mark for data device: sending event. [ 3.875066] lvm[440]: Insufficient free space: 3 extents needed, but only 2 available [ 3.886365] lvm[440]: Failed command for vg00-thinpool-tpool. [ 3.896824] lvm[440]: WARNING: Thin pool vg00-thinpool-tpool data is now 95.12% full. [ 4.436433] device-mapper: thin: 253:2: switching pool to out-of-data-space (queue IO) mode [ 3.980617] lvm[440]: Insufficient free space: 4 extents needed, but only 2 available [ 3.992068] lvm[440]: Failed command for vg00-thinpool-tpool. [ 4.066058] lvm[440]: Insufficient free space: 4 extents needed, but only 2 available [ 4.083457] lvm[440]: Failed command for vg00-thinpool-tpool. [ 4.092540] lvm[440]: WARNING: Thin pool vg00-thinpool-tpool data is now 100.00% full. [ 4.271764] kdump.sh[509]: ^MChecking for memory holes : [ 0.0 %] / ^MChecking for memory s [ 4.295810] kdump.sh[509]: The dumpfile is saved to /kdumproot/mnt/var/crash/127.0.0.1-2022-05-24-14:42:53//vmcore-incomplete. [ 4.304127] kdump.sh[509]: makedumpfile Completed. [ 12.564731] lvm[440]: Insufficient free space: 4 extents needed, but only 2 available [ 12.581630] lvm[440]: Failed command for vg00-thinpool-tpool. [ 42.569627] lvm[440]: Insufficient free space: 4 extents needed, but only 2 available [ 42.587529] lvm[440]: Failed command for vg00-thinpool-tpool. [ 67.085196] device-mapper: thin: 253:2: switching pool to out-of-data-space (error IO) mode

[ 67.126206] EXT4-fs warning (device dm-3): ext4_end_bio:345: I/O error 3 writing to inode 32389 starting block 28672)

[ 67.142081] Buffer I/O error on device dm-3, logical block 26625 [ 67.143062] Buffer I/O error on device dm-3, logical block 26626 [ 67.143062] Buffer I/O error on device dm-3, logical block 26627 [ 67.143062] Buffer I/O error on device dm-3, logical block 26628 [ 67.173719] Buffer I/O error on device dm-3, logical block 26629 [ 67.174703] Buffer I/O error on device dm-3, logical block 26630 [ 67.174703] Buffer I/O error on device dm-3, logical block 26631 [ 67.174703] Buffer I/O error on device dm-3, logical block 26632 [ 67.203393] Buffer I/O error on device dm-3, logical block 26633 [ 67.204375] Buffer I/O error on device dm-3, logical block 26634 [ 67.218086] EXT4-fs warning (device dm-3): ext4_end_bio:345: I/O error 3 writing to inode 32389 starting block 29184) [ 67.230461] EXT4-fs warning (device dm-3): ext4_end_bio:345: I/O error 3 writ: [ 67.243262] EXT4-fs warning (device dm-3): ext4_end_bio:345: I/O error 3 writing to inode 32389 starting block 32768) [ 67.257469] EXT4-fs warning (device dm-3): ext4_end_bio:345: I/O error 3 writing to inode 32389 starting block 33280) [ 67.270994] EXT4-fs warning (device dm-3): ext4_end_bio:345: I/O error 3 writing to inode 32389 starting block 34816) [ 67.283991] EXT4-fs warning (device dm-3): ext4_end_bio:345: I/O error 3 writing to inode 32389 starting block 36864) [ 67.296368] EXT4-fs warning (device dm-3): ext4_end_bio:345: I/O error 3 writing to inode 32389 starting block 37376) [ 67.310025] EXT4-fs warning (device dm-3): ext4_end_bio:345: I/O error 3 writing to inode 32389 starting block 38912) [ 67.323535] EXT4-fs warning (device dm-3): ext4_end_bio:345: I/O error 3 writing to inode 32389 starting block 43520) [ 67.338894] JBD2: Detected IO errors while flushing file data on dm-3-8 [ 66.856873] lvm[440]: Insufficient free space: 4 extents needed, but only 2 available [ 66.872656] lvm[440]: Failed command for vg00-thinpool-tpool. [ 66.884612] kdump.sh[512]: sync: error syncing '/kdumproot/mnt/var/crash/127.0.0.1-2022-05-24-14:42:53//vmcore': Input/output error [ 66.902511] kdump[514]: sync vmcore failed, exitcode:1 [ 66.914007] kdump[516]: saving vmcore failed > > > 3) If thin pool size-autoextend fails, the user space program will not know due to > > buffered IO. So "sync -f vmcore" is used during kdump in 2nd kernel, > > to force sync vmcore data into disk. > > It would be good if this "sync -f vmcore" fix is sent as a separate patch. > This is needed anyway irrespective of thin pool support. > OK, I will split it to a seperate patch. Thanks, Tao Liu > Thanks > Vivek > > > > > According to my testing, the memory consumption procedure for lvm2 thinp is the thin pool > > size-autoextend phase. For fedora and rhel9, the default crashkernel value is enough. But > > for rhel8, the default crashkernel value 1G-4G:160M is not enough, so it should > > be handled particularly. > > > > v1 -> v2: > > > > 1) Modified the usage of lvs cmd when check if target is lvm2 thinp > > device. > > 2) Removed the sync flag way of mounting for lvm2 thinp target > > during kdump, use "sync -f vmcore" to force sync data, and handle > > the error if fails. > > > > Tao Liu (4): > > Add lvm2 thin provision dump target checker > > Add lvm2-monitor.service for kdump when lvm2 thinp enabled > > lvm.conf should be check modified if lvm2 thinp enabled > > Fix the sync issue for dump_fs > > > > dracut-kdump.sh | 10 ++++++++-- > > dracut-lvm2-monitor.service | 15 +++++++++++++++ > > dracut-module-setup.sh | 16 ++++++++++++++++ > > kdump-lib-initramfs.sh | 20 ++++++++++++++++++++ > > kdumpctl | 1 + > > kexec-tools.spec | 2 ++ > > 6 files changed, 62 insertions(+), 2 deletions(-) > > create mode 100644 dracut-lvm2-monitor.service > > > > -- > > 2.33.1 > > >

On Tue, May 24, 2022 at 11:12:23AM -0400, Vivek Goyal wrote: > On Tue, May 24, 2022 at 11:02:12PM +0800, Tao Liu wrote: > > On Tue, May 24, 2022 at 10:24:32AM -0400, Vivek Goyal wrote: > > > On Tue, May 24, 2022 at 10:12:40PM +0800, Tao Liu wrote: > > > > Thin provision is a mechanism that you can allocate a lvm volume which has > > > > a large virtual size for file systems but actually in a small physical > > > > size. The physical size can be autoextended in use if thin pool reached a > > > > threshold specified in /etc/lvm/lvm.conf. > > > > > > > > There are 3 works should be handled when enable lvm2 thinp for kdump: > > > > > > > > 1) Check if the dump target device or directory is thinp device. > > > > 2) Monitor the thin pool and autoextend its size when it reached the threshold > > > > during kdump. > > > > > > > Hi Vivek, > > > > > Have you tested that auto-extend logic is working fine? > > > > > > Secondly, can you please also test what happens if thin pool gets full > > > and there is no more space for extension. Is system hanging? If yes, > > > that's not a good situation. We want to reboot back after saving dump > > > automatically. > > > > > > If it does hang, We need to add logic to configure xfs error handling so > > > that it does not retry infinitely. > > > > I have tested the autoextend logic locally, which works fine. If thinpool > > gets full, it will first reach "device-mapper: thin: 253:2: switching pool > > to out-of-data-space (queue IO) mode", then 60s later it will switch to > > "device-mapper: thin: 253:2: switching pool to out-of-data-space (error IO) > > mode", then continues. So it will hang for 60s mostly, please see the > > dmesg log below. > > > > As Zdenek suggested, we can use "lvchange --errorwhenfull y|n vgname/thinpoolname" > > to skip the waiting, but I think it is not harmful for now. > > > > [ 3.627063] kdump[506]: saving vmcore-dmesg.txt complete > > [ 3.635826] kdump[508]: saving vmcore > > [ 4.248430] device-mapper: thin: 253:2: reached low water mark for data device: sending event. > > [ 3.875066] lvm[440]: Insufficient free space: 3 extents needed, but only 2 available > > [ 3.886365] lvm[440]: Failed command for vg00-thinpool-tpool. > > [ 3.896824] lvm[440]: WARNING: Thin pool vg00-thinpool-tpool data is now 95.12% full. > > [ 4.436433] device-mapper: thin: 253:2: switching pool to out-of-data-space (queue IO) mode > > [ 3.980617] lvm[440]: Insufficient free space: 4 extents needed, but only 2 available > > [ 3.992068] lvm[440]: Failed command for vg00-thinpool-tpool. > > [ 4.066058] lvm[440]: Insufficient free space: 4 extents needed, but only 2 available > > [ 4.083457] lvm[440]: Failed command for vg00-thinpool-tpool. > > [ 4.092540] lvm[440]: WARNING: Thin pool vg00-thinpool-tpool data is now 100.00% full. > > [ 4.271764] kdump.sh[509]: ^MChecking for memory holes : [ 0.0 %] / ^MChecking for memory s > > [ 4.295810] kdump.sh[509]: The dumpfile is saved to /kdumproot/mnt/var/crash/127.0.0.1-2022-05-24-14:42:53//vmcore-incomplete. > > [ 4.304127] kdump.sh[509]: makedumpfile Completed. > > [ 12.564731] lvm[440]: Insufficient free space: 4 extents needed, but only 2 available > > [ 12.581630] lvm[440]: Failed command for vg00-thinpool-tpool. > > [ 42.569627] lvm[440]: Insufficient free space: 4 extents needed, but only 2 available > > [ 42.587529] lvm[440]: Failed command for vg00-thinpool-tpool. > > [ 67.085196] device-mapper: thin: 253:2: switching pool to out-of-data-space (error IO) mode > > [..] > > [ 67.126206] EXT4-fs warning (device dm-3): ext4_end_bio:345: I/O error 3 writing to inode 32389 starting block 28672) > > You are testing with EXT4 filesystem. Please test with XFS as well. While > using xfs on top of thin devices, docker had run into issues when thin > pool gets full. EXT4 seemed to be fine and I think it remoted file > system read-only and continued. Yes, I have tested with xfs, which works fine too. Please see the dmesg log: [ 3.426025] XFS (dm-3): Mounting V5 Filesystem [ 3.599421] XFS (dm-3): Starting recovery (logdev: internal) [ 3.624443] XFS (dm-3): Ending recovery (logdev: internal) [ 3.647705] xfs filesystem being mounted at /kdumproot/mnt supports timestamps until 2038 (0x7fffffff) .... [ 3.552870] kdump[514]: saving vmcore [ 4.082238] device-mapper: thin: 253:2: reached low water mark for data device: sending event. [ 4.142192] device-mapper: thin: 253:2: switching pool to out-of-data-space (queue IO) mode [ 3.679640] lvm[446]: WARNING: Sum of all thin volume sizes (300.00 MiB) exceeds the size of thin pools and the size of whole volume group (48.00 MiB). [ 3.690712] lvm[446]: Size of logical volume vg00/thinpool_tdata changed from 12.00 MiB (3 extents) to 20.00 MiB (5 extents). [ 4.211179] device-mapper: thin: 253:2: switching pool to write mode [ 4.227481] device-mapper: thin: 253:2: growing the data device from 192 to 320 blocks [ 3.789801] lvm[446]: Logical volume vg00/thinpool_tdata successfully resized. [ 4.269802] device-mapper: thin: 253:2: reached low water mark for data device: sending event. [ 4.335903] device-mapper: thin: 253:2: switching pool to out-of-data-space (queue IO) mode [ 3.948400] lvm[446]: WARNING: Sum of all thin volume sizes (300.00 MiB) exceeds the size of thin pools and the size of whole volume group (48.00 MiB). [ 3.972514] lvm[446]: Size of logical volume vg00/thinpool_tdata changed from 20.00 MiB (5 extents) to 32.00 MiB (8 extents). [ 4.468128] device-mapper: thin: 253:2: switching pool to write mode [ 4.485914] device-mapper: thin: 253:2: growing the data device from 320 to 512 blocks [ 4.049409] lvm[446]: Logical volume vg00/thinpool_tdata successfully resized. [ 4.528457] device-mapper: thin: 253:2: reached low water mark for data device: sending event. [ 4.605303] device-mapper: thin: 253:2: switching pool to out-of-data-space (queue IO) mode [ 4.224389] lvm[446]: Insufficient free space: 4 extents needed, but only 2 available [ 4.238011] lvm[446]: Failed command for vg00-thinpool-tpool. [ 4.251287] lvm[446]: WARNING: Thin pool vg00-thinpool-tpool data is now 100.00% full. [ 4.272262] lvm[446]: Insufficient free space: 4 extents needed, but only 2 available [ 4.293131] lvm[446]: Failed command for vg00-thinpool-tpool. [ 4.339268] kdump.sh[515]: ^MChecking for memory holes : [ 0.0 %] / ^MChecking for memory holes : [100.0 %] | ^MExcluding unnecessary pages : [100.0 %] \ ^MCopying data : [ 97.8 %] - eta: 0s^MCopying data : [100.0 %] / eta: 0s^MCopying data : [100.0 %] | eta: 0s [ 4.365706] kdump.sh[515]: The dumpfile is saved to /kdumproot/mnt/var/crash/127.0.0.1-2022-05-25-00:26:53//vmcore-incomplete. [ 4.374532] kdump.sh[515]: makedumpfile Completed. [ 23.344673] lvm[446]: Insufficient free space: 4 extents needed, but only 2 available [ 23.361064] lvm[446]: Failed command for vg00-thinpool-tpool. [ 53.344919] lvm[446]: Insufficient free space: 4 extents needed, but only 2 available [ 53.372743] lvm[446]: Failed command for vg00-thinpool-tpool. [ 67.034930] device-mapper: thin: 253:2: switching pool to out-of-data-space (error IO) mode [ 67.049185] dm-3: writeback error on inode 134, offset 176128, sector 1664 [ 67.049388] dm-3: writeback error on inode 134, offset 21590016, sector 42448 [ 67.058358] dm-3: writeback error on inode 134, offset 25784320, sector 58752 [ 67.067440] dm-3: writeback error on inode 134, offset 29978624, sector 64128 [ 67.077099] dm-3: writeback error on inode 134, offset 34172928, sector 67328 [ 67.086122] dm-3: writeback error on inode 134, offset 34340864, sector 82816 [ 67.095074] dm-3: writeback error on inode 134, offset 37294080, sector 88704 [ 67.104907] dm-3: writeback error on inode 134, offset 40325120, sector 90368 [ 67.120230] dm-3: writeback error on inode 134, offset 770048, sector 1784 [ 66.670306] kdump.sh[562]: sync: error syncing '/kdumproot/mnt/var/crash/127.0.0.1-2022-05-25-00:26:53//vmcore': Input/output error [ 66.694671] kdump[570]: sync vmcore failed, exitcode:1 [ 66.709217] lvm[446]: Insufficient free space: 4 extents needed, but only 2 available [ 66.757336] kdump[572]: saving vmcore failed

Thanks, Tao Liu > > Thanks > Vivek > > > [ 67.142081] Buffer I/O error on device dm-3, logical block 26625 > > [ 67.143062] Buffer I/O error on device dm-3, logical block 26626 > > [ 67.143062] Buffer I/O error on device dm-3, logical block 26627 > > [ 67.143062] Buffer I/O error on device dm-3, logical block 26628 > > [ 67.173719] Buffer I/O error on device dm-3, logical block 26629 > > [ 67.174703] Buffer I/O error on device dm-3, logical block 26630 > > [ 67.174703] Buffer I/O error on device dm-3, logical block 26631 > > [ 67.174703] Buffer I/O error on device dm-3, logical block 26632 > > [ 67.203393] Buffer I/O error on device dm-3, logical block 26633 > > [ 67.204375] Buffer I/O error on device dm-3, logical block 26634 > > [ 67.218086] EXT4-fs warning (device dm-3): ext4_end_bio:345: I/O error 3 writing to inode 32389 starting block 29184) > > [ 67.230461] EXT4-fs warning (device dm-3): ext4_end_bio:345: I/O error 3 writ: > > [ 67.243262] EXT4-fs warning (device dm-3): ext4_end_bio:345: I/O error 3 writing to inode 32389 starting block 32768) > > [ 67.257469] EXT4-fs warning (device dm-3): ext4_end_bio:345: I/O error 3 writing to inode 32389 starting block 33280) > > [ 67.270994] EXT4-fs warning (device dm-3): ext4_end_bio:345: I/O error 3 writing to inode 32389 starting block 34816) > > [ 67.283991] EXT4-fs warning (device dm-3): ext4_end_bio:345: I/O error 3 writing to inode 32389 starting block 36864) > > [ 67.296368] EXT4-fs warning (device dm-3): ext4_end_bio:345: I/O error 3 writing to inode 32389 starting block 37376) > > [ 67.310025] EXT4-fs warning (device dm-3): ext4_end_bio:345: I/O error 3 writing to inode 32389 starting block 38912) > > [ 67.323535] EXT4-fs warning (device dm-3): ext4_end_bio:345: I/O error 3 writing to inode 32389 starting block 43520) > > [ 67.338894] JBD2: Detected IO errors while flushing file data on dm-3-8 > > [ 66.856873] lvm[440]: Insufficient free space: 4 extents needed, but only 2 available > > [ 66.872656] lvm[440]: Failed command for vg00-thinpool-tpool. > > [ 66.884612] kdump.sh[512]: sync: error syncing '/kdumproot/mnt/var/crash/127.0.0.1-2022-05-24-14:42:53//vmcore': Input/output error > > [ 66.902511] kdump[514]: sync vmcore failed, exitcode:1 > > [ 66.914007] kdump[516]: saving vmcore failed > > > > > > > > > 3) If thin pool size-autoextend fails, the user space program will not know due to > > > > buffered IO. So "sync -f vmcore" is used during kdump in 2nd kernel, > > > > to force sync vmcore data into disk. > > > > > > It would be good if this "sync -f vmcore" fix is sent as a separate patch. > > > This is needed anyway irrespective of thin pool support. > > > > > > > OK, I will split it to a seperate patch. > > > > Thanks, > > Tao Liu > > > > > Thanks > > > Vivek > > > > > > > > > > > According to my testing, the memory consumption procedure for lvm2 thinp is the thin pool > > > > size-autoextend phase. For fedora and rhel9, the default crashkernel value is enough. But > > > > for rhel8, the default crashkernel value 1G-4G:160M is not enough, so it should > > > > be handled particularly. > > > > > > > > v1 -> v2: > > > > > > > > 1) Modified the usage of lvs cmd when check if target is lvm2 thinp > > > > device. > > > > 2) Removed the sync flag way of mounting for lvm2 thinp target > > > > during kdump, use "sync -f vmcore" to force sync data, and handle > > > > the error if fails. > > > > > > > > Tao Liu (4): > > > > Add lvm2 thin provision dump target checker > > > > Add lvm2-monitor.service for kdump when lvm2 thinp enabled > > > > lvm.conf should be check modified if lvm2 thinp enabled > > > > Fix the sync issue for dump_fs > > > > > > > > dracut-kdump.sh | 10 ++++++++-- > > > > dracut-lvm2-monitor.service | 15 +++++++++++++++ > > > > dracut-module-setup.sh | 16 ++++++++++++++++ > > > > kdump-lib-initramfs.sh | 20 ++++++++++++++++++++ > > > > kdumpctl | 1 + > > > > kexec-tools.spec | 2 ++ > > > > 6 files changed, 62 insertions(+), 2 deletions(-) > > > > create mode 100644 dracut-lvm2-monitor.service > > > > > > > > -- > > > > 2.33.1 > > > > > > > > > >

On Wed, May 25, 2022 at 09:54:30AM +0800, Tao Liu wrote: > On Tue, May 24, 2022 at 11:12:23AM -0400, Vivek Goyal wrote: > > On Tue, May 24, 2022 at 11:02:12PM +0800, Tao Liu wrote: > > > On Tue, May 24, 2022 at 10:24:32AM -0400, Vivek Goyal wrote: > > > > On Tue, May 24, 2022 at 10:12:40PM +0800, Tao Liu wrote: > > > > > Thin provision is a mechanism that you can allocate a lvm volume which has > > > > > a large virtual size for file systems but actually in a small physical > > > > > size. The physical size can be autoextended in use if thin pool reached a > > > > > threshold specified in /etc/lvm/lvm.conf. > > > > > > > > > > There are 3 works should be handled when enable lvm2 thinp for kdump: > > > > > > > > > > 1) Check if the dump target device or directory is thinp device. > > > > > 2) Monitor the thin pool and autoextend its size when it reached the threshold > > > > > during kdump. > > > > > > > > > > Hi Vivek, > > > > > > > Have you tested that auto-extend logic is working fine? > > > > > > > > Secondly, can you please also test what happens if thin pool gets full > > > > and there is no more space for extension. Is system hanging? If yes, > > > > that's not a good situation. We want to reboot back after saving dump > > > > automatically. > > > > > > > > If it does hang, We need to add logic to configure xfs error handling so > > > > that it does not retry infinitely. > > > > > > I have tested the autoextend logic locally, which works fine. If thinpool > > > gets full, it will first reach "device-mapper: thin: 253:2: switching pool > > > to out-of-data-space (queue IO) mode", then 60s later it will switch to > > > "device-mapper: thin: 253:2: switching pool to out-of-data-space (error IO) > > > mode", then continues. So it will hang for 60s mostly, please see the > > > dmesg log below. > > > > > > As Zdenek suggested, we can use "lvchange --errorwhenfull y|n vgname/thinpoolname" > > > to skip the waiting, but I think it is not harmful for now. > > > > > > [ 3.627063] kdump[506]: saving vmcore-dmesg.txt complete > > > [ 3.635826] kdump[508]: saving vmcore > > > [ 4.248430] device-mapper: thin: 253:2: reached low water mark for data device: sending event. > > > [ 3.875066] lvm[440]: Insufficient free space: 3 extents needed, but only 2 available > > > [ 3.886365] lvm[440]: Failed command for vg00-thinpool-tpool. > > > [ 3.896824] lvm[440]: WARNING: Thin pool vg00-thinpool-tpool data is now 95.12% full. > > > [ 4.436433] device-mapper: thin: 253:2: switching pool to out-of-data-space (queue IO) mode > > > [ 3.980617] lvm[440]: Insufficient free space: 4 extents needed, but only 2 available > > > [ 3.992068] lvm[440]: Failed command for vg00-thinpool-tpool. > > > [ 4.066058] lvm[440]: Insufficient free space: 4 extents needed, but only 2 available > > > [ 4.083457] lvm[440]: Failed command for vg00-thinpool-tpool. > > > [ 4.092540] lvm[440]: WARNING: Thin pool vg00-thinpool-tpool data is now 100.00% full. > > > [ 4.271764] kdump.sh[509]: ^MChecking for memory holes : [ 0.0 %] / ^MChecking for memory s > > > [ 4.295810] kdump.sh[509]: The dumpfile is saved to /kdumproot/mnt/var/crash/127.0.0.1-2022-05-24-14:42:53//vmcore-incomplete. > > > [ 4.304127] kdump.sh[509]: makedumpfile Completed. > > > [ 12.564731] lvm[440]: Insufficient free space: 4 extents needed, but only 2 available > > > [ 12.581630] lvm[440]: Failed command for vg00-thinpool-tpool. > > > [ 42.569627] lvm[440]: Insufficient free space: 4 extents needed, but only 2 available > > > [ 42.587529] lvm[440]: Failed command for vg00-thinpool-tpool. > > > [ 67.085196] device-mapper: thin: 253:2: switching pool to out-of-data-space (error IO) mode > > > > [..] > > > [ 67.126206] EXT4-fs warning (device dm-3): ext4_end_bio:345: I/O error 3 writing to inode 32389 starting block 28672) > > > > You are testing with EXT4 filesystem. Please test with XFS as well. While > > using xfs on top of thin devices, docker had run into issues when thin > > pool gets full. EXT4 seemed to be fine and I think it remoted file > > system read-only and continued. > > Yes, I have tested with xfs, which works fine too. Please see the dmesg log: > > [ 3.426025] XFS (dm-3): Mounting V5 Filesystem > [ 3.599421] XFS (dm-3): Starting recovery (logdev: internal) > [ 3.624443] XFS (dm-3): Ending recovery (logdev: internal) > [ 3.647705] xfs filesystem being mounted at /kdumproot/mnt supports timestamps until 2038 (0x7fffffff) > .... > [ 3.552870] kdump[514]: saving vmcore > [ 4.082238] device-mapper: thin: 253:2: reached low water mark for data device: sending event. > [ 4.142192] device-mapper: thin: 253:2: switching pool to out-of-data-space (queue IO) mode > [ 3.679640] lvm[446]: WARNING: Sum of all thin volume sizes (300.00 MiB) exceeds the size of thin pools and the size of whole volume group (48.00 MiB). > [ 3.690712] lvm[446]: Size of logical volume vg00/thinpool_tdata changed from 12.00 MiB (3 extents) to 20.00 MiB (5 extents). > [ 4.211179] device-mapper: thin: 253:2: switching pool to write mode > [ 4.227481] device-mapper: thin: 253:2: growing the data device from 192 to 320 blocks > [ 3.789801] lvm[446]: Logical volume vg00/thinpool_tdata successfully resized. > [ 4.269802] device-mapper: thin: 253:2: reached low water mark for data device: sending event. > [ 4.335903] device-mapper: thin: 253:2: switching pool to out-of-data-space (queue IO) mode > [ 3.948400] lvm[446]: WARNING: Sum of all thin volume sizes (300.00 MiB) exceeds the size of thin pools and the size of whole volume group (48.00 MiB). > [ 3.972514] lvm[446]: Size of logical volume vg00/thinpool_tdata changed from 20.00 MiB (5 extents) to 32.00 MiB (8 extents). > [ 4.468128] device-mapper: thin: 253:2: switching pool to write mode > [ 4.485914] device-mapper: thin: 253:2: growing the data device from 320 to 512 blocks > [ 4.049409] lvm[446]: Logical volume vg00/thinpool_tdata successfully resized. > [ 4.528457] device-mapper: thin: 253:2: reached low water mark for data device: sending event. > [ 4.605303] device-mapper: thin: 253:2: switching pool to out-of-data-space (queue IO) mode > [ 4.224389] lvm[446]: Insufficient free space: 4 extents needed, but only 2 available > [ 4.238011] lvm[446]: Failed command for vg00-thinpool-tpool. > [ 4.251287] lvm[446]: WARNING: Thin pool vg00-thinpool-tpool data is now 100.00% full. > [ 4.272262] lvm[446]: Insufficient free space: 4 extents needed, but only 2 available > [ 4.293131] lvm[446]: Failed command for vg00-thinpool-tpool. > [ 4.339268] kdump.sh[515]: ^MChecking for memory holes : [ 0.0 %] / ^MChecking for memory holes : [100.0 %] | ^MExcluding unnecessary pages : [100.0 %] \ ^MCopying data : [ 97.8 %] - eta: 0s^MCopying data : [100.0 %] / eta: 0s^MCopying data : [100.0 %] | eta: 0s > [ 4.365706] kdump.sh[515]: The dumpfile is saved to /kdumproot/mnt/var/crash/127.0.0.1-2022-05-25-00:26:53//vmcore-incomplete. > [ 4.374532] kdump.sh[515]: makedumpfile Completed. > [ 23.344673] lvm[446]: Insufficient free space: 4 extents needed, but only 2 available > [ 23.361064] lvm[446]: Failed command for vg00-thinpool-tpool. > [ 53.344919] lvm[446]: Insufficient free space: 4 extents needed, but only 2 available > [ 53.372743] lvm[446]: Failed command for vg00-thinpool-tpool. > [ 67.034930] device-mapper: thin: 253:2: switching pool to out-of-data-space (error IO) mode > [ 67.049185] dm-3: writeback error on inode 134, offset 176128, sector 1664 > [ 67.049388] dm-3: writeback error on inode 134, offset 21590016, sector 42448 > [ 67.058358] dm-3: writeback error on inode 134, offset 25784320, sector 58752 > [ 67.067440] dm-3: writeback error on inode 134, offset 29978624, sector 64128 > [ 67.077099] dm-3: writeback error on inode 134, offset 34172928, sector 67328 > [ 67.086122] dm-3: writeback error on inode 134, offset 34340864, sector 82816 > [ 67.095074] dm-3: writeback error on inode 134, offset 37294080, sector 88704 > [ 67.104907] dm-3: writeback error on inode 134, offset 40325120, sector 90368 > [ 67.120230] dm-3: writeback error on inode 134, offset 770048, sector 1784 > [ 66.670306] kdump.sh[562]: sync: error syncing '/kdumproot/mnt/var/crash/127.0.0.1-2022-05-25-00:26:53//vmcore': Input/output error > [ 66.694671] kdump[570]: sync vmcore failed, exitcode:1 > [ 66.709217] lvm[446]: Insufficient free space: 4 extents needed, but only 2 available > [ 66.757336] kdump[572]: saving vmcore failed And system rebooted by itself after this?

This is strange. Has xfs default behavior changed now? It used to try infinitely by default if thin pool got full. Eric, would have any idea. What has changed. Thanks Vivek > > Thanks, > Tao Liu > > > > > Thanks > > Vivek > > > > > [ 67.142081] Buffer I/O error on device dm-3, logical block 26625 > > > [ 67.143062] Buffer I/O error on device dm-3, logical block 26626 > > > [ 67.143062] Buffer I/O error on device dm-3, logical block 26627 > > > [ 67.143062] Buffer I/O error on device dm-3, logical block 26628 > > > [ 67.173719] Buffer I/O error on device dm-3, logical block 26629 > > > [ 67.174703] Buffer I/O error on device dm-3, logical block 26630 > > > [ 67.174703] Buffer I/O error on device dm-3, logical block 26631 > > > [ 67.174703] Buffer I/O error on device dm-3, logical block 26632 > > > [ 67.203393] Buffer I/O error on device dm-3, logical block 26633 > > > [ 67.204375] Buffer I/O error on device dm-3, logical block 26634 > > > [ 67.218086] EXT4-fs warning (device dm-3): ext4_end_bio:345: I/O error 3 writing to inode 32389 starting block 29184) > > > [ 67.230461] EXT4-fs warning (device dm-3): ext4_end_bio:345: I/O error 3 writ: > > > [ 67.243262] EXT4-fs warning (device dm-3): ext4_end_bio:345: I/O error 3 writing to inode 32389 starting block 32768) > > > [ 67.257469] EXT4-fs warning (device dm-3): ext4_end_bio:345: I/O error 3 writing to inode 32389 starting block 33280) > > > [ 67.270994] EXT4-fs warning (device dm-3): ext4_end_bio:345: I/O error 3 writing to inode 32389 starting block 34816) > > > [ 67.283991] EXT4-fs warning (device dm-3): ext4_end_bio:345: I/O error 3 writing to inode 32389 starting block 36864) > > > [ 67.296368] EXT4-fs warning (device dm-3): ext4_end_bio:345: I/O error 3 writing to inode 32389 starting block 37376) > > > [ 67.310025] EXT4-fs warning (device dm-3): ext4_end_bio:345: I/O error 3 writing to inode 32389 starting block 38912) > > > [ 67.323535] EXT4-fs warning (device dm-3): ext4_end_bio:345: I/O error 3 writing to inode 32389 starting block 43520) > > > [ 67.338894] JBD2: Detected IO errors while flushing file data on dm-3-8 > > > [ 66.856873] lvm[440]: Insufficient free space: 4 extents needed, but only 2 available > > > [ 66.872656] lvm[440]: Failed command for vg00-thinpool-tpool. > > > [ 66.884612] kdump.sh[512]: sync: error syncing '/kdumproot/mnt/var/crash/127.0.0.1-2022-05-24-14:42:53//vmcore': Input/output error > > > [ 66.902511] kdump[514]: sync vmcore failed, exitcode:1 > > > [ 66.914007] kdump[516]: saving vmcore failed > > > > > > > > > > > > 3) If thin pool size-autoextend fails, the user space program will not know due to > > > > > buffered IO. So "sync -f vmcore" is used during kdump in 2nd kernel, > > > > > to force sync vmcore data into disk. > > > > > > > > It would be good if this "sync -f vmcore" fix is sent as a separate patch. > > > > This is needed anyway irrespective of thin pool support. > > > > > > > > > > OK, I will split it to a seperate patch. > > > > > > Thanks, > > > Tao Liu > > > > > > > Thanks > > > > Vivek > > > > > > > > > > > > > > According to my testing, the memory consumption procedure for lvm2 thinp is the thin pool > > > > > size-autoextend phase. For fedora and rhel9, the default crashkernel value is enough. But > > > > > for rhel8, the default crashkernel value 1G-4G:160M is not enough, so it should > > > > > be handled particularly. > > > > > > > > > > v1 -> v2: > > > > > > > > > > 1) Modified the usage of lvs cmd when check if target is lvm2 thinp > > > > > device. > > > > > 2) Removed the sync flag way of mounting for lvm2 thinp target > > > > > during kdump, use "sync -f vmcore" to force sync data, and handle > > > > > the error if fails. > > > > > > > > > > Tao Liu (4): > > > > > Add lvm2 thin provision dump target checker > > > > > Add lvm2-monitor.service for kdump when lvm2 thinp enabled > > > > > lvm.conf should be check modified if lvm2 thinp enabled > > > > > Fix the sync issue for dump_fs > > > > > > > > > > dracut-kdump.sh | 10 ++++++++-- > > > > > dracut-lvm2-monitor.service | 15 +++++++++++++++ > > > > > dracut-module-setup.sh | 16 ++++++++++++++++ > > > > > kdump-lib-initramfs.sh | 20 ++++++++++++++++++++ > > > > > kdumpctl | 1 + > > > > > kexec-tools.spec | 2 ++ > > > > > 6 files changed, 62 insertions(+), 2 deletions(-) > > > > > create mode 100644 dracut-lvm2-monitor.service > > > > > > > > > > -- > > > > > 2.33.1 > > > > > > > > > > > > > > >