On Wed, May 25, 2022 at 7:43 PM Vivek Goyal <vgoyal(a)redhat.com> wrote:
On Wed, May 25, 2022 at 09:54:30AM +0800, Tao Liu wrote:
> On Tue, May 24, 2022 at 11:12:23AM -0400, Vivek Goyal wrote:
> > On Tue, May 24, 2022 at 11:02:12PM +0800, Tao Liu wrote:
> > > On Tue, May 24, 2022 at 10:24:32AM -0400, Vivek Goyal wrote:
> > > > On Tue, May 24, 2022 at 10:12:40PM +0800, Tao Liu wrote:
> > > > > Thin provision is a mechanism that you can allocate a lvm volume
which has
> > > > > a large virtual size for file systems but actually in a small
physical
> > > > > size. The physical size can be autoextended in use if thin pool
reached a
> > > > > threshold specified in /etc/lvm/lvm.conf.
> > > > >
> > > > > There are 3 works should be handled when enable lvm2 thinp for
kdump:
> > > > >
> > > > > 1) Check if the dump target device or directory is thinp
device.
> > > > > 2) Monitor the thin pool and autoextend its size when it reached
the threshold
> > > > > during kdump.
> > > >
> > >
> > > Hi Vivek,
> > >
> > > > Have you tested that auto-extend logic is working fine?
> > > >
> > > > Secondly, can you please also test what happens if thin pool gets
full
> > > > and there is no more space for extension. Is system hanging? If yes,
> > > > that's not a good situation. We want to reboot back after saving
dump
> > > > automatically.
> > > >
> > > > If it does hang, We need to add logic to configure xfs error handling
so
> > > > that it does not retry infinitely.
> > >
> > > I have tested the autoextend logic locally, which works fine. If thinpool
> > > gets full, it will first reach "device-mapper: thin: 253:2: switching
pool
> > > to out-of-data-space (queue IO) mode", then 60s later it will switch
to
> > > "device-mapper: thin: 253:2: switching pool to out-of-data-space
(error IO)
> > > mode", then continues. So it will hang for 60s mostly, please see
the
> > > dmesg log below.
> > >
> > > As Zdenek suggested, we can use "lvchange --errorwhenfull y|n
vgname/thinpoolname"
> > > to skip the waiting, but I think it is not harmful for now.
> > >
> > > [ 3.627063] kdump[506]: saving vmcore-dmesg.txt complete
> > > [ 3.635826] kdump[508]: saving vmcore
> > > [ 4.248430] device-mapper: thin: 253:2: reached low water mark for data
device: sending event.
> > > [ 3.875066] lvm[440]: Insufficient free space: 3 extents needed, but
only 2 available
> > > [ 3.886365] lvm[440]: Failed command for vg00-thinpool-tpool.
> > > [ 3.896824] lvm[440]: WARNING: Thin pool vg00-thinpool-tpool data is
now 95.12% full.
> > > [ 4.436433] device-mapper: thin: 253:2: switching pool to
out-of-data-space (queue IO) mode
> > > [ 3.980617] lvm[440]: Insufficient free space: 4 extents needed, but
only 2 available
> > > [ 3.992068] lvm[440]: Failed command for vg00-thinpool-tpool.
> > > [ 4.066058] lvm[440]: Insufficient free space: 4 extents needed, but
only 2 available
> > > [ 4.083457] lvm[440]: Failed command for vg00-thinpool-tpool.
> > > [ 4.092540] lvm[440]: WARNING: Thin pool vg00-thinpool-tpool data is
now 100.00% full.
> > > [ 4.271764] kdump.sh[509]: ^MChecking for memory holes
: [ 0.0 %] / ^MChecking for memory s
> > > [ 4.295810] kdump.sh[509]: The dumpfile is saved to
/kdumproot/mnt/var/crash/127.0.0.1-2022-05-24-14:42:53//vmcore-incomplete.
> > > [ 4.304127] kdump.sh[509]: makedumpfile Completed.
> > > [ 12.564731] lvm[440]: Insufficient free space: 4 extents needed, but
only 2 available
> > > [ 12.581630] lvm[440]: Failed command for vg00-thinpool-tpool.
> > > [ 42.569627] lvm[440]: Insufficient free space: 4 extents needed, but
only 2 available
> > > [ 42.587529] lvm[440]: Failed command for vg00-thinpool-tpool.
> > > [ 67.085196] device-mapper: thin: 253:2: switching pool to
out-of-data-space (error IO) mode
> >
> > [..]
> > > [ 67.126206] EXT4-fs warning (device dm-3): ext4_end_bio:345: I/O error
3 writing to inode 32389 starting block 28672)
> >
> > You are testing with EXT4 filesystem. Please test with XFS as well. While
> > using xfs on top of thin devices, docker had run into issues when thin
> > pool gets full. EXT4 seemed to be fine and I think it remoted file
> > system read-only and continued.
>
> Yes, I have tested with xfs, which works fine too. Please see the dmesg log:
>
> [ 3.426025] XFS (dm-3): Mounting V5 Filesystem
> [ 3.599421] XFS (dm-3): Starting recovery (logdev: internal)
> [ 3.624443] XFS (dm-3): Ending recovery (logdev: internal)
> [ 3.647705] xfs filesystem being mounted at /kdumproot/mnt supports timestamps
until 2038 (0x7fffffff)
> ....
> [ 3.552870] kdump[514]: saving vmcore
> [ 4.082238] device-mapper: thin: 253:2: reached low water mark for data device:
sending event.
> [ 4.142192] device-mapper: thin: 253:2: switching pool to out-of-data-space
(queue IO) mode
> [ 3.679640] lvm[446]: WARNING: Sum of all thin volume sizes (300.00 MiB) exceeds
the size of thin pools and the size of whole volume group (48.00 MiB).
> [ 3.690712] lvm[446]: Size of logical volume vg00/thinpool_tdata changed from
12.00 MiB (3 extents) to 20.00 MiB (5 extents).
> [ 4.211179] device-mapper: thin: 253:2: switching pool to write mode
> [ 4.227481] device-mapper: thin: 253:2: growing the data device from 192 to 320
blocks
> [ 3.789801] lvm[446]: Logical volume vg00/thinpool_tdata successfully resized.
> [ 4.269802] device-mapper: thin: 253:2: reached low water mark for data device:
sending event.
> [ 4.335903] device-mapper: thin: 253:2: switching pool to out-of-data-space
(queue IO) mode
> [ 3.948400] lvm[446]: WARNING: Sum of all thin volume sizes (300.00 MiB) exceeds
the size of thin pools and the size of whole volume group (48.00 MiB).
> [ 3.972514] lvm[446]: Size of logical volume vg00/thinpool_tdata changed from
20.00 MiB (5 extents) to 32.00 MiB (8 extents).
> [ 4.468128] device-mapper: thin: 253:2: switching pool to write mode
> [ 4.485914] device-mapper: thin: 253:2: growing the data device from 320 to 512
blocks
> [ 4.049409] lvm[446]: Logical volume vg00/thinpool_tdata successfully resized.
> [ 4.528457] device-mapper: thin: 253:2: reached low water mark for data device:
sending event.
> [ 4.605303] device-mapper: thin: 253:2: switching pool to out-of-data-space
(queue IO) mode
> [ 4.224389] lvm[446]: Insufficient free space: 4 extents needed, but only 2
available
> [ 4.238011] lvm[446]: Failed command for vg00-thinpool-tpool.
> [ 4.251287] lvm[446]: WARNING: Thin pool vg00-thinpool-tpool data is now 100.00%
full.
> [ 4.272262] lvm[446]: Insufficient free space: 4 extents needed, but only 2
available
> [ 4.293131] lvm[446]: Failed command for vg00-thinpool-tpool.
> [ 4.339268] kdump.sh[515]: ^MChecking for memory holes :
[ 0.0 %] / ^MChecking for memory holes : [100.0
%] | ^MExcluding unnecessary pages : [100.0 %] \
^MCopying data : [ 97.8 %] -
eta: 0s^MCopying data : [100.0 %] / eta:
0s^MCopying data : [100.0 %] | eta: 0s
> [ 4.365706] kdump.sh[515]: The dumpfile is saved to
/kdumproot/mnt/var/crash/127.0.0.1-2022-05-25-00:26:53//vmcore-incomplete.
> [ 4.374532] kdump.sh[515]: makedumpfile Completed.
> [ 23.344673] lvm[446]: Insufficient free space: 4 extents needed, but only 2
available
> [ 23.361064] lvm[446]: Failed command for vg00-thinpool-tpool.
> [ 53.344919] lvm[446]: Insufficient free space: 4 extents needed, but only 2
available
> [ 53.372743] lvm[446]: Failed command for vg00-thinpool-tpool.
> [ 67.034930] device-mapper: thin: 253:2: switching pool to out-of-data-space
(error IO) mode
> [ 67.049185] dm-3: writeback error on inode 134, offset 176128, sector 1664
> [ 67.049388] dm-3: writeback error on inode 134, offset 21590016, sector 42448
> [ 67.058358] dm-3: writeback error on inode 134, offset 25784320, sector 58752
> [ 67.067440] dm-3: writeback error on inode 134, offset 29978624, sector 64128
> [ 67.077099] dm-3: writeback error on inode 134, offset 34172928, sector 67328
> [ 67.086122] dm-3: writeback error on inode 134, offset 34340864, sector 82816
> [ 67.095074] dm-3: writeback error on inode 134, offset 37294080, sector 88704
> [ 67.104907] dm-3: writeback error on inode 134, offset 40325120, sector 90368
> [ 67.120230] dm-3: writeback error on inode 134, offset 770048, sector 1784
> [ 66.670306] kdump.sh[562]: sync: error syncing
'/kdumproot/mnt/var/crash/127.0.0.1-2022-05-25-00:26:53//vmcore': Input/output
error
> [ 66.694671] kdump[570]: sync vmcore failed, exitcode:1
> [ 66.709217] lvm[446]: Insufficient free space: 4 extents needed, but only 2
available
> [ 66.757336] kdump[572]: saving vmcore failed
And system rebooted by itself after this?
Yes, according to my test, it rebooted normally. BTW, the test system
is Fedora-Cloud-Base-33-1.2.x86_64.raw.xz, whose kernel version is
5.8.15-301.fc33.x86_64. Please see the attachment for a complete dmesg
log.
Thanks,
Tao Liu
This is strange. Has xfs default behavior changed now? It used to
try
infinitely by default if thin pool got full.
Eric, would have any idea. What has changed.
Thanks
Vivek
>
> Thanks,
> Tao Liu
>
> >
> > Thanks
> > Vivek
> >
> > > [ 67.142081] Buffer I/O error on device dm-3, logical block 26625
> > > [ 67.143062] Buffer I/O error on device dm-3, logical block 26626
> > > [ 67.143062] Buffer I/O error on device dm-3, logical block 26627
> > > [ 67.143062] Buffer I/O error on device dm-3, logical block 26628
> > > [ 67.173719] Buffer I/O error on device dm-3, logical block 26629
> > > [ 67.174703] Buffer I/O error on device dm-3, logical block 26630
> > > [ 67.174703] Buffer I/O error on device dm-3, logical block 26631
> > > [ 67.174703] Buffer I/O error on device dm-3, logical block 26632
> > > [ 67.203393] Buffer I/O error on device dm-3, logical block 26633
> > > [ 67.204375] Buffer I/O error on device dm-3, logical block 26634
> > > [ 67.218086] EXT4-fs warning (device dm-3): ext4_end_bio:345: I/O error
3 writing to inode 32389 starting block 29184)
> > > [ 67.230461] EXT4-fs warning (device dm-3): ext4_end_bio:345: I/O error
3 writ:
> > > [ 67.243262] EXT4-fs warning (device dm-3): ext4_end_bio:345: I/O error
3 writing to inode 32389 starting block 32768)
> > > [ 67.257469] EXT4-fs warning (device dm-3): ext4_end_bio:345: I/O error
3 writing to inode 32389 starting block 33280)
> > > [ 67.270994] EXT4-fs warning (device dm-3): ext4_end_bio:345: I/O error
3 writing to inode 32389 starting block 34816)
> > > [ 67.283991] EXT4-fs warning (device dm-3): ext4_end_bio:345: I/O error
3 writing to inode 32389 starting block 36864)
> > > [ 67.296368] EXT4-fs warning (device dm-3): ext4_end_bio:345: I/O error
3 writing to inode 32389 starting block 37376)
> > > [ 67.310025] EXT4-fs warning (device dm-3): ext4_end_bio:345: I/O error
3 writing to inode 32389 starting block 38912)
> > > [ 67.323535] EXT4-fs warning (device dm-3): ext4_end_bio:345: I/O error
3 writing to inode 32389 starting block 43520)
> > > [ 67.338894] JBD2: Detected IO errors while flushing file data on
dm-3-8
> > > [ 66.856873] lvm[440]: Insufficient free space: 4 extents needed, but
only 2 available
> > > [ 66.872656] lvm[440]: Failed command for vg00-thinpool-tpool.
> > > [ 66.884612] kdump.sh[512]: sync: error syncing
'/kdumproot/mnt/var/crash/127.0.0.1-2022-05-24-14:42:53//vmcore': Input/output
error
> > > [ 66.902511] kdump[514]: sync vmcore failed, exitcode:1
> > > [ 66.914007] kdump[516]: saving vmcore failed
> > >
> > > >
> > > > > 3) If thin pool size-autoextend fails, the user space program
will not know due to
> > > > > buffered IO. So "sync -f vmcore" is used during
kdump in 2nd kernel,
> > > > > to force sync vmcore data into disk.
> > > >
> > > > It would be good if this "sync -f vmcore" fix is sent as a
separate patch.
> > > > This is needed anyway irrespective of thin pool support.
> > > >
> > >
> > > OK, I will split it to a seperate patch.
> > >
> > > Thanks,
> > > Tao Liu
> > >
> > > > Thanks
> > > > Vivek
> > > >
> > > > >
> > > > > According to my testing, the memory consumption procedure for
lvm2 thinp is the thin pool
> > > > > size-autoextend phase. For fedora and rhel9, the default
crashkernel value is enough. But
> > > > > for rhel8, the default crashkernel value 1G-4G:160M is not
enough, so it should
> > > > > be handled particularly.
> > > > >
> > > > > v1 -> v2:
> > > > >
> > > > > 1) Modified the usage of lvs cmd when check if target is lvm2
thinp
> > > > > device.
> > > > > 2) Removed the sync flag way of mounting for lvm2 thinp target
> > > > > during kdump, use "sync -f vmcore" to force sync
data, and handle
> > > > > the error if fails.
> > > > >
> > > > > Tao Liu (4):
> > > > > Add lvm2 thin provision dump target checker
> > > > > Add lvm2-monitor.service for kdump when lvm2 thinp enabled
> > > > > lvm.conf should be check modified if lvm2 thinp enabled
> > > > > Fix the sync issue for dump_fs
> > > > >
> > > > > dracut-kdump.sh | 10 ++++++++--
> > > > > dracut-lvm2-monitor.service | 15 +++++++++++++++
> > > > > dracut-module-setup.sh | 16 ++++++++++++++++
> > > > > kdump-lib-initramfs.sh | 20 ++++++++++++++++++++
> > > > > kdumpctl | 1 +
> > > > > kexec-tools.spec | 2 ++
> > > > > 6 files changed, 62 insertions(+), 2 deletions(-)
> > > > > create mode 100644 dracut-lvm2-monitor.service
> > > > >
> > > > > --
> > > > > 2.33.1
> > > > >
> > > >
> > >
> >
>