9:11 a.m.
I'm trying to overcommit storage space on one our hypervisors (and reclaim
space if possible after memory peak times) and it seems like kernel in
Fedora 33 isn't doing automatic discard on swap partitions? The swap is
enabled by:
$ swapon --discard /dev/sda
The guest (on RHEL 8 HV host) is started with virtio-scsi controller to
propagate the trim requests to the host storage, the volumes I tried were
both of raw and qcow2 format.
The 'fstrim -av' isn't propagated to SWAP mount points (expected
probably). The only thing that helps to shrink the host volume allocated
for guest swap is 'swapoff -a && swapon --discard /dev/sda' (but I assume
TRIM is sent to the driver in this case, and that the problem isn't in the
host libvirt setup).
But otherwise, freeing the guest memory manually, minimizing the swap usage,
etc. (experiments with tmpfs) doesn't result in freeing the allocated storage
blocks on host.
Am I doing something wrong, is this expected or is this a bug?
FTR, I tested "kernel-core-5.10.11-200.fc33.x86_64" and
"kernel-core-5.10.12-200.fc33.x86_64" but this is not a regression -- I'm
experimenting with TRIM in this setup for the first time.
Dmesg output seems to indicate page cluster trim is enabled (c symbol):
[ 3936.914260] Adding 33554428k swap on /dev/sda. Priority:-2 extents:1 across:33554428k
DscFS
Thanks,
Pavel
1:42 p.m.
On Sat, Feb 6, 2021 at 8:12 AM Pavel Raiskup <praiskup(a)redhat.com> wrote:
I'm trying to overcommit storage space on one our hypervisors (and reclaim
space if possible after memory peak times) and it seems like kernel in
Fedora 33 isn't doing automatic discard on swap partitions?
You might ask about it on linux-mm(a)kvack.org
$ swapon --discard /dev/sda
--discard without options enables both available options, so you get
an initial discard and then an async discard when pages are freed.
Quite a lot of applications don't seem to free their dirty pages until
they quit, which might also be worth asking about.
The 'fstrim -av' isn't propagated to SWAP mount points
(expected
probably).
It's not a file system so that doesn't work.
But otherwise, freeing the guest memory manually, minimizing the swap
usage,
etc. (experiments with tmpfs) doesn't result in freeing the allocated storage
blocks on host.
I think it's up to the application to free the pages, if they aren't
freed somehow then the kernel can't know what blocks to issue discards
for.
--
Chris Murphy
9:56 a.m.
New subject: Continuously trim SWAP in Fedora guest over virtio-scsi
controller
On Sat, Feb 06, 2021 at 04:11:46PM +0100, Pavel Raiskup wrote:
I'm trying to overcommit storage space on one our hypervisors
(and reclaim
space if possible after memory peak times) and it seems like kernel in
Fedora 33 isn't doing automatic discard on swap partitions? The swap is
enabled by:
$ swapon --discard /dev/sda
The guest (on RHEL 8 HV host) is started with virtio-scsi controller to
propagate the trim requests to the host storage, the volumes I tried were
both of raw and qcow2 format.
The 'fstrim -av' isn't propagated to SWAP mount points (expected
probably). The only thing that helps to shrink the host volume allocated
for guest swap is 'swapoff -a && swapon --discard /dev/sda' (but I
assume
TRIM is sent to the driver in this case, and that the problem isn't in the
host libvirt setup).
fstrim will only trim a regular filesystem. AFAIK swap is not
involved in any way when you do fstrim. On the other hand swapon --discard
is correct, but you may need to check that --discard=pages was
selected. I have no idea what particular operations will cause a TRIM
to actually be sent though - it could also depend on the kernel
version. It seems unlikely that the kernel would send TRIM for every
single freed page because that would be very expensive on some
devices.
If you want to monitor what requests are seen on the hypervisor side,
I'd suggest running this command on the hypervisor:
nbdkit memory 10G --filter=log logfile=/var/tmp/log
Modify the guest's libvirt XML (virsh edit guestname) to add:
<disk type='network' device='disk'>
<driver name='qemu' type='raw'/>
<source protocol='nbd'>
<host name='localhost' port='10809'/>
</source>
<target dev='sdb' bus='scsi'/>
</disk>
(You might have to open port 10809 on the firewall, or else use
a Unix domain socket)
Then put the swap on /dev/sdb in the guest.
Then do whatever experiments you need and you should see (or perhaps
not see) Trim requests in /var/tmp/log.
You might nee dto play with <driver ... discard="unmap" ...> setting
in the libvirt XML too. I can't remember if that's the default or
not.
Rich.
But otherwise, freeing the guest memory manually, minimizing the swap
usage,
etc. (experiments with tmpfs) doesn't result in freeing the allocated storage
blocks on host.
Am I doing something wrong, is this expected or is this a bug?
FTR, I tested "kernel-core-5.10.11-200.fc33.x86_64" and
"kernel-core-5.10.12-200.fc33.x86_64" but this is not a regression -- I'm
experimenting with TRIM in this setup for the first time.
Dmesg output seems to indicate page cluster trim is enabled (c symbol):
[ 3936.914260] Adding 33554428k swap on /dev/sda. Priority:-2 extents:1 across:33554428k
DscFS
Thanks,
Pavel
_______________________________________________
devel mailing list -- devel(a)lists.fedoraproject.org
To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines. Supports shell scripting,
bindings from many languages. http://libguestfs.org
1234
days inactive
1236
days old
2 comments
3 participants
participants (3)
-
Chris Murphy -
Pavel Raiskup -
Richard W.M. Jones