Hi folks! I'm chasing down some interesting behavior on our Ubuntu 20.04 VMs. We're running sssd 2.3.3.
The people who run the underlying disks for our VM hosts have been complaining about a spike in disk write latency every 10min.
After investigation, I figured out the issue.
By default, every 10 mins Debian systems run a cronjob, which kicks off a script called debian-sa1. By default, this script is basically a no-op. It doesn't write to disk.
But running this cronjob (which runs as root by default) causes sssd_nss to get started up (on Ubuntu, it's socket activated, as you probably well know).
When sssd_nss starts up, it writes the memory-mapped cache in /var/lib/sss/mc/.
For us, it writes about 25 MiB of data. (Is that a lot?)
By default, sssd_nss has a 5-minute timeout, so, if nothing else looks up any user info, sssd_nss shuts down.
So, basically, every 10 mins, sssd_nss wakes up and writes 25 MiB. Across many thousands of VMs, this adds up to a lot of writes!
Oh RHEL, seems like sssd_nss runs continuously, not socket-activated. I am thinking of just doing the same on Ubuntu. This will keep the memory-mapped cache active all the time, which is fine.
It'll prevent all those disk writes.
However, I wanted to see if you had any other suggestions. Thanks!