URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
elkoniu commented: """ Just run another test run to include `batch_size = 1000`. @sumit-bose I have only one domain in `sssd.conf`: ``` [sssd] config_file_version = 2 services = nss, pam, ifp, sudo debug_level = 0x3ff0 domains = files user = root
[nss] timeout = 30000 debug_level = 0x3ff0 debug_microseconds = true
[pam] timeout = 30000 debug_level = 0x3ff0
[sudo] timeout = 30000 debug_level = 0x3ff0
[ifp] timeout = 30000 debug_level = 0x3ff0
[domain/files] timeout = 30000 debug_level = 0x3ff0 debug_microseconds = true id_provider = files ```
Time needed to mark domain as inconsistent:
|Version|30 + 100 users|30 + 250 users|30 + 500 users| |:---:|:---:|:---:|:---:| |SSSD master|204ms|558ms|1168ms| |SSSD batch_size = 10|0,5ms|0,5ms|70ms| |SSSD batch_size = 1000|0,4ms|0,5ms|0,6ms|
Result: * Each variant of batching speeds up marking domain as inconsistent and solves https://github.com/SSSD/sssd/issues/5091
Time needed to add / remove 100, 250 and 500 users in bash for loop: |Version|Add 100|Del 100|Add 250|Del 250|Add 500|Del 500| |:---:|:---:|:---:|:---:|:---:|:---:|:---:| |Raw files|0m 7.463s|0m 7.618s|0m 19.567s|0m 18.589s|0m 40.271s|0m 39.313s| |SSSD master|0m 58.315s|1m 2.276s|4m 6.709s|4m 2.857s|14m 2.478s|14m 54.913s| |SSSD batch_size = 10| |SSSD batch_size = 1000|
![test-result](https://user-images.githubusercontent.com/2312108/123186997-9502b700-d499-11...)
"""
See the full comment at https://github.com/SSSD/sssd/pull/5552#issuecomment-867257247