+1. Do you have snapshot administration rights to remove the giant one which will happen after deleting compose.old?

On 2/22/23 12:51, Kevin Fenzi wrote: > Greetings. > > As some of you may know, our fedora_koji volume is hitting up against > some limits (namely the netapp 100TB per volume limit). If it hits 100TB > used, the netapp folks tell me it will go offline and we will need to do > special things to free up any space and get it working again. > Obviously, we wish to avoid that. > > So, I think we can move /mnt/fedora_koji/koji/compose with minimal > disruption and give us a bunch of room and actually make things faster. > > Here's my tenative plan: > > * create ~15-20TB volume on one of our ssd aggregates. > * rsync all of /mnt/fedora_koji/koji/compose/ to it. > * Schedule a changeover time/date. > * Make sure no composes or updates pushes are running. > (This should be possible after branched/rawhide, but before updates > and before we are making rc's) > * Do another sync of content so the new copy is up to date. > (I am not sure how long a rsync will take, but we can figure it out) > * move the old directory to compose.old > * mount the new space on koji01/02, kojipkgs01/02, all compose channel > builders, compose-x86-01. Nothing else should need it. > * Wait a short while > * delete compose.old > > This should free up about 13TB or so on the main volume, reduce snapshot > churn on it, make composes faster because they will be on ssd instead of > sas drives, and all around be nicer. > > I think this can be done during some day without really causing much > outage. Because the koji space is so tight I would like to do it soon, > and I think it best to do it before we are too close to release. > So, later this week or early next week? > > Thoughts? +1s? alternative ideas? I just want to make sure our ostree use cases are considered here. I think we are already on our own separate volume, so maybe this has no impact, but I do know at least the mount paths include `compose` in them so I'll list out what we do and the desire for it to continue to work: 1. pungi composes - composing into compose/ostree/repo 2. coreos-ostree-importer - importing into /mnt/koji/compose/ostree/repo - https://github.com/coreos/fedora-coreos-releng-automation/blob/main/coreo... 3. fedora-ostree-pruner - pruning /mnt/koji/compose/ostree/repo - https://github.com/coreos/fedora-coreos-releng-automation/blob/main/fedor...

4. the `fedora-compose` ostree repo (accessible via clients for testing purposes) - https://src.fedoraproject.org/rpms/fedora-repos/blob/rawhide/f/fedora-com... Dusty _______________________________________________ infrastructure mailing list -- infrastructure(a)lists.fedoraproject.org To unsubscribe send an email to infrastructure-leave(a)lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedora... Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue

Greetings. As some of you may know, our fedora_koji volume is hitting up against some limits (namely the netapp 100TB per volume limit). If it hits 100TB used, the netapp folks tell me it will go offline and we will need to do special things to free up any space and get it working again. Obviously, we wish to avoid that. So, I think we can move /mnt/fedora_koji/koji/compose with minimal disruption and give us a bunch of room and actually make things faster. Here's my tenative plan: * create ~15-20TB volume on one of our ssd aggregates. * rsync all of /mnt/fedora_koji/koji/compose/ to it. * Schedule a changeover time/date. * Make sure no composes or updates pushes are running. (This should be possible after branched/rawhide, but before updates and before we are making rc's) * Do another sync of content so the new copy is up to date. (I am not sure how long a rsync will take, but we can figure it out) * move the old directory to compose.old * mount the new space on koji01/02, kojipkgs01/02, all compose channel builders, compose-x86-01. Nothing else should need it. * Wait a short while * delete compose.old This should free up about 13TB or so on the main volume, reduce snapshot churn on it, make composes faster because they will be on ssd instead of sas drives, and all around be nicer. I think this can be done during some day without really causing much outage. Because the koji space is so tight I would like to do it soon, and I think it best to do it before we are too close to release. So, later this week or early next week? Thoughts? +1s? alternative ideas?

kevin _______________________________________________ infrastructure mailing list -- infrastructure(a)lists.fedoraproject.org To unsubscribe send an email to infrastructure-leave(a)lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedora... Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue