Re: "ABSENT" test results from Fedora CI: status update

Hey folks! Just an update on an issue I know some packagers are running into. In the last few days, some Fedora CI jobs have been getting stuck and timing out. Packagers will usually experience this as a test result showing as "ABSENT" in Bodhi, e.g. for this update: https://bodhi.fedoraproject.org/updates/FEDORA-2024-ce1fb5a0db the fedora-ci.koji-build.tier0.functional result shows as ABSENT (red line, question mark icon which shows ABSENT if you hover over it). We don't have any CI tests that are universally gating by default, but some packages have opted in to gating on some CI tests, and some of these have been stuck in updates-testing for a few days due to this problem. It seems the issue was caused by a kernel 6.8 bug which broke quite a lot of stuff last week - see https://lore.kernel.org/all/6a150ddd-3267-4f89-81bd-6807700c57c1@redhat.com/ . The fix for that bug was in kernel 6.8rc6, which is now in both Fedora 40 and Rawhide. Miroslav Vadkerti is aware of this and is working to try and clean things up (update the base images for Fedora CI and get all the timed-out tests re-run). I don't know what the ETA is, but just wanted to let folks know it's at least being worked on.