Hey,
Thanks to Packit I noticed that a lot of our jobs are running longer than usual, and a quick glance at the Copr task queue[0] tells me there's something fishy going on. I opened a couple of jobs[1][2][3] and all of them seem to be stuck in the same step - signing the build RPMs:
builder-live.log: RPMResults finished
backend.log: [2023-12-05 16:22:41,769][ INFO][PID:2234386] Calling '/bin/sign -u rpmsoftwaremanagement#CI-libdnf5-pr1008@copr.fedorahosted.org -p' (attempt #2) [2023-12-05 16:22:43,127][WARNING][PID:2234386] Command '/bin/sign -u rpmsoftwaremanagement#CI-libdnf5-pr1008@copr.fedorahosted.org -p' failed with: unknown key: rpmsoftwaremanagement#CI-libdnf5-pr1008@copr.fedorahosted.org [2023-12-05 16:22:43,129][WARNING][PID:2234386] Going to sleep 20s and re-try. [2023-12-05 16:23:03,130][ INFO][PID:2234386] Calling '/bin/sign -u rpmsoftwaremanagement#CI-libdnf5-pr1008@copr.fedorahosted.org -p' (attempt #3) [2023-12-05 16:23:04,949][WARNING][PID:2234386] Command '/bin/sign -u rpmsoftwaremanagement#CI-libdnf5-pr1008@copr.fedorahosted.org -p' failed with: unknown key: rpmsoftwaremanagement#CI-libdnf5-pr1008@copr.fedorahosted.org [2023-12-05 16:23:04,950][WARNING][PID:2234386] Going to sleep 20s and re-try.
Looks like the last 20s sleep takes _way_ longer than 20s. Is this a known issue?
[0] https://copr.fedorainfracloud.org/status/running/ [1] https://copr.fedorainfracloud.org/coprs/packit/cockpit-project-cockpit-19698... [2] https://copr.fedorainfracloud.org/coprs/rpmsoftwaremanagement/CI-libdnf5-pr1... [3] https://copr.fedorainfracloud.org/coprs/eliasofwaffle/gnome-shell-patched/bu...
Dne 06. 12. 23 v 12:52 František Šumšal napsal(a):
Hey,
Thanks to Packit I noticed that a lot of our jobs are running longer than usual, and a quick glance at the Copr task queue[0] tells me there's something fishy going on. I opened a couple of jobs[1][2][3] and all of
...
Looks like the last 20s sleep takes _way_ longer than 20s. Is this a known issue?
Yes. Pavel, Jakub and Jirka was working on this most of today's date.
The culprit was that gpg key was not created for a project. We found that 'gpg2 --list-secret-keys` took 35 seconds to finish, but the client had a timeout 24 seconds. We solved it temporarily by spawning a beefier VM for sign server. Now gpg2 returns in 4 secs. It gives us time to look at how to solve it properly. In the meantime, your builds should build without a problem and the current queue is slowly processing.