Am Freitag, den 08.02.2019, 08:41 -0800 schrieb Kevin Fenzi:
Just wanted to update everyone on our current status.
As you know from the other thread:
* The mass rebuild happened and finished.
* The mass rebuild side tag was merged into the f30-pending tag
(to make sure everything was signed).
* In the middle of the night our autosign box stopped processing.
* The next morning this was noticed and a request for a replacement
motherboard was sent.
* Since that was going to take a day, we setup another machine to do
autosigning.
* That machine started processing the backlog, but also stopped
processing a few times (waiting on koji).
* Finally we set back up the normal listening process and I retagged
all
the f30-pending builds to it would "see" they needed processing.
Currently it's processing along pretty fast, but it does make sure
koji
has the signed rpms written out, which can take a few seconds on
larger
packages.
I'm hopeful that it will catch back up today and we can go back to
normal.
There will likely be a short outage next week to move back to the now
replaced hardware, but that should be only a few minutes.
kevin
Thank you for the extra work, Kevin and all other being involved with
it!
Just asking as this came to my mind from a different thread:
> Looks like that it was only the mass rebuild packages that got
stuck
in
> the signing queue and other builds were processed normally (they
moved
> from f30-pending to f30), so they are going to be out of order
when
the
> mass rebuild signing/tagging into f30 finishes.
>
> Is there any chance releng could figure out the list of packages
> affected once the signing/tagging is finished and fix all of them? I
can
> already tell that it's affecting at least 40-50 GNOME builds
done in
> that time frame.
Yes, there's a releng script to fix this very case... can run
after
everything is finished signing/tagging.
Has that issue been adressed already, too?
Björn