On 2/9/19 4:48 AM, Björn 'besser82' Esser wrote:
Am Freitag, den 08.02.2019, 08:41 -0800 schrieb Kevin Fenzi:
> Just wanted to update everyone on our current status.
>
> As you know from the other thread:
>
> * The mass rebuild happened and finished.
> * The mass rebuild side tag was merged into the f30-pending tag
> (to make sure everything was signed).
> * In the middle of the night our autosign box stopped processing.
> * The next morning this was noticed and a request for a replacement
> motherboard was sent.
> * Since that was going to take a day, we setup another machine to do
> autosigning.
> * That machine started processing the backlog, but also stopped
> processing a few times (waiting on koji).
> * Finally we set back up the normal listening process and I retagged
> all
> the f30-pending builds to it would "see" they needed processing.
>
> Currently it's processing along pretty fast, but it does make sure
> koji
> has the signed rpms written out, which can take a few seconds on
> larger
> packages.
>
> I'm hopeful that it will catch back up today and we can go back to
> normal.
>
> There will likely be a short outage next week to move back to the now
> replaced hardware, but that should be only a few minutes.
>
> kevin
Thank you for the extra work, Kevin and all other being involved with
it!
Just asking as this came to my mind from a different thread:
>> Looks like that it was only the mass rebuild packages that got stuck
in
>> the signing queue and other builds were processed normally (they
moved
>> from f30-pending to f30), so they are going to be out of order when
the
>> mass rebuild signing/tagging into f30 finishes.
>>
>> Is there any chance releng could figure out the list of packages
>> affected once the signing/tagging is finished and fix all of them? I
can
>> already tell that it's affecting at least 40-50 GNOME builds done in
>> that time frame.
> Yes, there's a releng script to fix this very case... can run after
> everything is finished signing/tagging.
Has that issue been adressed already, too?
Yes, I already fixed all those.
kevin