10:41 a.m.
Just wanted to update everyone on our current status.
As you know from the other thread:
* The mass rebuild happened and finished.
* The mass rebuild side tag was merged into the f30-pending tag
(to make sure everything was signed).
* In the middle of the night our autosign box stopped processing.
* The next morning this was noticed and a request for a replacement
motherboard was sent.
* Since that was going to take a day, we setup another machine to do
autosigning.
* That machine started processing the backlog, but also stopped
processing a few times (waiting on koji).
* Finally we set back up the normal listening process and I retagged all
the f30-pending builds to it would "see" they needed processing.
Currently it's processing along pretty fast, but it does make sure koji
has the signed rpms written out, which can take a few seconds on larger
packages.
I'm hopeful that it will catch back up today and we can go back to normal.
There will likely be a short outage next week to move back to the now
replaced hardware, but that should be only a few minutes.
kevin
1:11 p.m.
On pe, 08 helmi 2019, Kevin Fenzi wrote:
Just wanted to update everyone on our current status.
As you know from the other thread:
* The mass rebuild happened and finished.
* The mass rebuild side tag was merged into the f30-pending tag
(to make sure everything was signed).
* In the middle of the night our autosign box stopped processing.
* The next morning this was noticed and a request for a replacement
motherboard was sent.
* Since that was going to take a day, we setup another machine to do
autosigning.
* That machine started processing the backlog, but also stopped
processing a few times (waiting on koji).
* Finally we set back up the normal listening process and I retagged all
the f30-pending builds to it would "see" they needed processing.
Currently it's processing along pretty fast, but it does make sure koji
has the signed rpms written out, which can take a few seconds on larger
packages.
I'm hopeful that it will catch back up today and we can go back to normal.
Looks like my build got caught up with this:
$ fedpkg build
Building freeipa-4.7.2-3.fc30 for rawhide
Created task: 32666402
Task info: https://koji.fedoraproject.org/koji/taskinfo?taskID=32666402
Watching tasks (this may be safely interrupted)...
32666402 build (rawhide, /rpms/freeipa.git:4dd28889a240255e780926cb9d4b2aa2ef3d3432):
free
32666402 build (rawhide, /rpms/freeipa.git:4dd28889a240255e780926cb9d4b2aa2ef3d3432): free
-> open (buildvm-15.phx2.fedoraproject.org)
32666403 buildSRPMFromSCM (/rpms/freeipa.git:4dd28889a240255e780926cb9d4b2aa2ef3d3432):
open (buildvm-s390x-06.s390.fedoraproject.org)
32666480 buildArch (freeipa-4.7.2-3.fc30.src.rpm, s390x): open
(buildvm-s390x-13.s390.fedoraproject.org)
32666478 buildArch (freeipa-4.7.2-3.fc30.src.rpm, aarch64): open
(buildvm-aarch64-19.arm.fedoraproject.org)
32666477 buildArch (freeipa-4.7.2-3.fc30.src.rpm, ppc64le): open
(buildvm-ppc64le-09.ppc.fedoraproject.org)
32666479 buildArch (freeipa-4.7.2-3.fc30.src.rpm, i686): open
(buildvm-31.phx2.fedoraproject.org)
32666481 buildArch (freeipa-4.7.2-3.fc30.src.rpm, armv7hl): open
(buildvm-armv7-24.arm.fedoraproject.org)
32666476 buildArch (freeipa-4.7.2-3.fc30.src.rpm, x86_64): open
(buildhw-07.phx2.fedoraproject.org)
32666403 buildSRPMFromSCM (/rpms/freeipa.git:4dd28889a240255e780926cb9d4b2aa2ef3d3432):
open (buildvm-s390x-06.s390.fedoraproject.org) -> closed
0 free 7 open 1 done 0 failed
32666479 buildArch (freeipa-4.7.2-3.fc30.src.rpm, i686): open
(buildvm-31.phx2.fedoraproject.org) -> closed
0 free 6 open 2 done 0 failed
32666480 buildArch (freeipa-4.7.2-3.fc30.src.rpm, s390x): open
(buildvm-s390x-13.s390.fedoraproject.org) -> closed
0 free 5 open 3 done 0 failed
32666476 buildArch (freeipa-4.7.2-3.fc30.src.rpm, x86_64): open
(buildhw-07.phx2.fedoraproject.org) -> closed
0 free 4 open 4 done 0 failed
32666477 buildArch (freeipa-4.7.2-3.fc30.src.rpm, ppc64le): open
(buildvm-ppc64le-09.ppc.fedoraproject.org) -> closed
0 free 3 open 5 done 0 failed
32666478 buildArch (freeipa-4.7.2-3.fc30.src.rpm, aarch64): open
(buildvm-aarch64-19.arm.fedoraproject.org) -> closed
0 free 2 open 6 done 0 failed
32666402 build (rawhide, /rpms/freeipa.git:4dd28889a240255e780926cb9d4b2aa2ef3d3432): open
(buildvm-15.phx2.fedoraproject.org) -> closed
0 free 1 open 7 done 0 failed
32666898 tagBuild (noarch): closed
Could not execute build: ('Connection aborted.', RemoteDisconnected('Remote
end closed connection without response'))
And now I have a build without any tags.
--
/ Alexander Bokovoy
Sr. Principal Software Engineer
Security / Identity Management Engineering
Red Hat Limited, Finland
3:32 p.m.
On 2/8/19 11:11 AM, Alexander Bokovoy wrote:
Looks like my build got caught up with this:
$ fedpkg build
Building freeipa-4.7.2-3.fc30 for rawhide
Created task: 32666402
Task info: https://koji.fedoraproject.org/koji/taskinfo?taskID=32666402
...snip...
And now I have a build without any tags.
No, that all looks normal, the build was tagged into f30-pending as
expected. So once the autosigner gets to it, it will be signed and moved
to f30. So yeah, that is delayed due to the signing queue, but otherwise
it looks ok to me.
https://koji.fedoraproject.org/koji/buildinfo?buildID=1206959
kevin
4 a.m.
On 08. 02. 19 17:41, Kevin Fenzi wrote:
Just wanted to update everyone on our current status.
...
I'm hopeful that it will catch back up today and we can go back to normal.
Is this somehow relevant?
https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-2019020...
2019-02-09 08:44:00 [ERROR ] Compose run failed: RPM(s) not found for sigs:
['CFC659B9']. Check log for details. Unsigned packages:
dar-2.6.0-2.fc30
dar-debuginfo-2.6.0-2.fc30
dar-debugsource-2.6.0-2.fc30
electrum-3.3.2-3.fc30
libdar-2.6.0-2.fc30
libdar-debuginfo-2.6.0-2.fc30
libdar-devel-2.6.0-2.fc30
mysqltuner-1.7.19-2.git.59e5f40.fc30
openscap-1.3.0_alpha2-2.fc30
openscap-containers-1.3.0_alpha2-2.fc30
openscap-debuginfo-1.3.0_alpha2-2.fc30
openscap-debugsource-1.3.0_alpha2-2.fc30
openscap-devel-1.3.0_alpha2-2.fc30
openscap-engine-sce-1.3.0_alpha2-2.fc30
openscap-engine-sce-debuginfo-1.3.0_alpha2-2.fc30
openscap-engine-sce-devel-1.3.0_alpha2-2.fc30
openscap-perl-1.3.0_alpha2-2.fc30
openscap-perl-debuginfo-1.3.0_alpha2-2.fc30
openscap-python3-1.3.0_alpha2-2.fc30
openscap-scanner-1.3.0_alpha2-2.fc30
openscap-scanner-debuginfo-1.3.0_alpha2-2.fc30
openscap-utils-1.3.0_alpha2-2.fc30
openscap-utils-debuginfo-1.3.0_alpha2-2.fc30
python-oslo-messaging-8.0.0-1.fc29
python-oslo-messaging-doc-8.0.0-1.fc29
python2-oslo-messaging-8.0.0-1.fc29
python2-oslo-messaging-tests-8.0.0-1.fc29
python3-oslo-messaging-8.0.0-1.fc29
python3-oslo-messaging-tests-8.0.0-1.fc29
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
12:33 p.m.
On 2/9/19 2:00 AM, Miro Hrončok wrote:
On 08. 02. 19 17:41, Kevin Fenzi wrote:
> Just wanted to update everyone on our current status.
>
> ...
>
> I'm hopeful that it will catch back up today and we can go back to
> normal.
Is this somehow relevant?
This was caused by my fixing upgrade path on f30.
ie, there was an older 'dar' tagged in, so I tagged in the 'newer' (by
NVR) one. It was signed, but apparently the written out rpms were
garbage collected already.
I'm fixing these up and will fire a new rawhide after.
kevin
--
https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-2019020...
2019-02-09 08:44:00 [ERROR ] Compose run failed: RPM(s) not found for
sigs: ['CFC659B9']. Check log for details. Unsigned packages:
dar-2.6.0-2.fc30
dar-debuginfo-2.6.0-2.fc30
dar-debugsource-2.6.0-2.fc30
electrum-3.3.2-3.fc30
libdar-2.6.0-2.fc30
libdar-debuginfo-2.6.0-2.fc30
libdar-devel-2.6.0-2.fc30
mysqltuner-1.7.19-2.git.59e5f40.fc30
openscap-1.3.0_alpha2-2.fc30
openscap-containers-1.3.0_alpha2-2.fc30
openscap-debuginfo-1.3.0_alpha2-2.fc30
openscap-debugsource-1.3.0_alpha2-2.fc30
openscap-devel-1.3.0_alpha2-2.fc30
openscap-engine-sce-1.3.0_alpha2-2.fc30
openscap-engine-sce-debuginfo-1.3.0_alpha2-2.fc30
openscap-engine-sce-devel-1.3.0_alpha2-2.fc30
openscap-perl-1.3.0_alpha2-2.fc30
openscap-perl-debuginfo-1.3.0_alpha2-2.fc30
openscap-python3-1.3.0_alpha2-2.fc30
openscap-scanner-1.3.0_alpha2-2.fc30
openscap-scanner-debuginfo-1.3.0_alpha2-2.fc30
openscap-utils-1.3.0_alpha2-2.fc30
openscap-utils-debuginfo-1.3.0_alpha2-2.fc30
python-oslo-messaging-8.0.0-1.fc29
python-oslo-messaging-doc-8.0.0-1.fc29
python2-oslo-messaging-8.0.0-1.fc29
python2-oslo-messaging-tests-8.0.0-1.fc29
python3-oslo-messaging-8.0.0-1.fc29
python3-oslo-messaging-tests-8.0.0-1.fc29
10:01 a.m.
On Sat, 2019-02-09 at 10:33 -0800, Kevin Fenzi wrote:
On 2/9/19 2:00 AM, Miro Hrončok wrote:
> On 08. 02. 19 17:41, Kevin Fenzi wrote:
> > Just wanted to update everyone on our current status.
> >
> > ...
> >
> > I'm hopeful that it will catch back up today and we can go back
> > to
> > normal.
>
> Is this somehow relevant?
This was caused by my fixing upgrade path on f30.
ie, there was an older 'dar' tagged in, so I tagged in the 'newer'
(by
NVR) one. It was signed, but apparently the written out rpms were
garbage collected already.
I'm fixing these up and will fire a new rawhide after.
Hi, I don't if it helps but I don't know you notice that Fedora-
Rawhide- 20190207 [1] still running and pungi.log [2] last write was
2019-02-08 16:45:52 ( 2 days ago) ...
Best regards,
[1]
https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-2019020...
[2]
https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-2019020...
kevin
--
>
>
https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-2019020...
>
>
> 2019-02-09 08:44:00 [ERROR ] Compose run failed: RPM(s) not found
> for
> sigs: ['CFC659B9']. Check log for details. Unsigned packages:
> dar-2.6.0-2.fc30
> dar-debuginfo-2.6.0-2.fc30
> dar-debugsource-2.6.0-2.fc30
> electrum-3.3.2-3.fc30
> libdar-2.6.0-2.fc30
> libdar-debuginfo-2.6.0-2.fc30
> libdar-devel-2.6.0-2.fc30
> mysqltuner-1.7.19-2.git.59e5f40.fc30
> openscap-1.3.0_alpha2-2.fc30
> openscap-containers-1.3.0_alpha2-2.fc30
> openscap-debuginfo-1.3.0_alpha2-2.fc30
> openscap-debugsource-1.3.0_alpha2-2.fc30
> openscap-devel-1.3.0_alpha2-2.fc30
> openscap-engine-sce-1.3.0_alpha2-2.fc30
> openscap-engine-sce-debuginfo-1.3.0_alpha2-2.fc30
> openscap-engine-sce-devel-1.3.0_alpha2-2.fc30
> openscap-perl-1.3.0_alpha2-2.fc30
> openscap-perl-debuginfo-1.3.0_alpha2-2.fc30
> openscap-python3-1.3.0_alpha2-2.fc30
> openscap-scanner-1.3.0_alpha2-2.fc30
> openscap-scanner-debuginfo-1.3.0_alpha2-2.fc30
> openscap-utils-1.3.0_alpha2-2.fc30
> openscap-utils-debuginfo-1.3.0_alpha2-2.fc30
> python-oslo-messaging-8.0.0-1.fc29
> python-oslo-messaging-doc-8.0.0-1.fc29
> python2-oslo-messaging-8.0.0-1.fc29
> python2-oslo-messaging-tests-8.0.0-1.fc29
> python3-oslo-messaging-8.0.0-1.fc29
> python3-oslo-messaging-tests-8.0.0-1.fc29
>
_______________________________________________
devel mailing list -- devel(a)lists.fedoraproject.org
To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines:
https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
--
Sérgio M. B.
11:02 a.m.
On 2/10/19 8:01 AM, Sérgio Basto wrote:
On Sat, 2019-02-09 at 10:33 -0800, Kevin Fenzi wrote:
> On 2/9/19 2:00 AM, Miro Hrončok wrote:
>> On 08. 02. 19 17:41, Kevin Fenzi wrote:
>>> Just wanted to update everyone on our current status.
>>>
>>> ...
>>>
>>> I'm hopeful that it will catch back up today and we can go back
>>> to
>>> normal.
>>
>> Is this somehow relevant?
>
> This was caused by my fixing upgrade path on f30.
> ie, there was an older 'dar' tagged in, so I tagged in the 'newer'
> (by
> NVR) one. It was signed, but apparently the written out rpms were
> garbage collected already.
>
> I'm fixing these up and will fire a new rawhide after.
Hi, I don't if it helps but I don't know you notice that Fedora-
Rawhide- 20190207 [1] still running and pungi.log [2] last write was
2019-02-08 16:45:52 ( 2 days ago) ...
It crashed/died and the status didn't update.
It's not still composing, it will just be stuck in started forever. ;(
Yesterdays failed in an anaconda traceback on the kde live, so I am
untagging the new anaconda(s) and restarting it.
Signing should be all caught up as of yesterday.
kevin
--
Best regards,
[1]
https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-2019020...
[2]
https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-2019020...
> kevin
> --
>>
>>
https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-2019020...
>>
>>
>> 2019-02-09 08:44:00 [ERROR ] Compose run failed: RPM(s) not found
>> for
>> sigs: ['CFC659B9']. Check log for details. Unsigned packages:
>> dar-2.6.0-2.fc30
>> dar-debuginfo-2.6.0-2.fc30
>> dar-debugsource-2.6.0-2.fc30
>> electrum-3.3.2-3.fc30
>> libdar-2.6.0-2.fc30
>> libdar-debuginfo-2.6.0-2.fc30
>> libdar-devel-2.6.0-2.fc30
>> mysqltuner-1.7.19-2.git.59e5f40.fc30
>> openscap-1.3.0_alpha2-2.fc30
>> openscap-containers-1.3.0_alpha2-2.fc30
>> openscap-debuginfo-1.3.0_alpha2-2.fc30
>> openscap-debugsource-1.3.0_alpha2-2.fc30
>> openscap-devel-1.3.0_alpha2-2.fc30
>> openscap-engine-sce-1.3.0_alpha2-2.fc30
>> openscap-engine-sce-debuginfo-1.3.0_alpha2-2.fc30
>> openscap-engine-sce-devel-1.3.0_alpha2-2.fc30
>> openscap-perl-1.3.0_alpha2-2.fc30
>> openscap-perl-debuginfo-1.3.0_alpha2-2.fc30
>> openscap-python3-1.3.0_alpha2-2.fc30
>> openscap-scanner-1.3.0_alpha2-2.fc30
>> openscap-scanner-debuginfo-1.3.0_alpha2-2.fc30
>> openscap-utils-1.3.0_alpha2-2.fc30
>> openscap-utils-debuginfo-1.3.0_alpha2-2.fc30
>> python-oslo-messaging-8.0.0-1.fc29
>> python-oslo-messaging-doc-8.0.0-1.fc29
>> python2-oslo-messaging-8.0.0-1.fc29
>> python2-oslo-messaging-tests-8.0.0-1.fc29
>> python3-oslo-messaging-8.0.0-1.fc29
>> python3-oslo-messaging-tests-8.0.0-1.fc29
>>
>
>
> _______________________________________________
> devel mailing list -- devel(a)lists.fedoraproject.org
> To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> List Guidelines:
> https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
5:10 a.m.
On Sun, 2019-02-10 at 09:02 -0800, Kevin Fenzi wrote:
On 2/10/19 8:01 AM, Sérgio Basto wrote:
> On Sat, 2019-02-09 at 10:33 -0800, Kevin Fenzi wrote:
> > On 2/9/19 2:00 AM, Miro Hrončok wrote:
> > > On 08. 02. 19 17:41, Kevin Fenzi wrote:
> > > > Just wanted to update everyone on our current status.
> > > >
> > > > ...
> > > >
> > > > I'm hopeful that it will catch back up today and we can go back
> > > > to
> > > > normal.
> > >
> > > Is this somehow relevant?
> >
> > This was caused by my fixing upgrade path on f30.
> > ie, there was an older 'dar' tagged in, so I tagged in the
'newer'
> > (by
> > NVR) one. It was signed, but apparently the written out rpms were
> > garbage collected already.
> >
> > I'm fixing these up and will fire a new rawhide after.
>
> Hi, I don't if it helps but I don't know you notice that Fedora-
> Rawhide- 20190207 [1] still running and pungi.log [2] last write was
> 2019-02-08 16:45:52 ( 2 days ago) ...
It crashed/died and the status didn't update.
It's not still composing, it will just be stuck in started forever. ;(
Yesterdays failed in an anaconda traceback on the kde live, so I am
untagging the new anaconda(s) and restarting it.
Is there a bug/link to the logs ?
Signing should be all caught up as of yesterday.
kevin
--
> Best regards,
>
> [1]
> https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-2019020...
>
> [2]
>
https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-2019020...
>
>
> > kevin
> > --
> > >
>
https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-2019020...
> > >
> > > 2019-02-09 08:44:00 [ERROR ] Compose run failed: RPM(s) not found
> > > for
> > > sigs: ['CFC659B9']. Check log for details. Unsigned packages:
> > > dar-2.6.0-2.fc30
> > > dar-debuginfo-2.6.0-2.fc30
> > > dar-debugsource-2.6.0-2.fc30
> > > electrum-3.3.2-3.fc30
> > > libdar-2.6.0-2.fc30
> > > libdar-debuginfo-2.6.0-2.fc30
> > > libdar-devel-2.6.0-2.fc30
> > > mysqltuner-1.7.19-2.git.59e5f40.fc30
> > > openscap-1.3.0_alpha2-2.fc30
> > > openscap-containers-1.3.0_alpha2-2.fc30
> > > openscap-debuginfo-1.3.0_alpha2-2.fc30
> > > openscap-debugsource-1.3.0_alpha2-2.fc30
> > > openscap-devel-1.3.0_alpha2-2.fc30
> > > openscap-engine-sce-1.3.0_alpha2-2.fc30
> > > openscap-engine-sce-debuginfo-1.3.0_alpha2-2.fc30
> > > openscap-engine-sce-devel-1.3.0_alpha2-2.fc30
> > > openscap-perl-1.3.0_alpha2-2.fc30
> > > openscap-perl-debuginfo-1.3.0_alpha2-2.fc30
> > > openscap-python3-1.3.0_alpha2-2.fc30
> > > openscap-scanner-1.3.0_alpha2-2.fc30
> > > openscap-scanner-debuginfo-1.3.0_alpha2-2.fc30
> > > openscap-utils-1.3.0_alpha2-2.fc30
> > > openscap-utils-debuginfo-1.3.0_alpha2-2.fc30
> > > python-oslo-messaging-8.0.0-1.fc29
> > > python-oslo-messaging-doc-8.0.0-1.fc29
> > > python2-oslo-messaging-8.0.0-1.fc29
> > > python2-oslo-messaging-tests-8.0.0-1.fc29
> > > python3-oslo-messaging-8.0.0-1.fc29
> > > python3-oslo-messaging-tests-8.0.0-1.fc29
> > >
> >
> > _______________________________________________
> > devel mailing list -- devel(a)lists.fedoraproject.org
> > To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
> > Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> > List Guidelines:
> > https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives:
> > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
_______________________________________________
devel mailing list -- devel(a)lists.fedoraproject.org
To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
12:27 p.m.
On 2/11/19 3:10 AM, Martin Kolman wrote:
On Sun, 2019-02-10 at 09:02 -0800, Kevin Fenzi wrote:
> It crashed/died and the status didn't update.
>
> It's not still composing, it will just be stuck in started forever. ;(
>
> Yesterdays failed in an anaconda traceback on the kde live, so I am
> untagging the new anaconda(s) and restarting it.
Is there a bug/link to the logs ?
Yeah, sorry:
filed:
https://bugzilla.redhat.com/show_bug.cgi?id=1674605
kevin
6:48 a.m.
Am Freitag, den 08.02.2019, 08:41 -0800 schrieb Kevin Fenzi:
Just wanted to update everyone on our current status.
As you know from the other thread:
* The mass rebuild happened and finished.
* The mass rebuild side tag was merged into the f30-pending tag
(to make sure everything was signed).
* In the middle of the night our autosign box stopped processing.
* The next morning this was noticed and a request for a replacement
motherboard was sent.
* Since that was going to take a day, we setup another machine to do
autosigning.
* That machine started processing the backlog, but also stopped
processing a few times (waiting on koji).
* Finally we set back up the normal listening process and I retagged
all
the f30-pending builds to it would "see" they needed processing.
Currently it's processing along pretty fast, but it does make sure
koji
has the signed rpms written out, which can take a few seconds on
larger
packages.
I'm hopeful that it will catch back up today and we can go back to
normal.
There will likely be a short outage next week to move back to the now
replaced hardware, but that should be only a few minutes.
kevin
Thank you for the extra work, Kevin and all other being involved with
it!
Just asking as this came to my mind from a different thread:
> Looks like that it was only the mass rebuild packages that got
stuck
in
> the signing queue and other builds were processed normally (they
moved
> from f30-pending to f30), so they are going to be out of order
when
the
> mass rebuild signing/tagging into f30 finishes.
>
> Is there any chance releng could figure out the list of packages
> affected once the signing/tagging is finished and fix all of them? I
can
> already tell that it's affecting at least 40-50 GNOME builds
done in
> that time frame.
Yes, there's a releng script to fix this very case... can run
after
everything is finished signing/tagging.
Has that issue been adressed already, too?
Björn
12:34 p.m.
On 2/9/19 4:48 AM, Björn 'besser82' Esser wrote:
Am Freitag, den 08.02.2019, 08:41 -0800 schrieb Kevin Fenzi:
> Just wanted to update everyone on our current status.
>
> As you know from the other thread:
>
> * The mass rebuild happened and finished.
> * The mass rebuild side tag was merged into the f30-pending tag
> (to make sure everything was signed).
> * In the middle of the night our autosign box stopped processing.
> * The next morning this was noticed and a request for a replacement
> motherboard was sent.
> * Since that was going to take a day, we setup another machine to do
> autosigning.
> * That machine started processing the backlog, but also stopped
> processing a few times (waiting on koji).
> * Finally we set back up the normal listening process and I retagged
> all
> the f30-pending builds to it would "see" they needed processing.
>
> Currently it's processing along pretty fast, but it does make sure
> koji
> has the signed rpms written out, which can take a few seconds on
> larger
> packages.
>
> I'm hopeful that it will catch back up today and we can go back to
> normal.
>
> There will likely be a short outage next week to move back to the now
> replaced hardware, but that should be only a few minutes.
>
> kevin
Thank you for the extra work, Kevin and all other being involved with
it!
Just asking as this came to my mind from a different thread:
>> Looks like that it was only the mass rebuild packages that got stuck
in
>> the signing queue and other builds were processed normally (they
moved
>> from f30-pending to f30), so they are going to be out of order when
the
>> mass rebuild signing/tagging into f30 finishes.
>>
>> Is there any chance releng could figure out the list of packages
>> affected once the signing/tagging is finished and fix all of them? I
can
>> already tell that it's affecting at least 40-50 GNOME builds done in
>> that time frame.
> Yes, there's a releng script to fix this very case... can run after
> everything is finished signing/tagging.
Has that issue been adressed already, too?
Yes, I already fixed all those.
kevin
4:07 a.m.
Hi,
On 08-02-19 17:41, Kevin Fenzi wrote:
Just wanted to update everyone on our current status.
As you know from the other thread:
* The mass rebuild happened and finished.
* The mass rebuild side tag was merged into the f30-pending tag
(to make sure everything was signed).
* In the middle of the night our autosign box stopped processing.
* The next morning this was noticed and a request for a replacement
motherboard was sent.
* Since that was going to take a day, we setup another machine to do
autosigning.
* That machine started processing the backlog, but also stopped
processing a few times (waiting on koji).
* Finally we set back up the normal listening process and I retagged all
the f30-pending builds to it would "see" they needed processing.
Currently it's processing along pretty fast, but it does make sure koji
has the signed rpms written out, which can take a few seconds on larger
packages.
I'm hopeful that it will catch back up today and we can go back to normal.
There will likely be a short outage next week to move back to the now
replaced hardware, but that should be only a few minutes.
Thank you and the entire infra team for quickly fixing this!
And more in general a big thank you to the entire infra team for all
the heroic work you are doing on an almost daily basis.
Regards,
Hans
10:13 a.m.
On Sun, 10 Feb 2019 at 21:17, Kevin Fenzi <kevin(a)scrye.com> wrote:
Just wanted to update everyone on our current status.
As you know from the other thread:
* The mass rebuild happened and finished.
Is there anywhere some summary stats about mass update?
How many packages has been sent to rebuild vs those which rebuild failed?
kloczek
--
Tomasz Kłoczko | LinkedIn: http://lnkd.in/FXPWxH
12:35 p.m.
On 2/11/19 8:13 AM, Tomasz Kłoczko wrote:
On Sun, 10 Feb 2019 at 21:17, Kevin Fenzi <kevin(a)scrye.com>
wrote:
> Just wanted to update everyone on our current status.
>
> As you know from the other thread:
>
> * The mass rebuild happened and finished.
>
Is there anywhere some summary stats about mass update?
How many packages has been sent to rebuild vs those which rebuild failed?
There is:
https://kojipkgs.fedoraproject.org/mass-rebuild/f30-failures.html
and
https://kojipkgs.fedoraproject.org/mass-rebuild/f30-need-rebuild.html
The FTBFS bugs are being filed now I think...
kevin
2:46 a.m.
On Mon, Feb 11, 2019 at 10:35:55AM -0800, Kevin Fenzi wrote:
>>
>> * The mass rebuild happened and finished.
>
> Is there anywhere some summary stats about mass update?
> How many packages has been sent to rebuild vs those which rebuild failed?
There is:
https://kojipkgs.fedoraproject.org/mass-rebuild/f30-failures.html
and
https://kojipkgs.fedoraproject.org/mass-rebuild/f30-need-rebuild.html
The FTBFS bugs are being filed now I think...
My packages were caught in the crossfire:
https://bugzilla.redhat.com/show_bug.cgi?id=1675617
https://bugzilla.redhat.com/show_bug.cgi?id=1675618
https://bugzilla.redhat.com/show_bug.cgi?id=1675634
All of them failed on armv7hl, with some package (for example
glibc-2.29-1.fc30.1.armv7hl.rpm) missing in the mock step.
How should maintainers proceed in such situation?
--
Jan Pazdziora
Senior Principal Software Engineer, Security Engineering, Red Hat
3:19 a.m.
On Tue, Feb 12, 2019 at 8:49 AM Jan Pazdziora <jpazdziora(a)redhat.com> wrote:
On Mon, Feb 11, 2019 at 10:35:55AM -0800, Kevin Fenzi wrote:
> >>
> >> * The mass rebuild happened and finished.
> >
> > Is there anywhere some summary stats about mass update?
> > How many packages has been sent to rebuild vs those which rebuild failed?
>
> There is:
>
> https://kojipkgs.fedoraproject.org/mass-rebuild/f30-failures.html
>
> and
>
> https://kojipkgs.fedoraproject.org/mass-rebuild/f30-need-rebuild.html
>
> The FTBFS bugs are being filed now I think...
My packages were caught in the crossfire:
https://bugzilla.redhat.com/show_bug.cgi?id=1675617
https://bugzilla.redhat.com/show_bug.cgi?id=1675618
https://bugzilla.redhat.com/show_bug.cgi?id=1675634
All of them failed on armv7hl, with some package (for example
glibc-2.29-1.fc30.1.armv7hl.rpm) missing in the mock step.
How should maintainers proceed in such situation?
If the problem is now fixed you should just be able to do "fedpkg
build" to rebuild it and it'll proceed as usual.
5:14 a.m.
On Tue, Feb 12, 2019 at 09:19:03AM +0000, Peter Robinson wrote:
On Tue, Feb 12, 2019 at 8:49 AM Jan Pazdziora
<jpazdziora(a)redhat.com> wrote:
>
> All of them failed on armv7hl, with some package (for example
> glibc-2.29-1.fc30.1.armv7hl.rpm) missing in the mock step.
>
> How should maintainers proceed in such situation?
If the problem is now fixed you should just be able to do "fedpkg
build" to rebuild it and it'll proceed as usual.
Thanks, that passed now.
--
Jan Pazdziora
Senior Principal Software Engineer, Security Engineering, Red Hat
1915
days inactive
1919
days old
16 comments
10 participants
participants (10)
-
Alexander Bokovoy -
Björn 'besser82' Esser -
Hans de Goede -
Jan Pazdziora -
Kevin Fenzi -
Martin Kolman -
Miro Hrončok -
Peter Robinson -
Sérgio Basto -
Tomasz Kłoczko