1:05 p.m.
several of my `koji --scratch --arch-overide=s390x ...` builds have failed
with error; reading package header (after the rebuildSRPM)
Latest is https://koji.fedoraproject.org/koji/taskinfo?taskID=41622496
(guess I could reopen my s390x disk full ticket. Or open a new one.)
--
Kaleb
6:41 p.m.
On Tue, Feb 18, 2020 at 02:05:07PM -0500, Kaleb Keithley wrote:
several of my `koji --scratch --arch-overide=s390x ...` builds have
failed
with error; reading package header (after the rebuildSRPM)
Latest is https://koji.fedoraproject.org/koji/taskinfo?taskID=41622496
(guess I could reopen my s390x disk full ticket. Or open a new one.)
Sure, but thats not going to magicially make us solve the problem. ;(
It seems to happen sporadically, when the s390x builders are under heavy
load. It seems to happen to the Zvm instances more than the Kvm
instances. :(
I think they might be in odd states, so I am going to try and reboot
them tonight when things aren't as busy and see if that helps any.
kevin
8:57 p.m.
On Tue, Feb 18, 2020 at 04:41:42PM -0800, Kevin Fenzi wrote:
On Tue, Feb 18, 2020 at 02:05:07PM -0500, Kaleb Keithley wrote:
> several of my `koji --scratch --arch-overide=s390x ...` builds have failed
> with error; reading package header (after the rebuildSRPM)
>
> Latest is https://koji.fedoraproject.org/koji/taskinfo?taskID=41622496
>
> (guess I could reopen my s390x disk full ticket. Or open a new one.)
Sure, but thats not going to magicially make us solve the problem. ;(
It seems to happen sporadically, when the s390x builders are under heavy
load. It seems to happen to the Zvm instances more than the Kvm
instances. :(
I think they might be in odd states, so I am going to try and reboot
them tonight when things aren't as busy and see if that helps any.
FWIW, I have rebooted all the s390x builders now.
Hopefully that will help with this issue...
kevin
12:05 p.m.
On Tue, Feb 18, 2020 at 9:59 PM Kevin Fenzi <kevin(a)scrye.com> wrote:
On Tue, Feb 18, 2020 at 04:41:42PM -0800, Kevin Fenzi wrote:
> On Tue, Feb 18, 2020 at 02:05:07PM -0500, Kaleb Keithley wrote:
> > several of my `koji --scratch --arch-overide=s390x ...` builds have
failed
> > with error; reading package header (after the rebuildSRPM)
> >
> > Latest is https://koji.fedoraproject.org/koji/taskinfo?taskID=41622496
> >
> > (guess I could reopen my s390x disk full ticket. Or open a new one.)
>
> Sure, but thats not going to magicially make us solve the problem. ;(
>
> It seems to happen sporadically, when the s390x builders are under heavy
> load. It seems to happen to the Zvm instances more than the Kvm
> instances. :(
>
> I think they might be in odd states, so I am going to try and reboot
> them tonight when things aren't as busy and see if that helps any.
FWIW, I have rebooted all the s390x builders now.
Hopefully that will help with this issue...
The original issue seemed to clear itself up. Or at least by random chance
I got a builder that wasn't full.
But since you rebooted I've fired off a build, only to have it die at the
very end writing the rpm files. :-(
https://koji.fedoraproject.org/koji/getfile?taskID=41651353&volume=DE...
it's not the kernel, but never the less, these aren't quick builds,
especially on s390x so it's very frustrating.
Let's push it back up the hill....
--
Kaleb
12:17 p.m.
On Wed, 19 Feb 2020, Kaleb Keithley wrote:
long ago
diskcheck
was in Centos 2, and early Fedora
/var/ftp/pub/nfs/mirror/redhat/fedora/1/SRPMS/diskcheck-1.4-5.src.rpm
Why not build and install it so the admins would get warnings
?
Also, when a build fails for:
error: create archive failed on file <snip>
cpio: write failed - No space left on device
why not also emit a fedbus, or related message ?
-- Russ herrold
3:40 p.m.
On Wed, Feb 19, 2020 at 01:17:29PM -0500, R P Herrold wrote:
On Wed, 19 Feb 2020, Kaleb Keithley wrote:
long ago
diskcheck
was in Centos 2, and early Fedora
/var/ftp/pub/nfs/mirror/redhat/fedora/1/SRPMS/diskcheck-1.4-5.src.rpm
Why not build and install it so the admins would get warnings
?
Because it's not that simple.
Also, when a build fails for:
error: create archive failed on file <snip>
cpio: write failed - No space left on device
why not also emit a fedbus, or related message ?
Sure we could, but what action do we take?
kojid keeps track of buildroots it has still around and culls those from
failed or successfull builds. It also won't accept new builds if there's
too low a space. The problem shows up when there's a ton of small
buildroots (koschei) or multiple large builds unpack on the same builder
and mess up each other.
Hopefully adjust koschei will fix most of this... but after that we
should just tweak the kojid parameters until we get something that
works. (I note that I saw no signs of this while koschei was off).
kevin
3:37 p.m.
On Wed, Feb 19, 2020 at 01:05:59PM -0500, Kaleb Keithley wrote:
The original issue seemed to clear itself up. Or at least by random chance
I got a builder that wasn't full.
Its not got anything to do will full as far as we can tell.
It seems sporadic. The best kind of bugs to try and solve. ;(
But since you rebooted I've fired off a build, only to have it die at the
very end writing the rpm files. :-(
https://koji.fedoraproject.org/koji/getfile?taskID=41651353&volume=DE...
it's not the kernel, but never the less, these aren't quick builds,
especially on s390x so it's very frustrating.
That issue is disk space, which is unfortunately a complex issue on
builders.
That builder (BTW, please post links to top level taskid. it makes it
easier to find things) has currently 59 buildroots on it since I
rebooted it yesterday, taking up about 81GB of space. I suspect most of
those builds are koschei ones, but it could be some are things like
libreoffice or large module builds. Hopefully we can get koschei to stop
taking up space soon... but then past that we need to lower the time to
keep failed buildroots around or prevent so many builds using the same
builders. ie, think of the case where your build and say libreoffice
were using the same builder... it could easily take up all space between
the two, but be fine one at a time.
Let's push it back up the hill....
Sorry for the hassles. ;(
kevin
6:10 a.m.
New subject: disks full build errors again, was Re: s390x builder problem? disk
full, or ?
<top post>
Hi,
I've had four ceph builds die in the last 12ish hours. One of them was a
scratch build on x86_64; the others were regular builds, one on ppc64le,
and the other two on x86_64.
I don't know if this is a new problem or just repeat occurrences of a long
standing problem.
Anyway, just FYI. (I can provide the tasks, if you're unable to find them
easily in koji.)
</top post>
On Tue, Feb 18, 2020 at 2:05 PM Kaleb Keithley <kkeithle(a)redhat.com> wrote:
several of my `koji --scratch --arch-overide=s390x ...` builds have
failed with error; reading package header (after the rebuildSRPM)
Latest is https://koji.fedoraproject.org/koji/taskinfo?taskID=41622496
(guess I could reopen my s390x disk full ticket. Or open a new one.)
--
Kaleb
8:28 a.m.
New subject: disks full build errors again, was Re: s390x builder problem?
disk full, or ?
Now five. ppc64le this time.
On Tue, Mar 24, 2020 at 7:10 AM Kaleb Keithley <kkeithle(a)redhat.com> wrote:
<top post>
Hi,
I've had four ceph builds die in the last 12ish hours. One of them was a
scratch build on x86_64; the others were regular builds, one on ppc64le,
and the other two on x86_64.
I don't know if this is a new problem or just repeat occurrences of a long
standing problem.
Anyway, just FYI. (I can provide the tasks, if you're unable to find them
easily in koji.)
</top post>
On Tue, Feb 18, 2020 at 2:05 PM Kaleb Keithley <kkeithle(a)redhat.com>
wrote:
>
> several of my `koji --scratch --arch-overide=s390x ...` builds have
> failed with error; reading package header (after the rebuildSRPM)
>
> Latest is https://koji.fedoraproject.org/koji/taskinfo?taskID=41622496
>
> (guess I could reopen my s390x disk full ticket. Or open a new one.)
>
> --
>
> Kaleb
>
>
12:12 p.m.
New subject: disks full build errors again, was Re: s390x builder problem?
disk full, or ?
On Tue, Mar 24, 2020 at 09:28:35AM -0400, Kaleb Keithley wrote:
Now five. ppc64le this time.
I've cleaned these up now.
Mostly it was due to the upgrade on the builders this weekend pulling in
mock 2.1 and enabling it's 'bootstrap' mode, so it made bootstrap cache
files for everything. I've downgraded to 1.4.21 until we can make the
needed changes in koji and removed all these leftover caches.
Just out of curiosity... why is ceph so gigantic? It looks like it's
70-80GB unpacked, which I think makes it bigger than libreoffice.
Are there really big test files? Just a lot of code?
kevin
12:46 p.m.
New subject: disks full build errors again, was Re: s390x builder problem?
disk full, or ?
On Tue, Mar 24, 2020 at 1:13 PM Kevin Fenzi <kevin(a)scrye.com> wrote:
On Tue, Mar 24, 2020 at 09:28:35AM -0400, Kaleb Keithley wrote:
> Now five. ppc64le this time.
I've cleaned these up now.
Mostly it was due to the upgrade on the builders this weekend pulling in
mock 2.1 and enabling it's 'bootstrap' mode, so it made bootstrap cache
files for everything. I've downgraded to 1.4.21 until we can make the
needed changes in koji and removed all these leftover caches.
Just out of curiosity... why is ceph so gigantic? It looks like it's
70-80GB unpacked, which I think makes it bigger than libreoffice.
Are there really big test files? Just a lot of code?
It has always been big. The source tarball has gotten nearly 50% bigger
just since 14.1.0. (About one year ago.)
There are a lot of things bundled into it too for the cases where the
platform doesn't have the bleeding edge dependencies that they seem to
think they need to use. IMO they kinda paint themselves into a corner with
some of those, e.g. boost. They don't necessarily always build them though
so some of that is just a bloated source tarball. :-/
Even if current rawhide updates to boost-1.71 (which it really should do I
suppose) it'll still be bundled into the source. (Note that the ceph build
only builds selected bits of boost, not the whole thing.) There are
probably other, similar kinds of things in there, but I don't have detailed
knowledge of all the things that are bundled in it.
Convincing them to pick a set of common denominator dependencies that are
in all the Linux distributions they intend to support and code to that has
— AFAIK — fallen on deaf ears. I know I'm not the only one who has tried.
--
Kaleb
1487
days inactive
1522
days old
10 comments
3 participants
participants (3)
-
Kaleb Keithley -
Kevin Fenzi -
R P Herrold