9:03 a.m.
Hi all,
where is documented what system/hw is used on (primary) Koji builders?
I'm interested in memory, storage, filesystem, host operating system, guest
operating system (if those are VMs), etc.
The only thing I was able to find is version of mock in the log output.
FWIW, I'd like to reproduce hang in gnulib's test-lock [1] test case.
Unless I'm able to reproduce that somehow, I'll disable the test for the time
being (done in 'coreutils' and I guess elsewhere, too).
[1] https://koji.fedoraproject.org/koji/taskinfo?taskID=16970779
Thanks,
Pavel
9:57 a.m.
On Tue, 20 Dec 2016 16:03:49 +0100
Pavel Raiskup <praiskup(a)redhat.com> wrote:
Hi all,
where is documented what system/hw is used on (primary) Koji builders?
I'm interested in memory, storage, filesystem, host operating system,
guest operating system (if those are VMs), etc.
The only thing I was able to find is version of mock in the log
output.
FWIW, I'd like to reproduce hang in gnulib's test-lock [1] test case.
Unless I'm able to reproduce that somehow, I'll disable the test for
the time being (done in 'coreutils' and I guess elsewhere, too).
[1] https://koji.fedoraproject.org/koji/taskinfo?taskID=16970779
I don't think we have any document like that, but:
buildvm-01 to buildvm-32:
These are Fedora 25 Vm's with 15GB mem, 6 cpus, 256GB disk of which
128 is swap (in prep for testing tmpfs mock plugin, which is currently
NOT enabled).
arm*:
calexida armv7 SOC's. Fedora 25. Each has ~300GB disk, 4GB ram, 4 cpus.
buildhw*:
dell poweredge fx blades with 128GB ram, Fedora 25, 48 cpus, 2 400GB
drives in raid1.
buildppc:
VM on power8, Fedora 25, 10GB mem, 5 cpus, 150GB disk.
buildvm-aarch64:
VM's on HP moonshot cartridges. Fedora 24 (soon 25), 25GB memory,
4CPUs, 150GB disk.
In the cases where there are VM's, they are running on RHEL7.
It looks like your task hung on x86_64, i686 and ppc64.
The first two shoudl be easy to come by. Let us know (via an
infrastructure ticket) if you need a ppc64 vm.
kevin
10:04 a.m.
On Tue, 20 Dec 2016 16:03:49 +0100
Pavel Raiskup <praiskup(a)redhat.com> wrote:
Hi all,
where is documented what system/hw is used on (primary) Koji builders?
I'm interested in memory, storage, filesystem, host operating system,
guest operating system (if those are VMs), etc.
The only thing I was able to find is version of mock in the log
output.
FWIW, I'd like to reproduce hang in gnulib's test-lock [1] test case.
Unless I'm able to reproduce that somehow, I'll disable the test for
the time being (done in 'coreutils' and I guess elsewhere, too).
oh no, test-lock again :-( A year (or two) ago it turned to be a kernel
bug - https://bugzilla.redhat.com/show_bug.cgi?id=1155291
Dan
7:13 a.m.
On 21/12/16 16:04, Dan Horák wrote:
On Tue, 20 Dec 2016 16:03:49 +0100
Pavel Raiskup <praiskup(a)redhat.com> wrote:
> Hi all,
>
> where is documented what system/hw is used on (primary) Koji builders?
> I'm interested in memory, storage, filesystem, host operating system,
> guest operating system (if those are VMs), etc.
>
> The only thing I was able to find is version of mock in the log
> output.
>
> FWIW, I'd like to reproduce hang in gnulib's test-lock [1] test case.
> Unless I'm able to reproduce that somehow, I'll disable the test for
> the time being (done in 'coreutils' and I guess elsewhere, too).
oh no, test-lock again :-( A year (or two) ago it turned to be a kernel
bug - https://bugzilla.redhat.com/show_bug.cgi?id=1155291
I saw Kamil recently disabled that test for the coreutils update.
Note the test itself may have issues as discussed at:
https://lists.gnu.org/archive/html/bug-gnulib/2015-07/msg00032.html
7:13 a.m.
On 22/12/16 13:13, Pádraig Brady wrote:
On 21/12/16 16:04, Dan Horák wrote:
> On Tue, 20 Dec 2016 16:03:49 +0100
> Pavel Raiskup <praiskup(a)redhat.com> wrote:
>
>> Hi all,
>>
>> where is documented what system/hw is used on (primary) Koji builders?
>> I'm interested in memory, storage, filesystem, host operating system,
>> guest operating system (if those are VMs), etc.
>>
>> The only thing I was able to find is version of mock in the log
>> output.
>>
>> FWIW, I'd like to reproduce hang in gnulib's test-lock [1] test case.
>> Unless I'm able to reproduce that somehow, I'll disable the test for
>> the time being (done in 'coreutils' and I guess elsewhere, too).
>
> oh no, test-lock again :-( A year (or two) ago it turned to be a kernel
> bug - https://bugzilla.redhat.com/show_bug.cgi?id=1155291
I saw Kamil recently disabled that test for the coreutils update.
Note the test itself may have issues as discussed at:
https://lists.gnu.org/archive/html/bug-gnulib/2015-07/msg00032.html
Note this test was just changed upstream to use locks instead of volatiles,
which significantly improved performance on a 40 core NUMA system at least:
http://git.savannah.gnu.org/gitweb/?p=gnulib.git;a=commitdiff;h=480d374
cheers,
Pádraig
4 a.m.
On Sunday, December 25, 2016 13:13:31 Pádraig Brady wrote:
On 22/12/16 13:13, Pádraig Brady wrote:
> On 21/12/16 16:04, Dan Horák wrote:
>> On Tue, 20 Dec 2016 16:03:49 +0100
>>
>> Pavel Raiskup <praiskup(a)redhat.com> wrote:
>>> Hi all,
>>>
>>> where is documented what system/hw is used on (primary) Koji builders?
>>> I'm interested in memory, storage, filesystem, host operating system,
>>> guest operating system (if those are VMs), etc.
>>>
>>> The only thing I was able to find is version of mock in the log
>>> output.
>>>
>>> FWIW, I'd like to reproduce hang in gnulib's test-lock [1] test
case.
>>> Unless I'm able to reproduce that somehow, I'll disable the test
for
>>> the time being (done in 'coreutils' and I guess elsewhere, too).
>>
>> oh no, test-lock again :-( A year (or two) ago it turned to be a kernel
>> bug - https://bugzilla.redhat.com/show_bug.cgi?id=1155291
>
> I saw Kamil recently disabled that test for the coreutils update.
> Note the test itself may have issues as discussed at:
> https://lists.gnu.org/archive/html/bug-gnulib/2015-07/msg00032.html
Note this test was just changed upstream to use locks instead of volatiles,
which significantly improved performance on a 40 core NUMA system at least:
http://git.savannah.gnu.org/gitweb/?p=gnulib.git;a=commitdiff;h=480d374
Thanks for letting me know! I have applied the above patch on the coreutils
package in Fedora rawhide:
http://pkgs.fedoraproject.org/cgit/rpms/coreutils.git/commit/?id=b3b3da0c
... and it seems to have resolved the issue with hanging Koji builds.
Kamil
> cheers,
> Pádraig
7:40 a.m.
On Monday, January 2, 2017 11:00:44 AM CET Kamil Dudka wrote:
> Note this test was just changed upstream to use locks instead of
volatiles,
> which significantly improved performance on a 40 core NUMA system at least:
> http://git.savannah.gnu.org/gitweb/?p=gnulib.git;a=commitdiff;h=480d374
Thanks for letting me know! I have applied the above patch on the coreutils
package in Fedora rawhide:
http://pkgs.fedoraproject.org/cgit/rpms/coreutils.git/commit/?id=b3b3da0c
... and it seems to have resolved the issue with hanging Koji builds.
As we discussed in person with Kamil, unfortunately the upstream fix
doesn't help. And also most probably, I was wrong before when I told that
this is _not_ arch specific; it looks like it actually is ppc64le only
and, indeed!, the test_rwlock() test took incredibly long before the
upstream fix (not only on ppc64le). So a good performance speedup done
upstream, thanks!
But yes, there's still problem with test_rwlock() on ppc64le. Usually I
see this in build.log [1]:
...
+ set +x
1 Starting test_lock ... OK
9 Starting test_rwlock ... OK
10 Starting test_recursive_lock ... OK
13 Starting test_once ... OK
13 ====
...
but sometimes there's infinite hang [2]:
...
+ set +x
1 Starting test_lock ... OK
[HANG HERE]
That integer number on each line means "how many seconds left when the test
(being written to stdout) finished" [3].
Kamil pointed out that the test-lock.c might be wrong. The point is that
there are multiple independent read-only threads (up to 10?) in "infinite
loop until all read-write threads finish". Which causes probably writer's
starvation. POSIX only says that "Implementations may favor writers over
readers to avoid writer starvation.", but it is not guaranteed in general.
More precisely, we seem to have PTHREAD_RWLOCK_PREFER_READER_NP in glibc
by default, so I'm not sure whether we shouldn't set
PTHREAD_RWLOCK_PREFER_WRITER_NP in test-lock.c somewhere... at least if
possible. Also, the question is whether it isn't really glibc bug,
because schedulers non-ppc64le architectures look to be more "fair"
regardless the default.
Also, I was unable to reproduce anywhere else than in Koji.
[1] https://koji.fedoraproject.org/koji/taskinfo?taskID=17147902
[2] https://koji.fedoraproject.org/koji/taskinfo?taskID=17147995
[3]
http://pkgs.fedoraproject.org/cgit/rpms/gettext.git/commit/?h=private-pra...
Pavel
8:47 a.m.
On Monday, January 2, 2017 2:40:34 PM CET Pavel Raiskup wrote:
Also, the question is whether it isn't really glibc bug, because
schedulers non-ppc64le architectures look to be more "fair" regardless
the default.
Sorry, not really -- at least i686 (build cross-compiled from x86_64
probably) is also affected:
https://koji.fedoraproject.org/koji/taskinfo?taskID=17147577
Pavel
5:06 a.m.
On Monday, January 2, 2017 3:47:52 PM CET Pavel Raiskup wrote:
On Monday, January 2, 2017 2:40:34 PM CET Pavel Raiskup wrote:
> Also, the question is whether it isn't really glibc bug, because
> schedulers non-ppc64le architectures look to be more "fair" regardless
> the default.
Sorry, not really -- at least i686 (build cross-compiled from x86_64
probably) is also affected:
https://koji.fedoraproject.org/koji/taskinfo?taskID=17147577
FTR, I just filed:
https://bugzilla.redhat.com/show_bug.cgi?id=1410052
Who knows, maybe that could move this forward.
That's a result of gnulib discussion:
https://lists.gnu.org/archive/html/bug-gnulib/2017-01/msg00009.html
Pavel
6:58 a.m.
On Wednesday, January 4, 2017 12:06:40 PM CET Pavel Raiskup wrote:
On Monday, January 2, 2017 3:47:52 PM CET Pavel Raiskup wrote:
> On Monday, January 2, 2017 2:40:34 PM CET Pavel Raiskup wrote:
> > Also, the question is whether it isn't really glibc bug, because
> > schedulers non-ppc64le architectures look to be more "fair"
regardless
> > the default.
>
> Sorry, not really -- at least i686 (build cross-compiled from x86_64
> probably) is also affected:
> https://koji.fedoraproject.org/koji/taskinfo?taskID=17147577
It is nice to see that the build finished in the end (even though it took ~2
days). So not really infinite ...
... but what worries me is that both builds, even this one,
https://koji.fedoraproject.org/koji/taskinfo?taskID=17147897
finished almost at the same time; both tests took about 48 hours?
Is there some timeout, signal, or scheduler attempt to change something after 48
hours? :)
Pavel
FTR, I just filed:
https://bugzilla.redhat.com/show_bug.cgi?id=1410052
Who knows, maybe that could move this forward.
That's a result of gnulib discussion:
https://lists.gnu.org/archive/html/bug-gnulib/2017-01/msg00009.html
Pavel
_______________________________________________
devel mailing list -- devel(a)lists.fedoraproject.org
To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
7:14 a.m.
On Wed, 04 Jan 2017 13:58:38 +0100
Pavel Raiskup <praiskup(a)redhat.com> wrote:
On Wednesday, January 4, 2017 12:06:40 PM CET Pavel Raiskup wrote:
> On Monday, January 2, 2017 3:47:52 PM CET Pavel Raiskup wrote:
> > On Monday, January 2, 2017 2:40:34 PM CET Pavel Raiskup wrote:
> > > Also, the question is whether it isn't really glibc bug, because
> > > schedulers non-ppc64le architectures look to be more "fair"
> > > regardless the default.
> >
> > Sorry, not really -- at least i686 (build cross-compiled from
> > x86_64 probably) is also affected:
> > https://koji.fedoraproject.org/koji/taskinfo?taskID=17147577
It is nice to see that the build finished in the end (even though it
took ~2 days). So not really infinite ...
... but what worries me is that both builds, even this one,
https://koji.fedoraproject.org/koji/taskinfo?taskID=17147897
finished almost at the same time; both tests took about 48 hours?
Is there some timeout, signal, or scheduler attempt to change
something after 48 hours? :)
there is a timeout set in koji builder config that tells mock to kill
builds after 48 after hours
Dan
> Pavel
>
> > FTR, I just filed:
> > https://bugzilla.redhat.com/show_bug.cgi?id=1410052
> > Who knows, maybe that could move this forward.
> >
> > That's a result of gnulib discussion:
> > https://lists.gnu.org/archive/html/bug-gnulib/2017-01/msg00009.html
> >
> > Pavel
> > _______________________________________________
> > devel mailing list -- devel(a)lists.fedoraproject.org
> > To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
> >
>
> _______________________________________________
> devel mailing list -- devel(a)lists.fedoraproject.org
> To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
4:24 a.m.
Dne 20.12.2016 v 16:03 Pavel Raiskup napsal(a):
The only thing I was able to find is version of mock in the log
output.
Hmm, that is nice idea. Plugin for Mock which will print:
* number of CPU
* RAM + swap
* storage size available
I will try to work on that.
--
Miroslav Suchy, RHCA
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys
7:53 a.m.
On Monday, January 2, 2017 11:24:35 AM CET Miroslav Suchý wrote:
Dne 20.12.2016 v 16:03 Pavel Raiskup napsal(a):
> The only thing I was able to find is version of mock in the log output.
Hmm, that is nice idea. Plugin for Mock which will print:
* number of CPU
* RAM + swap
* storage size available
I will try to work on that.
Perfect, thanks!
Also it would be nice to see:
1. a note that multiple builds might be run concurrently on single Koji builder
2. what is the HW host running the builder VM
But I see those are not requests for mock.. Thus some wiki showing
builder's hostname to specs would be useful.
Also #2 is useless, because such hardware is usually not available for
maintainers ... (without ticket against fedora koji maintainers). But
some link for ticket creating would be useful on such wiki page.
Pavel
10:43 a.m.
Dne 2.1.2017 v 11:24 Miroslav Suchý napsal(a):
Hmm, that is nice idea. Plugin for Mock which will print:
* number of CPU
* RAM + swap
* storage size available
I will try to work on that.
Done. You can expect it in next version of Mock:
https://github.com/rpm-software-management/mock/wiki/Plugin-HwInfo
--
Miroslav Suchy, RHCA
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys
2669
days inactive
2684
days old
13 comments
6 participants
participants (6)
-
Dan Horák -
Kamil Dudka -
Kevin Fenzi -
Miroslav Suchý -
Pavel Raiskup -
Pádraig Brady