1:08 a.m.
Hello,
recently, package 'community-mysql' started to FTBFS, because of lack
of resources on armv7hl builders in KOJI.
Packages like 'mariadb' and 'community-mysql' are quite resource
hungry, both for disk and memory.
Occasionally it happens that out of random reasons, some builder here
and there don't have enough resources. Usually because that particular
builder was under heavy load.
However this time, the armv7hl builds lack the resources in 100%
builds for days in various times on various hosts.
(1) Did the KOJI armv7hl builder's configuration change recently ?
(2) Should I file an issue somewhere?
Generic https://pagure.io/releng/new_issue ?
First spotted: F35 mass rebuild.
Reproducible: 100%, on all F33..Rawhide
---
https://koji.fedoraproject.org/koji/taskinfo?taskID=73160741
https://koji.fedoraproject.org/koji/taskinfo?taskID=73161223
https://koji.fedoraproject.org/koji/taskinfo?taskID=72957127
https://koji.fedoraproject.org/koji/taskinfo?taskID=72968988
https://koji.fedoraproject.org/koji/taskinfo?taskID=73120117
https://koji.fedoraproject.org/koji/taskinfo?taskID=73120137
--
Michal Schorm
Software Engineer
Core Services - Databases Team
Red Hat
--
4:15 a.m.
On Tue, Aug 3, 2021 at 7:09 AM Michal Schorm <mschorm(a)redhat.com> wrote:
Hello,
recently, package 'community-mysql' started to FTBFS, because of lack
of resources on armv7hl builders in KOJI.
Packages like 'mariadb' and 'community-mysql' are quite resource
hungry, both for disk and memory.
Occasionally it happens that out of random reasons, some builder here
and there don't have enough resources. Usually because that particular
builder was under heavy load.
However this time, the armv7hl builds lack the resources in 100%
builds for days in various times on various hosts.
(1) Did the KOJI armv7hl builder's configuration change recently ?
(2) Should I file an issue somewhere?
Generic https://pagure.io/releng/new_issue ?
First spotted: F35 mass rebuild.
Reproducible: 100%, on all F33..Rawhide
I suspect it's not the builder that's running out of resources but a
single process exceeding the per process memory limit for 32 bit
applications.
4:31 a.m.
I suspect it's not the builder that's running out of
resources but a
single process exceeding the per process memory limit for 32 bit
applications.
And is there any suggested way how to handle this situation?
Some spec build macro or something?
Thank you
On Tue, Aug 3, 2021 at 11:16 AM Peter Robinson <pbrobinson(a)gmail.com> wrote:
> On Tue, Aug 3, 2021 at 7:09 AM Michal Schorm <mschorm(a)redhat.com> wrote:
> >
> > Hello,
> >
> > recently, package 'community-mysql' started to FTBFS, because of lack
> > of resources on armv7hl builders in KOJI.
> >
> > Packages like 'mariadb' and 'community-mysql' are quite
resource
> > hungry, both for disk and memory.
> >
> > Occasionally it happens that out of random reasons, some builder here
> > and there don't have enough resources. Usually because that particular
> > builder was under heavy load.
> >
> > However this time, the armv7hl builds lack the resources in 100%
> > builds for days in various times on various hosts.
> >
> > (1) Did the KOJI armv7hl builder's configuration change recently ?
> >
> > (2) Should I file an issue somewhere?
> > Generic https://pagure.io/releng/new_issue ?
> >
> > First spotted: F35 mass rebuild.
> > Reproducible: 100%, on all F33..Rawhide
>
I suspect it's not the builder that's running out of
resources but a
single process exceeding the per process memory limit for 32 bit
applications.
> _______________________________________________
> devel mailing list -- devel(a)lists.fedoraproject.org
> To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
> Do not reply to spam on the list, report it:
> https://pagure.io/fedora-infrastructure
>
--
S pozdravom/ Best regards
Lukáš Javorský
Associate Software Engineer, Core service - Databases
Red Hat <https://www.redhat.com>
Purkyňova 115 (TPB-C)
612 00 Brno - Královo Pole
ljavorsk(a)redhat.com
<https://www.redhat.com>
4:43 a.m.
On 03/08/2021 11:31, Lukas Javorsky wrote:
And is there any suggested way how to handle this situation?
Special hacks for ARMv7:
1. Build in one thread.
2. Reduce debug info: -g -> -g1 or even -g0.
Some spec build macro or something?
%ifarch %{arm}
%global optflags %(echo %{optflags} | sed 's/-g /-g0 /')
%endif
--
Sincerely,
Vitaly Zaitsev (vitaly(a)easycoding.org)
8:24 a.m.
Thank you for the suggestions.
I'll test reducing the debuginfo.
However I still believe something else is in play, as such errors are
normally very rare.
The package has already been built without issues; it was the F35 mass
rebuild which only bumped release, after which I noticed the FTBFS.
I also maintain a mysql module based on the same code. That's a lot of
builds already, so I would likely spot build issues, if they would
occur often.
Is there a way to check how much resources the KOJI build actually consumed ?
Michal
--
Michal Schorm
Software Engineer
Core Services - Databases Team
Red Hat
--
On Tue, Aug 3, 2021 at 11:44 AM Vitaly Zaitsev via devel
<devel(a)lists.fedoraproject.org> wrote:
On 03/08/2021 11:31, Lukas Javorsky wrote:
> And is there any suggested way how to handle this situation?
Special hacks for ARMv7:
1. Build in one thread.
2. Reduce debug info: -g -> -g1 or even -g0.
> Some spec build macro or something?
%ifarch %{arm}
%global optflags %(echo %{optflags} | sed 's/-g /-g0 /')
%endif
--
Sincerely,
Vitaly Zaitsev (vitaly(a)easycoding.org)
_______________________________________________
devel mailing list -- devel(a)lists.fedoraproject.org
To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
10:50 a.m.
On Tue, Aug 03, 2021 at 03:24:18PM +0200, Michal Schorm wrote:
Thank you for the suggestions.
I'll test reducing the debuginfo.
However I still believe something else is in play, as such errors are
normally very rare.
The package has already been built without issues; it was the F35 mass
rebuild which only bumped release, after which I noticed the FTBFS.
I also maintain a mysql module based on the same code. That's a lot of
builds already, so I would likely spot build issues, if they would
occur often.
Is there a way to check how much resources the KOJI build actually consumed ?
The only recent change in builders was moving them to f34.
That was in early April. Since then they have only had normal f34
updates.
You can see the hardware stats of the builder in the hw_info.log file
thats next to build.log/root.log in koji. So for example, the
community-mysql mass rebuild failure has:
https://kojipkgs.fedoraproject.org//work/tasks/6534/72336534/hw_info.log
sigh.
Yes, stupidly they updated to the normal kernel before the last reboot.
So, they only see 3GB of their 40GB memory. :(
I'll fix and reboot them today. I have no idea why this wasn't seen
before now. I guess it shows that they are not really memory bound
normally. :(
kevin
11:31 a.m.
On Tue, Aug 03, 2021 at 08:50:24AM -0700, Kevin Fenzi wrote:
Yes, stupidly they updated to the normal kernel before the last reboot.
So, they only see 3GB of their 40GB memory. :(
I'll fix and reboot them today. I have no idea why this wasn't seen
before now. I guess it shows that they are not really memory bound
normally. :(
Just to note that I fixed and rebooted all the armv7 builders yesterday.
Sadly, this seems to have brought back
https://bugzilla.redhat.com/show_bug.cgi?id=1920183
for python builds again.
Hopefully it fixes the community-mysql issue?
kevin
1:49 p.m.
On Wed, Aug 4, 2021 at 6:31 PM Kevin Fenzi <kevin(a)scrye.com> wrote:
Hopefully it fixes the community-mysql issue?
Yes, I got a successful build on the first try without touching the
Rawhide code.
https://koji.fedoraproject.org/koji/taskinfo?taskID=73279974
Thank you very much for your help, Kevin !
Michal
--
Michal Schorm
Software Engineer
Core Services - Databases Team
Red Hat
--
On Wed, Aug 4, 2021 at 6:31 PM Kevin Fenzi <kevin(a)scrye.com> wrote:
>
> On Tue, Aug 03, 2021 at 08:50:24AM -0700, Kevin Fenzi wrote:
> >
> > Yes, stupidly they updated to the normal kernel before the last reboot.
> > So, they only see 3GB of their 40GB memory. :(
> >
> > I'll fix and reboot them today. I have no idea why this wasn't seen
> > before now. I guess it shows that they are not really memory bound
> > normally. :(
>
> Just to note that I fixed and rebooted all the armv7 builders yesterday.
>
> Sadly, this seems to have brought back
> https://bugzilla.redhat.com/show_bug.cgi?id=1920183
> for python builds again.
>
Hopefully it fixes the community-mysql issue?
>
> kevin
> _______________________________________________
> devel mailing list -- devel(a)lists.fedoraproject.org
> To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
> Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
1015
days inactive
1016
days old
7 comments
5 participants
participants (5)
-
Kevin Fenzi -
Lukas Javorsky -
Michal Schorm -
Peter Robinson -
Vitaly Zaitsev