4:14 p.m.
On Thu, 2019-12-19 at 21:56 +0000, devel-
request(a)lists.fedoraproject.org wrote:
Neal,
> I'm generally happy with this idea. I'm one of
the maintainers of
> rpm-config-SUSE (the equivalent of redhat-rpm-config for SUSE
> distributions) and I somewhat saw the development of this feature
> across rpm, rpm-config-SUSE, and the implementation in openSUSE
> Tumbleweed.
Understood. FWIW, I work closely with Martin L, Martin J, Jan and
Richi @ SuSE. Martin L and I in particular coordinate on mass build
testing & related failure analysis and bugfixing.
> However, the implementation of LTO in openSUSE caused
major problems
> for "weaker" architectures like ARM and RISC-V. In Fedora, ARM is
> co-primary with the rest of the architectures (ARM is allowed to be
> broken in openSUSE from time to time). The major problems we
> encountered was increased miscompilation errors and timeouts due to
> builds taking even longer with LTO straining ARM build environments.
Yes. The 32bit architectures in particular are expected to be slightly
problematical due to the limited address space. Even with Jan's work
in this area, I expect things like firefox to simply be too big to
compile/link using LTO on the 32bit platforms.
When I discussed this with Martin L and one of the Ubuntu engineers
(Matthias) back in Sept, the general plan we agreed on was to first get
the Fedora test builds in reasonable shape on x86_64 that we could use
as a baseline (and we're just about there). Then do an i686 build for
comparison purposes.
For packages we find problematical on the 32bit platforms, we'll be
able to trivially opt-out of LTO for that package on those 32bit
platforms. It's a one-liner in the .spec file.
> And with RISC-V, is LTO even implemented well in GCC? In
OpenMandriva
> (where I help maintain rpm-openmandriva-setup, its equivalent of
> redhat-rpm-config), we're still using GCC for RISC-V since Clang is
> still horribly broken there, and we had to disable LTO to get a usable
> port going. Fedora RISC-V is going to be in for a world of pain if
> we're enabling it without any consideration for them.
GCC's implementation of LTO is target independent.
RISC-V really isn't on my radar at this point. Though if we have
issues specific to the code generator on RISC-V, I'm sure Jim Wilson
(SiFive) and I can work it out. We've been working together on GCC for
nearly 30 years at this point. I would _not_ expect RISC-V's code
generator to be as solid as other platforms simply because it hasn't
been used much yet.
> So what's the state of things for non-x86? Generally
speaking, my
> experience indicates everything is broken...
My tester is x86_64 based. There's no inherent reason it couldn't work
on another platform other than needing lots of machines to turn around
builds quickly enough to matter. My hope has always been to spin up
i686 and ppc64le as I can get access to reasonable hardware for those
targets.
I've spoken with ARM's & IBM's GCC engineers about this upcoming
change, they're aware and supportive of the plan. We typically lean
heavily on them to take care of any GCC issues that are specific to
their targets.
jeff
4:24 p.m.
New subject: devel Digest, Vol 190, Issue 186
On Thu, Dec 19, 2019 at 4:14 PM Jeff Law <law(a)redhat.com> wrote:
On Thu, 2019-12-19 at 21:56 +0000, devel-
request(a)lists.fedoraproject.org wrote:
>
Neal,
>
> I'm generally happy with this idea. I'm one of the maintainers of
> rpm-config-SUSE (the equivalent of redhat-rpm-config for SUSE
> distributions) and I somewhat saw the development of this feature
> across rpm, rpm-config-SUSE, and the implementation in openSUSE
> Tumbleweed.
Understood. FWIW, I work closely with Martin L, Martin J, Jan and
Richi @ SuSE. Martin L and I in particular coordinate on mass build
testing & related failure analysis and bugfixing.
Martin Liška is the one I've primarily interfaced with throughout the
implementation.
I don't know if you know about this, but there's a tracker bug for
LTO-related failures: https://bugzilla.opensuse.org/1133084
We should probably make sure this is cross-referenced as LTO is
implemented in Fedora.
>
> However, the implementation of LTO in openSUSE caused major problems
> for "weaker" architectures like ARM and RISC-V. In Fedora, ARM is
> co-primary with the rest of the architectures (ARM is allowed to be
> broken in openSUSE from time to time). The major problems we
> encountered was increased miscompilation errors and timeouts due to
> builds taking even longer with LTO straining ARM build environments.
Yes. The 32bit architectures in particular are expected to be slightly
problematical due to the limited address space. Even with Jan's work
in this area, I expect things like firefox to simply be too big to
compile/link using LTO on the 32bit platforms.
When I discussed this with Martin L and one of the Ubuntu engineers
(Matthias) back in Sept, the general plan we agreed on was to first get
the Fedora test builds in reasonable shape on x86_64 that we could use
as a baseline (and we're just about there). Then do an i686 build for
comparison purposes.
For packages we find problematical on the 32bit platforms, we'll be
able to trivially opt-out of LTO for that package on those 32bit
platforms. It's a one-liner in the .spec file.
Are we going to use the same %_lto_cflags mechanism that was
implemented in openSUSE, or are we going to do something different?
>
> And with RISC-V, is LTO even implemented well in GCC? In OpenMandriva
> (where I help maintain rpm-openmandriva-setup, its equivalent of
> redhat-rpm-config), we're still using GCC for RISC-V since Clang is
> still horribly broken there, and we had to disable LTO to get a usable
> port going. Fedora RISC-V is going to be in for a world of pain if
> we're enabling it without any consideration for them.
GCC's implementation of LTO is target independent.
RISC-V really isn't on my radar at this point. Though if we have
issues specific to the code generator on RISC-V, I'm sure Jim Wilson
(SiFive) and I can work it out. We've been working together on GCC for
nearly 30 years at this point. I would _not_ expect RISC-V's code
generator to be as solid as other platforms simply because it hasn't
been used much yet.
>
> So what's the state of things for non-x86? Generally speaking, my
> experience indicates everything is broken...
My tester is x86_64 based. There's no inherent reason it couldn't work
on another platform other than needing lots of machines to turn around
builds quickly enough to matter. My hope has always been to spin up
i686 and ppc64le as I can get access to reasonable hardware for those
targets.
I've spoken with ARM's & IBM's GCC engineers about this upcoming
change, they're aware and supportive of the plan. We typically lean
heavily on them to take care of any GCC issues that are specific to
their targets.
I'd like to see at *least* AArch64 spun to see how well this goes. I'm
reasonably confident that POWER8+ would be fine, since we don't have
ppc64be anymore (and I *know* that one was broken, since I reported it
years ago: https://bugzilla.redhat.com/1515934).
--
真実はいつも一つ！/ Always, there's only one truth!
4:47 p.m.
New subject: devel Digest, Vol 190, Issue 186
On Thu, 2019-12-19 at 16:24 -0600, Neal Gompa wrote:
On Thu, Dec 19, 2019 at 4:14 PM Jeff Law <law(a)redhat.com>
wrote:
> On Thu, 2019-12-19 at 21:56 +0000, devel-
> request(a)lists.fedoraproject.org wrote:
>
> Neal,
>
>
> > I'm generally happy with this idea. I'm one of the maintainers of
> > rpm-config-SUSE (the equivalent of redhat-rpm-config for SUSE
> > distributions) and I somewhat saw the development of this feature
> > across rpm, rpm-config-SUSE, and the implementation in openSUSE
> > Tumbleweed.
> Understood. FWIW, I work closely with Martin L, Martin J, Jan and
> Richi @ SuSE. Martin L and I in particular coordinate on mass build
> testing & related failure analysis and bugfixing.
>
Martin Liška is the one I've primarily interfaced with throughout the
implementation.
I don't know if you know about this, but there's a tracker bug for
LTO-related failures: https://bugzilla.opensuse.org/1133084
I am aware of it and
have referenced it several times in my own work.
We should probably make sure this is cross-referenced as LTO is
implemented in Fedora.
Sure. That's easy enough to do. Some of the
information is dated, but
there still useful nuggets in there.
They punted lots of stuff though. In particular they punted the
configure problem and middle-end issued diagnostics, which was terribly
unfortunate.
> > However, the implementation of LTO in openSUSE caused major problems
> > for "weaker" architectures like ARM and RISC-V. In Fedora, ARM is
> > co-primary with the rest of the architectures (ARM is allowed to be
> > broken in openSUSE from time to time). The major problems we
> > encountered was increased miscompilation errors and timeouts due to
> > builds taking even longer with LTO straining ARM build environments.
> Yes. The 32bit architectures in particular are expected to be slightly
> problematical due to the limited address space. Even with Jan's work
> in this area, I expect things like firefox to simply be too big to
> compile/link using LTO on the 32bit platforms.
>
> When I discussed this with Martin L and one of the Ubuntu engineers
> (Matthias) back in Sept, the general plan we agreed on was to first get
> the Fedora test builds in reasonable shape on x86_64 that we could use
> as a baseline (and we're just about there). Then do an i686 build for
> comparison purposes.
>
> For packages we find problematical on the 32bit platforms, we'll be
> able to trivially opt-out of LTO for that package on those 32bit
> platforms. It's a one-liner in the .spec file.
>
>
Are we going to use the same %_lto_cflags mechanism that was
implemented in openSUSE, or are we going to do something different?
Ideally the
same. I don't own redhat-rpm-config, but certainly my
preference is use the same mechanisms.
I'd like to see at *least* AArch64 spun to see how well this goes. I'm
reasonably confident that POWER8+ would be fine, since we don't have
ppc64be anymore (and I *know* that one was broken, since I reported it
years ago: https://bugzilla.redhat.com/1515934).
Obviously aarch64 will be built
and any issues resolved as is the case
with other architectures. We do this every spring cycle with the
introduction of a major GCC release and we work closely with the GCC
engineers at ARM and IBM along the way.
Building 9000 packages 3X each (gcc-10+LTO, gcc-10, gcc-9 baseline)
takes significant hardware. If there's hardware I can use (and you
should be thinking on the order of dozens of dedicated machines), then
I'm happy to use them. I mentioned ppc64le simply because I can get
access to a goodly number of them.
Thanks for that reference. Big endian ppc. Ugh. But in the end it's
just code.
Jeff
11:59 a.m.
New subject: devel Digest, Vol 190, Issue 186
Hi,
When replying to digests, I'd appreciate if you could please make an
effort to have the posts thread properly for the rest of us. Fedora
mailing list guidance on this:
https://fedoraproject.org/wiki/Mailing_list_guidelines#Replying_to_Digests
Thanks,
--Robbie
12:04 p.m.
New subject: digested list etiquette [ was: devel Digest, Vol 190, Issue 186
]
On Fri, 2019-12-20 at 12:59 -0500, Robbie Harwood wrote:
Hi,
When replying to digests, I'd appreciate if you could please make an
effort to have the posts thread properly for the rest of us. Fedora
mailing list guidance on this:
https://fedoraproject.org/wiki/Mailing_list_guidelines#Replying_to_Digests
Will do.
Sorry for making things harder on everyone in this regard.
jeff
9:52 a.m.
New subject: devel Digest, Vol 190, Issue 186
On Thu, Dec 19, 2019 at 5:48 PM Jeff Law <law(a)redhat.com> wrote:
On Thu, 2019-12-19 at 16:24 -0600, Neal Gompa wrote:
> On Thu, Dec 19, 2019 at 4:14 PM Jeff Law <law(a)redhat.com> wrote:
> > On Thu, 2019-12-19 at 21:56 +0000, devel-
> > request(a)lists.fedoraproject.org wrote:
> >
> > Neal,
> >
> >
> > > I'm generally happy with this idea. I'm one of the maintainers of
> > > rpm-config-SUSE (the equivalent of redhat-rpm-config for SUSE
> > > distributions) and I somewhat saw the development of this feature
> > > across rpm, rpm-config-SUSE, and the implementation in openSUSE
> > > Tumbleweed.
> > Understood. FWIW, I work closely with Martin L, Martin J, Jan and
> > Richi @ SuSE. Martin L and I in particular coordinate on mass build
> > testing & related failure analysis and bugfixing.
> >
>
> Martin Liška is the one I've primarily interfaced with throughout the
> implementation.
>
> I don't know if you know about this, but there's a tracker bug for
> LTO-related failures: https://bugzilla.opensuse.org/1133084
I am aware of it and have referenced it several times in my own work.
>
> We should probably make sure this is cross-referenced as LTO is
> implemented in Fedora.
Sure. That's easy enough to do. Some of the information is dated, but
there still useful nuggets in there.
They punted lots of stuff though. In particular they punted the
configure problem and middle-end issued diagnostics, which was terribly
unfortunate.
>
> > > However, the implementation of LTO in openSUSE caused major problems
> > > for "weaker" architectures like ARM and RISC-V. In Fedora, ARM
is
> > > co-primary with the rest of the architectures (ARM is allowed to be
> > > broken in openSUSE from time to time). The major problems we
> > > encountered was increased miscompilation errors and timeouts due to
> > > builds taking even longer with LTO straining ARM build environments.
> > Yes. The 32bit architectures in particular are expected to be slightly
> > problematical due to the limited address space. Even with Jan's work
> > in this area, I expect things like firefox to simply be too big to
> > compile/link using LTO on the 32bit platforms.
> >
> > When I discussed this with Martin L and one of the Ubuntu engineers
> > (Matthias) back in Sept, the general plan we agreed on was to first get
> > the Fedora test builds in reasonable shape on x86_64 that we could use
> > as a baseline (and we're just about there). Then do an i686 build for
> > comparison purposes.
> >
> > For packages we find problematical on the 32bit platforms, we'll be
> > able to trivially opt-out of LTO for that package on those 32bit
> > platforms. It's a one-liner in the .spec file.
> >
> >
>
> Are we going to use the same %_lto_cflags mechanism that was
> implemented in openSUSE, or are we going to do something different?
Ideally the same. I don't own redhat-rpm-config, but certainly my
preference is use the same mechanisms.
>
>
> I'd like to see at *least* AArch64 spun to see how well this goes. I'm
> reasonably confident that POWER8+ would be fine, since we don't have
> ppc64be anymore (and I *know* that one was broken, since I reported it
> years ago: https://bugzilla.redhat.com/1515934).
Obviously aarch64 will be built and any issues resolved as is the case
with other architectures. We do this every spring cycle with the
introduction of a major GCC release and we work closely with the GCC
engineers at ARM and IBM along the way.
Building 9000 packages 3X each (gcc-10+LTO, gcc-10, gcc-9 baseline)
takes significant hardware. If there's hardware I can use (and you
should be thinking on the order of dozens of dedicated machines), then
I'm happy to use them. I mentioned ppc64le simply because I can get
access to a goodly number of them.
Is there a reason you couldn't use aarch64 AWS instances?
josh
10:33 a.m.
New subject: devel Digest, Vol 190, Issue 186
On Sat, 2019-12-21 at 10:52 -0500, Josh Boyer wrote:
On Thu, Dec 19, 2019 at 5:48 PM Jeff Law <law(a)redhat.com>
wrote:
> On Thu, 2019-12-19 at 16:24 -0600, Neal Gompa wrote:
> > On Thu, Dec 19, 2019 at 4:14 PM Jeff Law <law(a)redhat.com> wrote:
> > > On Thu, 2019-12-19 at 21:56 +0000, devel-
> > > request(a)lists.fedoraproject.org wrote:
> > > > > Neal,
> > > > > > > > I'm generally happy with this idea.
I'm one of the maintainers of
> > > > rpm-config-SUSE (the equivalent of redhat-rpm-config for SUSE
> > > > distributions) and I somewhat saw the development of this feature
> > > > across rpm, rpm-config-SUSE, and the implementation in openSUSE
> > > > Tumbleweed.
> > > Understood. FWIW, I work closely with Martin L, Martin J, Jan and
> > > Richi @ SuSE. Martin L and I in particular coordinate on mass build
> > > testing & related failure analysis and bugfixing.
> > > >
> Martin Liška is the one I've primarily interfaced with throughout the
> > implementation.
> > > I don't know if you know about this, but
there's a tracker bug for
> > LTO-related failures: https://bugzilla.opensuse.org/1133084
> I am aware of it and have referenced it several times in my own work.
> > We should probably make sure this is
cross-referenced as LTO is
> > implemented in Fedora.
> Sure. That's easy enough to do. Some of the information is dated, but
> there still useful nuggets in there.
> They punted lots of stuff though. In particular they
punted the
> configure problem and middle-end issued diagnostics, which was terribly
> unfortunate.
> > > > However, the implementation of LTO in
openSUSE caused major problems
> > > > for "weaker" architectures like ARM and RISC-V. In Fedora,
ARM is
> > > > co-primary with the rest of the architectures (ARM is allowed to be
> > > > broken in openSUSE from time to time). The major problems we
> > > > encountered was increased miscompilation errors and timeouts due to
> > > > builds taking even longer with LTO straining ARM build environments.
> > > Yes. The 32bit architectures in particular are expected to be slightly
> > > problematical due to the limited address space. Even with Jan's work
> > > in this area, I expect things like firefox to simply be too big to
> > > compile/link using LTO on the 32bit platforms.
> > > > > When I discussed
this with Martin L and one of the Ubuntu engineers
> > > (Matthias) back in Sept, the general plan we agreed on was to first get
> > > the Fedora test builds in reasonable shape on x86_64 that we could use
> > > as a baseline (and we're just about there). Then do an i686 build
for
> > > comparison purposes.
> > > > > For packages we
find problematical on the 32bit platforms, we'll be
> > > able to trivially opt-out of LTO for that package on those 32bit
> > > platforms. It's a one-liner in the .spec file.
> > > > > > > Are we going to use the
same %_lto_cflags mechanism that was
> > implemented in openSUSE, or are we going to do something different?
> Ideally the same. I don't own redhat-rpm-config, but certainly my
> preference is use the same mechanisms.
> > > I'd like to see at
*least* AArch64 spun to see how well this goes. I'm
> > reasonably confident that POWER8+ would be fine, since we don't have
> > ppc64be anymore (and I *know* that one was broken, since I reported it
> > years ago: https://bugzilla.redhat.com/1515934).
> Obviously aarch64 will be built and any issues resolved as is the case
> with other architectures. We do this every spring cycle with the
> introduction of a major GCC release and we work closely with the GCC
> engineers at ARM and IBM along the way.
> Building 9000 packages 3X each (gcc-10+LTO, gcc-10,
gcc-9 baseline)
> takes significant hardware. If there's hardware I can use (and you
> should be thinking on the order of dozens of dedicated machines), then
> I'm happy to use them. I mentioned ppc64le simply because I can get
> access to a goodly number of them.
Is there a reason you couldn't use aarch64 AWS instances?
I've actually
applied for aws credits which could be used for this.
THe instances in the free tier are too small to really be useful.
jeff
1595
days inactive
1597
days old
6 comments
4 participants
participants (4)
-
Jeff Law -
Josh Boyer -
Neal Gompa -
Robbie Harwood