2:09 a.m.
Hello,
mame package uses mame -validate as %check step. It has recently started
to cause problems on rawhide/aarch64:
- I had to cancel 0.258 build after 16 hours [1]
- 0.259 build is stuck since ca. 1600 UTC yesterday [2]
Other branches built fine for aarch64, and so did other arches for
rawhide. Additionally, there are no rawhide koschei builds since August
making it harder to find the potential culprit.
How can I investigate further? I might need to disable %check if the
reason cannot be found.
Best regards,
Julian
[1] https://koji.fedoraproject.org/koji/taskinfo?taskID=106453404
[2] https://koji.fedoraproject.org/koji/taskinfo?taskID=106887762
2:32 a.m.
Am 30.09.23 um 09:09 schrieb Julian Sikorski:
Hello,
mame package uses mame -validate as %check step. It has recently started
to cause problems on rawhide/aarch64:
- I had to cancel 0.258 build after 16 hours [1]
- 0.259 build is stuck since ca. 1600 UTC yesterday [2]
Other branches built fine for aarch64, and so did other arches for
rawhide. Additionally, there are no rawhide koschei builds since August
making it harder to find the potential culprit.
How can I investigate further? I might need to disable %check if the
reason cannot be found.
Best regards,
Julian
[1] https://koji.fedoraproject.org/koji/taskinfo?taskID=106453404
[2] https://koji.fedoraproject.org/koji/taskinfo?taskID=106887762
The build is still stuck. Do we have aarch64 machines accessible to
maintainers which could be used to see what the problem is? The last
lines in the build log are (and have been for a while):
Executing(%check): /bin/sh -e /var/tmp/rpm-tmp.cLY2Y2
+ umask 022
+ cd /builddir/build/BUILD
+ cd mame-mame0259
+ ./mame -validate
Best regards,
Julian
2:37 a.m.
On Sun, 1 Oct 2023 09:32:10 +0200
Julian Sikorski <belegdol(a)gmail.com> wrote:
Am 30.09.23 um 09:09 schrieb Julian Sikorski:
> Hello,
>
> mame package uses mame -validate as %check step. It has recently started
> to cause problems on rawhide/aarch64:
> - I had to cancel 0.258 build after 16 hours [1]
> - 0.259 build is stuck since ca. 1600 UTC yesterday [2]
> Other branches built fine for aarch64, and so did other arches for
> rawhide. Additionally, there are no rawhide koschei builds since August
> making it harder to find the potential culprit.
> How can I investigate further? I might need to disable %check if the
> reason cannot be found.
>
> Best regards,
> Julian
>
> [1] https://koji.fedoraproject.org/koji/taskinfo?taskID=106453404
> [2] https://koji.fedoraproject.org/koji/taskinfo?taskID=106887762
The build is still stuck. Do we have aarch64 machines accessible to
maintainers which could be used to see what the problem is? The last
we do have, please see
https://fedoraproject.org/wiki/Test_Machine_Resources_For_Package_Maintai...
Dan
> lines in the build log are (and have been for a while):
>
> Executing(%check): /bin/sh -e /var/tmp/rpm-tmp.cLY2Y2
> + umask 022
> + cd /builddir/build/BUILD
> + cd mame-mame0259
> + ./mame -validate
>
> Best regards,
> Julian
> _______________________________________________
> devel mailing list -- devel(a)lists.fedoraproject.org
> To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
> Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
3:17 a.m.
Am 01.10.23 um 09:37 schrieb Dan Horák:
On Sun, 1 Oct 2023 09:32:10 +0200
Julian Sikorski <belegdol(a)gmail.com> wrote:
> Am 30.09.23 um 09:09 schrieb Julian Sikorski:
>> Hello,
>>
>> mame package uses mame -validate as %check step. It has recently started
>> to cause problems on rawhide/aarch64:
>> - I had to cancel 0.258 build after 16 hours [1]
>> - 0.259 build is stuck since ca. 1600 UTC yesterday [2]
>> Other branches built fine for aarch64, and so did other arches for
>> rawhide. Additionally, there are no rawhide koschei builds since August
>> making it harder to find the potential culprit.
>> How can I investigate further? I might need to disable %check if the
>> reason cannot be found.
>>
>> Best regards,
>> Julian
>>
>> [1] https://koji.fedoraproject.org/koji/taskinfo?taskID=106453404
>> [2] https://koji.fedoraproject.org/koji/taskinfo?taskID=106887762
>
> The build is still stuck. Do we have aarch64 machines accessible to
> maintainers which could be used to see what the problem is? The last
we do have, please see
https://fedoraproject.org/wiki/Test_Machine_Resources_For_Package_Maintai...
Dan
Thanks! Unfortunately it appears that aarch64 is only available as f38
and f37 which both finish the validation successfully. Rawhide is only
available as x86_64 which does not exhibit the issue either.
I will see if doing a rawhide mock build on f38 host is sufficient. This
is what koji seems to be doing anyway.
Best regards,
Julian
7:21 a.m.
Am 01.10.23 um 10:17 schrieb Julian Sikorski:
Am 01.10.23 um 09:37 schrieb Dan Horák:
> On Sun, 1 Oct 2023 09:32:10 +0200
> Julian Sikorski <belegdol(a)gmail.com> wrote:
>
>> Am 30.09.23 um 09:09 schrieb Julian Sikorski:
>>> Hello,
>>>
>>> mame package uses mame -validate as %check step. It has recently
>>> started
>>> to cause problems on rawhide/aarch64:
>>> - I had to cancel 0.258 build after 16 hours [1]
>>> - 0.259 build is stuck since ca. 1600 UTC yesterday [2]
>>> Other branches built fine for aarch64, and so did other arches for
>>> rawhide. Additionally, there are no rawhide koschei builds since August
>>> making it harder to find the potential culprit.
>>> How can I investigate further? I might need to disable %check if the
>>> reason cannot be found.
>>>
>>> Best regards,
>>> Julian
>>>
>>> [1] https://koji.fedoraproject.org/koji/taskinfo?taskID=106453404
>>> [2] https://koji.fedoraproject.org/koji/taskinfo?taskID=106887762
>>
>> The build is still stuck. Do we have aarch64 machines accessible to
>> maintainers which could be used to see what the problem is? The last
>
> we do have, please see
> https://fedoraproject.org/wiki/Test_Machine_Resources_For_Package_Maintai...
>
>
> Dan
Thanks! Unfortunately it appears that aarch64 is only available as f38
and f37 which both finish the validation successfully. Rawhide is only
available as x86_64 which does not exhibit the issue either.
I will see if doing a rawhide mock build on f38 host is sufficient. This
is what koji seems to be doing anyway.
Best regards,
Julian
I managed to reproduce this on the aarch64 machine in mock. Validation
seems to get stuck right away at:
(gdb) bt
#0 0x0000aaaab5bddb08 in ___ZN4bgfx12VertexLayoutC1Ev_bti_veneer ()
#1 0x0000fffff5870b2c in __libc_start_main_impl () from /lib64/libc.so.6
#2 0x0000aaaaaef01570 in _start ()
One question: the binary in /builddir/build/BUILD still has symbols in,
correct?
Best regards,
Julian
11:08 a.m.
>>>> mame package uses mame -validate as %check step. It
has recently started
>>>> to cause problems on rawhide/aarch64:
>>>> - 0.259 build is stuck since ca. 1600 UTC yesterday
[2]
I managed to reproduce this on the aarch64 machine in mock.
Validation seems to get stuck right away at:
(gdb) bt
#0 0x0000aaaab5bddb08 in ___ZN4bgfx12VertexLayoutC1Ev_bti_veneer ()
#1 0x0000fffff5870b2c in __libc_start_main_impl () from /lib64/libc.so.6
#2 0x0000aaaaaef01570 in _start ()
One question: the binary in /builddir/build/BUILD still has symbols in, correct?
The backtrace above does not contain a filename or line number for frame #0,
so it looks like the app (or a library that it uses) was compiled without "-g",
or the debug information has been stripped or separated after building.
Of course the app does contain some global symbol information
that is required for run-time dynamic linking ("readelf --symbols
--use-dynamic ./my_app"). It is remarkable that the symbol
for frame #0 '___ZN4bgfx12VertexLayoutC1Ev_bti_veneer'
begins with three underscores instead of only 1.
11:30 a.m.
Am 01.10.23 um 18:08 schrieb John Reiser:
>>>>> mame package uses mame -validate as %check step.
It has recently
>>>>> started
>>>>> to cause problems on rawhide/aarch64:
>>>>> - 0.259 build is stuck since ca. 1600 UTC yesterday [2]
> I managed to reproduce this on the aarch64 machine in mock. Validation
> seems to get stuck right away at:
>
> (gdb) bt
> #0 0x0000aaaab5bddb08 in ___ZN4bgfx12VertexLayoutC1Ev_bti_veneer ()
> #1 0x0000fffff5870b2c in __libc_start_main_impl () from /lib64/libc.so.6
> #2 0x0000aaaaaef01570 in _start ()
>
> One question: the binary in /builddir/build/BUILD still has symbols
> in, correct?
The backtrace above does not contain a filename or line number for frame
#0,
so it looks like the app (or a library that it uses) was compiled
without "-g",
or the debug information has been stripped or separated after building.
Of course the app does contain some global symbol information
that is required for run-time dynamic linking ("readelf --symbols
--use-dynamic ./my_app"). It is remarkable that the symbol
for frame #0 '___ZN4bgfx12VertexLayoutC1Ev_bti_veneer'
begins with three underscores instead of only 1.
I have to build with -g1 as building with normal -g2 causes the
compilation to exhaust all available memory.
Best regards,
Julian
11:14 a.m.
On Sun, 1 Oct 2023 14:21:23 +0200
Julian Sikorski <belegdol(a)gmail.com> wrote:
Am 01.10.23 um 10:17 schrieb Julian Sikorski:
> Am 01.10.23 um 09:37 schrieb Dan Horák:
>> On Sun, 1 Oct 2023 09:32:10 +0200
>> Julian Sikorski <belegdol(a)gmail.com> wrote:
>>
>>> Am 30.09.23 um 09:09 schrieb Julian Sikorski:
>>>> Hello,
>>>>
>>>> mame package uses mame -validate as %check step. It has recently
>>>> started
>>>> to cause problems on rawhide/aarch64:
>>>> - I had to cancel 0.258 build after 16 hours [1]
>>>> - 0.259 build is stuck since ca. 1600 UTC yesterday [2]
>>>> Other branches built fine for aarch64, and so did other arches for
>>>> rawhide. Additionally, there are no rawhide koschei builds since August
>>>> making it harder to find the potential culprit.
>>>> How can I investigate further? I might need to disable %check if the
>>>> reason cannot be found.
>>>>
>>>> Best regards,
>>>> Julian
>>>>
>>>> [1] https://koji.fedoraproject.org/koji/taskinfo?taskID=106453404
>>>> [2] https://koji.fedoraproject.org/koji/taskinfo?taskID=106887762
>>>
>>> The build is still stuck. Do we have aarch64 machines accessible to
>>> maintainers which could be used to see what the problem is? The last
>>
>> we do have, please see
>> https://fedoraproject.org/wiki/Test_Machine_Resources_For_Package_Maintai...
>>
>>
>> Dan
>
> Thanks! Unfortunately it appears that aarch64 is only available as f38
> and f37 which both finish the validation successfully. Rawhide is only
> available as x86_64 which does not exhibit the issue either.
> I will see if doing a rawhide mock build on f38 host is sufficient. This
> is what koji seems to be doing anyway.
>
> Best regards,
> Julian
I managed to reproduce this on the aarch64 machine in mock. Validation
seems to get stuck right away at:
(gdb) bt
#0 0x0000aaaab5bddb08 in ___ZN4bgfx12VertexLayoutC1Ev_bti_veneer ()
hmm, couldn't it be related to the PAC & BTI feature [1]? I believe
someone mentioned some issues with it, probably here on the devel
list and the workaround was to disable the related compiler flags ...
#1 0x0000fffff5870b2c in __libc_start_main_impl () from
/lib64/libc.so.6
#2 0x0000aaaaaef01570 in _start ()
One question: the binary in /builddir/build/BUILD still has symbols in,
correct?
yes, the files under /builddir/build/BUILDROOT/... are split into
stripped binaries and debuginfos
Best regards,
Julian
[1] https://fedoraproject.org/wiki/Changes/Aarch64_PointerAuthentication
Dan
11:32 a.m.
Am 01.10.23 um 18:14 schrieb Dan Horák:
On Sun, 1 Oct 2023 14:21:23 +0200
Julian Sikorski <belegdol(a)gmail.com> wrote:
> Am 01.10.23 um 10:17 schrieb Julian Sikorski:
>> Am 01.10.23 um 09:37 schrieb Dan Horák:
>>> On Sun, 1 Oct 2023 09:32:10 +0200
>>> Julian Sikorski <belegdol(a)gmail.com> wrote:
>>>
>>>> Am 30.09.23 um 09:09 schrieb Julian Sikorski:
>>>>> Hello,
>>>>>
>>>>> mame package uses mame -validate as %check step. It has recently
>>>>> started
>>>>> to cause problems on rawhide/aarch64:
>>>>> - I had to cancel 0.258 build after 16 hours [1]
>>>>> - 0.259 build is stuck since ca. 1600 UTC yesterday [2]
>>>>> Other branches built fine for aarch64, and so did other arches for
>>>>> rawhide. Additionally, there are no rawhide koschei builds since
August
>>>>> making it harder to find the potential culprit.
>>>>> How can I investigate further? I might need to disable %check if the
>>>>> reason cannot be found.
>>>>>
>>>>> Best regards,
>>>>> Julian
>>>>>
>>>>> [1] https://koji.fedoraproject.org/koji/taskinfo?taskID=106453404
>>>>> [2] https://koji.fedoraproject.org/koji/taskinfo?taskID=106887762
>>>>
>>>> The build is still stuck. Do we have aarch64 machines accessible to
>>>> maintainers which could be used to see what the problem is? The last
>>>
>>> we do have, please see
>>>
https://fedoraproject.org/wiki/Test_Machine_Resources_For_Package_Maintai...
>>>
>>>
>>> Dan
>>
>> Thanks! Unfortunately it appears that aarch64 is only available as f38
>> and f37 which both finish the validation successfully. Rawhide is only
>> available as x86_64 which does not exhibit the issue either.
>> I will see if doing a rawhide mock build on f38 host is sufficient. This
>> is what koji seems to be doing anyway.
>>
>> Best regards,
>> Julian
>
> I managed to reproduce this on the aarch64 machine in mock. Validation
> seems to get stuck right away at:
>
> (gdb) bt
> #0 0x0000aaaab5bddb08 in ___ZN4bgfx12VertexLayoutC1Ev_bti_veneer ()
hmm, couldn't it be related to the PAC & BTI feature [1]? I believe
someone mentioned some issues with it, probably here on the devel
list and the workaround was to disable the related compiler flags ...
The feature dates back to F33 whereas the breakage started after F39 was
branched. I discussed this with upstream [1] and the suspicion is that
the linker is misbehaving. I am attempting a build using lld to confirm.
Best regards,
Julian
[1] https://github.com/mamedev/mame/issues/11587
> #1 0x0000fffff5870b2c in __libc_start_main_impl () from /lib64/libc.so.6
> #2 0x0000aaaaaef01570 in _start ()
>
> One question: the binary in /builddir/build/BUILD still has symbols in,
> correct?
yes, the files under /builddir/build/BUILDROOT/... are split into
stripped binaries and debuginfos
> Best regards,
> Julian
[1] https://fedoraproject.org/wiki/Changes/Aarch64_PointerAuthentication
Dan
_______________________________________________
devel mailing list -- devel(a)lists.fedoraproject.org
To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
2:41 p.m.
Am 01.10.23 um 18:32 schrieb Julian Sikorski:
Am 01.10.23 um 18:14 schrieb Dan Horák:
> On Sun, 1 Oct 2023 14:21:23 +0200
> Julian Sikorski <belegdol(a)gmail.com> wrote:
>
>> Am 01.10.23 um 10:17 schrieb Julian Sikorski:
>>> Am 01.10.23 um 09:37 schrieb Dan Horák:
>>>> On Sun, 1 Oct 2023 09:32:10 +0200
>>>> Julian Sikorski <belegdol(a)gmail.com> wrote:
>>>>
>>>>> Am 30.09.23 um 09:09 schrieb Julian Sikorski:
>>>>>> Hello,
>>>>>>
>>>>>> mame package uses mame -validate as %check step. It has recently
>>>>>> started
>>>>>> to cause problems on rawhide/aarch64:
>>>>>> - I had to cancel 0.258 build after 16 hours [1]
>>>>>> - 0.259 build is stuck since ca. 1600 UTC yesterday [2]
>>>>>> Other branches built fine for aarch64, and so did other arches
for
>>>>>> rawhide. Additionally, there are no rawhide koschei builds since
>>>>>> August
>>>>>> making it harder to find the potential culprit.
>>>>>> How can I investigate further? I might need to disable %check if
the
>>>>>> reason cannot be found.
>>>>>>
>>>>>> Best regards,
>>>>>> Julian
>>>>>>
>>>>>> [1]
https://koji.fedoraproject.org/koji/taskinfo?taskID=106453404
>>>>>> [2]
https://koji.fedoraproject.org/koji/taskinfo?taskID=106887762
>>>>>
>>>>> The build is still stuck. Do we have aarch64 machines accessible to
>>>>> maintainers which could be used to see what the problem is? The last
>>>>
>>>> we do have, please see
>>>>
https://fedoraproject.org/wiki/Test_Machine_Resources_For_Package_Maintai...
>>>>
>>>>
>>>> Dan
>>>
>>> Thanks! Unfortunately it appears that aarch64 is only available as f38
>>> and f37 which both finish the validation successfully. Rawhide is only
>>> available as x86_64 which does not exhibit the issue either.
>>> I will see if doing a rawhide mock build on f38 host is sufficient.
>>> This
>>> is what koji seems to be doing anyway.
>>>
>>> Best regards,
>>> Julian
>>
>> I managed to reproduce this on the aarch64 machine in mock. Validation
>> seems to get stuck right away at:
>>
>> (gdb) bt
>> #0 0x0000aaaab5bddb08 in ___ZN4bgfx12VertexLayoutC1Ev_bti_veneer ()
>
> hmm, couldn't it be related to the PAC & BTI feature [1]? I believe
> someone mentioned some issues with it, probably here on the devel
> list and the workaround was to disable the related compiler flags ...
The feature dates back to F33 whereas the breakage started after F39 was
branched. I discussed this with upstream [1] and the suspicion is that
the linker is misbehaving. I am attempting a build using lld to confirm.
Best regards,
Julian
[1] https://github.com/mamedev/mame/issues/11587
Build using -fuse-ld=lld has succeeded:
https://koji.fedoraproject.org/koji/taskinfo?taskID=106961583
I have reported the problem to binutils upstream:
https://sourceware.org/bugzilla/show_bug.cgi?id=30930
Thanks for the help tracking this down.
Best regards,
Julian
>
>> #1 0x0000fffff5870b2c in __libc_start_main_impl () from
>> /lib64/libc.so.6
>> #2 0x0000aaaaaef01570 in _start ()
>>
>> One question: the binary in /builddir/build/BUILD still has symbols in,
>> correct?
>
> yes, the files under /builddir/build/BUILDROOT/... are split into
> stripped binaries and debuginfos
>> Best regards,
>> Julian
>
> [1] https://fedoraproject.org/wiki/Changes/Aarch64_PointerAuthentication
>
>
> Dan
> _______________________________________________
> devel mailing list -- devel(a)lists.fedoraproject.org
> To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
> Do not reply to spam, report it:
> https://pagure.io/fedora-infrastructure/new_issue
_______________________________________________
devel mailing list -- devel(a)lists.fedoraproject.org
To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it:
https://pagure.io/fedora-infrastructure/new_issue
243
days inactive
244
days old
9 comments
3 participants
participants (3)
-
Dan Horák -
John Reiser -
Julian Sikorski