7:51 a.m.
= Proposed Self Contained Change: RPM MPI Requires Provides =
https://fedoraproject.org/wiki/Changes/RpmMPIReqProv
Change owner(s): Sandro Mani <manisandro at gmail dot com>
Have the rpm-build find-provides and find-requires scripts encode the MPI compiler name in
the provides string of a binary to distinguish otherwise identical provides between
packages $foo, $foo-openmpi and $foo-mpich.
== Detailed Description ==
Currently, the packages libfoo, libfoo-openmpi and libfoo-mpich providing the library
libfoo.so all have a provides string of i.e.
libfoo.so()(64bit)
While yum used a shortest-package-name rule to choose which package to pick, dnf does not
have any rules, and seems to just pick the first match it comes across. Currently the only
solution would be to filter the provides from the -openmpi, -mpich packages and add
explicit Requires: where needed. I'd like to propose to extend the provides string in
such way that it also encodes the MPI implementation, i.e.:
$ rpm -qp --provides libfoo
libfoo.so()(64bit)
$ rpm -qp --provides libfoo-mpich
libfoo.so()(64bit)(mpich-x86_64)
$ rpm -qp --provides libfoo-openmpi
libfoo.so()(64bit)(openmpi-x86_64)
To this end, I'm proposing to adapt the find-requires and find-provides scripts as (or
similarly to):
find-provides
find-requires
Discussion of these changes are tracked in bug #1232504.
This change is intended to coordinate the rebuild of all MPI related packages to ensure
all such packages consistently use the new provides format.
== Scope ==
* Proposal owners:
- Work on find-provides and find-requires based on feedback.
- As soon as updated find-provides and find-requires shipped with rpm-build, do a
mass-rebuild of all MPI packages.
* Other developers: N/A (not a System Wide Change)
* Release engineering: N/A (not a System Wide Change)
* Policies and guidelines: N/A (not a System Wide Change)
* Trademark approval: N/A (not needed for this Change)
--
Jan Kuřík
_______________________________________________
devel-announce mailing list
devel-announce(a)lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel-announce
4:12 a.m.
Hello
I've got an initial implementation of this using the rpm dependency
generator hooks, as suggested in the other thread [1].
The resulting scripts are here:
https://smani.fedorapeople.org/rpm-mpi-hooks/
There is just one problem: an elf binary in an $MPI_HOME subfolder will
now trigger both the elf as well as the mpi dependency generator,
resulting in both
libfoo.so()(64bit)
libfoo.so()(64bit)(openmpi-x86_64)
being generated for /usr/lib64/openmpi/lib/libfoo.so. However, to
achieve the goal of disambiguating the provides of libfoo and
libfoo-openmpi, libfoo-openmpi should only provide
"libfoo.so()(64bit)(openmpi-x86_64)".
The only solution which comes to mind is adding
%__elf_exclude_path ^%{_prefix}/lib(64)/(openmpi|mpich)/.*$
to elf.attr. Is this acceptable?
Thanks,
Sandro
[1] https://lists.fedoraproject.org/pipermail/devel/2015-June/211570.html
3:28 p.m.
On 07/07/2015 03:12 AM, Sandro Mani wrote:
Hello
I've got an initial implementation of this using the rpm dependency generator
hooks, as suggested in the other thread [1].
The resulting scripts are here: https://smani.fedorapeople.org/rpm-mpi-hooks/
There is just one problem: an elf binary in an $MPI_HOME subfolder will now
trigger both the elf as well as the mpi dependency generator, resulting in both
libfoo.so()(64bit)
libfoo.so()(64bit)(openmpi-x86_64)
being generated for /usr/lib64/openmpi/lib/libfoo.so. However, to achieve the
goal of disambiguating the provides of libfoo and libfoo-openmpi,
libfoo-openmpi should only provide "libfoo.so()(64bit)(openmpi-x86_64)".
The only solution which comes to mind is adding
%__elf_exclude_path ^%{_prefix}/lib(64)/(openmpi|mpich)/.*$
to elf.attr. Is this acceptable?
It appears to be sufficient to define this macro anywhere, not just in
elf.attr. So I think it could be added in to a rpm macros file in
openmpi/mpich-devel. Also I think you need (64)? or (|64).
Cool stuff, thanks for working on this.
Thanks,
Sandro
[1] https://lists.fedoraproject.org/pipermail/devel/2015-June/211570.html
--
Orion Poplawski
Technical Manager 303-415-9701 x222
NWRA, Boulder/CoRA Office FAX: 303-415-9702
3380 Mitchell Lane orion(a)nwra.com
Boulder, CO 80301 http://www.nwra.com
6:03 p.m.
New subject: F23 Self Contained Change: RPM MPI Requires Provides -> Review request
On 08.07.2015 22:28, Orion Poplawski wrote:
It appears to be sufficient to define this macro anywhere, not just
in
elf.attr. So I think it could be added in to a rpm macros file in
openmpi/mpich-devel.
Ah cool, actually it seems to also work if I add it directly
to
mpi.attrs, meaning that all pieces are nicely confined in one place.
Also I think you need (64)? or (|64).
Correct, fixed. I'm
seeing this also in cmake:
$ grep '(64)' /usr/lib/rpm/fileattrs/cmake.attr
%__cmake_path ^/usr/lib(64)/cmake/.*/.*(Config\.cmake|-config\.cmake)$
suppose it should be fixed there too.
I've now posted the review request for the rpm-mpi-hooks package at
https://bugzilla.redhat.com/show_bug.cgi?id=1241282
A quick double check of the mpi.req and mpi.prov as part of the review
would also be appreciated.
Once the package is approved, openmpi and mpich will need to
BuildRequire: rpm-mpi-hooks, and openmpi-devel and mpich-devel will need
to Require: rpm-mpi-hooks.
And finally, once that is done too, all *-openmpi and *-mpich packages
will need to be rebuilt.
Thanks,
Sandro
1:31 p.m.
New subject: F23 Self Contained Change: RPM MPI Requires Provides -> Review request
On 07/08/2015 05:03 PM, Sandro Mani wrote:
On 08.07.2015 22:28, Orion Poplawski wrote:
> It appears to be sufficient to define this macro anywhere, not just in
> elf.attr. So I think it could be added in to a rpm macros file in
> openmpi/mpich-devel.
Ah cool, actually it seems to also work if I add it directly to mpi.attrs,
meaning that all pieces are nicely confined in one place.
Agreed.
> Also I think you need (64)? or (|64).
Correct, fixed. I'm seeing this also in cmake:
$ grep '(64)' /usr/lib/rpm/fileattrs/cmake.attr
%__cmake_path ^/usr/lib(64)/cmake/.*/.*(Config\.cmake|-config\.cmake)$
suppose it should be fixed there too.
Thanks. Fixed in rawhide now.
--
Orion Poplawski
Technical Manager 303-415-9701 x222
NWRA, Boulder/CoRA Office FAX: 303-415-9702
3380 Mitchell Lane orion(a)nwra.com
Boulder, CO 80301 http://www.nwra.com
2:39 p.m.
New subject: F23 Self Contained Change: RPM MPI Requires Provides -> Review request
On 07/08/2015 05:03 PM, Sandro Mani wrote:
Once the package is approved, openmpi and mpich will need to BuildRequire:
rpm-mpi-hooks, and openmpi-devel and mpich-devel will need to Require:
rpm-mpi-hooks.
And finally, once that is done too, all *-openmpi and *-mpich packages will
need to be rebuilt.
We (i.e. you :) ) should build this up in a copr first for testing.
--
Orion Poplawski
Technical Manager 303-415-9701 x222
NWRA, Boulder/CoRA Office FAX: 303-415-9702
3380 Mitchell Lane orion(a)nwra.com
Boulder, CO 80301 http://www.nwra.com
4:07 p.m.
New subject: F23 Self Contained Change: RPM MPI Requires Provides -> Review request
On 09.07.2015 21:39, Orion Poplawski wrote:
On 07/08/2015 05:03 PM, Sandro Mani wrote:
> Once the package is approved, openmpi and mpich will need to BuildRequire:
> rpm-mpi-hooks, and openmpi-devel and mpich-devel will need to Require:
> rpm-mpi-hooks.
>
> And finally, once that is done too, all *-openmpi and *-mpich packages will
> need to be rebuilt.
We (i.e. you :) ) should build this up in a copr first for testing.
Yes good idea, I'll start firing builds as soon as the pending issues
with rpm-mpi-hooks are resolved.
1:18 p.m.
On 07/07/2015 03:12 AM, Sandro Mani wrote:
Hello
I've got an initial implementation of this using the rpm dependency generator
hooks, as suggested in the other thread [1].
The resulting scripts are here: https://smani.fedorapeople.org/rpm-mpi-hooks/
There is just one problem: an elf binary in an $MPI_HOME subfolder will now
trigger both the elf as well as the mpi dependency generator, resulting in both
libfoo.so()(64bit)
libfoo.so()(64bit)(openmpi-x86_64)
Also, isn't the -x86_64 redundant? Also though I guess we don't have an mpi
variable MPI_NAME.
Also, your trick of using:
for module in $(module avail 2>&1 | grep "^mpi/"); do
to find the available mpi modules needs a "-t" option for Lmod. Fortunately
this also works with environment-modules:
for module in $(module -t avail 2>&1 | grep "^mpi/"); do
Also, it doesn't seem to get all of the requires quite right. For
scorep-openmpi I have:
Provides: libscorep_adapter_compiler_event.so.2()(64bit)(openmpi-x86_64)
but:
Requires: libscorep_adapter_compiler_event.so.2()(64bit)
is being emitted. This appears to be coming from:
./fileattrs/libsymlink.attr:%__libsymlink_requires
%{_rpmconfigdir}/elfdeps --provides --soname-only
So it looks like we need to contend with that as well.
# cat fileattrs/libsymlink.attr
# Make libfoo.so symlinks require the soname-provide of the target library
%__libsymlink_requires %{_rpmconfigdir}/elfdeps --provides --soname-only
%__libsymlink_magic ^symbolic link to `.*lib.*\.so\..*'$
%__libsymlink_exclude_path ^.*[[:digit:]]$
Perhaps with:
%global __libsymlink_exclude_path ^%{_prefix}/lib(64)?/(openmpi|mpich)/.*$
in mpi packages.
--
Orion Poplawski
Technical Manager 303-415-9701 x222
NWRA, Boulder/CoRA Office FAX: 303-415-9702
3380 Mitchell Lane orion(a)nwra.com
Boulder, CO 80301 http://www.nwra.com
2:14 p.m.
On 09.07.2015 20:18, Orion Poplawski wrote:
Also, isn't the -x86_64 redundant? Also though I guess we
don't have an mpi
variable MPI_NAME.
Yes it is redundant, but it is the prettiest variable I could
find,
given the lack of MPI_NAME.
Also, your trick of using:
for module in $(module avail 2>&1 | grep "^mpi/"); do
to find the available mpi modules needs a "-t" option for Lmod. Fortunately
this also works with environment-modules:
for module in $(module -t avail 2>&1 | grep "^mpi/"); do
Ok
Also, it doesn't seem to get all of the requires quite right.
For
scorep-openmpi I have:
Provides: libscorep_adapter_compiler_event.so.2()(64bit)(openmpi-x86_64)
but:
Requires: libscorep_adapter_compiler_event.so.2()(64bit)
is being emitted. This appears to be coming from:
./fileattrs/libsymlink.attr:%__libsymlink_requires
%{_rpmconfigdir}/elfdeps --provides --soname-only
So it looks like we need to contend with that as well.
# cat fileattrs/libsymlink.attr
# Make libfoo.so symlinks require the soname-provide of the target library
%__libsymlink_requires %{_rpmconfigdir}/elfdeps --provides --soname-only
%__libsymlink_magic ^symbolic link to `.*lib.*\.so\..*'$
%__libsymlink_exclude_path ^.*[[:digit:]]$
Perhaps with:
%global __libsymlink_exclude_path ^%{_prefix}/lib(64)?/(openmpi|mpich)/.*$
in mpi packages.
Or directly in mpi.attrs?
Thanks,
Sandro
2:42 p.m.
On 07/09/2015 01:14 PM, Sandro Mani wrote:
On 09.07.2015 20:18, Orion Poplawski wrote:
> Also, it doesn't seem to get all of the requires quite right. For
> scorep-openmpi I have:
>
> Provides: libscorep_adapter_compiler_event.so.2()(64bit)(openmpi-x86_64)
>
> but:
>
> Requires: libscorep_adapter_compiler_event.so.2()(64bit)
>
> is being emitted. This appears to be coming from:
>
> ./fileattrs/libsymlink.attr:%__libsymlink_requires
> %{_rpmconfigdir}/elfdeps --provides --soname-only
>
> So it looks like we need to contend with that as well.
>
> # cat fileattrs/libsymlink.attr
> # Make libfoo.so symlinks require the soname-provide of the target library
> %__libsymlink_requires %{_rpmconfigdir}/elfdeps --provides
> --soname-only
> %__libsymlink_magic ^symbolic link to `.*lib.*\.so\..*'$
> %__libsymlink_exclude_path ^.*[[:digit:]]$
>
>
> Perhaps with:
>
> %global __libsymlink_exclude_path ^%{_prefix}/lib(64)?/(openmpi|mpich)/.*$
>
> in mpi packages.
Or directly in mpi.attrs?
No, because there is a default %__libsymlink_exclude_path that we don't want
to override in general (think people building rpms locally with
openmpi/mpich-devel installed).
--
Orion Poplawski
Technical Manager 303-415-9701 x222
NWRA, Boulder/CoRA Office FAX: 303-415-9702
3380 Mitchell Lane orion(a)nwra.com
Boulder, CO 80301 http://www.nwra.com
4:06 p.m.
On 09.07.2015 21:42, Orion Poplawski wrote:
On 07/09/2015 01:14 PM, Sandro Mani wrote:
> On 09.07.2015 20:18, Orion Poplawski wrote:
>> Also, it doesn't seem to get all of the requires quite right. For
>> scorep-openmpi I have:
>>
>> Provides: libscorep_adapter_compiler_event.so.2()(64bit)(openmpi-x86_64)
>>
>> but:
>>
>> Requires: libscorep_adapter_compiler_event.so.2()(64bit)
>>
>> is being emitted. This appears to be coming from:
>>
>> ./fileattrs/libsymlink.attr:%__libsymlink_requires
>> %{_rpmconfigdir}/elfdeps --provides --soname-only
>>
>> So it looks like we need to contend with that as well.
>>
>> # cat fileattrs/libsymlink.attr
>> # Make libfoo.so symlinks require the soname-provide of the target library
>> %__libsymlink_requires %{_rpmconfigdir}/elfdeps --provides
>> --soname-only
>> %__libsymlink_magic ^symbolic link to `.*lib.*\.so\..*'$
>> %__libsymlink_exclude_path ^.*[[:digit:]]$
>>
>>
>> Perhaps with:
>>
>> %global __libsymlink_exclude_path ^%{_prefix}/lib(64)?/(openmpi|mpich)/.*$
>>
>> in mpi packages.
> Or directly in mpi.attrs?
No, because there is a default %__libsymlink_exclude_path that we don't want
to override in general (think people building rpms locally with
openmpi/mpich-devel installed).
Ah yes sorry didn't read the contents of fileattrs/libsymlink.attr
properly. But couldn't that be handled with a
%__libsymlink_path ^.*\.so$
%__libsymlink_flags magic_and_path
instead of the %__libsymlink_exclude_path in libsymlink.attr?
4:17 p.m.
On 07/09/2015 03:06 PM, Sandro Mani wrote:
On 09.07.2015 21:42, Orion Poplawski wrote:
> On 07/09/2015 01:14 PM, Sandro Mani wrote:
>> On 09.07.2015 20:18, Orion Poplawski wrote:
>>> Also, it doesn't seem to get all of the requires quite right. For
>>> scorep-openmpi I have:
>>>
>>> Provides: libscorep_adapter_compiler_event.so.2()(64bit)(openmpi-x86_64)
>>>
>>> but:
>>>
>>> Requires: libscorep_adapter_compiler_event.so.2()(64bit)
>>>
>>> is being emitted. This appears to be coming from:
>>>
>>> ./fileattrs/libsymlink.attr:%__libsymlink_requires
>>> %{_rpmconfigdir}/elfdeps --provides --soname-only
>>>
>>> So it looks like we need to contend with that as well.
>>>
>>> # cat fileattrs/libsymlink.attr
>>> # Make libfoo.so symlinks require the soname-provide of the target library
>>> %__libsymlink_requires %{_rpmconfigdir}/elfdeps --provides
>>> --soname-only
>>> %__libsymlink_magic ^symbolic link to `.*lib.*\.so\..*'$
>>> %__libsymlink_exclude_path ^.*[[:digit:]]$
>>>
>>>
>>> Perhaps with:
>>>
>>> %global __libsymlink_exclude_path
>>> ^%{_prefix}/lib(64)?/(openmpi|mpich)/.*$
>>>
>>> in mpi packages.
>> Or directly in mpi.attrs?
> No, because there is a default %__libsymlink_exclude_path that we don't want
> to override in general (think people building rpms locally with
> openmpi/mpich-devel installed).
>
Ah yes sorry didn't read the contents of fileattrs/libsymlink.attr properly.
But couldn't that be handled with a
%__libsymlink_path ^.*\.so$
%__libsymlink_flags magic_and_path
instead of the %__libsymlink_exclude_path in libsymlink.attr?
Could be. File a bug against redhat-rpm-config for that then.
--
Orion Poplawski
Technical Manager 303-415-9701 x222
NWRA, Boulder/CoRA Office FAX: 303-415-9702
3380 Mitchell Lane orion(a)nwra.com
Boulder, CO 80301 http://www.nwra.com
9:09 a.m.
On 09.07.2015 23:17, Orion Poplawski wrote:
On 07/09/2015 03:06 PM, Sandro Mani wrote:
>
> On 09.07.2015 21:42, Orion Poplawski wrote:
>> On 07/09/2015 01:14 PM, Sandro Mani wrote:
>>> On 09.07.2015 20:18, Orion Poplawski wrote:
>>>> Also, it doesn't seem to get all of the requires quite right. For
>>>> scorep-openmpi I have:
>>>>
>>>> Provides: libscorep_adapter_compiler_event.so.2()(64bit)(openmpi-x86_64)
>>>>
>>>> but:
>>>>
>>>> Requires: libscorep_adapter_compiler_event.so.2()(64bit)
>>>>
>>>> is being emitted. This appears to be coming from:
>>>>
>>>> ./fileattrs/libsymlink.attr:%__libsymlink_requires
>>>> %{_rpmconfigdir}/elfdeps --provides --soname-only
>>>>
>>>> So it looks like we need to contend with that as well.
>>>>
>>>> # cat fileattrs/libsymlink.attr
>>>> # Make libfoo.so symlinks require the soname-provide of the target
library
>>>> %__libsymlink_requires %{_rpmconfigdir}/elfdeps --provides
>>>> --soname-only
>>>> %__libsymlink_magic ^symbolic link to `.*lib.*\.so\..*'$
>>>> %__libsymlink_exclude_path ^.*[[:digit:]]$
>>>>
>>>>
>>>> Perhaps with:
>>>>
>>>> %global __libsymlink_exclude_path
>>>> ^%{_prefix}/lib(64)?/(openmpi|mpich)/.*$
>>>>
>>>> in mpi packages.
>>> Or directly in mpi.attrs?
>> No, because there is a default %__libsymlink_exclude_path that we don't want
>> to override in general (think people building rpms locally with
>> openmpi/mpich-devel installed).
>>
> Ah yes sorry didn't read the contents of fileattrs/libsymlink.attr properly.
> But couldn't that be handled with a
>
> %__libsymlink_path ^.*\.so$
> %__libsymlink_flags magic_and_path
>
> instead of the %__libsymlink_exclude_path in libsymlink.attr?
>
Could be. File a bug against redhat-rpm-config for that then.
FWIW, filed as #1241737, waiting for a reply.
10:40 a.m.
On 07/14/2015 08:09 AM, Sandro Mani wrote:
On 09.07.2015 23:17, Orion Poplawski wrote:
> On 07/09/2015 03:06 PM, Sandro Mani wrote:
>>
>> Ah yes sorry didn't read the contents of fileattrs/libsymlink.attr properly.
>> But couldn't that be handled with a
>>
>> %__libsymlink_path ^.*\.so$
>> %__libsymlink_flags magic_and_path
>>
>> instead of the %__libsymlink_exclude_path in libsymlink.attr?
>>
> Could be. File a bug against redhat-rpm-config for that then.
>
FWIW, filed as #1241737, waiting for a reply.
And quickly applied. Excellent, thanks.
--
Orion Poplawski
Technical Manager 303-415-9701 x222
NWRA, Boulder/CoRA Office FAX: 303-415-9702
3380 Mitchell Lane orion(a)nwra.com
Boulder, CO 80301 http://www.nwra.com
10:50 a.m.
On 17.07.2015 17:40, Orion Poplawski wrote:
On 07/14/2015 08:09 AM, Sandro Mani wrote:
>
> On 09.07.2015 23:17, Orion Poplawski wrote:
>> On 07/09/2015 03:06 PM, Sandro Mani wrote:
>>> Ah yes sorry didn't read the contents of fileattrs/libsymlink.attr
properly.
>>> But couldn't that be handled with a
>>>
>>> %__libsymlink_path ^.*\.so$
>>> %__libsymlink_flags magic_and_path
>>>
>>> instead of the %__libsymlink_exclude_path in libsymlink.attr?
>>>
>> Could be. File a bug against redhat-rpm-config for that then.
>>
> FWIW, filed as #1241737, waiting for a reply.
And quickly applied. Excellent, thanks.
Yep - I'm now building things in copr [1].
[1] https://copr.fedoraproject.org/coprs/smani/rpm-mpi-hooks-testing/
10:41 a.m.
On 07/17/2015 09:50 AM, Sandro Mani wrote:
Yep - I'm now building things in copr [1].
[1] https://copr.fedoraproject.org/coprs/smani/rpm-mpi-hooks-testing/
Great. Looks like you need to build a newer openmpi package in your copr
since I updating it to 1.8.7 in rawhide.
--
Orion Poplawski
Technical Manager 303-415-9701 x222
NWRA, Boulder/CoRA Office FAX: 303-415-9702
3380 Mitchell Lane orion(a)nwra.com
Boulder, CO 80301 http://www.nwra.com
10:45 a.m.
On 21.07.2015 17:41, Orion Poplawski wrote:
On 07/17/2015 09:50 AM, Sandro Mani wrote:
> Yep - I'm now building things in copr [1].
>
>
> [1] https://copr.fedoraproject.org/coprs/smani/rpm-mpi-hooks-testing/
Great. Looks like you need to build a newer openmpi package in your copr
since I updating it to 1.8.7 in rawhide.
Yes - I'm having some problems with
yum in copr not resolving the
dependencies, i.e.
DEBUG util.py:378: Error: Package: blacs-openmpi-2.0.2-8.1.fc24.x86_64
(coprbecloudfedoraprojectorg_results_smani_rpmmpihookstesting_fedorarawhidex86_64_)
DEBUG util.py:378: Requires: libmpi_mpifh.so.2()(64bit)(openmpi-x86_64)
DEBUG util.py:378: Available: openmpi-1.8.6-1.2.fc24.x86_64
(coprbecloudfedoraprojectorg_results_smani_rpmmpihookstesting_fedorarawhidex86_64_)
DEBUG util.py:378: libmpi_mpifh.so.2()(64bit)(openmpi-x86_64)
DEBUG util.py:378: Available: openmpi-devel-1.8.6-1.2.fc24.x86_64
(coprbecloudfedoraprojectorg_results_smani_rpmmpihookstesting_fedorarawhidex86_64_)
DEBUG util.py:378: libmpi_mpifh.so.2()(64bit)(openmpi-x86_64)
DEBUG util.py:378: Available: openmpi-1.8.7-1.fc24.i686 (fedora)
DEBUG util.py:378: Not found
DEBUG util.py:378: Available: openmpi-devel-1.8.7-1.fc24.i686 (fedora)
DEBUG util.py:378: Not found
but cannot reproduce this locally (neither with dnf nor with yum-deprecated). Haven't
had time to investigate properly yet though (oh and the -devel packages definitely should
not be providing the libraries).
11:03 a.m.
On 07/21/2015 09:45 AM, Sandro Mani wrote:
On 21.07.2015 17:41, Orion Poplawski wrote:
> On 07/17/2015 09:50 AM, Sandro Mani wrote:
>> Yep - I'm now building things in copr [1].
>>
>>
>> [1] https://copr.fedoraproject.org/coprs/smani/rpm-mpi-hooks-testing/
> Great. Looks like you need to build a newer openmpi package in your copr
> since I updating it to 1.8.7 in rawhide.
Yes - I'm having some problems with yum in copr not resolving the
dependencies, i.e.
DEBUG util.py:378: Error: Package: blacs-openmpi-2.0.2-8.1.fc24.x86_64
(coprbecloudfedoraprojectorg_results_smani_rpmmpihookstesting_fedorarawhidex86_64_)
DEBUG util.py:378: Requires:
libmpi_mpifh.so.2()(64bit)(openmpi-x86_64)
DEBUG util.py:378: Available: openmpi-1.8.6-1.2.fc24.x86_64
(coprbecloudfedoraprojectorg_results_smani_rpmmpihookstesting_fedorarawhidex86_64_)
DEBUG util.py:378: libmpi_mpifh.so.2()(64bit)(openmpi-x86_64)
DEBUG util.py:378: Available: openmpi-devel-1.8.6-1.2.fc24.x86_64
(coprbecloudfedoraprojectorg_results_smani_rpmmpihookstesting_fedorarawhidex86_64_)
DEBUG util.py:378: libmpi_mpifh.so.2()(64bit)(openmpi-x86_64)
DEBUG util.py:378: Available: openmpi-1.8.7-1.fc24.i686 (fedora)
DEBUG util.py:378: Not found
DEBUG util.py:378: Available: openmpi-devel-1.8.7-1.fc24.i686
(fedora)
DEBUG util.py:378: Not found
but cannot reproduce this locally (neither with dnf nor with yum-deprecated).
Haven't had time to investigate properly yet though (oh and the -devel
packages definitely should not be providing the libraries).
Again, you need to build a new openmpi in your copr that's newer than the
1.8.7-1 in Fedora now.
For the -devel issue, we need:
%__mpi_magic ^(setuid )?(setgid )?(sticky )?ELF (32|64)-bit.*$
%__mpi_flags exeonly
to match the %__elf_* options.
--
Orion Poplawski
Technical Manager 303-415-9701 x222
NWRA, Boulder/CoRA Office FAX: 303-415-9702
3380 Mitchell Lane orion(a)nwra.com
Boulder, CO 80301 http://www.nwra.com
11:06 a.m.
On 21.07.2015 18:03, Orion Poplawski wrote:
On 07/21/2015 09:45 AM, Sandro Mani wrote:
>
> On 21.07.2015 17:41, Orion Poplawski wrote:
>> On 07/17/2015 09:50 AM, Sandro Mani wrote:
>>> Yep - I'm now building things in copr [1].
>>>
>>>
>>> [1] https://copr.fedoraproject.org/coprs/smani/rpm-mpi-hooks-testing/
>> Great. Looks like you need to build a newer openmpi package in your copr
>> since I updating it to 1.8.7 in rawhide.
> Yes - I'm having some problems with yum in copr not resolving the
> dependencies, i.e.
>
> DEBUG util.py:378: Error: Package: blacs-openmpi-2.0.2-8.1.fc24.x86_64
> (coprbecloudfedoraprojectorg_results_smani_rpmmpihookstesting_fedorarawhidex86_64_)
>
> DEBUG util.py:378: Requires:
> libmpi_mpifh.so.2()(64bit)(openmpi-x86_64)
> DEBUG util.py:378: Available: openmpi-1.8.6-1.2.fc24.x86_64
> (coprbecloudfedoraprojectorg_results_smani_rpmmpihookstesting_fedorarawhidex86_64_)
>
> DEBUG util.py:378: libmpi_mpifh.so.2()(64bit)(openmpi-x86_64)
> DEBUG util.py:378: Available: openmpi-devel-1.8.6-1.2.fc24.x86_64
> (coprbecloudfedoraprojectorg_results_smani_rpmmpihookstesting_fedorarawhidex86_64_)
>
> DEBUG util.py:378: libmpi_mpifh.so.2()(64bit)(openmpi-x86_64)
> DEBUG util.py:378: Available: openmpi-1.8.7-1.fc24.i686 (fedora)
> DEBUG util.py:378: Not found
> DEBUG util.py:378: Available: openmpi-devel-1.8.7-1.fc24.i686
> (fedora)
> DEBUG util.py:378: Not found
>
> but cannot reproduce this locally (neither with dnf nor with yum-deprecated).
> Haven't had time to investigate properly yet though (oh and the -devel
> packages definitely should not be providing the libraries).
Again, you need to build a new openmpi in your copr that's newer than the
1.8.7-1 in Fedora now.
For the -devel issue, we need:
%__mpi_magic ^(setuid )?(setgid )?(sticky )?ELF (32|64)-bit.*$
%__mpi_flags exeonly
to match the %__elf_* options.
Ahh sorry didn't realize that this was related
to the version mismatch.
Ok thanks!
4:56 p.m.
Ok I've now got one full build of all MPI packages [1]. Investigating
the output, things are looking good, except for the fact that I realized
that I'll also need to handle binaries MPI_PYTHON_SITEARCH and
MPI_FORTRAN_MOD_DIR - these directories are outside MPI_HOME and hence
currently don't get the enhanced requires/provides strings. Since change
deadline is tomorrow, I'd like to see the rebuilds starting to land in
rawhide+F23 despite of this remaining issue, and then I'll look at
cleanly adapting the scripts to also handle the (few) remaining cases.
I've prepared a set of git am-able patches to apply to the various
packages here [2]. The rebuild sequence needed is posted below.
Know FTBFS are paraview (protobuf incompatibility) and netgen-mesher
(someone had the great idea of doing something like #define Status int
in a public header and I haven't yet got around to finding out who).
Affected by the missing handling of MPI_PYTHON_SITEARCH and
MPI_FORTRAN_MOD_DIR are elpa, sundials, pypar and mpi4py.
So if any proven packager could fire the rebuilds, that would be much
appreciated.
Thanks
Sandro
[1] https://copr.fedoraproject.org/coprs/smani/rpm-mpi-hooks-testing2
[2] https://smani.fedorapeople.org/rpm-mpi-hooks/patches/
Rebuild sequence (these are four groups, items within a group can be
built together):
mpich
openmpi
mpi4py
valgrind
pypar
boost
dl_poly
hpl
netgen-mesher
espresso
scotch
hdf5
scalapack
gromacs
scorep
sundials
Ray
orsa
mathgl
gmsh
elk
elpa
freefem++
netcdf
MUMPS
gpaw
ga
nwchem
cp2k
netcdf-cxx4
netcdf-fortran
coin-or-Ipopt
5:05 p.m.
On 27.07.2015 23:56, Sandro Mani wrote:
Ok I've now got one full build of all MPI packages [1].
Investigating
the output, things are looking good, except for the fact that I
realized that I'll also need to handle binaries MPI_PYTHON_SITEARCH
and MPI_FORTRAN_MOD_DIR - these directories are outside MPI_HOME and
hence currently don't get the enhanced requires/provides strings.
Since change deadline is tomorrow, I'd like to see the rebuilds
starting to land in rawhide+F23 despite of this remaining issue, and
then I'll look at cleanly adapting the scripts to also handle the
(few) remaining cases.
I've prepared a set of git am-able patches to apply to the various
packages here [2]. The rebuild sequence needed is posted below.
Know FTBFS are paraview (protobuf incompatibility) and netgen-mesher
(orrection: that would be gmsh
(someone had the great idea of doing something like #define Status
int
in a public header and I haven't yet got around to finding out who).
Affected by the missing handling of MPI_PYTHON_SITEARCH and
MPI_FORTRAN_MOD_DIR are elpa, sundials, pypar and mpi4py.
So if any proven packager could fire the rebuilds, that would be much
appreciated.
Thanks
Sandro
[1] https://copr.fedoraproject.org/coprs/smani/rpm-mpi-hooks-testing2
[2] https://smani.fedorapeople.org/rpm-mpi-hooks/patches/
Rebuild sequence (these are four groups, items within a group can be
built together):
mpich
openmpi
mpi4py
valgrind
pypar
boost
dl_poly
hpl
netgen-mesher
espresso
scotch
hdf5
scalapack
gromacs
scorep
sundials
Ray
orsa
mathgl
gmsh
elk
elpa
freefem++
netcdf
MUMPS
gpaw
ga
nwchem
cp2k
netcdf-cxx4
netcdf-fortran
coin-or-Ipopt
10:45 p.m.
On 07/27/2015 04:05 PM, Sandro Mani wrote:
On 27.07.2015 23:56, Sandro Mani wrote:
> Ok I've now got one full build of all MPI packages [1]. Investigating
> the output, things are looking good, except for the fact that I
> realized that I'll also need to handle binaries MPI_PYTHON_SITEARCH
> and MPI_FORTRAN_MOD_DIR - these directories are outside MPI_HOME and
> hence currently don't get the enhanced requires/provides strings.
> Since change deadline is tomorrow, I'd like to see the rebuilds
> starting to land in rawhide+F23 despite of this remaining issue, and
> then I'll look at cleanly adapting the scripts to also handle the
> (few) remaining cases.
>
> I've prepared a set of git am-able patches to apply to the various
> packages here [2]. The rebuild sequence needed is posted below.
>
> Know FTBFS are paraview (protobuf incompatibility) and netgen-mesher
(orrection: that would be gmsh
> (someone had the great idea of doing something like #define Status int
> in a public header and I haven't yet got around to finding out who).
> Affected by the missing handling of MPI_PYTHON_SITEARCH and
> MPI_FORTRAN_MOD_DIR are elpa, sundials, pypar and mpi4py.
>
> So if any proven packager could fire the rebuilds, that would be much
> appreciated.
Where are we with this? I see some F23 builds but no updates yet.
>
> Thanks
> Sandro
>
>
> [1] https://copr.fedoraproject.org/coprs/smani/rpm-mpi-hooks-testing2
> [2] https://smani.fedorapeople.org/rpm-mpi-hooks/patches/
>
>
> Rebuild sequence (these are four groups, items within a group can be
> built together):
> mpich
> openmpi
>
> mpi4py
> valgrind
> pypar
> boost
> dl_poly
> hpl
> netgen-mesher
> espresso
> scotch
> hdf5
> scalapack
> gromacs
> scorep
> sundials
> Ray
> orsa
> mathgl
> gmsh
> elk
>
> elpa
> freefem++
> netcdf
> MUMPS
> gpaw
> ga
>
> nwchem
> cp2k
> netcdf-cxx4
> netcdf-fortran
> coin-or-Ipopt
>
--
Orion Poplawski
Technical Manager 303-415-9701 x222
NWRA/CoRA Division FAX: 303-415-9702
3380 Mitchell Lane orion(a)cora.nwra.com
Boulder, CO 80301 http://www.cora.nwra.com
6:31 a.m.
On Sat, Aug 15, 2015 at 5:45 AM, Orion Poplawski <orion(a)cora.nwra.com>
wrote:
On 07/27/2015 04:05 PM, Sandro Mani wrote:
>
>
> On 27.07.2015 23:56, Sandro Mani wrote:
>
>> Ok I've now got one full build of all MPI packages [1]. Investigating
>> the output, things are looking good, except for the fact that I
>> realized that I'll also need to handle binaries MPI_PYTHON_SITEARCH
>> and MPI_FORTRAN_MOD_DIR - these directories are outside MPI_HOME and
>> hence currently don't get the enhanced requires/provides strings.
>> Since change deadline is tomorrow, I'd like to see the rebuilds
>> starting to land in rawhide+F23 despite of this remaining issue, and
>> then I'll look at cleanly adapting the scripts to also handle the
>> (few) remaining cases.
>>
>> I've prepared a set of git am-able patches to apply to the various
>> packages here [2]. The rebuild sequence needed is posted below.
>>
>> Know FTBFS are paraview (protobuf incompatibility) and netgen-mesher
>>
>
> (orrection: that would be gmsh
>
> (someone had the great idea of doing something like #define Status int
>> in a public header and I haven't yet got around to finding out who).
>> Affected by the missing handling of MPI_PYTHON_SITEARCH and
>> MPI_FORTRAN_MOD_DIR are elpa, sundials, pypar and mpi4py.
>>
>> So if any proven packager could fire the rebuilds, that would be much
>> appreciated.
>>
>
Where are we with this? I see some F23 builds but no updates yet.
Formally, this Change has been removed from the scope of F23 as it was not
ready for Alpha, which is a requirement to have a Change in a release.
However Zbyszek told me yesterday that he is going to work on it and would
like to push this F23. So, lets see when it will be ready and then decide
whether it can still meet the F23.
Regards,
Jan
>> Thanks
>> Sandro
>>
>>
>> [1] https://copr.fedoraproject.org/coprs/smani/rpm-mpi-hooks-testing2
>> [2] https://smani.fedorapeople.org/rpm-mpi-hooks/patches/
>>
>>
>> Rebuild sequence (these are four groups, items within a group can be
>> built together):
>> mpich
>> openmpi
>>
>> mpi4py
>> valgrind
>> pypar
>> boost
>> dl_poly
>> hpl
>> netgen-mesher
>> espresso
>> scotch
>> hdf5
>> scalapack
>> gromacs
>> scorep
>> sundials
>> Ray
>> orsa
>> mathgl
>> gmsh
>> elk
>>
>> elpa
>> freefem++
>> netcdf
>> MUMPS
>> gpaw
>> ga
>>
>> nwchem
>> cp2k
>> netcdf-cxx4
>> netcdf-fortran
>> coin-or-Ipopt
>>
>>
>
--
Orion Poplawski
Technical Manager 303-415-9701 x222
NWRA/CoRA Division FAX: 303-415-9702
3380 Mitchell Lane orion(a)cora.nwra.com
Boulder, CO 80301 http://www.cora.nwra.com
--
devel mailing list
devel(a)lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
6:49 a.m.
On 15.08.2015 13:31, Jan Kurik wrote:
On Sat, Aug 15, 2015 at 5:45 AM, Orion Poplawski
<orion(a)cora.nwra.com
<mailto:orion@cora.nwra.com>> wrote:
On 07/27/2015 04:05 PM, Sandro Mani wrote:
On 27.07.2015 23 <tel:27.07.2015%2023>:56, Sandro Mani wrote:
Ok I've now got one full build of all MPI packages [1].
Investigating
the output, things are looking good, except for the fact
that I
realized that I'll also need to handle binaries
MPI_PYTHON_SITEARCH
and MPI_FORTRAN_MOD_DIR - these directories are outside
MPI_HOME and
hence currently don't get the enhanced requires/provides
strings.
Since change deadline is tomorrow, I'd like to see the
rebuilds
starting to land in rawhide+F23 despite of this remaining
issue, and
then I'll look at cleanly adapting the scripts to also
handle the
(few) remaining cases.
I've prepared a set of git am-able patches to apply to the
various
packages here [2]. The rebuild sequence needed is posted
below.
Know FTBFS are paraview (protobuf incompatibility) and
netgen-mesher
(orrection: that would be gmsh
(someone had the great idea of doing something like
#define Status int
in a public header and I haven't yet got around to finding
out who).
Affected by the missing handling of MPI_PYTHON_SITEARCH and
MPI_FORTRAN_MOD_DIR are elpa, sundials, pypar and mpi4py.
So if any proven packager could fire the rebuilds, that
would be much
appreciated.
Where are we with this? I see some F23 builds but no updates yet.
Formally, this Change has been removed from the scope of F23 as it was
not ready for Alpha, which is a requirement to have a Change in a release.
However Zbyszek told me yesterday that he is going to work on it and
would like to push this F23. So, lets see when it will be ready and
then decide whether it can still meet the F23.
Regards,
Jan
So the issues I pointed out in my previous mail (conerning
MPI_PYTHON_SITEARCH and MPI_FORTRAN_MOD_DIR) above have been resolved. A
batch of F24 rebuilts has been done by Zbyszek, hitting some build
failures along the way. Most of these are now fixed, still needing a fix
is nwchem which fails [1] due to what appears to be a fortran "use
before decleared" issue. Once a full set of working F23 package is
built, the plan is to submit one big mass-update.
If anyone knowledgable of fortran has an idea how to fix the nwchem
issue, that should resolve the last pending issue. Appears to have been
triggered by a recent change of some other component, since it rebuilt
fine just a few weeks back when testing in copr.
Sandro
[1] http://koji.fedoraproject.org/koji/buildinfo?buildID=676763
8:31 a.m.
On Sat, Aug 15, 2015 at 2:49 PM, Sandro Mani <manisandro(a)gmail.com> wrote:
So the issues I pointed out in my previous mail (conerning
MPI_PYTHON_SITEARCH and MPI_FORTRAN_MOD_DIR) above have been resolved. A
batch of F24 rebuilts has been done by Zbyszek, hitting some build failures
along the way. Most of these are now fixed, still needing a fix is nwchem
which fails [1] due to what appears to be a fortran "use before decleared"
issue. Once a full set of working F23 package is built, the plan is to
submit one big mass-update.
If anyone knowledgable of fortran has an idea how to fix the nwchem issue,
that should resolve the last pending issue. Appears to have been triggered
by a recent change of some other component, since it rebuilt fine just a few
weeks back when testing in copr.
Sandro
[1] http://koji.fedoraproject.org/koji/buildinfo?buildID=676763
I haven't touched fortran in quite some time, so take this with with a
grain of salt (preferably more).
The issue likely has to do with some transliterations (?) from 32-bit
to 64-bit and vice versa when using particular libraries. Upstream has
released quite a few patches for build 26243, which might address
this:
http://www.nwchem-sw.org/index.php/Download#Patches_for_the_26243_revisio...
my money would be on Tddft_grad.patch.gz but I don't bet as I don't
have any luck...
7:36 a.m.
Sandro Mani <manisandro(a)gmail.com> writes:
> Where are we with this? I see some F23 builds but no updates
yet.
>
>
> Formally, this Change has been removed from the scope of F23 as it
> was not ready for Alpha, which is a requirement to have a Change in
> a release.
> However Zbyszek told me yesterday that he is going to work on it and
> would like to push this F23. So, lets see when it will be ready and
> then decide whether it can still meet the F23.
>
> Regards,
> Jan
So the issues I pointed out in my previous mail (conerning
MPI_PYTHON_SITEARCH and MPI_FORTRAN_MOD_DIR) above have been
resolved. A batch of F24 rebuilts has been done by Zbyszek, hitting
some build failures along the way. Most of these are now fixed, still
needing a fix is nwchem which fails [1] due to what appears to be a
fortran "use before decleared" issue. Once a full set of working F23
package is built, the plan is to submit one big mass-update.
Should MPI packages be OK to push in f23 currently? I've just had
failures from auto-qa like this, though the build actually ran an MPI
test successfully:
not ok - depcheck for Koji build scalasca-2.2.2-2.fc23 # FAIL
---
arch: x86_64
details:
output: |-
conflicting requests
nothing provides libmpi.so.12()(64bit)(mpich-x86_64) needed by
scalasca-mpich-2.2.2-2.fc23.x86_64
nothing provides libmpicxx.so.12()(64bit)(mpich-x86_64) needed by
scalasca-mpich-2.2.2-2.fc23.x86_64
conflicting requests
nothing provides libmpi.so.12()(64bit)(mpich-x86_64) needed by
scalasca-mpich-2.2.2-2.fc23.x86_64
nothing provides libmpicxx.so.12()(64bit)(mpich-x86_64) needed by
scalasca-mpich-2.2.2-2.fc23.x86_64
conflicting requests
nothing provides libmpi.so.1()(64bit)(openmpi-x86_64) needed by
scalasca-openmpi-2.2.2-2.fc23.x86_64
nothing provides libmpi_cxx.so.1()(64bit)(openmpi-x86_64) needed by
scalasca-openmpi-2.2.2-2.fc23.x86_64
If anyone knowledgable of fortran has an idea how to fix the nwchem
issue, that should resolve the last pending issue. Appears to have
been triggered by a recent change of some other component, since it
rebuilt fine just a few weeks back when testing in copr.
Sandro
[1] http://koji.fedoraproject.org/koji/buildinfo?buildID=676763
I can take a look if no-one else is going to.
11:11 a.m.
On 08/19/2015 06:36 AM, Dave Love wrote:
Should MPI packages be OK to push in f23 currently? I've just
had
failures from auto-qa like this, though the build actually ran an MPI
test successfully:
not ok - depcheck for Koji build scalasca-2.2.2-2.fc23 # FAIL
---
arch: x86_64
details:
output: |-
conflicting requests
nothing provides libmpi.so.12()(64bit)(mpich-x86_64) needed by
scalasca-mpich-2.2.2-2.fc23.x86_64
nothing provides libmpicxx.so.12()(64bit)(mpich-x86_64) needed by
scalasca-mpich-2.2.2-2.fc23.x86_64
conflicting requests
nothing provides libmpi.so.12()(64bit)(mpich-x86_64) needed by
scalasca-mpich-2.2.2-2.fc23.x86_64
nothing provides libmpicxx.so.12()(64bit)(mpich-x86_64) needed by
scalasca-mpich-2.2.2-2.fc23.x86_64
conflicting requests
nothing provides libmpi.so.1()(64bit)(openmpi-x86_64) needed by
scalasca-openmpi-2.2.2-2.fc23.x86_64
nothing provides libmpi_cxx.so.1()(64bit)(openmpi-x86_64) needed by
scalasca-openmpi-2.2.2-2.fc23.x86_64
You're going to need to wait for the big MPI update:
https://admin.fedoraproject.org/updates/rpm-mpi-hooks-3-1.fc23,mathgl-2.3...
--
Orion Poplawski
Technical Manager 303-415-9701 x222
NWRA, Boulder/CoRA Office FAX: 303-415-9702
3380 Mitchell Lane orion(a)nwra.com
Boulder, CO 80301 http://www.nwra.com
11:47 a.m.
On Wed, Aug 19, 2015 at 10:11:04AM -0600, Orion Poplawski wrote:
On 08/19/2015 06:36 AM, Dave Love wrote:
> Should MPI packages be OK to push in f23 currently? I've just had
> failures from auto-qa like this, though the build actually ran an MPI
> test successfully:
>
> not ok - depcheck for Koji build scalasca-2.2.2-2.fc23 # FAIL
> ---
> arch: x86_64
> details:
> output: |-
> conflicting requests
> nothing provides libmpi.so.12()(64bit)(mpich-x86_64) needed by
scalasca-mpich-2.2.2-2.fc23.x86_64
> nothing provides libmpicxx.so.12()(64bit)(mpich-x86_64) needed by
scalasca-mpich-2.2.2-2.fc23.x86_64
> conflicting requests
> nothing provides libmpi.so.12()(64bit)(mpich-x86_64) needed by
scalasca-mpich-2.2.2-2.fc23.x86_64
> nothing provides libmpicxx.so.12()(64bit)(mpich-x86_64) needed by
scalasca-mpich-2.2.2-2.fc23.x86_64
> conflicting requests
> nothing provides libmpi.so.1()(64bit)(openmpi-x86_64) needed by
scalasca-openmpi-2.2.2-2.fc23.x86_64
> nothing provides libmpi_cxx.so.1()(64bit)(openmpi-x86_64) needed by
scalasca-openmpi-2.2.2-2.fc23.x86_64
You're going to need to wait for the big MPI update:
https://admin.fedoraproject.org/updates/rpm-mpi-hooks-3-1.fc23,mathgl-2.3...
The update is now
https://admin.fedoraproject.org/updates/rpm-mpi-hooks-3-2.fc23
If you made an update with scalasca for F23, please cancel it and
I'll add scalasca to the big update. (I cannot check the status
because admin.fedoraproject.org is overloaded.)
Zbyszek
3167
days inactive
3229
days old
27 comments
6 participants
participants (6)
-
Alexander Ploumistos -
Dave Love -
Jan Kurik -
Orion Poplawski -
Sandro Mani -
Zbigniew Jędrzejewski-Szmek