Hi all,
I'm currently working on packaging the whole ROS stack by automatically generating SPEC files with the help of the rosinstall_generator. I think the project is far enough to be shared on this list, but it is still work in progress.
The SPEC files are generated with a python script. You can check it out here: https://pagure.io/ros
I've created a COPR for ROS Kinetic that contains all packages from desktop_full and move_base for F24+ (currently only for x86_64): https://copr.fedorainfracloud.org/coprs/thofmann/ros/
The scripts should also work for ROS Lunar (and possibly other releases), but I've only tested Kinetic so far.
I decided to package each ROS package into a separate Fedora package, because imho that's the best mapping from ROS to Fedora: Each package has its own sources, different dependencies, and some packages require different build arguments. This way, it's also much easier to add new (ROS) packages to Fedora.
I picked /usr/lib64/ros as ROS_ROOT. Currently, this does not allow having two separate ROS installations. But moving it to e.g., /usr/lib64/ros/kinetic should be as simple as adapting the template file. Is that something we would want to have?
There are still some remaining tasks on my list: - Provides: for ros-$pkgname (currently, the packages are all called ros-kinetic-$pkgname) - Clean up dependencies, many packages have explicit runtime dependencies that they don't need because they are automatic dependencies on the sonames. Some packages also have devel packages as runtime dependency, which is of course incorrect. - Splitting devel packages. I'm not sure how useful this would be, but I think it is required by the packaging guidelines. - I thought about adding the ROS lib dir to /etc/ld.so.conf.d, but I'm not sure whether this is a good idea because most libraries have unversioned sonames. On the other hand, all the packages currently Provide: their (unversioned) libraries, which may have unexpected effects, e.g., a program linking (and requiring) a ros lib without sourcing the ROS environment. - As it is, the ROS packages have a lot of bundled third-party libraries. This may require some work to separate if at all possible. - I haven't really checked any existing Fedora ROS packages other than catkin_pkg.
In the long run, it would be awesome to get those packages into Fedora (one-by-one), maybe even with different modules for the different ROS versions - modularity could be very useful here!
In the meantime, I'd appreciate any feedback, test results, RFEs, package requests, etc. The best place for this would be the repository: https://pagure.io/ros/issues
We (team Carologistics, RCLL) have successfully tested some of those packages on Fedora 25 at the last RoboCup in Japan, but of course more tests would be nice, as we only use a few packages.
Please let me know what you think!
Kind regards, Till
On 08/16/2017 04:19 PM, Till Hofmann wrote:
Hi all,
I'm currently working on packaging the whole ROS stack by automatically generating SPEC files with the help of the rosinstall_generator. I think the project is far enough to be shared on this list, but it is still work in progress.
The SPEC files are generated with a python script. You can check it out here: https://pagure.io/ros
I've created a COPR for ROS Kinetic that contains all packages from desktop_full and move_base for F24+ (currently only for x86_64): https://copr.fedorainfracloud.org/coprs/thofmann/ros/
The scripts should also work for ROS Lunar (and possibly other releases), but I've only tested Kinetic so far.
I decided to package each ROS package into a separate Fedora package, because imho that's the best mapping from ROS to Fedora: Each package has its own sources, different dependencies, and some packages require different build arguments. This way, it's also much easier to add new (ROS) packages to Fedora.
I picked /usr/lib64/ros as ROS_ROOT. Currently, this does not allow having two separate ROS installations. But moving it to e.g., /usr/lib64/ros/kinetic should be as simple as adapting the template file. Is that something we would want to have?
There are still some remaining tasks on my list:
- Provides: for ros-$pkgname (currently, the packages are all called
ros-kinetic-$pkgname)
- Clean up dependencies, many packages have explicit runtime
dependencies that they don't need because they are automatic dependencies on the sonames. Some packages also have devel packages as runtime dependency, which is of course incorrect.
- Splitting devel packages. I'm not sure how useful this would be, but I
think it is required by the packaging guidelines.
- I thought about adding the ROS lib dir to /etc/ld.so.conf.d, but I'm
not sure whether this is a good idea because most libraries have unversioned sonames. On the other hand, all the packages currently Provide: their (unversioned) libraries, which may have unexpected effects, e.g., a program linking (and requiring) a ros lib without sourcing the ROS environment.
- As it is, the ROS packages have a lot of bundled third-party
libraries. This may require some work to separate if at all possible.
- I haven't really checked any existing Fedora ROS packages other than
catkin_pkg.
Wow, thanks for taking this on. I haven't had any cycles to work on ROS in a long time, but this is work I support.
* I do think that making -devel packages might be useful, but if you can make the case as to why they aren't useful (as opposed to "harder to script"), you can probably get an exception for ROS packages.
* I don't think you want ROS libs in the ld cache, that seems like a recipe for disaster. You also probably want to disable the autoprovides generation and instead do something like:
Provides: ros(library.so)
or
Provides: ros-kinetic(library.so)
* The unbundling is going to suck, except in the rare cases where the bundled code is unmodified. Thankfully, you should be able to use the bundled code as long as you follow the Fedora Packaging Guidelines around that: https://fedoraproject.org/wiki/Bundled_Libraries
~tom
Thanks for the feedback!
On 08/16/2017 10:29 PM, Tom Callaway wrote:
- I do think that making -devel packages might be useful, but if you can
make the case as to why they aren't useful (as opposed to "harder to script"), you can probably get an exception for ROS packages.
My first thought was that in most use cases, people would use those ROS packages as base to build their own software on top of it, which would always require the devel package anyway. But on second thought, that's only for some of the packages, and people may want to just run ROS without a lot of custom software, so having separate devel packages is probably useful, I just haven't started working on the split.
- I don't think you want ROS libs in the ld cache, that seems like a
recipe for disaster. You also probably want to disable the autoprovides generation and instead do something like:
Provides: ros(library.so)
or
Provides: ros-kinetic(library.so)
That seems like a good idea, although I would have liked to use more of RPM's implicit dependency management instead of essentially re-adding all those Provides manually. Do you have an idea how we could add those Provides automatically, or at least in a scripted way without looking at the resulting packages first? So far I can generate all SPEC files without looking into the generated RPMs, (with the exceptions that have custom configs and/or templates) it would be nice if we could keep it that way.
- The unbundling is going to suck, except in the rare cases where the
bundled code is unmodified. Thankfully, you should be able to use the bundled code as long as you follow the Fedora Packaging Guidelines around that: https://fedoraproject.org/wiki/Bundled_Libraries
Yes, that's what I was aiming for. Unbundling would be ideal, but probably not feasible. But this is also a step that has to be done separately for each package, so far I've been trying to avoid this whenever possible.
Kind regards, Till
On 08/16/2017 05:08 PM, Till Hofmann wrote:
Do you have an idea how we could add those Provides automatically, or at least in a scripted way without looking at the resulting packages first? So far I can generate all SPEC files without looking into the generated RPMs, (with the exceptions that have custom configs and/or templates) it would be nice if we could keep it that way.
Hmm. It _might_ be possible to write a generic converter macro that parsed the generated provides and replaced any Provides for unversioned libraries with the special ros Provide.
Or, you could make something that is the ros equivalent of /usr/lib/rpm/perl.prov. You can look through /usr/lib/rpm/ at all the other .prov files for inspiration. (This seems like the better path forward)
~tom
On 08/16/2017 10:29 PM, Tom Callaway wrote:
- I don't think you want ROS libs in the ld cache, that seems like a
recipe for disaster. You also probably want to disable the autoprovides generation and instead do something like:
Provides: ros(library.so)
or
Provides: ros-kinetic(library.so)
I've figured how to do that with a __ros_provides script (once I understood the RPM macro system, this was actually quite simple). But I'm not sure how to translate the Requires: accordingly. I have ROS libs that "Provides: ros(libfoo.so)", but I don't know how to create a "Requires: ros(libfoo.so)" in a package that uses libfoo.
So far I see two options: 1) Run /usr/lib/rpmdeps from my __ros_requires script and translate ROS libs accordingly. But this would mean I would have to check for each dependency whether it is part of a ROS package. 2) Don't use rpmdeps but instead run ldd manually and filter ldd's output. This makes the distinction between ROS libs and other libs easier, but I would have to recreate the behavior of rpmdeps, which sounds like a terrible idea.
So neither option seems really satisfactory. Any other ideas?
Kind regards, Till
Hello Till:
On 16/08/17 22:19, Till Hofmann wrote:
Hi all,
I'm currently working on packaging the whole ROS stack by automatically generating SPEC files with the help of the rosinstall_generator. I think the project is far enough to be shared on this list, but it is still work in progress.
I just want to mention that there was a previous effort to package ROS in Fedora in the past, mostly done by Rich Mattes: - https://fedoraproject.org/wiki/SIGs/Robotics/ROS_Packaging
A similar work has been done in Debian recently: - https://wiki.debian.org/DebianScience/Robotics/ROS/Motivation
You can check their approach which does not rely in ROS distribution (ROS debian packages[1] has no ROS distribution in their names) and also map one ROS package with one downstream distribution package.
Some patches in Debian and specially the work done in Fedora before could be useful to you.
Good luck!
[1] https://wiki.debian.org/DebianScience/Robotics/ROS/Packages
On Thu, Aug 17, 2017 at 5:20 PM, Jose Luis Rivero jrivero@osrfoundation.org wrote:
Hello Till:
On 16/08/17 22:19, Till Hofmann wrote:
Hi all,
I'm currently working on packaging the whole ROS stack by automatically generating SPEC files with the help of the rosinstall_generator. I think the project is far enough to be shared on this list, but it is still work in progress.
I just want to mention that there was a previous effort to package ROS in Fedora in the past, mostly done by Rich Mattes:
A similar work has been done in Debian recently:
You can check their approach which does not rely in ROS distribution (ROS debian packages[1] has no ROS distribution in their names) and also map one ROS package with one downstream distribution package.
Some patches in Debian and specially the work done in Fedora before could be useful to you.
Good luck!
[1] https://wiki.debian.org/DebianScience/Robotics/ROS/Packages
Hi Till,
This is great, thanks for taking this on! Expanding on what Jose mentioned: there have been a couple efforts to support ROS packaging in Fedora, none of which made it all the way to a usable product. A little bit of history:
As Jose said, I did try to package ROS in Fedora, around the fuerte timeframe in 2012. As a result of that effort, about half of the core ROS stack as outlined on the robotics sig wiki page is packaged for fedora, and the other half is sitting in review requests. When I did it, I took the approach that there would be one ros distro provided in the core Fedora repositories, and that they would be installed like any other system package (e.g. libs in /usr/lib<qual>, config in /etc, data in /usr/share.) I figured that if anyone needed support for a separate ros distro, they could continue to build manually and install to /opt. I was also creating the specfiles manually, so that I could create proper subpackages (-devel, -doc, etc.) in each individual case. I think that this effort closely maps to the debian science efforts, though they got a lot farther than I did. I did recently look at trying to bring the debian science patches to fedora to restart the packaging process, but I haven't had the time to dedicate to such an effort.
Around the 2014 timeframe, we started to investigate the possibility of having the ros buildfarm generate fedora RPMs in parallel to the Ubuntu debs, so that all of the ros distros would be available in a manner similar to Ubuntu. Scott Logan did the work there, and landed patches to the ROS 'bloom' tool enabling it to generate spec files for fedora. That infrastructure is in place, but we got hung up in working with the OSRF to enable RPM building in their buildfarm, and then that fizzled out as well.
As it stands today, a lot of ROS dependencies are packaged in Fedora, making it possible to build from source pretty easily. I think that to move forward safely, you need to decide for the long term whether you want to support: - ROS installation via OSRF resources - ROS installation via copr - ROS packages in the distribution proper
Supporting ROS as a copr gives you a lot of leeway to do things that don't conform to the packaging guidelines, such as installing parallel stacks to /opt, creating single rpms for each package, etc. To bring the packages into fedora proper, as Tom was getting at, you need to reconcile how to support multiple software stacks within the package guidelines (and possibly get some exemptions to the guidelines,) and how to auto-generate specs that meet the packaging guidelines (proper division of -devel packages, etc.)
In case that wasn't enough to worry about, there is also a "Modularity" effort going on in Fedora. I have not been able to figure out what they're doing, but [1] claims that it's modules can have different lifecycles than the base distribution. It may provide a model to maintain multiple ROS stacks in parallel, but I don't really have any more information than that.
I'm happy to add you to the existing ROS packages that have been reviewed, add you to the robotics-sig user group, and continue to discuss on the mailing list.
As a side note, I've recently been poking at ros2 packaging[2]. The ecosystem is a lot smaller at the moment, and it may be able to re-use some of the concepts and tooling that get worked out for ros 1.
[1] https://docs.pagure.org/modularity/ [2] https://copr.fedorainfracloud.org/coprs/g/robotics-sig/ros2/
Hi
/*Rich Mattes*/ wrote on Sun, 20 Aug 2017 12:29:12 -0400:
<...> In case that wasn't enough to worry about, there is also a "Modularity" effort going on in Fedora. I have not been able to figure out what they're doing, but [1] claims that it's modules can have different lifecycles than the base distribution. It may provide a model to maintain multiple ROS stacks in parallel, but I don't really have any more information than that.
Yes, it provides a model to maintain multiple ROS stacks in parallel. However, currently, they are most likely assuming that only a single stack can be installed at a given time (if they just 'assume' it, it should be fine. But if they enforce it, you'll be unable to install multiple stacks simultaneously even if their packages doesn't conflict).
Regards, Hedayat
<...> [1] https://docs.pagure.org/modularity/
Hi all,
thanks again to everyone for the valuable feedback!
On 08/20/2017 06:29 PM, Rich Mattes wrote:
On Thu, Aug 17, 2017 at 5:20 PM, Jose Luis Rivero jrivero@osrfoundation.org wrote:
Hello Till:
On 16/08/17 22:19, Till Hofmann wrote:
Hi all,
I'm currently working on packaging the whole ROS stack by automatically generating SPEC files with the help of the rosinstall_generator. I think the project is far enough to be shared on this list, but it is still work in progress.
I just want to mention that there was a previous effort to package ROS in Fedora in the past, mostly done by Rich Mattes:
A similar work has been done in Debian recently:
You can check their approach which does not rely in ROS distribution (ROS debian packages[1] has no ROS distribution in their names) and also map one ROS package with one downstream distribution package.
Some patches in Debian and specially the work done in Fedora before could be useful to you.
Good luck!
[1] https://wiki.debian.org/DebianScience/Robotics/ROS/Packages
Hi Till,
This is great, thanks for taking this on! Expanding on what Jose mentioned: there have been a couple efforts to support ROS packaging in Fedora, none of which made it all the way to a usable product. A little bit of history:
As Jose said, I did try to package ROS in Fedora, around the fuerte timeframe in 2012. As a result of that effort, about half of the core ROS stack as outlined on the robotics sig wiki page is packaged for fedora, and the other half is sitting in review requests. When I did it, I took the approach that there would be one ros distro provided in the core Fedora repositories, and that they would be installed like any other system package (e.g. libs in /usr/lib<qual>, config in /etc, data in /usr/share.) I figured that if anyone needed support for a separate ros distro, they could continue to build manually and install to /opt. I was also creating the specfiles manually, so that I could create proper subpackages (-devel, -doc, etc.) in each individual case. I think that this effort closely maps to the debian science efforts, though they got a lot farther than I did. I did recently look at trying to bring the debian science patches to fedora to restart the packaging process, but I haven't had the time to dedicate to such an effort. > Around the 2014 timeframe, we started to investigate the possibility of having the ros buildfarm generate fedora RPMs in parallel to the Ubuntu debs, so that all of the ros distros would be available in a manner similar to Ubuntu. Scott Logan did the work there, and landed patches to the ROS 'bloom' tool enabling it to generate spec files for fedora. That infrastructure is in place, but we got hung up in working with the OSRF to enable RPM building in their buildfarm, and then that fizzled out as well.
Thanks for the summary, I've seen the Fedora wiki pages on ROS, although I didn't realize that many of those are already available in the repositories. I'm not seeing a list of the ROS packages packaged for Fedora, is that available somewhere? The build farm approach sounds like an interesting alternative, do you have a link where I can get further information about that?
I decided to start packaging from scratch rather than building on top of the existing packages for several reasons: 1. Since fuerte, there were quite some changes ROS in general and the build system in particular. I thought it would be easier to start from scratch, rather than building on top of the fuerte packages. 2. I want to avoid maintaining >200 SPEC files manually, and therefore came up with the SPEC file generation approach. Of course modifications may be necessary for some packages, but so far the number of package-specific changes is surprisingly small. 3. For me, the approach to put ROS packages in a separate directory (e.g., /usr/lib<qual>/ros) seemed like a better idea for now, because of missing SO versions and because it seems like this is still expected by ROS in many places. As an example it's not possible to provide separate install paths for the different target types such as binaries, libraries, devel files and so on. Additionally, in my understanding, packaging ROS in /usr/lib<qual>/ros is not prohibited by the packaging guidelines. Also, even if we package ROS in /usr/lib<qual>/ros, it provides several benefits over a local build in /opt: It is much easier to set up, especially if you have to manage multiple robots, it provides automatic updates, you can easily select and un-select packages that you want to have installed, you don't need to build locally, and there are probably many more reasons. In fact, I started packaging ROS because I was annoyed by managing 6 ROS installations on our robots, and only afterwards realized that the approach may be useful for distro-wide packages.
Anyway, I'd like to have a closer look at the existing packages, because I assume that they carry some patches that may still be useful, even when generating SPEC files.
As it stands today, a lot of ROS dependencies are packaged in Fedora, making it possible to build from source pretty easily. I think that to move forward safely, you need to decide for the long term whether you want to support:
- ROS installation via OSRF resources
- ROS installation via copr
- ROS packages in the distribution proper
Supporting ROS as a copr gives you a lot of leeway to do things that don't conform to the packaging guidelines, such as installing parallel stacks to /opt, creating single rpms for each package, etc. To bring the packages into fedora proper, as Tom was getting at, you need to reconcile how to support multiple software stacks within the package guidelines (and possibly get some exemptions to the guidelines,) and how to auto-generate specs that meet the packaging guidelines (proper division of -devel packages, etc.)
In the long run, my goal is to provide proper distribution packages for ROS, with COPR as a playing ground for development and testing. Splitting devel files into a sub-package should be fairly easy. Since the packaging guidelines regarding bundled libraries changed recently (last year?) and now allow bundled libraries as long as the package provides the respective bundled(library)s, unbundling all libraries is also not necessary anymore. Finding out all bundled libraries may result in a lot package-specific work, but at least we have an easy way to add these to the generated SPEC files. I think these are the main problems regarding the packaging guidelines.
In case that wasn't enough to worry about, there is also a "Modularity" effort going on in Fedora. I have not been able to figure out what they're doing, but [1] claims that it's modules can have different lifecycles than the base distribution. It may provide a model to maintain multiple ROS stacks in parallel, but I don't really have any more information than that.
I think modularity only helps us, and we could just decide to ignore it for now. In my understanding, it could provide us an easy way to support multiple ROS releases for the same Fedora release, e.g., having both kinetic and lunar modules for F26 without the hack of having the version in the name. That would mean that there is no ros-kinetic-desktop package, but instead two ros-desktop packages in separate modules.
I'm happy to add you to the existing ROS packages that have been reviewed, add you to the robotics-sig user group, and continue to discuss on the mailing list.
Yes, that would be nice as it would already give me more insight in the current status of ROS in Fedora. Please also add me to the user group.
I can grant access to the ROS pagure and my ROS COPR to anyone interested.
As a side note, I've recently been poking at ros2 packaging[2]. The ecosystem is a lot smaller at the moment, and it may be able to re-use some of the concepts and tooling that get worked out for ros 1.
I was totally unaware of ros2, thanks for the hint!
Kind regards, Till
robotics@lists.fedoraproject.org