Hello,
I would like to suggest a great addition to the EPEL, the z Systems architecture (s390x). I have seem a great number of requests around EPEL for z Systems, its definitely a growing platform around Linux as IBM keeps offering better and more efficient servers that are basically a datacenter in a box - which is a great way for customers to reduce costs by consolidating Linux workloads. EPEL repository to s390x can be a great way to allow customers to try an unsupported package that still not available to RHEL on z Systems - that can in time mature, and can skip some of the processes to make into the RHEL for z Systems distro.
The idea is to offer more technology options (Fedora Packages) to customers using RHEL on z Systems - this can flag us what packages are customers really interested into for RHEL on z Systems that are still not available in the Enterprise distro. Please let’s brainstorm this suggestion here, IBMers, community members, Fedora members, RedHatter, you are all welcome to join this discussion.
Thank you,
Kind Regards,
Filipe Miranda fmiranda@redhat.com Global Lead for Red Hat Products on IBM z Systems and Power Systems GPG Info: DF13EF21 (C79D 437C D56C ABC0 6878 D4D6 8BD5 2B72 DF13 EF21) RHC{E,DS,A,VA} Red Hat Inc. M. 949 572 3463 W. 650 254 4170
On 23 June 2015 at 10:29, Filipe Miranda fmiranda@redhat.com wrote:
Hello,
I would like to suggest a great addition to the EPEL, the z Systems architecture (s390x). I have seem a great number of requests around EPEL for z Systems, its definitely a growing platform around Linux as IBM keeps offering better and more efficient servers that are basically a datacenter in a box - which is a great way for customers to reduce costs by consolidating Linux workloads. EPEL repository to s390x can be a great way to allow customers to try an unsupported package that still not available to RHEL on z Systems - that can in time mature, and can skip some of the processes to make into the RHEL for z Systems distro.
The idea is to offer more technology options (Fedora Packages) to customers using RHEL on z Systems - this can flag us what packages are customers really interested into for RHEL on z Systems that are still not available in the Enterprise distro. Please let’s brainstorm this suggestion here, IBMers, community members, Fedora members, RedHatter, you are all welcome to join this discussion.
1) How would such items be built? The rest of EPEL is built with our builders in PHX2. We don't have an s390 system in PHX2. Build engineering requires access and the ability to 'rebuild' builders when needed. EPEL has no 2) How would developers test these builds? Much of the pushback we have in EPEL for other architectures is that developers have no systems which they can test to see why something is broken or not. This means we end up with a large amount of spec files with ExcludeArch: ppc etc etc. The lack of hardware and cost-free software (eg CentOS port) pretty much make any build problem get the hammer of ExcludeArch: 3) There isn't really a way to tell you which packages customers are interested in for two reasons: a) If we build packages for an arch, then every EPEL package gets built for it (except those with the ExcludeArch hammer) b) User logs of yum does not show which package is requested... it will show an estimate of users which for PPC is mainly engineers inside of RH and IBM and a handful of sites. From talking with people using PPC EPEL, they mirror it internally and no usage is seen.
Thank you,
Kind Regards,
Filipe Miranda fmiranda@redhat.com Global Lead for Red Hat Products on IBM z Systems and Power Systems GPG Info: DF13EF21 (C79D 437C D56C ABC0 6878 D4D6 8BD5 2B72 DF13 EF21) RHC{E,DS,A,VA} Red Hat Inc. M. 949 572 3463 W. 650 254 4170
epel-devel mailing list epel-devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/epel-devel
On Jun 23, 2015, at 1:17 PM, Stephen John Smoogen smooge@gmail.com wrote:
On 23 June 2015 at 10:29, Filipe Miranda fmiranda@redhat.com wrote:
Hello,
I would like to suggest a great addition to the EPEL, the z Systems architecture (s390x). I have seem a great number of requests around EPEL for z Systems, its definitely a growing platform around Linux as IBM keeps offering better and more efficient servers that are basically a datacenter in a box - which is a great way for customers to reduce costs by consolidating Linux workloads. EPEL repository to s390x can be a great way to allow customers to try an unsupported package that still not available to RHEL on z Systems - that can in time mature, and can skip some of the processes to make into the RHEL for z Systems distro.
The idea is to offer more technology options (Fedora Packages) to customers using RHEL on z Systems - this can flag us what packages are customers really interested into for RHEL on z Systems that are still not available in the Enterprise distro. Please let’s brainstorm this suggestion here, IBMers, community members, Fedora members, RedHatter, you are all welcome to join this discussion.
- How would such items be built? The rest of EPEL is built with our
builders in PHX2. We don't have an s390 system in PHX2. Build engineering requires access and the ability to 'rebuild' builders when needed. EPEL has no
We (Red Hat) have a z Systems server in Westford, and the Fedora for s390x is built over there, maybe we could use the same environment (same LPAR, just add a few VMs with Fedora to do so) maybe Dan Horak can help sort his out.
- How would developers test these builds? Much of the pushback we
have in EPEL for other architectures is that developers have no systems which they can test to see why something is broken or not. This means we end up with a large amount of spec files with ExcludeArch: ppc etc etc. The lack of hardware and cost-free software (eg CentOS port) pretty much make any build problem get the hammer of ExcludeArch)
This can be addressed maybe by IBM, by offering build systems to help developers test their packages.
- There isn't really a way to tell you which packages customers are
interested in for two reasons: a) If we build packages for an arch, then every EPEL package gets built for it (except those with the ExcludeArch hammer)
The idea is to start with what IBM thinks it will be good to have there, then if the project gets traction, we can have feedback from customers using bugzilla.
b) User logs of yum does not show which package is requested... it will show an estimate of users which for PPC is mainly engineers inside of RH and IBM and a handful of sites. From talking with people using PPC EPEL, they mirror it internally and no usage is seen.
Thank you for sharing these thoughts Stephen, let’s see what others have to say about it,
Thank you,
Kind Regards,
Filipe Miranda fmiranda@redhat.com Global Lead for Red Hat Products on IBM z Systems and Power Systems GPG Info: DF13EF21 (C79D 437C D56C ABC0 6878 D4D6 8BD5 2B72 DF13 EF21) RHC{E,DS,A,VA} Red Hat Inc. M. 949 572 3463 W. 650 254 4170
epel-devel mailing list epel-devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/epel-devel
-- Stephen J Smoogen. _______________________________________________ epel-devel mailing list epel-devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/epel-devel
On Tue, 23 Jun 2015 13:52:41 -0400 Filipe Miranda fmiranda@redhat.com wrote:
On Jun 23, 2015, at 1:17 PM, Stephen John Smoogen smooge@gmail.com wrote:
On 23 June 2015 at 10:29, Filipe Miranda fmiranda@redhat.com wrote:
Hello,
I would like to suggest a great addition to the EPEL, the z Systems architecture (s390x). I have seem a great number of requests around EPEL for z Systems, its definitely a growing platform around Linux as IBM keeps offering better and more efficient servers that are basically a datacenter in a box - which is a great way for customers to reduce costs by consolidating Linux workloads. EPEL repository to s390x can be a great way to allow customers to try an unsupported package that still not available to RHEL on z Systems - that can in time mature, and can skip some of the processes to make into the RHEL for z Systems distro.
The idea is to offer more technology options (Fedora Packages) to customers using RHEL on z Systems - this can flag us what packages are customers really interested into for RHEL on z Systems that are still not available in the Enterprise distro. Please let’s brainstorm this suggestion here, IBMers, community members, Fedora members, RedHatter, you are all welcome to join this discussion.
- How would such items be built? The rest of EPEL is built with our
builders in PHX2. We don't have an s390 system in PHX2. Build engineering requires access and the ability to 'rebuild' builders when needed. EPEL has no
We (Red Hat) have a z Systems server in Westford, and the Fedora for s390x is built over there, maybe we could use the same environment (same LPAR, just add a few VMs with Fedora to do so) maybe Dan Horak can help sort his out.
yes, technically Koji allows the builders to be anywhere on the Internet and with some caching there shouldn't any visible slowdown when compared to local builders. This is how Fedora/s390x infrastructure is designed.
- How would developers test these builds? Much of the pushback we
have in EPEL for other architectures is that developers have no systems which they can test to see why something is broken or not. This means we end up with a large amount of spec files with ExcludeArch: ppc etc etc. The lack of hardware and cost-free software (eg CentOS port) pretty much make any build problem get the hammer of ExcludeArch)
This can be addressed maybe by IBM, by offering build systems to help developers test their packages.
Ack, that's something for IBM, from Red Hat side it would require providing RHEL for System z subscriptions to such devel systems. Currently we have one public guest running Fedora available to the community, so it should be solvable. In addition to real HW there are 2 solutions capable running current Linux distributions under emulation.
From the quality point of view we are able to build 90+ percent of 16k+
packages in recent Fedora, so I don't expect many new issues related to running in EL environment. But I can be wrong and it is also a place where IBM engineers can step in :-) I would rather expect runtime issue than build time ones.
- There isn't really a way to tell you which packages customers are
interested in for two reasons: a) If we build packages for an arch, then every EPEL package gets built for it (except those with the ExcludeArch hammer)
The idea is to start with what IBM thinks it will be good to have there, then if the project gets traction, we can have feedback from customers using bugzilla.
b) User logs of yum does not show which package is requested... it will show an estimate of users which for PPC is mainly engineers inside of RH and IBM and a handful of sites. From talking with people using PPC EPEL, they mirror it internally and no usage is seen.
Thank you for sharing these thoughts Stephen, let’s see what others have to say about it,
There are other questions like do we know that current content of EPEL will build, etc.
And I also suppose we are talking about EPEL-7 (and up).
Dan
Thanks Filipe for initiating the discussion.
Dan Horák dan@danny.cz wrote on 2015-06-23 02:31:23 PM:
On Tue, 23 Jun 2015 13:52:41 -0400 Filipe Miranda fmiranda@redhat.com wrote:
This can be addressed maybe by IBM, by offering build systems to help developers test their packages.
Ack, that's something for IBM, from Red Hat side it would require providing RHEL for System z subscriptions to such devel systems. Currently we have one public guest running Fedora available to the community, so it should be solvable. In addition to real HW there are 2 solutions capable running current Linux distributions under emulation.
Dan, could you elaborate on the emulation aspect? Do you mean IBM zPDT and Hercules? I am curious if you are using emulation in the build farm today.
IBM is currently engaging open-source software companies to encourage support for the platform. IBM partners can essentially get access to hardware for development purposes at a discount.
For the community, we have a program called Community Development System for Linux on z, whereby open source projects can sign up for free access to Linux guests on our z Systems (for a limited time):
http://www-03.ibm.com/systems/z/os/linux/support/community.html
During registration, you can request a RHEL installation. I am not sure whether it comes with a RHN subscription.
Other community hardware access options are now being considered, and I hope something more streamlined can be announced soon.
From the quality point of view we are able to build 90+ percent of 16k+ packages in recent Fedora, so I don't expect many new issues related to running in EL environment. But I can be wrong and it is also a place where IBM engineers can step in :-) I would rather expect runtime issue than build time ones.
I would imagine that, when EPEL RPMs are built on z for the first time, a small percentage of them could be broken. We have some experience in porting and building for z, and can certainly assist the package maintainers with debugging build-time problems.
a) If we build packages for an arch, then every EPEL package gets built for it (except those with the ExcludeArch hammer)
The idea is to start with what IBM thinks it will be good to have there, then if the project gets traction, we can have feedback from customers using bugzilla.
Unless build farm resource is a concern, I actually like the idea of building all EPEL packages at once. Like Dan said, there probably won't be many build issues. But yes, we could also supply a list of the more important packages, to seed the repository.
There are other questions like do we know that current content of EPEL will build, etc.
And I also suppose we are talking about EPEL-7 (and up).
EL7 and up is a good start, although it would be really nice to support EL6 as well; many people still have production systems running RHEL 6.
Regards, Bryan -- Bryan Chan bryan.chan@ca.ibm.com
Excerpts from Bryan Chan's message of 2015-06-24 08:29 +10:00:
Dan Horák dan@danny.cz wrote on 2015-06-23 02:31:23 PM:
On Tue, 23 Jun 2015 13:52:41 -0400 Filipe Miranda fmiranda@redhat.com wrote:
This can be addressed maybe by IBM, by offering build systems to help developers test their packages.
Ack, that's something for IBM, from Red Hat side it would require providing RHEL for System z subscriptions to such devel systems. Currently we have one public guest running Fedora available to the community, so it should be solvable. In addition to real HW there are 2 solutions capable running current Linux distributions under emulation.
Dan, could you elaborate on the emulation aspect? Do you mean IBM zPDT and Hercules? I am curious if you are using emulation in the build farm today.
IBM is currently engaging open-source software companies to encourage support for the platform. IBM partners can essentially get access to hardware for development purposes at a discount.
For the community, we have a program called Community Development System for Linux on z, whereby open source projects can sign up for free access to Linux guests on our z Systems (for a limited time):
http://www-03.ibm.com/systems/z/os/linux/support/community.html
During registration, you can request a RHEL installation. I am not sure whether it comes with a RHN subscription.
Other community hardware access options are now being considered, and I hope something more streamlined can be announced soon.
One possibility is to make some z/VM guests available to approved Fedora contributors using beaker.fedoraproject.org, the same way that Red Hat engineers can use our internal Beaker instance to reserve z/VM guests.
I would love to see the arches that people don't have at home, like ppc64 and aarch64, on beaker.fedoraproject.org too.
The only big unresolved issue with beaker.fedoraproject.org right now is how to hook up FAS authentication. I haven't had a chance to figure that out yet.
On Wednesday, June 24, 2015 09:11:17 AM Dan Callaghan wrote:
Excerpts from Bryan Chan's message of 2015-06-24 08:29 +10:00:
Dan Horák dan@danny.cz wrote on 2015-06-23 02:31:23 PM:
On Tue, 23 Jun 2015 13:52:41 -0400
Filipe Miranda fmiranda@redhat.com wrote:
This can be addressed maybe by IBM, by offering build systems to help developers test their packages.
Ack, that's something for IBM, from Red Hat side it would require providing RHEL for System z subscriptions to such devel systems. Currently we have one public guest running Fedora available to the community, so it should be solvable. In addition to real HW there are 2 solutions capable running current Linux distributions under emulation.
Dan, could you elaborate on the emulation aspect? Do you mean IBM zPDT and Hercules? I am curious if you are using emulation in the build farm today.
IBM is currently engaging open-source software companies to encourage support for the platform. IBM partners can essentially get access to hardware for development purposes at a discount.
For the community, we have a program called Community Development System for Linux on z, whereby open source projects can sign up for free access to Linux guests on our z Systems (for a limited time):
http://www-03.ibm.com/systems/z/os/linux/support/community.html
During registration, you can request a RHEL installation. I am not sure whether it comes with a RHN subscription.
Other community hardware access options are now being considered, and I hope something more streamlined can be announced soon.
One possibility is to make some z/VM guests available to approved Fedora contributors using beaker.fedoraproject.org, the same way that Red Hat engineers can use our internal Beaker instance to reserve z/VM guests.
I would love to see the arches that people don't have at home, like ppc64 and aarch64, on beaker.fedoraproject.org too.
The only big unresolved issue with beaker.fedoraproject.org right now is how to hook up FAS authentication. I haven't had a chance to figure that out yet.
A bigger issue would be having a z series machine that can be tied into fedora's beaker instance. Considering that there is none in Fedora space today. I do not imagine IT or Red Hat Security allowing some tunnel into the systems inside of Red Hat. Assuming we can find hardware then sure would be great.
Dennis
On Tue, 23 Jun 2015 18:29:06 -0400 Bryan Chan bryan.chan@ca.ibm.com wrote:
Thanks Filipe for initiating the discussion.
Dan Horák dan@danny.cz wrote on 2015-06-23 02:31:23 PM:
On Tue, 23 Jun 2015 13:52:41 -0400 Filipe Miranda fmiranda@redhat.com wrote:
This can be addressed maybe by IBM, by offering build systems to help developers test their packages.
Ack, that's something for IBM, from Red Hat side it would require providing RHEL for System z subscriptions to such devel systems. Currently we have one public guest running Fedora available to the community, so it should be solvable. In addition to real HW there are 2 solutions capable running current Linux distributions under emulation.
Dan, could you elaborate on the emulation aspect? Do you mean IBM zPDT and Hercules? I am curious if you are using emulation in the build farm today.
yeah, I mean zPDT and Hercules. I haven't tried F-22 on Hercules yet, but F-21 worked fine there. And I suppose zPDT won't have issues as well. For the Fedora builds we use native builders on a LPAR from the Red Hat zEC12.
IBM is currently engaging open-source software companies to encourage support for the platform. IBM partners can essentially get access to hardware for development purposes at a discount.
For the community, we have a program called Community Development System for Linux on z, whereby open source projects can sign up for free access to Linux guests on our z Systems (for a limited time):
http://www-03.ibm.com/systems/z/os/linux/support/community.html
During registration, you can request a RHEL installation. I am not sure whether it comes with a RHN subscription.
Other community hardware access options are now being considered, and I hope something more streamlined can be announced soon.
That sounds promising, the IBM Power team is promoting the architecture on the OpenPOWER hubs (now 4 places across the world) where open source projects can get their own guests to port/fix their projects.
From the quality point of view we are able to build 90+ percent of 16k+ packages in recent Fedora, so I don't expect many new issues related to running in EL environment. But I can be wrong and it is also a place where IBM engineers can step in :-) I would rather expect runtime issue than build time ones.
I would imagine that, when EPEL RPMs are built on z for the first time, a small percentage of them could be broken. We have some experience in porting and building for z, and can certainly assist the package maintainers with debugging build-time problems.
a) If we build packages for an arch, then every EPEL package gets built for it (except those with the ExcludeArch hammer)
The idea is to start with what IBM thinks it will be good to have there, then if the project gets traction, we can have feedback from customers using bugzilla.
Unless build farm resource is a concern, I actually like the idea of building all EPEL packages at once. Like Dan said, there probably won't be many build issues. But yes, we could also supply a list of the more important packages, to seed the repository.
One of the first steps towards EPEL should taking the current source RPMs and start building them using mock (which is one level below the koji build system for proper EPEL) and publish the results. This step is important even when there won't be an agreement to merge s390x into EPEL, because the other options like "secondary EPEL", where the builds will cloned in similar way as being done for Fedora, or maybe even using COPR [1] for the builds will profit from it. Or rather it is the necessary first step for any solution.
There are other questions like do we know that current content of EPEL will build, etc.
And I also suppose we are talking about EPEL-7 (and up).
EL7 and up is a good start, although it would be really nice to support EL6 as well; many people still have production systems running RHEL 6.
yes, I can see the importance of EPEL-6, but lets start with something simpler :-)
[1] https://copr.fedoraproject.org/
Dan
Dan Horák dan@danny.cz wrote on 2015-06-24 08:53:24 AM:
yeah, I mean zPDT and Hercules. I haven't tried F-22 on Hercules yet, but F-21 worked fine there. And I suppose zPDT won't have issues as well. For the Fedora builds we use native builders on a LPAR from the Red Hat zEC12.
Good to hear. These are certainly possibilities if there isn't enough native capacity.
One of the first steps towards EPEL should taking the current source RPMs and start building them using mock (which is one level below the koji build system for proper EPEL) and publish the results. This step is important even when there won't be an agreement to merge s390x into EPEL, because the other options like "secondary EPEL", where the builds will cloned in similar way as being done for Fedora, or maybe even using COPR [1] for the builds will profit from it. Or rather it is the necessary first step for any solution.
I have set up mock on one of my z test systems, but I ran into a snag; I couldn't get a working mock configuration for epel-7-s390x. I tried copying epel-7-x86_64.cfg and changing the repo URLs, but it failed to set up the chroot jail because it couldn't find @buildsys-build. I also tried to change the chroot_setup_cmd to use another package group; no luck either. Could you help me out on that front?
Could you elaborate on "secondary EPEL" and cloning builds? I am not familiar with those concepts.
Bryan
Hi,
I would like to suggest a great addition to the EPEL, the z Systems architecture (s390x). I have seem a great number of requests around EPEL for z Systems, its definitely a growing platform around Linux as IBM keeps offering better and more efficient servers that are basically a datacenter in a box - which is a great way for customers to reduce costs by consolidating Linux workloads. EPEL repository to s390x can be a great way to allow customers to try an unsupported package that still not available to RHEL on z Systems - that can in time mature, and can skip some of the processes to make into the RHEL for z Systems distro.
Thanks for the sales pitch... please check your sales hat at the EPEL door :)
Where is the "great number of requests" for EPEL for system-Z? Can you quantify the demand?
The idea is to offer more technology options (Fedora Packages) to customers using RHEL on z Systems - this can flag us what packages are customers really interested into for RHEL on z Systems that are still not available in the Enterprise distro. Please let’s brainstorm this suggestion here, IBMers, community members, Fedora members, RedHatter, you are all welcome to join this discussion.
So the concerns I have regarding this that I've not really yet seen addressed on this thread are: 1) who's going to do this work? Setting up a new EPEL architecture isn't an insignificant amount of work (like a new arch in Fedora). There's an expectation of people to be around to assist in problems with package builds etc 2) even with the free emulators mentioned in the thread how would people test this on those. I don't believe there's CentOS for 390x yet 3) what RHEL releases do you expect to support?
Peter
epel-devel@lists.fedoraproject.org
-
Bryan Chan -
Dan Callaghan -
Dan Horák -
Dennis Gilmore -
Filipe Miranda -
Peter Robinson -
Stephen John Smoogen