Hello Fedora EPEL maintainers!
First I don't feel comfortable announcing this, I'm not happy about the situation and so I don't want to be the lightning rod :-). But I believe that we can come to acceptable Copr/Mock solution and this needs to be discussed... so here we are.
By the end of the year 2021 we have to fix our default EPEL 8 Mock configuration (mock-core-configs.rpm, /etc/mock/epel-8-*.cfg) as CentOS 8 goes EOL by then.
The same thing needs to happen in Fedora Copr, with the epel-8-* chroots (side note, in Fedora Copr we use the mock-core-configs package for builds without any deployment specific modifications).
I am proposing (as PR against mock upstream ATM [1]) to switch the default epel-* configuration from CentOS+EPEL to RHEL+EPEL as soon as possible (see the pull request [1]).
This would bring some consequences, namely newly with epel-8* chroots,
- builds will require a valid Red Hat subscription (the no-cost variant is OK as well, though [2]) - we will miss some build-time packages that Red Hat is not shipping, at least at the beginning till they are added (to RHEL CRB, or other currently unknown place). - cross-arch compilation can not be used, Red Hat subscriptions don't allow that (using QEMU and rpm --forcearch), [3]
The positive thing is that the default configuration will be much closer to the official EPEL builds (because Fedora Koji EPEL builds are actually done also against RHEL).
For the Fedora Copr builders, we already have the necessary Red Hat subscriptions in hand (will be deployed by the end of the year). So we will only loose the opportunity to build on emulated epel-8-armhfp permanently, and epel-8-s390x temporarily (as we already work on the native s390x support).
Any thoughts? Feedback is needed here.
[1] https://github.com/rpm-software-management/mock/pull/802 [2] https://rpm-software-management.github.io/mock/Feature-rhelchroots [3] https://bugzilla.redhat.com/show_bug.cgi?id=1912847
Pavel
On 22. 11. 21 15:00, Pavel Raiskup wrote:
- builds will require a valid Red Hat subscription (the no-cost variant is OK as well, though [2])
I cannot help myself but I consider this very unpleasant for EPEL packagers.
Getting and configuring the subscription was always so unfriendly for me that I've been using EPEL mocks even for my RHEL work. This basically means using EPEL mocks will once again be as complicated as using RHEL.
However, enough of my personal views. Since we have not used RHEL for copr/mock EPEL buidlroots until now, but we used a downstream freely-available RHEL-copy (CentOS Linux), could we not continue doing so by using e.g. AlmaLinux?
Dne 22. 11. 21 v 15:10 Miro Hrončok napsal(a):
However, enough of my personal views. Since we have not used RHEL for copr/mock EPEL buidlroots until now, but we used a downstream freely-available RHEL-copy (CentOS Linux), could we not continue doing so by using e.g. AlmaLinux?
For day to day work I would suggest to move to centos-stream + epel-next (hmm, we do not have a config for that).
But EPEL is built against RHEL (not Alma, not Rocky). So we either use default config which will differ from Koji or we have to fiddle with entitlements:
https://rpm-software-management.github.io/mock/Feature-rhelchroots
https://developers.redhat.com/blog/2021/02/10/how-to-activate-your-no-cost-r...
Miroslav
On 22. 11. 21 15:25, Miroslav Suchý wrote:
But EPEL is built against RHEL (not Alma, not Rocky).
True. As well as it is true today that it is not built against CentOS Linux (and yet we do that in mock).
On Mon, Nov 22, 2021 at 8:27 AM Miroslav Suchý msuchy@redhat.com wrote:
Dne 22. 11. 21 v 15:10 Miro Hrončok napsal(a):
However, enough of my personal views. Since we have not used RHEL for copr/mock EPEL buidlroots until now, but we used a downstream freely-available RHEL-copy (CentOS Linux), could we not continue doing so by using e.g. AlmaLinux?
For day to day work I would suggest to move to centos-stream + epel-next (hmm, we do not have a config for that).
But EPEL is built against RHEL (not Alma, not Rocky). So we either use default config which will differ from Koji or we have to fiddle with entitlements:
https://rpm-software-management.github.io/mock/Feature-rhelchroots
https://developers.redhat.com/blog/2021/02/10/how-to-activate-your-no-cost-r...
Miroslav _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Agreed, for quick local builds it's fine to use epel8-next (c8s) to verify it builds and then submit the koji build to the epel8 (rhel8) target. If the local build works but the koji build doesn't, you likely have a candidate for an official epel8-next koji build.
On Mon, Nov 22, 2021 at 8:11 AM Miro Hrončok mhroncok@redhat.com wrote:
On 22. 11. 21 15:00, Pavel Raiskup wrote:
- builds will require a valid Red Hat subscription (the no-cost variant is OK as well, though [2])
I cannot help myself but I consider this very unpleasant for EPEL packagers.
Getting and configuring the subscription was always so unfriendly for me that I've been using EPEL mocks even for my RHEL work. This basically means using EPEL mocks will once again be as complicated as using RHEL.
However, enough of my personal views. Since we have not used RHEL for copr/mock EPEL buidlroots until now, but we used a downstream freely-available RHEL-copy (CentOS Linux), could we not continue doing so by using e.g. AlmaLinux?
-- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
I'm not aware of a RHEL clone that offers all the architectures that EPEL does. As far as I can tell, the three most popular (Alma, Rocky, Oracle) only offer x86_64 and aarch64 but are missing ppc64le and s390x. That said CentOS Linux 8 doesn't offer s390x either, so we already have this problem, but switch the EPEL mock chroot to one of those clones would make the situation worse by also dropping ppc64le.
On Mon, Nov 22, 2021 at 11:55 AM Carl George carl@redhat.com wrote:
On Mon, Nov 22, 2021 at 8:11 AM Miro Hrončok mhroncok@redhat.com wrote:
On 22. 11. 21 15:00, Pavel Raiskup wrote:
- builds will require a valid Red Hat subscription (the no-cost variant is OK as well, though [2])
I cannot help myself but I consider this very unpleasant for EPEL packagers.
Getting and configuring the subscription was always so unfriendly for me that I've been using EPEL mocks even for my RHEL work. This basically means using EPEL mocks will once again be as complicated as using RHEL.
However, enough of my personal views. Since we have not used RHEL for copr/mock EPEL buidlroots until now, but we used a downstream freely-available RHEL-copy (CentOS Linux), could we not continue doing so by using e.g. AlmaLinux?
-- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
I'm not aware of a RHEL clone that offers all the architectures that EPEL does. As far as I can tell, the three most popular (Alma, Rocky, Oracle) only offer x86_64 and aarch64 but are missing ppc64le and s390x. That said CentOS Linux 8 doesn't offer s390x either, so we already have this problem, but switch the EPEL mock chroot to one of those clones would make the situation worse by also dropping ppc64le.
I've been informed that AlmaLinux is working on ppc64le support, and it's supposed to be released "soon". Perhaps Jack can provide an update here?
Le 22/11/2021 à 15:52, Carl George a écrit :
On Mon, Nov 22, 2021 at 8:11 AM Miro Hrončok mhroncok@redhat.com wrote:
On 22. 11. 21 15:00, Pavel Raiskup wrote:
- builds will require a valid Red Hat subscription (the no-cost variant is OK as well, though [2])
I cannot help myself but I consider this very unpleasant for EPEL packagers.
Getting and configuring the subscription was always so unfriendly for me that I've been using EPEL mocks even for my RHEL work. This basically means using EPEL mocks will once again be as complicated as using RHEL.
However, enough of my personal views. Since we have not used RHEL for copr/mock EPEL buidlroots until now, but we used a downstream freely-available RHEL-copy (CentOS Linux), could we not continue doing so by using e.g. AlmaLinux?
-- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok _______________________________________________
I'm not aware of a RHEL clone that offers all the architectures that EPEL does. As far as I can tell, the three most popular (Alma, Rocky, Oracle) only offer x86_64 and aarch64 but are missing ppc64le and s390x. That said CentOS Linux 8 doesn't offer s390x either, so we already have this problem, but switch the EPEL mock chroot to one of those clones would make the situation worse by also dropping ppc64le.
The success of COPR is based on it's easy way to build packages, especially for EPEL, do whatever you want guys but please at the end, try to not choose the bullshito way.
Le 22/11/2021 à 15:52, Carl George a écrit :
On Mon, Nov 22, 2021 at 8:11 AM Miro Hrončok mhroncok@redhat.com wrote:
On 22. 11. 21 15:00, Pavel Raiskup wrote:
- builds will require a valid Red Hat subscription (the no-cost
variant is OK as well, though [2])
I cannot help myself but I consider this very unpleasant for EPEL packagers.
Getting and configuring the subscription was always so unfriendly for me that I've been using EPEL mocks even for my RHEL work. This basically means using EPEL mocks will once again be as complicated as using RHEL.
However, enough of my personal views. Since we have not used RHEL for copr/mock EPEL buidlroots until now, but we used a downstream freely-available RHEL-copy (CentOS Linux), could we not continue doing so by using e.g. AlmaLinux?
-- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok _______________________________________________
I'm not aware of a RHEL clone that offers all the architectures that EPEL does. As far as I can tell, the three most popular (Alma, Rocky, Oracle) only offer x86_64 and aarch64 but are missing ppc64le and s390x. That said CentOS Linux 8 doesn't offer s390x either, so we already have this problem, but switch the EPEL mock chroot to one of those clones would make the situation worse by also dropping ppc64le.
The success of COPR is based on it's easy way to build packages, especially for EPEL, do whatever you want guys but please at the end, try to not choose the bullshito way.
On Mon, Nov 22, 2021 at 03:10:58PM +0100, Miro Hrončok wrote:
- builds will require a valid Red Hat subscription (the no-cost variant is OK as well, though [2])
I cannot help myself but I consider this very unpleasant for EPEL packagers.
Getting and configuring the subscription was always so unfriendly for me that I've been using EPEL mocks even for my RHEL work. This basically means using EPEL mocks will once again be as complicated as using RHEL.
I'm not opposed to working with our friends over at Alma on this, but I think we _also_ should fix this. Would it help if we made it easy to add a RHEL developer entitlement (the 16 no-cost RHEL license thing) to a Fedora Account (perhaps with a check-box agreement in the account system, like the FPCA), and made it trivial in mock use that?
I mean, seriously, RH should make this easy for Fedora packagers.
On Mon, Nov 22, 2021 at 9:01 AM Pavel Raiskup praiskup@redhat.com wrote:
Hello Fedora EPEL maintainers!
First I don't feel comfortable announcing this, I'm not happy about the situation and so I don't want to be the lightning rod :-). But I believe that we can come to acceptable Copr/Mock solution and this needs to be discussed... so here we are.
By the end of the year 2021 we have to fix our default EPEL 8 Mock configuration (mock-core-configs.rpm, /etc/mock/epel-8-*.cfg) as CentOS 8 goes EOL by then.
The same thing needs to happen in Fedora Copr, with the epel-8-* chroots (side note, in Fedora Copr we use the mock-core-configs package for builds without any deployment specific modifications).
I am proposing (as PR against mock upstream ATM [1]) to switch the default epel-* configuration from CentOS+EPEL to RHEL+EPEL as soon as possible (see the pull request [1]).
This would seem to be an exceptionally bad idea. Red Hat elected to force this CentOS Stream release structure down everyone's throats unannounced, compelling default mock behavior to use the stream is a logical consequence. Trying to outsmart Red Hat on this is going to break EPEL dependencies and keep breaking EPEL for Stream deployments.
This would bring some consequences, namely newly with epel-8* chroots,
- builds will require a valid Red Hat subscription (the no-cost variant is OK as well, though [2])
- we will miss some build-time packages that Red Hat is not shipping, at least at the beginning till they are added (to RHEL CRB, or other currently unknown place).
- cross-arch compilation can not be used, Red Hat subscriptions don't allow that (using QEMU and rpm --forcearch), [3]
Which is precisely why pointing it to the 'stream' release seems the only workable solution.
The positive thing is that the default configuration will be much closer to the official EPEL builds (because Fedora Koji EPEL builds are actually done also against RHEL).
That does seem desirable. Frankly, I don't expect this mid-stream pivot to last more than 2 years: it was tried before with Red Hat 9, and discarded thoroughly for RHEL 2.1. Point releases are too useful for environments that cannot afford CI/CD with the underlying operating system, and don't want to spend the time re-inventing point releases themselves.
For the Fedora Copr builders, we already have the necessary Red Hat subscriptions in hand (will be deployed by the end of the year). So we will only loose the opportunity to build on emulated epel-8-armhfp permanently, and epel-8-s390x temporarily (as we already work on the native s390x support).
Any thoughts? Feedback is needed here.
[1] https://github.com/rpm-software-management/mock/pull/802 [2] https://rpm-software-management.github.io/mock/Feature-rhelchroots [3] https://bugzilla.redhat.com/show_bug.cgi?id=1912847
Pavel
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Dne 22. 11. 21 v 17:57 Nico Kadel-Garcia napsal(a):
Which is precisely why pointing it to the 'stream' release seems the only workable solution.
That is EPEL-next
https://fedoraproject.org/wiki/EPEL_Next#Introduction
Miroslav
Dne 22. 11. 21 v 15:00 Pavel Raiskup napsal(a):
Hello Fedora EPEL maintainers!
First I don't feel comfortable announcing this, I'm not happy about the situation and so I don't want to be the lightning rod :-). But I believe that we can come to acceptable Copr/Mock solution and this needs to be discussed... so here we are.
By the end of the year 2021 we have to fix our default EPEL 8 Mock configuration (mock-core-configs.rpm, /etc/mock/epel-8-*.cfg) as CentOS 8 goes EOL by then.
I wrote down the possible options and their pros and cons and I done my best to catch all the feedback here.
https://docs.google.com/document/d/1wF7-7_y6Ac_oB-kCFdE6VBWPW8o8zjXd2Z0SGy4V...
Miroslav
On Thu, Nov 25, 2021 at 3:05 AM Miroslav Suchý msuchy@redhat.com wrote:
Dne 22. 11. 21 v 15:00 Pavel Raiskup napsal(a):
Hello Fedora EPEL maintainers!
First I don't feel comfortable announcing this, I'm not happy about the situation and so I don't want to be the lightning rod :-). But I believe that we can come to acceptable Copr/Mock solution and this needs to be discussed... so here we are.
By the end of the year 2021 we have to fix our default EPEL 8 Mock configuration (mock-core-configs.rpm, /etc/mock/epel-8-*.cfg) as CentOS 8 goes EOL by then.
I wrote down the possible options and their pros and cons and I done my best to catch all the feedback here.
https://docs.google.com/document/d/1wF7-7_y6Ac_oB-kCFdE6VBWPW8o8zjXd2Z0SGy4V...
Miroslav
That seems to be a succinct listing. I think you left out my suggestion.of "support people re-inventing point releases for CentOS", which is what major CentOS users will do using internal mirrors. due to concern about unexpected and unwelcome updates of CentOS Stream, while they assess whether AlmaLinux or Rocky are reliable and stable enough to use. It's not an uncommon behavior for EPEL itself, partly because of EPEL's bad habit of deleting RPMs without warning and stripping out all previous releases. That's caused me problems with chromium and firefox when updates were incompatible with contemporary regression testing systems.
The difficulty with switching mock to AlmaLinux or Rocky is that there is likely to be significant phase lag with new point releases by Red Hat, and that it will inflict quite a bandwidth burden for all the "mock" setups in the field. Can they scale up to handle that?
On Thu, Nov 25, 2021 at 6:19 AM Nico Kadel-Garcia nkadel@gmail.com wrote:
On Thu, Nov 25, 2021 at 3:05 AM Miroslav Suchý msuchy@redhat.com wrote:
Dne 22. 11. 21 v 15:00 Pavel Raiskup napsal(a):
Hello Fedora EPEL maintainers!
First I don't feel comfortable announcing this, I'm not happy about the situation and so I don't want to be the lightning rod :-). But I believe that we can come to acceptable Copr/Mock solution and this needs to be discussed... so here we are.
By the end of the year 2021 we have to fix our default EPEL 8 Mock configuration (mock-core-configs.rpm, /etc/mock/epel-8-*.cfg) as CentOS 8 goes EOL by then.
I wrote down the possible options and their pros and cons and I done my best to catch all the feedback here.
https://docs.google.com/document/d/1wF7-7_y6Ac_oB-kCFdE6VBWPW8o8zjXd2Z0SGy4V...
Miroslav
That seems to be a succinct listing. I think you left out my suggestion.of "support people re-inventing point releases for CentOS", which is what major CentOS users will do using internal mirrors. due to concern about unexpected and unwelcome updates of CentOS Stream, while they assess whether AlmaLinux or Rocky are reliable and stable enough to use. It's not an uncommon behavior for EPEL itself, partly because of EPEL's bad habit of deleting RPMs without warning and stripping out all previous releases. That's caused me problems with chromium and firefox when updates were incompatible with contemporary regression testing systems.
It's not a "bad habit", it happens because when packages are retired, keeping the packages there does a disservice to the community by effectively forcing a maintenance burden when there's no maintainer. As for stripping out previous releases, that's just how Pungi and Bodhi do update composes at the moment. Someday that'll be fixed, but then we'd have to come up with a policy on how many because there are storage concerns for mirrors if we kept everything published forever.
The difficulty with switching mock to AlmaLinux or Rocky is that there is likely to be significant phase lag with new point releases by Red Hat, and that it will inflict quite a bandwidth burden for all the "mock" setups in the field. Can they scale up to handle that?
Insofar as "phase lag with new point releases", AlmaLinux made their release 48 hours after Red Hat did with RHEL. So, frankly, I'm not worried there with AlmaLinux.
For bandwidth burdens, mirror networks are designed to alleviate that burden and both have those in place.
-- 真実はいつも一つ！/ Always, there's only one truth!
On Thu, Nov 25, 2021 at 8:26 AM Neal Gompa ngompa13@gmail.com wrote:
On Thu, Nov 25, 2021 at 6:19 AM Nico Kadel-Garcia nkadel@gmail.com wrote:
On Thu, Nov 25, 2021 at 3:05 AM Miroslav Suchý msuchy@redhat.com wrote:
Dne 22. 11. 21 v 15:00 Pavel Raiskup napsal(a):
Hello Fedora EPEL maintainers!
First I don't feel comfortable announcing this, I'm not happy about the situation and so I don't want to be the lightning rod :-). But I believe that we can come to acceptable Copr/Mock solution and this needs to be discussed... so here we are.
By the end of the year 2021 we have to fix our default EPEL 8 Mock configuration (mock-core-configs.rpm, /etc/mock/epel-8-*.cfg) as CentOS 8 goes EOL by then.
I wrote down the possible options and their pros and cons and I done my best to catch all the feedback here.
https://docs.google.com/document/d/1wF7-7_y6Ac_oB-kCFdE6VBWPW8o8zjXd2Z0SGy4V...
Miroslav
That seems to be a succinct listing. I think you left out my suggestion.of "support people re-inventing point releases for CentOS", which is what major CentOS users will do using internal mirrors. due to concern about unexpected and unwelcome updates of CentOS Stream, while they assess whether AlmaLinux or Rocky are reliable and stable enough to use. It's not an uncommon behavior for EPEL itself, partly because of EPEL's bad habit of deleting RPMs without warning and stripping out all previous releases. That's caused me problems with chromium and firefox when updates were incompatible with contemporary regression testing systems.
It's not a "bad habit", it happens because when packages are retired, keeping the packages there does a disservice to the community by effectively forcing a maintenance burden when there's no maintainer. As for stripping out previous releases, that's just how Pungi and Bodhi do update composes at the moment. Someday that'll be fixed, but then we'd have to come up with a policy on how many because there are storage concerns for mirrors if we kept everything published forever.
It causes problems and confusion for people who need to lock down evisting versions for deployment. And it happens for packages that are not retired, but merely updated. I was bitten by it myself with chromium updates last year. It forces users of EPEL to maintain internal repos, or out of band access to previously accessible RPMs. It's destabilizing and breaks the use of bills-of-material based deployments with complete lists of all desired RPMs.
Storage and bandwidth concerns are legitimate concerns, as is potentially continuing to publish older releases with known vulnerabilities or bugs. But neither Fedora nor RHEL simply discard previously published versions this way, they aggregate new releases. I consider this a longstanding bug for EPEL, and one of the reasons I set up internal mirrors in large deployments.
The difficulty with switching mock to AlmaLinux or Rocky is that there is likely to be significant phase lag with new point releases by Red Hat, and that it will inflict quite a bandwidth burden for all the "mock" setups in the field. Can they scale up to handle that?
Insofar as "phase lag with new point releases", AlmaLinux made their release 48 hours after Red Hat did with RHEL. So, frankly, I'm not worried there with AlmaLinux.
48 hours is pretty danged good. I hesitate to rely on a new OS publisher with so little track record, even if they've been very good in their first year. I'm very curious how well they'll do with RHEL 8.6, without a published CentOS example to compare with. And I'd be very, very wary of the "planning fallacy", where people underestimate the time of future tasks, despite knowledge that previous tasks have sometimes taken far longer.
For bandwidth burdens, mirror networks are designed to alleviate that burden and both have those in place.
Sure, and they're very helpful. But to the one managing such networks, a massive increase in bandwidth use or in the *breadth* of content used for a particular mirrored repo for a particular product is very helpful to plan for and avoid local choke points. I'm especially thinking of the cache on relevant proxy servers.
On Thu, Nov 25, 2021 at 2:02 PM Nico Kadel-Garcia nkadel@gmail.com wrote:
On Thu, Nov 25, 2021 at 8:26 AM Neal Gompa ngompa13@gmail.com wrote:
On Thu, Nov 25, 2021 at 6:19 AM Nico Kadel-Garcia nkadel@gmail.com wrote:
On Thu, Nov 25, 2021 at 3:05 AM Miroslav Suchý msuchy@redhat.com wrote:
Dne 22. 11. 21 v 15:00 Pavel Raiskup napsal(a):
Hello Fedora EPEL maintainers!
First I don't feel comfortable announcing this, I'm not happy about the situation and so I don't want to be the lightning rod :-). But I believe that we can come to acceptable Copr/Mock solution and this needs to be discussed... so here we are.
By the end of the year 2021 we have to fix our default EPEL 8 Mock configuration (mock-core-configs.rpm, /etc/mock/epel-8-*.cfg) as CentOS 8 goes EOL by then.
I wrote down the possible options and their pros and cons and I done my best to catch all the feedback here.
https://docs.google.com/document/d/1wF7-7_y6Ac_oB-kCFdE6VBWPW8o8zjXd2Z0SGy4V...
Miroslav
That seems to be a succinct listing. I think you left out my suggestion.of "support people re-inventing point releases for CentOS", which is what major CentOS users will do using internal mirrors. due to concern about unexpected and unwelcome updates of CentOS Stream, while they assess whether AlmaLinux or Rocky are reliable and stable enough to use. It's not an uncommon behavior for EPEL itself, partly because of EPEL's bad habit of deleting RPMs without warning and stripping out all previous releases. That's caused me problems with chromium and firefox when updates were incompatible with contemporary regression testing systems.
It's not a "bad habit", it happens because when packages are retired, keeping the packages there does a disservice to the community by effectively forcing a maintenance burden when there's no maintainer. As for stripping out previous releases, that's just how Pungi and Bodhi do update composes at the moment. Someday that'll be fixed, but then we'd have to come up with a policy on how many because there are storage concerns for mirrors if we kept everything published forever.
It causes problems and confusion for people who need to lock down evisting versions for deployment. And it happens for packages that are not retired, but merely updated. I was bitten by it myself with chromium updates last year. It forces users of EPEL to maintain internal repos, or out of band access to previously accessible RPMs. It's destabilizing and breaks the use of bills-of-material based deployments with complete lists of all desired RPMs.
Storage and bandwidth concerns are legitimate concerns, as is potentially continuing to publish older releases with known vulnerabilities or bugs. But neither Fedora nor RHEL simply discard previously published versions this way, they aggregate new releases. I consider this a longstanding bug for EPEL, and one of the reasons I set up internal mirrors in large deployments.
This is not true. Fedora and EPEL share the same system, and have the same issues. The only difference is that the release repo is frozen in Fedora, so only the updates repo is affected this way. So there's at most two versions of a package at any time.
RHEL *does* maintain multiple old versions, but their system is completely different and supports that capability.
On Sun, Nov 28, 2021 at 7:06 PM Neal Gompa ngompa13@gmail.com wrote:
On Thu, Nov 25, 2021 at 2:02 PM Nico Kadel-Garcia nkadel@gmail.com wrote:
On Thu, Nov 25, 2021 at 8:26 AM Neal Gompa ngompa13@gmail.com wrote:
On Thu, Nov 25, 2021 at 6:19 AM Nico Kadel-Garcia nkadel@gmail.com wrote:
On Thu, Nov 25, 2021 at 3:05 AM Miroslav Suchý msuchy@redhat.com wrote:
Dne 22. 11. 21 v 15:00 Pavel Raiskup napsal(a):
Hello Fedora EPEL maintainers!
First I don't feel comfortable announcing this, I'm not happy about the situation and so I don't want to be the lightning rod :-). But I believe that we can come to acceptable Copr/Mock solution and this needs to be discussed... so here we are.
By the end of the year 2021 we have to fix our default EPEL 8 Mock configuration (mock-core-configs.rpm, /etc/mock/epel-8-*.cfg) as CentOS 8 goes EOL by then.
I wrote down the possible options and their pros and cons and I done my best to catch all the feedback here.
https://docs.google.com/document/d/1wF7-7_y6Ac_oB-kCFdE6VBWPW8o8zjXd2Z0SGy4V...
Miroslav
That seems to be a succinct listing. I think you left out my suggestion.of "support people re-inventing point releases for CentOS", which is what major CentOS users will do using internal mirrors. due to concern about unexpected and unwelcome updates of CentOS Stream, while they assess whether AlmaLinux or Rocky are reliable and stable enough to use. It's not an uncommon behavior for EPEL itself, partly because of EPEL's bad habit of deleting RPMs without warning and stripping out all previous releases. That's caused me problems with chromium and firefox when updates were incompatible with contemporary regression testing systems.
It's not a "bad habit", it happens because when packages are retired, keeping the packages there does a disservice to the community by effectively forcing a maintenance burden when there's no maintainer. As for stripping out previous releases, that's just how Pungi and Bodhi do update composes at the moment. Someday that'll be fixed, but then we'd have to come up with a policy on how many because there are storage concerns for mirrors if we kept everything published forever.
It causes problems and confusion for people who need to lock down evisting versions for deployment. And it happens for packages that are not retired, but merely updated. I was bitten by it myself with chromium updates last year. It forces users of EPEL to maintain internal repos, or out of band access to previously accessible RPMs. It's destabilizing and breaks the use of bills-of-material based deployments with complete lists of all desired RPMs.
Storage and bandwidth concerns are legitimate concerns, as is potentially continuing to publish older releases with known vulnerabilities or bugs. But neither Fedora nor RHEL simply discard previously published versions this way, they aggregate new releases. I consider this a longstanding bug for EPEL, and one of the reasons I set up internal mirrors in large deployments.
This is not true. Fedora and EPEL share the same system, and have the same issues. The only difference is that the release repo is frozen in Fedora, so only the updates repo is affected this way. So there's at most two versions of a package at any time.
You're correct. With the current setup, it's also relatively simple to revert to the "frozen" release, which handles most of the regression situations. And Fedora releases are nowhere near so long-lived as RHEL and EPEL, so it tends to be less of a long-lived problem.
RHEL *does* maintain multiple old versions, but their system is completely different and supports that capability.
What would it take to get Fedora, or at least EPEL, to preserve old releases in the default published repos? I appreciate that it would require thought and expand them noticeably, especially for bulky and frequently updating components like chromium. I admit to not having numbers on how much churn happens there: does anyone have numbers?
Nico Kadel-Garcia
On Sun, 28 Nov 2021 at 19:32, Nico Kadel-Garcia nkadel@gmail.com wrote:
On Sun, Nov 28, 2021 at 7:06 PM Neal Gompa ngompa13@gmail.com wrote:
On Thu, Nov 25, 2021 at 2:02 PM Nico Kadel-Garcia nkadel@gmail.com wrote:
On Thu, Nov 25, 2021 at 8:26 AM Neal Gompa ngompa13@gmail.com wrote:
On Thu, Nov 25, 2021 at 6:19 AM Nico Kadel-Garcia nkadel@gmail.com wrote:
On Thu, Nov 25, 2021 at 3:05 AM Miroslav Suchý msuchy@redhat.com wrote:
Dne 22. 11. 21 v 15:00 Pavel Raiskup napsal(a): > Hello Fedora EPEL maintainers! > > First I don't feel comfortable announcing this, I'm not happy about the > situation and so I don't want to be the lightning rod :-). But I believe > that we can come to acceptable Copr/Mock solution and this needs to be > discussed... so here we are. > > By the end of the year 2021 we have to fix our default EPEL 8 Mock > configuration (mock-core-configs.rpm, /etc/mock/epel-8-*.cfg) as CentOS 8 > goes EOL by then.
I wrote down the possible options and their pros and cons and I done my best to catch all the feedback here.
https://docs.google.com/document/d/1wF7-7_y6Ac_oB-kCFdE6VBWPW8o8zjXd2Z0SGy4V...
Miroslav
That seems to be a succinct listing. I think you left out my suggestion.of "support people re-inventing point releases for CentOS", which is what major CentOS users will do using internal mirrors. due to concern about unexpected and unwelcome updates of CentOS Stream, while they assess whether AlmaLinux or Rocky are reliable and stable enough to use. It's not an uncommon behavior for EPEL itself, partly because of EPEL's bad habit of deleting RPMs without warning and stripping out all previous releases. That's caused me problems with chromium and firefox when updates were incompatible with contemporary regression testing systems.
It's not a "bad habit", it happens because when packages are retired, keeping the packages there does a disservice to the community by effectively forcing a maintenance burden when there's no maintainer. As for stripping out previous releases, that's just how Pungi and Bodhi do update composes at the moment. Someday that'll be fixed, but then we'd have to come up with a policy on how many because there are storage concerns for mirrors if we kept everything published forever.
It causes problems and confusion for people who need to lock down evisting versions for deployment. And it happens for packages that are not retired, but merely updated. I was bitten by it myself with chromium updates last year. It forces users of EPEL to maintain internal repos, or out of band access to previously accessible RPMs. It's destabilizing and breaks the use of bills-of-material based deployments with complete lists of all desired RPMs.
Storage and bandwidth concerns are legitimate concerns, as is potentially continuing to publish older releases with known vulnerabilities or bugs. But neither Fedora nor RHEL simply discard previously published versions this way, they aggregate new releases. I consider this a longstanding bug for EPEL, and one of the reasons I set up internal mirrors in large deployments.
This is not true. Fedora and EPEL share the same system, and have the same issues. The only difference is that the release repo is frozen in Fedora, so only the updates repo is affected this way. So there's at most two versions of a package at any time.
You're correct. With the current setup, it's also relatively simple to revert to the "frozen" release, which handles most of the regression situations. And Fedora releases are nowhere near so long-lived as RHEL and EPEL, so it tends to be less of a long-lived problem.
RHEL *does* maintain multiple old versions, but their system is completely different and supports that capability.
What would it take to get Fedora, or at least EPEL, to preserve old releases in the default published repos? I appreciate that it would require thought and expand them noticeably, especially for bulky and frequently updating components like chromium. I admit to not having numbers on how much churn happens there: does anyone have numbers?
In order to keep older package releases, it would require changes to the compose tool pungi. It would also have to make it so it worked for EPEL versus Fedora. [Fedora Linux releases have grown to the point that many mirrors can barely carry the OS as is.. adding in older packages is out of the question for them.] I do not have numbers on how often packages churn or which ones churn the most.
On Mon, Nov 29, 2021 at 6:42 AM Stephen John Smoogen smooge@gmail.com wrote:
On Sun, 28 Nov 2021 at 19:32, Nico Kadel-Garcia nkadel@gmail.com wrote:
On Sun, Nov 28, 2021 at 7:06 PM Neal Gompa ngompa13@gmail.com wrote:
On Thu, Nov 25, 2021 at 2:02 PM Nico Kadel-Garcia nkadel@gmail.com wrote:
On Thu, Nov 25, 2021 at 8:26 AM Neal Gompa ngompa13@gmail.com wrote:
On Thu, Nov 25, 2021 at 6:19 AM Nico Kadel-Garcia nkadel@gmail.com wrote:
On Thu, Nov 25, 2021 at 3:05 AM Miroslav Suchý msuchy@redhat.com wrote: > > Dne 22. 11. 21 v 15:00 Pavel Raiskup napsal(a): > > Hello Fedora EPEL maintainers! > > > > First I don't feel comfortable announcing this, I'm not happy about the > > situation and so I don't want to be the lightning rod :-). But I believe > > that we can come to acceptable Copr/Mock solution and this needs to be > > discussed... so here we are. > > > > By the end of the year 2021 we have to fix our default EPEL 8 Mock > > configuration (mock-core-configs.rpm, /etc/mock/epel-8-*.cfg) as CentOS 8 > > goes EOL by then. > > > I wrote down the possible options and their pros and cons and I done my best to catch all the feedback here. > > https://docs.google.com/document/d/1wF7-7_y6Ac_oB-kCFdE6VBWPW8o8zjXd2Z0SGy4V... > > Miroslav
That seems to be a succinct listing. I think you left out my suggestion.of "support people re-inventing point releases for CentOS", which is what major CentOS users will do using internal mirrors. due to concern about unexpected and unwelcome updates of CentOS Stream, while they assess whether AlmaLinux or Rocky are reliable and stable enough to use. It's not an uncommon behavior for EPEL itself, partly because of EPEL's bad habit of deleting RPMs without warning and stripping out all previous releases. That's caused me problems with chromium and firefox when updates were incompatible with contemporary regression testing systems.
It's not a "bad habit", it happens because when packages are retired, keeping the packages there does a disservice to the community by effectively forcing a maintenance burden when there's no maintainer. As for stripping out previous releases, that's just how Pungi and Bodhi do update composes at the moment. Someday that'll be fixed, but then we'd have to come up with a policy on how many because there are storage concerns for mirrors if we kept everything published forever.
It causes problems and confusion for people who need to lock down evisting versions for deployment. And it happens for packages that are not retired, but merely updated. I was bitten by it myself with chromium updates last year. It forces users of EPEL to maintain internal repos, or out of band access to previously accessible RPMs. It's destabilizing and breaks the use of bills-of-material based deployments with complete lists of all desired RPMs.
Storage and bandwidth concerns are legitimate concerns, as is potentially continuing to publish older releases with known vulnerabilities or bugs. But neither Fedora nor RHEL simply discard previously published versions this way, they aggregate new releases. I consider this a longstanding bug for EPEL, and one of the reasons I set up internal mirrors in large deployments.
This is not true. Fedora and EPEL share the same system, and have the same issues. The only difference is that the release repo is frozen in Fedora, so only the updates repo is affected this way. So there's at most two versions of a package at any time.
You're correct. With the current setup, it's also relatively simple to revert to the "frozen" release, which handles most of the regression situations. And Fedora releases are nowhere near so long-lived as RHEL and EPEL, so it tends to be less of a long-lived problem.
RHEL *does* maintain multiple old versions, but their system is completely different and supports that capability.
What would it take to get Fedora, or at least EPEL, to preserve old releases in the default published repos? I appreciate that it would require thought and expand them noticeably, especially for bulky and frequently updating components like chromium. I admit to not having numbers on how much churn happens there: does anyone have numbers?
In order to keep older package releases, it would require changes to the compose tool pungi. It would also have to make it so it worked for EPEL versus Fedora. [Fedora Linux releases have grown to the point that many mirrors can barely carry the OS as is.. adding in older packages is out of the question for them.] I do not have numbers on how often packages churn or which ones churn the most.
EPEL is big enough that I begged Troy Dawson to untag duplicate copies of KDE Plasma so that I could have enough free space for a second mirror job to work. Between EPEL and EPEL Next for 8, it's pretty huge. I would say with all the EPEL repositories, it matches the size of Fedora Linux release trees. I don't think anyone would be too happy with more. And keep in mind that EPEL trees last for 10 years on the mirror network before being frozen. That's a lot of updates over the years.
If Fedora and EPEL were to have older versions, we'd have to have a dedicated CDN endpoint for them, because mirrors would seriously have trouble taking it.
copr-devel@lists.fedorahosted.org
-
Carl George -
Jean-Marc Liger -
Jean-Marc Liger -
Matthew Miller -
Miro Hrončok -
Miroslav Suchý -
Neal Gompa -
Nico Kadel-Garcia -
Pavel Raiskup -
Stephen John Smoogen