Wiki - https://fedoraproject.org/wiki/Changes/OpensslDeprecateEngine
This is a proposed Change for Fedora Linux. This document represents a proposed Change. As part of the Changes process, proposals are publicly announced in order to receive community feedback. This proposal will only be implemented if approved by the Fedora Engineering Steering Committee.
== Summary == We disable building the packages using ENGINE API in OpenSSL without breaking ABI.
== Owner == * Name: [[User:Dbelyavs| Dmitry Belyavskiy]] * Email: dbelyavs@redhat.com
== Detailed Description == We are going to deprecate OpenSSL engine support. Engines are not FIPS compatible and corresponding API is deprecated since OpenSSL 3.0. The engine functionality we are aware of (PKCS#11, TPM) is either covered by providers or will be covered soon.
We don't plan to remove the API from libcrypto.so. We are going to prevent creating the new packages dependent on OpenSSL ENGINE API and remove ENGINE dependencies from the existing packages.
During discussion of the previous proposal - to completely remove the ENGINE API - there were many relevant arguments why it shouldn't be done. We agree with them but still want to deprecate the ENGINE support to simplify removing it in the earliest release when it's feasible.
== Feedback ==
== Benefit to Fedora == We get rid of deprecated functionality and enforce using up-to-date API. Engine support is deprecated in OpenSSL upstream, and after provider migration caused some deficiencies with engine support. No new features will be added to engine. So we reduce maintenance burden and potentially attack surface.
It follows approach planned for CentOS 10.
== Scope == * Proposal owners: maintainers of packages enumerated here: https://clang.fedorapeople.org/c10s-engine-users/ plus probably owners of some Fedora-only packages
For most of the packages the maintainers will just have to rebuild their packages after the OpenSSL change lands in compose. For several packages some patches should be implemented to prevent compilation errors.
* Other developers: -
* Release engineering: [https://pagure.io/releng/issues #Releng issue number] This change probably requires mass-rebuild.
* Policies and guidelines: We need reject/modify packages providing OpenSSL engines
* Trademark approval: N/A (not needed for this Change)
* Alignment with Community Initiatives:
== Upgrade/compatibility impact == None. Users will be encouraged to switch their configurations to use providers instead but existing engines will continue working.
== How To Test == OpenSSL libcrypto.so exports the same ENGINE_* symbols as for f40. Applications relying on the ENGINE API can't be built but still work.
== User Experience == Users will be encouraged to reconfigure systems to providers if they use engines. No other changes are expected.
== Dependencies == In theory, all OpenSSL-dependent packages. In practice, only those that explicitly use ENGINE api.
== Contingency Plan == Returning the engine header file to allow old applications to be built.
* Contingency mechanism: (What to do? Who will do it?) rebuild OpenSSL and dependent packages * Contingency deadline: beta freeze? * Blocks release? Yes
== Documentation == TBD
== Release Notes == TBD
On Tue, Apr 02, 2024 at 10:45:32AM +0100, Aoife Moloney wrote:
== Summary == We disable building the packages using ENGINE API in OpenSSL without breaking ABI.
"Without breaking ABI" is a improvement. Everything else — not so much.
== Detailed Description == We are going to deprecate OpenSSL engine support. Engines are not FIPS compatible and corresponding API is deprecated since OpenSSL 3.0. The engine functionality we are aware of (PKCS#11, TPM) is either covered by providers or will be covered soon.
This doesn't really answer the problems that were raised in the previous round of discussion. Even if the plan is that some functionality will be provided "soon", dropping engine support _right now_ will cause problems for developers and for users: first the providers need to become available and have the complete feature set, then the upstreams have add the relevant provider code and document things so that users know how to adapt, and then we need to integrate all of that downstream in packaging.
If we start be ripping out the engine functionality, we'll have a window, of unpredictable length, where the functionality will not be available. This is bad for users, but also breaks the promise that Fedora is the best environment for upstream development.
We don't plan to remove the API from libcrypto.so. We are going to prevent creating the new packages dependent on OpenSSL ENGINE API and remove ENGINE dependencies from the existing packages.
IIUC, this is implemented by dropping the header files. This is not acceptable: it would break package rebuilds.
== Benefit to Fedora == We get rid of deprecated functionality and enforce using up-to-date API. Engine support is deprecated in OpenSSL upstream, and after provider migration caused some deficiencies with engine support. No new features will be added to engine. So we reduce maintenance burden and potentially attack surface.
It follows approach planned for CentOS 10.
Such things are always a tradeoff. The benefit of removing the deprecated code obviously reduced the maintanance burder a bit. But is is really that much? OTOH, it disables features that are being used or developed in a bunch of important projects. It seems very premature to take this step for F41, i.e. sometime within the next two—three months. It seems that the ecosystem isn't ready.
The tradeoff for CentOS 10 is different: the first release is still a while away and most people will not jump onto the 10.0 release anyway. By the time CentOS is used in anger, the ecosystem might be ready. This is one of the cases where using Fedora as a pre-release for CentOS as a pre-release for RHEL is detrimental to Fedora.
== How To Test == OpenSSL libcrypto.so exports the same ENGINE_* symbols as for f40. Applications relying on the ENGINE API can't be built but still work.
That's incompatible with package rebuilds…
An acceptable approach would be to split out the headers to a separate -devel file, e.g. openssl-engine-devel, mark it as Provides: deprecated(). Existing packages which need the engine headers can adjust to use the new header and new packages are prevented by the Packaging Guidelines from adding a dependency on deprecated packages.
Zbyszek
Hi Zbigniew!
On Tue, Apr 2, 2024 at 1:15 PM Zbigniew Jędrzejewski-Szmek < zbyszek@in.waw.pl> wrote:
On Tue, Apr 02, 2024 at 10:45:32AM +0100, Aoife Moloney wrote:
== Summary == We disable building the packages using ENGINE API in OpenSSL without breaking ABI.
"Without breaking ABI" is a improvement. Everything else — not so much.
== Detailed Description == We are going to deprecate OpenSSL engine support. Engines are not FIPS compatible and corresponding API is deprecated since OpenSSL 3.0. The engine functionality we are aware of (PKCS#11, TPM) is either covered by providers or will be covered soon.
This doesn't really answer the problems that were raised in the previous round of discussion. Even if the plan is that some functionality will be provided "soon", dropping engine support _right now_ will cause problems for developers and for users: first the providers need to become available and have the complete feature set, then the upstreams have add the relevant provider code and document things so that users know how to adapt, and then we need to integrate all of that downstream in packaging.
If we start be ripping out the engine functionality, we'll have a window, of unpredictable length, where the functionality will not be available. This is bad for users, but also breaks the promise that Fedora is the best environment for upstream development.
Thanks. In the period between the proposal was written and published the TPM2 provider has landed in Fedora. PKCS#11 provider is already here for a while.
Should I update the Wiki page to adjust this point?
We don't plan to remove the API from libcrypto.so. We are going to
prevent creating the new packages dependent on OpenSSL ENGINE API and remove ENGINE dependencies from the existing packages.
IIUC, this is implemented by dropping the header files. This is not acceptable: it would break package rebuilds.
It's our goal - to eliminate the engine dependency. While a package can be built, it gets lower priority.
== Benefit to Fedora ==
We get rid of deprecated functionality and enforce using up-to-date API. Engine support is deprecated in OpenSSL upstream, and after provider migration caused some deficiencies with engine support. No new features will be added to engine. So we reduce maintenance burden and potentially attack surface.
It follows approach planned for CentOS 10.
Such things are always a tradeoff. The benefit of removing the deprecated code obviously reduced the maintanance burder a bit. But is is really that much? OTOH, it disables features that are being used or developed in a bunch of important projects. It seems very premature to take this step for F41, i.e. sometime within the next two—three months. It seems that the ecosystem isn't ready.
I want to encourage the ecosystem to move to a viable target.
The tradeoff for CentOS 10 is different: the first release is still a while away and most people will not jump onto the 10.0 release anyway. By the time CentOS is used in anger, the ecosystem might be ready. This is one of the cases where using Fedora as a pre-release for CentOS as a pre-release for RHEL is detrimental to Fedora.
I agree.
== How To Test ==
OpenSSL libcrypto.so exports the same ENGINE_* symbols as for f40. Applications relying on the ENGINE API can't be built but still work.
That's incompatible with package rebuilds…
An acceptable approach would be to split out the headers to a separate -devel file, e.g. openssl-engine-devel, mark it as Provides: deprecated(). Existing packages which need the engine headers can adjust to use the new header and new packages are prevented by the Packaging Guidelines from adding a dependency on deprecated packages.
Thanks! I like this idea and can update the Wiki page accordingly.
Hi Zbigniew!
On Tue, Apr 2, 2024 at 1:15 PM Zbigniew Jędrzejewski-Szmek < zbyszek(a)in.waw.pl> wrote:
Thanks. In the period between the proposal was written and published the TPM2 provider has landed in Fedora. PKCS#11 provider is already here for a while.
The fact that such packages are physically present is not enough - they need to implement all the needed features, and they need to be mature enough to just work out of the box. Neither of these are true today, and providers just do not work for very simple use cases like signing a UKI with a yubikey. At the very least a couple more years of development and testing is needed before they are anywhere near ready to drop support for engines, that actually do work out of the box. Not to mention third party engines that are specific to internal/private build systems - if any such system runs Fedora as the build host, they'd have to migrate to Debian/Ubuntu to keep working.
Dear Luca
On Tue, Apr 2, 2024 at 4:32 PM Luca Boccassi bluca@debian.org wrote:
Hi Zbigniew!
On Tue, Apr 2, 2024 at 1:15 PM Zbigniew Jędrzejewski-Szmek < zbyszek(a)in.waw.pl> wrote:
Thanks. In the period between the proposal was written and published the TPM2 provider has landed in Fedora. PKCS#11 provider is already here for a while.
The fact that such packages are physically present is not enough - they need to implement all the needed features, and they need to be mature enough to just work out of the box. Neither of these are true today, and providers just do not work for very simple use cases like signing a UKI with a yubikey. At the very least a couple more years of development and testing is needed before they are anywhere near ready to drop support for engines, that actually do work out of the box. Not to mention third party engines that are specific to internal/private build systems - if any such system runs Fedora as the build host, they'd have to migrate to Debian/Ubuntu to keep working.
The TPM2 package is suitable for all required operations, AFAIK. I'm also sure about the PKCS11 provider which I follow close enough.
Please raise detailed issues if you have something particular. I remember that you mentioned a particular issue about PKCS#11, could you please try the current version? My colleagues working on PKCS#11 are not aware of any Yubikey issues, BTW.
Third-party engines may be a problem but as we don't break ABI, it's not a problem of the moment.
On Tue, Apr 2, 2024 at 3:12 PM Dmitry Belyavskiy dbelyavs@redhat.com wrote:
Third-party engines may be a problem but as we don't break ABI, it's not a problem of the moment.
The fact you are removing the headers means it is a problem for 3rd party engines who build from source (and everyone should at least occasionally be building from source as part of their CI).
I consider removing the headers as breaking the API, as the headers define the API. The headers already mark the engine APIs as deprecated. Orgs with resources should be starting their migration, although some will defer it to the next quarter (and the next....)
I expect you have no visibility into 3rd party engines, nor how big an issue this will be (and can make no statistically valid claim it will not be an issue unless you have real numbers to share). However, *after* RHEL10 is released with engine support removed RH's TAMs may have a better idea (I am WAGing most 3rd party engines will be being used by large customers, and they are likely to make any concerns known).
I believe this should be part of OpenSSL 4.0. It will be a clear change. There is no compelling reason for this to happen today via the headers. Instead this should be a marketing campaign by OpenSSL to remind everyone that engines are going away with OpenSSL 4.0 with every new set of release notes (first item, in double bold), and that orgs need to start their migration. And then do it again. And then do it again. And when OpenSSL 4.0 is released, you can remind everyone you warned them.
Dear Gary,
On Tue, Apr 2, 2024 at 5:39 PM Gary Buhrmaster gary.buhrmaster@gmail.com wrote:
On Tue, Apr 2, 2024 at 3:12 PM Dmitry Belyavskiy dbelyavs@redhat.com wrote:
Third-party engines may be a problem but as we don't break ABI, it's not
a problem of the moment.
The fact you are removing the headers means it is a problem for 3rd party engines who build from source (and everyone should at least occasionally be building from source as part of their CI).
I consider removing the headers as breaking the API, as the headers define the API.
I agree with moving the engine header package to a separate devel package instead of removing it.
The headers already mark the engine APIs as deprecated. Orgs with resources should be starting their migration, although some will defer it to the next quarter (and the next....)
Yes. That's why I think the pressure in this direction is worth the effort.
I believe this should be part of OpenSSL 4.0.
It will be a clear change. There is no compelling reason for this to happen today via the headers. Instead this should be a marketing campaign by OpenSSL to remind everyone that engines are going away with OpenSSL 4.0 with every new set of release notes (first item, in double bold), and that orgs need to start their migration. And then do it again. And then do it again. And when OpenSSL 4.0 is released, you can remind everyone you warned them.
I agree that removing the engine stuff is clearly 4.0 stuff (or earlier version if .so version changes). But there are steps that will indicate our moving in this direction and will allow reducing the number of packages to be changed.
[Replying to two mails at once to conserve some electrons.]
On Tue, Apr 02, 2024 at 04:03:31PM +0200, Dmitry Belyavskiy wrote:
Thanks. In the period between the proposal was written and published the TPM2 provider has landed in Fedora. PKCS#11 provider is already here for a while.
Should I update the Wiki page to adjust this point?
Please do.
== How To Test ==
OpenSSL libcrypto.so exports the same ENGINE_* symbols as for f40. Applications relying on the ENGINE API can't be built but still work.
That's incompatible with package rebuilds…
An acceptable approach would be to split out the headers to a separate -devel file, e.g. openssl-engine-devel, mark it as Provides: deprecated(). Existing packages which need the engine headers can adjust to use the new header and new packages are prevented by the Packaging Guidelines from adding a dependency on deprecated packages.
Thanks! I like this idea and can update the Wiki page accordingly.
Thanks!
On Tue, Apr 02, 2024 at 05:12:20PM +0200, Dmitry Belyavskiy wrote:
On Tue, Apr 2, 2024 at 4:32 PM Luca Boccassi bluca@debian.org wrote: [...] The TPM2 package is suitable for all required operations, AFAIK. I'm also sure about the PKCS11 provider which I follow close enough.
Please raise detailed issues if you have something particular. I remember that you mentioned a particular issue about PKCS#11, could you please try the current version? My colleagues working on PKCS#11 are not aware of any Yubikey issues, BTW.
Third-party engines may be a problem but as we don't break ABI, it's not a problem of the moment.
On Wed, Apr 03, 2024 at 09:50:27AM +0200, Clemens Lang wrote:
I did try using the current pkcs11-provider with my Yubikey to create a signature using openssl dgst -sign 'pkcs11:serial=18c9662a9c930e9e;id=%02;type=private'. It worked just fine for me, including prompting for the PIN, twice.
I did have to enable the PKCS11 provider in my openssl.cnf, but that could also be done programmatically at runtime by applications> should they choose to do so.
I was not able to reproduce the problems you faced in the systemd upstream ticket you referred to earlier. It is possible that they have been fixed upstream in the meantime.
Thank you both, it sounds like this should work. In systemd, we'll need to adjust the code to use providers, but that should be doable.
OK, so with discussed changes, I'm +1.
Zbyszek
Dear Zbyszek,
Thanks, I updated the Wiki page correspondingly.
On Wed, Apr 3, 2024 at 5:56 PM Zbigniew Jędrzejewski-Szmek < zbyszek@in.waw.pl> wrote:
[Replying to two mails at once to conserve some electrons.]
On Tue, Apr 02, 2024 at 04:03:31PM +0200, Dmitry Belyavskiy wrote:
Thanks. In the period between the proposal was written and published the TPM2 provider has landed in Fedora. PKCS#11 provider is already here for a while.
Should I update the Wiki page to adjust this point?
Please do.
== How To Test ==
OpenSSL libcrypto.so exports the same ENGINE_* symbols as for f40. Applications relying on the ENGINE API can't be built but still work.
That's incompatible with package rebuilds…
An acceptable approach would be to split out the headers to a separate -devel file, e.g. openssl-engine-devel, mark it as Provides:
deprecated().
Existing packages which need the engine headers can adjust to use the new header and new packages are prevented by the Packaging Guidelines from adding a dependency on deprecated packages.
Thanks! I like this idea and can update the Wiki page accordingly.
Thanks!
On Tue, Apr 02, 2024 at 05:12:20PM +0200, Dmitry Belyavskiy wrote:
On Tue, Apr 2, 2024 at 4:32 PM Luca Boccassi bluca@debian.org wrote: [...] The TPM2 package is suitable for all required operations, AFAIK. I'm also sure about the PKCS11 provider which I follow close enough.
Please raise detailed issues if you have something particular. I remember that you mentioned a particular issue about PKCS#11, could you please try the current version? My colleagues working on PKCS#11 are not aware of any Yubikey issues,
BTW.
Third-party engines may be a problem but as we don't break ABI, it's not
a
problem of the moment.
On Wed, Apr 03, 2024 at 09:50:27AM +0200, Clemens Lang wrote:
I did try using the current pkcs11-provider with my Yubikey to create a signature using openssl dgst -sign 'pkcs11:serial=18c9662a9c930e9e;id=%02;type=private'. It worked just fine for me, including prompting for the PIN, twice.
I did have to enable the PKCS11 provider in my openssl.cnf, but that could also be done programmatically at runtime by applications> should they choose to do so.
I was not able to reproduce the problems you faced in the systemd upstream ticket you referred to earlier. It is possible that they have been fixed upstream in the meantime.
Thank you both, it sounds like this should work. In systemd, we'll need to adjust the code to use providers, but that should be doable.
OK, so with discussed changes, I'm +1.
Zbyszek
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Hi,
On 2. Apr 2024, at 16:31, Luca Boccassi bluca@debian.org wrote:
The fact that such packages are physically present is not enough - they need to implement all the needed features, and they need to be mature enough to just work out of the box. Neither of these are true today, and providers just do not work for very simple use cases like signing a UKI with a yubikey. At the very least a couple more years of development and testing is needed before they are anywhere near ready to drop support for engines, that actually do work out of the box. Not to mention third party engines that are specific to internal/private build systems - if any such system runs Fedora as the build host, they'd have to migrate to Debian/Ubuntu to keep working.
I did try using the current pkcs11-provider with my Yubikey to create a signature using openssl dgst -sign 'pkcs11:serial=18c9662a9c930e9e;id=%02;type=private'. It worked just fine for me, including prompting for the PIN, twice.
I did have to enable the PKCS11 provider in my openssl.cnf, but that could also be done programmatically at runtime by applications should they choose to do so.
I was not able to reproduce the problems you faced in the systemd upstream ticket you referred to earlier. It is possible that they have been fixed upstream in the meantime.
There will always be some effort related to such a transition, but that effort will have to happen one way or the other eventually. I suspect if Fedora decides to keep ENGINE support, we’ll have the exact same discussion in a few years when OpenSSL 4.0 is released, and people will demand that the rebase to 4.0 that removes engine support should be a system-wide change proposal because it breaks engines.
On Wed, Apr 03, 2024 at 09:50:27AM +0200, Clemens Lang wrote:
There will always be some effort related to such a transition, but that effort will have to happen one way or the other eventually. I suspect if Fedora decides to keep ENGINE support, we’ll have the exact same discussion in a few years when OpenSSL 4.0 is released, and people will demand that the rebase to 4.0 that removes engine support should be a system-wide change proposal because it breaks engines.
Given that the ENGINE API is deprecated upstream since OpenSSL 3.0, the API is optional upstream, and its use has produced compiler warnings since it was introduced in Fedora 36, it seems perfectly reasonable to disable this API in Fedora 41.
We have to deal with a very large numbers of FTBFS bugs from e.g. Python API breaks every other release cycle, or the latest compiler flag tuning. The fact that the transition creates work for other package maintainers is obviously not a reasonable blocker for a Fedora Change.
Regards, Joe
Joe Orton wrote:
Given that the ENGINE API is deprecated upstream since OpenSSL 3.0, the API is optional upstream, and its use has produced compiler warnings since it was introduced in Fedora 36, it seems perfectly reasonable to disable this API in Fedora 41.
I disagree. Disabling an API that is still widely used and for which there is still no complete replacement (several replies have pointed out issues still preventing "providers" from working in all use cases in which "engines" work) is NOT reasonable.
We have to deal with a very large numbers of FTBFS bugs from e.g. Python API breaks every other release cycle, or the latest compiler flag tuning. The fact that the transition creates work for other package maintainers is obviously not a reasonable blocker for a Fedora Change.
And that is exactly the kind of cultural issue we need to solve. The Python 3 transition is exactly an example of a transition that was handled horribly, kicking out lots of useful packages from Fedora just because they were not ported to Python 3. Python 2, or a fork like Tauthon (which has the advantage that it supports some Python 3 features, so some Python-3-only libraries / library versions can be backported to Tauthon more easily than to stock Python 2), should have been retained as a compatibility platform in Fedora. (There is technically still a "python27" package, but the modules available for it are intentionally limited and there are strict rules on what packages are allowed to depend on it.) It should NEVER be considered reasonable to break other people's work.
Kevin Kofler
Dear Kevin
On Wed, Apr 3, 2024 at 8:13 PM Kevin Kofler via devel < devel@lists.fedoraproject.org> wrote:
Joe Orton wrote:
Given that the ENGINE API is deprecated upstream since OpenSSL 3.0, the API is optional upstream, and its use has produced compiler warnings since it was introduced in Fedora 36, it seems perfectly reasonable to disable this API in Fedora 41.
I disagree. Disabling an API that is still widely used and for which there is still no complete replacement (several replies have pointed out issues still preventing "providers" from working in all use cases in which "engines" work) is NOT reasonable.
You are 100% correct. That's why disabling this API is not on the table for now anymore.