2:35 p.m.
https://fedoraproject.org/wiki/Changes/krb5_crypto_modernization
krb5 will be removing support for DES, 3DES, crc-32, and MD4 entirely;
they will not be allowed in session keys or long-term keys.
Additionally, RC4 and MD5 will be marked deprecated and dangerous.
== Owner ==
* Name: [[User:rharwood|Robbie Harwood]]
* Email: rharwood at fp dot o
== Detailed Description ==
For formal deprecation announcements, there have been two RFCs. First
was [https://tools.ietf.org/html/rfc6649 rfc6649], which deprecated
single-DES (and export strength RC4). More recently, there has also
been [https://tools.ietf.org/html/rfc8429 rfc8429] which deprecated
the rest of RC4 as well as 3DES. (Note that 3DES was paired with
SHA1, RC4 with MD5, and DES with MD4, MD5, or crc32, and that these
were the only usages of these digests in krb5's Kerberos
implementation.)
Of particular note, single-DES has been known-bad since the mid 90s.
It's now so bad that it can be entirely broken in 13 hours for $300:
[https://crack.sh/kerb5/ https://crack.sh/kerb5/] But even though
other encryption types in this list are less bad, none are what we'd
call good - especially since krb5 supports much stronger encryption,
such as AES256/SHA2.
Finally, field data suggests that complete removal is preferable to
disabling by default. Admins have unfortunately been using
`allow_weak_crypto` (which restored disabled-by-default single-DES
functionality to krb5) to this day. A mere config setting doesn't
suffice to convey how bad this is, nor does it indicate the
appropriate level of urgency of migration. For that reason, and in
anticipation that our crypto backends might one day also remove these
algorithms, we are going straight to removal as well, rather than
repeating `allow_weak_crypto` or similar.
Unfortunately, it's not possible for RC4 to be removed entirely at
this time because it is still in demand for Active Directory
integration (which also affects freeIPA and samba). For this reason,
it's moved to `allow_weak_crypto`.
== Benefit to Fedora ==
Improved crpytography is good for security. Fedora will be leading on
this change; no other distributions or OS releases have limited the
bad ciphers that Kerberos can use.
== Scope ==
* Proposal owners: rharwood
* Other developers: N/A (not a System Wide Change)
* Release engineering: [https://pagure.io/releng/issue/8012 #8012]
* Policies and guidelines: N/A (not a System Wide Change)
* Trademark approval: N/A (not needed for this Change)
== Upgrade/compatibility impact ==
Users of RC4 who need to continue doing so will need to add
`allow_weak_crypto = True` to /etc/krb5.conf.
== How To Test ==
If you'd like to test either this change or your site's readiness for
more modern cryptography, that's great! `krb5.conf(5)` has several
encryption type (enctype) -related settings - of particular note is
permitted_enctypes. Set to only removed enctyptes to verify they
don't work anymore; set to only modern enctypes to verify that you're
ready. (Then just perform your normal tasks with krb5, such as
`kinit`).
C-friendly users might wish to take a look at t_enctypes.c in the krb5
source tree. (How this will be affected depends on how the removal
work internally, which I haven't worked out yet.)
== User Experience ==
Ideally no change! Worst case some users will see krb5 produce error
messages about bad enctypes not being able to be used (has no enctype,
could not fullfill enctype, etc.). These pains are the feeling of the
world grinding forward security-wise.
== Dependencies ==
N/A (not a System Wide Change)
== Contingency Plan ==
* Contingency mechanism: (What to do? Who will do it?) N/A (not a
System Wide Change)
* Contingency deadline: N/A (not a System Wide Change)
* Blocks release? N/A (not a System Wide Change)
* Blocks product? N/A
== Documentation ==
N/A (not a System Wide Change)
== Release Notes ==
krb5 removes support for several known-bad encryption types.
Hopefully users will see no changes.
--
Ben Cotton
Fedora Program Manager
TZ=America/Indiana/Indianapolis
4:31 a.m.
Hi,
personally, I don't like the advertisment for that commercial service, see the given
price and link. It does not comply with FLOSS policies, therefore I commented in the
releng ticket.
Just my 5ct, Raphael
3:33 p.m.
Raphael, I'm confused how this doesn't comply. Their source code lives here:
https://github.com/h1kari/des_kpt
Also, let's keep comments on this thread *and CC me* - then there's only one place
to look for replies (and I'm not subscribed to this list). In any case, as I stated
there, if you have an alternate phrasing that doesn't come across to you as
"advertising", I'm open to suggestion.
Thanks,
--Robbie
4:31 a.m.
On Fri, 2018-12-21 at 15:35 -0500, Ben Cotton wrote:
https://fedoraproject.org/wiki/Changes/krb5_crypto_modernization
krb5 will be removing support for DES, 3DES, crc-32, and MD4
entirely;
they will not be allowed in session keys or long-term keys.
Additionally, RC4 and MD5 will be marked deprecated and dangerous.
== Owner ==
* Name: [[|Robbie Harwood]]
* Email: rharwood at fp dot o
== Detailed Description ==
For formal deprecation announcements, there have been two
RFCs. First
was [https://tools.ietf.org/html/rfc6649 rfc6649], which deprecated
single-DES (and export strength RC4). More recently, there has also
been [https://tools.ietf.org/html/rfc8429 rfc8429] which deprecated
the rest of RC4 as well as 3DES. (Note that 3DES was paired with
SHA1, RC4 with MD5, and DES with MD4, MD5, or crc32, and that these
were the only usages of these digests in krb5's Kerberos
implementation.)
Of particular note, single-DES has been known-bad since the mid 90s.
It's now so bad that it can be entirely broken in 13 hours for $300:
[https://crack.sh/kerb5/ https://crack.sh/kerb5/] But even though
other encryption types in this list are less bad, none are what we'd
call good - especially since krb5 supports much stronger encryption,
such as AES256/SHA2.
Finally, field data suggests that complete removal is preferable to
disabling by default. Admins have unfortunately been using
`allow_weak_crypto` (which restored disabled-by-default single-DES
functionality to krb5) to this day. A mere config setting doesn't
suffice to convey how bad this is, nor does it indicate the
appropriate level of urgency of migration. For that reason, and in
anticipation that our crypto backends might one day also remove these
algorithms, we are going straight to removal as well, rather than
repeating `allow_weak_crypto` or similar.
Unfortunately, it's not possible for RC4 to be removed entirely at
this time because it is still in demand for Active Directory
integration (which also affects freeIPA and samba). For this reason,
it's moved to `allow_weak_crypto`.
== Benefit to Fedora ==
Improved crpytography is good for security. Fedora will be leading
on
this change; no other distributions or OS releases have limited the
bad ciphers that Kerberos can use.
== Scope ==
* Proposal owners: rharwood
* Other developers: N/A (not a System Wide Change)
* Release engineering: [https://pagure.io/releng/issue/8012 #8012]
* Policies and guidelines: N/A (not a System Wide Change)
* Trademark approval: N/A (not needed for this Change)
== Upgrade/compatibility impact ==
Users of RC4 who need to continue doing so will need to add
`allow_weak_crypto = True` to /etc/krb5.conf.
== How To Test ==
If you'd like to test either this change or your site's readiness for
more modern cryptography, that's great! `krb5.conf(5)` has several
encryption type (enctype) -related settings - of particular note is
permitted_enctypes. Set to only removed enctyptes to verify they
don't work anymore; set to only modern enctypes to verify that you're
ready. (Then just perform your normal tasks with krb5, such as
`kinit`).
C-friendly users might wish to take a look at t_enctypes.c in the
krb5
source tree. (How this will be affected depends on how the removal
work internally, which I haven't worked out yet.)
Hi,
How does this ties with crypto policies? libkrb5 is already under
crypto policies and has these ciphers disabled by default. Is this
change about removing them from the code or removing them from the
capabilities of the KDC which is not covered by crypto policies?
== User Experience ==
Ideally no change! Worst case some users will see krb5 produce error
messages about bad enctypes not being able to be used (has no
enctype,
could not fullfill enctype, etc.). These pains are the feeling of
the
world grinding forward security-wise.
I guess there will be different experience in client side where these
are already disabled by default in fedora, and in server side which
they are not.
regards,
Nikos
12:47 p.m.
Nikos Mavrogiannopoulos <nmav(a)redhat.com> writes:
How does this ties with crypto policies? libkrb5 is already under
crypto policies and has these ciphers disabled by default. Is this
change about removing them from the code or removing them from the
capabilities of the KDC which is not covered by crypto policies?
> == User Experience ==
>
> Ideally no change! Worst case some users will see krb5 produce error
> messages about bad enctypes not being able to be used (has no
> enctype, could not fullfill enctype, etc.). These pains are the
> feeling of the world grinding forward security-wise.
I guess there will be different experience in client side where these
are already disabled by default in fedora, and in server side which
they are not.
To address I think both your questions: the defaults we set matter a lot
less than we think they do. I observe plenty of machines running
single-DES in the wild - despite this being permitted at no level of
crypto-policies. This is obviously not the fault of crypto-policies in
any way; users have explicitly chosen to route around the system's
notion of "permitted" - and not without reason, in many cases.
The two biggest culprits for this (beyond general legacy infrastructure)
are Active Directory and AFS. Active Directory uses RC4
(crypto-policies LEGACY) as an interop cipher; this is why it's not
actually removed in this change. AFS has been using single-DES (for
interop) for quite some time. AFS does support more modern
cryptography, but its adoption is far from universal.
But I think the biggest takeaway here is that requiring users to perform
additional configuration in order to hurt themselves won't actually stop
them all from hurting themselves - especially not in the age of
automation. Maybe it would have been more effective if we'd followed
the 2009 `allow_weak_crypto` designation with a 2010 removal or some
threat thereof. Hard for me to know; in 2009 I was busy being an
obnoxious teenager.
Thanks,
--Robbie
12:44 p.m.
>>>> "BC" == Ben Cotton
<bcotton(a)redhat.com> writes:
BC> == Detailed Description ==
[elided]
Is it just me or does this not actually say clearly what is changing?
The first paragraph talks about two RFCs. The second paragraph talks
about how easy it is to break single DES. The third paragraph talks
about how disabled by default is undesirable. The fourth paragraph
talks about it not being possible to remove RC4.
But nothing says what is changing. The release notes section says:
BC> krb5 removes support for several known-bad encryption types.
BC> Hopefully users will see no changes.
but it doesn't really say what "removes support" means, or exactly which
encryption types will no longer be supported.
BC> == User Experience ==
BC> Ideally no change! Worst case some users will see krb5 produce
BC> error messages about bad enctypes not being able to be used (has no
BC> enctype, could not fullfill enctype, etc.). These pains are the
BC> feeling of the world grinding forward security-wise.
I think this is an extremely optimistic description of the worst case
behavior.
There is a good document for migrating from old encryption types at
https://web.mit.edu/kerberos/krb5-1.16/doc/admin/advanced/retiring-des.html
Note that the last paragraph there mentions that single-DES is still
basically OK for the K/M principal and it's not required to use
allow_weak_crypto to access a database encrypted with single-DES. I
can't tell if this change actually intends to remove all support for the
single-DES algorithm, but if so then it is highly important to
communicate the additional need to roll over the master key and stash
files.
It would also be nice to document a way to find principals which do not
have a sufficiently strong key.
I personally have a kerberos database which has existed for a very long
time now and am still dealing with the 3DES deprecation (and the fact
that the F29 crypto policies changed to remove it from
supported_enctypes).
- J<
3:05 p.m.
>>>> "JLT" == Jason L Tibbitts
<tibbs(a)math.uh.edu> writes:
JLT> Is it just me or does this not actually say clearly what is
JLT> changing?
Seems it's just me; somehow that's in the summary but not in the
detailed description. Seems odd for the details to have less
information than the short version, but I guess that's how it goes.
My questions about whether K/M will need to be rotated (even though the
existing kerberos docs say that it doesn't) and how a site can easily
tell if it has old principals still stand.
- J<
5:07 p.m.
BC> == Detailed Description ==
Is it just me or does this not actually say clearly what is changing?
The first paragraph talks about two RFCs. The second paragraph talks
about how easy it is to break single DES. The third paragraph talks
about how disabled by default is undesirable. The fourth paragraph
talks about it not being possible to remove RC4.
But nothing says what is changing. The release notes section says:
BC> krb5 removes support for several known-bad encryption types.
BC> Hopefully users will see no changes.
but it doesn't really say what "removes support" means, or exactly which
encryption types will no longer be supported.
Per your follow-up email, I'm not clear on whether you want changes here. If you do,
speak up, especially if you have suggestions.
BC> == User Experience ==
BC> Ideally no change! Worst case some users will see krb5 produce
BC> error messages about bad enctypes not being able to be used (has no
BC> enctype, could not fullfill enctype, etc.). These pains are the
BC> feeling of the world grinding forward security-wise.
I think this is an extremely optimistic description of the worst case
behavior.
I really don't think that "it won't work and there'll be error
messages" is an "extremely optimistic description". If you have langauge
changes, I'm happy to hear them out.
There is a good document for migrating from old encryption types at
https://web.mit.edu/kerberos/krb5-1.16/doc/admin/advanced/retiring-des.html
Note that the last paragraph there mentions that single-DES is still
basically OK for the K/M principal and it's not required to use
allow_weak_crypto to access a database encrypted with single-DES.
That reading only applies to sites with a stash file nearby (though sites set up otherwise
remain relatively uncommon in my impression, or are freeipa and will have AES keys).
I
can't tell if this change actually intends to remove all support for the
single-DES algorithm, but if so then it is highly important to
communicate the additional need to roll over the master key and stash
files.
Session keys for sure, but ideally all of it. I tried to state this in the Summary; if I
was insufficiently clear I'm happy to make changes! (Added rollover langauge to
"Upgrade/Compatability Impact" section.)
It would also be nice to document a way to find principals which do
not
have a sufficiently strong key.
This is good feedback. I will explore and see if I can make something.
I personally have a kerberos database which has existed for a very
long
time now and am still dealing with the 3DES deprecation (and the fact
that the F29 crypto policies changed to remove it from
supported_enctypes).
I'm sure you're aware, then, that there's only so much I can do as a packager
here. In particular, the KDC can't push keytab updates (because servers with keytabs
are not required to be able to communicate with the KDC). The UX on this is *going* to be
bad for anyone with weak crypto in use - which is part of why I want to pull it all out at
once, so we don't have to do this again for a while.
Please CC me on all future mails; I'm not subscribed to this list.
Thanks,
--Robbie
7:28 p.m.
On Thu, 2019-01-03 at 23:07 +0000, Robbie Harwood wrote:
> BC> == Detailed Description ==
>
>
> Is it just me or does this not actually say clearly what is changing?
> The first paragraph talks about two RFCs. The second paragraph talks
> about how easy it is to break single DES. The third paragraph talks
> about how disabled by default is undesirable. The fourth paragraph
> talks about it not being possible to remove RC4.
>
> But nothing says what is changing. The release notes section says:
>
> BC> krb5 removes support for several known-bad encryption types.
> BC> Hopefully users will see no changes.
>
> but it doesn't really say what "removes support" means, or exactly
which
> encryption types will no longer be supported.
Per your follow-up email, I'm not clear on whether you want changes here. If you do,
speak up, especially if you have suggestions.
> BC> == User Experience ==
>
> BC> Ideally no change! Worst case some users will see krb5 produce
> BC> error messages about bad enctypes not being able to be used (has no
> BC> enctype, could not fullfill enctype, etc.). These pains are the
> BC> feeling of the world grinding forward security-wise.
>
> I think this is an extremely optimistic description of the worst case
> behavior.
I really don't think that "it won't work and there'll be error
messages" is an "extremely optimistic description". If you have
langauge changes, I'm happy to hear them out.
Well, the wording doesn't actually say "it won't work", does it? It
just says "users will see krb5 produce error messages". You could at
least plausibly read that as "it will produce error messages...but work
anyway". It would certainly be *clearer* if it explicitly said "stuff
may not work without at least some kind of configuration change".
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
10:40 p.m.
>>>> "RH" == Robbie Harwood
<rharwood(a)redhat.com> writes:
RH> Per your follow-up email, I'm not clear on whether you want changes
RH> here. If you do, speak up, especially if you have suggestions.
Well, it was just odd that the summary had information not contained at
all within the detailed description. Since this was a change that
particularly interests me, I skipped straight to the details and then
was left with the question about exactly what was changing.
RH> I really don't think that "it won't work and there'll be error
RH> messages" is an "extremely optimistic description".
But to be fair, MIT krb5 is not known for having great error output.
Not being able to start at all because the K/M has an enctype which is
acceptable and not at all deprecated according to the documentation that
exists today _and_ failing in the way the software tends to fail (with
obscure and sometimes numeric messages) would be... tough. Not that I
think anyone would just do dnf system-upgrade on their master KDC.
RH> If you have langauge changes, I'm happy to hear them out.
I'm just not optimistic that it would be much better than "may fail in a
completely inexplicable fashion". Having a couple of principals that
can't get tickets because they lack a supported enctype is one thing.
(I've seen that and the error messages you get are OK, I guess.) But
having your entire database not load because it doesn't the enctype on
K/M is a different thing entirely. Note that currently things do work
fine if the enctype on K/M isn't in supported_enctypes. It doesn't even
warn. Presumably because supported_enctypes is related to issuing
tickets, not decrypting the database. But if the algorithms are
disabled completely, then... I don't know what would happen. That's why
I was asking whether the algorithms will still exist.
> There is a good document for migrating from old encryption types
at
> https://web.mit.edu/kerberos/krb5-1.16/doc/admin/advanced/retiring-des.html
> Note that the last paragraph there mentions that single-DES is still
> basically OK for the K/M principal and it's not required to use
> allow_weak_crypto to access a database encrypted with single-DES.
RH> That reading only applies to sites with a stash file nearby (though
RH> sites set up otherwise remain relatively uncommon in my impression,
RH> or are freeipa and will have AES keys).
I guess I can see that if I read it a few times. The idea is that if
the stash file is there on the disk, then having raw access to the
database probably means you can also just read the stash and so the
encryption type doesn't really matter much. If you're not using a stash
file then it then becomes more important. Still, rolling over K/M seems
relatively simple (though I haven't actually done it).
RH> (Added rollover langauge to "Upgrade/Compatability Impact" section.)
Thanks.
RH> I'm sure you're aware, then, that there's only so much I can do as a
RH> packager here.
Well certainly there isn't much you can do to fix old principals on
existing systems. But the current versions should be complaining loudly
when it has to issue a ticket for a principal that lacks a modern
enctype today, if not six months ago.
Also, I'm curious, because I read all of this as "Fedora is going to
patch upstream to disable certain algorithms because we don't think the
software should support them at all even if upstream does." If that's a
correct interpretation, then it's a bit disingenuous to say there's
nothing a packager can do when it's the packager who is introducing the
change. For example, if we're deviating from upsteam, then it's not a
stretch to patch in additional deprecation warnings for existing
releases (though that should have been done some time ago if the
deprecation is happening in F30). The package could provide additional
utilities to check for problematic principals and have them run when the
server starts. Certainly people have to check their logs, but the
server certainly has enough information to warn.
Of course, if I'm reading that wrong and this is merely an effect of
moving to version 1.18 then.... the current devel documentation doesn't
seem to say much about deprecation of additional algorithms. Which I
know doesn't mean much, but given how kerberos development progresses, I
would have expected the complete removal of an encryption type to be
telegraphed years in advance.
But after all that, the number of sites running a KDC on Fedora with old
principals has to be vanishingly small so I may be the only person who
would actually care.
- J<
10:57 p.m.
On Thu, 2019-01-03 at 22:40 -0600, Jason L Tibbitts III wrote:
> > > > >
But to be fair, MIT krb5 is not known for having great error output.
Not being able to start at all because the K/M has an enctype which is
acceptable and not at all deprecated according to the documentation that
exists today _and_ failing in the way the software tends to fail (with
obscure and sometimes numeric messages) would be... tough. Not that I
think anyone would just do dnf system-upgrade on their master KDC.
Anyone using FreeIPA and upgrading it is doing this, I guess. (Like
me...)
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
3:12 p.m.
Adam Williamson <adamwill(a)fedoraproject.org> writes:
On Thu, 2019-01-03 at 22:40 -0600, Jason L Tibbitts III wrote:
> But to be fair, MIT krb5 is not known for having great error output.
> Not being able to start at all because the K/M has an enctype which is
> acceptable and not at all deprecated according to the documentation that
> exists today _and_ failing in the way the software tends to fail (with
> obscure and sometimes numeric messages) would be... tough. Not that I
> think anyone would just do dnf system-upgrade on their master KDC.
Anyone using FreeIPA and upgrading it is doing this, I guess. (Like
me...)
We appreciate that you *do* do this because it means others will hit
fewer bugs that they need to roll back! But while krb5 version upgrades
are safe, distro upgrades aren't something that can be tested except in
a distro context.
Snapshots are always a good idea.
Thanks,
--Robbie
10:08 a.m.
On Mon, Jan 07, 2019 at 04:12:47PM -0500, Robbie Harwood wrote:
Adam Williamson <adamwill(a)fedoraproject.org> writes:
> On Thu, 2019-01-03 at 22:40 -0600, Jason L Tibbitts III wrote:
>
>> But to be fair, MIT krb5 is not known for having great error output.
>> Not being able to start at all because the K/M has an enctype which is
>> acceptable and not at all deprecated according to the documentation that
>> exists today _and_ failing in the way the software tends to fail (with
>> obscure and sometimes numeric messages) would be... tough. Not that I
>> think anyone would just do dnf system-upgrade on their master KDC.
>
> Anyone using FreeIPA and upgrading it is doing this, I guess. (Like
> me...)
We appreciate that you *do* do this because it means others will hit
fewer bugs that they need to roll back! But while krb5 version upgrades
are safe, distro upgrades aren't something that can be tested except in
a distro context.
Distro context? But we are talking about Fedora here, this couldn't
be less “distro context”.
Distro upgrade is a suggested way to keep one's installation at
supportable release (as opposed to reinstall), so distro-upgrade path
MUST be tested and working.
--
Tomasz Torcz 72->| 80->|
xmpp: zdzichubg(a)chrome.pl 72->| 80->|
12:45 p.m.
Tomasz Torcz <tomek(a)pipebreaker.pl> writes:
On Mon, Jan 07, 2019 at 04:12:47PM -0500, Robbie Harwood wrote:
> Adam Williamson <adamwill(a)fedoraproject.org> writes:
>
> > On Thu, 2019-01-03 at 22:40 -0600, Jason L Tibbitts III wrote:
> >
> >> But to be fair, MIT krb5 is not known for having great error output.
> >> Not being able to start at all because the K/M has an enctype which is
> >> acceptable and not at all deprecated according to the documentation that
> >> exists today _and_ failing in the way the software tends to fail (with
> >> obscure and sometimes numeric messages) would be... tough. Not that I
> >> think anyone would just do dnf system-upgrade on their master KDC.
> >
> > Anyone using FreeIPA and upgrading it is doing this, I guess. (Like
> > me...)
>
> We appreciate that you *do* do this because it means others will hit
> fewer bugs that they need to roll back! But while krb5 version upgrades
> are safe, distro upgrades aren't something that can be tested except in
> a distro context.
Distro context? But we are talking about Fedora here, this couldn't
be less “distro context”.
Distro upgrade is a suggested way to keep one's installation at
supportable release (as opposed to reinstall), so distro-upgrade path
MUST be tested and working.
I don't disagree with any of that. What I mean to say is that if you
hold all other package versions the same, krb5 upgrades are always safe
- but we can't, as krb5, make guarantees about what any other packages
do, especially our dependencies. The testing needs to happen when the
distro is assembled - i.e., in the context of the distro, not
beforehand.
Thanks,
--Robbie
1:31 p.m.
On ma, 14 tammi 2019, Robbie Harwood wrote:
Tomasz Torcz <tomek(a)pipebreaker.pl> writes:
> On Mon, Jan 07, 2019 at 04:12:47PM -0500, Robbie Harwood wrote:
>> Adam Williamson <adamwill(a)fedoraproject.org> writes:
>>
>> > On Thu, 2019-01-03 at 22:40 -0600, Jason L Tibbitts III wrote:
>> >
>> >> But to be fair, MIT krb5 is not known for having great error output.
>> >> Not being able to start at all because the K/M has an enctype which is
>> >> acceptable and not at all deprecated according to the documentation
that
>> >> exists today _and_ failing in the way the software tends to fail (with
>> >> obscure and sometimes numeric messages) would be... tough. Not that I
>> >> think anyone would just do dnf system-upgrade on their master KDC.
>> >
>> > Anyone using FreeIPA and upgrading it is doing this, I guess. (Like
>> > me...)
>>
>> We appreciate that you *do* do this because it means others will hit
>> fewer bugs that they need to roll back! But while krb5 version upgrades
>> are safe, distro upgrades aren't something that can be tested except in
>> a distro context.
>
> Distro context? But we are talking about Fedora here, this couldn't
> be less “distro context”.
> Distro upgrade is a suggested way to keep one's installation at
> supportable release (as opposed to reinstall), so distro-upgrade path
> MUST be tested and working.
I don't disagree with any of that. What I mean to say is that if you
hold all other package versions the same, krb5 upgrades are always safe
- but we can't, as krb5, make guarantees about what any other packages
do, especially our dependencies. The testing needs to happen when the
distro is assembled - i.e., in the context of the distro, not
beforehand.
With FreeIPA we attempt to do that in Fedora and it helped to find
quite
a few issues with FreeIPA components already. If there are specific
additional tests that can be exercised in deploying IPA server and a
client and then upgrading both of them in a sequence, using IPA Web UI
and other tools, feel free to suggest those scenarios.
For existing tests see
https://pagure.io/fedora-qa/os-autoinst-distri-fedora/blob/master/f/tests,
tests with freeipa and domain_controller in the names
--
/ Alexander Bokovoy
Sr. Principal Software Engineer
Security / Identity Management Engineering
Red Hat Limited, Finland
5:01 p.m.
Jason L Tibbitts III <tibbs(a)math.uh.edu> writes:
>>>>> "RH" == Robbie Harwood
<rharwood(a)redhat.com> writes:
RH> I really don't think that "it won't work and there'll be error
RH> messages" is an "extremely optimistic description".
But to be fair, MIT krb5 is not known for having great error output.
I've spent a nontrivial amount of time working on improving that, but am
always willing to process more bugs in the documentation/errors area.
krb5 only has five bugzillas in Fedora, so don't sweat giving me more to
do :)
RH> If you have langauge changes, I'm happy to hear them out.
I'm just not optimistic that it would be much better than "may fail in
a completely inexplicable fashion". Having a couple of principals
that can't get tickets because they lack a supported enctype is one
thing. (I've seen that and the error messages you get are OK, I
guess.) But having your entire database not load because it doesn't
the enctype on K/M is a different thing entirely. Note that currently
things do work fine if the enctype on K/M isn't in supported_enctypes.
It doesn't even warn. Presumably because supported_enctypes is
related to issuing tickets, not decrypting the database.
Correct.
But if the algorithms are disabled completely, then... I don't
know
what would happen.
My memory is that it complains about a bad enctype on K/M, but it's been
a while. If you want, as part of this change I can make sure it
generates a useful error message.
RH> I'm sure you're aware, then, that there's only so
much I can do as
RH> a packager here.
Well certainly there isn't much you can do to fix old principals on
existing systems. But the current versions should be complaining
loudly when it has to issue a ticket for a principal that lacks a
modern enctype today, if not six months ago.
Yes but also no. The problem is where to complain.
For most users, Kerberos functionally exists in library form (with some
CLI for various functions). Libraries can't log anywhere - which is why
krb5 has KRB5_TRACE. But unless something isn't working, there's no
reason to go look at the KRB5_TRACE output.
For folks running KDCs, we do have existing logging. But as an admin,
you know that I'm already logging every issuance - including their
etypes. (Now, the raw numbers aren't very helpful unless you're used to
looking at them - but that's why I've got
https://bugzilla.redhat.com/show_bug.cgi?id=1664156 .) This is pretty
chatty! I don't think that putting more messages in here would be
useful, but I could be persuaded otherwise.
Also, I'm curious, because I read all of this as "Fedora is
going to
patch upstream to disable certain algorithms because we don't think
the software should support them at all even if upstream does." If
that's a correct interpretation, then it's a bit disingenuous to say
there's nothing a packager can do when it's the packager who is
introducing the change. For example, if we're deviating from upsteam,
then it's not a stretch to patch in additional deprecation warnings
for existing releases (though that should have been done some time ago
if the deprecation is happening in F30).
The comment you're quoting is intended to refer to pushing changes to
non-KDC machines in the network (such as keytabs). My point was that I
can't make everything seamless. See above for a discussion of
"deprecation warnings".
The package could provide additional utilities to check for
problematic principals and have them run when the server starts.
Certainly people have to check their logs, but the server certainly
has enough information to warn.
As previously stated, I'm investigating providing tooling.
Of course, if I'm reading that wrong and this is merely an effect
of
moving to version 1.18 then.... the current devel documentation
doesn't seem to say much about deprecation of additional algorithms.
Which I know doesn't mean much, but given how kerberos development
progresses, I would have expected the complete removal of an
encryption type to be telegraphed years in advance.
Arguably, we telegraphed single-DES removal in 2009 (and there have been
two RFCs about this that I cited), but you're right that Kerberos has
never been a "move fast and deprecate things" kind of situation.
This has worked to our detriment and we shouldn't let tradition be the
enemy of improvement.
I don't want to go on too long about this, but the perception that
Kerberos is somehow "maintenance mode" isn't quite right. A lot of the
changes we've made recently haven't really been end-user visible (like
SPAKE), but things in the pipe will be more so (this crypto change,
FIDO/U2F support, etc.).
But after all that, the number of sites running a KDC on Fedora with
old principals has to be vanishingly small so I may be the only person
who would actually care.
Maybe in Fedora, but we're leading here. The concerns you're raising
are important for other distros that make these changes down the line.
I may not be happy about them (because they make this harder), but we
still need to talk them through :)
For Fedora specifically, I do think we have a lot of end users (even
just from kinit followed by fedpkg), a smaller-but-not-insignificant
number of service users (people running Kerberized services), and a
pretty small number of folks running full-on KDCs (whose testing is very
helpful) in comparison to other distros.
Thanks,
--Robbie
9:49 p.m.
>>>> "RH" == Robbie Harwood
<rharwood(a)redhat.com> writes:
RH> I've spent a nontrivial amount of time working on improving that,
RH> but am always willing to process more bugs in the
RH> documentation/errors area.
I know, and I don't mean to denigrate any work that's been done in
making the MIT KRB stack better. It's certainly a whole lot better than
it used to be. I've been using it since, I don't know, sometime in the
previous century.
RH> My memory is that it complains about a bad enctype on K/M, but it's
RH> been a while. If you want, as part of this change I can make sure
RH> it generates a useful error message.
Certainly don't add a message on my account, but on the general
principle that if krb5kdc can't start at all then it would try to say
why.
> Well certainly there isn't much you can do to fix old
principals on
> existing systems. But the current versions should be complaining
> loudly when it has to issue a ticket for a principal that lacks a
> modern enctype today, if not six months ago.
RH> Yes but also no. The problem is where to complain.
Maybe I wasn't clear, but I was saying that maybe the KDC should
complain when it has to issue a ticket with a key type that's
deprecated.
RH> For folks running KDCs, we do have existing logging.
Then I have to say that I'm not seeing it. For some background, here's
what I was recently working on.
I set up a test slave KDC with Fedora 29 and did some testing against
it. Trying to get a ticket using an old principal generated this bit of
log:
Dec 07 19:08:30 XXX krb5kdc[846](info): AS_REQ (7 etypes {18 17 16 23 1 3 2}) YYY:
NEEDED_PREAUTH: ZZZ for krbtgt/QQQ, Additional pre-authentication required
Dec 07 19:08:30 XXX krb5kdc[846](info): DISPATCH: repeated (retransmitted?) request from
YYY, resending previous response
The last line is repeated a number of times. There are no other logs
related to this principal.
The problem is that the principal in question hasn't been touched in
twelve years (which I guess you can see from the list of etypes.) That
coupled with the change (which I don't think was your doing) to
/etc/crypto-policies/back-ends/krb5.config to remove des3-cbc-sha1
caused the KDC to fail to issue those tickets.
So obviously I figured out what the problem was, but the logged info
wasn't remotely helpful in figuring that out.
Now, krb5kdc couldn't really know that F29 was going to drop something
from supported_enctypes, and there's no setting like
"deprecated_enctypes" which would have told it what to complain about
(though that doesn't sound like a terrible idea). But if the server had
said what the problem was, or there was something which bugged me about
principals having only deprecated etypes, then I wouldn't have had any
confusion at all.
RH> But as an admin, you know that I'm already logging every issuance -
RH> including their etypes.
Yes, but of course that doesn't help you if it doesn't issue anything.
I guess if you know the list of etypes by heart you could see the
problem from the logged info. And if it does issue something today but
will fail in F30 due to removal of some etypes, there's no complaint at
all.
RH> I don't think that putting more messages in here would be useful,
RH> but I could be persuaded otherwise.
Doesn't technically have to be more messages, even adding more
information to the existing message would help. Adding ", Only
deprecated etypes issued" to the end of the ISSUE lines would sort of be
in keeping with the existing format.
But really, whether it's in something that gets logged or some separate
tool I can run or something, it would just be nice to have some
established procedure to know that that you're going to be hit by
expiring crypto before you actually get hit by it.
> Of course, if I'm reading that wrong and this is merely an
effect of
> moving to version 1.18 then.... the current devel documentation
> doesn't seem to say much about deprecation of additional algorithms.
> Which I know doesn't mean much, but given how kerberos development
> progresses, I would have expected the complete removal of an
> encryption type to be telegraphed years in advance.
RH> Arguably, we telegraphed single-DES removal in 2009 (and there have
RH> been two RFCs about this that I cited), but you're right that
RH> Kerberos has never been a "move fast and deprecate things" kind of
RH> situation.
Yes, single-DES removal has been yelled about loudly for a long time.
There's a section in every release announcement about it. But isn't the
current proposal about removing more than just 1DES?
RH> I don't want to go on too long about this, but the perception that
RH> Kerberos is somehow "maintenance mode" isn't quite right.
I certainly don't harbor the belief that MIT Kerberos is just in
maintenance mode. I was trying to say that it moves carefully and with
significant effort put into backwards compatibility.
- J<
2:37 p.m.
Jason L Tibbitts III <tibbs(a)math.uh.edu> writes:
>>>>> "RH" == Robbie Harwood
<rharwood(a)redhat.com> writes:
>>
>> Well certainly there isn't much you can do to fix old principals on
>> existing systems. But the current versions should be complaining
>> loudly when it has to issue a ticket for a principal that lacks a
>> modern enctype today, if not six months ago.
RH> For folks running KDCs, we do have existing logging.
The problem is that the principal in question hasn't been touched in
twelve years (which I guess you can see from the list of etypes.) That
coupled with the change (which I don't think was your doing) to
/etc/crypto-policies/back-ends/krb5.config to remove des3-cbc-sha1
caused the KDC to fail to issue those tickets.
Ah, I see, you're talking about the case when the enctype is already not
permitted. That all makes sense then.
RH> I don't think that putting more messages in here would be
useful,
RH> but I could be persuaded otherwise.
Doesn't technically have to be more messages, even adding more
information to the existing message would help. Adding ", Only
deprecated etypes issued" to the end of the ISSUE lines would sort of be
in keeping with the existing format.
But really, whether it's in something that gets logged or some separate
tool I can run or something, it would just be nice to have some
established procedure to know that that you're going to be hit by
expiring crypto before you actually get hit by it.
This is good feedback, thanks. My plan is to try for an external "realm
check" type tool - but I need to make it first :)
>> Of course, if I'm reading that wrong and this is merely
an effect of
>> moving to version 1.18 then.... the current devel documentation
>> doesn't seem to say much about deprecation of additional algorithms.
>> Which I know doesn't mean much, but given how kerberos development
>> progresses, I would have expected the complete removal of an
>> encryption type to be telegraphed years in advance.
RH> Arguably, we telegraphed single-DES removal in 2009 (and there have
RH> been two RFCs about this that I cited), but you're right that
RH> Kerberos has never been a "move fast and deprecate things" kind of
RH> situation.
Yes, single-DES removal has been yelled about loudly for a long time.
There's a section in every release announcement about it. But isn't the
current proposal about removing more than just 1DES?
It may help understanding my motivations here to know that upstream
won't consider removal until some distro has done so and they have a
sense of what it entails. Until then, it's a bit Catch-22: no one wants
to divert from upstream and upstream doesn't want to break
downstreams... We've yet to remove anything of import from any Kerberos
implementation; MIT still includes Kerberos4 support, for instance.
The MIT-specific telegraphing is single-DES specific. The RFCs (which
also include RC4 and 3DES) are applicable to the entire ecosystem. But
ignoring all that, fundamentally I don't want to do this same process
three times - and I can't imagine admins do, either. Which is to say: I
think that the only telegraphing I can meaningfully do is administrative
(RFCs, release notes, change proposals, etc.), and then try to smooth
the roadbumps.
Thanks,
--Robbie
4:45 p.m.
>>>> "RH" == Robbie Harwood
<rharwood(a)redhat.com> writes:
RH> Ah, I see, you're talking about the case when the enctype is already
RH> not permitted. That all makes sense then.
Right. Basically, if any one of these:
* Warnings in previous versions about principals without modern etypes
* Logging in the new version to say why tickets wouldn't issue for
principals with old etypes
* A checkup tool in either the old or current versions to tell me what's
gone wrong
had existed then there would have been no confusion. Certainly I was
able to figure it out but... if someone had just done an OS update
without proper testing then they could be in a pretty bad position.i
So basically the big issue as I see it is that there's simply nothing to
tell you that things are going to break, and after the update there's
nothing that tells you why things are broken. And I was concerned that
if some encryption routines go away completely then it would be possible
to be in a state where you can't even decrypt the database.
- J<
7:20 a.m.
On Tue, Jan 08, 2019 at 04:45:53PM -0600, Jason L Tibbitts III wrote:
>>>>> "RH" == Robbie Harwood
<rharwood(a)redhat.com> writes:
RH> Ah, I see, you're talking about the case when the enctype is already
RH> not permitted. That all makes sense then.
Hi,
after re-reading this thread, I'm still unclear on some issues. Please
correct me if I'm wrong.
- The plan is to patch the Fedora package to remove support for some
algorithms above and beyond what upstream is removing right now.
- Current implementation in F29 does not warn that those algorithms
will become unimplemented.
- Because of the combination of two previous points, users who simply
upgrade to F30 without paying attention will have to temporarily
downgrade to the F29 version, perform key roll-over, and only then
upgrade.
If this understanding is correct, this seems like a trap for the unwary.
What about turning the removal into a big warning in the logs each
time kdc is started and needs to use those deprecated algorithms in
F30, and only doing the removal in F31?
Zbyszek
>
> Right. Basically, if any one of these:
>
> * Warnings in previous versions about principals without modern etypes
> * Logging in the new version to say why tickets wouldn't issue for
> principals with old etypes
> * A checkup tool in either the old or current versions to tell me what's
> gone wrong
>
> had existed then there would have been no confusion. Certainly I was
> able to figure it out but... if someone had just done an OS update
> without proper testing then they could be in a pretty bad position.i
>
> So basically the big issue as I see it is that there's simply nothing to
> tell you that things are going to break, and after the update there's
> nothing that tells you why things are broken. And I was concerned that
> if some encryption routines go away completely then it would be possible
> to be in a state where you can't even decrypt the database.
>
> - J<
> _______________________________________________
> devel mailing list -- devel(a)lists.fedoraproject.org
> To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
2:10 p.m.
Zbigniew Jędrzejewski-Szmek <zbyszek(a)in.waw.pl> writes:
On Tue, Jan 08, 2019 at 04:45:53PM -0600, Jason L Tibbitts III
wrote:
> >>>>> "RH" == Robbie Harwood <rharwood(a)redhat.com>
writes:
>
> RH> Ah, I see, you're talking about the case when the enctype is already
> RH> not permitted. That all makes sense then.
Hi,
after re-reading this thread, I'm still unclear on some issues. Please
correct me if I'm wrong.
- The plan is to patch the Fedora package to remove support for some
algorithms above and beyond what upstream is removing right now.
Upstream has never removed an algorithm. Hopefully the Fedora changes
will allow us to do so, both by providing the code, and by showing that
the fallout isn't catastrophic.
- Current implementation in F29 does not warn that those algorithms
will become unimplemented.
- Because of the combination of two previous points, users who simply
upgrade to F30 without paying attention will have to temporarily
downgrade to the F29 version, perform key roll-over, and only then
upgrade.
That's not a consequence of the previous two points but is nontheless
true.
If this understanding is correct, this seems like a trap for the
unwary.
So is keeping any of these algorithms around.
What about turning the removal into a big warning in the logs each
time kdc is started and needs to use those deprecated algorithms in
F30,
Still in progress. It may prove too invasive to be feasible.
If I can't, then I'll be providing external tooling that will check for
some common-ish legacy cruft in setups that might break.
Also, I don't know if you've ever looked at krb5kdc logs, but they're
really chatty - I doubt anyone reads regularly them unless they're
trying to debug a problem.
and only doing the removal in F31?
It may come to that. If it does, then I want to be sure everything is
lined up for it, and we won't slip to fc32 or later.
Finally, I do want to call out here that announcing crypto deprecations
in the previous release is not the norm for Fedora. Since fc20, there
have been seven crypto Changes (mostly to do with adding crypto-policies
support), none of which performed any notification other than what was
in the release notes:
https://fedoraproject.org/wiki/Changes/RemoveSSL3andRc4
https://fedoraproject.org/wiki/Changes/CryptoPolicyKrb5
https://fedoraproject.org/wiki/Changes/NSSCryptoPolicies
https://fedoraproject.org/wiki/Changes/OpenSSH_Crypto_Policy
https://fedoraproject.org/wiki/Changes/Remove_SSH-1_from_OpenSSH
https://fedoraproject.org/wiki/Changes/OpenSSH_Server_Crypto_Policy
https://fedoraproject.org/wiki/Changes/StrongCryptoSettings
Furthermore, at no point have there even been Change/Feature pages for
all the TLS changes relating to vulnerable ciphers.
Thanks,
--Robbie
12:50 a.m.
> after re-reading this thread, I'm still unclear on some
issues. Please
> correct me if I'm wrong.
>
> - The plan is to patch the Fedora package to remove support for some
> algorithms above and beyond what upstream is removing right now.
Upstream has never removed an algorithm. Hopefully the Fedora changes
will allow us to do so, both by providing the code, and by showing that
the fallout isn't catastrophic.
> - Current implementation in F29 does not warn that those algorithms
> will become unimplemented.
>
> - Because of the combination of two previous points, users who simply
> upgrade to F30 without paying attention will have to temporarily
> downgrade to the F29 version, perform key roll-over, and only then
> upgrade.
That's not a consequence of the previous two points but is nontheless
true.
> If this understanding is correct, this seems like a trap for the unwary.
So is keeping any of these algorithms around.
Maybe a middle ground can be found here:
- remove support for legacy algorithms
- let it blow in production in a helpful manner
- provide legacy algorithms support for key roll-over only
For the last point, that could either be a version of the library
outside of the default library path, or a static program dedicated to
that task. This way you pull the rug under the users' feet but you
don't break their legs in the process. And maybe this is something
that should be discussed upstream before making such a change in
Fedora so they finally have a means to remove legacy algorithms and
still offer an upgrade path towards proper crypto for existing
installations.
Dridi
1:14 p.m.
Jason L Tibbitts III <tibbs(a)math.uh.edu> writes:
>>>>> "RH" == Robbie Harwood
<rharwood(a)redhat.com> writes:
RH> Ah, I see, you're talking about the case when the enctype is already
RH> not permitted. That all makes sense then.
Right. Basically, if any one of these:
* Warnings in previous versions about principals without modern etypes
* Logging in the new version to say why tickets wouldn't issue for
principals with old etypes
* A checkup tool in either the old or current versions to tell me what's
gone wrong
had existed then there would have been no confusion. Certainly I was
able to figure it out but... if someone had just done an OS update
without proper testing then they could be in a pretty bad position.i
So basically the big issue as I see it is that there's simply nothing
to tell you that things are going to break, and after the update
there's nothing that tells you why things are broken. And I was
concerned that if some encryption routines go away completely then it
would be possible to be in a state where you can't even decrypt the
database.
I have added warnings for:
- deprecated enctype issuance on KDC
- deprecated enctypes in klist output
- deprecated enctype in stash / K/M
to the latest rawhide builds (krb5-1.17-3.fc30).
If I backport this to fc29, will that assuage people's concerns? Or do
I need to defer the change until fc31?
Thanks,
--Robbie
1:24 p.m.
>>>> "RH" == Robbie Harwood
<rharwood(a)redhat.com> writes:
RH> If I backport this to fc29, will that assuage people's concerns?
I think it would certainly help and I wouldn't complain. In fact, I'd
love to start running that as soon as I can. However, it wouldn't help
anyone who does a (supposedly supported) F28->F30 update. I do not know
if that is a significant concern.
- J<
4:01 p.m.
Jason L Tibbitts III <tibbs(a)math.uh.edu> writes:
>>>>> "RH" == Robbie Harwood
<rharwood(a)redhat.com> writes:
RH> If I backport this to fc29, will that assuage people's concerns?
I think it would certainly help and I wouldn't complain. In fact, I'd
love to start running that as soon as I can. However, it wouldn't
help anyone who does a (supposedly supported) F28->F30 update. I do
not know if that is a significant concern.
Nothing would break, so it's technically correct to leave out right now
I think. Since my f28 builds match f29 right now, I just did both:
fc28: https://bodhi.fedoraproject.org/updates/FEDORA-2019-0956d60ffd
fc29: https://bodhi.fedoraproject.org/updates/FEDORA-2019-d226c3ecf1
I've also updated the change page to mention that these warnings are
present (and the versions they start appearing in).
Thanks,
--Robbie
4:08 p.m.
On Thu, Jan 17, 2019 at 05:01:45PM -0500, Robbie Harwood wrote:
Jason L Tibbitts III <tibbs(a)math.uh.edu> writes:
>>>>>> "RH" == Robbie Harwood <rharwood(a)redhat.com>
writes:
>
> RH> If I backport this to fc29, will that assuage people's concerns?
>
> I think it would certainly help and I wouldn't complain. In fact, I'd
> love to start running that as soon as I can. However, it wouldn't
> help anyone who does a (supposedly supported) F28->F30 update. I do
> not know if that is a significant concern.
Nothing would break, so it's technically correct to leave out right now
I think. Since my f28 builds match f29 right now, I just did both:
fc28: https://bodhi.fedoraproject.org/updates/FEDORA-2019-0956d60ffd
fc29: https://bodhi.fedoraproject.org/updates/FEDORA-2019-d226c3ecf1
I've also updated the change page to mention that these warnings are
present (and the versions they start appearing in).
So that gives 4+ months of warnings (assuming the F28 and F29 updates
go to stable soon), which should be enough for people to take notice.
It seems it's fine to remove in F30 as originally planned.
Zbyszek
1923
days inactive
1950
days old
25 comments
10 participants
participants (10)
-
Adam Williamson -
Alexander Bokovoy -
Ben Cotton -
Dridi Boukelmoune -
Jason L Tibbitts III -
Nikos Mavrogiannopoulos -
Raphael Groner -
Robbie Harwood -
Tomasz Torcz -
Zbigniew Jędrzejewski-Szmek