6:14 a.m.
Hi Daiki & Frantisek,
There's a new error that is appearing in the libnbd test suite when
testing TLS-PSK. Regular TLS (with X.509 certs) works fine. It seems
to have started since I upgraded the kernel on my machine from 5.19.0 ->
6.1.0, and I think it is related to kTLS.
You may be able to reproduce it fairly easily in Fedora Rawhide, or in
Fedora 37 by upgrading the kernel, nbdkit and libnbd to Rawhide versions.
$ uname -a
Linux pick.home.annexia.org 6.1.0-0.rc6.46.fc38.x86_64 #1 SMP PREEMPT_DYNAMIC Mon Nov 21
16:07:44 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
$ nbdkit --version
nbdkit 1.33.3 (nbdkit-1.33.3-1.fc38)
$ nbdinfo --version
nbdinfo 1.15.7
libnbd 1.15.7
To reproduce it:
$ psktool -u bob -p keys.psk
Generating a random key for user 'bob'
Key stored to keys.psk
$ nbdkit --tls=require --tls-psk=keys.psk null \
--run 'nbdinfo "nbds://bob@localhost/?tls-psk-file=keys.psk"
'
nbdkit: null[1]: error: gnutls_record_recv: Error in the pull function.
nbdkit: null[1]: error: reading option: conn->recv: Input/output error
nbdinfo: nbd_connect_uri: gnutls_record_recv: Error in the pull function.
For lots more debugging, use this command instead:
$ nbdkit -fv --tls=require --tls-psk=keys.psk \
-D nbdkit.tls.log=99 -D nbdkit.tls.session=1 null \
--run 'LIBNBD_DEBUG=1 nbdinfo
"nbds://bob@localhost/?tls-psk-file=keys.psk" '
The reason I believe it is related to kTLS is because if I do:
# modprobe -r tls
then the error goes away. Loading the module makes the error appear
again. (Note that the module appears to be loaded on boot, so this
error will happen for all Rawhide users unless they take special
action.)
Are there ways to debug kTLS? It seems like there is no kernel output
related to the above failure.
Are there ways to override GnuTLS automatic detection of kTLS, to
temporarily disable it, even when the kernel module is loaded?
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW
6:43 a.m.
New subject: Potential kTLS issue with TLS-PSK, GnuTLS + Rawhide - how to
debug it?
On Fri, Nov 25, 2022 at 1:14 PM Richard W.M. Jones <rjones(a)redhat.com> wrote:
Hi Daiki & Frantisek,
There's a new error that is appearing in the libnbd test suite when
testing TLS-PSK. Regular TLS (with X.509 certs) works fine. It seems
to have started since I upgraded the kernel on my machine from 5.19.0 ->
6.1.0, and I think it is related to kTLS.
You may be able to reproduce it fairly easily in Fedora Rawhide, or in
Fedora 37 by upgrading the kernel, nbdkit and libnbd to Rawhide versions.
$ uname -a
Linux pick.home.annexia.org 6.1.0-0.rc6.46.fc38.x86_64 #1 SMP PREEMPT_DYNAMIC Mon Nov
21 16:07:44 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
$ nbdkit --version
nbdkit 1.33.3 (nbdkit-1.33.3-1.fc38)
$ nbdinfo --version
nbdinfo 1.15.7
libnbd 1.15.7
To reproduce it:
$ psktool -u bob -p keys.psk
Generating a random key for user 'bob'
Key stored to keys.psk
$ nbdkit --tls=require --tls-psk=keys.psk null \
--run 'nbdinfo "nbds://bob@localhost/?tls-psk-file=keys.psk"
'
nbdkit: null[1]: error: gnutls_record_recv: Error in the pull function.
nbdkit: null[1]: error: reading option: conn->recv: Input/output error
nbdinfo: nbd_connect_uri: gnutls_record_recv: Error in the pull function.
For lots more debugging, use this command instead:
$ nbdkit -fv --tls=require --tls-psk=keys.psk \
-D nbdkit.tls.log=99 -D nbdkit.tls.session=1 null \
--run 'LIBNBD_DEBUG=1 nbdinfo
"nbds://bob@localhost/?tls-psk-file=keys.psk" '
The reason I believe it is related to kTLS is because if I do:
# modprobe -r tls
then the error goes away. Loading the module makes the error appear
again. (Note that the module appears to be loaded on boot, so this
error will happen for all Rawhide users unless they take special
action.)
Are there ways to debug kTLS? It seems like there is no kernel output
related to the above failure.
Are there ways to override GnuTLS automatic detection of kTLS, to
temporarily disable it, even when the kernel module is loaded?
For disabling KTLS, try putting
```
[global]
ktls = false
```
into `/etc/crypto-policies/local.d/gnutls-no-ktls.config`,
and follow up with an `update-crypto-policies --set`.
7:22 a.m.
New subject: Potential kTLS issue with TLS-PSK, GnuTLS + Rawhide - how to
debug it?
On Fri, Nov 25, 2022 at 01:43:18PM +0100, Alexander Sosedkin wrote:
On Fri, Nov 25, 2022 at 1:14 PM Richard W.M. Jones
<rjones(a)redhat.com> wrote:
>
> Hi Daiki & Frantisek,
>
> There's a new error that is appearing in the libnbd test suite when
> testing TLS-PSK. Regular TLS (with X.509 certs) works fine. It seems
> to have started since I upgraded the kernel on my machine from 5.19.0 ->
> 6.1.0, and I think it is related to kTLS.
>
> You may be able to reproduce it fairly easily in Fedora Rawhide, or in
> Fedora 37 by upgrading the kernel, nbdkit and libnbd to Rawhide versions.
>
> $ uname -a
> Linux pick.home.annexia.org 6.1.0-0.rc6.46.fc38.x86_64 #1 SMP PREEMPT_DYNAMIC Mon
Nov 21 16:07:44 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
>
> $ nbdkit --version
> nbdkit 1.33.3 (nbdkit-1.33.3-1.fc38)
> $ nbdinfo --version
> nbdinfo 1.15.7
> libnbd 1.15.7
>
> To reproduce it:
>
> $ psktool -u bob -p keys.psk
> Generating a random key for user 'bob'
> Key stored to keys.psk
>
> $ nbdkit --tls=require --tls-psk=keys.psk null \
> --run 'nbdinfo
"nbds://bob@localhost/?tls-psk-file=keys.psk" '
> nbdkit: null[1]: error: gnutls_record_recv: Error in the pull function.
> nbdkit: null[1]: error: reading option: conn->recv: Input/output error
> nbdinfo: nbd_connect_uri: gnutls_record_recv: Error in the pull function.
>
> For lots more debugging, use this command instead:
>
> $ nbdkit -fv --tls=require --tls-psk=keys.psk \
> -D nbdkit.tls.log=99 -D nbdkit.tls.session=1 null \
> --run 'LIBNBD_DEBUG=1 nbdinfo
"nbds://bob@localhost/?tls-psk-file=keys.psk" '
>
> The reason I believe it is related to kTLS is because if I do:
>
> # modprobe -r tls
>
> then the error goes away. Loading the module makes the error appear
> again. (Note that the module appears to be loaded on boot, so this
> error will happen for all Rawhide users unless they take special
> action.)
>
> Are there ways to debug kTLS? It seems like there is no kernel output
> related to the above failure.
>
> Are there ways to override GnuTLS automatic detection of kTLS, to
> temporarily disable it, even when the kernel module is loaded?
For disabling KTLS, try putting
```
[global]
ktls = false
```
into `/etc/crypto-policies/local.d/gnutls-no-ktls.config`,
and follow up with an `update-crypto-policies --set`.
Aha! Actually I already have:
$ cat /etc/crypto-policies/local.d/gnutls-ktls.config
[global]
ktls = true
which I must have added a while back and forgotten about.
Removing that file disables kTLS. So in fact what I said above isn't
right - this will not affect Rawhide users unless they also enable
kTLS explicitly, which is good to know.
Still interested in debugging the kTLS problem.
I've been having a look at straces, and what's interesting is that the
client sends and the server receives the cleartext message
successfully (as we would expect because the kernel is doing the
encryption and decryption below the socket level), but something
(GnuTLS?) believes that there was a socket error, even though there
isn't one.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW
7:57 a.m.
New subject: Potential kTLS issue with TLS-PSK, GnuTLS + Rawhide - how to
debug it?
Turns out this is fixed in upstream gnutls (not the version in
Rawhide). The commit which fixes it is:
commit 67843b3a8e28e4c74296caea2d1019065c87afb3
Author: Frantisek Krenzelok <krenzelok.frantisek(a)gmail.com>
Date: Mon Sep 5 13:05:17 2022 +0200
KTLS: fallback to default
If an error occurs during setting of keys either initial or key update
then fallback to default mode of operation (disable ktls) and let the
user know
Signed-off-by: Frantisek Krenzelok <krenzelok.frantisek(a)gmail.com>
lib/handshake.c | 7 ++++++-
lib/tls13/key_update.c | 23 +++++++++++++++++++----
2 files changed, 25 insertions(+), 5 deletions(-)
With full debugging you can see the message caused by this commit:
nbdkit: null[1]: debug: gnutls: 4: HSK[0x7fc9e00010a0]: TLS 1.3 set read key with cipher
suite: GNUTLS_CHACHA20_POLY1305_SHA256
nbdkit: null[1]: debug: gnutls: 13: BUF[HSK]: Emptied buffer
nbdkit: null[1]: debug: gnutls: 13: BUF[HSK]: Emptied buffer
nbdkit: null[1]: debug: gnutls: 5: REC[0x7fc9e00010a0]: Start of epoch cleanup
nbdkit: null[1]: debug: gnutls: 5: REC[0x7fc9e00010a0]: Epoch #0 freed
nbdkit: null[1]: debug: gnutls: 5: REC[0x7fc9e00010a0]: Epoch #1 freed
nbdkit: null[1]: debug: gnutls: 5: REC[0x7fc9e00010a0]: End of epoch cleanup
nbdkit: null[1]: debug: gnutls: 1: disabling KTLS: failed to set keys
Is this because kTLS doesn't support PSK?
Anyway I will file a bug to add this commit to Rawhide.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines. Supports shell scripting,
bindings from many languages. http://libguestfs.org
8:58 a.m.
New subject: Potential kTLS issue with TLS-PSK, GnuTLS + Rawhide - how to debug it?
On Friday, 25 November 2022 13:57:26 GMT Richard W.M. Jones wrote:
Turns out this is fixed in upstream gnutls (not the version in
Rawhide). The commit which fixes it is:
commit 67843b3a8e28e4c74296caea2d1019065c87afb3
Author: Frantisek Krenzelok <krenzelok.frantisek(a)gmail.com>
Date: Mon Sep 5 13:05:17 2022 +0200
KTLS: fallback to default
If an error occurs during setting of keys either initial or key update
then fallback to default mode of operation (disable ktls) and let the
user know
Signed-off-by: Frantisek Krenzelok <krenzelok.frantisek(a)gmail.com>
lib/handshake.c | 7 ++++++-
lib/tls13/key_update.c | 23 +++++++++++++++++++----
2 files changed, 25 insertions(+), 5 deletions(-)
With full debugging you can see the message caused by this commit:
nbdkit: null[1]: debug: gnutls: 4: HSK[0x7fc9e00010a0]: TLS 1.3 set read key
with cipher suite: GNUTLS_CHACHA20_POLY1305_SHA256
nbdkit: null[1]: debug:
gnutls: 13: BUF[HSK]: Emptied buffer
nbdkit: null[1]: debug: gnutls: 13: BUF[HSK]: Emptied buffer
nbdkit: null[1]: debug: gnutls: 5: REC[0x7fc9e00010a0]: Start of epoch
cleanup
nbdkit: null[1]: debug: gnutls: 5: REC[0x7fc9e00010a0]: Epoch #0
freed nbdkit: null[1]: debug: gnutls: 5: REC[0x7fc9e00010a0]: Epoch
#1
freed nbdkit: null[1]: debug: gnutls: 5: REC[0x7fc9e00010a0]: End of epoch
cleanup nbdkit: null[1]: debug: gnutls: 1: disabling KTLS: failed to set
keys
Is this because kTLS doesn't support PSK?
kTLS doesn't do key management, so it doesn't know the difference between
PSK
and X.509 (it gets the symmetric keys from userspace after the key exchange is
done).
However, looking at https://gitlab.com/gnutls/gnutls/-/blob/master/lib/system/
ktls.c, GnuTLS only supports using kTLS with GNUTLS_CIPHER_AES_128_GCM and
GNUTLS_CIPHER_AES_256_GCM cipher suites, while your debugging output shows
that this connection is using GNUTLS_CHACHA20_POLY1305_SHA256.
As a result, the upstream fix you point to does the right thing because you
can't use GnuTLS's kTLS support on this connection (at all), and the only
thing that makes sense is to disable kTLS.
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/
include/uapi/linux/tls.h suggests that kTLS has some degree of support for
ciphers other than the two GnuTLS enables it for, but I don't know enough
about TLS to know whether enabling all the ciphers current kernels support in
GnuTLS would enable it to be used for this connection.
--
Simon Farnsworth
9:11 p.m.
New subject: Potential kTLS issue with TLS-PSK, GnuTLS + Rawhide - how to
debug it?
Hello Simon,
Thanks for looking into it!
Simon Farnsworth <simon(a)farnz.org.uk> writes:
On Friday, 25 November 2022 13:57:26 GMT Richard W.M. Jones wrote:
> Turns out this is fixed in upstream gnutls (not the version in
> Rawhide). The commit which fixes it is:
>
> commit 67843b3a8e28e4c74296caea2d1019065c87afb3
> Author: Frantisek Krenzelok <krenzelok.frantisek(a)gmail.com>
> Date: Mon Sep 5 13:05:17 2022 +0200
>
> KTLS: fallback to default
>
> If an error occurs during setting of keys either initial or key update
> then fallback to default mode of operation (disable ktls) and let the
> user know
>
> Signed-off-by: Frantisek Krenzelok <krenzelok.frantisek(a)gmail.com>
>
> lib/handshake.c | 7 ++++++-
> lib/tls13/key_update.c | 23 +++++++++++++++++++----
> 2 files changed, 25 insertions(+), 5 deletions(-)
>
> With full debugging you can see the message caused by this commit:
>
> nbdkit: null[1]: debug: gnutls: 4: HSK[0x7fc9e00010a0]: TLS 1.3 set read key
> with cipher suite: GNUTLS_CHACHA20_POLY1305_SHA256
nbdkit: null[1]: debug:
> gnutls: 13: BUF[HSK]: Emptied buffer
> nbdkit: null[1]: debug: gnutls: 13: BUF[HSK]: Emptied buffer
> nbdkit: null[1]: debug: gnutls: 5: REC[0x7fc9e00010a0]: Start of epoch
> cleanup
nbdkit: null[1]: debug: gnutls: 5: REC[0x7fc9e00010a0]: Epoch #0
> freed nbdkit: null[1]: debug: gnutls: 5: REC[0x7fc9e00010a0]: Epoch #1
> freed nbdkit: null[1]: debug: gnutls: 5: REC[0x7fc9e00010a0]: End of epoch
> cleanup nbdkit: null[1]: debug: gnutls: 1: disabling KTLS: failed to set
> keys
> Is this because kTLS doesn't support PSK?
>
kTLS doesn't do key management, so it doesn't know the difference between PSK
and X.509 (it gets the symmetric keys from userspace after the key exchange is
done).
However, looking at https://gitlab.com/gnutls/gnutls/-/blob/master/lib/system/
ktls.c, GnuTLS only supports using kTLS with GNUTLS_CIPHER_AES_128_GCM and
GNUTLS_CIPHER_AES_256_GCM cipher suites, while your debugging output shows
that this connection is using GNUTLS_CHACHA20_POLY1305_SHA256.
Yes, that is the case. With gnutls-cli:
$ gnutls-cli -d1 -p 5556 localhost --pskusername psk_identity --pskkey "..."
--priority NORMAL:-KX-ALL:+ECDHE-PSK
|<1>| disabling KTLS: failed to set keys
If you disable CHACHA20-POLY1305, it works without that message:
$ gnutls-cli -d1 -p 5556 localhost --pskusername psk_identity --pskkey "..."
--priority NORMAL:-CHACHA20-POLY1305:-KX-ALL:+ECDHE-PSK
I've filed an issue so those ciphersuites are eventually supported:
https://gitlab.com/gnutls/gnutls/-/issues/1434
Regards,
--
Daiki Ueno
524
days inactive
527
days old
5 comments
4 participants
participants (4)
-
Alexander Sosedkin -
Daiki Ueno -
Richard W.M. Jones -
Simon Farnsworth