So, as part of our outages yesterday I reinstalled bastion01 (and 02 a few days before) with rhel9. This means it's ssh host key changed.
However, if you are setup right this should be a non event. :)
There's at least 2 ways you can confirm the new new is right:
1. Enable sshfp:
Add in your .ssh/config the following to the entry for bastion/fedora-infrastructure hosts:
VerifyHostKeyDNS yes
This will get the ssh fingerprint from dns and confirm it matches.
2. Add our ssh cert authority to your ~/.ssh/known_hosts file. This can be found at: https://admin.fedoraproject.org/ssh_known_hosts Just add those lines to your known_hosts and ssh will verify and trust any ssh host key thats signed by those certificate authorities. (which we do for all hosts).
Sorry for any trouble.
kevin
Hi -
So, as part of our outages yesterday I reinstalled bastion01 (and 02 a few days before) with rhel9. This means it's ssh host key changed.
There was no way of saving & restoring the former host key?
- Enable sshfp:
Add in your .ssh/config the following to the entry for bastion/fedora-infrastructure hosts: VerifyHostKeyDNS yes This will get the ssh fingerprint from dns and confirm it matches.
(This is enabled but f37 ssh still rejects the change.)
- FChE
On Thu, Mar 30, 2023 at 11:57:45AM -0400, Frank Ch. Eigler wrote:
Hi -
So, as part of our outages yesterday I reinstalled bastion01 (and 02 a few days before) with rhel9. This means it's ssh host key changed.
There was no way of saving & restoring the former host key?
I could get it back. But at this point I think it's better to figure out how to never hit it again. ;)
- Enable sshfp:
Add in your .ssh/config the following to the entry for bastion/fedora-infrastructure hosts: VerifyHostKeyDNS yes This will get the ssh fingerprint from dns and confirm it matches.
(This is enabled but f37 ssh still rejects the change.)
Odd. It definitely used to work. I can look more when I get a chance. I did update the SSHFP records.
kevin
Kevin Fenzi wrote:
On Thu, Mar 30, 2023 at 11:57:45AM -0400, Frank Ch. Eigler wrote:
Hi -
So, as part of our outages yesterday I reinstalled bastion01 (and 02 a few days before) with rhel9. This means it's ssh host key changed.
There was no way of saving & restoring the former host key?
I could get it back. But at this point I think it's better to figure out how to never hit it again. ;)
- Enable sshfp:
Add in your .ssh/config the following to the entry for bastion/fedora-infrastructure hosts: VerifyHostKeyDNS yes This will get the ssh fingerprint from dns and confirm it matches.
(This is enabled but f37 ssh still rejects the change.)
Odd. It definitely used to work. I can look more when I get a chance. I did update the SSHFP records.
The VerifyHostKeyDNS does require secure DNS to avoid any confirmation prompt. Without DNSSEC, `VerifyHostKeyDNS yes` is the same as `VerifyHostKeyDNS ask`.
Perhaps that's the issue in Frank's case?
Hi -
The VerifyHostKeyDNS does require secure DNS to avoid any confirmation prompt. Without DNSSEC, `VerifyHostKeyDNS yes` is the same as `VerifyHostKeyDNS ask`.
OK, that's one thing to check/fix.
Perhaps that's the issue in Frank's case?
Plus: bastion-iad01.fedoraproject.org. appears to lack the SSHFP records.
- FChE
On Thu, Mar 30, 2023 at 04:11:45PM -0400, Frank Ch. Eigler wrote:
Hi -
The VerifyHostKeyDNS does require secure DNS to avoid any confirmation prompt. Without DNSSEC, `VerifyHostKeyDNS yes` is the same as `VerifyHostKeyDNS ask`.
OK, that's one thing to check/fix.
Ah yeah, that could well be the case.
Perhaps that's the issue in Frank's case?
Plus: bastion-iad01.fedoraproject.org. appears to lack the SSHFP records.
We should drop that from dns. It was only used when we were moving from phx2 to iad2. There should be 3 things in dns:
bastion01.fedoraproject.org bastion02.fedoraproject.org bastion.fedoraproject.org
with the last one being a CNAME for whichever one is default (currently 01).
Anyhow, the ssh access SOP should be updated with all this info.
Probibly sshfp isn't worth doing these days and we should just stick with the certificate signed config.
kevin
On 30/03/2023 22:25, Kevin Fenzi wrote:
On Thu, Mar 30, 2023 at 04:11:45PM -0400, Frank Ch. Eigler wrote:
Hi -
The VerifyHostKeyDNS does require secure DNS to avoid any confirmation prompt. Without DNSSEC, `VerifyHostKeyDNS yes` is the same as `VerifyHostKeyDNS ask`.
OK, that's one thing to check/fix.
Ah yeah, that could well be the case.
Perhaps that's the issue in Frank's case?
Plus: bastion-iad01.fedoraproject.org. appears to lack the SSHFP records.
We should drop that from dns. It was only used when we were moving from phx2 to iad2. There should be 3 things in dns:
bastion01.fedoraproject.org bastion02.fedoraproject.org bastion.fedoraproject.org
with the last one being a CNAME for whichever one is default (currently 01).
Anyhow, the ssh access SOP should be updated with all this info.
Probibly sshfp isn't worth doing these days and we should just stick with the certificate signed config.
+1 to that .. While it's possible to automate SSHFP (to a kind of delegated/dynamic zone), I prefer just relying on signed host certificate : just trust the CA once and enjoy forever :)
FWIW, that's also what we do for centos.org infra
We should drop that from dns. [...] Anyhow, the ssh access SOP should be updated with all this info.
I looked for the SOP and found this: https://docs.fedoraproject.org/en-US/infra/sysadmin_guide/sshaccess/ It still mentions bastion-iad01. Am I on the wrong docs? It looks like the right place.
I also stumbled upon https://fedora-infra-docs.readthedocs.io/ in my bookmarks and it's been squatted. It's built from https://github.com/primitivea/fedora-infra-docs, but that's a different issue altogether.
Aurélien
On Fri, Mar 31, 2023 at 10:30:41AM +0200, Aurelien Bompard wrote:
We should drop that from dns. [...] Anyhow, the ssh access SOP should be updated with all this info.
I looked for the SOP and found this: https://docs.fedoraproject.org/en-US/infra/sysadmin_guide/sshaccess/ It still mentions bastion-iad01. Am I on the wrong docs? It looks like the right place.
Thats the right place, no one updated it. ;(
https://pagure.io/infra-docs-fpo/pull-request/195
adds a bunch of things. Reviews welcome. :)
I also stumbled upon https://fedora-infra-docs.readthedocs.io/ in my bookmarks and it's been squatted. It's built from https://github.com/primitivea/fedora-infra-docs, but that's a different issue altogether.
Ugh. Well, we used to use that, but retired it.
Is there any way we could ask them to remove it? Or are we just out of luck since we dropped it?
kevin
infrastructure@lists.fedoraproject.org