11:27 a.m.
Greetings.
We have a request (
https://pagure.io/fedora-infrastructure/issue/5372 ) to setup ssl cert
pinning for ostree deliverables. It's also been a long wishlist item
to have that for rpm deliverables too. Unfortunately there's a bunch of
moving parts here that we need to sort out before we can move this
forward.
First some background/info:
* kojipkgs.fedoraproject.org currently uses a valid digisign cert. It
needs this because browsers download from it directly, our builders
download from it directly, etc.
* pkgs/koji currently use certs signed by the Fedora Koji CA (which
expires in 2024). This is currently needed by koji to do builds and
the upload cgi for lookaside.
* We are hoping to deploy soon a pair of freeipa servers in production
that get information from fas and allow us to issue kerberos tickets.
koji can already authenticate via this method.
* There's an outstanding ticket about having a verified way to get
source: https://pagure.io/fedora-infrastructure/issue/2324
Questions we need to figure out:
* Are we going to retire/replace the koji CA? My thought was yes, but I
think Dennis wasn't on board with this. Can anyone who wants to save
it speak up? :)
* The upload cgi would need to auth with kerberos and sigul would need
to auth with kerberos for this to work.
* If we are not completely retiring the koji CA, are we replacing it?
* Is ostree going to stay distributed at kojipkgs ? Or is it going to
move somewhere else? we should figure out the final place for it
before we go setting up cert pinning.
* The simple way to do pinning is for the application(s) to include a
hard coded list of valid certs. I guess this would require changes in
librepo and somewhere in ostree?
* The complex way to do pinning would be to setup
https://en.wikipedia.org/wiki/HTTP_Public_Key_Pinning
For this we would need to get backup keys for our cert(s) that are
used for this and setup webservers to send the right headers. This
would also need (more complex) changes in librepo and/or somewhere in
ostree. This would also optionally get us reports of violations.
Thoughts? Comments?
kevin
11:57 a.m.
Hi,
...snip...
Questions we need to figure out:
* Are we going to retire/replace the koji CA? My thought was yes, but I
think Dennis wasn't on board with this. Can anyone who wants to save
it speak up? :)
I want to kill this CA. If there's anyone that sees problems with this, talk to
me and I'll see how to resolve them, as I have a plan for all the issues I
have so far foreseen.
* The upload cgi would need to auth with kerberos and sigul would need
to auth with kerberos for this to work.
Upload CGI is no issue as it uses http auth, so is just configuration.
I have a sigul patch for krb support that I'm going to merge soon.
* If we are not completely retiring the koji CA, are we replacing it?
Not if it's up to me.
* Is ostree going to stay distributed at kojipkgs ? Or is it going to
move somewhere else? we should figure out the final place for it
before we go setting up cert pinning.
* The simple way to do pinning is for the application(s) to include a
hard coded list of valid certs. I guess this would require changes in
librepo and somewhere in ostree?
As far as I know, yum/dnf supports setting a cafile for repos, so we
can just update fedora-repos.
* The complex way to do pinning would be to setup
https://en.wikipedia.org/wiki/HTTP_Public_Key_Pinning
For this we would need to get backup keys for our cert(s) that are
used for this and setup webservers to send the right headers. This
would also need (more complex) changes in librepo and/or somewhere in
ostree. This would also optionally get us reports of violations.
I would prefer this, since that means the configuration is server-side and we
can phase over to a different CA or something at a later point in time way
easier.
Thoughts? Comments?
kevin
Regards,
Patrick Uiterwijk
12:04 p.m.
On Mon, 10 Oct 2016 16:57:25 +0000
Patrick Uiterwijk <puiterwijk(a)redhat.com> wrote:
...snip...
As far as I know, yum/dnf supports setting a cafile for repos, so we
can just update fedora-repos.
That doesn't help. If we are using a well known cert, it's already
valid based on the system ca's, and IMHO it would be very poor to use a
self signed cert for this. So, either librepo carries a static list for
our base repos or we add support for HPKP.
> * The complex way to do pinning would be to setup
> https://en.wikipedia.org/wiki/HTTP_Public_Key_Pinning
> For this we would need to get backup keys for our cert(s) that are
> used for this and setup webservers to send the right headers. This
> would also need (more complex) changes in librepo and/or
> somewhere in ostree. This would also optionally get us reports of
> violations.
I would prefer this, since that means the configuration is
server-side and we can phase over to a different CA or something at a
later point in time way easier.
Still will need HPKP support in the clients... but yeah, it has
advantages.
kevin
12:16 p.m.
On Mon, Oct 10, 2016, at 01:04 PM, Kevin Fenzi wrote:
On Mon, 10 Oct 2016 16:57:25 +0000
Patrick Uiterwijk <puiterwijk(a)redhat.com> wrote:
...snip...
> As far as I know, yum/dnf supports setting a cafile for repos, so we
> can just update fedora-repos.
That doesn't help. If we are using a well known cert, it's already
valid based on the system ca's, and IMHO it would be very poor to use a
self signed cert for this. So, either librepo carries a static list for
our base repos or we add support for HPKP.
I would s/static list/custom root CA set/.
It's worth emphasizing that having a custom CA root set is
a very standard thing to do - we're basically just doing
exactly what ca-certificates does, just with a different configuration.
And it's supported today by both librepo and ostree, and many
HTTP libraries in general. And both of those code paths are used
today for Red Hat Enterprise Linux Atomic Host, talking
to the default Akamai CDN.
HPKP is (as far as I know) mostly a browser thing[1]; it isn't
implemented by libcurl for example. Though it does
make sense for general outside-of-the-browser use cases,
from an operating system perspective, we already have
a mechanism to ship the configuration out-of-band to
client systems by embedding it into the installation media
and the update stream. The same as we do for GPG.
Basically, I think it's easiest if we think of the keys/configuration for
CA-pinning the same as GPG.
[1] Though I did find https://github.com/tam7t/hpkp - kind of curious
what Digital Ocean is doing with it if anything
12:58 p.m.
On Mon, 10 Oct 2016 13:16:23 -0400
Colin Walters <walters(a)verbum.org> wrote:
On Mon, Oct 10, 2016, at 01:04 PM, Kevin Fenzi wrote:
> On Mon, 10 Oct 2016 16:57:25 +0000
> Patrick Uiterwijk <puiterwijk(a)redhat.com> wrote:
>
> ...snip...
>
> > As far as I know, yum/dnf supports setting a cafile for repos, so
> > we can just update fedora-repos.
>
> That doesn't help. If we are using a well known cert, it's already
> valid based on the system ca's, and IMHO it would be very poor to
> use a self signed cert for this. So, either librepo carries a
> static list for our base repos or we add support for HPKP.
I would s/static list/custom root CA set/.
It's worth emphasizing that having a custom CA root set is
a very standard thing to do - we're basically just doing
exactly what ca-certificates does, just with a different
configuration.
And it's supported today by both librepo and ostree, and many
HTTP libraries in general. And both of those code paths are used
today for Red Hat Enterprise Linux Atomic Host, talking
to the default Akamai CDN.
HPKP is (as far as I know) mostly a browser thing[1]; it isn't
implemented by libcurl for example. Though it does
make sense for general outside-of-the-browser use cases,
from an operating system perspective, we already have
a mechanism to ship the configuration out-of-band to
client systems by embedding it into the installation media
and the update stream. The same as we do for GPG.
Basically, I think it's easiest if we think of the keys/configuration
for CA-pinning the same as GPG.
But does that not mean anyone going to the same place with a browser or
command line downloading specific packages will get a "sorry, this cert
is not trusted" ? Thats not such a big deal for ostree's, but for rpms,
people do this all the time.
kevin
1:31 p.m.
On Mon, Oct 10, 2016, at 01:58 PM, Kevin Fenzi wrote:
But does that not mean anyone going to the same place with a browser or
command line downloading specific packages will get a "sorry, this cert
is not trusted" ? Thats not such a big deal for ostree's, but for rpms,
people do this all the time.
Yes, there are two things someone could do then:
1) Go to any of the many non-ca-pinned URLs
I wasn't proposing switching any of the existing URLs, but adding
a new one, and we should ensure that the exact same view is
available with a regular ca-certificates signed cert
2) Use curl --cafile or equivalent (or hack it with curl -k etc.)
2:17 p.m.
On Tue, 11 Oct 2016 14:31:55 -0400
Colin Walters <walters(a)verbum.org> wrote:
On Mon, Oct 10, 2016, at 01:58 PM, Kevin Fenzi wrote:
>
> But does that not mean anyone going to the same place with a
> browser or command line downloading specific packages will get a
> "sorry, this cert is not trusted" ? Thats not such a big deal for
> ostree's, but for rpms, people do this all the time.
Yes, there are two things someone could do then:
1) Go to any of the many non-ca-pinned URLs
I wasn't proposing switching any of the existing URLs, but adding
a new one, and we should ensure that the exact same view is
available with a regular ca-certificates signed cert
2) Use curl --cafile or equivalent (or hack it with curl -k etc.)
Sure, but they won't. They will complain that we have an invalid cert
and we will need to explain to them whats going on. ;)
Instead of shipping a fedora-ca that you verify against, why not do
what chrom*/firefox do and have a hardcoded list of hashes that must be
in the cert?
https://wiki.mozilla.org/SecurityEngineering/Public_Key_Pinning
https://src.chromium.org/viewvc/chrome/trunk/src/net/http/transport_secur...
If we ever switched from using Digicert for our certs we would need to
change this, but otherwise it should protect from the Rogue CA threat
(unless it was Digicert I guess).
Also in the same file chom*/firefox set a list of sites to assume ssl,
which would also be nice to hard code.
kevin
8:42 a.m.
On Wed, Oct 12, 2016, at 03:17 PM, Kevin Fenzi wrote:
Sure, but they won't. They will complain that we have an invalid
cert
and we will need to explain to them whats going on. ;)
I still think this would be mostly covered if the yum repo files
and the ostree remote config had a comment like:
# This URL is ca-pinned, see https://fedoraproject.org/wiki/CAPinning
But:
Instead of shipping a fedora-ca that you verify against, why not do
what chrom*/firefox do and have a hardcoded list of hashes that must be
in the cert?
https://wiki.mozilla.org/SecurityEngineering/Public_Key_Pinning
https://src.chromium.org/viewvc/chrome/trunk/src/net/http/transport_secur...
The Chrome and Firefox implementations seem different - Firefox
hashes the certs, but Chrome seems to have a CA whitelist, which
is actually a lot easier to read.
Anyways, there's a higher level question here - you're arguing
for pinning to Digicert rather than a custom CA. That seems good
enough, but I think we need a recovery mechanism in case Digicert
explodes.
So I'd propose pinning to a 3 set of CAs:
- Digicert
- Some other well-regarded CA vendor
- A Fedora-infra custom CA (doesn't have to be deployed, just a backup plan)
And as for a specific implementation mechanism, we'd have just
those CAs in /etc/pki/tls/certs/fedora-infra.crt or so, and that file
would be in the fedora-repos package. The argument for this again
is that librepo and ostree already have all of the code for this, and so does curl etc.
Doing the hashes of the certs like Firefox does is certainly possible,
but it requires custom logic in the HTTP layer, and there's no
standard configuration file formats for the data, etc.
Also in the same file chom*/firefox set a list of sites to assume
ssl,
which would also be nice to hard code.
Yeah, we could follow up with this adding Fedora sites to the browser
lists. Chrome's version seems saner to me.
9:16 a.m.
On Fri, Oct 14, 2016, at 08:42 AM, Colin Walters wrote:
Anyways, there's a higher level question here - you're arguing
for pinning to Digicert rather than a custom CA. That seems good
enough, but I think we need a recovery mechanism in case Digicert
explodes.
So I'd propose pinning to a 3 set of CAs:
- Digicert
- Some other well-regarded CA vendor
- A Fedora-infra custom CA (doesn't have to be deployed, just a backup plan)
Any further thoughts here?
> And as for a specific implementation mechanism, we'd have just
> those CAs in /etc/pki/tls/certs/fedora-infra.crt or so, and that file
> would be in the fedora-repos package. The argument for this again
> is that librepo and ostree already have all of the code for this, and so does curl
etc.
>
> Doing the hashes of the certs like Firefox does is certainly possible,
> but it requires custom logic in the HTTP layer, and there's no
> standard configuration file formats for the data, etc.
>
> > Also in the same file chom*/firefox set a list of sites to assume ssl,
> > which would also be nice to hard code.
>
> Yeah, we could follow up with this adding Fedora sites to the browser
> lists. Chrome's version seems saner to me.
11:10 a.m.
On Mon, 21 Nov 2016 10:16:55 -0500
Colin Walters <walters(a)verbum.org> wrote:
On Fri, Oct 14, 2016, at 08:42 AM, Colin Walters wrote:
>
> Anyways, there's a higher level question here - you're arguing
> for pinning to Digicert rather than a custom CA. That seems good
> enough, but I think we need a recovery mechanism in case Digicert
> explodes.
>
> So I'd propose pinning to a 3 set of CAs:
>
> - Digicert
> - Some other well-regarded CA vendor
> - A Fedora-infra custom CA (doesn't have to be deployed, just a
> backup plan)
Any further thoughts here?
> And as for a specific implementation mechanism, we'd have just
> those CAs in /etc/pki/tls/certs/fedora-infra.crt or so, and that
> file would be in the fedora-repos package. The argument for this
> again is that librepo and ostree already have all of the code for
> this, and so does curl etc.
I suppose thats workable if all the stakeholders agree.
So, we would need:
1. certs in fedora-repos package
2. librepo would need changes to know to check those.
3. ostree would need changes to know to check those.
I've not heard from librepo folks, should we ask them before moving
forward here? fedora-repos would need ack from Dennis (who is currently
on vacation), but I don't think that should be a problem.
kevin
2:45 p.m.
On Wed, Nov 23, 2016, at 12:10 PM, Kevin Fenzi wrote:
I suppose thats workable if all the stakeholders agree.
To confirm, are you agreeing with:
So I'd propose pinning to a 3 set of CAs:
- Digicert
- Some other well-regarded CA vendor
- A Fedora-infra custom CA (doesn't have to be deployed, just a
backup plan)
You were arguing earlier to pin to just digicert I think (though
I can't find that now).
We could probably move forward with Digicert + 1-2 other
vendors as well. Maybe to be conservative 2. We can easily
add a custom CA to the set as well at any point.
10:20 a.m.
On Wed, 23 Nov 2016 15:45:55 -0500
Colin Walters <walters(a)verbum.org> wrote:
On Wed, Nov 23, 2016, at 12:10 PM, Kevin Fenzi wrote:
> I suppose thats workable if all the stakeholders agree.
To confirm, are you agreeing with:
> So I'd propose pinning to a 3 set of CAs:
>
> - Digicert
> - Some other well-regarded CA vendor
> - A Fedora-infra custom CA (doesn't have to be deployed, just a
> backup plan)
You were arguing earlier to pin to just digicert I think (though
I can't find that now).
Yeah. I am not sure the process we will need to use to get some other
CA vendor. RH has a relationship with digicert, so we get our certs via
that. When using another vendor we may have to go through some
red-tape. So, I can't commit for a time when this would be ready.
We could probably move forward with Digicert + 1-2 other
vendors as well. Maybe to be conservative 2. We can easily
add a custom CA to the set as well at any point.
We should make sure that the librepo/dnf folks are on board with this
plan before moving forward. :)
kevin
2:32 p.m.
On Mon, Nov 28, 2016, at 11:20 AM, Kevin Fenzi wrote:
Yeah. I am not sure the process we will need to use to get some other
CA vendor. RH has a relationship with digicert, so we get our certs via
that. When using another vendor we may have to go through some
red-tape. So, I can't commit for a time when this would be ready.
OK, can you file the issue/request and link me to it?
> We could probably move forward with Digicert + 1-2 other
> vendors as well. Maybe to be conservative 2. We can easily
> add a custom CA to the set as well at any point.
We should make sure that the librepo/dnf folks are on board with this
plan before moving forward. :)
Sure, I sent Honza and Igor a mail.
1 p.m.
On Mon, 28 Nov 2016 15:32:02 -0500
Colin Walters <walters(a)verbum.org> wrote:
On Mon, Nov 28, 2016, at 11:20 AM, Kevin Fenzi wrote:
>
> Yeah. I am not sure the process we will need to use to get some
> other CA vendor. RH has a relationship with digicert, so we get our
> certs via that. When using another vendor we may have to go through
> some red-tape. So, I can't commit for a time when this would be
> ready.
OK, can you file the issue/request and link me to it?
> > We could probably move forward with Digicert + 1-2 other
> > vendors as well. Maybe to be conservative 2. We can easily
> > add a custom CA to the set as well at any point.
>
> We should make sure that the librepo/dnf folks are on board with
> this plan before moving forward. :)
Sure, I sent Honza and Igor a mail.
Hum. I was writing up an email on this, and something occurred to me.
The various browsers already have our digicert cert hard coded.
So, if we ever had problems with that cert and had to switch to the
secondary or tertiary certs, all browser access would be broken. ;(
So, perhaps we should be more targeted here and only do this for some
particular endpoints? mirrors.fedoraproject.org and
dl.fedoraproject.org ? That way if we had to fall back to another cert
only those would be broken for browsers.
Or should I just not worry too much about it because anything that
causes us to switch from the primary cert would likely be a massive
blowup anyhow?
kevin
4:14 p.m.
On Tue, Nov 29, 2016, at 02:00 PM, Kevin Fenzi wrote:
The various browsers already have our digicert cert hard coded.
Hum, really? Reference?
$ pwd
/home/walters/src/github/mozilla/gecko-dev
$ git rev-parse HEAD
a8b5f53e7df90df655a0982e94087ee83290c22e
$ git grep fedoraproject.org
Shows me what looks like a HSTS preload entry, but not a ca/cert
pin. (Remember HSTS preload == use https:// by default)
11:29 a.m.
On Tue, 06 Dec 2016 17:14:48 -0500
Colin Walters <walters(a)verbum.org> wrote:
On Tue, Nov 29, 2016, at 02:00 PM, Kevin Fenzi wrote:
>
> The various browsers already have our digicert cert hard coded.
Hum, really? Reference?
$ pwd
/home/walters/src/github/mozilla/gecko-dev
$ git rev-parse HEAD
a8b5f53e7df90df655a0982e94087ee83290c22e
$ git grep fedoraproject.org
Shows me what looks like a HSTS preload entry, but not a ca/cert
pin. (Remember HSTS preload == use https:// by default)
ok, I finally found some time to sit down and do some digging and... I
can't find it now. ;(
I cannot even see the HSTS preload there in firefox/chromium, but that
might have gotten kicked out due to a error with
paste.fedoraproject.org I can investigate more.
So, sorry for that sidetrack/delay... we can move forward AFAICT.
kevin
3:51 p.m.
On Tue, Nov 29, 2016, at 02:00 PM, Kevin Fenzi wrote:
The various browsers already have our digicert cert hard coded.
So, if we ever had problems with that cert and had to switch to the
secondary or tertiary certs, all browser access would be broken. ;(
So, perhaps we should be more targeted here and only do this for some
particular endpoints? mirrors.fedoraproject.org and
dl.fedoraproject.org ? That way if we had to fall back to another cert
only those would be broken for browsers.
I don't understand this btw - the CA pinning we're talking about
would only be for software mechanisms like dnf/rpm-ostree and possibly docker/flatpak.
I'm certainly not advocating changing any other tools right now,
although one could theroetically consider things like the `bodhi` command
line tools (or possibly changing the underlying shared libraries).
4:38 p.m.
On 9 December 2016 at 16:51, Colin Walters <walters(a)verbum.org> wrote:
On Tue, Nov 29, 2016, at 02:00 PM, Kevin Fenzi wrote:
> The various browsers already have our digicert cert hard coded.
> So, if we ever had problems with that cert and had to switch to the
> secondary or tertiary certs, all browser access would be broken. ;(
>
> So, perhaps we should be more targeted here and only do this for some
> particular endpoints? mirrors.fedoraproject.org and
> dl.fedoraproject.org ? That way if we had to fall back to another cert
> only those would be broken for browsers.
I don't understand this btw - the CA pinning we're talking about
would only be for software mechanisms like dnf/rpm-ostree and possibly docker/flatpak.
I'm certainly not advocating changing any other tools right now,
although one could theroetically consider things like the `bodhi` command
line tools (or possibly changing the underlying shared libraries).
I don't think anyone is understanding each other.. because that isn't
what I was getting from this thread until now.
_______________________________________________
infrastructure mailing list -- infrastructure(a)lists.fedoraproject.org
To unsubscribe send an email to infrastructure-leave(a)lists.fedoraproject.org
--
Stephen J Smoogen.
11:37 a.m.
On Fri, Dec 9, 2016, at 05:38 PM, Stephen John Smoogen wrote:
I don't think anyone is understanding each other.. because that
isn't
what I was getting from this thread until now.
The thread has been 95% just me and Kevin on and off over the last 6
months. I asked him for clarification. Now, aAre you going to help?
If so, can you elaborate on what you are/aren't understanding?
12:49 p.m.
On 13 December 2016 at 12:37, Colin Walters <walters(a)verbum.org> wrote:
On Fri, Dec 9, 2016, at 05:38 PM, Stephen John Smoogen wrote:
> I don't think anyone is understanding each other.. because that isn't
> what I was getting from this thread until now.
The thread has been 95% just me and Kevin on and off over the last 6
months. I asked him for clarification. Now, aAre you going to help?
If so, can you elaborate on what you are/aren't understanding?
So the parts I think I am seeing different answers are:
1. What are we trying to accomplish and where?
2. What infrastructure is needed to accomplish this?
3. When does this need to be done?
4. What is the budget cost for setting up these certificates,
reporting systems, pinned ips etc
5. Who is paying for this (hardware, certificates, domain names, etc).
6. Who is doing the implementation work and where?
To me Kevin, Patrick and you have said similar things but that you
each have different ideas about what they mean and so aren't actually
communicating.
_______________________________________________
infrastructure mailing list -- infrastructure(a)lists.fedoraproject.org
To unsubscribe send an email to infrastructure-leave(a)lists.fedoraproject.org
--
Stephen J Smoogen.
4:04 p.m.
On Tue, Dec 13, 2016, at 01:49 PM, Stephen John Smoogen wrote:
So the parts I think I am seeing different answers are:
1. What are we trying to accomplish and where?
2. What infrastructure is needed to accomplish this?
I think this stuff is pretty well covered in the thread and should
be hashed out by the people who are going to do the work.
3. When does this need to be done?
I don't quite understand how this process stretched to 6
months already...but soon, please - I'd like Fedora to
be a reference architecture. If it takes another
few months something is clearly going wrong.
4. What is the budget cost for setting up these certificates,
reporting systems, pinned ips etc
5. Who is paying for this (hardware, certificates, domain names, etc).
Fedora infra.
6. Who is doing the implementation work and where?
I can take point on making and testing the client side changes.
9:53 p.m.
FYI, I marked this thread to reply to, but I simply have not had time
lately with last week on site at the datacenter and this weekend
prepping for the flag day and this week helping people with fallout
from the flag day.
I'll try and get back to this this week, but please have some patience.
kevin
8:16 a.m.
On Tue, Dec 13, 2016, at 10:53 PM, Kevin Fenzi wrote:
FYI, I marked this thread to reply to, but I simply have not had
time
lately with last week on site at the datacenter and this weekend
prepping for the flag day and this week helping people with fallout
from the flag day.
I'll try and get back to this this week, but please have some patience.
That's fine! This seems like something we can get done if someone has
a chance to focus on it for a day or two.
To summarize then, my understanding is:
- Fedora chooses 1-2 other CA providers to use as backup, and acquires
certs from those providers for at least:
* mirrors.fedoraproject.org
* kojipkgs.fedoraproject.org
(Or maybe it's simpler to just do all of *fedoraproject.org, either way)
- I will take care of prepping a patch for just the ostree portion of
Atomic Host to use this configuration
- We'll create a wiki page collaboratively describing this, and
post to fedora-devel how to enable it with the rpm-md configuration,
and have interested testers try it
- At some point later, we change the fedora-repos package to enable
that configuration by default
One thing this likely will break is people who run things like
`sed -i -e s,baseurl=.*,http://myinternalmirror.corp.example.com
/etc/yum.repos.d/fedora.repo`,
but I think we'll get past those types of minor things over time;
the security win is worth it.
11:40 a.m.
On Wed, 14 Dec 2016 09:16:47 -0500
Colin Walters <walters(a)verbum.org> wrote:
On Tue, Dec 13, 2016, at 10:53 PM, Kevin Fenzi wrote:
> FYI, I marked this thread to reply to, but I simply have not had
> time lately with last week on site at the datacenter and this
> weekend prepping for the flag day and this week helping people with
> fallout from the flag day.
>
> I'll try and get back to this this week, but please have some
> patience.
That's fine! This seems like something we can get done if someone has
a chance to focus on it for a day or two.
yeah.
To summarize then, my understanding is:
- Fedora chooses 1-2 other CA providers to use as backup, and acquires
certs from those providers for at least:
* mirrors.fedoraproject.org
* kojipkgs.fedoraproject.org
(Or maybe it's simpler to just do all of *fedoraproject.org,
either way)
Yep.
- I will take care of prepping a patch for just the ostree portion
of
Atomic Host to use this configuration
- We'll create a wiki page collaboratively describing this, and
post to fedora-devel how to enable it with the rpm-md
configuration, and have interested testers try it
- At some point later, we change the fedora-repos package to enable
that configuration by default
Did you hear back from the dnf/librepo folks? Is there a ticket or
email thread I can follow along for their side of things?
I'll fire off some emails here to figure out what other CA we could use
and how (our current approved process is to get them all via digicert,
so we need to find out a new process), as well as run it by infosec
folks to make sure we didn't forget anything.
One thing this likely will break is people who run things like
`sed -i -e
s,baseurl=.*,http://myinternalmirror.corp.example.com /etc/yum.repos.d/fedora.repo`,
but I think we'll get past those types of minor things over time; the
security win is worth it.
Yeah, I think we cant worry too much about those cases, and I think
they are pretty small these days.
kevin
11:37 a.m.
On Fri, 09 Dec 2016 16:51:25 -0500
Colin Walters <walters(a)verbum.org> wrote:
On Tue, Nov 29, 2016, at 02:00 PM, Kevin Fenzi wrote:
> The various browsers already have our digicert cert hard coded.
> So, if we ever had problems with that cert and had to switch to the
> secondary or tertiary certs, all browser access would be broken. ;(
>
> So, perhaps we should be more targeted here and only do this for
> some particular endpoints? mirrors.fedoraproject.org and
> dl.fedoraproject.org ? That way if we had to fall back to another
> cert only those would be broken for browsers.
I don't understand this btw - the CA pinning we're talking about
would only be for software mechanisms like dnf/rpm-ostree and
possibly docker/flatpak.
Right now for say dnf, it would hit mirrors.fedoraproject.org (for the
metalink) and possibly dl.fedoraproject.org (if it happened to get it
at the end of the metalink). I was saying instead of pinning our
wildcard *.fedoraproject.org cert (which we use for a number of sites /
places) we could just get specific non wildcard ones for these sites.
However, pondering on it more those would still have to be on the
proxies, so I am not sure it would buy us in the end.
kevin
5:20 p.m.
On Monday, October 10, 2016 10:27:29 AM CDT Kevin Fenzi wrote:
Greetings.
We have a request (
https://pagure.io/fedora-infrastructure/issue/5372 ) to setup ssl cert
pinning for ostree deliverables. It's also been a long wishlist item
to have that for rpm deliverables too. Unfortunately there's a bunch of
moving parts here that we need to sort out before we can move this
forward.
First some background/info:
* kojipkgs.fedoraproject.org currently uses a valid digisign cert. It
needs this because browsers download from it directly, our builders
download from it directly, etc.
* pkgs/koji currently use certs signed by the Fedora Koji CA (which
expires in 2024). This is currently needed by koji to do builds and
the upload cgi for lookaside.
The koji CA expires in 2018
* We are hoping to deploy soon a pair of freeipa servers in
production
that get information from fas and allow us to issue kerberos tickets.
koji can already authenticate via this method.
* There's an outstanding ticket about having a verified way to get
source: https://pagure.io/fedora-infrastructure/issue/2324
Questions we need to figure out:
* Are we going to retire/replace the koji CA? My thought was yes, but I
think Dennis wasn't on board with this. Can anyone who wants to save
it speak up? :)
I am against kerberos being the only auth mechanism. I suspect
that some
people either cant for reasons beyond our control or won't set up kerberos for
auth
* The upload cgi would need to auth with kerberos and sigul would
need
to auth with kerberos for this to work.
* If we are not completely retiring the koji CA, are we replacing it?
If not
retired it has to be replaced, could be certs from freeipa that auto
renew with certmonger, which i suspect users would like better than entering
their kerberos password once a day.
* Is ostree going to stay distributed at kojipkgs ? Or is it going
to
move somewhere else? we should figure out the final place for it
before we go setting up cert pinning.
No, it needs to go on the mirrors when we
sort out how to mirror it, and the
client and mirrormanager support it
* The simple way to do pinning is for the application(s) to include
a
hard coded list of valid certs. I guess this would require changes in
librepo and somewhere in ostree?
* The complex way to do pinning would be to setup
https://en.wikipedia.org/wiki/HTTP_Public_Key_Pinning
For this we would need to get backup keys for our cert(s) that are
used for this and setup webservers to send the right headers. This
would also need (more complex) changes in librepo and/or somewhere in
ostree. This would also optionally get us reports of violations.
Thoughts? Comments?
Not against making changes, but I do not think that they totally fit into long
term goals
Dennis
kevin
2:34 p.m.
I meant to reply to this eariler. ;)
On Mon, 10 Oct 2016 17:20:06 -0500
Dennis Gilmore <dennis(a)ausil.us> wrote:
On Monday, October 10, 2016 10:27:29 AM CDT Kevin Fenzi wrote:
> Greetings.
>
> We have a request (
> https://pagure.io/fedora-infrastructure/issue/5372 ) to setup ssl
> cert pinning for ostree deliverables. It's also been a long
> wishlist item to have that for rpm deliverables too. Unfortunately
> there's a bunch of moving parts here that we need to sort out
> before we can move this forward.
>
> First some background/info:
>
> * kojipkgs.fedoraproject.org currently uses a valid digisign cert.
> It needs this because browsers download from it directly, our
> builders download from it directly, etc.
>
> * pkgs/koji currently use certs signed by the Fedora Koji CA (which
> expires in 2024). This is currently needed by koji to do builds
> and the upload cgi for lookaside.
The koji CA expires in 2018
ok
> * We are hoping to deploy soon a pair of freeipa servers in
> production that get information from fas and allow us to issue
> kerberos tickets. koji can already authenticate via this method.
>
> * There's an outstanding ticket about having a verified way to get
> source: https://pagure.io/fedora-infrastructure/issue/2324
>
> Questions we need to figure out:
>
> * Are we going to retire/replace the koji CA? My thought was yes,
> but I think Dennis wasn't on board with this. Can anyone who wants
> to save it speak up? :)
I am against kerberos being the only auth mechanism. I suspect that
some people either cant for reasons beyond our control or won't set
up kerberos for auth
I'm interested in what these reasons could be.
Maintaining 2 authentication stacks is not something we really want to
do if we can at all avoid it. And kerberos should be a good deal more
secure than certs I would think, and we still don't have a great setup
to manage the certs.
> * The upload cgi would need to auth with kerberos and sigul
would
> need to auth with kerberos for this to work.
>
> * If we are not completely retiring the koji CA, are we replacing
> it?
If not retired it has to be replaced, could be certs from freeipa
that auto renew with certmonger, which i suspect users would like
better than entering their kerberos password once a day.
well, we can adjust the kerberos settings. If they can renew for a week
would that be sufficent?
> * Is ostree going to stay distributed at kojipkgs ? Or is it
going
> to move somewhere else? we should figure out the final place for it
> before we go setting up cert pinning.
No, it needs to go on the mirrors when we sort out how to mirror it,
and the client and mirrormanager support it
Right, so doing some cert pinning right now might be not a great idea
if we are just going to move it.
> * The simple way to do pinning is for the application(s) to
include
> a hard coded list of valid certs. I guess this would require
> changes in librepo and somewhere in ostree?
>
> * The complex way to do pinning would be to setup
> https://en.wikipedia.org/wiki/HTTP_Public_Key_Pinning
> For this we would need to get backup keys for our cert(s) that are
> used for this and setup webservers to send the right headers. This
> would also need (more complex) changes in librepo and/or
> somewhere in ostree. This would also optionally get us reports of
> violations.
>
> Thoughts? Comments?
Not against making changes, but I do not think that they totally fit
into long term goals
Well, I can definitely see the need for cert pinning and moving to
kerberos, but perhaps you can expand on the long term goals that this
doesn't address?
kevin
2:34 p.m.
On jueves, 13 de octubre de 2016 1:34:42 PM CDT Kevin Fenzi wrote:
I meant to reply to this eariler. ;)
I just now saw the reply
:(
On Mon, 10 Oct 2016 17:20:06 -0500
Dennis Gilmore <dennis(a)ausil.us> wrote:
> On Monday, October 10, 2016 10:27:29 AM CDT Kevin Fenzi wrote:
> > Greetings.
> >
> > We have a request (
> > https://pagure.io/fedora-infrastructure/issue/5372 ) to setup ssl
> > cert pinning for ostree deliverables. It's also been a long
> > wishlist item to have that for rpm deliverables too. Unfortunately
> > there's a bunch of moving parts here that we need to sort out
> > before we can move this forward.
> >
> > First some background/info:
> >
> > * kojipkgs.fedoraproject.org currently uses a valid digisign cert.
> > It needs this because browsers download from it directly, our
> > builders download from it directly, etc.
> >
> > * pkgs/koji currently use certs signed by the Fedora Koji CA (which
> >
> > expires in 2024). This is currently needed by koji to do builds
> >
> > and the upload cgi for lookaside.
>
> The koji CA expires in 2018
ok
> > * We are hoping to deploy soon a pair of freeipa servers in
> > production that get information from fas and allow us to issue
> > kerberos tickets. koji can already authenticate via this method.
> >
> > * There's an outstanding ticket about having a verified way to get
> >
> > source: https://pagure.io/fedora-infrastructure/issue/2324
> >
> > Questions we need to figure out:
> >
> > * Are we going to retire/replace the koji CA? My thought was yes,
> > but I think Dennis wasn't on board with this. Can anyone who wants
> > to save it speak up? :)
>
> I am against kerberos being the only auth mechanism. I suspect that
> some people either cant for reasons beyond our control or won't set
> up kerberos for auth
I'm interested in what these reasons could be.
The machines managed by their employers IT who forbid using any external
kerberos. The users just do not like kerberos.
Maintaining 2 authentication stacks is not something we really want
to
do if we can at all avoid it. And kerberos should be a good deal more
secure than certs I would think, and we still don't have a great setup
to manage the certs.
sure. it could be IPA makes it simple for us.
> > * The upload cgi would need to auth with kerberos and sigul
would
> > need to auth with kerberos for this to work.
> >
> > * If we are not completely retiring the koji CA, are we replacing
> > it?
>
> If not retired it has to be replaced, could be certs from freeipa
> that auto renew with certmonger, which i suspect users would like
> better than entering their kerberos password once a day.
well, we can adjust the kerberos settings. If they can renew for a week
would that be sufficent?
possibly, I am not the users. though with kerberos based
ssh for access to git
and fedora infra resources puts another benefit to having kerberos, as well as
possibly sso on all the web apps.
> > * Is ostree going to stay distributed at kojipkgs ? Or is
it going
> > to move somewhere else? we should figure out the final place for it
> >
> > before we go setting up cert pinning.
>
> No, it needs to go on the mirrors when we sort out how to mirror it,
> and the client and mirrormanager support it
Right, so doing some cert pinning right now might be not a great idea
if we are just going to move it.
There is bound to be a lot of moving coming. what is any different about any
of this that we do not need to do the same for rpm repos?
> > * The simple way to do pinning is for the application(s) to
include
> > a hard coded list of valid certs. I guess this would require
> > changes in librepo and somewhere in ostree?
> >
> > * The complex way to do pinning would be to setup
> >
> > https://en.wikipedia.org/wiki/HTTP_Public_Key_Pinning
> > For this we would need to get backup keys for our cert(s) that are
> > used for this and setup webservers to send the right headers. This
> > would also need (more complex) changes in librepo and/or
> >
> > somewhere in ostree. This would also optionally get us reports of
> > violations.
> >
> > Thoughts? Comments?
>
> Not against making changes, but I do not think that they totally fit
> into long term goals
Well, I can definitely see the need for cert pinning and moving to
kerberos, but perhaps you can expand on the long term goals that this
doesn't address?
Well can we do cert pinning with mirrors? I guess we could to mirrormanager.
Does it matter with signed commits or repodata? some content will never or
rarely be available via https
Dennis
kevin
3:18 a.m.
On 10/13/2016 09:34 PM, Kevin Fenzi wrote:
>> * If we are not completely retiring the koji CA, are we
replacing
>> it?
> If not retired it has to be replaced, could be certs from freeipa
> that auto renew with certmonger, which i suspect users would like
> better than entering their kerberos password once a day.
well, we can adjust the kerberos settings. If they can renew for a week
would that be sufficent?
Couldn't users simply generate keytab for themselves? Koji client
supports keytabs directly (via setting in koji.conf or --keytab param),
for other services it should be possible to run "kinit -k" (which can
even be ran from startup stripts, cron etc.)
--
Mikolaj Izdebski
IRC: mizdebsk
2683
days inactive
2748
days old
infrastructure@lists.fedoraproject.org
28 comments
6 participants
participants (6)
-
Colin Walters -
Dennis Gilmore -
Kevin Fenzi -
Mikolaj Izdebski -
Patrick Uiterwijk -
Stephen John Smoogen