7:59 p.m.
It seems to me that Fedora has three severe distribution wide issues
relating to security updates:
1. Updates, even critical security updates, are very slow. Getting an
update out involves creating a build and an update (which is
reasonably fast for most packages), pushing the update to
updates-testing, getting karma, and pushing to updates. The latter
three can happen in parallel in the best case. The big problem as I
see it is that pushing updates is painfully slow. For reasons that
I'm sure make a lot of internal sense, updates seem to get stuck
behind broken composes and such all the time. Can this be made
better?
IMO it should be possible for a security update to be checked,
incrementally, for sanity with respect to the rest of the package
repository, and then pushed out, without needing to do whatever
expensive work a compose does. If some other update breaks the
compose, I don't think this should cause security updates to get
stuck.
2. There doesn't appear to be a working process to get updates out
quickly. As a current and pressing example, there is no build for
Firefox 49.0.2.
3. AFAIK Fedora has no means by which it can participate in embargoed
updates. For this to work, I think there ought to be private git
branches, a way to get Koji to make a private build from a private git
branch, and a way to get private karma on a private update. Then,
when an embargo is lifted, the packager could merge the private branch
in, the various infrastructure bits could notice that the very same
git commit is now public and permit all of the private builds,
updates, and karma to become public and allow an immediate push to
updates.
8:03 p.m.
On Tue, 2016-10-25 at 17:59 -0700, Andrew Lutomirski wrote:
2. There doesn't appear to be a working process to get updates
out
quickly. As a current and pressing example, there is no build for
Firefox 49.0.2.
There isn't really a single 'security update process', no. Releasing
security updates is a package maintainer responsibility. I've seen
various people pontificating in various different places about why
there's no Firefox 49.0.2 build yet, but no-one seems to have taken the
rather obvious step of *asking the package maintainer*, or at least no-
one's done so and then posted the result rather than guessing about it.
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
8:08 p.m.
On Tue, Oct 25, 2016 at 6:03 PM, Adam Williamson
<adamwill(a)fedoraproject.org> wrote:
On Tue, 2016-10-25 at 17:59 -0700, Andrew Lutomirski wrote:
> 2. There doesn't appear to be a working process to get updates out
> quickly. As a current and pressing example, there is no build for
> Firefox 49.0.2.
There isn't really a single 'security update process', no. Releasing
security updates is a package maintainer responsibility. I've seen
various people pontificating in various different places about why
there's no Firefox 49.0.2 build yet, but no-one seems to have taken the
rather obvious step of *asking the package maintainer*, or at least no-
one's done so and then posted the result rather than guessing about it.
Fair enough.
FWIW, I think that #1 and #3 are more severe than #2 in the sense that
#2 is sporadic and there are various mechanisms that mostly work to
notify maintainers about updates, whereas #1 and #3 are problematic
even when the maintainer responds instantaneously.
9:02 p.m.
Wouldn't package maintainers get the CVE bug notification from Bugzilla
about FF that I pointed to? Given that, the assumption that maintainers
are away seems reasonable. Ergo, I sent an e-mail to the list.
PS. I also checked FF package git repo, which had no recent commits.
--
Bojan
8:37 p.m.
On Tue, 25 Oct 2016 17:59:24 -0700
Andrew Lutomirski <luto(a)mit.edu> wrote:
It seems to me that Fedora has three severe distribution wide issues
relating to security updates:
1. Updates, even critical security updates, are very slow. Getting an
update out involves creating a build and an update (which is
reasonably fast for most packages), pushing the update to
updates-testing, getting karma, and pushing to updates. The latter
three can happen in parallel in the best case. The big problem as I
see it is that pushing updates is painfully slow. For reasons that
I'm sure make a lot of internal sense, updates seem to get stuck
behind broken composes and such all the time. Can this be made
better?
It has been. In the last month or so, a bunch of bodhi masher bugs have
been tracked down and we have autosigning in place.
I think people don't realize all the stuff we have to do to push
updates.
IMO it should be possible for a security update to be checked,
incrementally, for sanity with respect to the rest of the package
repository, and then pushed out, without needing to do whatever
expensive work a compose does. If some other update breaks the
compose, I don't think this should cause security updates to get
stuck.
Nope. We have talked about having some kind of fast track, but IMHO, we
should just get the normal process faster.
2. There doesn't appear to be a working process to get updates out
quickly. As a current and pressing example, there is no build for
Firefox 49.0.2.
Ask the maintainer(s)? there could be any number of reasons for this...
3. AFAIK Fedora has no means by which it can participate in
embargoed
updates. For this to work, I think there ought to be private git
branches, a way to get Koji to make a private build from a private git
branch, and a way to get private karma on a private update. Then,
when an embargo is lifted, the packager could merge the private branch
in, the various infrastructure bits could notice that the very same
git commit is now public and permit all of the private builds,
updates, and karma to become public and allow an immediate push to
updates.
Yep. Thats a gigantic pile of work there for sure.
kevin
5:23 a.m.
On Tuesday, October 25, 2016 7:37:32 PM CEST Kevin Fenzi wrote:
> 3. AFAIK Fedora has no means by which it can participate in
embargoed
> updates. For this to work, I think there ought to be private git
> branches, a way to get Koji to make a private build from a private git
> branch, and a way to get private karma on a private update. Then,
> when an embargo is lifted, the packager could merge the private branch
> in, the various infrastructure bits could notice that the very same
> git commit is now public and permit all of the private builds,
> updates, and karma to become public and allow an immediate push to
> updates.
Yep. Thats a gigantic pile of work there for sure.
That's too vague statement, really. Can you make a better estimation? As
far as I understand, there are processes in Debian which allow them
preparing CVE builds so they are able to provide "testing" builds to users
immediately after the public announcement.
Seems like we need to:
[x] have another git repository, say prepared-rpms/PACKAGE
- that's clone of rpms/PACKAGE
- permission bits inherited from rpms/PACKAGE, but not publicly
available for cloning, + security guys
[x] Making sure that only one repo is writeable:
- if prepared-rpms/PKG exists, rpms/PKG is read only
- the info "why" it is read-only shouldn't be public information
[x] Support for "private" builds in koji. Maintainer should be able to re-tag
security/prepared build into public tag once thing is publicly announced.
[x] Nothing changes in bodhi, if we consder 'testing' to be enough for
security purpose. But yeah, there could be 'testing-security' optionally
available for users...
Note that this is not security-only. That's the reason for
'prepared-rpms' prefix, e.g. if we had something like that in Fedora, we
could test/use this feature several times a year as we are informed by
PostgreSQL upstream about upcoming releases, we have tarball in advance
... but now it is shame we are not able to announce updates immediately
with upstream. We are not allowed to share the tarballs with upstream before
announcement, of course.
Pavel
6:33 a.m.
On 10/26/2016 12:23 PM, Pavel Raiskup wrote:
On Tuesday, October 25, 2016 7:37:32 PM CEST Kevin Fenzi wrote:
>> 3. AFAIK Fedora has no means by which it can participate in embargoed
>> updates. For this to work, I think there ought to be private git
>> branches, a way to get Koji to make a private build from a private git
>> branch, and a way to get private karma on a private update. Then,
>> when an embargo is lifted, the packager could merge the private branch
>> in, the various infrastructure bits could notice that the very same
>> git commit is now public and permit all of the private builds,
>> updates, and karma to become public and allow an immediate push to
>> updates.
>
> Yep. Thats a gigantic pile of work there for sure.
That's too vague statement, really. Can you make a better estimation? As
far as I understand, there are processes in Debian which allow them
preparing CVE builds so they are able to provide "testing" builds to users
immediately after the public announcement.
Debian has a completely separate installation of its equivalent to Koji
(the dak part, the builders are separate from archive management).
Nowadays, it's source code is mostly up-to-date to what the main archive
uses, but there are still lingering data synchronization issues.
Debian does not build from SCM, but directly from maintainer-uploaded
source packages, so there is no need to have a private SCM.
The security archive is separate as well, and it is served by a
Debian-maintained push mirror network, unlike the main archive, where
most users use third-party mirrors.
For Fedora, I would suggest to replicate the separate security archive
with its push mirrors. The way the Fedora updates repository is updated
seems to cause far more delays than what is lost due to build delays
(the only part the embargoed builders could improve).
Florian
6:45 a.m.
On Wed, Oct 26, 2016 at 7:33 AM, Florian Weimer <fweimer(a)redhat.com> wrote:
Debian has a completely separate installation of its equivalent to Koji (the
dak part, the builders are separate from archive management). Nowadays, it's
source code is mostly up-to-date to what the main archive uses, but there
are still lingering data synchronization issues.
That's a lot of work for not a lot of benefit. Our Koji system should
be able to be extended to handle these extra use-cases. And as it is,
we're trying to get rid of duplicate Koji instances.
For Fedora, I would suggest to replicate the separate security archive with
its push mirrors. The way the Fedora updates repository is updated seems to
cause far more delays than what is lost due to build delays (the only part
the embargoed builders could improve).
There's no point to this, since our updateinfo metadata includes the
classifications, and the Debian/Ubuntu approach means that the same
package needs to be pushed to both the security repository and the
regular updates one. This is not great for keeping things sane for
mirrors.
However, extending Koji to support "hidden builds" is certainly a good
idea. From the SCM side, I'm not sure if gitolite supports making
private branches. At least it should be possible to give each package
a shadow git repository in a separate namespace (as mentioned earlier)
that would be private for doing these kinds of builds and releases.
Upon merge to the public repository, the builds could be marked public
to Koji (since the commit hashes should remain the same post-merge
anyway, we can track using that).
--
真実はいつも一つ！/ Always, there's only one truth!
6:54 a.m.
On Wed, Oct 26, 2016 at 7:45 AM, Neal Gompa <ngompa13(a)gmail.com> wrote:
On Wed, Oct 26, 2016 at 7:33 AM, Florian Weimer
<fweimer(a)redhat.com> wrote:
>
>
> Debian has a completely separate installation of its equivalent to Koji (the
> dak part, the builders are separate from archive management). Nowadays, it's
> source code is mostly up-to-date to what the main archive uses, but there
> are still lingering data synchronization issues.
That's a lot of work for not a lot of benefit. Our Koji system should
be able to be extended to handle these extra use-cases. And as it is,
we're trying to get rid of duplicate Koji instances.
>
> For Fedora, I would suggest to replicate the separate security archive with
> its push mirrors. The way the Fedora updates repository is updated seems to
> cause far more delays than what is lost due to build delays (the only part
> the embargoed builders could improve).
>
There's no point to this, since our updateinfo metadata includes the
classifications, and the Debian/Ubuntu approach means that the same
package needs to be pushed to both the security repository and the
regular updates one. This is not great for keeping things sane for
mirrors.
However, extending Koji to support "hidden builds" is certainly a good
idea. From the SCM side, I'm not sure if gitolite supports making
private branches. At least it should be possible to give each package
a shadow git repository in a separate namespace (as mentioned earlier)
that would be private for doing these kinds of builds and releases.
Upon merge to the public repository, the builds could be marked public
to Koji (since the commit hashes should remain the same post-merge
anyway, we can track using that).
This is directed at the whole thread, not Neal.
Here is the problem with all of this. We have plenty of ideas, but
nobody willing to do the work. So you can all continue to throw out
ideas and question estimates, but until someone signs up to actually
start coding things across whatever stacks are needed, nothing will
change.
If anyone is really interested in solving this problem, it's time to
roll up your sleeves. We need code now, not ideas.
josh
7:03 a.m.
On 10/26/2016 01:45 PM, Neal Gompa wrote:
> For Fedora, I would suggest to replicate the separate security
archive with
> its push mirrors. The way the Fedora updates repository is updated seems to
> cause far more delays than what is lost due to build delays (the only part
> the embargoed builders could improve).
>
There's no point to this, since our updateinfo metadata includes the
classifications,
This doesn't help if the data is published only once per day.
and the Debian/Ubuntu approach means that the same
package needs to be pushed to both the security repository and the
regular updates one. This is not great for keeping things sane for
mirrors.
Of course, the end user systems would have the security repository
enabled, so timely pushes to the updates repository are not needed
anymore to make updates available.
It should be possible to emulate this setup with a COPR repository even
today.
However, extending Koji to support "hidden builds" is
certainly a good
idea.
Trust me, it's not. Embargoes are against the spirit of Fedora, and a
general hassle for everyone involved. Deploying a lot of infrastructure
for the one or two cases per year where it comes in handy does not make
sense.
Florian
7:31 a.m.
On Wednesday, October 26, 2016 2:03:20 PM CEST Florian Weimer wrote:
> However, extending Koji to support "hidden builds" is
certainly a good
> idea.
Trust me, it's not. Embargoes are against the spirit of Fedora, and a
general hassle for everyone involved.
Vague argument, sorry. Please elaborate what's against Fedora.
The "status quo" is bad, our _users_ are the ones who suffer from delays
in CVE fixes. We should care take about users, in the first place.
Deploying a lot of infrastructure for the one or two cases per year
where it comes in handy does not make sense.
That's more than _two cases_ a year and I'm talking about _single_
package, at least I've heard mariadb/mysql guys have it similarly.
Upstream maintainers are *asking us* in advance to report them back
whether the release tarball works for us! And they would willingly fix
the release tarballs or help us with issues, but we are unable to respond
(that's shame). Yeah, I'm able to do my local build on x86_64 machine
(and build in i686 mock), but that's everything I can do; if the aarch64
fails for me, then we'll have nontrivial delay..
Side note: Really typical for Fedora devel, we always suggest people to
do either of those:
- Do nothing.
- Wait. Other (in many times unrelated project) will fix this. (== never)
- Deal with that, that's not what Fedora is about.
I'm OK to accept the fact "hidden builds" are not perfect approach,
that's
right, but that could be relatively easy for implementation. Better to have
something in the beginning rather than wait for "expensive feature" which
we'll
never have enough manpower for.
Pavel
7:42 a.m.
On 10/26/2016 02:31 PM, Pavel Raiskup wrote:
On Wednesday, October 26, 2016 2:03:20 PM CEST Florian Weimer wrote:
>> However, extending Koji to support "hidden builds" is certainly a good
>> idea.
>
> Trust me, it's not. Embargoes are against the spirit of Fedora, and a
> general hassle for everyone involved.
Vague argument, sorry. Please elaborate what's against Fedora.
Looks like I was wrong. I couldn't find an explicit commitment towards
transparency and openness. Debian has one when it comes to bugs, but
Fedora does not.
The "status quo" is bad, our _users_ are the ones who
suffer from delays
in CVE fixes. We should care take about users, in the first place.
The question is where these delays come from.
If a (allegedly non-critical) Firefox update is delayed by weeks, this
is certainly not caused by not being able to build embargoed package
updates for Fedora.
In fact, I strongly suggest that any delay by more than one day is *not*
caused by lack of embargo support in the Fedora infrastructure because
the delay added by fedpkg push && fedpkg build rarely amounts to that.
There are certainly security updates which require more time to prepare,
but those typically require coordination over several packages and
involve potentially backwards-incompatible changes. This is hardly
something which is feasible to do under embargo.
> Deploying a lot of infrastructure for the one or two cases per
year
> where it comes in handy does not make sense.
That's more than _two cases_ a year
Could you name a few cases where this would make a difference? If there
are so many, surely it's easy to come up with a couple of examples.
Than we could look at Fedora update delays for those and see to what
extent building fixed packages in secret, during the embargo period,
would reduce those delays.
Upstream maintainers are *asking us* in advance to report them back
whether the release tarball works for us! And they would willingly fix
the release tarballs or help us with issues, but we are unable to respond
(that's shame). Yeah, I'm able to do my local build on x86_64 machine
(and build in i686 mock), but that's everything I can do; if the aarch64
fails for me, then we'll have nontrivial delay..
Again, concrete examples here (and not for rebases targeting rawhide),
would be more convincing.
I'm OK to accept the fact "hidden builds" are not
perfect approach, that's
right, but that could be relatively easy for implementation.
I haven't tried to implement anything in Koji, so I'm not sure how easy
it is to get there. There seem to be a lot of corner cases to consider:
Should fedpkg new-sources uploads default to private? Are all packages
allowed to see all embargoed builds? Do we need one temporary build
root per embargoed fix to deal with dependency chains?
These look like difficult decisions, and that's not even about
implementation at all.
Thanks,
Florian
9:01 a.m.
On Monday, October 31, 2016 1:42:12 PM CET Florian Weimer wrote:
On 10/26/2016 02:31 PM, Pavel Raiskup wrote:
> On Wednesday, October 26, 2016 2:03:20 PM CEST Florian Weimer wrote:
>>> However, extending Koji to support "hidden builds" is certainly a
good
>>> idea.
>>
>> Trust me, it's not. Embargoes are against the spirit of Fedora, and a
>> general hassle for everyone involved.
>
> Vague argument, sorry. Please elaborate what's against Fedora.
Looks like I was wrong. I couldn't find an explicit commitment towards
transparency and openness.
Hms, can't comment this personally.
Debian has one when it comes to bugs, but Fedora does not.
So how debian treats embargo bugs? As I understood it previously, it
looked like they are able to build in advance, or not? Do they consider
this "open" enough?
Do we consider temporary window before transparentl publishing everything to be
an issue against openness or transparency? To making all of this
acceptable (for me), it requires of course that the package "push" action
is a kind of "atomic":
- builds are published only if CVE is published in the end
- builds are published only if the sources were already published
(immediately before)
- there must be obvious that packer did not build something from crafted
SRPM
> The "status quo" is bad, our _users_ are the ones who
suffer from delays
> in CVE fixes. We should care take about users, in the first place.
The question is where these delays come from.
If a (allegedly non-critical) Firefox update is delayed by weeks, this
is certainly not caused by not being able to build embargoed package
updates for Fedora.
In fact, I strongly suggest that any delay by more than one day is *not*
caused by lack of embargo support in the Fedora infrastructure because
the delay added by fedpkg push && fedpkg build rarely amounts to that.
There are certainly security updates which require more time to prepare,
but those typically require coordination over several packages and
involve potentially backwards-incompatible changes. This is hardly
something which is feasible to do under embargo.
Yep, accepted. But we don't have to cover all use cases.
>> Deploying a lot of infrastructure for the one or two cases
per year
>> where it comes in handy does not make sense.
>
> That's more than _two cases_ a year
Could you name a few cases where this would make a difference? If there
are so many, surely it's easy to come up with a couple of examples.
Starting the list of examples by PostgreSQL, there are several updates a
year (some are security related). Distro packagers are nicely asked to
have a look at the tarballs and perform testing in advance.
Than we could look at Fedora update delays for those and see to what
extent building fixed packages in secret, during the embargo period,
would reduce those delays.
*Full* update delays are usually related to karma++. But once the build
is in 'testing', it is fine from the security POV.
So in the end, the delay is basically between upstream announcement and
(git push; fedpkg build; fedpkg update). It is usually not a huge delay,
because we plan this according to upstream schedule -> but having the
update prepared with full respect to upstream would ease the preparation
phase.
> Upstream maintainers are *asking us* in advance to report them
back
> whether the release tarball works for us! And they would willingly fix
> the release tarballs or help us with issues, but we are unable to respond
> (that's shame). Yeah, I'm able to do my local build on x86_64 machine
> (and build in i686 mock), but that's everything I can do; if the aarch64
> fails for me, then we'll have nontrivial delay..
Again, concrete examples here (and not for rebases targeting rawhide),
would be more convincing.
Personally I can mention only PostgreSQL, there was such upgrade last week
(while I posted my previous message actually). That's not only about
Rawhide (F23+).
> I'm OK to accept the fact "hidden builds" are not
perfect approach, that's
> right, but that could be relatively easy for implementation.
I haven't tried to implement anything in Koji, so I'm not sure how easy
it is to get there. There seem to be a lot of corner cases to consider:
Should fedpkg new-sources uploads default to private? Are all packages
allowed to see all embargoed builds? Do we need one temporary build
root per embargoed fix to deal with dependency chains?
IMO: No && s/packages/packagers/ No && Yes.
These look like difficult decisions, and that's not even about
implementation at all.
No doubts.. Doesn't sound easy at all, so if there was easier way to solve the
problem?
Pavel
7:45 a.m.
On Wednesday, October 26, 2016 1:33:34 PM CEST Florian Weimer wrote:
Debian does not build from SCM, but directly from maintainer-uploaded
source packages, so there is no need to have a private SCM.
Do we have a good marketing for the fact that we are that "superior"
compared to Debian then? Sounds like a main thing for for distro comparison
article: It sounds like this is much, *much* more difficult to get malicious
software into distribution (without noticing) for Fedora packager than for
Debian packager, right?
Pavel
7:45 a.m.
On 10/26/2016 02:45 PM, Pavel Raiskup wrote:
On Wednesday, October 26, 2016 1:33:34 PM CEST Florian Weimer wrote:
> Debian does not build from SCM, but directly from maintainer-uploaded
> source packages, so there is no need to have a private SCM.
Do we have a good marketing for the fact that we are that "superior"
compared to Debian then? Sounds like a main thing for for distro comparison
article: It sounds like this is much, *much* more difficult to get malicious
software into distribution (without noticing) for Fedora packager than for
Debian packager, right?
You need people to actually look at stuff that's being uploaded. I
don't think there is a key difference between Fedora and Debian as far
as this aspect is concerned. D
In addition, Koji likely allows you to create tagged builds which came
from SRPMs, so I don't think there is an actually difference here in
terms of attack surface (at least not in Fedora's favor).
Florian
8:01 a.m.
On Monday, October 31, 2016 1:45:22 PM CET Florian Weimer wrote:
On 10/26/2016 02:45 PM, Pavel Raiskup wrote:
> On Wednesday, October 26, 2016 1:33:34 PM CEST Florian Weimer wrote:
>> Debian does not build from SCM, but directly from maintainer-uploaded
>> source packages, so there is no need to have a private SCM.
>
> Do we have a good marketing for the fact that we are that "superior"
> compared to Debian then? Sounds like a main thing for for distro comparison
> article: It sounds like this is much, *much* more difficult to get malicious
> software into distribution (without noticing) for Fedora packager than for
> Debian packager, right?
You need people to actually look at stuff that's being uploaded. I
don't think there is a key difference between Fedora and Debian as far
as this aspect is concerned. D
In addition, Koji likely allows you to create tagged builds which came
from SRPMs, so I don't think there is an actually difference here in
terms of attack surface (at least not in Fedora's favor).
Do you mean that this is allowed by policy or that this is "implemented"?
Pavel
8:12 a.m.
On 10/31/2016 02:01 PM, Pavel Raiskup wrote:
On Monday, October 31, 2016 1:45:22 PM CET Florian Weimer wrote:
> On 10/26/2016 02:45 PM, Pavel Raiskup wrote:
>> On Wednesday, October 26, 2016 1:33:34 PM CEST Florian Weimer wrote:
>>> Debian does not build from SCM, but directly from maintainer-uploaded
>>> source packages, so there is no need to have a private SCM.
>>
>> Do we have a good marketing for the fact that we are that "superior"
>> compared to Debian then? Sounds like a main thing for for distro comparison
>> article: It sounds like this is much, *much* more difficult to get malicious
>> software into distribution (without noticing) for Fedora packager than for
>> Debian packager, right?
>
> You need people to actually look at stuff that's being uploaded. I
> don't think there is a key difference between Fedora and Debian as far
> as this aspect is concerned. D
>
> In addition, Koji likely allows you to create tagged builds which came
> from SRPMs, so I don't think there is an actually difference here in
> terms of attack surface (at least not in Fedora's favor).
Do you mean that this is allowed by policy or that this is "implemented"?
I don't think Koji implements the necessary build provenience checks to
implement a different policy.
Thanks,
Florian
11:24 a.m.
On Mon, 31 Oct 2016 14:12:24 +0100
Florian Weimer <fweimer(a)redhat.com> wrote:
On 10/31/2016 02:01 PM, Pavel Raiskup wrote:
> On Monday, October 31, 2016 1:45:22 PM CET Florian Weimer wrote:
>> On 10/26/2016 02:45 PM, Pavel Raiskup wrote:
>>> On Wednesday, October 26, 2016 1:33:34 PM CEST Florian Weimer
>>> wrote:
>>>> Debian does not build from SCM, but directly from
>>>> maintainer-uploaded source packages, so there is no need to have
>>>> a private SCM.
>>>
>>> Do we have a good marketing for the fact that we are that
>>> "superior" compared to Debian then? Sounds like a main thing for
>>> for distro comparison article: It sounds like this is much,
>>> *much* more difficult to get malicious software into distribution
>>> (without noticing) for Fedora packager than for Debian packager,
>>> right?
>>
>> You need people to actually look at stuff that's being uploaded. I
>> don't think there is a key difference between Fedora and Debian as
>> far as this aspect is concerned. D
>>
>> In addition, Koji likely allows you to create tagged builds which
>> came from SRPMs, so I don't think there is an actually difference
>> here in terms of attack surface (at least not in Fedora's favor).
>
> Do you mean that this is allowed by policy or that this is
> "implemented"?
I don't think Koji implements the necessary build provenience checks
to implement a different policy.
koji only allows src.rpms to be used for scratch builds or if you have
the koji admin permission. Otherwise you must build all official builds
from our packages git with a known hash.
So, no, regular packagers cannot tag src.rpm builds into anything.
kevin
5:02 p.m.
On 10/31/2016 02:12 PM, Florian Weimer wrote:
On 10/31/2016 02:01 PM, Pavel Raiskup wrote:
> On Monday, October 31, 2016 1:45:22 PM CET Florian Weimer wrote:
>> On 10/26/2016 02:45 PM, Pavel Raiskup wrote:
>>> On Wednesday, October 26, 2016 1:33:34 PM CEST Florian Weimer wrote:
>>>> Debian does not build from SCM, but directly from maintainer-uploaded
>>>> source packages, so there is no need to have a private SCM.
>>>
>>> Do we have a good marketing for the fact that we are that
"superior"
>>> compared to Debian then? Sounds like a main thing for for distro
>>> comparison
>>> article: It sounds like this is much, *much* more difficult to get
>>> malicious
>>> software into distribution (without noticing) for Fedora packager
>>> than for
>>> Debian packager, right?
>>
>> You need people to actually look at stuff that's being uploaded. I
>> don't think there is a key difference between Fedora and Debian as far
>> as this aspect is concerned. D
>>
>> In addition, Koji likely allows you to create tagged builds which came
>> from SRPMs, so I don't think there is an actually difference here in
>> terms of attack surface (at least not in Fedora's favor).
>
> Do you mean that this is allowed by policy or that this is "implemented"?
I don't think Koji implements the necessary build provenience checks to
implement a different policy.
I'm wrong, Koji handles this. Thanks to Kevin for point this out to me.
Florian
9:10 a.m.
On 26 October 2016 at 07:33, Florian Weimer <fweimer(a)redhat.com> wrote:
For Fedora, I would suggest to replicate the separate security
archive with
its push mirrors. The way the Fedora updates repository is updated seems to
cause far more delays than what is lost due to build delays (the only part
the embargoed builders could improve).
Just to clarify because people seem to think we have push mirrors all
the time. Our mirroring system is a pull mirror system where sites
either copy via http or rsync the files from the master mirrors.
Attempts to put in a push mirroring system have failed mainly because
other than a few sites most mirrors have no interest (or ability) in
running any distribution specific software or allowing some other
service push data to them.
Florian
_______________________________________________
devel mailing list -- devel(a)lists.fedoraproject.org
To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
--
Stephen J Smoogen.
7:21 a.m.
On Wed, Oct 26, 2016 at 12:23 PM, Pavel Raiskup <praiskup(a)redhat.com> wrote:
On Tuesday, October 25, 2016 7:37:32 PM CEST Kevin Fenzi wrote:
> > 3. AFAIK Fedora has no means by which it can participate in embargoed
> > updates. For this to work, I think there ought to be private git
> > branches, a way to get Koji to make a private build from a private git
> > branch, and a way to get private karma on a private update. Then,
> > when an embargo is lifted, the packager could merge the private branch
> > in, the various infrastructure bits could notice that the very same
> > git commit is now public and permit all of the private builds,
> > updates, and karma to become public and allow an immediate push to
> > updates.
>
> Yep. Thats a gigantic pile of work there for sure.
That's too vague statement, really. Can you make a better estimation?
We had this discussion in the past and the result from the Board was
that Board won't block hidden private builds (ticket #144, not sure
it's publicly accessible). That time it was me who created the ticket
on behalf of OpenJDK folks. Of course if someone implements it and we
don't have Board but Council and Council can have different opinion on
this. I was told there was some support for private builds in the past
in our tools but it was removed long time ago as there was no use for
that and it complicated the codebase.
Jaroslav
10:54 a.m.
On Wed, 26 Oct 2016 12:23:58 +0200
Pavel Raiskup <praiskup(a)redhat.com> wrote:
On Tuesday, October 25, 2016 7:37:32 PM CEST Kevin Fenzi wrote:
> > 3. AFAIK Fedora has no means by which it can participate in
> > embargoed updates. For this to work, I think there ought to be
> > private git branches, a way to get Koji to make a private build
> > from a private git branch, and a way to get private karma on a
> > private update. Then, when an embargo is lifted, the packager
> > could merge the private branch in, the various infrastructure
> > bits could notice that the very same git commit is now public and
> > permit all of the private builds, updates, and karma to become
> > public and allow an immediate push to updates.
>
> Yep. Thats a gigantic pile of work there for sure.
That's too vague statement, really. Can you make a better
estimation? As far as I understand, there are processes in Debian
which allow them preparing CVE builds so they are able to provide
"testing" builds to users immediately after the public announcement.
Well, I don't think I would be the one to provide an estimate, it would
be the authors of all the tools we use that are affected.
Seems like we need to:
[x] have another git repository, say prepared-rpms/PACKAGE
- that's clone of rpms/PACKAGE
- permission bits inherited from rpms/PACKAGE, but not publicly
available for cloning, + security guys
[x] Making sure that only one repo is writeable:
- if prepared-rpms/PKG exists, rpms/PKG is read only
- the info "why" it is read-only shouldn't be public information
[x] Support for "private" builds in koji. Maintainer should be able
to re-tag security/prepared build into public tag once thing is
publicly announced.
[x] Nothing changes in bodhi, if we consder 'testing' to be enough
for security purpose. But yeah, there could be 'testing-security'
optionally available for users...
Note that this is not security-only. That's the reason for
'prepared-rpms' prefix, e.g. if we had something like that in Fedora,
we could test/use this feature several times a year as we are
informed by PostgreSQL upstream about upcoming releases, we have
tarball in advance ... but now it is shame we are not able to
announce updates immediately with upstream. We are not allowed to
share the tarballs with upstream before announcement, of course.
So whats the lag here there? they announce and it's an hour or two
until you have finished builds and submitted updates? Is an hour or two
really worth all this... complexity?
kevin
2:36 a.m.
On Monday, October 31, 2016 9:54:37 AM CET Kevin Fenzi wrote:
> Note that this is not security-only. That's the reason for
> 'prepared-rpms' prefix, e.g. if we had something like that in Fedora,
> we could test/use this feature several times a year as we are
> informed by PostgreSQL upstream about upcoming releases, we have
> tarball in advance ... but now it is shame we are not able to
> announce updates immediately with upstream. We are not allowed to
> share the tarballs with upstream before announcement, of course.
So whats the lag here there? they announce and it's an hour or two
until you have finished builds and submitted updates? Is an hour or two
really worth all this... complexity?
It's ~1 hour in the best case, but that requires one of package
maintainers is up&running during release time.. which is not very
comfortable. Also the maintainer needs to be ready for failures, fix
mistakes, etc.
Considering we have several days time window for "production" package
preparation, it is unnecessary rush post mortem. Whether is it worth
implementing? Dunno. If the case I described is that rare across
Fedora package set then I wish this became more standard.
Pavel
10:25 a.m.
On Tue, Oct 25, 2016 at 07:37:32PM -0600, Kevin Fenzi wrote:
Nope. We have talked about having some kind of fast track, but IMHO,
we
should just get the normal process faster.
Getting the normal process faster would help in a large number of other
areas. Right now, when we have issues in a spin or other deliverable,
we have to first update the RPM, wait for it to get pushed to testing,
wait for it to get pushed to stable, and then wait for a new release
artifact (iso, qcow, whatever) with that RPM included. This means it
can take several days from fix to testing the fix, and if it takes
several tries to get right, it can easily take a week to address a
simpel problem. It'd be nice if we could reduce that turn-around time
to hours, if not minutes.
--
Matthew Miller
<mattdm(a)fedoraproject.org>
Fedora Project Leader
1:37 a.m.
On Wed, Oct 26, 2016 at 4:25 PM, Matthew Miller
<mattdm(a)fedoraproject.org> wrote:
On Tue, Oct 25, 2016 at 07:37:32PM -0600, Kevin Fenzi wrote:
> Nope. We have talked about having some kind of fast track, but IMHO, we
> should just get the normal process faster.
Getting the normal process faster would help in a large number of other
areas. Right now, when we have issues in a spin or other deliverable,
we have to first update the RPM, wait for it to get pushed to testing,
wait for it to get pushed to stable, and then wait for a new release
artifact (iso, qcow, whatever) with that RPM included. This means it
can take several days from fix to testing the fix, and if it takes
several tries to get right, it can easily take a week to address a
simpel problem. It'd be nice if we could reduce that turn-around time
to hours, if not minutes.
If it takes several goes to get right it's clearly not a simple
problem! And for RCs rel-eng and QA have a process that circumvents
all of the testing phase to get it straight into the compose so most
of what you outline there for delays is actually garbage. We don't use
this process for nightlies because they're not meant to be fire drills
so should follow the right process.
Peter
6:31 a.m.
On Thu, Oct 27, 2016 at 07:37:48AM +0100, Peter Robinson wrote:
> simpel problem. It'd be nice if we could reduce that
turn-around time
> to hours, if not minutes.
If it takes several goes to get right it's clearly not a simple
problem!
I don't think that's so clear -- there's lots of cases where something
seems easy but the details turn out to be fiddly.
And for RCs rel-eng and QA have a process that circumvents
all of the testing phase to get it straight into the compose so most
of what you outline there for delays is actually garbage. We don't use
this process for nightlies because they're not meant to be fire drills
so should follow the right process.
That's what this is all about -- it would be ideal if the "right
process" were also a fast process, so that it doesn't have to be
special cases, circumvention, and firedrills.
--
Matthew Miller
<mattdm(a)fedoraproject.org>
Fedora Project Leader
9 p.m.
If there existed updates-urgent and updates-urgent-testing repositories
for packages like kernel (example: Dirty COW patch-to-testing wait time
was rather long; note that some people cannot install unsigned kernel
packages from koji due to grub2 bugs), FF etc., maybe these large (and
possibly failing) composes could be avoided? Something like fast track
in RHEL.
So, a small and fast couple of repos for really, really important
stuff. Unless I'm misunderstanding how repos work...
--
Bojan
9:28 p.m.
On Wed, 26 Oct 2016 13:00:07 +1100
Bojan Smojver <bojan(a)rexursive.com> wrote:
If there existed updates-urgent and updates-urgent-testing
repositories for packages like kernel (example: Dirty COW
patch-to-testing wait time was rather long; note that some people
cannot install unsigned kernel packages from koji due to grub2 bugs),
FF etc., maybe these large (and possibly failing) composes could be
avoided? Something like fast track in RHEL.
So, a small and fast couple of repos for really, really important
stuff. Unless I'm misunderstanding how repos work...
We worked on something like this last year:
https://fedorahosted.org/rel-eng/ticket/5886
https://fedoraproject.org/wiki/Urgent_updates_policy
We concluded that we needed to use bodhi for it, and were unsure how
much bodhi work it would be to get it to be able to do this, so we went
looking for other easier wins.
kevin
9:50 p.m.
I'm thinking, why not just have these as dump repositories (i.e. just
signed packages) and then have dnf on each system stitch up a repo from
them using createrepo locally. Then you don't need to teach bodhi
anything. And the number of such urgent packages would always be very
low. Essentially an intersection of critical path and high severity
CVEs.
In the meantime, when the regular bodhi composer job sees them, it
picks them up and puts them into updates/updates-testing, as required.
--
Bojan
10:04 p.m.
On Wed, 26 Oct 2016 13:50:11 +1100
Bojan Smojver <bojan(a)rexursive.com> wrote:
I'm thinking, why not just have these as dump repositories (i.e.
just
signed packages) and then have dnf on each system stitch up a repo
from them using createrepo locally. Then you don't need to teach bodhi
anything. And the number of such urgent packages would always be very
low. Essentially an intersection of critical path and high severity
CVEs.
How would dnf know there are packages there without any repodata?
How would it know what key they should be signed by?
Every fedora dnf on every run hits the master mirror for an index?
There would also be no multilib, so people with i686/x86_64 installed
machines could see errors/not update. Also no drpms, but perhaps thats
not a show stopper.
In the meantime, when the regular bodhi composer job sees them, it
picks them up and puts them into updates/updates-testing, as required.
Sure.
kevin
11:08 p.m.
Dnf would need to be taught to do these things, of course
The "on the fly" repositories could be defined like any other, using
repo files. They could be signed by the same keys updates/updates-
testing repos use. Not sure why master mirrors would be required.
Wouldn't regular mirrors work, minus the repomd.xml checksum?
Multilib would be a problem, if this is something that only bodhi can
do.
I guess if that's not doable, then having a jump-the-queue bodhi
mashing of urgent repos would be the only option.
--
Bojan
11:57 p.m.
On 10/26/2016 04:28 AM, Kevin Fenzi wrote:
On Wed, 26 Oct 2016 13:00:07 +1100
Bojan Smojver <bojan(a)rexursive.com> wrote:
> If there existed updates-urgent and updates-urgent-testing
> repositories for packages like kernel (example: Dirty COW
> patch-to-testing wait time was rather long; note that some people
> cannot install unsigned kernel packages from koji due to grub2 bugs),
> FF etc., maybe these large (and possibly failing) composes could be
> avoided? Something like fast track in RHEL.
>
> So, a small and fast couple of repos for really, really important
> stuff. Unless I'm misunderstanding how repos work...
We worked on something like this last year:
https://fedorahosted.org/rel-eng/ticket/5886
https://fedoraproject.org/wiki/Urgent_updates_policy
We concluded that we needed to use bodhi for it, and were unsure how
much bodhi work it would be to get it to be able to do this, so we went
looking for other easier wins.
Would it make sense to get *all* updates delivered faster to users instead?
Thanks,
Florian
6 a.m.
On Tue, Oct 25, 2016 at 10:00 PM, Bojan Smojver <bojan(a)rexursive.com> wrote:
If there existed updates-urgent and updates-urgent-testing
repositories
for packages like kernel (example: Dirty COW patch-to-testing wait time
was rather long; note that some people cannot install unsigned kernel
packages from koji due to grub2 bugs), FF etc., maybe these large (and
That part doesn't make sense. If the kernel is built in koji, it has
been signed with the key we use for secure boot purposes. The RPM
signature is separate and doesn't impact the grub2 issue you mention.
So official builds in koji of the kernel are usable by anyone.
josh
3:52 p.m.
I guess I must have misread this as all kernels built in koji, not just
scratch builds. Ouch.
--
Bojan
11:56 p.m.
On 10/26/2016 02:59 AM, Andrew Lutomirski wrote:
3. AFAIK Fedora has no means by which it can participate in
embargoed
updates.
Is this really relevant? For how many updates a year would it make a
difference? I get that participating in embargoes is important to some
people, but in the grand scheme of things, they just complicate things
tremendously.
Doing community QA on embargoed packages is rather difficult, and QA is
usually what takes the most time, not getting the update package updated
or built.
The other avoidable delay is getting the updated repository to users,
and having pre-packaged and pre-built updates won't help there, either.
Florian
10:22 a.m.
On Tue, Oct 25, 2016 at 05:59:24PM -0700, Andrew Lutomirski wrote:
1. Updates, even critical security updates, are very slow. Getting
an
update out involves creating a build and an update (which is
reasonably fast for most packages), pushing the update to
There are two open tickets on issues related to this:
Release Engineering:
https://fedorahosted.org/rel-eng/ticket/5886
Security Team:
https://fedorahosted.org/fedora-security-team/ticket/3
Both of these are stalled and could use help.
--
Matthew Miller
<mattdm(a)fedoraproject.org>
Fedora Project Leader
9:37 p.m.
There are some good thoughts in this thread. A few people have
suggested that getting the update process to go faster would really
help with these problems, and I agree.
Patrick Uiterwijk has recently made quite a few contributions to Bodhi
that a) make it more reliable, and b) allow it to gate on automatic
signatures. I believe these two areas of improvement in Bodhi will get
us closer to automating Bodhi. I'm sure we still have bugs that we will
need to shake out to get it there, but that's the goal I would like to
see us achieve and I don't think it's unrealistic. I expect that
automating Bodhi will make our update process a little bit faster, and
I'm sure there are many other opportunities for improvement that we can
tackle as well. Contributions welcome!
By the way, I am planning to release Bodhi 2.3.0 tomorrow which has the
automatic signing gating feature that Patrick wrote for us. The draft
release notes are here:
https://github.com/fedora-infra/bodhi/pull/1054
Once that's in place, there are nice little icons that render on Update
pages to indicate whether the builds are signed yet or not. Once the
builds are signed, the Update can then progress from pending to testing
during the next push.
5:41 a.m.
1. and 2.: Yes, it often takes at least 3 days for security critical updates in important
packages (e.g. kernel update to 4.8.3) to land.
I think the real challenge here is to continue shipping quality software while reducing
time to ship. Scratch builds and release-monitoring.org (Anitya) have improved this a lot,
effectively providing finished update builds within <24 hours after upstream update
release – at least for some packages with upstream tracked there. Parallel to bodhi/koji
preparing the update the maintainer can review code and have the update ready to submit to
updates-testing.
Some reasons why updates take some time to be shipped to the user, plus some solutions or
open questions:
* time goes by before maintainer and build system know there is an update. Solution:
release-monitoring.org. Packages just have to use it.
* building the package takes some time. Solution: Auto-build feature. Package maintainers
just have to use it. Doesn't work well with all packages.
* mainainer response: Many packages have only one effective maintainer or none at all.
Solution: Not easy, since we don't have enough maintainers
* when submitting a build to updates-testing or updates, the repo quite often is in
"locked" state causing a delay of e.g. 10 hours. This should be fixed.
* testing: more people could contribute to test packages
* mirrors are often outdated. I've seen delays of more than 2 days earlier this year.
From my subjective perspective this has improved, but getting updates to all mirrors in
<24 hours is probably too hard to do.
6:13 a.m.
On Wed, Nov 2, 2016 at 10:41 AM, Christian Stadelmann
<genodeftest(a)fedoraproject.org> wrote:
1. and 2.: Yes, it often takes at least 3 days for security critical
updates in important packages (e.g. kernel update to 4.8.3) to land.
I think the real challenge here is to continue shipping quality software while reducing
time to ship. Scratch builds and release-monitoring.org (Anitya) have improved this a lot,
effectively providing finished update builds within <24 hours after upstream update
release – at least for some packages with upstream tracked there. Parallel to bodhi/koji
preparing the update the maintainer can review code and have the update ready to submit to
updates-testing.
Some reasons why updates take some time to be shipped to the user, plus some solutions or
open questions:
* time goes by before maintainer and build system know there is an update. Solution:
release-monitoring.org. Packages just have to use it.
* building the package takes some time. Solution: Auto-build feature. Package maintainers
just have to use it. Doesn't work well with all packages.
They fix some of the issues but can often cause just as many more. For
this to be effective we need also a decent CI platform to run
automated tests against API/ABIs and other such things. The auto build
also doesn't have the knowledge to deal with things like patches IE
does the patch need to be rebased or dropped due to upstream status
(is the patch Fedora specific, was it pulled from upstream awaiting a
new release etc).
* mainainer response: Many packages have only one effective
maintainer or none at all. Solution: Not easy, since we don't have enough maintainers
There's ways to address this, but also a lot of the CVEs for packages
are local exploit only and a lot of the packages are also corner
cases. In the case of something that's remotely exploitable there's
often people around with the ability to deal with any of the packages
and people (I regularly assist) are able to assist.
* when submitting a build to updates-testing or updates, the repo
quite often is in "locked" state causing a delay of e.g. 10 hours. This should
be fixed.
That doesn't stop an update from being submitted, updates once they
are submitted are only locked when they are being actively processed
and pushed out as part of an updates push. There were some bugs in
bodhi for this but they are now fixed.
* testing: more people could contribute to test packages
This is a problem for all updates! And even when people are testing
(all my systems always have updates-testing enabled) it requires them
to provide active feedback through karma for their testing to be
effective.
* mirrors are often outdated. I've seen delays of more than 2
days earlier this year. From my subjective perspective this has improved, but getting
updates to all mirrors in <24 hours is probably too hard to do.
It's also something completely out of our control, mirrors are in
control and pull from the primary mirrors, we have no functionality to
push to them.
2742
days inactive
2749
days old
39 comments
14 participants
participants (14)
-
Adam Williamson -
Andrew Lutomirski -
Bojan Smojver -
Christian Stadelmann -
Florian Weimer -
Jaroslav Reznik -
Josh Boyer -
Kevin Fenzi -
Matthew Miller -
Neal Gompa -
Pavel Raiskup -
Peter Robinson -
Randy Barlow -
Stephen John Smoogen