On 1/31/19 4:24 AM, Igor Gnatenko wrote:
ProblemsProblem №1: Build-only packages
Some ecosystems have many build-only packages (packages which are used to
build other packages, without having them installed on end-user systems).
Those ecosystems include Java, Rust and Go.
It is extremely hard to keep up maintaining them due to lack of
time/people. Upstreams are usually changing quickly (sometimes when you
update just one such package, you’d need to update tens of the
dependencies). Few facts about such packages:
-
They are almost always outdated in released versions of distribution;
-
They are often FTBFS (e.g. because there was compiler update but not
package update, broken deps, broken APIs due to deps rebases, …).
In Rust ecosystem, we abuse Rawhide to build and store such packages there
and then generate end-user application (e.g. ripgrep) which uses some of
those packages and produces binary for all supported releases. Those
applications have high quality and are supported.
Rawhide gating makes this much more complicated because builds appear in
buildroot slower, updating group of packages would need side tags and it’s
just painful to work with.
Note that the rawhide gating plans include self service side tags, so
you can make one anytime you want to update a collection of packages.
This of course doesn't solve the above problem, but it's worth
mentioning I think.
https://pagure.io/fesco/issue/2068
And, after all, those packages shouldn’t be shipped to users.
This of course means however that users who want to build their own
verions of the applications or whatever can't start from our repos, they
have to use whatever setup we use for developing them. I don't know any
answers here but it seems to me we keep making it harder for people to
use Fedora to develop applications.
...snip...
Solution
-
Separate Koji + Koschei deployed in Fedora infrastructure cloud;
Turns out we are going to be retiring our cloud, so no, this is not a
place to put it. :)
-
Builders are optimized for the best performance (see below
<
https://docs.google.com/document/d/1DtNTCa-gXc0ILDDHPyAcAca2GGGlENavfgPwG...
);
-
People can have their own targets where they can break things as they
wish without affecting others;
Would that be entirely self service? Or ?
-
Package changes are eventually contributed to Fedora proper by merging
changes to Fedora dist-git and rebuilding packages in official Fedora Koji;
Where is the source for the changes from? Would this need it's own
pagure/src/dist-git? Or would it pull from PR's in prod dist git? Or ?
-
All improvements can eventually be contributed back to official Fedora
infra.
Sounds great!
Ideas
All ideas which you’ll find below are just an ideas and do not have to be
implemented.
Idea №1: Automated legal review
openSUSE released their review app called Cavil
<
https://github.com/openSUSE/cavil> which we could try running in automated
way.
I guess this would be up to legal to look at and tell us if it's worth
doing.
Idea №2: Automated package review
Currently review process is burden and we could run automated legal review,
create a repo, run set of checks and notify person. Somewhat similar to
fresque <
https://github.com/fedora-infra/fresque>. In future might be
integrated with approval process and auto-imported into Fedora.
Sounds great if someone has time to build the app(s)
Idea №3: Building packages for multiple distributions
In Rust ecosystem, we managed to get have same packaging guidelines in
openSUSE, Mageia and Fedora. We could automatically build some packages on
multiple distributions and be kinda upstream.
So, this setup will need to distribute things... would it need mirrors
as well? or ?
Idea №4: Building custom images with user content
Deploy (or build) a tool(s) to enable user-built install, appliance and
container images with their content (modulo restrictions similar to COPR)
on top of Fedora.
Yeah, this has been attempted about 3 times, but if someone has cycles
to work on a app/tool again, great!
Implementation details
-
Koji and Koschei deployed in
fedorainfracloud.org
Nope. There are options of course, but the cloud won't be one. ;)
We could do a remote cloud provider possibly, or we are considering
another openshift cluster with kubevirt or perhaps just some simple
virthost/ansible controlled thing. In any case I think if it's important
we can get resources.
-
A few builders constantly running, with a possibility to add more
builders as needed and as available cloud resources allow
koji doesn't have much way to dynamically deal with builders...
-
Deployed and maintained by proposal owners - not supported by Fedora
infrastructure
-
Other contributors can have access granted by proposal owners (for the
time being only users in “packagers” group will be eligible to get access)
-
Possibility to have some builders running outsides of Fedora
infrastructure - contributed by proposal owners
I'd be carefull of this, as it can cause a lot of issues, but of course
up to you as you are doing the work. :)
-
Koji builds can be done from SCM (src.fp.o, pagure.io, github, …) or
from SRPM upload
Ah, so it would not have it's own git/pagure?
FAQWhy not COPR?
-
COPR has been starved of resources for years, which has impaired its
ability to provide reliable service at scale. Fedora Infrastructure refuses
to consider supporting it officially and Fedora Release Engineering seems
to have some undefined issues with COPR.
Thats kind of glossing over a lot. The reason infrastructure doesn't
want to support it is that its running on a ancient version of openstack
thats very hard to fix when broken. The only issues I know otherwise is
that copr is just designed differently from koji. koji is set to record
everything and keep anything needed to rebuild things, but cope is much
more free form.
In any case with our cloud retirement there will be changes to copr.
Not determined yet, but hopefully it will have more resources and a
solid base to run on.
For the case of this proposal, I agree copr might not be a good fit, as
you are trying to fold all the changes back to koji, so best to just run
a koji to make it as close as possible.
-
It is not official build system of Fedora which is not helping with Problem
№2: Testing of new rpm/koji/mock features/configuration
<
https://docs.google.com/document/d/1DtNTCa-gXc0ILDDHPyAcAca2GGGlENavfgPwG...
Note that if you also are testing new things you could break packagers
that are trying to do other work...
...snip...
How can you make Koji builds faster?
-
By tuning Koji for performance, not correctness and reliability.
-
By building only on a single, fast architecture.
This may well bite you when you merge back to koji and build for all
arches.
...snip...
Overall sounds very nice... we will have to work out details if everyone
is on board with it. Do note that it's going to be a lot of work for you
all... :)
kevin