This idea sounds pretty good to me.
As a less active contributor: How can I help out? Do you have a task
list? (Sorry if this got already answered in one of the follow up
emails.)
Cheers,
Dan
Igor Gnatenko <ignatenkobrain(a)fedoraproject.org> writes:
> MayBe I …(can do something useful)?
>
> Hello,
>
> We've been discussing some (hopefully) nice idea with Mikolaj, Neal and
> Jakub how we could improve packager (and user) experience and we have some
> proposal which will be described below.
>
> We would like to ask you to read it, understand it and ask us any questions
> you have. From the Fedora Infrastructure we would like to ask for some
> machines to implement this idea (you can find some more information in
> "Implementation details" part).
>
> I would like to apologize for HTML email, but it is much easier to have it
> that way to have better visibility and reading easiness.
>
> Feel free to reply to this email or comment in google doc (there is a link
> on the bottom).
> Proposal Owners
>
> -
>
> Mikolaj Izdebski (mizdebsk) <mizdebsk(a)redhat.com> - Java SIG, Fedora
> infrastructure
> -
>
> Igor Gnatenko (ignatenkobrain) <ignatenkobrain(a)fedoraproject.org> - Rust
> SIG, Golang SIG, Neuro SIG, RPM and libsolv contributor
> -
>
> Neal Gompa (ngompa) <ngompa13(a)gmail.com> - Rust SIG, Golang SIG, RPM
> contributor
> -
>
> Jakub Čajka (jcajka) <jcajka(a)fedoraproject.org> - Golang SIG, Container
> SIG
>
>
> This proposal aligns with the objective of improving the packager experience
> <
https://fedoraproject.org/wiki/Objectives/Packager_Experience> by
> developing a platform whereby the proposal owners and others can experiment
> with improvements that can be funneled back into the official production
> infrastructure.
> ProblemsProblem №1: Build-only packages
>
> Some ecosystems have many build-only packages (packages which are used to
> build other packages, without having them installed on end-user systems).
> Those ecosystems include Java, Rust and Go.
>
> It is extremely hard to keep up maintaining them due to lack of
> time/people. Upstreams are usually changing quickly (sometimes when you
> update just one such package, you’d need to update tens of the
> dependencies). Few facts about such packages:
>
> -
>
> They are almost always outdated in released versions of distribution;
> -
>
> They are often FTBFS (e.g. because there was compiler update but not
> package update, broken deps, broken APIs due to deps rebases, …).
>
>
> In Rust ecosystem, we abuse Rawhide to build and store such packages there
> and then generate end-user application (e.g. ripgrep) which uses some of
> those packages and produces binary for all supported releases. Those
> applications have high quality and are supported.
>
> Rawhide gating makes this much more complicated because builds appear in
> buildroot slower, updating group of packages would need side tags and it’s
> just painful to work with.
>
>
https://pagure.io/fesco/issue/2068
>
> And, after all, those packages shouldn’t be shipped to users.
> Problem №2: Testing of new rpm/koji/mock features/configuration
>
> When developing new features in RPM (e.g. rich dependencies) or testing
> different Koji configuration (e.g. removing gcc/gcc-c++ from the
> buildroot), it is required to make custom configuration and try building
> things.
> Problem №3: Developing modules
>
> Modules are built in MBS as a single unit. It is hard to develop big
> modules by progressively improving individual packages or small package
> groups. Scratch builds for modules are not implemented. Local builds work
> differently from official module builds, they don’t scale and don’t allow
> groups of people to work collaboratively. All these problems can be solved
> by first developing a flat repository of packages in a shared environment
> and then generating modulemd from such package set.
> Problem №4: Testing things before push
>
> Primary Fedora Koji and dist-git are not suited for packaging
> experimentation. Packagers are expected to experiment on their own systems
> and push changes of successful experiments only. But this approach doesn’t
> scale with number of maintained packages. Often you can find commits like
> “d’oh, forgot to upload sources” or “let’s try with this settings”. People
> need to build things somewhere quickly, do development and testing. And
> only after that, they should push commit(s) to Fedora.
> Solution
>
> -
>
> Separate Koji + Koschei deployed in Fedora infrastructure cloud;
> -
>
> Builders are optimized for the best performance (see below
>
<
https://docs.google.com/document/d/1DtNTCa-gXc0ILDDHPyAcAca2GGGlENavfgPwG...
> );
> -
>
> People can have their own targets where they can break things as they
> wish without affecting others;
> -
>
> Package changes are eventually contributed to Fedora proper by merging
> changes to Fedora dist-git and rebuilding packages in official Fedora Koji;
> -
>
> All improvements can eventually be contributed back to official Fedora
> infra.
>
> Ideas
>
> All ideas which you’ll find below are just an ideas and do not have to be
> implemented.
> Idea №1: Automated legal review
>
> openSUSE released their review app called Cavil
> <
https://github.com/openSUSE/cavil> which we could try running in automated
> way.
> Idea №2: Automated package review
>
> Currently review process is burden and we could run automated legal review,
> create a repo, run set of checks and notify person. Somewhat similar to
> fresque <
https://github.com/fedora-infra/fresque>. In future might be
> integrated with approval process and auto-imported into Fedora.
> Idea №3: Building packages for multiple distributions
>
> In Rust ecosystem, we managed to get have same packaging guidelines in
> openSUSE, Mageia and Fedora. We could automatically build some packages on
> multiple distributions and be kinda upstream.
> Idea №4: Building custom images with user content
>
> Deploy (or build) a tool(s) to enable user-built install, appliance and
> container images with their content (modulo restrictions similar to COPR)
> on top of Fedora.
> Implementation details
>
> -
>
> Koji and Koschei deployed in
fedorainfracloud.org
> -
>
> A few builders constantly running, with a possibility to add more
> builders as needed and as available cloud resources allow
> -
>
> Deployed and maintained by proposal owners - not supported by Fedora
> infrastructure
> -
>
> Other contributors can have access granted by proposal owners (for the
> time being only users in “packagers” group will be eligible to get access)
> -
>
> Possibility to have some builders running outsides of Fedora
> infrastructure - contributed by proposal owners
> -
>
> Koji builds can be done from SCM (src.fp.o, pagure.io, github, …) or
> from SRPM upload
>
> FAQWhy not COPR?
>
> -
>
> COPR has been starved of resources for years, which has impaired its
> ability to provide reliable service at scale. Fedora Infrastructure refuses
> to consider supporting it officially and Fedora Release Engineering seems
> to have some undefined issues with COPR.
> -
>
> It is not official build system of Fedora which is not helping with Problem
> №2: Testing of new rpm/koji/mock features/configuration
>
<
https://docs.google.com/document/d/1DtNTCa-gXc0ILDDHPyAcAca2GGGlENavfgPwG...
> .
> -
>
> COPR is not extensible enough, more specifically:
> -
>
> No query API (e.g. it is not possible to find out from which SCM
> commit the package has been built)
> -
>
> Builds always have access to all previous builds in a repository
> (i.e. not possible to control when/how repo is created)
> -
>
> GC doesn’t exist
> -
>
> No scratch builds
>
> Why not staging Koji?
>
> -
>
> Staging Koji is meant for testing Koji itself and not for package
> development.
> -
>
> Staging Koji is often in broken state where it is not possible to build
> anything.
> -
>
> No one can touch staging Koji, so it’s pointless for offering a useful
> service.
>
> How can you make Koji builds faster?
>
> -
>
> By tuning Koji for performance, not correctness and reliability.
> -
>
> By building only on a single, fast architecture.
> -
>
> By extensive caching. Files like RPM packages, repodata and files in
> lookaside cache don’t change after they are initially stored, so builders
> can cache them aggressively.
> -
>
> By keeping build repositories small. Some package sets don’t need to be
> built against full Fedora, but can use a minimal subset of Fedora. Such
> repositories can be generated by Koji in seconds, not minutes.
>
> Why not the Open Build Service (OBS)?
>
> -
>
> OBS is not yet packaged for Fedora officially. The upstream code lacks
> adaptations necessary to deploy and run on Fedora and in Fedora
> Infrastructure.
> -
>
> OBS is not official build system of Fedora, which does not help with Problem
> №2: Testing of new rpm/koji/mock features/configuration
>
<
https://docs.google.com/document/d/1DtNTCa-gXc0ILDDHPyAcAca2GGGlENavfgPwG...
> .
>
> What does MBI stand for?
>
> -
>
> M for middlestream, module, mizdebsk, …
> -
>
> B for build, bootstrap, …
> -
>
> I for infrastructure, initiative, ignatenkobrain, …
> -
>
> MBI might be pronounced as “maybe I …(can make something cool)?”
> -
>
> MBI is also initials of mizdebsk (Mikolaj Bozydar Izdebski)
> -
>
> MBI is also IBM spelled backwards :)
>
> Is it somehow related to Fedora Playground
> <
https://fedoraproject.org/wiki/Playground>?
>
> Yes, it is. Although it is more developer-focused. Users could enrol for
> some specific things like experimental Java/Rust packages.
>
>
> MBI (Playground 2.0)
>
<
https://docs.google.com/document/d/1DtNTCa-gXc0ILDDHPyAcAca2GGGlENavfgPwG...
> _______________________________________________
> devel mailing list -- devel(a)lists.fedoraproject.org
> To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct:
https://getfedora.org/code-of-conduct.html
> List Guidelines:
https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org