Yeah that seems sensible (although I'm not sure of the wording of "what it was meant to be done", but I think I get it). This would satisfy apps built with s2i as long as they are pinning their dependencies with something like poetry or pipenv. We are currently standardizing on poetry but any would do as long as deps are pinned). For s2i based apps, I see two ways of ensuring repeatability, one being stricter but more transparent than the other: 1. have the buildconfig track a production branch upstream, and rely on the build log to know which exact commit was built 2. have the buildconfig specify the commit hash, and change the buildconfig each time we want to deploy a new prod version Option 2 is more transparent because the commit to build is a var in ansible, but it means updating ansible each time we want to make a prod deployment. The workflow for option 1 is simpler because it's just a start-build, but we'll need the logs to know which commit in the prod branch was actually built, and it may be cumbersome to dig up during if something goes wrong. Any preference? As a dev I would be happy with both, they are still infinitely easier than building RPMs. Option 1 being easier for devs, my lazy self leans towards it, but Option 2 is fine as well. Another option that I did not think of? Aurélien

Let me rephrase that and please tell me if I'm correctly representing your thoughts. You loathe somebody else deciding which dependencies you must use. That's fair, it's a distro packager's hell. However in this case I think it's pretty different: we control both the pinning and the packaging (well, image building). In a way, using RPMs does not guarantee reproducibility either: if my app depends on libA-X.Y and it works when I build it, but then libA's maintainer decides to update to X.Z and it breaks my app when I rebuild the image, then having an RPM of my app does not help. To ensure reproducibility we need the versions of the RPMs used at build time, and that's pretty similar to the versions that pip would have pulled at image build time. So, in our case, I suppose storing the list of all versions used would suffice. Or even better: let's store the images themselves and version them. Can the internal OpenShift registry reliably do that? Do we need to switch to something external (quay.io?) to reduce the chance of everything failing at the same time? Then it does not matter whether we track a branch or a commit, we can rollback to the code that was used before by using the previous image. Provided there was no DB schema upgrade, but that's another can of worms. Aurélien

On Wed, May 25, 2022 at 12:54:57AM +0200, Aurelien Bompard wrote: Yeah, in a distro you need to integrate the thing with all the other things and come up with versions that they can all use/share. If something is pinned to a specific version it often indeed causes doom because all the other things have moved on or havent yet and you have to reconcile that with them. well, sure, but it makes it hard for someone else to package it if they want to. :) Indeed. Yeah, saving images sounds good to me for the openshift case. As long as we can rollout an old version we know was working, I think thats just fine. I would prefer not to depend on any external providers for that if we can at all. So, perhaps we need a little tinkering here... take some app...and confirm we can successfully rollout an older version. If we can do that, I think we are fine, pinning or not pinning. At least for the purposes of our deployment. I think that does make it harder for people to package our things, but I suppose thats really an upstream decision how tightly to pin. kevin

On Mon, May 23, 2022 at 6:17 PM Stephen Smoogen <ssmoogen(a)redhat.com&gt; wrote: I like this approach. I don't think there's real value in requiring that everything be packaged as an RPM, but we do want to make sure we can re-deploy correctly. What are the implications for pinning requirements here? Should we require that each application require specific versions of dependencies? I don't love that idea, but I love even less the idea of a stealthy change to a package turning our infrastructure into a cryptocurrency rig. -- Ben Cotton He / Him / His Fedora Program Manager Red Hat TZ=America/Indiana/Indianapolis

On Mon, May 30, 2022 at 11:20 PM Miroslav Suchý <msuchy(a)redhat.com&gt; wrote: The bigger problem is that those applications are *not* able to easily be deployed outside of Fedora infrastructure. One consequence of OpenShift based deployments is that it's become almost too easy to assume nobody else would ever want to run that code. Noggin and Bodhi both have had a number of code changes in the past couple of years that have made it increasingly difficult to use outside of Fedora deployments in their default code state. I've given up on Bodhi, but I'm still trying to get Noggin into good shape. I've been able to evaluate those issues by packaging them as RPMs, because RPM packaging forces a total decoupling of development, deployment, and configuration. None of that is true with our container based deployments. They're not discoverable, and if you can find them, they're not independently useful. Because of this, it becomes hard for community growth around these projects. -- 真実はいつも一つ！/ Always, there's only one truth!

On Mon, 30 May 2022 at 21:26, Neal Gompa <ngompa13(a)gmail.com&gt; wrote: I think we should be clearer about if community growth is expected. 15 to 10 years ago, we tried very hard to grow community support into helping to build things but found that the requirements of what was wanted in the applications always took more time and energy than volunteers had. Over the last 10 years, we have had less and less time to do this community growth work. Fedora Infrastructure's primary mission is to provide a working build system and produce N thousand deliverables every day (be this individually built RPMS, containers, spins, isos, raw images, etc etc). We are to do this while also maintaining the core community features which allow for mail, badges, and outsourced tools to work as best as possible. The amount of resources to actually make it work where we could have people do the marketing and community support needed to make generic apps or even build an infrastructure community is about 2x what we have people for IF we only had to keep the current applications going. However as soon as you hire someone in this space, you find that 3-4 new projects which were waiting to be resourced become high priority. And the timeframes to delivery are shortened as it is no longer acceptable to have a tool in development for N years before initial deployment like Bodhi and some others had. Instead you usually go from prototype to full production in 1 to 1.5 release cycles (6 to 9 months). In order to meet this, we usually have to bake all the 'business rules' of the time into the application. The idea being that when the rules change, the developer will just likely rewrite it from scratch to meet those. [Since one of the lessons learned from when we did 'long form development' was that a lot of code was rewritten every time a new framework which helped solve something better was found.] Finally, I think trying to decouple deployment, development, and configuration is going against the tide. "Infrastructure as Code" sounds great but it generally means that those 3 things are tied together deeply in every application and tool written these days. Like the tide, this is a cyclical pattern in the industry and it will roll out and leave a pile of bad example flotsam that few will remember the next time the tide comes in. -- Stephen Smoogen, Red Hat Automotive Let us be kind to one another, for most of us are fighting a hard battle. -- Ian MacClaren