On 07/07/2017 11:04 PM, Florian Weimer wrote: I don't think anyone has replied to this question so far. We can probably bring it down a bit (to ~95 MiB or thereabouts), but it obviously will grow again. The background of my question is whether we can ship the entire upstream Git history in the SRPM, and add a Git bundle on top of that with the Fedora changes (which can contain merges from the upstream release branch, among other things). There would not be any patch files anymore, just a baseline bundle (unchanged after the upstream release) and an updates bundle. There would have to be a tool to create the updates bundle from the original (non-dist-git) Git repository. It would also extract the glibc.spec file from the source Git repository and synthesize a Release: entry and a bit of %changelog contents (listing only those commits since the last manual %changelog update in the committed glibc.spec file). This is peripherally related to Carlos' ABI checking efforts. Once we follow that model, we could simply put the ABI definitions as plain XML files in a subdirectory in the non-dist-git repository. Changes to the ABI definitions could be reviewed along other changes to it. Thanks, Florian

On 03/07/2018 06:11 PM, Carlos O'Donell wrote: The goal is to have the complete development history in the SRPM, so you could use it for future development if you wanted. I would keep the actual scripts external. The source Git repository contains a glibc.spec file. All changes to the actual SRPM glibc.spec originate from there, with the help of the sync script. The sync script would do something like this: * Get the head commit from the base bundle in some way (we could put that into glibc.spec if anything else fails). * Generate a new incremental bundle from that commit to the current HEAD (or more likely the matching branch under origin/, to encourage that changes have been pushed to the source repository). * Enforce some syntax regarding commit messages (excluding content which comes in through merges from upstream release branches for Fedora). Downstream will probably reject merges altogether, except perhaps for hotfix branches. * Find the last commit which changed the %changelog tail in glibc.spec. * Synthesize an ever-increasing number on top of that (something like “git log --pretty=oneline | wc -l”) and use that to generate a release number which is strictly monotonically increasing. * Summarize the commits since the last %changelog edit, maybe with the help of some headers in the new commits. Add this to the %changelog as a synthesized entry. (From the destination Git, this will patch the existing top-most entry.) * Summarize the commits since the last sync commit to the destination repository and put that into the commit message, including bug numbers required to get past policy receive hooks which are present in some dist-git implementations, and finally commit to the destination dist-git repository. It's not completely trivial to implement this, but it's probably of higher long-term value than any polishing of the existing two-way sync scripts I've developed. 8-/ It's self-sufficient, perhaps except for the policy check against the upstream repository (for recognizing merge commits which do not need specially formatted commit messages). I would strongly suggest to have that part of the tooling externally, so that fixes become available immediately on all branches. At least in Fedora, the script could perhaps copy itself into the source RPM automatically, though, but it might not be entirely straightforward to recognize if the local version is indeed newer than the version being replaced. Yes, we could have an abi/ directory tree with those files there. Any ABI change would also have to update the baseline files. That may not work so well for rawhide, though. Thanks, Florian

On 03/09/2018 07:33 PM, Carlos O'Donell wrote: Oh, I thought you wanted to do continuous verification during development. Then the problem is that if you merge from rawhide, you need to make ABI edits at the same time, and it won't be clear which changes in the merge commit come from the upstream changes, and which are due to adjustments of downstream-only (internal) ABI adjustments as the result of resolving (semantic) merge conflicts. Thanks, Florian

On 03/14/2018 04:37 PM, Carlos O'Donell wrote: Sorry, I meant to write “when we merge changes from upstream master into rawhide”. Thanks, Florian

On 03/14/2018 09:56 PM, Carlos O'Donell wrote: I find (b) rather burdensome because the ABI update has to happen as part of the merge commit. So you have to do the merge, do a build, figure out the ABI differences, applied that to the real merge, and then build again. This is not my idea of a good development process. Thanks, Florian

On 05/14/2018 04:38 PM, Carlos O'Donell wrote: If everything is mechanical and easy to automate, why is it that the only person regularly releasing Fedora updates is me? Do you do a scratch build on all architectures? I don't think this is feasible because it turns every update into a multi-day effort. This would need something that dumps out the ABI differences in a base64 blob (similar to what we use to exfiltrate coredumps). Is this included in your patches? I didn't see it there. I think what you suggests requires local builds, not scratch builds in Koji. This means that one needs to reserve a machine capable of building the relevant Fedora branch. This is currently difficult to automate for various reasons. Thanks, Florian

I've decided to try a different approach first: https://bugzilla.redhat.com/show_bug.cgi?id=1578348 https://pagure.io/releng/issue/7498 This would eliminate the need for manual use of “git archive”, “git bundle”, or whatever is used for generating the data that goes in the source RPM. If Fedora releng decides to support this, I suggest that we structure the new patch process around that, and not custom tooling. (This is independent of where we store the ABI data and where we perform the validation.) Thanks, Florian