Wiki - https://fedoraproject.org/wiki/Changes/ReproduciblePackageBuilds Discussion.fpo - https://discussion.fedoraproject.org/t/f41-change-proposal-reproducible-pack...
== Summary ==
A post-build cleanup is integrated into the RPM build process so that common causes of build irreproducibility in packages are removed, making most of Fedora packages reproducible.
== Owner == * Name: Davide Cavalca * Name: Neil Hanlon * Name: [[User:churchyard|Miro Hrončok]] * Name: [[User:zbyszek|Zbigniew Jędrzejewski-Szmek]]
* Email: dcavalca@fedoraproject.org * Email: neil at shrug.pw * Email: mhroncok at redhat.com * Email: zbyszek at in.waw.pl
== Detailed Description == As of 2023 there is an active effort to implement [https://docs.fedoraproject.org/en-US/reproducible-builds/ Reproducible builds] in Fedora. Reproducible builds will allow our users to be able to independently verify that the RPMs have not been tampered with (either maliciously or via hardware/software fault): someone can do an independent rebuild of a package and confirm that they get identical binaries when building with the same versions of the compiler and other tools. This Change allows us to move forward in this direction by removing the common sources of irreproducibility.
[https://github.com/keszybz/add-determinism add-determinism] is a Rust program which, as its name suggests, adds determinism to files that are given as input by attempting to standardize metadata contained in binary or source files to ensure consistency and clamping to $SOURCE_DATE_EPOCH in all instances. `add-determinism` is the "Fedora version" of [https://salsa.debian.org/reproducible-builds/strip-nondeterminism strip-nondeterminism] from the Debian project. Since strip-nondeterminism is written in perl, it is undesirable for use in Fedora, as we don't want to pull perl in the buildroot for every package.
It's worth noting that this Change does not intend to impose any specific reproducibility requirements on Fedora packages. Once this Change is implemented and we have been through a mass rebuild and can verify that the common causes of irreproducibility have indeed been removed, we can consider further steps. But that will be at least one release later.
This change does add a small amount of time to the processing of RPMs at the end of a build. Accordingly, packages containing large quantities or sizes of files can be slower, but this effect is not expected to be noticeable. `add-determinism` takes steps to ensure it does not interfere with other buildroot post processors like `mangle-shebangs`, `python-hardlink`, `python-bytecompile`. It defaults to not doing any modifications in case it doesn't understand the input file or there are any other problems.
A mechanism to opt-out will be provided: to either completely disable the postprocessing step or to disable specific "handlers" (i.e. implementations of cleanup for specific file types, for example static archives). See [https://github.com/keszybz/add-determinism/blob/main/rpm/macros.build-reprod... macros.build-reproducibility].
=== Related Changes === * [https://fedoraproject.org/wiki/Changes/ReproducibleBuildsClampMtimes Clamp build mtimes to SOURCE_DATE_EPOCH] * [https://fedoraproject.org/wiki/Changes/RPM-4.20 RPM 4.20] — this pulls in changes to `%autosetup -S git` which removed a source of irreproducibility.
== Feedback ==
== Benefit to Fedora == Adding determinism (i.e., removing non-determinsim) enables the Fedora community to have confidence that, if given the same source code, build environment, build instructions, and metadata from the build artifacts, any party can recreate copies of the artifacts that are identical except for the signatures and some parts of metadata.
Reproducibility of builds leads to packages of higher quality. It turns out that quite often those irreproducible bits are caused by an error or sloppiness in the code. In particular, any dependence on architecture in noarch packages is almost always unwanted and/or a bug. Test builds that check reproducibility will expose such instances.
Reproducibility of builds makes it easier to develop packages: when a small change is made and a package is rebuilt (in the same environment), then with a reproducible package, the only difference is directly caused by the change. If the package is different every time it is rebuilt, making a comparison is much harder.
Build reproducibility for noarch ''sub''packages solves the problem where package builds on different architectures are different, causing mock to reject the whole build. In particular, this issue occurs for [https://docs.fedoraproject.org/en-US/packaging-guidelines/Python_Appendix/#_... pyc files]. This will now be solved without requiring opt-in from individual packages.
== Scope ==
* Proposal Owners: ** Integrate `add-determinism` as a BuildRoot Policy script ** Add a dependency on `marshalparser` to `python3` (probably conditionalized on `rpm-build`)
* Other Developers: ** Test their packages with the additional phase, report problems ** Potentially integrate changes to packages to enable reproducibility
* Release Engineering: Ideally we want this to happen before the mass rebuild, but that is not strictly required.
* Policies and Guidelines: Fedora Packaging Guidelines should be updated to include information on the add-determinism BuildRoot Policy. User documentation should be amended to include instructions on how to verify reproducibility for a given package, and what packages are known to be non-reproducible, and how to opt-out.
* Trademark approval: N/A (not needed for this Change)
* Alignment with Community Initiatives: All software and requests are consistent with the decision process and similar across other groups in Fedora. The Fedora Reproducibility Working group began at Flock 2023 in Cork.
== Upgrade/compatibility impact ==
No impact is expected.
== How To Test == To test on the level of individual files: * install `add-determinism` * call `SOURCE_DATE_EPOCH=… add-determinism -v ./path/to/file`
To test package builds: * build a local copy of `redhat-rpm-config` with https://src.fedoraproject.org/rpms/redhat-rpm-config/pull-request/293 * install `add-determinism` * build packages ;)
(This can be done on a normal system or in a mock chroot.)
== User Experience == No impact is expected.
== Dependencies ==
== Contingency Plan == * Contingency mechanism: ** In case of major problems, disable the change in `redhat-rpm-config`. ** In case of problems with specific packages, opt-out by setting a macro. * Contingency deadline: No limit really. * Blocks release? No.
== Documentation ==
* [https://docs.fedoraproject.org/en-US/reproducible-builds/ Fedora Reproducible Builds] * [https://github.com/keszybz/add-determinism/blob/main/rpm/macros.build-reprod... add-determinism macros.build-reproducibility] * [https://github.com/keszybz/add-determinism/tree/main?tab=readme-ov-file#buil... add-determinism README]
== Release Notes ==
Fedora package builds are now more deterministic, bringing the distribution closer to the goal of achieving fully reproducible builds for all of its packages.
On Fri, Apr 12, 2024 at 10:41:43PM +0100, Aoife Moloney wrote:
[https://github.com/keszybz/add-determinism add-determinism] is a Rust program which, as its name suggests, adds determinism to files that are given as input by attempting to standardize metadata contained in binary or source files to ensure consistency and clamping to $SOURCE_DATE_EPOCH in all instances. `add-determinism` is the "Fedora version" of [https://salsa.debian.org/reproducible-builds/strip-nondeterminism strip-nondeterminism] from the Debian project. Since strip-nondeterminism is written in perl, it is undesirable for use in Fedora, as we don't want to pull perl in the buildroot for every package.
https://github.com/keszybz/add-determinism looks like a package with a lot of Rust dependencies just to make some small changes to four different file types. Isn't there an easier way to do this? I would have thought a Python library would be more suitable as the most complicated bit is the *.pyc change which is done using Python code.
Rich.
On Sat, Apr 13, 2024 at 3:59 AM Richard W.M. Jones rjones@redhat.com wrote:
On Fri, Apr 12, 2024 at 10:41:43PM +0100, Aoife Moloney wrote:
[https://github.com/keszybz/add-determinism add-determinism] is a Rust program which, as its name suggests, adds determinism to files that are given as input by attempting to standardize metadata contained in binary or source files to ensure consistency and clamping to $SOURCE_DATE_EPOCH in all instances. `add-determinism` is the "Fedora version" of [https://salsa.debian.org/reproducible-builds/strip-nondeterminism strip-nondeterminism] from the Debian project. Since strip-nondeterminism is written in perl, it is undesirable for use in Fedora, as we don't want to pull perl in the buildroot for every package.
https://github.com/keszybz/add-determinism looks like a package with a lot of Rust dependencies just to make some small changes to four different file types. Isn't there an easier way to do this? I would have thought a Python library would be more suitable as the most complicated bit is the *.pyc change which is done using Python code.
Considering Debian's version is in Perl, yes, it's quite reasonable to consider that. It could have been written in Python, C++, or even shell (if you truly hated yourself). I'm not a big fan of Rust for this either. But unless someone offers to make another version that is more appealing, this is what we have.
-- 真実はいつも一つ！/ Always, there's only one truth!
On Sat, Apr 13, 2024 at 04:12:09AM -0500, Neal Gompa wrote:
On Sat, Apr 13, 2024 at 3:59 AM Richard W.M. Jones rjones@redhat.com wrote:
On Fri, Apr 12, 2024 at 10:41:43PM +0100, Aoife Moloney wrote:
[https://github.com/keszybz/add-determinism add-determinism] is a Rust program which, as its name suggests, adds determinism to files that are given as input by attempting to standardize metadata contained in binary or source files to ensure consistency and clamping to $SOURCE_DATE_EPOCH in all instances. `add-determinism` is the "Fedora version" of [https://salsa.debian.org/reproducible-builds/strip-nondeterminism strip-nondeterminism] from the Debian project. Since strip-nondeterminism is written in perl, it is undesirable for use in Fedora, as we don't want to pull perl in the buildroot for every package.
The proposal explicitly states that we don't want Perl in all buildroots. It doesn't say that explicitly, but we also don't want Python. In the past we made an effort to remove gcc and make [https://fedoraproject.org/wiki/Changes/Remove_make_from_BuildRoot, https://fedoraproject.org/wiki/Changes/Remove_GCC_from_BuildRoot], and Python would be a fairly big addition.
If we don't want to pull in an additional language framework, the options are either a compiled language or a scripting language that is already installed anyway, i.e. bash or awk. Considering that we want to do multiprocessing and/or multithreading to make things quick, a compiled language seems better. And among the compiled languages, I think Rust should be the default choice nowadays.
The resulting binary is a single file of 2.6 MB, which is more than it'd be if written in C, but still reasonably small.
https://github.com/keszybz/add-determinism looks like a package with a lot of Rust dependencies just to make some small changes to four different file types. Isn't there an easier way to do this? I would have thought a Python library would be more suitable as the most complicated bit is the *.pyc change which is done using Python code.
The program currently has just four "handlers", but I expect that we'll need to add some more. Those four address issues that were apparent after a "mini mass rebuild" of ~2k packages, but that covered only a subset of packages, and it's possible that those most obvious issues masked other issues, and we didn't analyze all failures in detail, etc. The program is structured so that it should be simple to add additional handlers.
Now I'll try to channel Fabio V.: all the crates that are dependencies are commonly used, i.e. they're in the common set of crates that are used by modern Rust code and we need them to be packaged anyway. The packaging effort for add-determinism was miniscule. (I made a mistake by packaging a snapshot, which the guidelines disallow like I did it, but when doing it correctly, it's really just a matter of running rust2rpm and filling in some blanks.)
Considering Debian's version is in Perl, yes, it's quite reasonable to consider that. It could have been written in Python, C++, or even shell (if you truly hated yourself). I'm not a big fan of Rust for this either. But unless someone offers to make another version that is more appealing, this is what we have.
Yes. But actually I think Rust is the optimal choice here. Writing this in Python would be possibly slightly nicer, but we don't want to pull the interpreter and packages into the buildroot. Python also has the problem (challenge?) that it needs to be bootstrapped once per year. The less packages are involved in the bootstrap, the easier it is. And if the brp was written in Python, we'd need to deal with that, and it would probably increase the number of builds which are done without the cleanup. Having this as an indepedent binary avoids some of the issues with bootstrap.
Please note that add-determinism has: - configurable logging - unit tests - integration tests - multiprocess support - atomic file replacements (for files without hardlinks) - rewriting of files in place (for files with hardlinks)
Doing this in bash would be doable, but not worth it, IMO.
Zbyszek
On Sat, Apr 13, 2024 at 1:18 PM Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl wrote:
Yes. But actually I think Rust is the optimal choice here. Writing this in Python would be possibly slightly nicer, but we don't want to pull the interpreter and packages into the buildroot. Python also has the problem (challenge?) that it needs to be bootstrapped once per year. The less packages are involved in the bootstrap, the easier it is. And if the brp was written in Python, we'd need to deal with that, and it would probably increase the number of builds which are done without the cleanup. Having this as an indepedent binary avoids some of the issues with bootstrap.
I think Rust *would* be a good choice here ... BUT add-determinism uses pyo3 to link to CPython, so it pulls in python3-libs anyway. So you get the downsides of pulling in Python without the upsides of using Rust ...
Fabio
On Sat, Apr 13, 2024 at 01:41:59PM +0200, Fabio Valentini wrote:
On Sat, Apr 13, 2024 at 1:18 PM Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl wrote:
Yes. But actually I think Rust is the optimal choice here. Writing this in Python would be possibly slightly nicer, but we don't want to pull the interpreter and packages into the buildroot. Python also has the problem (challenge?) that it needs to be bootstrapped once per year. The less packages are involved in the bootstrap, the easier it is. And if the brp was written in Python, we'd need to deal with that, and it would probably increase the number of builds which are done without the cleanup. Having this as an indepedent binary avoids some of the issues with bootstrap.
I think Rust *would* be a good choice here ... BUT add-determinism uses pyo3 to link to CPython, so it pulls in python3-libs anyway. So you get the downsides of pulling in Python without the upsides of using Rust ...
Yes, it currently pulls in python3-libs as a dependency, but not the rest of the Python stack. Ideally, the dependency on python3-libs will become optional, and we'll use it if found at runtime if found and ignore otherwise. (Anything that creates pyc files will have python installed, so it's fine if the pyc handler only works if there.) How to best do this is something that needs to be figured out…
Zbyszek
On Sat, Apr 13, 2024 at 01:38:49PM +0000, Zbigniew Jędrzejewski-Szmek wrote:
On Sat, Apr 13, 2024 at 01:41:59PM +0200, Fabio Valentini wrote:
On Sat, Apr 13, 2024 at 1:18 PM Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl wrote:
Yes. But actually I think Rust is the optimal choice here. Writing this in Python would be possibly slightly nicer, but we don't want to pull the interpreter and packages into the buildroot. Python also has the problem (challenge?) that it needs to be bootstrapped once per year. The less packages are involved in the bootstrap, the easier it is. And if the brp was written in Python, we'd need to deal with that, and it would probably increase the number of builds which are done without the cleanup. Having this as an indepedent binary avoids some of the issues with bootstrap.
I think Rust *would* be a good choice here ... BUT add-determinism uses pyo3 to link to CPython, so it pulls in python3-libs anyway. So you get the downsides of pulling in Python without the upsides of using Rust ...
Yes, it currently pulls in python3-libs as a dependency, but not the rest of the Python stack. Ideally, the dependency on python3-libs will become optional, and we'll use it if found at runtime if found and ignore otherwise. (Anything that creates pyc files will have python installed, so it's fine if the pyc handler only works if there.) How to best do this is something that needs to be figured out…
https://github.com/keszybz/add-determinism/pull/1 makes the dependency on libpython optional. One option would be to compile the binary twice, and use rich dependencies to install the heavyweight one if python3 is installed. If somebody has a better approach, I'm all ears.
Zbyszek
Dne 13. 04. 24 v 21:04 Zbigniew Jędrzejewski-Szmek napsal(a):
On Sat, Apr 13, 2024 at 01:38:49PM +0000, Zbigniew Jędrzejewski-Szmek wrote:
On Sat, Apr 13, 2024 at 01:41:59PM +0200, Fabio Valentini wrote:
On Sat, Apr 13, 2024 at 1:18 PM Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl wrote:
Yes. But actually I think Rust is the optimal choice here. Writing this in Python would be possibly slightly nicer, but we don't want to pull the interpreter and packages into the buildroot. Python also has the problem (challenge?) that it needs to be bootstrapped once per year. The less packages are involved in the bootstrap, the easier it is. And if the brp was written in Python, we'd need to deal with that, and it would probably increase the number of builds which are done without the cleanup. Having this as an indepedent binary avoids some of the issues with bootstrap.
I think Rust *would* be a good choice here ... BUT add-determinism uses pyo3 to link to CPython, so it pulls in python3-libs anyway. So you get the downsides of pulling in Python without the upsides of using Rust ...
Yes, it currently pulls in python3-libs as a dependency, but not the rest of the Python stack. Ideally, the dependency on python3-libs will become optional, and we'll use it if found at runtime if found and ignore otherwise. (Anything that creates pyc files will have python installed, so it's fine if the pyc handler only works if there.) How to best do this is something that needs to be figured out…
https://github.com/keszybz/add-determinism/pull/1 makes the dependency on libpython optional. One option would be to compile the binary twice, and use rich dependencies to install the heavyweight one if python3 is installed. If somebody has a better approach, I'm all ears.
Could you please share some comparison of what is impact on buildroot? How many packages added, download size, install size.
Thx a lot
Vít
On Mon, Apr 15, 2024 at 12:59:04PM +0200, Vít Ondruch wrote:
Could you please share some comparison of what is impact on buildroot? How many packages added, download size, install size.
$ rpm -qiR add-determinism-0.2.0-1.fc40.x86_64 ... Size : 2801276 (installed size, 2.7 MB) ... libc.so.6()(64bit) libgcc_s.so.1()(64bit) libbz2.so.1()(64bit) libzstd.so.1()(64bit) libpython3.12.so.1.0()(64bit) ...
So I think the size and deps are neglible, with the exception of libpython3, which is large (45 MB). As discussed, we'll make it optional. I amended the Proposal page now to say that explicitly.
Zbyszek
On sobota 13. dubna 2024 21:04:06, CEST Zbigniew Jędrzejewski-Szmek wrote:
On Sat, Apr 13, 2024 at 01:38:49PM +0000, Zbigniew Jędrzejewski-Szmek wrote:
On Sat, Apr 13, 2024 at 01:41:59PM +0200, Fabio Valentini wrote:
On Sat, Apr 13, 2024 at 1:18 PM Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl wrote:
Yes. But actually I think Rust is the optimal choice here. Writing this in Python would be possibly slightly nicer, but we don't want to pull the interpreter and packages into the buildroot. Python also has the problem (challenge?) that it needs to be bootstrapped once per year. The less packages are involved in the bootstrap, the easier it is. And if the brp was written in Python, we'd need to deal with that, and it would probably increase the number of builds which are done without the cleanup. Having this as an indepedent binary avoids some of the issues with bootstrap.
I think Rust *would* be a good choice here ... BUT add-determinism uses pyo3 to link to CPython, so it pulls in python3-libs anyway. So you get the downsides of pulling in Python without the upsides of using Rust ...
Yes, it currently pulls in python3-libs as a dependency, but not the rest of the Python stack. Ideally, the dependency on python3-libs will become optional, and we'll use it if found at runtime if found and ignore otherwise. (Anything that creates pyc files will have python installed, so it's fine if the pyc handler only works if there.) How to best do this is something that needs to be figured out…
https://github.com/keszybz/add-determinism/pull/1 makes the dependency on libpython optional. One option would be to compile the binary twice, and use rich dependencies to install the heavyweight one if python3 is installed. If somebody has a better approach, I'm all ears.
I do not claim these are better approaches, but you could:
- you could consider using tooling "from host", not "from chroot" (adding a dependency to Mock seems just fine then)
- use dynamic buildrequires to detect what plugins are needed
Pavel
Zbyszek
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
On Tue, Apr 16, 2024 at 09:42:27AM +0200, Pavel Raiskup wrote:
On sobota 13. dubna 2024 21:04:06, CEST Zbigniew Jędrzejewski-Szmek wrote:
On Sat, Apr 13, 2024 at 01:38:49PM +0000, Zbigniew Jędrzejewski-Szmek wrote:
On Sat, Apr 13, 2024 at 01:41:59PM +0200, Fabio Valentini wrote:
On Sat, Apr 13, 2024 at 1:18 PM Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl wrote:
Yes. But actually I think Rust is the optimal choice here. Writing this in Python would be possibly slightly nicer, but we don't want to pull the interpreter and packages into the buildroot. Python also has the problem (challenge?) that it needs to be bootstrapped once per year. The less packages are involved in the bootstrap, the easier it is. And if the brp was written in Python, we'd need to deal with that, and it would probably increase the number of builds which are done without the cleanup. Having this as an indepedent binary avoids some of the issues with bootstrap.
I think Rust *would* be a good choice here ... BUT add-determinism uses pyo3 to link to CPython, so it pulls in python3-libs anyway. So you get the downsides of pulling in Python without the upsides of using Rust ...
Yes, it currently pulls in python3-libs as a dependency, but not the rest of the Python stack. Ideally, the dependency on python3-libs will become optional, and we'll use it if found at runtime if found and ignore otherwise. (Anything that creates pyc files will have python installed, so it's fine if the pyc handler only works if there.) How to best do this is something that needs to be figured out…
https://github.com/keszybz/add-determinism/pull/1 makes the dependency on libpython optional. One option would be to compile the binary twice, and use rich dependencies to install the heavyweight one if python3 is installed. If somebody has a better approach, I'm all ears.
I do not claim these are better approaches, but you could:
- you could consider using tooling "from host", not "from chroot" (adding a dependency to Mock seems just fine then)
Hmm, how would that work? We call mock, which calls systemd-nspawn, which runs rpmbuild, and the build env is completely isolated from the host.
- use dynamic buildrequires to detect what plugins are needed
My problem is that the binary is linked to the libpython3.12.so shared library… The detection part is easy, the hard part is how to have the binary work when the shared lib is not installed.
Zbyszek
Dne 16. 04. 24 v 10:04 odp. Zbigniew Jędrzejewski-Szmek napsal(a):
Hmm, how would that work? We call mock, which calls systemd-nspawn, which runs rpmbuild, and the build env is completely isolated from the host.
By adding this functionality to Mock itself. It can be optional (--add-determinism). And then Mock can call
add-determinism $chroot/%buildroot/
On Wed, Apr 17, 2024 at 08:39:48AM +0200, Miroslav Suchý wrote:
Dne 16. 04. 24 v 10:04 odp. Zbigniew Jędrzejewski-Szmek napsal(a):
Hmm, how would that work? We call mock, which calls systemd-nspawn, which runs rpmbuild, and the build env is completely isolated from the host.
By adding this functionality to Mock itself. It can be optional (--add-determinism). And then Mock can call
add-determinism $chroot/%buildroot/
I don't think we should make this particular functionality special. We have a bunch of brps: + /usr/lib/rpm/check-buildroot + /usr/lib/rpm/redhat/brp-ldconfig + /usr/lib/rpm/brp-compress + /usr/lib/rpm/redhat/brp-strip-lto + /usr/lib/rpm/brp-strip-static-archive + /usr/lib/rpm/check-rpaths + /usr/lib/rpm/redhat/brp-mangle-shebangs + /usr/lib/rpm/brp-remove-la-files + /usr/lib/rpm/redhat/brp-python-bytecompile + /usr/lib/rpm/redhat/brp-python-hardlink And if such a feature is added, it should be generic to allow all and any of those to be moved out of the build env.
In some ways, that'd be nice, because we wouldn't have to install additional tools in the buildroot. But OTOH, those tools are rather small and bash/find/etc probably need to be installed anyway. And also, the package can configure and override the brps via macros. If they were called from the outside, we'd need to at least figure out a new protocol for that.
Certainly an interesting idea, but I'm not convinced, yet.
---
BTW.: + /usr/lib/rpm/brp-remove-la-files
This one means that various packages that remove .la files on their own don't need to do this. Maybe we could have a pp sweep over the specs and drop various 'find %{buildroot} -name *.la -delete'?
Zbyszek
Dne 17. 04. 24 v 9:20 dop. Zbigniew Jędrzejewski-Szmek napsal(a):
By adding this functionality to Mock itself. It can be optional (--add-determinism). And then Mock can call
add-determinism $chroot/%buildroot/
I don't think we should make this particular functionality special. We have a bunch of brps:
It depends... if you want to have this check/sanitization part of rpmbuild. When it is small,and does not inflate buildroot, then fine.
Over the years, I learn that people have different view where each component should go. :) I will not argue.
If you package add-determinism I can help you to add it to Mock. Likely as plugin:
https://rpm-software-management.github.io/mock/#plugins
that is called in `postbuild`
https://rpm-software-management.github.io/mock/Plugin-Hooks
And by helping I mean that I will create the initial PR and you (and others) will test the functionality. Deal?
On Wed, Apr 17, 2024 at 09:38:30AM +0200, Miroslav Suchý wrote:
Dne 17. 04. 24 v 9:20 dop. Zbigniew Jędrzejewski-Szmek napsal(a):
By adding this functionality to Mock itself. It can be optional (--add-determinism). And then Mock can call
add-determinism $chroot/%buildroot/
I don't think we should make this particular functionality special. We have a bunch of brps:
It depends... if you want to have this check/sanitization part of rpmbuild. When it is small,and does not inflate buildroot, then fine.
Over the years, I learn that people have different view where each component should go. :) I will not argue.
If you package add-determinism I can help you to add it to Mock. Likely as plugin:
https://rpm-software-management.github.io/mock/#plugins
that is called in `postbuild`
https://rpm-software-management.github.io/mock/Plugin-Hooks
And by helping I mean that I will create the initial PR and you (and others) will test the functionality. Deal?
Thank you for the offer. I _might_ take you up on it later, but for now, I think it's better to keep this inside of the buildroot.
I don't think that this functionality should be tied to mock. Right now, the helper runs for 'fedpkg local' (as all brps), but if it's moved to mock, then we'd need to at least call it from two places.
Zbyszek
On 4/17/24 12:54 AM, Zbigniew Jędrzejewski-Szmek wrote:
On Wed, Apr 17, 2024 at 09:38:30AM +0200, Miroslav Suchý wrote:
Dne 17. 04. 24 v 9:20 dop. Zbigniew Jędrzejewski-Szmek napsal(a):
By adding this functionality to Mock itself. It can be optional (--add-determinism). And then Mock can call
add-determinism $chroot/%buildroot/
I don't think we should make this particular functionality special. We have a bunch of brps:
It depends... if you want to have this check/sanitization part of rpmbuild. When it is small,and does not inflate buildroot, then fine.
Over the years, I learn that people have different view where each component should go. :) I will not argue.
If you package add-determinism I can help you to add it to Mock. Likely as plugin:
https://rpm-software-management.github.io/mock/#plugins
that is called in `postbuild`
https://rpm-software-management.github.io/mock/Plugin-Hooks
And by helping I mean that I will create the initial PR and you (and others) will test the functionality. Deal?
Thank you for the offer. I _might_ take you up on it later, but for now, I think it's better to keep this inside of the buildroot.
I don't think that this functionality should be tied to mock. Right now, the helper runs for 'fedpkg local' (as all brps), but if it's moved to mock, then we'd need to at least call it from two places.
A lot of our security libraries create cryptographic checksums on the binaries that need to be correct in order for them to run in FIPS mode. If this is actually changing the binaries, we'll need to rerun those checksums.
bob
Zbyszek
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
On 17/04/2024 09:20, Zbigniew Jędrzejewski-Szmek wrote:
In some ways, that'd be nice, because we wouldn't have to install additional tools in the buildroot. But OTOH, those tools are rather small and bash/find/etc probably need to be installed anyway.
It doesn't seem to work properly without the marshalparser Python package installed:
+ /usr/bin/add-determinism --brp -j48 /builddir/build/BUILDROOT/nheko-0.11.3-11.fc41.x86_64 Cannot initialize handler pyc: ModuleNotFoundError: No module named 'marshalparser' [src/multiprocess.rs:66:9] &cmd = Command { program: "/usr/bin/add-determinism", args: [ "/usr/bin/add-determinism", "--socket", "3", "--brp", "--handler", "ar,jar,javadoc", ], env: CommandEnv { clear: false, vars: { "SOURCE_DATE_EPOCH": Some( "1716249600", ), }, }, create_pidfd: false, } Bye! Bye! Bye! Bye!
Also, is it possible to suppress these "Bye!" messages?
On Wed, May 22, 2024 at 09:06:07AM +0200, Vitaly Zaitsev via devel wrote:
On 17/04/2024 09:20, Zbigniew Jędrzejewski-Szmek wrote:
In some ways, that'd be nice, because we wouldn't have to install additional tools in the buildroot. But OTOH, those tools are rather small and bash/find/etc probably need to be installed anyway.
It doesn't seem to work properly without the marshalparser Python package installed:
- /usr/bin/add-determinism --brp -j48
/builddir/build/BUILDROOT/nheko-0.11.3-11.fc41.x86_64 Cannot initialize handler pyc: ModuleNotFoundError: No module named 'marshalparser'
Yes. The plan is to pull in the marshalparser module from python3-devel (conditionalized on python not being bootstrapped). The lack of the marshalparser module means that the program doesn't handle .pyc files, but otherwise works.
Also, is it possible to suppress these "Bye!" messages?
Yeah, I left in the debugging message to have more clarity if things are working. They'll be gone in the next version.
Zbyszek
Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl wrote:
[…]
- use dynamic buildrequires to detect what plugins are needed
My problem is that the binary is linked to the libpython3.12.so shared library… The detection part is easy, the hard part is how to have the binary work when the shared lib is not installed.
Quick 'n' dirty: Have two binaries, unconditionally call add-determinism-python for *.pyc files, either from add-determinism or the BRP macro (which essentially should be called when %__brp_python_bytecompile is called?), rely on the packager to build-require add-determinism-python or require that from python3-devel (the missing binary should fail the build otherwise).
Tim
On Wed, Apr 17, 2024, 08:45 Tim Landscheidt tim@tim-landscheidt.de wrote:
Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl wrote:
[…]
- use dynamic buildrequires to detect what plugins are needed
My problem is that the binary is linked to the libpython3.12.so shared library… The detection part is easy, the hard part is how to have the binary work when the shared lib is not installed.
Quick 'n' dirty: Have two binaries, unconditionally call add-determinism-python for *.pyc files, either from add-determinism or the BRP macro (which essentially should be called when %__brp_python_bytecompile is called?), rely on the packager to build-require add-determinism-python or require that from python3-devel (the missing binary should fail the build otherwise).
Something like this could be even made automatic.
- split Python-specific functionality into a separate binary and subpackage of add-determinism - add only add-determinism to the default buildroot - add "Requires: (add-determinism-python if python3)" to add-determinism
That way the pyc processing functionality would only be pulled in iff python3 is already getting installed by something else.
Fabio
Tim
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
On Wed, Apr 17, 2024 at 5:30 AM Fabio Valentini decathorpe@gmail.com wrote:
On Wed, Apr 17, 2024, 08:45 Tim Landscheidt tim@tim-landscheidt.de wrote:
Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl wrote:
[…]
- use dynamic buildrequires to detect what plugins are needed
My problem is that the binary is linked to the libpython3.12.so shared library… The detection part is easy, the hard part is how to have the binary work when the shared lib is not installed.
Quick 'n' dirty: Have two binaries, unconditionally call add-determinism-python for *.pyc files, either from add-determinism or the BRP macro (which essentially should be called when %__brp_python_bytecompile is called?), rely on the packager to build-require add-determinism-python or require that from python3-devel (the missing binary should fail the build otherwise).
Something like this could be even made automatic.
- split Python-specific functionality into a separate binary and subpackage of add-determinism
- add only add-determinism to the default buildroot
- add "Requires: (add-determinism-python if python3)" to add-determinism
That way the pyc processing functionality would only be pulled in iff python3 is already getting installed by something else.
Because this is written in Rust instead of Python, you will need a build variant for *every* Python interpreter shipped in Fedora.
-- 真実はいつも一つ！/ Always, there's only one truth!
On Wed, Apr 17, 2024 at 08:58:46AM -0400, Neal Gompa wrote:
On Wed, Apr 17, 2024 at 5:30 AM Fabio Valentini decathorpe@gmail.com wrote:
On Wed, Apr 17, 2024, 08:45 Tim Landscheidt tim@tim-landscheidt.de wrote:
Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl wrote:
[…]
- use dynamic buildrequires to detect what plugins are needed
My problem is that the binary is linked to the libpython3.12.so shared library… The detection part is easy, the hard part is how to have the binary work when the shared lib is not installed.
Quick 'n' dirty: Have two binaries, unconditionally call add-determinism-python for *.pyc files, either from add-determinism or the BRP macro (which essentially should be called when %__brp_python_bytecompile is called?), rely on the packager to build-require add-determinism-python or require that from python3-devel (the missing binary should fail the build otherwise).
Something like this could be even made automatic.
- split Python-specific functionality into a separate binary and subpackage of add-determinism
- add only add-determinism to the default buildroot
- add "Requires: (add-determinism-python if python3)" to add-determinism
That way the pyc processing functionality would only be pulled in iff python3 is already getting installed by something else.
Because this is written in Rust instead of Python, you will need a build variant for *every* Python interpreter shipped in Fedora.
No, just one one, at any given time.
Or in other words, it's the same as any application using Python in Fedora: it is built against some version of Python, usually the default one.
(pyo3 has support for linking to the stable python abi, which theoretically would allow the program to be able to run with python versions newer than the one against which it was built. I didn't look at the details, so I don't know what would be needed to link it like that and whether that'd actually make things better for us.)
Zbyszek
On Wed, Apr 17, 2024 at 3:10 PM Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl wrote:
Because this is written in Rust instead of Python, you will need a build variant for *every* Python interpreter shipped in Fedora.
No, just one one, at any given time.
Assuming that the marshalparser package can parse pyc files from newer Python versions, I don't think this is necessary.
Or in other words, it's the same as any application using Python in Fedora: it is built against some version of Python, usually the default one.
(pyo3 has support for linking to the stable python abi, which theoretically would allow the program to be able to run with python versions newer than the one against which it was built. I didn't look at the details, so I don't know what would be needed to link it like that and whether that'd actually make things better for us.)
The functionality for building + linking only with the stable / limited "abi3" CPython ABI is only relevant for *extension modules*, i.e. Python modules that contain a native extension written in Rust. This is not the case here - it's a Rust program that calls into CPython (as opposed to the Python interpreter loading a native extension module, which is effectively a "plugin" which is not linked to libpython directly). add-determinism is linked *directly* to libpython3.x.so, so it is only ever compatible with the major version of the Python interpreter that it was built against.
Fabio
On Wed, Apr 17, 2024 at 11:28:53AM +0200, Fabio Valentini wrote:
On Wed, Apr 17, 2024, 08:45 Tim Landscheidt tim@tim-landscheidt.de wrote:
Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl wrote:
[…]
- use dynamic buildrequires to detect what plugins are needed
My problem is that the binary is linked to the libpython3.12.so shared library… The detection part is easy, the hard part is how to have the binary work when the shared lib is not installed.
Quick 'n' dirty: Have two binaries, unconditionally call add-determinism-python for *.pyc files, either from add-determinism or the BRP macro (which essentially should be called when %__brp_python_bytecompile is called?), rely on the packager to build-require add-determinism-python or require that from python3-devel (the missing binary should fail the build otherwise).
Something like this could be even made automatic.
- split Python-specific functionality into a separate binary and subpackage
of add-determinism
- add only add-determinism to the default buildroot
- add "Requires: (add-determinism-python if python3)" to add-determinism
That way the pyc processing functionality would only be pulled in iff python3 is already getting installed by something else.
Thanks to everyone who made suggestions.
A variant of the above is implemented in https://src.fedoraproject.org/rpms/rust-add-determinism/pull-request/1:
We' install either add-determinism or add-determinism-nopython. (The two packages provide the same files and declare Conflicts with each other). The appropriate variant will be pulled in via rich dependencies.
(The arrangement is a bit different than what is described above, i.e. one or the other binary is installed, because of some low-level implementation needs:
- I don't want to spawn a separate "handler" for each type of cleanup. Right now, the different handlers don't process files of the same type, so it doesn't matter much, but such cases can easily occur, for example a .pyc file stored in a .jar, or more realistically, a .html javadoc file embedded in a .jar file. If it's all in a single process, it's fairly easy to arrange how this should be sequenced.
- The multi-worker mode is now implemented by spawning N workers, and each worker accepts all file types. This makes the distribution of jobs trivial, we always have one coordinator and N workers. But if we had workers of different types, we'd suddenly need to figure out answers to questions like: how many workers of each type to spawn, e.g. does this particular package have both .jar and .a files, etc. It's much better to avoid this.
- Right now the discovery which handlers are used is made once, in the first binary that is called. If it cannot initialize some type of handler (right now mainly only the pyc handler, if python cannot be initialized or marshalparser cannot be imported, but some other handlers have checks on allowed $SOURCE_DATE_EPOCH values), a warning is printed and the handler is skipped. The workers are then initialized with an explicit handler list, and if any of the specified handlers doesn't initialize, this becomes a hard error. Overall, we get warnings just once, and every unexpected failure is an error. If initialization of handlers was moved down, things would become noisier and less tight.)
This approach solves the problem and is fairly simple and should be robust. If if turns out to be problematic we can always switch to some different solution in the future.
Zbyszek
P.S. https://github.com/keszybz/add-determinism/pull/5 was merged, which cuts down on unneeded features in dependencies.
On 4/13/24 06:41, Fabio Valentini wrote:
On Sat, Apr 13, 2024 at 1:18 PM Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl wrote:
Yes. But actually I think Rust is the optimal choice here. Writing this in Python would be possibly slightly nicer, but we don't want to pull the interpreter and packages into the buildroot. Python also has the problem (challenge?) that it needs to be bootstrapped once per year. The less packages are involved in the bootstrap, the easier it is. And if the brp was written in Python, we'd need to deal with that, and it would probably increase the number of builds which are done without the cleanup. Having this as an indepedent binary avoids some of the issues with bootstrap.
I think Rust*would* be a good choice here ... BUT add-determinism uses pyo3 to link to CPython, so it pulls in python3-libs anyway. So you get the downsides of pulling in Python without the upsides of using Rust ...
Would it be possible to shell out to Python instead of using pyo3? I assume add-determinism's Python handler is not that complicated given that marshalparser does most of the work.
On Wed, Apr 17, 2024 at 05:12:07PM -0500, Maxwell G wrote:
On 4/13/24 06:41, Fabio Valentini wrote:
On Sat, Apr 13, 2024 at 1:18 PM Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl wrote:
Yes. But actually I think Rust is the optimal choice here. Writing this in Python would be possibly slightly nicer, but we don't want to pull the interpreter and packages into the buildroot. Python also has the problem (challenge?) that it needs to be bootstrapped once per year. The less packages are involved in the bootstrap, the easier it is. And if the brp was written in Python, we'd need to deal with that, and it would probably increase the number of builds which are done without the cleanup. Having this as an indepedent binary avoids some of the issues with bootstrap.
I think Rust*would* be a good choice here ... BUT add-determinism uses pyo3 to link to CPython, so it pulls in python3-libs anyway. So you get the downsides of pulling in Python without the upsides of using Rust ...
Would it be possible to shell out to Python instead of using pyo3? I assume add-determinism's Python handler is not that complicated given that marshalparser does most of the work.
Possible — yes. Desirable — not clear.
Firstly, Python interpreter startup with module imports is relatively slow. In the current implementation we do that just once in each worker, and then read the list of files to process from a socket. People were worried about the postprocessing being slow, so I implemented a fairly nice multiprocessing scheme that cuts down on this avoidable overhead.
Secondly, calling a library function is generally preferred over calling a binary for the purposes of error handling. For example, right now I can catch an exception and handle it as appropriate. When calling a subprogram, we only get an error code and maybe some error message, and, for example, supressing or customizing the message for some classes of exceptions cannot be done cleanly.
Thirdly, bootstrapping might actually be easier with a link to a library. We might install older python3-libs together with a new versions of python3-libs and python3, since they don't conflict, but we can have only one /usr/bin/python…
But yeah, just calling an external python binary is certainly an option if the current approach doesn't work out. Thank you for the suggestion.
Zbyszek
Dne 13. 04. 24 v 1:16 odp. Zbigniew Jędrzejewski-Szmek napsal(a):
The proposal explicitly states that we don't want Perl in all buildroots.
How many seconds we save by NOT pulling Perl? Per each build? In total for whole release cycle?
How many seconds we loose (lost) by refactoring the code? And syncing with Debian? Or even worse finding issues that Debian will find and we not?
On Mon, Apr 15, 2024 at 11:10:28AM +0200, Miroslav Suchý wrote:
Dne 13. 04. 24 v 1:16 odp. Zbigniew Jędrzejewski-Szmek napsal(a):
The proposal explicitly states that we don't want Perl in all buildroots.
How many seconds we save by NOT pulling Perl? Per each build? In total for whole release cycle?
How many seconds we loose (lost) by refactoring the code?
I wasn't measuring the time exactly, but about two weeks of my time.
And syncing with Debian? Or even worse finding issues that Debian will find and we not?
Some divergence from Debian would happen anyway. Their build system is different enough that we have some non-overlapping issues. And in my implementation, I was a bit more conservative in what is changed, doing the minimal replacements. We'd either have to maintain some patches or wait until upstream accepts our changes and makes a release. Having a tool that is under our control allows us to update much more quickly.
For Perl, the issue is not so much the seconds and megabytes, but that a) Perl is now effectively a niche language, we made the choice to use Python for tooling instead a long time ago, and b) it'd pull in the interpreter and a set of modules. Considering that we need python marshalparser for the pyc files, that'd mean two interpreter stacks in some cases. Perl certainly has and will have its uses, but I don't think we should use it when adding completely new tooling to our build system.
Zbyszek
Dne 15. 04. 24 v 11:10 Miroslav Suchý napsal(a):
Dne 13. 04. 24 v 1:16 odp. Zbigniew Jędrzejewski-Szmek napsal(a):
The proposal explicitly states that we don't want Perl in all buildroots.
How many seconds we save by NOT pulling Perl? Per each build? In total for whole release cycle?
I know that I have saved quite significant time every build.
Also having buildroot minimal also makes sure that we have correctly specified explicit dependencies, therefore making the builds more reproducible. IOW adding Perl back into the build root would be against the goal of this change proposal.
Vít
How many seconds we loose (lost) by refactoring the code? And syncing with Debian? Or even worse finding issues that Debian will find and we not?
-- Miroslav Suchy, RHCA Red Hat, Manager, Packit and CPT, #brno, #fedora-buildsys
-- _______________________________________________ devel mailing list --devel@lists.fedoraproject.org To unsubscribe send an email todevel-leave@lists.fedoraproject.org Fedora Code of Conduct:https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines:https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives:https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it:https://pagure.io/fedora-infrastructure/new_issue
On Sat, Apr 13, 2024 at 11:16:37AM +0000, Zbigniew Jędrzejewski-Szmek wrote:
If we don't want to pull in an additional language framework, the options are either a compiled language or a scripting language that is already installed anyway, i.e. bash or awk. Considering that we want to do multiprocessing and/or multithreading to make things quick, a compiled language seems better. And among the compiled languages, I think Rust should be the default choice nowadays.
But this does seem to be using python? The readme says it uses marshalparser and the pyc handler imports it:
https://github.com/keszybz/add-determinism/blob/ffdc8364839b42e22e846dce4106...
so you still need python, right?
Brian
On Wed, Apr 17, 2024 at 10:41:22AM -0700, Brian C. Lane wrote:
On Sat, Apr 13, 2024 at 11:16:37AM +0000, Zbigniew Jędrzejewski-Szmek wrote:
If we don't want to pull in an additional language framework, the options are either a compiled language or a scripting language that is already installed anyway, i.e. bash or awk. Considering that we want to do multiprocessing and/or multithreading to make things quick, a compiled language seems better. And among the compiled languages, I think Rust should be the default choice nowadays.
But this does seem to be using python? The readme says it uses marshalparser and the pyc handler imports it:
https://github.com/keszybz/add-determinism/blob/ffdc8364839b42e22e846dce4106...
so you still need python, right?
Yes, kind of. Python and the marshalparser module are used for .pyc files. The marshalparser module is already there, and I didn't see a good reason to reimplement that code. But if there are no .pyc files, that code is not used.
Zbyszek
-
Aoife Moloney -
Brian C. Lane -
Fabio Valentini -
Maxwell G -
Miroslav Suchý -
Neal Gompa -
Pavel Raiskup -
Richard W.M. Jones -
Robert Relyea -
Tim Landscheidt -
Vitaly Zaitsev -
Vít Ondruch -
Zbigniew Jędrzejewski-Szmek