Good Morning Everyone,
You may remember that Nils, Adam and pingou have been investigating what it would take to get rid of maintaining the changelog and release fields manually in our spec files (but still have them in the produced RPMs).
We have already discussed the idea in a few threads: - https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/... announced the original idea - https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/... presented some of the possible approaches to do this
The result of these discussions and our investigations is `rpmautospec`
As its documentation says: `rpmautospec` is a program and library used to automatically generate the release and changelog fields in RPM spec files opting to use it.
`rpmautospec` is currently a proof of concept, we have deployed it in staging and with this email we would like to invite you to test it there.
We have written some documentation on how `rpmautospec` works and how you can opt in at: https://docs.pagure.org/Fedora-Infra.rpmautospec/
To interact with staging you will need: - to use `stg-koji` instead of `koji` - to use `fedpkg-stage` instead of `fedpkg` (``dnf install fedpkg-stage``) - to kinit against `STG.FEDORAPROJECT.ORG` - to test with rawhide as rpmautospec is only available there in staging at this point
To remove some of the warnings thrown by `fedpkg` or to simply keep `rpmbuild` working locally, you will have to install the `rpmautospec-rpm-macros` package available in your nearest bodhi (it's still hot from the oven at the time of writing this email so it hasn't made it yet to updates-testing).
Note, this task was very much time-bound for us and we reached this limit, so this is not something that we will be heavily working on in the coming 3 months. However, if there are sufficient people testing this in staging, providing feedback or contributing to it, we may (depending on that feedback) look at pushing this project forward later in the year.
Finally, while we've tried to be careful, we're sure that we've missed some use cases and that this software isn't bug free, so please bear with us if some of its edges are rough and report your issues/RFEs at: https://pagure.io/Fedora-Infra/rpmautospec/
Thanks in advance for your help and feedback,
Adam, Nils and Pierre
PS: - F32 update: https://bodhi.fedoraproject.org/updates/FEDORA-2020-7f41380eb9 - F31 update: https://bodhi.fedoraproject.org/updates/FEDORA-2020-3ee46bf2cd - F30 update: https://bodhi.fedoraproject.org/updates/FEDORA-2020-081a876918
_______________________________________________ devel-announce mailing list -- devel-announce@lists.fedoraproject.org To unsubscribe send an email to devel-announce-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel-announce@lists.fedorapro...
On 09. 04. 20 15:43, Pierre-Yves Chibon wrote:
We have written some documentation on how `rpmautospec` works and how you can opt in at:https://docs.pagure.org/Fedora-Infra.rpmautospec/
Thanks for working on this \o/
I will definitively try this out soon.
Here is one important concern I have before I dive in.
From the docs:
https://docs.pagure.org/Fedora-Infra.rpmautospec/principle.html
The plugin then installs and calls rpmautospec in the chroot.
https://docs.pagure.org/Fedora-Infra.rpmautospec/install.html
The Koji plugin rpmautospec_builder is meant to be installed on all the Koji builders running the buildSRPMFromSCM task
I suppose this means the "koji-builder-plugin-rpmautospec" is installed on the Koji builder and "rpmautospec" package is installed inside the mock chroot of the buildSRPMFromSCM task, right?
I've looked at your stg Koji builds: https://koji.stg.fedoraproject.org/koji/tasks?owner=pingou&state=all
homebank seem to use rpmautospec: https://src.stg.fedoraproject.org/rpms/homebank/blob/master/f/homebank.spec
The buildSRPMFromSCM task root.log has:
================================================================================ Package Arch Version Repo Size ================================================================================ Installing: rpmautospec noarch 0.1.2-1.fc33 build 8.8 k Installing dependencies: cracklib aarch64 2.9.6-21.fc31 build 83 k fipscheck aarch64 1.5.0-7.fc31 build 26 k fipscheck-lib aarch64 1.5.0-7.fc31 build 14 k gdbm-libs aarch64 1:1.18.1-1.fc32 build 54 k git-core aarch64 2.25.0-1.fc32 build 4.9 M ima-evm-utils aarch64 1.2.1-2.fc31 build 56 k koji noarch 1.20.0-1.fc32 build 149 k less aarch64 551-2.fc31 build 156 k libblkid aarch64 2.35-1.fc32 build 152 k libcomps aarch64 0.1.14-1.fc32 build 77 k libedit aarch64 3.1-31.20191231cvs.fc32 build 104 k libfdisk aarch64 2.35-1.fc32 build 200 k libmount aarch64 2.35-1.fc32 build 177 k libnsl2 aarch64 1.2.0-5.20180605git4a062cf.fc31 build 58 k libpwquality aarch64 1.4.2-1.fc32 build 101 k libsmartcols aarch64 2.35-1.fc32 build 118 k libtirpc aarch64 1.2.5-0.fc32 build 97 k libutempter aarch64 1.1.6-17.fc31 build 26 k libuuid aarch64 2.35-1.fc32 build 28 k openssh aarch64 8.1p1-3.fc32 build 438 k openssh-clients aarch64 8.1p1-3.fc32 build 596 k openssl aarch64 1:1.1.1d-5.fc32 build 645 k pam aarch64 1.3.1-21.fc32 build 662 k python-pip-wheel noarch 19.3.1-1.fc32 build 1.2 M python-setuptools-wheel noarch 41.6.0-1.fc32 build 282 k python3 aarch64 3.8.1-1.fc32 build 31 k python3-cffi aarch64 1.13.2-1.fc32 build 245 k python3-chardet noarch 3.0.4-14.fc32 build 194 k python3-cryptography aarch64 2.8-2.fc32 build 524 k python3-dateutil noarch 1:2.8.0-7.fc32 build 291 k python3-idna noarch 2.8-5.fc32 build 97 k python3-kerberos aarch64 1.3.0-7.fc32 build 32 k python3-koji noarch 1.20.0-1.fc32 build 290 k python3-libcomps aarch64 0.1.14-1.fc32 build 50 k python3-libs aarch64 3.8.1-1.fc32 build 7.8 M python3-ply noarch 3.11-6.fc32 build 104 k python3-pyOpenSSL noarch 19.0.0-5.fc32 build 92 k python3-pycparser noarch 2.19-1.fc32 build 125 k python3-pysocks noarch 1.7.1-3.fc32 build 35 k python3-requests noarch 2.22.0-7.fc32 build 112 k python3-requests-kerberos noarch 0.12.0-8.fc32 build 26 k python3-rpm aarch64 4.15.1-2.fc32 build 98 k python3-rpmautospec noarch 0.1.2-1.fc33 build 42 k python3-setuptools noarch 41.6.0-1.fc32 build 588 k python3-six noarch 1.14.0-1.fc32 build 36 k python3-urllib3 noarch 1.25.7-2.fc32 build 173 k rpm-sign-libs aarch64 4.15.1-2.fc32 build 26 k tss2 aarch64 1331-2.fc31 build 570 k util-linux aarch64 2.35-1.fc32 build 2.7 M
I am not entirely sure if the buildSRPMFromSCM task in a side tag is using that side tag's builds for itself, but if it does, my specific concern is that the entire stack of packages rpmautospec depends on has a runtime dependency on python(abi) = 3.8, so when we'll start bootstrapping Python 3.9 in a side tag, the plugin won't be installable yet, hence every "Python package" (including the Python interpreter) in the dep chain of "rpmautospec" might not be able to use this.
Is my conclusion correct?
On Thu, Apr 09, 2020 at 05:43:36PM +0200, Miro Hrončok wrote:
On 09. 04. 20 15:43, Pierre-Yves Chibon wrote:
We have written some documentation on how `rpmautospec` works and how you can opt in at:https://docs.pagure.org/Fedora-Infra.rpmautospec/
Thanks for working on this \o/
I will definitively try this out soon.
Here is one important concern I have before I dive in.
From the docs:
https://docs.pagure.org/Fedora-Infra.rpmautospec/principle.html
The plugin then installs and calls rpmautospec in the chroot.
https://docs.pagure.org/Fedora-Infra.rpmautospec/install.html
The Koji plugin rpmautospec_builder is meant to be installed on all the Koji builders running the buildSRPMFromSCM task
I suppose this means the "koji-builder-plugin-rpmautospec" is installed on the Koji builder and "rpmautospec" package is installed inside the mock chroot of the buildSRPMFromSCM task, right?
I've looked at your stg Koji builds: https://koji.stg.fedoraproject.org/koji/tasks?owner=pingou&state=all
homebank seem to use rpmautospec: https://src.stg.fedoraproject.org/rpms/homebank/blob/master/f/homebank.spec
The buildSRPMFromSCM task root.log has:
================================================================================ Package Arch Version Repo Size ================================================================================ Installing: rpmautospec noarch 0.1.2-1.fc33 build 8.8 k Installing dependencies:
...
python3 aarch64 3.8.1-1.fc32 build 31 k
...
I am not entirely sure if the buildSRPMFromSCM task in a side tag is using that side tag's builds for itself, but if it does, my specific concern is that the entire stack of packages rpmautospec depends on has a runtime dependency on python(abi) = 3.8, so when we'll start bootstrapping Python 3.9 in a side tag, the plugin won't be installable yet, hence every "Python package" (including the Python interpreter) in the dep chain of "rpmautospec" might not be able to use this.
Is my conclusion correct?
I actually cannot comment on this, it may be worth opening a koji ticket to ask if this is the case or not and if it is if they can think of a way to deal with this.
Random thought: Could both python versions be made available in the side-tag?
On the bright, I take your questions as you'd be interested in using this ;-)
Pierre
On 09. 04. 20 20:45, Pierre-Yves Chibon wrote:
I actually cannot comment on this, it may be worth opening a koji ticket to ask if this is the case or not and if it is if they can think of a way to deal with this.
I plan to test this in staging and follow up on that, but possibly after Easter.
Random thought: Could both python versions be made available in the side-tag?
Actually, with this:
https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproje...
Yes, they could. However, only one version of python3-setuptools and one version of python3-rpm etc. So that would not solve the problem.
On the bright, I take your questions as you'd be interested in using this
Indeed, very much. Getting rid of release/changelog would make cherry-picking Python fixes in the actual interpreters painless.
On 09. 04. 20 23:57, Miro Hrončok wrote:
On 09. 04. 20 20:45, Pierre-Yves Chibon wrote:
I actually cannot comment on this, it may be worth opening a koji ticket to ask if this is the case or not and if it is if they can think of a way to deal with this.
I plan to test this in staging and follow up on that, but possibly after Easter.
I was curious:
https://koji.stg.fedoraproject.org/koji/taskinfo?taskID=90028113
CallbackError: Error running postSCMCheckout callback from _koji_plugin__rpmautospec_builder: Traceback (most recent call last): File "/usr/lib/python3.7/site-packages/koji/plugin.py", line 195, in run_callbacks func(cbtype, *cb_args, **cb_kwargs) File "/usr/lib/koji-builder-plugins/rpmautospec_builder.py", line 91, in process_distgit_cb raise koji.BuildError("Installing `rpmautospec` into the build root failed.") koji.BuildError: Installing `rpmautospec` into the build root failed.
Error: Problem: package python3-rpmautospec-0.1.3-1.fc33.noarch requires python3-koji, but none of the providers can be installed - package rpmautospec-0.1.3-1.fc33.noarch requires python3-rpmautospec = 0.1.3-1.fc33, but none of the providers can be installed - package python3-koji-1.20.0-1.fc32.noarch requires python3-six, but none of the providers can be installed - conflicting requests - nothing provides this_is_broken_on_purpose needed by python3-six-1.12.0-8.fc33.noarch
On 10. 04. 20 1:50, Miro Hrončok wrote:
On 09. 04. 20 23:57, Miro Hrončok wrote:
On 09. 04. 20 20:45, Pierre-Yves Chibon wrote:
I actually cannot comment on this, it may be worth opening a koji ticket to ask if this is the case or not and if it is if they can think of a way to deal with this.
I plan to test this in staging and follow up on that, but possibly after Easter.
I was curious:
https://koji.stg.fedoraproject.org/koji/taskinfo?taskID=90028113
CallbackError: Error running postSCMCheckout callback from _koji_plugin__rpmautospec_builder: Traceback (most recent call last): File "/usr/lib/python3.7/site-packages/koji/plugin.py", line 195, in run_callbacks func(cbtype, *cb_args, **cb_kwargs) File "/usr/lib/koji-builder-plugins/rpmautospec_builder.py", line 91, in process_distgit_cb raise koji.BuildError("Installing `rpmautospec` into the build root failed.") koji.BuildError: Installing `rpmautospec` into the build root failed.
Error: Problem: package python3-rpmautospec-0.1.3-1.fc33.noarch requires python3-koji, but none of the providers can be installed - package rpmautospec-0.1.3-1.fc33.noarch requires python3-rpmautospec = 0.1.3-1.fc33, but none of the providers can be installed - package python3-koji-1.20.0-1.fc32.noarch requires python3-six, but none of the providers can be installed - conflicting requests - nothing provides this_is_broken_on_purpose needed by python3-six-1.12.0-8.fc33.noarch
So there are 2 reasonable things I think that can be done apart from rewriting rpmautospec into Rust ;)
## Koji side tag buildSRPMFromSCM self inavailability solution
Based on my testing, it is clear that builds from the on demand side tags (created with `fedpkg request-side-tag`) use the builds from its own side tag both in the actual buildArch tasks and in the buildSRPMFromSCM task.
OTOH I know we can have side tags that don't use their own builds in neither task, for example the side tags we use for mass rebuilds behave this way.
So the actual question is: Can we have side tags that do both of the following things at the same time?
- use their own builds in the buildArch tasks - don't see/use their own builds in the buildSRPMFromSCM tasks
(CCed Kevin, Mohan and Tomáš for this before I go file tickets.)
Pros:
1. I'd say that rpmautospec can't be the first thing ever that would need this kind fo treatment. We need to be able to bootstrap things that are used in buildSRPMFromSCM tasks, right?
2. No significant changes needed in rpmautospec.
Cons:
1. If Koji doesn't support this use case yet, it might require significant changes there.
2. If packages start using rpmautospec at scale, side tag must be always used to bootstrap anything in the dependency chain. While I'd very much like to see this happening regardless of rpmautospec, the truth is, that people do "simple chain builds" directly in rawhide all the time (e.g. gcc + annobin, python-cryptogtaphy-vectors + python-cryptogtaphy).
In the case of rpmautospec this means that if somebody bumps python3-urllib3 in rawhide first to go to bump python3-requests immediately after that, if the old requests don't install with the new urllib3 AND requests use %autorel, things will blow up, builds will need to be untagged and tagged to a new side tag just to untangle this.
## Move (most of) rpmautospec outside of the mock chroot
Another solution I can think of is that the koji-builder-plugin-rpmautospec package does all the nice things like figure out what to inject into the spec file and prepares everything for needed for rpmautospec *outside of the mock chroot*.
rpmautospec than in fact would only be responsible of actually putting the information into the specfile without using all the 3rd party Python libraries. It would be a simple script (see fedpkg-minimal that exists for this very reason) -- keeping it in written in Python remain an option (as long as it only uses stdlib and doesn't live in site-packages), but making it Bash (like fedpkg-minimal) would be even less fragile.
However, what are the reasons to determine the spec content from within the buildroot? Does it heavily depend on values of macros like %fedora or %dist?
What parts of the procedure are safe to do from a different system and what needs to be done from within?
Pros:
1. Much more robust solution, smaller package footprint in buildSRPMFromSCM means smaller "bungle surface" (like "attack surface" but without the ill will).
Cons:
2. Significant (design) changes in rpmautospec needed. 3. I don't know the exact reasons rpmautospec runs from within the chroot.
Hey Miro!
On Fri, 2020-04-10 at 11:01 +0200, Miro Hrončok wrote:
On 10. 04. 20 1:50, Miro Hrončok wrote:
On 09. 04. 20 23:57, Miro Hrončok wrote:
On 09. 04. 20 20:45, Pierre-Yves Chibon wrote:
I actually cannot comment on this, it may be worth opening a koji ticket to ask if this is the case or not and if it is if they can think of a way to deal with this.
I plan to test this in staging and follow up on that, but possibly after Easter.
I was curious:
https://koji.stg.fedoraproject.org/koji/taskinfo?taskID=90028113
[snipped errors from broken Python 3rd party packages]
So there are 2 reasonable things I think that can be done apart from rewriting rpmautospec into Rust ;)
It's a little late for April Fools. :)
[snipped the "special case buildSRPMFromSCM for side tag builds" approach]
## Move (most of) rpmautospec outside of the mock chroot
Another solution I can think of is that the koji-builder-plugin- rpmautospec package does all the nice things like figure out what to inject into the spec file and prepares everything for needed for rpmautospec *outside of the mock chroot*.
rpmautospec than in fact would only be responsible of actually putting the information into the specfile without using all the 3rd party Python libraries. It would be a simple script (see fedpkg-minimal that exists for this very reason) -- keeping it in written in Python remain an option (as long as it only uses stdlib and doesn't live in site-packages), but making it Bash (like fedpkg-minimal) would be even less fragile.
However, what are the reasons to determine the spec content from within the buildroot? Does it heavily depend on values of macros like %fedora or %dist?
What parts of the procedure are safe to do from a different system and what needs to be done from within?
Pros:
- Much more robust solution, smaller package footprint in
buildSRPMFromSCM means smaller "bungle surface" (like "attack surface" but without the ill will).
Cons:
- Significant (design) changes in rpmautospec needed.
- I don't know the exact reasons rpmautospec runs from within the
chroot.
I think rpmautospec separates concerns between what has to be done in the build root vs. what can be done outside pretty good already. ;)
Let me back up a little:
The reason rpmautospec does anything in the build root at all is because the code that computes the "next release" value has to know for which Fedora version (or more precisely, the %dist tag) the build in question is intended: if at all possible, the EVR about to be built should fit within that of the latest builds in the same or older Fedora versions on the one hand, and newer Fedora versions on the other.
Unfortunately (for rpmautospec), Koji doesn't really care about the Fedora or EPEL version it's building for, all it seems to know is tags and how they're derived from each other. The dist tag is only set in the build root, by /usr/lib/rpm/macros.d/macros.dist from the fedora- release-common package.
Fortunately, rpmautospec already does very little in the build root: The plugin queries Koji outside (this is where the one 3rd party Python package dep comes from) and stores the information gathered (latest build EVRs for each dist tag) as git tags. The code run in the build root computes the next release using the tags (dep on git-core), and with that generates the changelog (which needs info about the current and historic build EVRs).
So, to make this more robust against problems like you described, we should decouple what's run in the build root from a specific minor version of Python and the rpm Python package (and remove the superfluous Koji dependency). This shouldn't be much work, the biggest piece is probably to make it get by without using the rpm Python package (for comparing EVRs and to expand macros), but a tiny ctypes wrapper around rpmlib should solve this for us.
To me, that's good enough. Porting the pieces that have to run inside the build root to e.g. shell really wouldn't make them more robust, but just change one dependency which may not break for another.
Also, even if the worst came to the worst, say a severily broken Python or git-core package has snuck its way into the build roots, and all people with the power to untag these went on vacation, then any packager can easily unblock their build by temporarily reverting to a manually set release and changelog, the plugin will just go out of the way in this case.
Nils
On 14. 04. 20 13:39, Nils Philippsen wrote:
So, to make this more robust against problems like you described, we should decouple what's run in the build root from a specific minor version of Python and the rpm Python package (and remove the superfluous Koji dependency). This shouldn't be much work, the biggest piece is probably to make it get by without using the rpm Python package (for comparing EVRs and to expand macros), but a tiny ctypes wrapper around rpmlib should solve this for us.
Or even running `rpmdev-vercmp` and `rpm --eval` with subprocess:
Inspiration: https://github.com/hroncok/mini-mass-rebuild/blob/9759e77/obsolete_packages....
And: https://github.com/fedora-python/pyp2rpm/blob/33ab6c0c/pyp2rpm/utils.py#L152...
To me, that's good enough. Porting the pieces that have to run inside the build root to e.g. shell really wouldn't make them more robust, but just change one dependency which may not break for another.
Sure. Running on Python (stdlib only) is good enough.
Thanks for the update, I really like where this is going.
On Tue, 2020-04-14 at 13:58 +0200, Miro Hrončok wrote:
On 14. 04. 20 13:39, Nils Philippsen wrote:
So, to make this more robust against problems like you described, we should decouple what's run in the build root from a specific minor version of Python and the rpm Python package (and remove the superfluous Koji dependency). This shouldn't be much work, the biggest piece is probably to make it get by without using the rpm Python package (for comparing EVRs and to expand macros), but a tiny ctypes wrapper around rpmlib should solve this for us.
Or even running `rpmdev-vercmp` and `rpm --eval` with subprocess:
I'd rather not pull in rpmdevtools -- which depends on Perl and a couple 3rd party Perl packages. ;)
One thing I missed in my previous mail was that we already shell out to rpmspec ("rpm --specfile ...") to query the epoch/version of the spec file, but that should be okay, too -- it's part of rpm-build which we need for building the SRPM at that point anyway.
Nils
On Tue, Apr 14, 2020 at 9:06 AM Nils Philippsen nils@redhat.com wrote:
On Tue, 2020-04-14 at 13:58 +0200, Miro Hrončok wrote:
On 14. 04. 20 13:39, Nils Philippsen wrote:
So, to make this more robust against problems like you described, we should decouple what's run in the build root from a specific minor version of Python and the rpm Python package (and remove the superfluous Koji dependency). This shouldn't be much work, the biggest piece is probably to make it get by without using the rpm Python package (for comparing EVRs and to expand macros), but a tiny ctypes wrapper around rpmlib should solve this for us.
Or even running `rpmdev-vercmp` and `rpm --eval` with subprocess:
I'd rather not pull in rpmdevtools -- which depends on Perl and a couple 3rd party Perl packages. ;)
One thing I missed in my previous mail was that we already shell out to rpmspec ("rpm --specfile ...") to query the epoch/version of the spec file, but that should be okay, too -- it's part of rpm-build which we need for building the SRPM at that point anyway.
For what its worth, I want to get rid of the Perl dependency. I just have to decide which Python reimplementation of spectool to put in there... :)
On 14. 04. 20 15:04, Nils Philippsen wrote:
Or even running `rpmdev-vercmp` and `rpm --eval` with subprocess:
I'd rather not pull in rpmdevtools -- which depends on Perl and a couple 3rd party Perl packages.;)
Out of the frying pan and into the fire. Sorry, I forgot that rpmdev-vercmp is not part of rpm.
On Tue, Apr 14, 2020 at 03:14:59PM +0200, Miro Hrončok wrote:
On 14. 04. 20 15:04, Nils Philippsen wrote:
Or even running `rpmdev-vercmp` and `rpm --eval` with subprocess:
I'd rather not pull in rpmdevtools -- which depends on Perl and a couple 3rd party Perl packages.;)
Out of the frying pan and into the fire. Sorry, I forgot that rpmdev-vercmp is not part of rpm.
I don't know what specific features of rpmdev-vercmp tool you need, but version comparison is implemented in rpmvercmp() function of rpm library. I even have a tiny Perl wrapper, perl-RPM-VersionCompare, for that :)
-- Petr
On 14. 04. 20 15:30, Petr Pisar wrote:
I don't know what specific features of rpmdev-vercmp tool you need, but version comparison is implemented in rpmvercmp() function of rpm library.
Exactly that, but without the need to use ctypes.
On 4/14/20 5:17 PM, Miro Hrončok wrote:
On 14. 04. 20 15:30, Petr Pisar wrote:
I don't know what specific features of rpmdev-vercmp tool you need, but version comparison is implemented in rpmvercmp() function of rpm library.
Exactly that, but without the need to use ctypes.
Note thought that rpmvercmp() can only compare the individual E, V and R segments one at the time. To compare an entire EVR string as a whole against another, you'll need something like rpm-python rpm.labelCompare():
https://github.com/rpm-software-management/rpm/blob/master/python/header-py....
- Panu -
Hi,
Another solution would be to just save the state from which autobump is computed in the srpm, and bump from this state at the rpmbuild level. As long as the previous counter value is available computing a new one is just a few lines of lua macro code
That would remove any adherence to koji, mock or Fedora git.
Regards,
Note: I'm subscribed to the list, there's no need to CC me. :)
On Mon, Apr 20, 2020 at 11:45:17AM +0200, Nicolas Mailhot wrote:
Hi,
Another solution would be to just save the state from which autobump is computed in the srpm, and bump from this state at the rpmbuild level.
How would that look? an untracked by git file with called 'releasever' and having the releasever number in it?
As long as the previous counter value is available computing a new one is just a few lines of lua macro code
That would remove any adherence to koji, mock or Fedora git.
Can you provide an example?
kevin
On 09. 04. 20 15:43, Pierre-Yves Chibon wrote:
To remove some of the warnings thrown by `fedpkg` or to simply keep `rpmbuild` working locally, you will have to install the `rpmautospec-rpm-macros` package available in your nearest bodhi (it's still hot from the oven at the time of writing this email so it hasn't made it yet to updates-testing).
There is also this warning:
$ fedpkg-stage build Could not execute build: Package 3dprinter-udev-rules-0.2.2-1.fc33 has already been built Note: You can skip this check with --skip-nvr-check. See help for more info.
Because locally, it always thinks I'm at release 1.fc33, but this has been already built. With --skip-nvr-check, it works, obviously.
I suppose this could be fixed (workarounded?) in fedpkg (it could default to --skip-nvr-check if it finds the %autorel macro in the spec).
On Fri, Apr 10, 2020 at 01:58:20AM +0200, Miro Hrončok wrote:
On 09. 04. 20 15:43, Pierre-Yves Chibon wrote:
To remove some of the warnings thrown by `fedpkg` or to simply keep `rpmbuild` working locally, you will have to install the `rpmautospec-rpm-macros` package available in your nearest bodhi (it's still hot from the oven at the time of writing this email so it hasn't made it yet to updates-testing).
There is also this warning:
$ fedpkg-stage build Could not execute build: Package 3dprinter-udev-rules-0.2.2-1.fc33 has already been built Note: You can skip this check with --skip-nvr-check. See help for more info.
Because locally, it always thinks I'm at release 1.fc33, but this has been already built. With --skip-nvr-check, it works, obviously.
I suppose this could be fixed (workarounded?) in fedpkg (it could default to --skip-nvr-check if it finds the %autorel macro in the spec).
Yes we're aware of this one and I thought we documented it but apparently we only covered the part about the missing macros.
I'll add this to the doc, thanks!
Pierre
On 10. 04. 20 9:03, Pierre-Yves Chibon wrote:
On Fri, Apr 10, 2020 at 01:58:20AM +0200, Miro Hrončok wrote:
On 09. 04. 20 15:43, Pierre-Yves Chibon wrote:
To remove some of the warnings thrown by `fedpkg` or to simply keep `rpmbuild` working locally, you will have to install the `rpmautospec-rpm-macros` package available in your nearest bodhi (it's still hot from the oven at the time of writing this email so it hasn't made it yet to updates-testing).
There is also this warning:
$ fedpkg-stage build Could not execute build: Package 3dprinter-udev-rules-0.2.2-1.fc33 has already been built Note: You can skip this check with --skip-nvr-check. See help for more info.
Because locally, it always thinks I'm at release 1.fc33, but this has been already built. With --skip-nvr-check, it works, obviously.
I suppose this could be fixed (workarounded?) in fedpkg (it could default to --skip-nvr-check if it finds the %autorel macro in the spec).
Yes we're aware of this one and I thought we documented it but apparently we only covered the part about the missing macros.
I'll add this to the doc, thanks!
I have found a slight issue with this approach.
1. Packager A clones package P (has %autorel) 2. Packager B pushes+builds some changes in package P 3. Packager A runs `fedpkg build --skip-nvr-check` without pulling first
At 3, the old version of the package gets rebuilt and gets a higher EVR.
See the build/3dprinter-udev-rules-0-0.2.2-7.fc33 and ...-6.fc33 tags in:
https://src.stg.fedoraproject.org/rpms/3dprinter-udev-rules/commits/master
On Wed, May 13, 2020 at 11:45:26PM +0200, Miro Hrončok wrote:
On 10. 04. 20 9:03, Pierre-Yves Chibon wrote:
On Fri, Apr 10, 2020 at 01:58:20AM +0200, Miro Hrončok wrote:
On 09. 04. 20 15:43, Pierre-Yves Chibon wrote:
To remove some of the warnings thrown by `fedpkg` or to simply keep `rpmbuild` working locally, you will have to install the `rpmautospec-rpm-macros` package available in your nearest bodhi (it's still hot from the oven at the time of writing this email so it hasn't made it yet to updates-testing).
There is also this warning:
$ fedpkg-stage build Could not execute build: Package 3dprinter-udev-rules-0.2.2-1.fc33 has already been built Note: You can skip this check with --skip-nvr-check. See help for more info.
Because locally, it always thinks I'm at release 1.fc33, but this has been already built. With --skip-nvr-check, it works, obviously.
I suppose this could be fixed (workarounded?) in fedpkg (it could default to --skip-nvr-check if it finds the %autorel macro in the spec).
Yes we're aware of this one and I thought we documented it but apparently we only covered the part about the missing macros.
I'll add this to the doc, thanks!
I have found a slight issue with this approach.
- Packager A clones package P (has %autorel)
- Packager B pushes+builds some changes in package P
- Packager A runs `fedpkg build --skip-nvr-check` without pulling first
At 3, the old version of the package gets rebuilt and gets a higher EVR.
See the build/3dprinter-udev-rules-0-0.2.2-7.fc33 and ...-6.fc33 tags in:
https://src.stg.fedoraproject.org/rpms/3dprinter-udev-rules/commits/master
It took me a little time to find it back, but I knew we had documented this as this is something we are aware off: https://docs.pagure.org/fedora-infra.rpmautospec/peculiarities.html#multiple...
I wonder if we could teach fedpkg to check if there are new commits in a branch before it runs a fedpkg build and issue an error message if that is the case (with the --I-know-what-I-am-doing argument of course).
Pierre
On 14. 05. 20 9:13, Pierre-Yves Chibon wrote:
I have found a slight issue with this approach.
- Packager A clones package P (has %autorel)
- Packager B pushes+builds some changes in package P
- Packager A runs `fedpkg build --skip-nvr-check` without pulling first
At 3, the old version of the package gets rebuilt and gets a higher EVR.
See the build/3dprinter-udev-rules-0-0.2.2-7.fc33 and ...-6.fc33 tags in:
https://src.stg.fedoraproject.org/rpms/3dprinter-udev-rules/commits/master
It took me a little time to find it back, but I knew we had documented this as this is something we are aware off: https://docs.pagure.org/fedora-infra.rpmautospec/peculiarities.html#multiple...
Good. Sorry I've missed it.
I wonder if we could teach fedpkg to check if there are new commits in a branch before it runs a fedpkg build and issue an error message if that is the case (with the --I-know-what-I-am-doing argument of course).
It already stops if git knows your are not at origin/<branch>, but you need to fetch in order for this to work. So fedpkg would need to either:
- always fetch (which is slow) - always ask the API (which might be slow, but it already asks koji)
On Thu, May 14, 2020 at 11:00:05AM +0200, Miro Hrončok wrote:
On 14. 05. 20 9:13, Pierre-Yves Chibon wrote:
I have found a slight issue with this approach.
- Packager A clones package P (has %autorel)
- Packager B pushes+builds some changes in package P
- Packager A runs `fedpkg build --skip-nvr-check` without pulling first
At 3, the old version of the package gets rebuilt and gets a higher EVR.
See the build/3dprinter-udev-rules-0-0.2.2-7.fc33 and ...-6.fc33 tags in:
https://src.stg.fedoraproject.org/rpms/3dprinter-udev-rules/commits/master
It took me a little time to find it back, but I knew we had documented this as this is something we are aware off: https://docs.pagure.org/fedora-infra.rpmautospec/peculiarities.html#multiple...
Good. Sorry I've missed it.
I wonder if we could teach fedpkg to check if there are new commits in a branch before it runs a fedpkg build and issue an error message if that is the case (with the --I-know-what-I-am-doing argument of course).
It already stops if git knows your are not at origin/<branch>, but you need to fetch in order for this to work. So fedpkg would need to either:
- always fetch (which is slow)
- always ask the API (which might be slow, but it already asks koji)
I was thinking maybe query: https://src.fedoraproject.org/api/0/rpms/copr-backend/git/branches?with_comm... I would be one more http query for sure.
Pierre
Dne 09. 04. 20 v 15:43 Pierre-Yves Chibon napsal(a):
PS:
Could be the source package renamed s/python-rpmautospec/rpmautospec/. The `python-` prefix does not help with the discoverability at all.
Vít
On Tue, Apr 14, 2020 at 12:46:11PM +0200, Vít Ondruch wrote:
Dne 09. 04. 20 v 15:43 Pierre-Yves Chibon napsal(a):
PS:
Could be the source package renamed s/python-rpmautospec/rpmautospec/. The `python-` prefix does not help with the discoverability at all.
Considering that rpmautospec is mostly a python module with two koji plugins and a small CLI, it seemed appropriate to us to follow the python naming guidelines, and this is how we understand them.
Pierre
On Tuesday, 14 April 2020 14.47.15 WEST Pierre-Yves Chibon wrote:
Considering that rpmautospec is mostly a python module with two koji plugins and a small CLI, it seemed appropriate to us to follow the python naming guidelines, and this is how we understand them.
Pierre
Sure. :-)
But since it has a program associated it is probably a good idea to have
Provides: rpmautospec
since if you are a user you do not care about the python dependency.
On 14. 04. 20 16:20, José Abílio Matos wrote:
On Tuesday, 14 April 2020 14.47.15 WEST Pierre-Yves Chibon wrote:
Considering that rpmautospec is mostly a python module with two koji plugins
and a small CLI, it seemed appropriate to us to follow the python naming
guidelines, and this is how we understand them.
Pierre
Sure. :-)
But since it has a program associated it is probably a good idea to have
Provides: rpmautospec
since if you are a user you do not care about the python dependency.
There is an "rpmautospec" subpackage already. The discussion is about SRPM name.
On 14. 04. 20 15:47, Pierre-Yves Chibon wrote:
On Tue, Apr 14, 2020 at 12:46:11PM +0200, Vít Ondruch wrote:
Dne 09. 04. 20 v 15:43 Pierre-Yves Chibon napsal(a):
PS:
Could be the source package renamed s/python-rpmautospec/rpmautospec/. The `python-` prefix does not help with the discoverability at all.
Considering that rpmautospec is mostly a python module with two koji plugins and a small CLI, it seemed appropriate to us to follow the python naming guidelines, and this is how we understand them.
Being the current de-facto maintainer of the Python Packaging guidelines, let me followup on that.
The package produces several installable packages:
- koji-builder-plugin-rpmautospec - koji-hub-plugin-rpmautospec - python3-rpmautospec - rpmautospec - rpmautospec-rpm-macros
And all of them seem to follow the naming guidelines and conventions.
As for the component/SRPM name, the guidelines are pretty loose here. Both "rpmautospec" and "python-rpmautospec" SRPM names are valid depending on how you perceive the package.
The Python guidelines explicitly say:
"This rule [using the python- prefix for SRPM] does not apply to applications."
And obviously, the source package does contain both application and other things, hence it is really up to the maintainer.
Dne 14. 04. 20 v 16:30 Miro Hrončok napsal(a):
On 14. 04. 20 15:47, Pierre-Yves Chibon wrote:
On Tue, Apr 14, 2020 at 12:46:11PM +0200, Vít Ondruch wrote:
Dne 09. 04. 20 v 15:43 Pierre-Yves Chibon napsal(a):
PS: - F32 update: https://bodhi.fedoraproject.org/updates/FEDORA-2020-7f41380eb9 - F31 update: https://bodhi.fedoraproject.org/updates/FEDORA-2020-3ee46bf2cd - F30 update: https://bodhi.fedoraproject.org/updates/FEDORA-2020-081a876918
Could be the source package renamed s/python-rpmautospec/rpmautospec/. The `python-` prefix does not help with the discoverability at all.
Considering that rpmautospec is mostly a python module with two koji plugins and a small CLI
In your initial email, you have mentioned rpmautospec and rpmautospec-rpm-macros. I was interested in how things are done and non of the packages available in Fedora correspond to the SRPM. This makes the project undiscoverable (not mentioning the proposed rewrite in Rust :wink:)
Vít
, it seemed appropriate to us to follow the python naming guidelines, and this is how we understand them.
Being the current de-facto maintainer of the Python Packaging guidelines, let me followup on that.
The package produces several installable packages:
- koji-builder-plugin-rpmautospec
- koji-hub-plugin-rpmautospec
- python3-rpmautospec
- rpmautospec
- rpmautospec-rpm-macros
And all of them seem to follow the naming guidelines and conventions.
As for the component/SRPM name, the guidelines are pretty loose here. Both "rpmautospec" and "python-rpmautospec" SRPM names are valid depending on how you perceive the package.
The Python guidelines explicitly say:
"This rule [using the python- prefix for SRPM] does not apply to applications."
And obviously, the source package does contain both application and other things, hence it is really up to the maintainer.
Will this work with modules? There is the context already increased for every build ...
Vít
Dne 09. 04. 20 v 15:43 Pierre-Yves Chibon napsal(a):
Good Morning Everyone,
You may remember that Nils, Adam and pingou have been investigating what it would take to get rid of maintaining the changelog and release fields manually in our spec files (but still have them in the produced RPMs).
We have already discussed the idea in a few threads:
- https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/... announced the original idea
- https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/... presented some of the possible approaches to do this
The result of these discussions and our investigations is `rpmautospec`
As its documentation says: `rpmautospec` is a program and library used to automatically generate the release and changelog fields in RPM spec files opting to use it.
`rpmautospec` is currently a proof of concept, we have deployed it in staging and with this email we would like to invite you to test it there.
We have written some documentation on how `rpmautospec` works and how you can opt in at: https://docs.pagure.org/Fedora-Infra.rpmautospec/
To interact with staging you will need:
- to use `stg-koji` instead of `koji`
- to use `fedpkg-stage` instead of `fedpkg` (``dnf install fedpkg-stage``)
- to kinit against `STG.FEDORAPROJECT.ORG`
- to test with rawhide as rpmautospec is only available there in staging at this point
To remove some of the warnings thrown by `fedpkg` or to simply keep `rpmbuild` working locally, you will have to install the `rpmautospec-rpm-macros` package available in your nearest bodhi (it's still hot from the oven at the time of writing this email so it hasn't made it yet to updates-testing).
Note, this task was very much time-bound for us and we reached this limit, so this is not something that we will be heavily working on in the coming 3 months. However, if there are sufficient people testing this in staging, providing feedback or contributing to it, we may (depending on that feedback) look at pushing this project forward later in the year.
Finally, while we've tried to be careful, we're sure that we've missed some use cases and that this software isn't bug free, so please bear with us if some of its edges are rough and report your issues/RFEs at: https://pagure.io/Fedora-Infra/rpmautospec/
Thanks in advance for your help and feedback,
Adam, Nils and Pierre
PS:
- F32 update: https://bodhi.fedoraproject.org/updates/FEDORA-2020-7f41380eb9
- F31 update: https://bodhi.fedoraproject.org/updates/FEDORA-2020-3ee46bf2cd
- F30 update: https://bodhi.fedoraproject.org/updates/FEDORA-2020-081a876918
devel-announce mailing list -- devel-announce@lists.fedoraproject.org To unsubscribe send an email to devel-announce-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel-announce@lists.fedorapro...
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
-
José Abílio Matos -
Kevin Fenzi -
Miro Hrončok -
Neal Gompa -
Nicolas Mailhot -
Nils Philippsen -
Panu Matilainen -
Petr Pisar -
Pierre-Yves Chibon -
Vít Ondruch