[Fedora-packaging] Re: RPM 4.19: Dynamic subpackages with Python extras

On Thu, Mar 30, 2023 at 11:56 PM Miro Hrončok <mhroncok(a)redhat.com <mailto:mhroncok@redhat.com>> wrote: Hello Python packagers. RPM 4.19 introduces this feature: https://rpm-software-management.github.io/rpm/manual/dynamic_specs.html <https://rpm-software-management.github.io/rpm/manual/dynamic_specs.html> I decided to write this email to gather my thoughts. I believe that with this, we can turn manual Python extras subpackages like this:    %package -n python3-...    Summary:        %{summary}    %description -n python3-... %_description    %pyproject_extras_subpkg -n python3-xxx extra1 extra2 (See https://docs.fedoraproject.org/en-US/packaging-guidelines/Python/#Extras <https://docs.fedoraproject.org/en-US/packaging-guidelines/Python/#Extras> for what that means.) Into something like this:    %package -n python3-...    Summary:        %{summary}    %description -n python3-... %_description    ...    %install    %pyproject_install    ...    %pyproject_generate_extras_subpkgs -n python3-xxx The %pyproject_generate_extras_subpkgs macro would parse the installed .dist-info directory to find out what extras are available and generate subpackages for all of them. (Obviously, the macro name is open up for discussion.) Overall this changes the concept of manually opting-in to extras to manually opting-out. I have a slight preference over the current status quo, as the packager needs to be aware of the package's extras and modify the SPEC accordingly, whereas in the event of implementing an opt-out approach, things could easily go in the way of enabling everything till something breaks. At the moment, when working on a package, I am choosing which extras to enable, leading me to also search about requirements, purpose of each extra etc. Being able to enable everything and exclude what I don't need, would lead to the path of least resistance which is basically to enable everything and not think about potential issues (as there wouldn't be any at this point). Now I can imagine "fail to install" bugzilla's over various extras with the extra caveat that in order to fix a package we would require to carry the Obsoletes tag in the SPEC for two releases. Coupled with the fact that an auto-generated list of extras makes difficult to figure out if an extra has been removed in order to fix it on the package level makes this approach less than ideal in my head.

Of course not all packages have extras and one can reasonably argue that packagers who are utilizing the %pyproject_extras_subpkg macro would be well aware of the implications if they change to the new macro. While this change would definitely make things easier for packagers who'd like a cleaner SPEC (and continuing on the path that pyproject macros have set to align the upstream metadata with the rpm generated metadata) I believe it can be more error prone than the current approach and the potential risks outweigh the benefits. I'm happy to be proven wrong of course.

An API would be required to exclude extras:   - that are not useful for other packages    (for example build/development requirements, commonly named dev, doc or test)   - that have requirements that are not packaged in Fedora For example (mimicking the API of %pyproject_check_import):    %pyproject_generate_extras_subpkgs -n python3-xxx -e test -e 'nonfree*' ---------------- However, extras are also currently manually passed to %pyproject_buildrequires:    %generate_buildrequires    %pyproject_buildrequires -x extra1 -x extra2 -x test It should already be possible to implement automatic extras discovery in %pyproject_buildrequires with older RPM versions and allow it to be used this way:    %generate_buildrequires    %pyproject_buildrequires <FLAG_TO_ENABLE_ALL_EXTRAS> -X 'nonfree*' RPM macros can only accept short flags, so <FLAG_TO_ENABLE_ALL_EXTRAS> can either be -x '*' (if we start treating -x values as globs, which is backwards compatible and probably generally useful), or a single-letter switch such as -a (but honestly we are running out of meaningful letters). (When -X is used, <FLAG_TO_ENABLE_ALL_EXTRAS> can probably be implied. However, an explicit form needs to exist for packages that don't need to exclude any extras at all.) Eventually, I'd like to make <FLAG_TO_ENABLE_ALL_EXTRAS> the default, once RPM 4.19 is omnipresent. ---------------- Combined, this would mean that the packager needs to:   1. specify extras that are not supposed to be used as BRs   2. specify extras that are not supposed to be packaged In the ideal word (2) is a superset of (1). Should %pyproject_generate_extras_subpkgs somehow inherit the -Xes from %pyproject_buildrequires? When a package has extra1, extra2, nonfree1, nonfree2 and test extras, one could do:    %generate_buildrequires    %pyproject_buildrequires <FLAG_TO_ENABLE_ALL_EXTRAS> -X 'nonfree*'    ...    %pyproject_install    ...    %pyproject_generate_extras_subpkgs -X test That would mean:   - extra1 is BRed and packaged   - extra2 is BRed and packaged   - test is BRed but not packaged   - nonfree1 is neither   - nonfree2 is neither Your thoughts on the implementation are quite sound and consistent, I don't really have any comments on that as you have taken all the possibilities into account and provided examples. If those thoughts move forward with an implementation I'll provide more comprehensive feedback on that. Also %pyproject_generate_extras_subpkgs somehow inheriting the excluded extras from %pyproject_buildrequires is reasonable, I dislike the archaic %global approach as well. ---------------- Alternatively the information could be supplied by %globals:    %global _python_ignored_extras nonfree*    %global _python_unpackaged_extras test However, I somehow dislike this approach. ---------------- I'd appreciate your thoughts on the matter. -- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok _______________________________________________ packaging mailing list -- packaging(a)lists.fedoraproject.org <mailto:packaging@lists.fedoraproject.org> To unsubscribe send an email to packaging-leave(a)lists.fedoraproject.org <mailto:packaging-leave@lists.fedoraproject.org> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ <https://docs.fedoraproject.org/en-US/project/code-of-conduct/> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines <https://fedoraproject.org/wiki/Mailing_list_guidelines> List Archives: https://lists.fedoraproject.org/archives/list/packaging@lists.fedoraproje... <https://lists.fedoraproject.org/archives/list/packaging@lists.fedoraproje... Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue <https://pagure.io/fedora-infrastructure/new_issue> -- Regards, Charalampos Stratakis Senior Software Engineer Python Maintenance Team, Red Hat _______________________________________________ packaging mailing list -- packaging(a)lists.fedoraproject.org To unsubscribe send an email to packaging-leave(a)lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/packaging@lists.fedoraproje... Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue