Hello all, I recently upstreamed some fixes to a new RPM dependency generator that will be available as an option for distributions to enable. The new generator uses python .egg data to generate Provides and Requires in the form of pythonXegg(Y), where X is the Python major version and Y is the module.

On Tue, Nov 17, 2015 at 3:26 AM, Nick Coghlan <ncoghlan(a)gmail.com&gt; wrote: > I'm not clear on what you mean by depending on an egg. Eggs are a > binary format that isn't compatible with Linux distro packaging > policies, since they lose too much structural information regarding > where files should be installed for policy compliance - that's one of > the major reasons we defined the wheel format as a next generation > replacement: https://www.python.org/dev/peps/pep-0427/. I don't think I've ever heard of this, much less seen it in use. And it looks like the format was approved well over two years ago, so I'm *really* surprised nothing has happened since then.

Variants of this generator are in use in both Mandriva and Mageia,

and I wanted to give the Python SIG in Fedora the opportunity to try it out and see if we might want to enable it in Fedora. Essentially, for things that have egg info that includes the requirements and the name of the module, it makes it simpler to figure out what how to install Python packages in a more cross-distro manner.

> On Nov 17, 2015, at 7:54 AM, Nick Coghlan <ncoghlan(a)gmail.com&gt; wrote: > > On 17 November 2015 at 22:05, Neal Gompa <ngompa13(a)gmail.com&gt; wrote: >> and >> I wanted to give the Python SIG in Fedora the opportunity to try it >> out and see if we might want to enable it in Fedora. >> >> Essentially, for things that have egg info that includes the >> requirements and the name of the module, it makes it simpler to figure >> out what how to install Python packages in a more cross-distro manner. > > Having a plugin to automatically generate appropriate upstream > Provides/Requires to allow installation based on PyPI distribution > names definitely has the potential to be useful. It was only the > choice of "python2egg" and "python3egg" as the name that threw me, > since that doesn't align with the upstream community terminology (the > egg format is specific to setuptools and easy_install) This sounds like it’s actually use the “other” Egg (because for maximum confusion setuptools eggs are actually like 3 different things). Right now both distutils and setuptools will generate an egg-info metadata when installing projects (though distutils generates a file and setuptools a directory). However wheels install a dist-info directory. I don’t have a better suggestion for name other than I’d prefer not to bake more things onto the “egg” terminology since we (Python packaging upstream) are trying to phase it out.

> On Nov 17, 2015, at 8:25 AM, Neal Gompa <ngompa13(a)gmail.com&gt; wrote: > Is the format inside of the .dist-info directory the same as the older > .egg-info and .egg-link directories? If so, it should be easy to add > to read that information too. Currently yes. In the future we’ll probably evolve it and then they’ll drift apart. Part of doing that though will be defining a way to determine what “version” of standard the directory is currently using (with the current version being a not very well defined version 0 or something).

On 17 November 2015 at 23:25, Neal Gompa <ngompa13(a)gmail.com&gt; wrote: > As for naming, I'm all ears for a better name, because if the "egg" > name is going away, I'd rather it not continue to say that. My suggestions would be either: python2dist(name)/python3dist(name) or: python2(name)/python3(name) The "dist" suffix comes from: * "distutils", the standard library's software distribution utilities * the "sdist" name used for uploading source archives to PyPI * the "-Dist" suffix used in the never-really-adopted metadata 1.2 spec [1] While dropping the suffix entirely seems like a potentially attractive option to me, it may also be ambiguous as to whether it's referring to import package names (eg. "python2(pkg_resources)") or distribution package names (e.g. "python2(setuptools)"). Cheers, Nick. [1] https://www.python.org/dev/peps/pep-0345/

> On Nov 17, 2015, at 9:26 AM, Neal Gompa <ngompa13(a)gmail.com&gt; wrote: > > The dependency generator uses pkg_resources (specifically > Distribution, FileMetadata, PathMetadata) to read data in the > .egg-info directory. That should still work with .dist-info, right? If > it does, then I can easily support it to read and parse the data. If > there's something else I have to do, please let me know, so I can add > it. > That’s correct. There’s a EggInfoMetadata class and a DistInfoMetadata class in pkg_resources.

As for naming, I'm all ears for a better name, because if the "egg" name is going away, I'd rather it not continue to say that.

On Nov 17, 2015 6:06 AM, "Nick Coghlan" <ncoghlan(a)gmail.com&gt; wrote: > > On 17 November 2015 at 23:25, Neal Gompa <ngompa13(a)gmail.com&gt; wrote: > > As for naming, I'm all ears for a better name, because if the "egg" > > name is going away, I'd rather it not continue to say that. > > My suggestions would be either: > > python2dist(name)/python3dist(name) > I'd favor this form due to the import names conflict that Nick mentions (not that I think anyone will code an auto generator that uses import names once this is accepted.. this seems good enough to carry the day.) One thing I would change, though: instead of python2dist(name) use python2.7dist(name). A module built for python 2.7 won't be able to import a module built for python2.6 (the python interpreter won't look in the directories in which the files are placed) so without both the major.minor numbers we won't be able to rely on the auto generated requires in the spec file. -Toshio

On Nov 17, 2015 6:47 AM, "Neal Gompa" <ngompa13(a)gmail.com&gt; wrote: > > That's already guaranteed by the auto-generated python(abi) requires, > and that would also make it hugely problematic to use in spec files in > any distro agnostic manner, so I don't think I will do that. > I could just be being dumb here but it doesn't seem like it would. The python(abi) will require a specific major.minor of the python package itself. It will not require a python-setuptools (for instance) that was built for the same version of python as the package you are installing. If it was true, then you wouldn't need the separate python2dist and python3dist as python(abi) would take care of the difference between packages built for python2 and python3. example autogenerated deps: python-setuptools Provides: python2dist(setuptools) Requires: python(abi) = 2.7 python2.6-setuptools Provides: python2dist(setuptools) Requires: python(abi) = 2.6 python2.6-foo Requires: python2dist(setuptools) Requires: python(abi) = 2.6 On my system, I have installed, python-2.7.0 (which provides python(abi)==2.7), python2.6-2.6.0 (which provides python(abi)==2.6), and python-setuptools (deps listed above). I want to install python2.6foo. The depsolver will satisfy python2dist(setuptools) with my already installed python-setuptools package and python(abi) == 2.6 with python2.6-2.6.0. Thus, the library will not function correctly because python-2.6 will not be able to import setuptools.

>>>> "NC" == Nick Coghlan <ncoghlan(a)gmail.com&gt; writes:

On 18 November 2015 at 02:29, Jason L Tibbitts III <tibbs(a)math.uh.edu&gt; wrote: >>>>>> "NC" == Nick Coghlan <ncoghlan(a)gmail.com&gt; writes: > > NC> If so, then there's some relevant work currently under way upstream > NC> to improve the interaction between Python installation tools and > NC> build systems to improve the metadata extraction process, rather > NC> than relying on implementation details of setuptools. > > If that's the case, could someone bang out a few paragraphs that we > could use as a blueprint for some packaging guidelines? An example > spec, or even just the file layout and some idea of how autogenerated > dependencies would work would be enough. I know this stuff is a bit > new, but we've been doing a really big overhaul of the python stuff and > we'd like to at least design to accommodate this rather than having to > do it all over again once this new format comes out. The main policy changes would be to update these two section to mention keeping dist-info directories: * https://fedoraproject.org/wiki/Packaging:Python#Files_to_include * https://fedoraproject.org/wiki/Packaging:Python#Reviewer_checklist It would also be desirable to state a preference for dist-info over egg-info. Thinking about it a bit further, I don't think the latest round of upstream changes should impact the metadata analysis step, as the metadata querying changes are designed to support getting at the metadata without building and installing the package first, and that's not a consideration for the RPM use case. However, they could potentially affect the py2/3_build macros, as we're looking to finally migrate away from *requiring* the presence of a setup.py file in every source tree, and instead allow out-of-tree build tools, with machine readable instructions for bootstrapping them into the build environment. The draft PEP for that is at https://github.com/pypa/interoperability-peps/pull/54/files and upstream discussions are on distutils-sig I'd been thinking using "pip install" instead of "setup.py install" in the build macros would be sufficient, but I now realise that isn't the case - if a project uses flit (for example) as its build utility, then we're going to need to generate a suitable BuildRequires in pyp2rpm and similar tools (perhaps using the "BuidlRequires: pythonX.Ydist(flit)" format). The build macros themselves could still delegate the task of working out the right build command to invoke to pip, though.

On Wed, Nov 18, 2015 at 5:27 AM, Neal Gompa <ngompa13(a)gmail.com&gt; wrote: > On Wed, Nov 18, 2015 at 2:48 AM, Nick Coghlan <ncoghlan(a)gmail.com&gt; wrote: >> I'd been thinking using "pip install" instead of "setup.py install" in >> the build macros would be sufficient, but I now realise that isn't the >> case - if a project uses flit (for example) as its build utility, then >> we're going to need to generate a suitable BuildRequires in pyp2rpm >> and similar tools (perhaps using the "BuidlRequires: >> pythonX.Ydist(flit)" format). The build macros themselves could still >> delegate the task of working out the right build command to invoke to >> pip, though. >> > > The main issue I see with that is how to make it so that python upgrades > aren't obnoxiously painful. If BuildRequires use pythonXdist(module) format, > but all *generated* runtime requirements use pythonX.Ydist(module) format, > this problem goes away. But as Toshio mentioned, how do we solve that in a > multi-version environment (like Enterprise Linux, for instance)? > Really, for this I think continuing to use package names is the right thing to do. Package names uniquely identify the package built for the default python version for that distribution+release which is what we want to make rebuilding a package on a newer release with a newer version of python as the default. Attempting to build this into the generated dependencies duplicates the features that relying on the name gives us. > Using pythonX.Ydist(module) for BuildRequires effectively locks the module > to a specific Python version until each and every maintainer upgrades them. > That is an awful thing to have to do, and no other programming environment > in any RPM-based distribution requires that. Most of the time, this is an > unnecessary burden on the package maintainers. > Note: we do want this when we're specifically building a package for a non-default version of python. If we want to have python2.7 in EPEL6 or python2.6 in Fedora24 or (python3.5 and python3.4 for that matter), then the package stack built for that non-default python needs to specify that the packages it requires also need to be those built for that stack. So all of those packages can BuildRequire the X.Y autogenerated deps. It's for the default stack that we want to have some notion of "default" within the dependencies

On 22 November 2015 at 04:18, Neal Gompa <ngompa13(a)gmail.com&gt; wrote: > Based on the feedback from you guys, I've made the changes to move to > pythonX.Ydist() in the dependency generator. That code has been > submitted as a pull request to the RPM GitHub repository[0]. I also > added a switch for those who want pythonXdist() Provides, but it is > opt-in rather than opt-out. The option is only for distributions that > intend to carry only one Python runtime per major version. Very cool, thank you! Cheers, Nick.