Re: [Fedora-packaging] How to automatically handle Python namespace packages (e.g. in %pyproject_save_files)

On Wed, Apr 14, 2021 at 5:18 AM Miro Hrončok <mhroncok(a)redhat.com&gt; wrote: > > Hello Pythonistas. > > I'd like to be able to automatically handle Python "namespace" packages from our > packaging macros. > > The problem: > > Several Python packages share a "namespace", let's take an artificial example > with food.spam and food.eggs Python packages. > > 1. the Python packages both have site-packages/food > 2. sometimes such packages also both have site-packages/food/__init__.py > (usually empty or mostly empty, but with different mtimes etc.) > > On RPM level, this means: > > 1. %{python3_sitelib}/food can be co-owned > OR it can be in an artificial python3-food(-filesystem) package [0] > OR it can be in an existing package that is always present [1] > 2. %{python3_sitelib}/food/__init__.py and > %{python3_sitelib}/food/__pycache__/__init__.*.pyc > will conflict if present in multiple packages, > they need to be removed or shared from the python3-food(-filesystem) package > > I want to solve this once for all, define the best practice, document it in the > packaging guidelines and possible automate this in %pyproject_save_files [2]. > > > My current idea is: > > - sharing directories is safe and easy, > let's do that instead of artificial packages (those are hard to automate) > - namespace packages should not need __init__py with modern Python 3, > let's discourage that This, unfortunately, needs to be done upstream, at least some of the time. There are three different ways to do namespace packages in Python. The modern Python 3 version does not require __init__.py files but the other two (from before Python 3.3 added namespace packages to the core interpreter. One is implemented via pkgutil from the python stdlib and the other is implemented via a setuptools feature) have logic in the __init__.py to turn the directory into a namespace. The Python 3.3+ and pkgutil methods of namespace packaging are largely compatible (Enough so I think your idea to convert pkgutil-based packages to Python-3.3+ versions will work) but the setuptools version is incompatible.[1]_ The problem with us trying to change the setuptools using python modules that we package to use the modern Python 3 occurs when a user installs a different package in the namespace from upstream. The user then has two packages which implement the namespace in incompatible ways. My testing shows that this will result in all of our packages failing to be found by python.[2]_ You could modify your proposal to deal with setuptools based namespaces in a different manner than the other two namespaces. This might cause more mistakes (as packagers will have to figure out if they're in the special case scenario of a setuptools based namespace) but it does simplify packaging in the other two cases. .. [1]_: https://packaging.python.org/guides/packaging-namespace-packages/#creatin... .. [2]_: Here's the procedure to test compatibility: mkdir -p site-3.3/food/spam site-pkgutil/food/eggs site-setuptools/food/potato echo "print('spam')" > site-3.3/food/spam/__init__.py echo "__path__ = __import__('pkgutil').extend_path(__path__, __name__)" > site-pkgutil/food/__init__.py echo "print('eggs')" > site-pkgutil/food/eggs/__init__.py echo "__import__('pkg_resources').declare_namespace(__name__)" > site-setuptools/food/__init__.py echo "print('potato')" > site-pkgutil/food/eggs/__init__.py # These are both compatible PYTHONPATH=site-3.3:site-pkgutil python3 -c 'import food.spam, food.eggs' PYTHONPATH=site-pkgutil:site-3.3 python3 -c 'import food.spam, food.eggs' # The setuptools namespace makes it so Python does not register the 3.3-style namespace at all PYTHONPATH=site-3.3:site-setuptools python3 -c 'import food.spam, food.potato' PYTHONPATH=site-setuptools:site-3.3 python3 -c 'import food.spam, food.potato' # The setuptools namespace causes the pkgutil namespace to silently fail PYTHONPATH=site-setuptools:site-pkgutil python3 -c 'import food.eggs, food.potato' PYTHONPATH=site-setuptools:site-pkgutil python3 -c 'import food.eggs, food.potato'