On 2020-08-12 17:22, Neal Gompa wrote:
On Wed, Aug 12, 2020 at 11:19 AM Petr Viktorin
<pviktori(a)redhat.com> wrote:
>
> On 2020-08-12 16:53, Neal Gompa wrote:
>> On Wed, Aug 12, 2020 at 10:23 AM Petr Viktorin <pviktori(a)redhat.com>
wrote:
>>>
>>> I'll move some discussion here, where it doesn't become part of the
>>> document:
>>>
>>>>
>>>> On 2020-08-11 14:19, Petr Viktorin wrote:
>>>>> These Guidelines represent a major rewrite and paradigm shift, and
not
>>>>> all packages are updated to reflect this.
>>>>> Older guidelines are still being kept up to date, and existing
packages
>>>>> **MAY** use them instead of this document:
>>>>> * 201x-era [Python packaging
>>>>>
guidelines](https://docs.fedoraproject.org/en-US/packaging-guidelines/Pyt...
>>>>> (for packages that use e.g. `%py3_install` or `setup.py install`)
>>>>> * [Python 2
>>>>>
appendix](https://docs.fedoraproject.org/en-US/packaging-guidelines/Pytho...
>>>>> (for e.g. `%py2_install`) (Python 2 packages require a FESCo
exception.)
>>>
>>> @Conan-Kudo:
>>>> Is there something you've done here to make the new macros
unequivocally better? Do you have automated flavor builds, for example? Right now there is
no effective difference.
>>>
>>>
>>> The new macros allow other build tools than just setuptools.
>>> They also use upstream metadata for BuildRequires & file lists.
>>>
>>> Of course, where possible the changes were backported to the existing
>>> macros. I don't necessarily see the macros as the main thing that
>>> changes. But when you look at a specfile, you can usually tell what
>>> conventions it uses by what macros you see.
>>>
>>
>> Those are all nice things, for sure. I think it'd be important to
>> spell that out somewhere.
>
> Do you mean providing a summary of the changes between the existing
> guidelines and these?
>
Yes. Also indicating the *why* for pyproject stuff is useful within
the context of why the old macros are deprecated.
Is that necessary for the beta release of the packaging guidelines?
(i.e. when they would be in effect, but optional?)
You're asking for quite a lot of work around describing a document that
might still change.
>>>>> ## PyPI parity
>>>>>
>>>>> Every Python package in Fedora **SHOULD** also be available
>>>>> on [the Python Package Index](https://pypi.org) (PyPI).
>>>>>
>>>>> The command `pip install PROJECTNAME` **MUST** install the same
package
>>>>> (possibly in a different version),
>>>>> install nothing,
>>>>> or fail with a reasonable error message.
>>>>>
>>>>> If this is not the case, the packager **MUST** contact upstream about
this.
>>>>> The goal is to get the project name registered or blocked on PyPI,
>>>>> or to otherwise ensure the rule is followed.
>>>>>
>>>>> To block a project name on PyPI, email
>>>>> [admin@pypi.org](mailto:admin@pypi.org),
>>>>> giving the project name and explaining the situation
>>>>> (for example: the package cannot currently be installed via `pip`).
>>>>> You can ask questions and discuss the process at the [Python
>>>>>
Discourse](https://discuss.python.org/t/block-names/4045).
>>>>>
>>>>> > NOTE: Project names that were in Fedora but not on PyPI
>>>>> > when these guidelines were proposed
>>>>> > are *blocked* from being uploaded to PyPI.
>>>>> > This prevents potential trolls from taking them,
>>>>> > but it also blocks legitimate owners.
>>>>> > If your package is affected, contact the Python SIG
>>>>> > or [file a PyPA
>>>>>
issue](https://github.com/pypa/pypi-support/issues/new?labels=PEP+541&...
>>>>>
>>>>> > and mention `@encukou`.
>>>>>
>>>>> This is necessary to protect users,
>>>>> avoid confusion,
>>>>> and enable automation as Fedora and upstream ecosystems grow more
>>>>> integrated.
>>>>>
>>>>> As always, [specific exceptions can be granted by the Packaging
>>>>>
Committee](https://docs.fedoraproject.org/en-US/packaging-guidelines/#_ge....
>>>
>>>
>>> @Conan-Kudo:
>>>> This is not reasonable to ask of packagers to do. Such a check is
difficult to do, and there is no particularly compelling reason for making everything
written in Python using Python build system available in PyPI. Unless you plan to provide
an automated system to inform PyPI of these things, this is not only unreasonable, it is
unenforceable.
>>>
>>> A lot of stuff in the guidelines is unenforceable, so let's focus on the
>>> "unreasonable" part.
>>>
>>> There is no reason to have everything available on PyPI, but I do
>>> believe it's reasonable to *reserve the name* in such cases.
>>>
>>> Here's a reason why I want this:
>>>
>>> The issue here is that Python tools have access to project names. For
>>> example, I can get the version of Requests using:
>>>
>>> >>> from importlib.metadata import version
>>> >>> version('requests')
>>> '2.22.0'
>>>
>>> Things like this are only useful if we use a common namespace. Upstream,
>>> that namespace *is* PyPI; there's little we can do about that.
>>> If project names mean something else in Fedora than in the wider
>>> ecosystem, we'll get user confusion at best.
>>>
>>> (If you use a private package index, like they do at CERN or some
>>> closed-source shops, there can be some collisions -- but in that case
>>> there's a limited number of software authors and users, and a lot more
>>> control over the package set. Conflicts in global repositories of
>>> free/open-source software are much harder to manage.)
>>>
>>>
>>>
>>> Lately, I think about another way to handle this: packages that aren't
>>> on PyPI could not ship the .dist-info at all, and so, they would not
>>> have things like `python3dist(...)` provides. They'd only be usable with
>>> Fedora tooling, not in the wider Python ecosystem.
>>> It's a major case to think out and test, but maybe it would be worth it?
>>>
>>
>> I think omitting the Provides is unfair.
>
> Why?
> In my view, `python3dist(...)` is using the namespace of the wider
> Python ecosystem.
> In Fedora, you can always use Fedora package names. But for Python
> tools, we need to use names from PyPI.
>
I disagree here. There is nothing inherent about pythonXdist()
metadata that indicates it is associated with PyPI. It's *nice* if
it's in PyPI, but not even Fedora projects themselves do it, so...
Well, this *is* the main change this document brings: it says that
`pythonXdist(...)` now *will* inherently use the PyPI namespace, the
default repository of open-source packages. I'm not documenting the
status quo. I am saying the Fedora projects that don't currently do this
are should be considered wrong and should be fixed.
I `pip install foo` (in a venv with system site packages), and the PyPI
package `foo` depends on `bar`, I should get the `bar` from PyPI even if
I have some unrelated, Fedora-only `python3-bar` package.
This is what people expect, because the Fedora and PyPI namespaces
already *almost* always overlap. When they don't, users are confused.
And as we automate stuff to use upstream metadata more, the tools will
"get confused" as well.
If you do need a Fedora-only namespace, regular package names are
completely adequate. But we need a *Python-ecosystem-wide* namespace.
And that is PyPI.
>> What I think we should do is figure out a system to make
such
>> reservations more automated, like making such requests and linking
>> them to package review bugs, packaging git repos, and a koji build to
>> indicate their validity. PyPI upstream can then automatically verify
>> the validity of the request and reserve them, and we could have a bug
>> filed to have the maintainer reach out to upstream (or bug filed to
>> upstream directly if the metadata has info on that) to submit stuff to
>> PyPI.
>
> Automation is always nice. But IMO, first we need to agree on what to
> automate, not start with the automation.
You're starting with the implication that the pythonXdist() name must
exist in PyPI, therefore you already know what must be automated (the
process of registering Fedora packages in PyPI that use pythonXdist()
names).
Not really: it would be better to automate releasing the packages to
PyPI, rather than blocking their names. That way, the package can be
more widely useful – e.g. outside system-wide distro installs.
And you can use twine for that.
I can see some cases where you want neither to release to PyPI nor to
block the name:
- you use a non-Python buildsystem, which doesn't generate
PyPI-compatible artifacts – but then it shouldn't generate .dist-info either
- you're packaging a Fedora-only thing – but then other packages can
depend on the Fedora package name
- you're packaging an application that other Python libraries should not
depend on – but then, .dist-info is quite useless and can be omitted