python-devel November 2016

python-devel@lists.fedoraproject.org

15 participants
11 discussions

Python 3.6 and blocking-by-default os.urandom()
by Nick Coghlan 05 Jan '17

05 Jan '17

Hi folks, The long discussions upstream regarding the default behaviour of Python's os.urandom() API in Python 3.6 have come to an end, with Guido opting to make os.urandom() block implicitly waiting for the system RNG to be ready, even on Linux (where /dev/urandom doesn't do that). While I can see his point from a cross-platform language runtime design perspective, I'm still not sure it's the right answer for the Fedora system Python, as it doesn't play nice with ABRT, and goes against Ted T'so's decision to keep /dev/urandom non-blocking at the kernel level (risking confusion for folks that are Linux developers & operators first, and Pythonistas second). If "os.urandom() was called before system RNG is ready" throws an uncaught BlockingIOError, then we'll get a nice ABRT-friendly Python stack trace for people to debug (and perhaps figure out their VM entropy pool isn't being seeded properly, or their ARM hardware design needs a better source of entropy). By contrast, if it just blocks, then folks will be faced with a system hang, which they'll need to trace back to CPython being blocked on a kernel getrandom() call, and then infer from that that the system RNG isn't ready (for whatever reason), and they should probably do something about that. Patching this behaviour should be relatively straightforward - Python 3.5 currently makes this call in non-blocking mode and falls back to reading /dev/urandom directly in that case, so we'd just be forward porting the same logic to 3.6 and raising an exception instead of falling back to the file descriptor. Accordingly, what I propose we do is as follows: 1. Raise the concern in the F26 system-wide change proposal for migrating to Python 3.6 2. Apply the patch when the 3.6 beta releases are added to Fedora Rawhide 3. Decide whether or not to keep the patch based on ABRT results and other feedback on the Rawhide builds. Note that if the feedback on Rawhide shows that the change is helping people to find and diagnose VMs and hardware with improperly seeded entropy pools, that's a *good* thing: this proposed change is replacing Python 3.5's "/dev/urandom and os.urandom() may silently return statistically less-than-fully-random random numbers if the kernel entropy pool isn't seeded properly" with "os.urandom() will fail noisily in those cases, so you can either switch to the random module, or fix your entropy pool seeding". Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

4 9

List of packages owning %{python3_sitelib}/__pycache__
by Athos Ribeiro 30 Dec '16

30 Dec '16

Hello, Guidelines say that %{python3_sitelib}/__pycache__ should not be owned by python packages because python3-libs already owns it [1]. That directory is actually owned by system-python-libs. While going through a package review, I realized that 50+ packages own %{python3_sitelib}/__pycache__. To avoid generating noise to packagers, I am just listing those packages here [2]. [1] https://fedoraproject.org/wiki/Packaging:Python#Byte_compiling [2] - List of packages (rawhide) owning %{python3_sitelib}/__pycache__: cairo-dock-plug-ins netstat-monitor pydot pyparsing python-apipkg python-args python-autopep8 python-bottle python-cma python-cmdln python-configobj python-configparser python-cookies python-cram python-cycler python-debian python-demjson python-dialog python-docopt python-empy python-entrypoints python-feedparser python-flask-assets python-flask-login python-flask-principal python-flask-whooshee python-flexmock python-fuse python-hwdata python-ipgetter python-jupyter-core python-markdown2 python-mimeparse python-MultipartPostHandler2 python-novaclient-os-networks python-novaclient-os-virtual-interfaces python-optcomplete python-pandocfilters python-path python-pefile python-pickleshare python-pika_pool python-plaintable python-polib python-pylcdsysinfo python-pyphen python-pytest-flakes python-pytest-pep8 python-q python-random2 python-responses python-scp python-setuptools_hg python-simplemediawiki root uwsgi -- Athos Ribeiro http://www.ime.usp.br/~athoscr

7 6

Python SIG on-boarding issue
by Dhanesh B. Sabane 08 Dec '16

08 Dec '16

CommOps is currently working on the Python SIG On-Boarding ticket [1] and one of the tasks we have identified is re-writing the Python SIG wiki page to make it more beginner-friendly. I was assigned with this particular task which is now complete [2] [3]. We received some feedback from mhroncok today during the CommOps meeting and following points were discussed: 1. Splitting Python SIG in two FAS groups: 1.1 *python-sig* for newcomers and interested members who hang out and help with Python on Fedora 1.2 *python-packagers* for proven members of the community / packagers who will also receive security-related bugs. Promising members from *python-sig* group can be promoted to *python-packagers* given that they have contributed to the package's git. 2. Python Features section on the wiki page needs some love. 3. Targeting specific information for newcomers to include in their self-introductions that would be helpful for SIG members to know where someone fits in the Python community and what their expertise is. We would like to hear your feedback on the draft and also on the aforementioned points. Everyone is welcome to leave a comment on the ticket[1]. [1] https://pagure.io/fedora-commops/issue/84 [2] https://fedoraproject.org/wiki/User:Dhanesh95/SIGs/Python [3] https://fedoraproject.org/wiki/User:Dhanesh95/SIGs/Python/Join --- Dhanesh B. Sabane

6 11

[RFC] RPM's Python dependency generator
by Igor Gnatenko 04 Dec '16

04 Dec '16

Hi, in short, it reads egg metadata and can generate Provides (which we already do now), Requires (which I want to talk about) and Recommends (which I don't care atm). Let's take simple package -- aiohttp. https://bugzilla.redhat.com/show_bug.cgi?id=1381750 As you can see, since some version multidict requirement was bumped to >= 2.0 and async_timeout requirement was added. Currently we specify all requirements during initial package and usually without versions which is breaking after some time. So, let's try it (I will skip unrelated parts). Before: python(abi) = 3.5 python3-chardet python3-multidict After: python(abi) = 3.5 python3.5dist(async-timeout) python3.5dist(chardet) python3.5dist(multidict) >= 2.0 Without more complicated packages (MNE, nipy, nilearn, moss) it's getting much more harder since I have there 10+ deps. We can't really track all changes in upstream code, so if we will enable dependency generator, it will do this work for us. Note that we can't just enable it in RPM, because it will break a lot of packages due to: * missing requires in egg metadata * extra requires in egg metadata (e.g. windows-modules) I would propose plan: 1. Create macro which will enable/disable depgen (e.g %python_enable_depgen) 2. Start enabling depgen and porting things (somehow reuse portingdb.xyz probably?) and submitting patches upstream 3. In 1-2 releases I hope we can use it for major amount of packages 4. Enable depgen by default in RPM, add disabling depgen for remaining packages Neal, you can share how Mageia did that as well and feel free to comment this ;) Thoughts? -- -Igor Gnatenko

7 10

upstream dev. asks suggestions about howto make packagers work easier (bundled libraries, etc.)
by Germano Massullo 26 Nov '16

26 Nov '16

We often deal with upstream developers that bundle libraries in their code, so to make a package we have to debundle them, etc. This time, an upstream dev. asked me what he could do to make easier the work of packagers. In this case the software is python-netjsongraph [1] that bundles javascript-d3 library and that is being reviewd at [2] I think it would be nice to make a discussion even for non Python packages, so we can elaborate a sort of vademecum that a packager could show to upstreams when there is a collaboration between them. Have a nice day [1]: https://github.com/interop-dev/django-netjsongraph [2]: https://bugzilla.redhat.com/show_bug.cgi?id=1369213

5 4

Upgrade pip in EPEL
by Orion Poplawski 18 Nov '16

18 Nov '16

I'm thinking about upgrading pip in EPEL, but I don't really know about any backwards compatibility ramifications. Can anyone enlighten me about possible issues? -- Orion Poplawski Technical Manager 303-415-9701 x222 NWRA, Boulder/CoRA Office FAX: 303-415-9702 3380 Mitchell Lane orion(a)nwra.com Boulder, CO 80301 http://www.nwra.com

3 3

Fwd: [fpc] #663: New python egg macros
by Orion Poplawski 16 Nov '16

16 Nov '16

Any comments from the python community on this? -------- Forwarded Message -------- Subject: [fpc] #663: New python egg macros Date: Wed, 16 Nov 2016 05:43:59 -0000 From: fpc <trac(a)fedorahosted.org> Reply-To: nobody(a)fedoraproject.org #663: New python egg macros ------------------------------+------------------- Reporter: orion | Owner: Status: new | Priority: normal Component: Guideline Draft | Keywords: Blocked By: | Blocking: ------------------------------+------------------- I've introduced two new macros to python-rpm-macros: %py{,2,3}_build_egg and %py{,2,3}_install_egg to implement the instructions here: https://fedoraproject.org/wiki/Packaging:Python_Eggs#Multiple_Versions Changes suggested are here: https://fedoraproject.org/w/index.php?title=PackagingDrafts%3APython_Eggs&d… Also, it seems like the setuptools docs mentions the {{{__requires__}}} method, so it seems like the caveats could be removed. -- Ticket URL: <https://fedorahosted.org/fpc/ticket/663> fpc <https://fedoraproject.org/wiki/Packaging:Committee> Fedora Packages Collection Task Tracking

1 0

Re: [opensuse-buildservice] Re: introducing new macros across the whole OBS?
by Neal Gompa 10 Nov '16

10 Nov '16

On Thu, Nov 10, 2016 at 8:24 AM, jan matejek <jmatejek(a)suse.cz> wrote: > On 10.11.2016 14:16, Neal Gompa wrote: >> As far as I know, openSUSE was the *only* distribution *not* doing >> single spec Python 2/3 packaging. Fedora[1] and Mageia[2] both do. > > and as far as I can tell, we're now the only ones to (try to) automate > this as opposed to writing everything in the spec file explicitly ;) > > Which, for instance, would allow us to support pypy across the board > with little additional effort. > > Maybe that's where pyp2rpm comes in at Fedora? Does it regenerate the > spec for every build? > > m. > Yes. We also have it hooked up to our COPR service to autogenerate them fairly regularly[1][2][3] as a way to help make pyp2rpm even more useful and reliable. [1]: https://copr.fedorainfracloud.org/coprs/g/copr/PyPi/ [2]: https://copr.fedorainfracloud.org/coprs/g/copr/PyPI2/ [3]: https://copr.fedorainfracloud.org/coprs/g/copr/PyPI3/ -- 真実はいつも一つ！/ Always, there's only one truth!

1 0

Re: introducing new macros across the whole OBS?
by Neal Gompa 10 Nov '16

10 Nov '16

On Thu, Nov 10, 2016 at 8:11 AM, jan matejek <jmatejek(a)suse.com> wrote: > On 10.11.2016 14:02, Neal Gompa wrote: > >> Why not pull in the dependency generator I upstreamed into RPM[1] into >> openSUSE? Fedora is using it now in Fedora 25 and Rawhide[2][3], and >> Mageia is using an earlier variant of it now. It establishes common >> names for Python packages that provide egg-info or dist-info data >> (egg/wheel metadata installed on the system) that can be used for >> standardizing how Python dependencies are referenced. > > this is certainly very useful and we're looking into it, but it doesn't > seem to solve my question? > > to restate, I need to configure BuildRequires (not install-time > requirements) and pull in, as automagically as possible, all rpm > packages of "python module foo" for all supported pythons (that is > python2 and python3 at the moment, pypy3 in the near future (hopefully)) > Having the standardized names means you can just read requirements.txt or whatever and generate the list and expect it to work. It's a step that'd be required for that. OBS would likely need to grow the ability to run something like pyp2rpm spec generator before starting the build, which means being able to read requirements.txt or setup.py for the deps and resolving them properly. > also, BuildRequires are evaluated by OBS before rpmbuild starts, so I > can't rely on the usual ways of injecting macros into the build environment. > > (i'd also be interested in what Fedora has to say about the single-spec > thing, [1]. ISTM you don't do anything similar at the moment? > but i suppose that's for a different thread altogether) > > [1] https://lists.opensuse.org/opensuse-packaging/2016-10/msg00031.html > As far as I know, openSUSE was the *only* distribution *not* doing single spec Python 2/3 packaging. Fedora[1] and Mageia[2] both do. [1]: https://fedoraproject.org/wiki/Packaging:Python#Example_common_spec_file [2]: https://wiki.mageia.org/en/Python_policy#Samples -- 真実はいつも一つ！/ Always, there's only one truth!

1 1

Re: [opensuse-buildservice] introducing new macros across the whole OBS?
by Neal Gompa 10 Nov '16

10 Nov '16

On Thu, Nov 10, 2016 at 7:49 AM, jan matejek <jmatejek(a)suse.com> wrote: > Hello, > in relation to the python single-spec initiative, I have designed a set > of new macros that allow significant automation in building Python packages. > > However, these are constrained by the %python_module macro used in > BuildRequires. > The purpose of this macro is to expand %{python_module foo} to > python2-foo, python3-foo, etc., based on which python flavors are build > targets for the package. Obviously, the buildservice itself must be able > to do this expansion, otherwise it can't resolve the build requirements. > > So far I have solved this by placing all the macros in prjconf. But this > is not viable if I want my macros to be available everywhere. I can > create a "python-macros" package for all the other macros, and > buildrequire it, in order to make the packages build in any distro that > has this package, but I can't do the same for %python_module. > > Is it possible to make the buildservice itself recognize this macro, or > place it in some sort of global prjconf? Ideally without triggering a > full rebuild of everything? > Or do you have any tips for alternate solutions? > > One thing that occured to me, instead of relying on macros, turn > "python-foo" packages into metapackages requiring the respective > "python2-foo", "python3-foo" etc., and then BuildRequire the > metapackage. I don't have a clear idea about how well this would work, > it seems rather problematic. > > I don't want to make packagers list the buildrequires by hand, because > the requirements can be numerous and we also want to be able to add more > python flavors and expand the build requirements accordingly. > > thanks for any tips > m. > Why not pull in the dependency generator I upstreamed into RPM[1] into openSUSE? Fedora is using it now in Fedora 25 and Rawhide[2][3], and Mageia is using an earlier variant of it now. It establishes common names for Python packages that provide egg-info or dist-info data (egg/wheel metadata installed on the system) that can be used for standardizing how Python dependencies are referenced. Fedora is currently only using it for Provides generation at the moment, but Mageia uses it for Provides and Requires generation. That would make it less onerous in general to package Python stuff, as it would gain useful dependency generation like Perl and Ruby have. Tools like pyp2rpm[4] can use these names for generating good spec files that can build packages properly. Petr Viktorin gave a presentation at Flock 2016[5][6] about the future of Python packaging in Fedora, with this as a key staple. [1]: https://github.com/rpm-software-management/rpm/blob/master/scripts/pythondi… [2]: http://pkgs.fedoraproject.org/cgit/rpms/rpm.git/tree/ [3]: https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproj… [4]: https://github.com/fedora-python/pyp2rpm [5]: https://speakerdeck.com/encukou/python-packaging-in-fedora [6]: https://www.youtube.com/watch?v=rp5jq-25nZg -- 真実はいつも一つ！/ Always, there's only one truth!

1 0

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

python-devel November 2016