Python 3.6 and blocking-by-default os.urandom()
by Nick Coghlan
Hi folks,
The long discussions upstream regarding the default behaviour of Python's
os.urandom() API in Python 3.6 have come to an end, with Guido opting to
make os.urandom() block implicitly waiting for the system RNG to be ready,
even on Linux (where /dev/urandom doesn't do that).
While I can see his point from a cross-platform language runtime design
perspective, I'm still not sure it's the right answer for the Fedora system
Python, as it doesn't play nice with ABRT, and goes against Ted T'so's
decision to keep /dev/urandom non-blocking at the kernel level (risking
confusion for folks that are Linux developers & operators first, and
Pythonistas second).
If "os.urandom() was called before system RNG is ready" throws an uncaught
BlockingIOError, then we'll get a nice ABRT-friendly Python stack trace for
people to debug (and perhaps figure out their VM entropy pool isn't being
seeded properly, or their ARM hardware design needs a better source of
entropy). By contrast, if it just blocks, then folks will be faced with a
system hang, which they'll need to trace back to CPython being blocked on a
kernel getrandom() call, and then infer from that that the system RNG isn't
ready (for whatever reason), and they should probably do something about
that.
Patching this behaviour should be relatively straightforward - Python 3.5
currently makes this call in non-blocking mode and falls back to reading
/dev/urandom directly in that case, so we'd just be forward porting the
same logic to 3.6 and raising an exception instead of falling back to the
file descriptor.
Accordingly, what I propose we do is as follows:
1. Raise the concern in the F26 system-wide change proposal for migrating
to Python 3.6
2. Apply the patch when the 3.6 beta releases are added to Fedora Rawhide
3. Decide whether or not to keep the patch based on ABRT results and other
feedback on the Rawhide builds.
Note that if the feedback on Rawhide shows that the change is helping
people to find and diagnose VMs and hardware with improperly seeded entropy
pools, that's a *good* thing: this proposed change is replacing Python
3.5's "/dev/urandom and os.urandom() may silently return statistically
less-than-fully-random random numbers if the kernel entropy pool isn't
seeded properly" with "os.urandom() will fail noisily in those cases, so
you can either switch to the random module, or fix your entropy pool
seeding".
Cheers,
Nick.
--
Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia
6 years, 11 months
List of packages owning %{python3_sitelib}/__pycache__
by Athos Ribeiro
Hello,
Guidelines say that %{python3_sitelib}/__pycache__ should not be owned
by python packages because python3-libs already owns it [1]. That
directory is actually owned by system-python-libs.
While going through a package review, I realized that 50+ packages own
%{python3_sitelib}/__pycache__. To avoid generating noise to packagers,
I am just listing those packages here [2].
[1] https://fedoraproject.org/wiki/Packaging:Python#Byte_compiling
[2] - List of packages (rawhide) owning %{python3_sitelib}/__pycache__:
cairo-dock-plug-ins
netstat-monitor
pydot
pyparsing
python-apipkg
python-args
python-autopep8
python-bottle
python-cma
python-cmdln
python-configobj
python-configparser
python-cookies
python-cram
python-cycler
python-debian
python-demjson
python-dialog
python-docopt
python-empy
python-entrypoints
python-feedparser
python-flask-assets
python-flask-login
python-flask-principal
python-flask-whooshee
python-flexmock
python-fuse
python-hwdata
python-ipgetter
python-jupyter-core
python-markdown2
python-mimeparse
python-MultipartPostHandler2
python-novaclient-os-networks
python-novaclient-os-virtual-interfaces
python-optcomplete
python-pandocfilters
python-path
python-pefile
python-pickleshare
python-pika_pool
python-plaintable
python-polib
python-pylcdsysinfo
python-pyphen
python-pytest-flakes
python-pytest-pep8
python-q
python-random2
python-responses
python-scp
python-setuptools_hg
python-simplemediawiki
root
uwsgi
--
Athos Ribeiro
http://www.ime.usp.br/~athoscr
6 years, 11 months
Python SIG on-boarding issue
by Dhanesh B. Sabane
CommOps is currently working on the Python SIG On-Boarding ticket [1] and one of the tasks we have identified is re-writing the Python SIG wiki page to make it more beginner-friendly. I was assigned with this particular task which is now complete [2] [3]. We received some feedback from mhroncok today during the CommOps meeting and following points were discussed:
1. Splitting Python SIG in two FAS groups:
1.1 *python-sig* for newcomers and interested members who hang out and help with Python on Fedora
1.2 *python-packagers* for proven members of the community / packagers who will also receive security-related bugs. Promising members from *python-sig* group can be promoted to *python-packagers* given that they have contributed to the package's git.
2. Python Features section on the wiki page needs some love.
3. Targeting specific information for newcomers to include in their self-introductions that would be helpful for SIG members to know where someone fits in the Python community and what their expertise is.
We would like to hear your feedback on the draft and also on the aforementioned points. Everyone is welcome to leave a comment on the ticket[1].
[1] https://pagure.io/fedora-commops/issue/84
[2] https://fedoraproject.org/wiki/User:Dhanesh95/SIGs/Python
[3] https://fedoraproject.org/wiki/User:Dhanesh95/SIGs/Python/Join
---
Dhanesh B. Sabane
7 years
[RFC] RPM's Python dependency generator
by Igor Gnatenko
Hi,
in short, it reads egg metadata and can generate Provides (which we
already do now), Requires (which I want to talk about) and Recommends
(which I don't care atm).
Let's take simple package -- aiohttp.
https://bugzilla.redhat.com/show_bug.cgi?id=1381750
As you can see, since some version multidict requirement was bumped to
>= 2.0 and async_timeout requirement was added. Currently we specify
all requirements during initial package and usually without versions
which is breaking after some time.
So, let's try it (I will skip unrelated parts).
Before:
python(abi) = 3.5
python3-chardet
python3-multidict
After:
python(abi) = 3.5
python3.5dist(async-timeout)
python3.5dist(chardet)
python3.5dist(multidict) >= 2.0
Without more complicated packages (MNE, nipy, nilearn, moss) it's
getting much more harder since I have there 10+ deps.
We can't really track all changes in upstream code, so if we will
enable dependency generator, it will do this work for us. Note that we
can't just enable it in RPM, because it will break a lot of packages
due to:
* missing requires in egg metadata
* extra requires in egg metadata (e.g. windows-modules)
I would propose plan:
1. Create macro which will enable/disable depgen (e.g %python_enable_depgen)
2. Start enabling depgen and porting things (somehow reuse
portingdb.xyz probably?) and submitting patches upstream
3. In 1-2 releases I hope we can use it for major amount of packages
4. Enable depgen by default in RPM, add disabling depgen for remaining packages
Neal, you can share how Mageia did that as well and feel free to comment this ;)
Thoughts?
--
-Igor Gnatenko
7 years
Upgrade pip in EPEL
by Orion Poplawski
I'm thinking about upgrading pip in EPEL, but I don't really know about any
backwards compatibility ramifications. Can anyone enlighten me about possible
issues?
--
Orion Poplawski
Technical Manager 303-415-9701 x222
NWRA, Boulder/CoRA Office FAX: 303-415-9702
3380 Mitchell Lane orion(a)nwra.com
Boulder, CO 80301 http://www.nwra.com
7 years
Re: introducing new macros across the whole OBS?
by Neal Gompa
On Thu, Nov 10, 2016 at 8:11 AM, jan matejek <jmatejek(a)suse.com> wrote:
> On 10.11.2016 14:02, Neal Gompa wrote:
>
>> Why not pull in the dependency generator I upstreamed into RPM[1] into
>> openSUSE? Fedora is using it now in Fedora 25 and Rawhide[2][3], and
>> Mageia is using an earlier variant of it now. It establishes common
>> names for Python packages that provide egg-info or dist-info data
>> (egg/wheel metadata installed on the system) that can be used for
>> standardizing how Python dependencies are referenced.
>
> this is certainly very useful and we're looking into it, but it doesn't
> seem to solve my question?
>
> to restate, I need to configure BuildRequires (not install-time
> requirements) and pull in, as automagically as possible, all rpm
> packages of "python module foo" for all supported pythons (that is
> python2 and python3 at the moment, pypy3 in the near future (hopefully))
>
Having the standardized names means you can just read requirements.txt
or whatever and generate the list and expect it to work. It's a step
that'd be required for that. OBS would likely need to grow the ability
to run something like pyp2rpm spec generator before starting the
build, which means being able to read requirements.txt or setup.py for
the deps and resolving them properly.
> also, BuildRequires are evaluated by OBS before rpmbuild starts, so I
> can't rely on the usual ways of injecting macros into the build environment.
>
> (i'd also be interested in what Fedora has to say about the single-spec
> thing, [1]. ISTM you don't do anything similar at the moment?
> but i suppose that's for a different thread altogether)
>
> [1] https://lists.opensuse.org/opensuse-packaging/2016-10/msg00031.html
>
As far as I know, openSUSE was the *only* distribution *not* doing
single spec Python 2/3 packaging. Fedora[1] and Mageia[2] both do.
[1]: https://fedoraproject.org/wiki/Packaging:Python#Example_common_spec_file
[2]: https://wiki.mageia.org/en/Python_policy#Samples
--
真実はいつも一つ！/ Always, there's only one truth!
7 years, 1 month
Re: [opensuse-buildservice] introducing new macros across the whole OBS?
by Neal Gompa
On Thu, Nov 10, 2016 at 7:49 AM, jan matejek <jmatejek(a)suse.com> wrote:
> Hello,
> in relation to the python single-spec initiative, I have designed a set
> of new macros that allow significant automation in building Python packages.
>
> However, these are constrained by the %python_module macro used in
> BuildRequires.
> The purpose of this macro is to expand %{python_module foo} to
> python2-foo, python3-foo, etc., based on which python flavors are build
> targets for the package. Obviously, the buildservice itself must be able
> to do this expansion, otherwise it can't resolve the build requirements.
>
> So far I have solved this by placing all the macros in prjconf. But this
> is not viable if I want my macros to be available everywhere. I can
> create a "python-macros" package for all the other macros, and
> buildrequire it, in order to make the packages build in any distro that
> has this package, but I can't do the same for %python_module.
>
> Is it possible to make the buildservice itself recognize this macro, or
> place it in some sort of global prjconf? Ideally without triggering a
> full rebuild of everything?
> Or do you have any tips for alternate solutions?
>
> One thing that occured to me, instead of relying on macros, turn
> "python-foo" packages into metapackages requiring the respective
> "python2-foo", "python3-foo" etc., and then BuildRequire the
> metapackage. I don't have a clear idea about how well this would work,
> it seems rather problematic.
>
> I don't want to make packagers list the buildrequires by hand, because
> the requirements can be numerous and we also want to be able to add more
> python flavors and expand the build requirements accordingly.
>
> thanks for any tips
> m.
>
Why not pull in the dependency generator I upstreamed into RPM[1] into
openSUSE? Fedora is using it now in Fedora 25 and Rawhide[2][3], and
Mageia is using an earlier variant of it now. It establishes common
names for Python packages that provide egg-info or dist-info data
(egg/wheel metadata installed on the system) that can be used for
standardizing how Python dependencies are referenced.
Fedora is currently only using it for Provides generation at the
moment, but Mageia uses it for Provides and Requires generation. That
would make it less onerous in general to package Python stuff, as it
would gain useful dependency generation like Perl and Ruby have. Tools
like pyp2rpm[4] can use these names for generating good spec files
that can build packages properly. Petr Viktorin gave a presentation at
Flock 2016[5][6] about the future of Python packaging in Fedora, with
this as a key staple.
[1]: https://github.com/rpm-software-management/rpm/blob/master/scripts/python...
[2]: http://pkgs.fedoraproject.org/cgit/rpms/rpm.git/tree/
[3]: https://lists.fedoraproject.org/archives/list/python-devel@lists.fedorapr...
[4]: https://github.com/fedora-python/pyp2rpm
[5]: https://speakerdeck.com/encukou/python-packaging-in-fedora
[6]: https://www.youtube.com/watch?v=rp5jq-25nZg
--
真実はいつも一つ！/ Always, there's only one truth!
7 years, 1 month
