10:56 a.m.
Hi Pythonistas.
Regarding our Python 3 C.UTF-8 locale coercing [1], aka PEP 538 [2].
As you probably know, we build RPM packages with the C locale. So
everytime we use python3 in the spec file, the coercing message is
shown. This can be more problematic than just spamming the build logs,
see for example the related rpmlint bug [3][4].
Our macros, such as %{python3_sitelib}, %{python3_version} etc. all call
python3 and generate the warning. Should we "fix" our macros to set the
LANG to C.UTF-8?
If we change the %{__python3} macro entirely, we might get rid of all of
those warnings and we will workaround the fact that we build RPM
packages with the C locale. On the other hand the packager would not be
able to set a desired locale because it will always be overwritten:
# The crazy test suite needs Czech locale
LANG=cs_CZ.utf8 %{__python3} -m pytest
Will become:
LANG=cs_CZ.utf8 LANG=C.utf8 /usr/bin/python3 -m pytest
So I would not do that.
But we can change all other macros in
/usr/lib/rpm/macros.d/macros.python3 to set the UTF-8 locale. Would that
be wise? Desired?
Any thoughts?
[1] https://fedoraproject.org/wiki/Changes/python3_c.utf-8_locale
[2] https://www.python.org/dev/peps/pep-0538/
[3] https://bugzilla.redhat.com/show_bug.cgi?id=1457786
[4] https://bugzilla.redhat.com/show_bug.cgi?id=1436345
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
1:26 p.m.
I think it is better if the build system sets LANG=C.UTF-8, not the
python specific macros.
On Thu, Jun 1, 2017 at 8:56 AM, Miro Hrončok <mhroncok(a)redhat.com> wrote:
> Hi Pythonistas.
>
> Regarding our Python 3 C.UTF-8 locale coercing [1], aka PEP 538 [2].
>
> As you probably know, we build RPM packages with the C locale. So everytime
> we use python3 in the spec file, the coercing message is shown. This can be
> more problematic than just spamming the build logs, see for example the
> related rpmlint bug [3][4].
>
> Our macros, such as %{python3_sitelib}, %{python3_version} etc. all call
> python3 and generate the warning. Should we "fix" our macros to set the
LANG
> to C.UTF-8?
>
> If we change the %{__python3} macro entirely, we might get rid of all of
> those warnings and we will workaround the fact that we build RPM packages
> with the C locale. On the other hand the packager would not be able to set a
> desired locale because it will always be overwritten:
>
> # The crazy test suite needs Czech locale
> LANG=cs_CZ.utf8 %{__python3} -m pytest
>
> Will become:
>
> LANG=cs_CZ.utf8 LANG=C.utf8 /usr/bin/python3 -m pytest
>
> So I would not do that.
>
> But we can change all other macros in /usr/lib/rpm/macros.d/macros.python3
> to set the UTF-8 locale. Would that be wise? Desired?
>
> Any thoughts?
>
>
> [1] https://fedoraproject.org/wiki/Changes/python3_c.utf-8_locale
> [2] https://www.python.org/dev/peps/pep-0538/
> [3] https://bugzilla.redhat.com/show_bug.cgi?id=1457786
> [4] https://bugzilla.redhat.com/show_bug.cgi?id=1436345
>
> --
> Miro Hrončok
> --
> Phone: +420777974800
> IRC: mhroncok
> _______________________________________________
> python-devel mailing list -- python-devel(a)lists.fedoraproject.org
> To unsubscribe send an email to python-devel-leave(a)lists.fedoraproject.org
1:49 p.m.
On Thu, Jun 01, 2017 at 11:26:55AM -0700, Toshio Kuratomi wrote:
I think it is better if the build system sets LANG=C.UTF-8, not the
python specific macros.
Yeah, setting it through python macros would mean that e.g. any
python program invoked during the build would see the obsolete non-utf8
encoding. Seems better to just set LANG=C.UTF-8 globally.
Zbyszek
>
> On Thu, Jun 1, 2017 at 8:56 AM, Miro Hrončok <mhroncok(a)redhat.com> wrote:
> > Hi Pythonistas.
> >
> > Regarding our Python 3 C.UTF-8 locale coercing [1], aka PEP 538 [2].
> >
> > As you probably know, we build RPM packages with the C locale. So everytime
> > we use python3 in the spec file, the coercing message is shown. This can be
> > more problematic than just spamming the build logs, see for example the
> > related rpmlint bug [3][4].
> >
> > Our macros, such as %{python3_sitelib}, %{python3_version} etc. all call
> > python3 and generate the warning. Should we "fix" our macros to set
the LANG
> > to C.UTF-8?
> >
> > If we change the %{__python3} macro entirely, we might get rid of all of
> > those warnings and we will workaround the fact that we build RPM packages
> > with the C locale. On the other hand the packager would not be able to set a
> > desired locale because it will always be overwritten:
> >
> > # The crazy test suite needs Czech locale
> > LANG=cs_CZ.utf8 %{__python3} -m pytest
> >
> > Will become:
> >
> > LANG=cs_CZ.utf8 LANG=C.utf8 /usr/bin/python3 -m pytest
> >
> > So I would not do that.
> >
> > But we can change all other macros in /usr/lib/rpm/macros.d/macros.python3
> > to set the UTF-8 locale. Would that be wise? Desired?
> >
> > Any thoughts?
> >
> >
> > [1] https://fedoraproject.org/wiki/Changes/python3_c.utf-8_locale
> > [2] https://www.python.org/dev/peps/pep-0538/
> > [3] https://bugzilla.redhat.com/show_bug.cgi?id=1457786
> > [4] https://bugzilla.redhat.com/show_bug.cgi?id=1436345
> >
> > --
> > Miro Hrončok
> > --
> > Phone: +420777974800
> > IRC: mhroncok
> > _______________________________________________
> > python-devel mailing list -- python-devel(a)lists.fedoraproject.org
> > To unsubscribe send an email to python-devel-leave(a)lists.fedoraproject.org
> _______________________________________________
> python-devel mailing list -- python-devel(a)lists.fedoraproject.org
> To unsubscribe send an email to python-devel-leave(a)lists.fedoraproject.org
2:37 p.m.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
On Thu, 2017-06-01 at 17:56 +0200, Miro Hrončok wrote:
Hi Pythonistas.
Regarding our Python 3 C.UTF-8 locale coercing [1], aka PEP 538 [2].
As you probably know, we build RPM packages with the C locale. So
everytime we use python3 in the spec file, the coercing message is
shown. This can be more problematic than just spamming the build
logs,
see for example the related rpmlint bug [3][4].
I would prefer to fix mock once
which will fix not only python builds,
but others as well!
Our macros, such as %{python3_sitelib}, %{python3_version} etc. all
call
python3 and generate the warning. Should we "fix" our macros to set
the
LANG to C.UTF-8?
If we change the %{__python3} macro entirely, we might get rid of all
of
those warnings and we will workaround the fact that we build RPM
packages with the C locale. On the other hand the packager would not
be
able to set a desired locale because it will always be overwritten:
# The crazy test suite needs Czech locale
LANG=cs_CZ.utf8 %{__python3} -m pytest
Will become:
LANG=cs_CZ.utf8 LANG=C.utf8 /usr/bin/python3 -m pytest
So I would not do that.
But we can change all other macros in
/usr/lib/rpm/macros.d/macros.python3 to set the UTF-8 locale. Would
that
be wise? Desired?
Any thoughts?
[1] https://fedoraproject.org/wiki/Changes/python3_c.utf-8_locale
[2] https://www.python.org/dev/peps/pep-0538/
[3] https://bugzilla.redhat.com/show_bug.cgi?id=1457786
[4] https://bugzilla.redhat.com/show_bug.cgi?id=1436345
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
_______________________________________________
python-devel mailing list -- python-devel(a)lists.fedoraproject.org
To unsubscribe send an email to python-devel-leave(a)lists.fedoraprojec
t.org
- --
- -Igor Gnatenko
-----BEGIN PGP SIGNATURE-----
iQIzBAEBCAAdFiEEhLFO09aHZVqO+CM6aVcUvRu8X0wFAlkwbP0ACgkQaVcUvRu8
X0xZnw/9EGT6r12y4wcITPH+Y8SYCkcKVDsIIofhbZLFP1f+bAjYVjnVWccn5Nih
uFcXbHXcHp+jji+gAxrG44tdZS212aMIZ57cqEIx7xlG/qf5z1Vx+ZOMDhrP61lx
l+MAvoCI0fGdVGX5R7qZX1Vun6K6E93viAyjSTn+liCeJr9F0PxJsVC8StNvBk2c
2ZER6lgAQ0Nck+Y7K4oN7l09uHuBmxMCDcOXGBHcl5M3Mwh65WZncY162h4nQc2V
X7+Gyq83kx1Ya/LIOuG2rM74QcaaQ4IaoI2FBVj8DWFdgF/goYFOFfd3HwqVPVEZ
TJ8KV3XlnkaMqE9ysvKw3FZErUAxkqVy18OlkdEVFv7AQCx/eZqfTGrgurqCUYZQ
FB7ZjYjdNSyu5RFiOKXJWbUTgfHKe6BugVsvcajbpW493A5iOpnurMEuEL0w5ucp
KRnMvmEKq69iJ+NqXcGJRBIYN28RjAoxoydXcV5heVkMab4HJoLOhp0HdEsteFiD
l/+YcHG7HkERiKG6AxjJRjz42v8rgpP14fRnAQb+yKVfqQ5u5Ld0jsEIs+ST1Pwl
aHqoC2QEYPsL1rDEnOiWG2J61giUd0/pXJMMhE9Ki5yWtgsImnpqqHXAPn/accRR
5EjYFDFhP5PH3BkaowCjgzehjUBpHQwcCn8qHmqWN5AZoG3WLb0=
=K+2G
-----END PGP SIGNATURE-----
4:02 p.m.
On 1.6.2017 20:49, Zbigniew Jędrzejewski-Szmek wrote:
On Thu, Jun 01, 2017 at 11:26:55AM -0700, Toshio Kuratomi wrote:
> I think it is better if the build system sets LANG=C.UTF-8, not the
> python specific macros.
Yeah, setting it through python macros would mean that e.g. any
python program invoked during the build would see the obsolete non-utf8
encoding. Seems better to just set LANG=C.UTF-8 globally.
Who could make that happen? Should I propose this at devel@?
Zbyszek
>
> On Thu, Jun 1, 2017 at 8:56 AM, Miro Hrončok <mhroncok(a)redhat.com> wrote:
>> Hi Pythonistas.
>>
>> Regarding our Python 3 C.UTF-8 locale coercing [1], aka PEP 538 [2].
>>
>> As you probably know, we build RPM packages with the C locale. So everytime
>> we use python3 in the spec file, the coercing message is shown. This can be
>> more problematic than just spamming the build logs, see for example the
>> related rpmlint bug [3][4].
>>
>> Our macros, such as %{python3_sitelib}, %{python3_version} etc. all call
>> python3 and generate the warning. Should we "fix" our macros to set the
LANG
>> to C.UTF-8?
>>
>> If we change the %{__python3} macro entirely, we might get rid of all of
>> those warnings and we will workaround the fact that we build RPM packages
>> with the C locale. On the other hand the packager would not be able to set a
>> desired locale because it will always be overwritten:
>>
>> # The crazy test suite needs Czech locale
>> LANG=cs_CZ.utf8 %{__python3} -m pytest
>>
>> Will become:
>>
>> LANG=cs_CZ.utf8 LANG=C.utf8 /usr/bin/python3 -m pytest
>>
>> So I would not do that.
>>
>> But we can change all other macros in /usr/lib/rpm/macros.d/macros.python3
>> to set the UTF-8 locale. Would that be wise? Desired?
>>
>> Any thoughts?
>>
>>
>> [1] https://fedoraproject.org/wiki/Changes/python3_c.utf-8_locale
>> [2] https://www.python.org/dev/peps/pep-0538/
>> [3] https://bugzilla.redhat.com/show_bug.cgi?id=1457786
>> [4] https://bugzilla.redhat.com/show_bug.cgi?id=1436345
>>
>> --
>> Miro Hrončok
>> --
>> Phone: +420777974800
>> IRC: mhroncok
>> _______________________________________________
>> python-devel mailing list -- python-devel(a)lists.fedoraproject.org
>> To unsubscribe send an email to python-devel-leave(a)lists.fedoraproject.org
> _______________________________________________
> python-devel mailing list -- python-devel(a)lists.fedoraproject.org
> To unsubscribe send an email to python-devel-leave(a)lists.fedoraproject.org
_______________________________________________
python-devel mailing list -- python-devel(a)lists.fedoraproject.org
To unsubscribe send an email to python-devel-leave(a)lists.fedoraproject.org
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
10:15 p.m.
On Thu, Jun 01, 2017 at 11:02:25PM +0200, Miro Hrončok wrote:
On 1.6.2017 20:49, Zbigniew Jędrzejewski-Szmek wrote:
>On Thu, Jun 01, 2017 at 11:26:55AM -0700, Toshio Kuratomi wrote:
>>I think it is better if the build system sets LANG=C.UTF-8, not the
>>python specific macros.
>
>Yeah, setting it through python macros would mean that e.g. any
>python program invoked during the build would see the obsolete non-utf8
>encoding. Seems better to just set LANG=C.UTF-8 globally.
Who could make that happen? Should I propose this at devel@?
It seems that mock uses $LANG to configure $LANG and all $LC_* variables
in the build environment. But when mock is invoked with LANG=C, it ignores
that and uses C.UTF-8 instead, because of the "Python 3 C.UTF-8 locale"
change in python. So with new-enough python, we'd get C.UTF-8 anyway on the
builders, once the builders are upgraded to F26. But then you'd still get
a warning on every mock invocation.
It seems that right solution, that would work while the builders are
still not F26, would be to configure their locale to C.UTF-8. It'd
just be a matter of dropping LANG=C.UTF-8 into /etc/locale.conf.
I think a ticket on https://pagure.io/fedora-infrastructure would be
the way to request the change.
Zbyszek
3:46 a.m.
On Fri, Jun 2, 2017 at 1:15 PM, Zbigniew Jędrzejewski-Szmek
<zbyszek(a)in.waw.pl> wrote:
It seems that right solution, that would work while the builders are
still not F26, would be to configure their locale to C.UTF-8. It'd
just be a matter of dropping LANG=C.UTF-8 into /etc/locale.conf.
I think a ticket on https://pagure.io/fedora-infrastructure would be
the way to request the change.
While this seems like the right next step to me, if it isn't feasible
for some reason, we may need to reconsider emitting the warning at the
Python level, and instead just silently change LC_CTYPE to the more
sensible setting.
The accepted version of the PEP upstream *only* sets LC_CTYPE, so the
chance of unintended side effects from the coercion is much lower that
it was with the approach that made the F26 Beta cut-off (which also
sets LANG).
Cheers,
Nick.
--
Nick Coghlan
Red Hat Platform Engineering, Brisbane
9:42 p.m.
On 2 June 2017 at 18:46, Nick Coghlan <ncoghlan(a)redhat.com> wrote:
On Fri, Jun 2, 2017 at 1:15 PM, Zbigniew Jędrzejewski-Szmek
<zbyszek(a)in.waw.pl> wrote:
> It seems that right solution, that would work while the builders are
> still not F26, would be to configure their locale to C.UTF-8. It'd
> just be a matter of dropping LANG=C.UTF-8 into /etc/locale.conf.
> I think a ticket on https://pagure.io/fedora-infrastructure would be
> the way to request the change.
While this seems like the right next step to me, if it isn't feasible
for some reason, we may need to reconsider emitting the warning at the
Python level, and instead just silently change LC_CTYPE to the more
sensible setting.
A follow-up on this: I just made the change upstream to turn these
warnings off by default, and add a "PYTHONCOERCECLOCALE=warn" runtime
flag to let folks opt in to the warnings as a tool for debugging
potential locale coercion and locale compatibility problems. We were
simply getting far too many false positives where code that was
previously working fine (because it was genuinely only needing to
handle 7-bit ASCII inputs and outputs) was failing solely due to the
new warnings, not because anything was actually broken.
There are also some significant improvements to the test suite to make
sure that it's properly covering the expected behaviour when there
*aren't* any coercion target locales available. While that's not
applicable to Fedora directly, it *will* apply to any future Python
3.6 SCL, as well as to any python36 packages put together for EPEL.
Tracker issue: https://bugs.python.org/issue30565
Squashed commit:
https://github.com/python/cpython/commit/eb81795d7d3a8c898fa89a376d63fc3b...
Cheers,
Nick.
--
Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia
2503
days inactive
2520
days old
python-devel@lists.fedoraproject.org
7 comments
6 participants
participants (6)
-
Igor Gnatenko -
Miro Hrončok -
Nick Coghlan -
Nick Coghlan -
Toshio Kuratomi -
Zbigniew Jędrzejewski-Szmek