3:19 a.m.
As mentioned in my self-introduction email, I'm interested in the idea
of filtered Fedora provided mirrors of PyPI/RubyGems/etc that contained
only packages that had been through Fedora Playground levels of review.
Having such selective mirrors available would mean that we could freely
choose between using the language level packaging tools and RPMs on a
project by project basis without having to recheck the licenses
ourselves. It would also provide a way for us to easily share the
results of our own licensing reviews when we bring in new dependencies
from upstream.
I believe this approach would also set up a good pipeline for bringing
new packages into Fedora:
Step 1: basic review
- if the package is even remotely acceptable to Fedora (e.g. suitable
license, not actively malicious), then flag it for inclusion in the
appropriate language specific mirror
Step 2: draft spec file
- make an initial cut at a spec file (e.g. autogenerated)
- provide as a COPR repo
- include in Fedora Playground
Step 3: full packaging policy compliance
- iterate on the spec file and the package to attain policy compliance
- incorporate into Fedora itself
Does this sound like an approach worth pursuing?
In the specific case of Python, devpi is an existing project that I
believe would be eminently suitable to providing a service like this
(using the whitelisting feature to indicate packages which had been
checked for Fedora Playground policy compatibility): http://doc.devpi.net/
Regards,
Nick.
--
Nick Coghlan
Red Hat Hosted & Shared Services
Software Engineering & Development, Brisbane
HSS Provisioning Architect
8:34 a.m.
Hi Nick
Dne 15.9.2014 10:19, Nick Coghlan napsal(a):
As mentioned in my self-introduction email, I'm interested in the
idea
of filtered Fedora provided mirrors of PyPI/RubyGems/etc that contained
only packages that had been through Fedora Playground levels of review.
Having such selective mirrors available would mean that we could freely
choose between using the language level packaging tools and RPMs on a
project by project basis without having to recheck the licenses
ourselves. It would also provide a way for us to easily share the
results of our own licensing reviews when we bring in new dependencies
from upstream.
While you plot this nicely and it sounds attractive, I am really against
this idea. Here are a few points which comes to my mind:
1) Speaking of Rubyists, you would need to convince them to start using
"source "https://fedora-rubygems-mirror.org" instead of standard
"source
"https://rubygems.org"" in their Gemfiles. Not sure how you would like
to achieve this. We could patch Bundler somehow to substitute the
original source on Fedora, but I don't want to imagine what source of
problems it would bring.
2) In Fedora, there are packages, which are modified from upstream
versions. For example, rubygem-listen [1] provides universal API to
provide notifications about file changes on file system. Upstream
decided, that they will have hard dependencies on other platform
specific libraries, which supports FS level notifications. We decided to
drop them, since it does not make sense to support libraries which does
nothing on Linux. Sometimes, we are forced to override overly tight
dependencies on other libraries. And now to the question. What would
happened with such gem? What the mirror should provide in such case?
Fedora's version or upstream version of gem? The former is quite
dangerous idea, since it would end up in two different packages of the
same version.
3) You need additional human/infrastructure resources.
Additionally to the point 3, if you have such resources, I would be
better to invest them into development of proper support of multiple
versions of packages in DNF. That would remove the need for most of the
SCLs and additional repositories in Copr, at least speaking of Ruby
ecosystem.
Nowadays, people are not using ruby193 collection, because they are in
love with Ruby 1.9.3. They are using the collection since it is bundled
with Ruby on Rails 3.2. And they are using SCLs just because it allows
them to install more versions of RoR on single system, not because they
believe that their worflow is now much easier. But you can achieve it
just fine with rpm -i having the RPMs. It is just not supported by
DNF/YUM, since DNF/YUM understands just "upgrade, don't leave old
version on system".
Vít
[1] http://rubygems.org/gems/listen
9:40 a.m.
Dne 15.9.2014 15:34, Vít Ondruch napsal(a):
Hi Nick
Dne 15.9.2014 10:19, Nick Coghlan napsal(a):
> As mentioned in my self-introduction email, I'm interested in the idea
> of filtered Fedora provided mirrors of PyPI/RubyGems/etc that contained
> only packages that had been through Fedora Playground levels of review.
>
> Having such selective mirrors available would mean that we could freely
> choose between using the language level packaging tools and RPMs on a
> project by project basis without having to recheck the licenses
> ourselves. It would also provide a way for us to easily share the
> results of our own licensing reviews when we bring in new dependencies
> from upstream.
While you plot this nicely and it sounds attractive, I am really against
this idea. Here are a few points which comes to my mind:
1) Speaking of Rubyists, you would need to convince them to start using
"source "https://fedora-rubygems-mirror.org" instead of standard
"source
"https://rubygems.org"" in their Gemfiles. Not sure how you would like
to achieve this. We could patch Bundler somehow to substitute the
original source on Fedora, but I don't want to imagine what source of
problems it would bring.
2) In Fedora, there are packages, which are modified from upstream
versions. For example, rubygem-listen [1] provides universal API to
provide notifications about file changes on file system. Upstream
decided, that they will have hard dependencies on other platform
specific libraries, which supports FS level notifications. We decided to
drop them, since it does not make sense to support libraries which does
nothing on Linux. Sometimes, we are forced to override overly tight
dependencies on other libraries. And now to the question. What would
happened with such gem? What the mirror should provide in such case?
Fedora's version or upstream version of gem? The former is quite
dangerous idea, since it would end up in two different packages of the
same version.
3) You need additional human/infrastructure resources.
Additionally to the point 3, if you have such resources, I would be
better to invest them into development of proper support of multiple
versions of packages in DNF. That would remove the need for most of the
SCLs and additional repositories in Copr, at least speaking of Ruby
ecosystem.
Nowadays, people are not using ruby193 collection, because they are in
love with Ruby 1.9.3. They are using the collection since it is bundled
with Ruby on Rails 3.2. And they are using SCLs just because it allows
them to install more versions of RoR on single system, not because they
believe that their worflow is now much easier. But you can achieve it
just fine with rpm -i having the RPMs. It is just not supported by
DNF/YUM, since DNF/YUM understands just "upgrade, don't leave old
version on system".
BTW I forgot to mention one issue. "gem install" or "bundle
--install"
works just fine for every user. But the issue is with binary extensions.
What should I do to install sqlite3 gem for example:
$ gem install sqlite3
Fetching: sqlite3-1.3.9.gem (100%)
Building native extensions. This could take a while...
ERROR: Error installing sqlite3:
ERROR: Failed to build gem native extension.
/usr/bin/ruby extconf.rb
mkmf.rb can't find header files for ruby at /usr/share/include/ruby.h
Gem files will remain installed in
/home/vondruch/.gem/ruby/gems/sqlite3-1.3.9 for inspection.
Results logged to
/home/vondruch/.gem/ruby/gems/sqlite3-1.3.9/ext/sqlite3/gem_make.out
There are two answers:
1) yum install rubygem-sqlite3
2) gem instal gem-nice-install && gem install sqlite3
where yum install knows everybody on Red Hat platform, gem-nice-install
knows virtually nobody. And to be honest, even me as a rubyist prefer 1)
since I have *no* compiler on my system and I hope it stays like that.
Vít
10:08 p.m.
On 09/16/2014 12:40 AM, Vít Ondruch wrote:
There are two answers:
1) yum install rubygem-sqlite3
2) gem instal gem-nice-install && gem install sqlite3
where yum install knows everybody on Red Hat platform, gem-nice-install
knows virtually nobody. And to be honest, even me as a rubyist prefer 1)
since I have *no* compiler on my system and I hope it stays like that.
The Python world defined its own binary package format in order to deal
with that problem (http://www.python.org/dev/peps/pep-0427/). That's
actually one of my other reasons for wanting a Fedora specific PyPI
mirror based on devpi - devpi is specifically defined in a way that
would let us build and push Fedora compatible wheel files for each
version of the distro (as well as each version of EPEL), while the
architecture independent files could remain in a shared index. That's a
solution that will work *today*, whereas providing a general solution
for arbitrary Linux distros will require further upstream engineering
effort (and is still only at the "kicking ideas around" stage).
For a broader picture of my perspective on how upstream and
redistributors can work together to provide a better developer
experience, it's worth having a look at my draft design for the next
generation Python packaging metadata format:
http://www.python.org/dev/peps/pep-0426/
Regards,
Nick.
--
Nick Coghlan
Red Hat Hosted & Shared Services
Software Engineering & Development, Brisbane
HSS Provisioning Architect
12:26 a.m.
----- Original Message -----
Dne 15.9.2014 15:34, Vít Ondruch napsal(a):
> Hi Nick
>
> Dne 15.9.2014 10:19, Nick Coghlan napsal(a):
>> As mentioned in my self-introduction email, I'm interested in the idea
>> of filtered Fedora provided mirrors of PyPI/RubyGems/etc that contained
>> only packages that had been through Fedora Playground levels of review.
>>
>> Having such selective mirrors available would mean that we could freely
>> choose between using the language level packaging tools and RPMs on a
>> project by project basis without having to recheck the licenses
>> ourselves. It would also provide a way for us to easily share the
>> results of our own licensing reviews when we bring in new dependencies
>> from upstream.
> While you plot this nicely and it sounds attractive, I am really against
> this idea. Here are a few points which comes to my mind:
>
> 1) Speaking of Rubyists, you would need to convince them to start using
> "source "https://fedora-rubygems-mirror.org" instead of standard
"source
> "https://rubygems.org"" in their Gemfiles. Not sure how you would
like
> to achieve this. We could patch Bundler somehow to substitute the
> original source on Fedora, but I don't want to imagine what source of
> problems it would bring.
IIRC that's what Maven does already.
>
> 2) In Fedora, there are packages, which are modified from upstream
> versions. For example, rubygem-listen [1] provides universal API to
> provide notifications about file changes on file system. Upstream
> decided, that they will have hard dependencies on other platform
> specific libraries, which supports FS level notifications. We decided to
> drop them, since it does not make sense to support libraries which does
> nothing on Linux. Sometimes, we are forced to override overly tight
> dependencies on other libraries. And now to the question. What would
> happened with such gem? What the mirror should provide in such case?
> Fedora's version or upstream version of gem? The former is quite
> dangerous idea, since it would end up in two different packages of the
> same version.
The idea is that the content provided by Fedora/RHEL/CentOS can be still modified/patched
version of upstream content. If we can automate most of the work, that's great but
I'm pretty sure we will have to maintain patches for various components. We will still
be downstream.
>
> 3) You need additional human/infrastructure resources.
>
> Additionally to the point 3, if you have such resources, I would be
> better to invest them into development of proper support of multiple
> versions of packages in DNF. That would remove the need for most of the
> SCLs and additional repositories in Copr, at least speaking of Ruby
> ecosystem.
Vitku, seems like we need another meeting. Nick, would you be interested in joining it?
There are various requirements from both the software management side as well as from
Developers/Users and we need to find a the best approach which suits developer needs and
still supports admins and the way they manage the system. The main idea for developers was
"Use what they're used to and support it on the OS" (which actually applies
for admins as well)
>
> Nowadays, people are not using ruby193 collection, because they are in
> love with Ruby 1.9.3. They are using the collection since it is bundled
> with Ruby on Rails 3.2. And they are using SCLs just because it allows
> them to install more versions of RoR on single system, not because they
> believe that their worflow is now much easier. But you can achieve it
> just fine with rpm -i having the RPMs. It is just not supported by
> DNF/YUM, since DNF/YUM understands just "upgrade, don't leave old
> version on system".
>
>
BTW I forgot to mention one issue. "gem install" or "bundle
--install"
works just fine for every user. But the issue is with binary extensions.
What should I do to install sqlite3 gem for example:
$ gem install sqlite3
Fetching: sqlite3-1.3.9.gem (100%)
Building native extensions. This could take a while...
ERROR: Error installing sqlite3:
ERROR: Failed to build gem native extension.
/usr/bin/ruby extconf.rb
mkmf.rb can't find header files for ruby at /usr/share/include/ruby.h
Gem files will remain installed in
/home/vondruch/.gem/ruby/gems/sqlite3-1.3.9 for inspection.
Results logged to
/home/vondruch/.gem/ruby/gems/sqlite3-1.3.9/ext/sqlite3/gem_make.out
There are two answers:
1) yum install rubygem-sqlite3
2) gem instal gem-nice-install && gem install sqlite3
where yum install knows everybody on Red Hat platform, gem-nice-install
knows virtually nobody. And to be honest, even me as a rubyist prefer 1)
since I have *no* compiler on my system and I hope it stays like that.
That's surprising to me cos I'm being told that maven, ruby, nodejs developers
prefer the upstream management tools for various reasons and they are not in favor of
RPMs.
Radek
Vít
_______________________________________________
env-and-stacks mailing list
env-and-stacks(a)lists.fedoraproject.org
https://lists.fedoraproject.org/mailman/listinfo/env-and-stacks
3:21 a.m.
On 09/16/2014 03:26 PM, Radek Vokál wrote:
----- Original Message -----
> Dne 15.9.2014 15:34, Vít Ondruch napsal(a):
> $ gem install sqlite3
> Fetching: sqlite3-1.3.9.gem (100%)
> Building native extensions. This could take a while...
> ERROR: Error installing sqlite3:
> ERROR: Failed to build gem native extension.
>
> /usr/bin/ruby extconf.rb
> mkmf.rb can't find header files for ruby at /usr/share/include/ruby.h
>
>
> Gem files will remain installed in
> /home/vondruch/.gem/ruby/gems/sqlite3-1.3.9 for inspection.
> Results logged to
> /home/vondruch/.gem/ruby/gems/sqlite3-1.3.9/ext/sqlite3/gem_make.out
>
>
> There are two answers:
>
> 1) yum install rubygem-sqlite3
> 2) gem instal gem-nice-install && gem install sqlite3
>
> where yum install knows everybody on Red Hat platform, gem-nice-install
> knows virtually nobody. And to be honest, even me as a rubyist prefer 1)
> since I have *no* compiler on my system and I hope it stays like that.
>
>
That's surprising to me cos I'm being told that maven, ruby, nodejs developers
prefer the upstream management tools for various reasons and they are not in favor of
RPMs.
For the folks that use pure noarch packages, this is true. There are
also various mechanisms for separating the build servers from the
production servers (like wrapping up entire virtual environments as
system packages:
https://hynek.me/articles/python-app-deployment-with-native-packages/)
It's not without it's problems though, and that's what I think
Env&Stacks is in a great position to solve by bringing upstream and
downstream contributors together to come up with a more coherent
solution, rather than each side continuing to work around the other.
Cheers,
Nick.
--
Nick Coghlan
Red Hat Hosted & Shared Services
Software Engineering & Development, Brisbane
HSS Provisioning Architect
8:05 p.m.
On 09/15/2014 11:34 PM, Vít Ondruch wrote:
1) Speaking of Rubyists, you would need to convince them to start
using
"source "https://fedora-rubygems-mirror.org" instead of standard
"source
"https://rubygems.org"" in their Gemfiles. Not sure how you would like
to achieve this. We could patch Bundler somehow to substitute the
original source on Fedora, but I don't want to imagine what source of
problems it would bring.
I'm not suggesting making this mandatory for anything (except perhaps as
part of the package review process at some indeterminate point in the
future). It would simply be about providing developers with the *option*
of pulling Python packages, Ruby gems, etc from a language specific
mirror if they so chose.
For many open source folks, pulling direct from upstream is going to be
simpler and easier. For us, pulling directly from upstream means having
to repeat the licensing review work that the Fedora community would
otherwise handle.
Accordingly, this new service would be primarily for the benefit of
folks in environments like ours where licensing compliance is of greater
concern, by providing a way for us to collaborate on that work *without*
having to muck about with changing package formats.
2) In Fedora, there are packages, which are modified from upstream
versions. For example, rubygem-listen [1] provides universal API to
provide notifications about file changes on file system. Upstream
decided, that they will have hard dependencies on other platform
specific libraries, which supports FS level notifications. We decided to
drop them, since it does not make sense to support libraries which does
nothing on Linux. Sometimes, we are forced to override overly tight
dependencies on other libraries. And now to the question. What would
happened with such gem? What the mirror should provide in such case?
Fedora's version or upstream version of gem? The former is quite
dangerous idea, since it would end up in two different packages of the
same version.
In Python's case, we've deliberately redesigned our versioning system to
cope with the needs of redistributors like Fedora to indicate patched
versions (see http://www.python.org/dev/peps/pep-0440/ for details).
That would allow us to publish the Fedora-specific patched versions on a
PyPI mirror without version conflicts.
In general, however, I would expect the packages on the language mirrors
to be unchanged from their upstream counterparts. If we need to apply
patches, then that's what the distro packaging formats are for.
3) You need additional human/infrastructure resources.
We (as in the software development teams for internal services at Red
Hat) are going to have to do this work for our own internal application
deployments anyway. My proposal is that instead of doing that in such a
way that it only benefits Red Hat associates, we should instead make it
possible to move that work upstream into Fedora where it can benefit the
entire Fedora/RHEL/CentOS ecosystem.
Additionally to the point 3, if you have such resources, I would be
better to invest them into development of proper support of multiple
versions of packages in DNF. That would remove the need for most of the
SCLs and additional repositories in Copr, at least speaking of Ruby
ecosystem.
Better for who? Parallel installs within the system directories aren't
particularly interesting to me. I *like* the software collections model,
because it decouples the language runtime lifecycle from that of the
underlying OS. It doesn't matter whether I'm deploying to Fedora, RHEL
6/7 or CentOS 6/7, the "py27" software collection is the "py27"
software
collection. The same goes for all the other language runtimes, database
runtimes and web servers that are available from softwarecollections.org
or commercially through Red Hat.
That reduced level of coupling is hugely beneficial for application
portability between dev workstations, community environments and
corporate environments.
Nowadays, people are not using ruby193 collection, because they are
in
love with Ruby 1.9.3. They are using the collection since it is bundled
with Ruby on Rails 3.2. And they are using SCLs just because it allows
them to install more versions of RoR on single system, not because they
believe that their worflow is now much easier. But you can achieve it
just fine with rpm -i having the RPMs. It is just not supported by
DNF/YUM, since DNF/YUM understands just "upgrade, don't leave old
version on system".
My interest is in managing the maintenance lifecycle of long lived
production services, not ad hoc parallel installs on developer
workstations. With software collections, I get a steady platform upgrade
cadence independent of the OS lifecycle, as well as a smooth layering of
packages updated at different rates. Roughly speaking, I'll be aiming
for a lifecycle like:
Application layer: our responsibility, upgrade every 2-12 weeks
Software collections layer: upgrade every 1-2 years
Operating system layer: upgrade every 3-5 years
The software collection layer also decouples our applications to some
degree from the OS layer, abstracting away many of the differences
between Fedora & the various versions of RHEL/CentOS that software
collections support.
Cheers,
Nick.
--
Nick Coghlan
Red Hat Hosted & Shared Services
Software Engineering & Development, Brisbane
HSS Provisioning Architect
3514
days inactive
3515
days old
env-and-stacks@lists.fedoraproject.org
6 comments
3 participants
participants (3)
-
Nick Coghlan -
Radek Vokal -
Vít Ondruch