Hi, as it was said on the fpc meeting, I'm writing to comment on the section "Some Notes" in Toshio's draft of new Ruby packaging guidelines [1].
[citing the lines from "Some Notes" one by one]
- "Need to move the rubygems library into the per-interpreter directories as it is a non-gem library." As we have said with Mo Morsi, Rubygems library should stay out of Ruby directory structure. Pros: -- Prooves to Rubygems and JRuby upstreams that Rubygems library can be unbundled from Ruby and it makes sense to work on merging the JRuby changes in Rubygems upstream to make one general Rubygems library (that should therefore be implementation-independent). -- As noted by Toshio on the fpc meeting, our system-wide Rubygems are currently only used by MRI Ruby. But as I've said, we need to take steps gradually to convince the upstreams of feasibility of such changes and not to break anything. It is true, that JRuby currently ships with it's own modified (non-compatible) version of Rubygems, but we are working to merge their changes into Rubygems upstream. So yes, currently in F17, there is only the MRI Ruby using the system-wide Rubygems, but the support for JRuby is comming (perhaps F18?). If we are discussing this only from F17 point of view, we still may want to package Rubinius there (it is on our todo list, although not that high as JRuby) and Rubinius _would_ be able to use the system-wide Rubygems - that is another reason why Rubygems should stay where they are even in Fedora 17.
Cons: -- Toshio says that he doesn't like special-casing and Rubygems should ship inside each of the Ruby implementations, until we make all the changes to have system-wide Rubygems, that work with all Ruby implementations (I hope I am not misinterpreting you, if yes, then please correct me). I'd like to add, that it is very hard to convince Rubygems upstream to make any changes that we need and we must have something to show them it's worth the work to merge the JRuby changes in. -- Any others?
- "Need to rebuild ruby and rubygems package to use the new location" -- I think that depends on whether the Rubygems library is moved, so let's put it aside for the moment and discuss it afterwards.
- "Need to rebuild rubygem packages to use the new interpreter-neutral rubygem library location." -- Same as above.
- "Should there be more information about jruby, etc in the introductory portion (naming and" [unfinished] -- I think it would be good to postpone this until we have better integration with the other Ruby implementations. So far, no one has requested any JRuby specific packages or anything connected with JRuby, so I would leave that for a separate discussion/fpc ticket.
- "gem2rpm and rpmdev-newrpmspec can be updated with the new template" -- Yes, we will do that once the guidelines are complete. I hope that the gem2rpm part of guidelines will then be added back.
Thanks for reading this through :)
Thanks for sending this to get the ball rolling!
On Thu, Mar 29, 2012 at 06:53:21AM -0400, Bohuslav Kabrda wrote:
Hi, as it was said on the fpc meeting, I'm writing to comment on the section "Some Notes" in Toshio's draft of new Ruby packaging guidelines [1].
[citing the lines from "Some Notes" one by one]
- "Need to move the rubygems library into the per-interpreter directories as it is a non-gem library."
As we have said with Mo Morsi, Rubygems library should stay out of Ruby directory structure. Pros: -- Prooves to Rubygems and JRuby upstreams that Rubygems library can be unbundled from Ruby and it makes sense to work on merging the JRuby changes in Rubygems upstream to make one general Rubygems library (that should therefore be implementation-independent). -- As noted by Toshio on the fpc meeting, our system-wide Rubygems are currently only used by MRI Ruby. But as I've said, we need to take steps gradually to convince the upstreams of feasibility of such changes and not to break anything. It is true, that JRuby currently ships with it's own modified (non-compatible) version of Rubygems, but we are working to merge their changes into Rubygems upstream. So yes, currently in F17, there is only the MRI Ruby using the system-wide Rubygems, but the support for JRuby is comming (perhaps F18?). If we are discussing this only from F17 point of view, we still may want to package Rubinius there (it is on our todo list, although not that high as JRuby) and Rubinius _would_ be able to use the system-wide Rubygems - that is another reason why Rubygems should stay where they are even in Fedora 17.
Cons: -- Toshio says that he doesn't like special-casing and Rubygems should ship inside each of the Ruby implementations, until we make all the changes to have system-wide Rubygems, that work with all Ruby implementations (I hope I am not misinterpreting you, if yes, then please correct me). I'd like to add, that it is very hard to convince Rubygems upstream to make any changes that we need and we must have something to show them it's worth the work to merge the JRuby changes in. -- Any others?
So, this is pretty close to my reasoning. I don't like special-casing things without a good reason. It makes explanations longer and trips people up when they look at a particular well-known package as an example of how to do something. (And meets resistance and causes extra work if that special-case is ever removed ;-). That said, I'm not overly attached to this -- if there is a *good* reason for special casing, that's fine with me.
A good reason for special casing should:
* apply to the rubygem library but not to other non-gem libraries (this is what makes it special) * help us achieve some goal (the goal in the times I can think of have been to get a particular package into Fedora but this could be a different goal) * differ as little from standard packaging guidelines and other, similar special cases as possible (to minimize the amount that packagers, reviewers, and sysadmins have to learn to deal with this specific special case)
I think we have pieces of this but I'm not quite to the point where I think we have all of these reasons covered.
- Proving to rubygem upstream that we can have common code. This seems to be one goal that we're trying to achieve. - Working towards a single code-base for ruby and jruby (and rubinius). Also seems to be a goal to achieve.
What we're not getting at with these two reasons is why the rubygem library is special compared to other non-gem libraries and whether a different way exists to do this that deviates less from the standard guidelines.
For the first question, the idea that the rubygem module is vastly more popular than other non-gem modules was advanced as a possibility. In talking with vondruch on IRC, he pushed this idea even further to encompass the idea that non-gem ruby libraries are a legacy format. rubygems should be considered the format for ruby libraries. A few non-gem libraries will exist because no one exists upstream to port them over but by and large they have very little importance. What sets the rubygem library itself apart here is the bootstrapping issue. The rubygem library can't be packaged as a gem because it is needed in order to load libraries packaged as rubygems.
If we decide that that's the basis of why the rubygem library is special, that leaves us with the last question of whether we can package this better. I can think of two alternatives that seem more standard and still satisfy the goals. I'll present them so we can either come up with new goals that will make them not work or we can see that the special-case in mind makes more sense:
1) As of ruby-1.9, the rubygem module is shipped as part of the ruby standard library which is an acknowledgement by upstream that rubygem is needed to install almost any other ruby library. We should simply patch the rubygem library in the ruby standard library with the changes to make it compatible across interpreters. The various ruby interpreters have to fork the standard library when they create their interpreters so the more we work with the standard library versions to make the standard library version of modules not need patches per interpreter, the more everyone benefits. We can do this work without pushing a third-party module for rubygems or installing into a separate location.
2) Ship a rubygem package that has separate subpackages for each interpreter that places the code into the vendorlib for each interpreter. Apply patches to this as needed to get to the place where we have common code running on all interpreters. This seems pretty close to what we're doing now -- it's just that we don't need to install into a separate location that all the ruby interpreters need to know about.
- "Need to rebuild ruby and rubygems package to use the new location"
-- I think that depends on whether the Rubygems library is moved, so let's put it aside for the moment and discuss it afterwards.
- "Need to rebuild rubygem packages to use the new interpreter-neutral rubygem library location."
-- Same as above.
- "Should there be more information about jruby, etc in the introductory portion (naming and" [unfinished]
-- I think it would be good to postpone this until we have better integration with the other Ruby implementations. So far, no one has requested any JRuby specific packages or anything connected with JRuby, so I would leave that for a separate discussion/fpc ticket.
- "gem2rpm and rpmdev-newrpmspec can be updated with the new template"
-- Yes, we will do that once the guidelines are complete. I hope that the gem2rpm part of guidelines will then be added back.
yep, all of these sound like reasonable answers.
-Toshio
Hi Toshio, thanks for your reply.
----- Original Message -----
Thanks for sending this to get the ball rolling!
On Thu, Mar 29, 2012 at 06:53:21AM -0400, Bohuslav Kabrda wrote:
Hi, as it was said on the fpc meeting, I'm writing to comment on the section "Some Notes" in Toshio's draft of new Ruby packaging guidelines [1].
[citing the lines from "Some Notes" one by one]
- "Need to move the rubygems library into the per-interpreter
directories as it is a non-gem library." As we have said with Mo Morsi, Rubygems library should stay out of Ruby directory structure. Pros: -- Prooves to Rubygems and JRuby upstreams that Rubygems library can be unbundled from Ruby and it makes sense to work on merging the JRuby changes in Rubygems upstream to make one general Rubygems library (that should therefore be implementation-independent). -- As noted by Toshio on the fpc meeting, our system-wide Rubygems are currently only used by MRI Ruby. But as I've said, we need to take steps gradually to convince the upstreams of feasibility of such changes and not to break anything. It is true, that JRuby currently ships with it's own modified (non-compatible) version of Rubygems, but we are working to merge their changes into Rubygems upstream. So yes, currently in F17, there is only the MRI Ruby using the system-wide Rubygems, but the support for JRuby is comming (perhaps F18?). If we are discussing this only from F17 point of view, we still may want to package Rubinius there (it is on our todo list, although not that high as JRuby) and Rubinius _would_ be able to use the system-wide Rubygems - that is another reason why Rubygems should stay where they are even in Fedora 17.
Cons: -- Toshio says that he doesn't like special-casing and Rubygems should ship inside each of the Ruby implementations, until we make all the changes to have system-wide Rubygems, that work with all Ruby implementations (I hope I am not misinterpreting you, if yes, then please correct me). I'd like to add, that it is very hard to convince Rubygems upstream to make any changes that we need and we must have something to show them it's worth the work to merge the JRuby changes in. -- Any others?
So, this is pretty close to my reasoning. I don't like special-casing things without a good reason. It makes explanations longer and trips people up when they look at a particular well-known package as an example of how to do something. (And meets resistance and causes extra work if that special-case is ever removed ;-). That said, I'm not overly attached to this -- if there is a *good* reason for special casing, that's fine with me.
A good reason for special casing should:
- apply to the rubygem library but not to other non-gem libraries
(this is what makes it special)
- help us achieve some goal (the goal in the times I can think of
have been to get a particular package into Fedora but this could be a different goal)
- differ as little from standard packaging guidelines and other,
similar special cases as possible (to minimize the amount that packagers, reviewers, and sysadmins have to learn to deal with this specific special case)
I think we have pieces of this but I'm not quite to the point where I think we have all of these reasons covered.
- Proving to rubygem upstream that we can have common code. This
seems to be one goal that we're trying to achieve.
- Working towards a single code-base for ruby and jruby (and
rubinius). Also seems to be a goal to achieve.
What we're not getting at with these two reasons is why the rubygem library is special compared to other non-gem libraries and whether a different way exists to do this that deviates less from the standard guidelines.
For the first question, the idea that the rubygem module is vastly more popular than other non-gem modules was advanced as a possibility. In talking with vondruch on IRC, he pushed this idea even further to encompass the idea that non-gem ruby libraries are a legacy format. rubygems should be considered the format for ruby libraries. A few non-gem libraries will exist because no one exists upstream to port them over but by and large they have very little importance. What sets the rubygem library itself apart here is the bootstrapping issue. The rubygem library can't be packaged as a gem because it is needed in order to load libraries packaged as rubygems.
If we decide that that's the basis of why the rubygem library is special, that leaves us with the last question of whether we can package this better. I can think of two alternatives that seem more standard and still satisfy the goals. I'll present them so we can either come up with new goals that will make them not work or we can see that the special-case in mind makes more sense:
- As of ruby-1.9, the rubygem module is shipped as part of the ruby standard library which is an acknowledgement by upstream that rubygem is needed to install almost any other ruby library. We should simply patch the rubygem library in the ruby standard library with the changes to make it compatible across interpreters.
That's not as easy as it sounds. In terms of sustainability, this may become a nightmare to support, because we are talking about a pretty decent amount of patches. That is why we are trying to make Rubygems upstream merge the JRuby modifications.
The various ruby interpreters have to fork the standard library when they create their interpreters so the more we work with the standard library versions to make the standard library version of modules not need patches per interpreter, the more everyone benefits. We can do this work without pushing a third-party module for rubygems or installing into a separate location.
I see your point here, but the standard library itself is not completely implemented in Ruby (in MRI Ruby, it has C extensions, in JRuby, it has jars). So we can put some effort into this, but it's not completely achievable, just considering the per-interpreter extensions. The Rubygems library is however 100 % Ruby and therefore can be adapted to work with all the interpreters (I took the time to go through all the JRuby modifications and it's obvious, that these can be merged with some amount of work). That is one of the reasons why it makes sense to take Rubygems out and make them interpreter-independent.
- Ship a rubygem package that has separate subpackages for each interpreter that places the code into the vendorlib for each interpreter. Apply patches to this as needed to get to the place where we have common code running on all interpreters. This seems pretty close to what we're doing now -- it's just that we don't need to install into a separate location that all the ruby interpreters need to know about.
Interpreter-dependent Rubygems were the case up to F16, but this whole idea is about not doing it this way. As I have already mentioned, applying interpreter-specific patches may become very hard to maintain (should we have one Rubygems package with subpackages), so I wouldn't see that as an option.
Really, having one Rubygems interpreter-independent package is the best direction that we can take this.
- "Need to rebuild ruby and rubygems package to use the new
location" -- I think that depends on whether the Rubygems library is moved, so let's put it aside for the moment and discuss it afterwards.
- "Need to rebuild rubygem packages to use the new
interpreter-neutral rubygem library location." -- Same as above.
- "Should there be more information about jruby, etc in the
introductory portion (naming and" [unfinished] -- I think it would be good to postpone this until we have better integration with the other Ruby implementations. So far, no one has requested any JRuby specific packages or anything connected with JRuby, so I would leave that for a separate discussion/fpc ticket.
- "gem2rpm and rpmdev-newrpmspec can be updated with the new
template" -- Yes, we will do that once the guidelines are complete. I hope that the gem2rpm part of guidelines will then be added back.
yep, all of these sound like reasonable answers.
-Toshio
-- packaging mailing list packaging@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/packaging
On Tue, Apr 3, 2012 at 10:45 PM, Bohuslav Kabrda bkabrda@redhat.com wrote:
Hi Toshio, ----- Original Message -----
- As of ruby-1.9, the rubygem module is shipped as part of the ruby
standard library which is an acknowledgement by upstream that rubygem is needed to install almost any other ruby library. We should simply patch the rubygem library in the ruby standard library with the changes to make it compatible across interpreters.
That's not as easy as it sounds. In terms of sustainability, this may become a nightmare to support, because we are talking about a pretty decent amount of patches. That is why we are trying to make Rubygems upstream merge the JRuby modifications.
We'll have those patches either way. Either the patches are applied against the ruby-rubygem package or against the ruby (ruby-libs) package. We want to have the patches merged upstream in either case.
The various ruby interpreters have to fork the standard library when they create their interpreters so the more we work with the standard library versions to make the standard library version of modules not need patches per interpreter, the more everyone benefits. We can do this work without pushing a third-party module for rubygems or installing into a separate location.
I see your point here, but the standard library itself is not completely implemented in Ruby (in MRI Ruby, it has C extensions, in JRuby, it has jars). So we can put some effort into this, but it's not completely achievable, just considering the per-interpreter extensions. The Rubygems library is however 100 % Ruby and therefore can be adapted to work with all the interpreters (I took the time to go through all the JRuby modifications and it's obvious, that these can be merged with some amount of work). That is one of the reasons why it makes sense to take Rubygems out and make them interpreter-independent.
I'm not suggesting that all of the std library be made interpreter independent. I'm just saying that making sure that the rubygem patches for interpreter independence that we're making get pushed into the stdlib version of rubygem.
- Ship a rubygem package that has separate subpackages for each
interpreter that places the code into the vendorlib for each interpreter. Apply patches to this as needed to get to the place where we have common code running on all interpreters. This seems pretty close to what we're doing now -- it's just that we don't need to install into a separate location that all the ruby interpreters need to know about.
Interpreter-dependent Rubygems were the case up to F16, but this whole idea is about not doing it this way. As I have already mentioned, applying interpreter-specific patches may become very hard to maintain (should we have one Rubygems package with subpackages), so I wouldn't see that as an option.
Really, having one Rubygems interpreter-independent package is the best direction that we can take this.
So you're not quite getting what I'm saying, I think.
The special case I'm seeing does not have to do with whether we work on patching rubygems to be interpreter independent. It doesn't have to do with us getting those patches integrated with upstream. I think we both agree that both working towards interpreter independence and having any patches we generate merged upstream are goals of this (they fall under criteria #1 for a good special case).
What both of my proposals do, essentially, is simply remove the special-case for where the code lives on the filesystem. They're saying "all non-gem libraries live in the stdlib of each interpreter if shipped with the interpreter or in the vendorlib if shipped as third-party code." As opposed to the proposal that the rubygem module is special and therefore has the additional clause "as a specialcase, the rubygem module lives in %{_datadir}/rubygem instead of either of those places". The code inside of any of these locations (with our choice of whether to patch upstream for interpreter independence or not) would be the same in all cases.
So what I'm trying to find out is why there's the perceived need to special case the filesystem location of the rubygem module... not why there's a need to patch the code. (and then whether that perceived need translated into a real need :-)
-Toshio
packaging@lists.fedoraproject.org
-
Bohuslav Kabrda -
Toshio Kuratomi