11:24 a.m.
Hi,
I recently had an idea of faking load paths [0] to satisfy bundler's
loading and help bring our behavior closer to upstream while still
unbundling the libraries.
To automatically test for the usual user's workflow with bundler and a
default gem (only RDoc so far),
I have created small TMT test suite [1]. It pulls the Ruby RPMs from my
copr repo
https://copr.fedorainfracloud.org/coprs/jackorp/ruby-rdoc-default-gem/ .
It is adjustable to fit your own copr if you wish to do your own changes
[2], TMT can also install RPMs from a local directory, but that's going
off-topic.
One of my TODO item here is to validate this against some rbenv Ruby
install.
To the loading itself and my examination.
Default gems seem to skip bundler somewhere and go their own path which
is more similar to regular `Kernel.require` without bundler.
Therefore, just having it on the default load path, where default gems
are expected to exist is enough.
However, regular Ruby loading also means that it expects a certain
layout that it won't search for itself if it can't find a subfile, see
Rake and `rdoc/task` below.
So, if we place `rdoc.rb` into `/usr/share/ruby/` that adjusts the load
path accordingly and then just passes on the require, that's enough if
the code is either
~~~
$ cat foo.rb
require 'rdoc'
p RDoc::VERSION # Demonstrate that RDoc is available
~~~
and run with bundler `bundle exec ruby ./foo.rb`.
or in CLI like this: `bundle exec ruby -rrdoc -e 'p RDoc::VERSION'`.
In both cases, the associated Gemfile is empty. No gems are recored in
the file.
My approach, at this point in time, falls apart with Rake and RDoc using
`require rdoc/task`, as there is no /usr/share/ruby/rdoc/task.rb. I am
unable to track down the exact code path,
but it seems to skip some places inside bundler, even rubygems, that
could otherwise make it possible to work (with a few minor adjustments
of course).
This demostrated a problem, we'd have to fake out the whole structure of
any gem we'd like to unbundle this way.
That, of course, is a possibility.
So the pros / cons of this approach currently seem:
+ we'd unbundle default gems away
+ no more symlinks.
+ closer to upstream behavior for the users
- We'd have to retain the structure and provide each "fake" path to a file.
* Though we are doing something a small bit similar with the symlinks.
- added complexity of writing correct paths into the files on their
correct places.
Alternatively, this is just an academic idea, populate the default load
path with default gems.
I, however, do not know where would the right place for that be, or what
would be the cons of this approach
(apart from potentially breaking expectations set accross the whole Ruby
environment for loadables).
Also to adjust the Kernel.require, but that feels very wrong (also
potentially unsafe, as the code is in C AFAICT, and very divergent from
upstream).
But I have quickly dismissed these as I have high doubts such patches
would ever be accepted upstream and maintainability of such approaches
is quetionable at best.
Additionally, I am not sure what should be the behaviour with the plain
rdoc executable and bundler executed like this: `bundle exec rdoc`
I will try to apply the approach of faking the whole gem directory and
file tree in /usr/share/ruby for RDoc and record my findings.
I am open to discussion, should you want to share your thoughts or ideas
regarding this topic.
Regards,
Jarek Prokop
[0]
https://src.fedoraproject.org/fork/jackorp/rpms/ruby/c/1f660500090ad6d176...
[1] https://gitlab.com/jackorp/test_default_gem_fedora_ruby
[2]
https://gitlab.com/jackorp/test_default_gem_fedora_ruby/-/blob/7658195644...
12:34 p.m.
New subject: Alternative way to package RDoc and other default gems that come
with Ruby
Hi and very much thank you for looking into this issue.
Dne 10. 08. 23 v 18:24 Jarek Prokop napsal(a):
Hi,
I recently had an idea of faking load paths [0] to satisfy bundler's
loading and help bring our behavior closer to upstream while still
unbundling the libraries.
To automatically test for the usual user's workflow with bundler and a
default gem (only RDoc so far),
I have created small TMT test suite [1]. It pulls the Ruby RPMs from
my copr repo
https://copr.fedorainfracloud.org/coprs/jackorp/ruby-rdoc-default-gem/ .
It is adjustable to fit your own copr if you wish to do your own
changes [2], TMT can also install RPMs from a local directory, but
that's going off-topic.
One of my TODO item here is to validate this against some rbenv Ruby
install.
To the loading itself and my examination.
Default gems seem to skip bundler somewhere and go their own path
which is more similar to regular `Kernel.require` without bundler.
Therefore, just having it on the default load path, where default gems
are expected to exist is enough.
However, regular Ruby loading also means that it expects a certain
layout that it won't search for itself if it can't find a subfile, see
Rake and `rdoc/task` below.
So, if we place `rdoc.rb` into `/usr/share/ruby/` that adjusts the
load path accordingly and then just passes on the require, that's
enough if the code is either
~~~
$ cat foo.rb
require 'rdoc'
p RDoc::VERSION # Demonstrate that RDoc is available
~~~
and run with bundler `bundle exec ruby ./foo.rb`.
or in CLI like this: `bundle exec ruby -rrdoc -e 'p RDoc::VERSION'`.
In both cases, the associated Gemfile is empty. No gems are recored in
the file.
My approach, at this point in time, falls apart with Rake and RDoc
using `require rdoc/task`, as there is no
/usr/share/ruby/rdoc/task.rb. I am unable to track down the exact code
path,
but it seems to skip some places inside bundler, even rubygems, that
could otherwise make it possible to work (with a few minor adjustments
of course).
At this place it is necessary to understand what Bundler actually does.
Please note that it is possible to run not only `bundle exec rdoc`, but
it also is possible to run `bundle exec bash` and there are several
interesting environment variables injected by Bundler:
~~~
$ bundle exec bash
[mockbuild@fbbdeb5a66084b7eae24c24f33923368 ~]$ env | sort
BUNDLER_ORIG_BUNDLER_SETUP=BUNDLER_ENVIRONMENT_PRESERVER_INTENTIONALLY_NIL
BUNDLER_ORIG_BUNDLER_VERSION=BUNDLER_ENVIRONMENT_PRESERVER_INTENTIONALLY_NIL
BUNDLER_ORIG_BUNDLE_BIN_PATH=BUNDLER_ENVIRONMENT_PRESERVER_INTENTIONALLY_NIL
BUNDLER_ORIG_BUNDLE_GEMFILE=BUNDLER_ENVIRONMENT_PRESERVER_INTENTIONALLY_NIL
BUNDLER_ORIG_GEM_HOME=BUNDLER_ENVIRONMENT_PRESERVER_INTENTIONALLY_NIL
BUNDLER_ORIG_GEM_PATH=BUNDLER_ENVIRONMENT_PRESERVER_INTENTIONALLY_NIL
BUNDLER_ORIG_MANPATH=BUNDLER_ENVIRONMENT_PRESERVER_INTENTIONALLY_NIL
BUNDLER_ORIG_PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/sbin
BUNDLER_ORIG_RB_USER_INSTALL=BUNDLER_ENVIRONMENT_PRESERVER_INTENTIONALLY_NIL
BUNDLER_ORIG_RUBYLIB=BUNDLER_ENVIRONMENT_PRESERVER_INTENTIONALLY_NIL
BUNDLER_ORIG_RUBYOPT=BUNDLER_ENVIRONMENT_PRESERVER_INTENTIONALLY_NIL
BUNDLER_SETUP=/usr/share/gems/gems/bundler-2.4.10/lib/bundler/setup
BUNDLER_VERSION=2.4.10
BUNDLE_BIN_PATH=/usr/share/gems/gems/bundler-2.4.10/libexec/bundle
BUNDLE_GEMFILE=/builddir/Gemfile
... snip ...
RUBYLIB=/usr/share/gems/gems/bundler-2.4.10/lib
RUBYOPT=-r/usr/share/gems/gems/bundler-2.4.10/lib/bundler/setup
... snip ...
~~~
As you can see, Bundler places itself on Ruby `$LOAD_PATH` via
`$RUBYLIB` and it loads itself via `$RUBYOPT`. Now should also be clear,
that you actually don't have to run `bundle exec`, it is enough to setup
the right env variables.
If I am not mistaken, historically Bundler already at this stage
injected the load paths via `RUBYLIB`. Apparently, it does not doing so
anymore (I think that is probably because they are handling too many
corner cases).
Anyway, what `bundler/setup` does at this stage is that it populates the
Ruby `$LOAD_PATH` (and loads the libraries, unless there is `:require =>
false`, but this is not really important). As a last step, Bundler
actually unloads the customized `require` method and resorts back to the
plain Ruby (!!!) version of require. Not even the RubyGems loader is
active. Nothing. So what is not on load path is not loaded.
These are references to the key steps:
https://github.com/rubygems/rubygems/blob/556349fd863ba586930804140fbcdd6...
https://github.com/rubygems/rubygems/blob/556349fd863ba586930804140fbcdd6...
This demostrated a problem, we'd have to fake out the whole
structure
of any gem we'd like to unbundle this way.
That, of course, is a possibility.
Given the above, I think this is necessity.
So the pros / cons of this approach currently seem:
+ we'd unbundle default gems away
+ no more symlinks.
+ closer to upstream behavior for the users
- We'd have to retain the structure and provide each "fake" path to a
file.
* Though we are doing something a small bit similar with the symlinks.
- added complexity of writing correct paths into the files on their
correct places.
One cons is that still, the `$LOAD_PATH` with this approach will be
different to what would be the default.
Alternatively, this is just an academic idea, populate the default
load path with default gems.
I, however, do not know where would the right place for that be, or
what would be the cons of this approach
(apart from potentially breaking expectations set accross the whole
Ruby environment for loadables).
Not sure about this. We would need to ensure that the paths has lower
priority in `$LOAD_PATH`.
Also to adjust the Kernel.require, but that feels very wrong (also
potentially unsafe, as the code is in C AFAICT, and very divergent
from upstream).
Just given the two links above, this is asking for even more troubles.
But I have quickly dismissed these as I have high doubts such patches
would ever be accepted upstream and maintainability of such approaches
is quetionable at best.
Additionally, I am not sure what should be the behaviour with the
plain rdoc executable and bundler executed like this: `bundle exec rdoc`
From the `env` output you can see, that at minimum, Bundler preserves
some of the original env variables, such as `BUNDLER_ORIG_PATH`, because
it obviously can modify `$PATH` env variable. It can also influence
`MANPATH`:
https://github.com/rubygems/rubygems/blob/556349fd863ba586930804140fbcdd6...
I was not aware of this behavior and it seems to complicate the matter :/
BTW you are using `find` and what not to create the stubs, but it could
be possible to use the already built Ruby and use them to e.g. evaluate
the .gemspec files to get the list of gem files. Or even try to identify
the unbundled default gems.
Vít
I will try to apply the approach of faking the whole gem directory
and
file tree in /usr/share/ruby for RDoc and record my findings.
I am open to discussion, should you want to share your thoughts or
ideas regarding this topic.
Regards,
Jarek Prokop
[0]
https://src.fedoraproject.org/fork/jackorp/rpms/ruby/c/1f660500090ad6d176...
[1] https://gitlab.com/jackorp/test_default_gem_fedora_ruby
[2]
https://gitlab.com/jackorp/test_default_gem_fedora_ruby/-/blob/7658195644...
_______________________________________________
ruby-sig mailing list --ruby-sig(a)lists.fedoraproject.org
To unsubscribe send an email toruby-sig-leave(a)lists.fedoraproject.org
Fedora Code of Conduct:https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines:https://fedoraproject.org/wiki/Mailing_list_guidelines
List
Archives:https://lists.fedoraproject.org/archives/list/ruby-sig@lists.fed...
Do not reply to spam, report it:https://pagure.io/fedora-infrastructure/new_issue
10:51 a.m.
New subject: Alternative way to package RDoc and other default gems that come
with Ruby
JFTR the changes were made into a PR
https://src.fedoraproject.org/rpms/ruby/pull-request/158
On 8/16/23 19:34, Vít Ondruch wrote:
Hi and very much thank you for looking into this issue.
Dne 10. 08. 23 v 18:24 Jarek Prokop napsal(a):
..snip..
>
> So the pros / cons of this approach currently seem:
> + we'd unbundle default gems away
> + no more symlinks.
> + closer to upstream behavior for the users
> - We'd have to retain the structure and provide each "fake" path to a
> file.
> * Though we are doing something a small bit similar with the symlinks.
> - added complexity of writing correct paths into the files on their
> correct places.
>
One cons is that still, the `$LOAD_PATH` with this approach will be
different to what would be the default.
Right, that is surely a deviation. How much that matters is another
question.
> Alternatively, this is just an academic idea, populate the default
> load path with default gems.
> I, however, do not know where would the right place for that be, or
> what would be the cons of this approach
> (apart from potentially breaking expectations set accross the whole
> Ruby environment for loadables).
>
Not sure about this. We would need to ensure that the paths has lower
priority in `$LOAD_PATH`.
With more uncovered facts as to the env modification, I am not even sure
if this would properly survive bundler's extensive ENV modification...
> But I have quickly dismissed these as I have high doubts such patches
> would ever be accepted upstream and maintainability of such
> approaches is quetionable at best.
>
> Additionally, I am not sure what should be the behaviour with the
> plain rdoc executable and bundler executed like this: `bundle exec rdoc`
>
From the `env` output you can see, that at minimum, Bundler preserves
some of the original env variables, such as `BUNDLER_ORIG_PATH`,
because it obviously can modify `$PATH` env variable. It can also
influence `MANPATH`:
https://github.com/rubygems/rubygems/blob/556349fd863ba586930804140fbcdd6...
I was not aware of this behavior and it seems to complicate the matter :/
Huh, I wasn't even aware there are gems distributing proper manpages, or
that rubygems is capable of installing them.
Well, what's more of a concern that I was hitting is the executable
generated by Rubygems, especially this line: `load
Gem.activate_bin_path('rdoc', 'rdoc', version)`, that one is the reason
the gemspec even need to be in the special directory...
BTW you are using `find` and what not to create the stubs, but it
could be possible to use the already built Ruby and use them to e.g.
evaluate the .gemspec files to get the list of gem files. Or even try
to identify the unbundled default gems.
I am not a fan of using the contents of the package for modifying the
package contents (to not rely on it, Ruby changes more than coreutils...).
We can use Ruby for a smarter way later, current way is OK for
demonstration of the concept.
Thinking about the conflict concern (
https://src.fedoraproject.org/rpms/ruby/pull-request/158#comment-154397
) of regular `rubygem-rdoc` and the default gem bundled with Ruby
sources gem `rubygem-rdoc`, where the gemspec would be gone from the
defaults/ directory this if the standalone gem RPM was to replace the
default gem from the Ruby upstream sources
We'd have to have a different RPM naming for these default gems. There
is not just RDoc in this case but also rubygem-json, rubygem-racc and
perhaps a few more that are also present in Fedora as standalone gems
and are defined as default gems as well.
IOW a default gem should not conflict with the same library installed as
a regular gem.
The 3.3.0 preview
https://github.com/ruby/ruby/blob/v3_3_0_preview1/doc/standard_library.rd...
specifies the behavior as "default gems can be updated used by rubygems"
and `gem update` is mentioned, The translation to RPM would be simple
`dnf update rubygem-some_default_gem`, but when I try `gem update` with
rbenv's ruby 3.2.0, and the URI gem (to "fix" the recent cve), I get
something that might not be clear to people, and let's skip for the
moment the fact that people might not even care about fixing such CVEs.
```
$ ruby -ruri -e 'p URI::VERSION'
"0.12.0"
$ gem update uri -N
Updating installed gems
Updating uri
Fetching uri-0.12.2.gem
Successfully installed uri-0.12.2
Gems updated: uri
$ ruby -ruri -e 'p URI::VERSION'
"0.12.2"
$ cat Gemfile
# frozen_string_literal: true
source "https://rubygems.org"
$ bundle exec ruby -ruri -e 'p URI::VERSION'
"0.12.0"
```
For bundler environment this gem update seems to be essentially a no-op.
Which makes sense, it does not replace the default files in the
/usr/share/ruby, that seem to have priority.
We can, instead of moving them away from the /usr/share/ruby, let the
files be on their place and create subpackages made in a way that does
not conflict with stand-alone gems/RPMified gems.
This would kind of retain the upstream behavior, but will generate a new
RPM, let's say names like `rubygem-default-rdoc`, `rubygem-default-json`
etc... It will provide the rubygem(rdoc) rubygem(json),
perhaps rubygem(default-rdoc), but I don't see the use of the latter, I
don't think we expect usage of bundler in any place of the Fedora Ruby
RPM ecosystem. This will allow for parallel installation of both the
default and regular gem. Minus of this approach is that updating
rubygem-rdoc will not fix problems present in rubygem-default-rdoc, but
neither will `gem update rdoc` when we are speaking in terms of the
bundler's environment. If the version will be higher I expect regular
Rubygem's resolver to resolve to the latest, I can validate that claim
later (next week probably, when I get enough sleep on this).
Thinking about it, default gems are a special case on their own.
A set of base libraries that people should be able to rely upon and it
might not necessarily be expected that an update to that gem would in
any way break API by replacing default gem. Therefore default gems are
different from their regular libraries solely based on their location
and the expectation placed on them by upstream.
I think we should keep splitting the default gems out (Why would I need
rubygem-typeprof on a prod system...), but provide them with a file
layout similar to upstream, that would be somewhat of a compromise I can
also see working.
As mentioned, if this sounds like a venue fit for investigation, I'll
inspect it next week.
Regards,
Jarek
Vít
265
days inactive
272
days old
ruby-sig@lists.fedoraproject.org
2 comments
2 participants
participants (2)
-
Jarek Prokop -
Vít Ondruch