[modularity] Modularizing the world fast and iteratively
by Adam Samalik
Starting with a summary: Let's 1) use something like dependency-report
scripts [1] to get coordinated, and 2) make the initial builds against
bootstrap so we get a working thing fast and can iterate.
I've realized that developing the initial set of modules for F27 could be a
bit tricky in the beginning as some things might require many basic
dependencies on top of Platform to run and build. These dependencies, like
autotools or dynamic languages for example, need to get modularized first.
In F26 Boltron we had a 'shared-userspace' and 'shared-build-dependencies'
modules that kind of randomly included many of the shared dependencies of
other modules. I don't think that's necessarily terrible (although I'm sure
contyk would argue it is!), but it's not ideal. We should make modules that
represent a thing, like autotools, python2, python3, ruby, gtk2, gtk3, git,
etc. But making all of these modules on a greenfield without a solid base
could be tricky. How do we help people to know what they should include,
and what's going to be somewhere else?
For this, I have written a set of scripts, and already named them badly
'dependency-report' [1]. We have tested it with FreeIPA maintainers to
generate a dependency report [2] for them, and we have already identified
many modules that FreeIPA need as dependencies. I think we could use this
to help us with the initial design.
This brings me to my second thought... to build what we want, we need to
modularize a lot of other stuff. And when we are ready with the initial
design - that means we have identified all the modules we want to have and
all their dependencies, we need to build them. But, for example, a pretty
commonly needed thing like autotools [3] has pretty crazy build
dependencies [4] including Java, gtk2, gtk3, erlang, X11, python2, python3,
etc. And many of these need autotools, of course. We need to do
bootstraping.
So instead of trying to make everything perfect from the begining, we could
build everything we need against the Bootstrap module - a module that is
used as a buildroot for Host and Platform and contains mostly everything we
need. To compare, there is a report without using bootstrap [5] and with
using bootstrap [6]. This way we'll have a pretty decent set of module very
soon.
The next step would be lookin at the build dependencies of these initial
modules, building them against bootstrap, and rebuilding the initial
modules against these new ones. And then do it for the new ones as well,
until we have everything in place.
Also, with this top-down approach, we'll be flexible with the frequent
changes of the Platform module as it's getting stabilized which also makes
the design of the low-level modules nearly impossible right now.
With this approach, we:
1) get coordinated and make modules without conflicts
2) have a working thing very soon
3) can iterate on expanding the base set over time
If you'd like to participate, please feel free to propose your module on
the F27 content plan [7], I'll make you a repository under the
modularity-modules space [8] which serves as an input to the scripts, and
include your module in there.
Cheers!
Adam
[1] https://github.com/fedora-modularity/dependency-report
[2]
https://github.com/fedora-modularity/dependency-report/tree/master/module...
[3]
https://github.com/fedora-modularity/dependency-report/tree/original-plan...
[4]
https://github.com/fedora-modularity/dependency-report/blob/original-plan...
[5]
https://github.com/fedora-modularity/dependency-report/tree/original-plan...
[6]
https://github.com/fedora-modularity/dependency-report/tree/bootstrap/glo...
[7] https://github.com/fedora-modularity/f27-content-tracking
[8] https://github.com/modularity-modules
--
Adam Šamalík
---------------------------
Software Engineer
Red Hat
4 years, 8 months
OCaml / aarch64 / binutils / coq mess
by Richard W.M. Jones
OCaml 4.05 was added to Fedora 27+ recently. Unfortunately, on
aarch64 only, it interacts badly with a change made in binutils 2.29
which tightens up the rules on relocations for PC-relative addresses.
More details:
https://caml.inria.fr/mantis/view.php?id=7585 detailed discussion
https://github.com/ocaml/ocaml/pull/1268 partial solution
https://sourceware.org/ml/binutils/2017-06/msg00171.html binutils change
This only affects dynamic loading of OCaml modules on aarch64, and in
particular it only affects the Coq theorem prover. This package has a
number of dependencies, so it affects several other packages too. I
believe the complete list is:
- coq Theorem prover
- frama-c Framework for source code analysis of C software
- gappalib-coq Coq support library for gappa
- why Software verification platform
I tried very hard to modify the OCaml aarch64 code generator to make
this work, but it seems as if the OCaml front end compiler doesn't
provide enough information for us to know if a particular symbol will
be used outside a module (it just assumes that all non-private symbols
can be used this way). Or something. Anyway I couldn't work out how
to fix it.
So ... I don't know what to do here, but I guess possible options
include:
(1) Continue having broken dependencies for the affected packages on
aarch64 for a bit and see if upstream come up with anything.
(2) ‘ExcludeArch: aarch64’ on coq and its dependencies. We would of
course file the relavent ExcludeArch bugs. The thing that stops me
doing this is that we're nowhere nearer to having a fix, so it just
punts the problem, plus aarch64 is an important target and Coq is an
important package.
(3) Disable the problematic coq modules. I'm not clear if they are
necessary however, since I've only used Coq as an end user, I've never
delved into how it works.
(4) Back out the binutils change? I guess it was made for good
reasons even though it breaks everything. Also it would be a
downstream change which is bad for Fedora, and we'd probably need to
mass-rebuild everything which is even worse.
(5) Your Plan Here.
Suggestions?
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW
4 years, 8 months
Urgent attention required; ImageMagick update breakage
by Michael Cronenworth
Hi all,
An ImageMagick update (6.9 => 7.0) with an SONAME bump and other breakage has been
pushed to F25 and higher.
First, the update introduces regressions on s390x and ppc64 arches.
- https://bugzilla.redhat.com/show_bug.cgi?id=1484578
- https://bugzilla.redhat.com/show_bug.cgi?id=1484579
Secondly, rebuilds are required for:
autotrace-0.31.1-44.fc26.src.rpm
converseen-0.9.6.2-1.fc27.src.rpm
dmtx-utils-0.7.4-2.fc27.src.rpm
drawtiming-0.7.1-20.fc26.src.rpm
F gtatool-2.2.0-4.fc27.src.rpm
imageinfo-0.05-25.fc26.src.rpm
inkscape-0.92.1-5.20170510bzr15686.fc27.src.rpm
kxstitch-1.2.0-7.fc26.src.rpm
perl-Image-SubImageFind-0.03-11.fc27.src.rpm
pfstools-2.0.6-1.fc27.src.rpm
F php-magickwand-1.0.9.2-9.fc24.src.rpm
F php-pecl-imagick-3.4.3-0.3.RC1.fc26.src.rpm
psiconv-0.9.8-20.fc26.src.rpm
q-7.11-27.fc27.src.rpm
ripright-0.11-3.fc26.src.rpm
rss-glx-0.9.1.p-29.fc26.src.rpm
F rubygem-rmagick-2.16.0-3.fc26.src.rpm
F synfig-1.2.0-5.fc27.src.rpm
(boost issues)
F synfigstudio-1.1.10-0.20160706gitd4e547.fc25.src.rpm
(needs synfig)
vdr-scraper2vdr-1.0.5-2.20170611git254122b.fc27.src.rpm
vips-8.5.6-1.fc27.src.rpm
WindowMaker-0.95.8-1.fc27.src.rpm
F cuneiform
(some c++ blowup)
k3d
The ones with F have possible compile issues.
Thirdly, two updates have been created. I've disabled autopush on them.
- https://bodhi.fedoraproject.org/updates/FEDORA-2017-0a1f3de4eb
- https://bodhi.fedoraproject.org/updates/FEDORA-2017-c276ee400a
It is really late here and I don't have the time to investigate the arch issues right now. I'll investigate when I can.
4 years, 8 months
Re: [DONE] Mass package change (python2- binary package renaming)
by Zbigniew Jędrzejewski-Szmek
Hi,
I pushed the changes and rebuilt the packages today.
In the process I found a significant bug:
for arch-full packages, Provides: %{oldname}%{_isa} = %{version}-%{release}
was generated, but Provides: %{oldname} = %{version}-%{release} is also
necessary. I added that in various packages and rebuilt them, but it's likely
that I missed some. I'll be looking into this tomorrow too, but if you
encounter an issue with some package, feel free to ping me so that I don't
miss it (or rebuild it yourself: the fix is trivial, just copy the Provides
line and remove %{?_isa}).
Some packages failed to build:
xmms2-0.8-46.fc28 No matching package to install: 'ecore-devel'
v8-6.2.91-2.fc28 s390x trouble
wiredtiger-2.9.0-5.fc28 core dump on arm64
python-urwid-1.3.0-11.fc28 tests
python-soaplib-0.8.1-16.fc28 tests
python-pylons-1.0.1-9.fc28 AttributeError: 'HTTPRequestEntityTooLarge' object has no attribute 'exception'
python-pyblock-0.53-15.fc28 error: comparison of constant '0' with boolean expression is always false [-Werror=bool-compare]
python-openstackclient-3.2.0-4.fc28 tests
python-mwlib-0.15.14-10.fc28 tests
python-libasyncns-0.7.1-18.fc28 libasyncns.c:137:13: error: 'ns_t_zxfr' undeclared (first use in this function); did you mean 'ns_t_axfr'?
python-lettuce-0.2.20-6.fc28 tests
python-keyczar-0.71c-10.fc28 tests
python-httpretty-0.8.14-5.20161011git70af1f8.fc28 tests
python-gensim-0.10.0-12.fc28 IndexError: only integers, slices (`:`), ellipsis (`...`), numpy.newaxis (`None`) and integer or boolean arrays are valid indices
python-flask-silk-0.1.2-10.fc28 ImportError: cannot import name Module
python-gear-0.5.9-7.fc28 pkg_resources.ContextualVersionConflict: (extras 0.0.3 (/usr/lib/python2.7/site-packages), Requirement.parse('extras>=1.0.0'), set(['testtools']))
python-djvulibre-0.3.9-11.fc28 error: invalid application of 'sizeof' to incomplete type 'enum ddjvu_status_e'
python-django-mptt-0.8.6-5.fc28 tests
python-django-countries-4.0-4.fc28 AttributeError: 'module' object has no attribute 'BaseSettings'
python-blist-1.3.6-13.fc28 tests on s390x
python-amqpclt-0.6-4.fc28 KeyError: u'No such transport: amqplib. Did you mean amqp?'
mesos-0.23.0-0.4ce5475.fc28.9 configure: error: can not find mvn on your path
libsvm-3.21-6.fc28 %add_maven_depmap JPP.tw.edu.ntu.csie-libsvm.pom tw.edu.ntu.csie/libsvm.jar
libkdtree++-0.7.0-17.fc28 ./kdtree++/kdtree.hpp:1200:28: error: no match for 'operator<<'
gdal-2.1.4-6.fc28 configure: error: C compiler cannot create executables
fts-rest-3.6.3-3.fc28 AttributeError: 'HTTPForbidden' object has no attribute 'exception'
csound-6.03.2-16.fc28 cc1: error: -Wformat-security ignored without -Wformat [-Werror=format-security]
bro-2.4.1-7.fc28 error: ambiguous overload for 'operator=='
cjdns-19.1-7.fc28 tests
atomic-reactor-1.6.23.2-3.fc28 AttributeError: module 'docker' has no attribute 'Client
gis-2.18.11-4.fc28 error: conflicting declaration 'PyObject* sipExportedExceptions__core [3]'
'
I also added missing BR:/usr/bin/2to3 to a bunch of packages which
other failed to build. Those are OK now:
958814 python-tablib-0.11.5-3.fc28 2to3
958774 python-setproctitle-1.1.10-4.fc28 2to3
958707 python-pypng-0.0.18-8.fc28 2to3
958648 python-openopt-0.5625-7.fc28 2to3
958645 python-openoffice-0.1-0.23.20110209.fc28 2to3
958512 python-google-apputils-0.4.2-10.fc28 2to3
958362 python-beautifulsoup4-4.6.0-3.fc28 2to3
958209 brltty-5.5-9.fc28 2to3
vtk and vdsm are still building as I'm writing this.
Zbyszek
4 years, 8 months
[SO-NAME BUMP] Updating jsoncpp on Rawhide and fc27
by Björn 'besser82' Esser
Hello folks,
I'm planning to update jsoncpp on Rawhide and fc27 during the next
days. After the builds have landed, I'll take care of rebuilding all
consumers against the new so name. Since the API / ABI has no publicly
consumed symbols removed, I don't expect any real problems.
Cheers,
Björn
4 years, 8 months
FAILED: BuildError: package ... not in list for tag f28-pending
by Michael Schwendt
It's been several hours since this entirely new package repo has been created,
but the koji build still fails. How long does it take for koji to learn
about this new package?
$ fedpkg build
Building ampache_browser-1.0.0-1.fc28 for rawhide
Created task: 21592583
Task info: https://koji.fedoraproject.org/koji/taskinfo?taskID=21592583
Watching tasks (this may be safely interrupted)...
21592583 build (rawhide, /rpms/ampache_browser:d7627e20f2a306bb1f53d39c182280c3c991fe15): open (buildhw-08.phx2.fedoraproject.org)
21592584 buildSRPMFromSCM (/rpms/ampache_browser:d7627e20f2a306bb1f53d39c182280c3c991fe15): free
21592584 buildSRPMFromSCM (/rpms/ampache_browser:d7627e20f2a306bb1f53d39c182280c3c991fe15): free -> open (buildvm-s390x-13.s390.fedoraproject.org)
21592584 buildSRPMFromSCM (/rpms/ampache_browser:d7627e20f2a306bb1f53d39c182280c3c991fe15): open (buildvm-s390x-13.s390.fedoraproject.org) -> closed
0 free 1 open 1 done 0 failed
21592583 build (rawhide, /rpms/ampache_browser:d7627e20f2a306bb1f53d39c182280c3c991fe15): open (buildhw-08.phx2.fedoraproject.org) -> FAILED: BuildError: package ampache_browser not in list for tag f28-pending
0 free 0 open 1 done 1 failed
21592583 build (rawhide, /rpms/ampache_browser:d7627e20f2a306bb1f53d39c182280c3c991fe15) failed
4 years, 8 months
Module Stream "Expansion"
by Ralph Bean
This is a writeup on a problem we’re facing with modularity, and some ideas on
how to resolve it.
# The "Problem"
Imagine I have an **httpd module**. To simplify things, let’s say that this
module has only one stream: **2.4**. Today, in the modulemd for this module, I
declare build and runtime dependencies like this:
dependencies:
buildrequires:
platform: f26
requires:
platform: f26
This works.
Now, imagine that Fedora 27 comes along. I want the httpd module to be
installable on f27, so I update those streams to point to f27. But now I can’t
ship updates to f26 anymore! Uh oh.
## Just use more streams?
We could just ask packagers to create more streams to deal with this. httpd
wouldn’t have a 2.4 stream. It would need to have a f26-2.4 stream and a
f27-2.4 stream.
But, this gets us exactly back into the place we were **before** we had
arbitrary branching! You need to have as many branches for *every component*
as you have releases of the distro (or, as you have "products"). Noooo!
## Solution: "Input" Modulemd Syntax Changes
We’re going to extend the modulemd syntax to allow specifying multiple
dependencies in an "input" modulemd (the one that packagers modify). When
submitted to the build system, the module-build-service (MBS) will *expand*
these values under the hood, and submit **multiple** module builds to
itself based on the expansion.
Here are some examples of modulemd files (maintained by humans) that might be
submitted to the MBS.
Build on **all** active streams of the platform module.
dependencies:
buildrequires:
platform: []
Build on **only** the f27 and f26 platform streams.
dependencies:
buildrequires:
platform: [ f27, f26 ]
Build on all active streams of platform **except** for the f26 stream.
dependencies:
buildrequires:
platform: [ -f26 ]
The following syntax, which builds on **only** the f26 stream, will still be supported:
dependencies:
buildrequires:
platform: f26
--
Take the following example (described above):
dependencies:
buildrequires:
platform: [ f27, f26 ]
Submitting this modulemd to the MBS would in turn generate **two** module
builds: one of our httpd module against the f27 stream and another against the
f26 stream. Each module build would be associated with its own unique
"flattened" *output* modulemd file that specifies exactly which platform stream
it was built against.
# New Problems
Having **multiple** module builds for a single dist-git commit of a modulemd
file poses new implementation problems.
## NSV uniqueness
Today, we uniquely identify a module build in *a variety of systems* with
**<name>-<stream>-<version>** (NSV) where version is derived from the dist-git
commit timestamp. Here we’ll have an httpd-2.4 module built on f26 and an
http-2.4 module built on f27: two different module builds with the same name,
stream, and version. How can we tell them apart?
We will introduce a new value called the **context** of a built module and
include it in the modulemd metadata that gets carried with the built module.
* For user facing cases (dnf installation) it will generally be hidden. The
old NSV value will still appear. If a user ever needs to surface the value,
client tooling can find it in the modulemd metadata. The additional
identifier will give users access to explicit pointers to the content that is
installed:
* For cloning a machine.
* For comparing two installed hosts.
* For reporting bugs.
* Some build and compose time systems will have to be modified to use the
context as part of a new unique identifier. **NSVC** will be the
**<name>-<stream>-<version>-<context>**. The systems in question are PDC,
koji/brew, pungi, and bodhi.
The context value will be a **hash**, generating as the first step in the build
process (but after expansion). Consider what metadata needs to be hashed: we
think that hashing the whole modulemd is problematic, because the modulemd can
and will be modified after build time.
Therefore, the *context_hash* value will need to be derived from only the stuff
that uniquely identifies the build and runtime context of the module -- name,
stream, version and crucially, its dependencies
## Buildtime/Runtime Dep Correlation
Another problem. We now have input modulemd files for a single stream that can
expand buildtime dependencies to ‘f27’ and ‘f26’, but what about the
runtime dependencies?
Consider the following yaml file:
dependencies:
buildrequires:
platform: [f28, f27, f26]
requires:
platform: [f28, f27, f26]
With our thinking caps on, this should obviously generate three module builds
(one that buildrequires f28 and run requires f28, a second that buildrequires
f27 and run requires f27, and a third for f26). The naive cross-product of all
streams is not valuable; the MBS needs some logic to **correlate** the build
time requirements with the run-time requirements. That logic will contain
assumptions about how this will be used, and it needs to be well-defined.
Let’s try to define that. Consider a more complicated yaml file that depends
on both multiple streams of platform and on multiple hypothetical streams of a
"shared-userspace" module:
dependencies:
buildrequires:
platform: [f28, f27, f26]
shared-userspace: [fancy, nonfancy]
requires:
platform: [f28, f27, f26]
shared-userspace: [fancy, nonfancy]
Here are some rules the MBS should obey:
* If a build-time dep contains an expansion, that dep **does not** have to also
appear as a run-time dep.
* If a build-time dep contains an expansion, and if that dep *also appears* as
a runtime dep, the runtime expansion **must** match the buildtime expansion
exactly.
Now that the streams from build-time and runtime *can be mapped one-to-one*,
the cross-product of the deps is taken to produce a set of modules builds. In
our example here, we get:
1. One build with platform f28 and shared-userspace fancy.
2. One build with platform f27 and shared-userspace fancy.
3. One build with platform f26 and shared-userspace fancy.
4. One build with platform f28 and shared-userspace nonfancy.
5. One build with platform f27 and shared-userspace nonfancy.
6. One build with platform f26 and shared-userspace nonfancy.
We like this.
--
But, a new** corner case** emerges! What if the *nonfancy* branch of
shared-userspace is only compatible with f27 and f28, but not with f26?
Build #6 will always fail - or, if it doesn’t fail it will be unusable.
We thought through a number of ways that corner cases like these could be
handled automatically, and are still working through them. At the moment, we
are *leaning towards* suggesting that a failing corner **requires** a new
stream of the module in question to handle the exclusion. I.e., no convenient
syntax to say "all combinations except for this one".
In this case, you would have two streams of the httpd-2.4 module:
httpd-2.4 and httpd-2.4_f26. The first modulemd file in the
2.4 stream would be nice, like this:
dependencies:
buildrequires:
platform: [f28, f27]
shared-userspace: [fancy, nonfancy]
requires:
platform: [f28, f27]
shared-userspace: [fancy, nonfancy]
The second in the 2.4_f26 stream would exist only to support the desired
corner case:
dependencies:
buildrequires:
platform: [f26]
shared-userspace: [fancy]
requires:
platform: [f26]
shared-userspace: [fancy]
--
We’re still working through more details, especially on this last part (and how
it may affect upgrades), but figured its best to share now. Any help
appreciated!
Yours-
-Ralph
4 years, 8 months
