9:25 a.m.
Hi,
We have several forms of non-Koji internal builds:
1) COPR (the most visible)
2) http://alt.fedoraproject.org/pub/alt/rawhide-kernel-nodebug/ (this
could be a COPR)
3) rpm-ostree: http://rpm-ostree.cloud.fedoraproject.org/#/
Now, I recently was allocated a new atomic01.qa.fedoraproject.org server
dedicated to rpm-ostree composes, and I'm happy to announce I now have
it composing trees:
fedora-atomic/rawhide/x86_64/server/docker-host =>
175510ab77ef39ec6000db2745c70444d55957e38c5ff28f890237e1f012ea5e
fedora-atomic/rawhide/x86_64/workstation/gnome/core =>
b6f3f3b53ef957c71e8d8bdc36c35519888d4177bf6a582ba37e33416ff84216
But it's on the QA network, and so I can't just serve the content
directly from it. Nor would I want to have the compose server doing
double duty as a static webserver anyways.
I could just rsync this content to the current
rpm-ostree.cloud.fedoraproject.org, but there are a few issues with
that:
1) It's a one off instance in the fedora cloud with no backup/monitoring
2) No redundancy
3) No tuning for static webserving
4) Most critically, no TLS
COPR only relatively recently gained TLS, which is fundamentally
necessary for retrieving arbitrary executable code that runs as root.
But COPR also (from what I understand from <puiterwijk>) is just one
machine.
What do you think about investing in having e.g.
altcdn.fedoraproject.org be a load-balanced static webserver farm
protected by TLS that could be written to by a trusted subset of
non-Koji build/compose tooling?
"altcdn" is the first thing that came to mind offhand for a name, feel
free to pick others =)
We'd need to determine the writing process; could be rsync-over-ssh
access to specific subdirectories gated by ssh key? Something else?
12:35 p.m.
On Sat, 21 Jun 2014 07:25:47 -0700
Colin Walters <walters(a)verbum.org> wrote:
Hi,
We have several forms of non-Koji internal builds:
1) COPR (the most visible)
2) http://alt.fedoraproject.org/pub/alt/rawhide-kernel-nodebug/ (this
could be a COPR)
3) rpm-ostree: http://rpm-ostree.cloud.fedoraproject.org/#/
Now, I recently was allocated a new atomic01.qa.fedoraproject.org
server dedicated to rpm-ostree composes, and I'm happy to announce I
now have it composing trees:
fedora-atomic/rawhide/x86_64/server/docker-host =>
175510ab77ef39ec6000db2745c70444d55957e38c5ff28f890237e1f012ea5e
fedora-atomic/rawhide/x86_64/workstation/gnome/core =>
b6f3f3b53ef957c71e8d8bdc36c35519888d4177bf6a582ba37e33416ff84216
But it's on the QA network, and so I can't just serve the content
directly from it. Nor would I want to have the compose server doing
double duty as a static webserver anyways.
Yeah, we can actually map an external IP into it if required, but I
agree it isn't great to have the build host also serving external
content.
I could just rsync this content to the current
rpm-ostree.cloud.fedoraproject.org, but there are a few issues with
that:
1) It's a one off instance in the fedora cloud with no
backup/monitoring 2) No redundancy
3) No tuning for static webserving
4) Most critically, no TLS
Yep.
COPR only relatively recently gained TLS, which is fundamentally
necessary for retrieving arbitrary executable code that runs as root.
But COPR also (from what I understand from <puiterwijk>) is just one
machine.
Correct.
What do you think about investing in having e.g.
altcdn.fedoraproject.org be a load-balanced static webserver farm
protected by TLS that could be written to by a trusted subset of
non-Koji build/compose tooling?
"altcdn" is the first thing that came to mind offhand for a name, feel
free to pick others =)
We'd need to determine the writing process; could be rsync-over-ssh
access to specific subdirectories gated by ssh key? Something else?
I think this is pretty much already in place with our /pub/alt/ space
(that the kernel nodebug uses above).
This space is on a netapp.
It is mirrored, but only by a few larger mirrors, most don't.
It's served by our 5 download servers in phx2 and 3 on I2 in rdu2.
So, we could sync the ostree stuff there as well... its a large number
of small files? Or can you expand on the size and content it will have
over time and how often updated?
kevin
3:53 p.m.
On Sat, Jun 21, 2014, at 10:35 AM, Kevin Fenzi wrote:
Yeah, we can actually map an external IP into it if required, but I
agree it isn't great to have the build host also serving external
content.
Though, separate from the actual content, it would be nice if it had a
webserver to output live compose status. That would be very low
priority requests.
I think this is pretty much already in place with our /pub/alt/
space
(that the kernel nodebug uses above).
This space is on a netapp.
It is mirrored, but only by a few larger mirrors, most don't.
It's served by our 5 download servers in phx2 and 3 on I2 in rdu2.
Cool. Would it make sense to sync COPR there as well?
So, we could sync the ostree stuff there as well... its a large
number
of small files? Or can you expand on the size and content it will have
over time and how often updated?
Yep, lots of small (almost all immutable) files. At least until
https://bugzilla.gnome.org/show_bug.cgi?id=721799 lands.
Here's some quick stats on the old repository:
# du -shc repo.old
46G repo.old
srv/rpm-ostree <uid=0>
# find repo.old | wc -l
560164
This however includes binary-level history over months for many
packages. The base/core tree is 92 commits, from 2014-02-22 23:27:03
+0000 to 2014-06-04 18:48:16 +0000.
A major source of bloat is having to regenerate the initramfs each time:
https://bugzilla.redhat.com/show_bug.cgi?id=1098457
92 commits * 20 MiB is almost 2G per tree, with 6 trees the initramfs
images account for 25% of the size.
gnome-continuous has an optimization for this, if you look at
https://git.gnome.org/browse/gnome-continuous/log/manifest.json you'll
see the "initramfs-depends": true, we only regenerate it for that.
The *new* repository I just composed has just 2 commits for each tree so
far:
# du -sh repo
3.5G repo
# find repo | wc -l
168744
Also, the way I'd like this to work is to have separate "release" versus
"integration" repositories. The release repository wouldn't have the
full binary history, it gets recomposed
The amount of space used though is flexible because we can also
prune/rebase the history.
The goal for how often updated would come down to the rate of change of
the RPM repositories it tracks; the goal is when an RPM is updated, the
tree regenerates.
11:52 a.m.
On Sat, 21 Jun 2014 13:53:12 -0700
Colin Walters <walters(a)verbum.org> wrote:
On Sat, Jun 21, 2014, at 10:35 AM, Kevin Fenzi wrote:
>
> Yeah, we can actually map an external IP into it if required, but I
> agree it isn't great to have the build host also serving external
> content.
Though, separate from the actual content, it would be nice if it had a
webserver to output live compose status. That would be very low
priority requests.
Well, we do have:
https://apps.fedoraproject.org/releng-dash/
which is a simple page that pulls info from our fedmsg database.
If we get your composes emitting fedmsgs we can query for and watch for
those too.
> I think this is pretty much already in place with our /pub/alt/
> space (that the kernel nodebug uses above).
>
> This space is on a netapp.
> It is mirrored, but only by a few larger mirrors, most don't.
> It's served by our 5 download servers in phx2 and 3 on I2 in rdu2.
Cool. Would it make sense to sync COPR there as well?
Well, we have talked about it... but all of copr is a pretty large
thing and only getting larger over time, so syncing could lead to slow
propagation.
So, we have some dedicated storage coming in for copr. (dell equalogics
unit). Which we can look at making a bit more HA with multiple
frontends, etc.
> So, we could sync the ostree stuff there as well... its a large
> number of small files? Or can you expand on the size and content it
> will have over time and how often updated?
Yep, lots of small (almost all immutable) files. At least until
https://bugzilla.gnome.org/show_bug.cgi?id=721799 lands.
Here's some quick stats on the old repository:
# du -shc repo.old
46G repo.old
srv/rpm-ostree <uid=0>
# find repo.old | wc -l
560164
This however includes binary-level history over months for many
packages. The base/core tree is 92 commits, from 2014-02-22 23:27:03
+0000 to 2014-06-04 18:48:16 +0000.
A major source of bloat is having to regenerate the initramfs each
time: https://bugzilla.redhat.com/show_bug.cgi?id=1098457
92 commits * 20 MiB is almost 2G per tree, with 6 trees the initramfs
images account for 25% of the size.
gnome-continuous has an optimization for this, if you look at
https://git.gnome.org/browse/gnome-continuous/log/manifest.json you'll
see the "initramfs-depends": true, we only regenerate it for that.
The *new* repository I just composed has just 2 commits for each tree
so far:
# du -sh repo
3.5G repo
# find repo | wc -l
168744
Also, the way I'd like this to work is to have separate "release"
versus "integration" repositories. The release repository wouldn't
have the full binary history, it gets recomposed
The amount of space used though is flexible because we can also
prune/rebase the history.
The goal for how often updated would come down to the rate of change
of the RPM repositories it tracks; the goal is when an RPM is
updated, the tree regenerates.
Yeah, my only worry here is that we have too much churn, but it sounds
like there's some mitigation there and you have already been working on
reducing that ;)
kevin
3:01 p.m.
On Thu, Jun 26, 2014, at 12:52 PM, Kevin Fenzi wrote:
Yeah, my only worry here is that we have too much churn, but it sounds
like there's some mitigation there and you have already been working on
reducing that ;)
I'm getting complaints about the download speed from the one-off
rpm-ostree.cloud instance, and there's still the outstanding concerns
about no TLS protection.
Is there anything we can do to move this forward?
11:55 a.m.
On Thu, 17 Jul 2014 16:01:56 -0400
Colin Walters <walters(a)verbum.org> wrote:
On Thu, Jun 26, 2014, at 12:52 PM, Kevin Fenzi wrote:
>
> Yeah, my only worry here is that we have too much churn, but it
> sounds like there's some mitigation there and you have already been
> working on reducing that ;)
I'm getting complaints about the download speed from the one-off
rpm-ostree.cloud instance, and there's still the outstanding concerns
about no TLS protection.
Is there anything we can do to move this forward?
Sorry for the delay here...
I can add you some space right now. How do you normally sync the
content? Manual rsync, or is there scripting?
Ideally we want some setup with no private keys stored on machines...
We just need to come up with a setup that we think is best and
implement it. Hit me on irc if you want to discuss ideas.
kevin
3569
days inactive
3596
days old
infrastructure@lists.fedoraproject.org
5 comments
2 participants
participants (2)
-
Colin Walters -
Kevin Fenzi