8:39 p.m.
dustymabe reported a new issue against the project: `atomic-wg` that you are following:
``
We have system containers storing things in /var/lib/containers/storage. We have
docker/moby containers storing things in /var/lib/docker. We might move to overlay2 on a
single big root partition in the future, we might not.
We have discussed these topics in a few maililng list threads
[[1](https://lists.projectatomic.io/projectatomic-archives/atomic-devel/2017-April/msg00031.html)]
[[2]()] and also ran out of storage on atomic host before
[[3](https://bugzilla.redhat.com/show_bug.cgi?id=1391725)].
Let's put some thought into this and try to figure out a proper strategy going
forward.
``
To reply, visit the link below or just reply to this email
https://pagure.io/atomic-wg/issue/281
9:13 a.m.
walters added a new comment to an issue you are following:
``
Another thread on list:
https://lists.fedoraproject.org/archives/list/cloud@lists.fedoraproject.o...
``
To reply, visit the link below or just reply to this email
https://pagure.io/atomic-wg/issue/281
11:47 a.m.
dustymabe added a new comment to an issue you are following:
``
Following from that thread the proposal from colin is that we move to one big / with
overlayfs by default for our cloud/vagrant images with all space being taken by the root
filesystme by default . This is something we had talked about in the past, but we have not
formally made this proposal until now. Pending negative feedback in the f26 timeframe,
I'm +1 to this.
The other part of this puzzle is making sure that all container storage *can* be mounted
on another block device/filesystem easily.
So proposal is:
- 1 large filesystem by default
- Ability to easily move container storage to another filesystem on system setup.
``
To reply, visit the link below or just reply to this email
https://pagure.io/atomic-wg/issue/281
12:02 p.m.
walters added a new comment to an issue you are following:
``
I'm going to do some work on this; though we're blocked a bit in testing by
[rawhide
issues](https://lists.fedoraproject.org/archives/list/devel@lists.fedorap....
``
To reply, visit the link below or just reply to this email
https://pagure.io/atomic-wg/issue/281
4:14 p.m.
walters added a new comment to an issue you are following:
``
I tested the `20170615.n.0` FAH iso; it looks like we'll need this:
```
diff --git a/docker.spec b/docker.spec
index b654082..b170151 100644
--- a/docker.spec
+++ b/docker.spec
@@ -569,10 +569,14 @@ echo 'STORAGE_DRIVER=overlay2' >>
%{repo}-storage-setup-workstation
ln -s %{repo}-storage-setup-workstation %{repo}-storage-setup-cloud
# create server override config
ln -s %{repo}-storage-setup-workstation %{repo}-storage-setup-server
-# create atomic override config
+# create atomic override config; see https://pagure.io/atomic-wg/issue/281
+%if 0%{?fedora} >= 27
+ln -s %{repo}-storage-setup-workstation %{repo}-storage-setup-atomichost
+%else
cp %{repo}-storage-setup-server %{repo}-storage-setup-atomichost
echo 'CONTAINER_ROOT_LV_NAME=docker-root-lv' >>
%{repo}-storage-setup-atomichost
echo 'CONTAINER_ROOT_LV_MOUNT_PATH=/var/lib/docker' >>
%{repo}-storage-setup-atomichost
+%endif #atomichost
%endif # custom_storage
popd
```
Basically all the editions become the same thing. Seem sane?
``
To reply, visit the link below or just reply to this email
https://pagure.io/atomic-wg/issue/281
11:36 a.m.
dustymabe added a new comment to an issue you are following:
``
seems sane to me - i asked @dwalsh and @vgoyal to weigh in - also i'll bring this up
in the next meeting we have.
``
To reply, visit the link below or just reply to this email
https://pagure.io/atomic-wg/issue/281
11:55 a.m.
jberkus added a new comment to an issue you are following:
``
@dustymabe seems sensible
``
To reply, visit the link below or just reply to this email
https://pagure.io/atomic-wg/issue/281
12:26 p.m.
vgoyal added a new comment to an issue you are following:
``
So this relies on that fact that atomic users will not use devicemapper graph driver. If
they really need to use devicemapper, they will need to add additional disk. (Which
requires extra effort and planning).
So has this been established that overlayfs is working well for container workload and
most of the users don't have the need to go back to devicemapper.
In F26, we wanted to retain capability to easily go back to devicemapper if things
don't work out well with overlayfs. Given F26 is not GA, I guess we don't even
have that data to make a decision?
``
To reply, visit the link below or just reply to this email
https://pagure.io/atomic-wg/issue/281
12:29 p.m.
walters added a new comment to an issue you are following:
``
@vgoyal: Not quite; the AH storage is the same as Server, we only use up to 15G. For real
baremetal machines, they'll have larger disks, and hence there will be reserved space
in the VG.
For cloud images, it's up to the operator; they can use cloud functionality to expand
the root disk, or they can allocate a separate disk.
``
To reply, visit the link below or just reply to this email
https://pagure.io/atomic-wg/issue/281
12:38 p.m.
vgoyal added a new comment to an issue you are following:
``
I think I am missing something. So rootfs which used to be 3G by default will now be 15G
by default.
Ok, so for cloud images we will continue to have volume group and when image is grown,
additional space will go in volume group and one can carve out space for thin pool. Yep,
that will work.
``
To reply, visit the link below or just reply to this email
https://pagure.io/atomic-wg/issue/281
12:48 p.m.
walters added a new comment to an issue you are following:
``
Thinking of this change as "3G → 15G by default" indeed is a good and simple way
to phrase it. It's not *entirely* accurate since we also need to factor in the
default size of the cloud images and the vagrant box. But it's a good first
approximation.
But also we're no longer using a separate partition by default for AH.
(Incidentally, I have a pending proposal to also use this same partitioning for
"Atomic Workstation", though for somewhat different reasons:
https://pagure.io/pungi-fedora/pull-request/257 )
For reference, here's the first commit:
http://pkgs.fedoraproject.org/cgit/rpms/fedora-productimg-atomic.git/comm...
``
To reply, visit the link below or just reply to this email
https://pagure.io/atomic-wg/issue/281
11:03 a.m.
walters added a new comment to an issue you are following:
``
Docker patch
[committed](http://pkgs.fedoraproject.org/cgit/rpms/docker.git/commit/?id=...
and [built](https://koji.fedoraproject.org/koji/buildinfo?buildID=910722).
``
To reply, visit the link below or just reply to this email
https://pagure.io/atomic-wg/issue/281
12:40 p.m.
dustymabe added a new comment to an issue you are following:
``
Discussed in the atomic working group meeting today:
*changing fedora 27 and beyond to default to overlayfs on the root partition will help us
simplify our storage setup and align better with other fedora variants. This seems like a
reasonable change to make assuming that overlayfs proves stable for the f26 cycle.*
``
To reply, visit the link below or just reply to this email
https://pagure.io/atomic-wg/issue/281
2:22 p.m.
walters added a new comment to an issue you are following:
``
I swear we were defaulting to XFS before, but I may have been confused by the fact that
it's hardcoded in the cloud kickstart.
Any objections to backporting this to f26?
```
diff --git a/installclass_atomic.py b/installclass_atomic.py
index cce1470..7363360 100644
--- a/installclass_atomic.py
+++ b/installclass_atomic.py
@@ -36,6 +36,7 @@ class AtomicInstallClass(FedoraBaseInstallClass):
name = "Atomic Host"
sortPriority = 11000
hidden = False
+ defaultFS = "xfs"
def setDefaultPartitioning(self, storage):
# 3GB is obviously arbitrary, but we have to pick some default.
```
(We could also consider backporting unified storage)
``
To reply, visit the link below or just reply to this email
https://pagure.io/atomic-wg/issue/281
2:33 p.m.
dustymabe added a new comment to an issue you are following:
``
Any objections to backporting this to f26?
I think if we make a good case for why XFS vs ext4 then backporting it to f26 would be a
reasonable thing to do. I'd hate to do so 'just because', though.
``
To reply, visit the link below or just reply to this email
https://pagure.io/atomic-wg/issue/281
2:37 p.m.
walters added a new comment to an issue you are following:
``
We already discussed the rationale as an aside in the larger partitioning discussion
earlier. F27 AH defaults to XFS already; that's what Fedora Server does, which also
matches the downstream RHEL Atomic Host. The dynamic inodes in particular are nice for
overlayfs.
``
To reply, visit the link below or just reply to this email
https://pagure.io/atomic-wg/issue/281
2:38 p.m.
walters added a new comment to an issue you are following:
``
https://koji.fedoraproject.org/koji/taskinfo?taskID=20464792
``
To reply, visit the link below or just reply to this email
https://pagure.io/atomic-wg/issue/281
3:28 p.m.
mattdm added a new comment to an issue you are following:
``
For what it's worth, I was talking with a Red Hat filesystems engineer and he strongly
recommended moving to XFS across all of Fedora. Pretty much the only downside is lack of
filesystem shrinking, and I don't think that's a big issue with AH. Not that we
have to do it because RH says so -- I mean that that's an opinion I have some faith
in.
``
To reply, visit the link below or just reply to this email
https://pagure.io/atomic-wg/issue/281
3:29 p.m.
mattdm added a new comment to an issue you are following:
``
I mean, not that I don't also have faith in you guys. :)
``
To reply, visit the link below or just reply to this email
https://pagure.io/atomic-wg/issue/281
4:28 p.m.
dustymabe added a new comment to an issue you are following:
``
Pretty much the only downside is lack of filesystem shrinking,
yeah, which i think sucks since we had that for ext4 and I actually
used it a decent amount. That's just a personal gripe, not one I've
really had to deal with too much in real life, though.
``
To reply, visit the link below or just reply to this email
https://pagure.io/atomic-wg/issue/281
2:03 p.m.
jberkus added a new comment to an issue you are following:
``
Issue #197 has been folded into this issue. Please do look at that issue, as it has some
specific use-cases we want to solve.
``
To reply, visit the link below or just reply to this email
https://pagure.io/atomic-wg/issue/281
2:27 p.m.
jberkus added a new comment to an issue you are following:
``
Specifically, we want to be sure that the full disk is allocate on AWS instances.
``
To reply, visit the link below or just reply to this email
https://pagure.io/atomic-wg/issue/281
3:40 p.m.
chrismurphy added a new comment to an issue you are following:
``
All the partitioning, sizing, and resizing concerns mentioned in this issue vanish with a
certain other filesystem, which does all resizes (grow, shrink, add and remove devices)
online and atomically and typically in a single command. Whether scripted or user issued,
the commands are shorter, easier to understand, complete faster and are safer.
Gotcha though is I haven't used it with overlayfs. A cursory search yields no hits.
But it seems sane to allow Docker to continue to use overlayfs for the shared page cache
benefit, and even snapshotting (if Docker supports that overlayfs feature now?).
But the main pro is that you can have separate fstrees read-only or read-write mounted,
but they share the same storage pool, without hard barriers between them.
*shrug*
``
To reply, visit the link below or just reply to this email
https://pagure.io/atomic-wg/issue/281
4:14 p.m.
jberkus added a new comment to an issue you are following:
``
"certain other filesystem"? Can you please be less opaque?
``
To reply, visit the link below or just reply to this email
https://pagure.io/atomic-wg/issue/281
4:33 p.m.
chrismurphy added a new comment to an issue you are following:
``
Ahh sorry, I kinda figured realistically there are only three options: ext4, XFS, and
Btrfs, and the only one not mentioned so far is Btrfs.
There's some hits of people using it in AWS contexts, but I have not yet run across
Btrfs + overlayfs. So, I started a thread on linux-btrfs(a)vger.kernel.org to see if
anyone's using containers with Btrfs + overlayfs. Insofar as I'm aware it's
not a pathological combination, I'm just gonna guess to the vast majority it seems
redundant, but they each bring different things to the table.
``
To reply, visit the link below or just reply to this email
https://pagure.io/atomic-wg/issue/281
4:35 p.m.
jberkus added a new comment to an issue you are following:
``
@chrismurphy ZFS is still out there, even if we can't touch it.
``
To reply, visit the link below or just reply to this email
https://pagure.io/atomic-wg/issue/281
4:49 p.m.
chrismurphy added a new comment to an issue you are following:
``
Yeah I wasn't considering anything we don't have in anaconda, but then also
anything not already in the Fedora kernel for some time now.
Plus ZFS lacks fs shrink, and so you can't remove block devices arbitrarily, it also
lacks online replication and seeding. So I even if it weren't for licensing it
wouldn't be the direction I'd go in.
``
To reply, visit the link below or just reply to this email
https://pagure.io/atomic-wg/issue/281
4:52 p.m.
jberkus added a new comment to an issue you are following:
``
Anyway, I refuse to weigh into another XFS vs. BtrFS debate, and starting one here
won't result in us having a storage plan.
``
To reply, visit the link below or just reply to this email
https://pagure.io/atomic-wg/issue/281
5:30 p.m.
chrismurphy added a new comment to an issue you are following:
``
I read a list of problems already with negative arguments, not supplied by me. And
I've presented something that obviates literally all of them. I see it as advice, not
debate.
``
To reply, visit the link below or just reply to this email
https://pagure.io/atomic-wg/issue/281
5:33 p.m.
jberkus added a new comment to an issue you are following:
``
I'm saying that the XFS vs. BtrFS decision was made elsewhere for Fedora, and we
can't change it *in this issue*.
``
To reply, visit the link below or just reply to this email
https://pagure.io/atomic-wg/issue/281
9:11 a.m.
dustymabe added a new comment to an issue you are following:
``
@jberkus
Specifically, we want to be sure that the full disk is allocate on AWS instances.
I just tried this with a [rawhide
qcow](https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-2...
(not on aws but with kvm/qemu and giving it a bigger disk). I don't end up with a
large disk, but just with the 3G and docker using the root filesystem + overlayfs.
```
[root@cloudhost sysconfig]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 40G 0 disk
├─sda1 8:1 0 300M 0 part /boot
└─sda2 8:2 0 39.7G 0 part
└─atomicos-root 253:0 0 3G 0 lvm /sysroot
```
There is some more configuration that needs to be done to make this happen.
``
To reply, visit the link below or just reply to this email
https://pagure.io/atomic-wg/issue/281
5:38 a.m.
walters added a new comment to an issue you are following:
``
Oh right, duh: https://pagure.io/fedora-kickstarts/pull-request/266
``
To reply, visit the link below or just reply to this email
https://pagure.io/atomic-wg/issue/281
2469
days inactive
2513
days old
31 comments
6 participants
participants (6)
-
Chris Murphy -
Colin Walters
-
Dusty Mabe
-
Josh Berkus
-
Matthew Miller
-
Vivek Goyal