> I agree switching from ext4 to XFS is likely not worthwhile. Whether Server WG goes with ext4 or XFS on LVM, it's worthwhile for Workstation WG to mimic it merely due to simplicity because then we don't need separate installers or composes.

The elephant in the room here seems to be LVM backing, I don't see anyone discussing that. Do desktop and server want to keep LVM backing by default if they don't go with btrfs? Do desktop and server have *differing* perspectives there? (Do we want to re-run the Fedora 18 tape where we switch to no LVM backing by default and then have to go back to LVM by default for some reason I've forgotten?)

by default we put ext4 on LVM

On Feb 26, 2014, at 12:53 PM, Michael Cronenworth <mike(a)cchtml.com&gt; wrote: > Chris Murphy wrote: >> by default we put ext4 on LVM > The tool works in this use-case unless something has broken it recently. It can be done, the convert tool should work, and Btrfs should work on any device mapper instance. However… In the context of the default ext4+LVM layout the conversion still means separate /boot, /, and /home file systems. A major benefit of the Btrfs layout is these are subvolumes, which instead draw space from one volume pool. And that's lost with a conversion strategy. It also means going from a Fedora "standard" layout to a distinctly non-standard one because our Btrfs layout isn't like the result you'd get from what you're talking about. Chris Murphy

On 2/27/14, 10:53 PM, James Wilson Harshaw IV wrote: > On 02/27/2014 11:20 PM, Chris Murphy wrote: >> On Feb 26, 2014, at 12:53 PM, Michael Cronenworth <mike(a)cchtml.com&gt; wrote: >> >>> Chris Murphy wrote: >>>> by default we put ext4 on LVM >>> The tool works in this use-case unless something has broken it recently. >> It can be done, the convert tool should work, and Btrfs should work on any device mapper instance. However… >> >> In the context of the default ext4+LVM layout the conversion still means separate /boot, /, and /home file systems. A major benefit of the Btrfs layout is these are subvolumes, which instead draw space from one volume pool. And that's lost with a conversion strategy. It also means going from a Fedora "standard" layout to a distinctly non-standard one because our Btrfs layout isn't like the result you'd get from what you're talking about. >> >> >> Chris Murphy > A question I have, forgive me if it seems stupid, is why not just > change the standard? From what I have seen is that btrfs is superior > to ext4 as even Theodore T'so (creator of ext4) said btrfs "offers > improvements in scalability, reliability, and ease of management". Yes, that is a thing on Wikipedia isn't it. ;) It's referring to a discussion held 5 years ago. > I > feel that eventually other filesystems may show to be better than > ext4 and we probably will need to change the standards for the sake > of quality. My fear is that we will be sacrificing quality for not > having too change much. I'll see your 5-year-old quote from the ext4 maintainer, and raise you a 1-day-old quote from a btrfs maintainer: "Just popping in here to say that btrfs is not ready to be default in Fedora yet. Optional is fine but not default. Thanks, Josef." If and when btrfs becomes a superior solution, I'm sure Fedora will be eager to ship it. -Eric

On Feb 27, 2014, at 10:56 PM, James Wilson Harshaw IV <jwharshaw(a)gmail.com&gt; wrote: > Yet what was the main point that it wasn't ready yet? It was easy to miss if you don't know the name Josef Bacik, but he's one of the upstream Btrfs maintainers. He said here in this thread yesterday that it's not ready as a default. That's basically the end of the discussion of whether to make it the default. > My point is we should choose the best solution, even if it takes a little more work to get it up and running. It's not the best solution as a default when the people working on it full time say it's not ready to be the default. > I want to know what it will take to make sure btrfs is good to go as default and then see if the end result will out weigh the effort put in. What it will take: it needs more time in the oven. Chris Murphy

On Feb 27, 2014, at 11:07 PM, James Wilson Harshaw IV <jwharshaw(a)gmail.com&gt; wrote: > I apologize, I guess I did not get the whole background out of it. > > What filesystems are we considering? It's XFS vs ext4 and Server WG has agreed on XFS on LVM.

> Basically, what I'm saying is that if Desktop would be OK with using > xfs-on-LVM as default with all choices demoted to custom partitioning > (no dropdown), as Server has currently agreed on, that'd be great. Or if > we could otherwise achieve agreement on something. I'll bring it up. I believe the momentum on using ext4-on-LVM is that it's the existing default, it's a known quantity, and we have the most experience with it as a project.

On Thu, 2014-02-27 at 23:16 -0700, Chris Murphy wrote:

> It's XFS vs ext4 and Server WG has agreed on XFS on LVM. As a server WG member I voted +1 on XFS as I have no particular objection to XFS as a filesystem, but I do think it seems a bit sub-optimal for us to wind up with server and desktop having defaults that are very similar but slightly different, for no apparently great reason.

Is xfs really so much better for servers, and ext4 so much better for desktops, that it's worth the extra development/maintenance to allow Desktop to use ext4 and Server to use xfs?

Basically, what I'm saying is that if Desktop would be OK with using xfs-on-LVM as default with all choices demoted to custom partitioning (no dropdown), as Server has currently agreed on, that'd be great.

Right now we seem to be sleepwalking into a situation where server and desktop diverge but no-one particularly *wants* that, which seems a bit off.

Once upon a time, Chris Murphy&lt;lists(a)colorremedies.com&gt; said: > >There are good reasons to use XFS by default for Server. Are they listed somewhere?

As you say they are 'plain' filesystems. Though I now regret not sending my small datapoint in before the Server WG decision. That's that a while ago, after using XFS for a long time we started putting new filesystems onto ext4 and in the past month we moved probably our largest remaining dataset (1.1TB) from XFS to ext4, the main reason has been flexibility with resizing. Particularly the XFS 32bit inode ceiling, (inode64 not working well with NFS).

On 28 February 2014 20:45, Adam Williamson <awilliam(a)redhat.com&gt; wrote: > On Thu, 2014-02-27 at 23:16 -0700, Chris Murphy wrote: >> On Feb 27, 2014, at 11:07 PM, James Wilson Harshaw IV <jwharshaw(a)gmail.com&gt; wrote: >> >> > I apologize, I guess I did not get the whole background out of it. >> > >> > What filesystems are we considering? >> >> It's XFS vs ext4 and Server WG has agreed on XFS on LVM. > > As a server WG member I voted +1 on XFS as I have no particular > objection to XFS as a filesystem, but I do think it seems a bit > sub-optimal for us to wind up with server and desktop having defaults > that are very similar but slightly different, for no apparently great > reason. > > ext4 and xfs are basically what I refer to as 'plain' filesystems (i.e. > not all souped-up btrfs/zfs stuff), they're stable, mature, and > generally good-enough for just about all cases. Is xfs really so much > better for servers, and ext4 so much better for desktops, that it's > worth the extra development/maintenance to allow Desktop to use ext4 and > Server to use xfs? > > Basically, what I'm saying is that if Desktop would be OK with using > xfs-on-LVM as default with all choices demoted to custom partitioning > (no dropdown), as Server has currently agreed on, that'd be great. Or if > we could otherwise achieve agreement on something. > As you say they are 'plain' filesystems. Though I now regret not sending my small datapoint in before the Server WG decision. That's that a while ago, after using XFS for a long time we started putting new filesystems onto ext4 and in the past month we moved probably our largest remaining dataset (1.1TB) from XFS to ext4, the main reason has been flexibility with resizing. Particularly the XFS 32bit inode ceiling, (inode64 not working well with NFS). 1TB doesn't sound very big. These are imaging datasets in a research environment, files going from small text to images at tens of MB (some bigger, but not the dominant type). Projects usually get their own FS (for a variety of reasons including backup, audit and budgeting reasons). And often it's not known how large a project will eventually be, so filesystems get extended as appropriate. With XFS we have to take care to avoid the 32bit inode ceiling, and most recently found a filesystem that refused to take any more files for some other reason, even after creating a new clean copy. We didn't get to the bottom of that, and moved the data to ext4. Which is not to say XFS is a bad decision, there's plenty of people using it for other tasks, but I looked through the Server WG meeting and couldn't see mention of the for/against arguments. If my ramble above demonstrates anything it's not really about XFS, it's that server admins have reasons for choosing an FS and the scope to look at and change to alternatives. Default FS on the Server is not actually a massive impact, LVM is a different decision and makes sense there. LVM on a workstation though, well, you can make it the default, but a couple of releases ago I started turning it off and will continue doing so. It adds an extra level of complication to management which doesn't gain you anything on a workstation.

On Mar 1, 2014, at 1:19 PM, Jon <jdisnard(a)gmail.com&gt; wrote: > The inability to shrink or reduce XFS is rather disappointing. I've > seen a few sarcastic remarks along the lines of (paraphrased): why > would anyone ever want to shrink a volume? In the context of server, and default installs, why is a valid question. > People do shrink volumes, and this lack of flexibility is an important > consideration I feel was ignored in the Server WG decision. What is the use case for volume shrinking in a server context? Dual boot is a total edge case for servers.

The servers were rented with a Fedora produced default/automatic/guided partitioning layout? If not, your example is out of scope. We are only talking about this context specifically, not arbitrary examples for shrinking a file system. The Fedora automatic/guided partition layout is a rootfs of 50GB, and any significant additional space goes to a separate /home. So you're saying you'd shrink a 50GB rootfs for encrypted data, rather than blow away the /home LV, make a new LV, encrypt it, then format it?

On Mar 2, 2014, at 9:35 PM, Nathanael Noblet <nathanael(a)gnat.ca&gt; wrote: > On 03/01/2014 04:57 PM, Chris Murphy wrote: >> The servers were rented with a Fedora produced default/automatic/guided partitioning layout? If not, your example is out of scope. We are only talking about this context specifically, not arbitrary examples for shrinking a file system. The Fedora automatic/guided partition layout is a rootfs of 50GB, and any significant additional space goes to a separate /home. So you're saying you'd shrink a 50GB rootfs for encrypted data, rather than blow away the /home LV, make a new LV, encrypt it, then format it? > They were CentOS 6 machines. So perhaps the defaults are different however this is something that happened to me and not being able to shrink a fs would have been problematic / costly for me. Granted the default there was /boot / and swap so I had a 900G / and nothing else thus the shrinking of the / fs. So I suppose that if the servers were fedora and had a /home LV this particular situation wouldn't have been an issue. > > I just wanted to point out that shrinking a partition is a valid use case is all. In our current default fedora layout I could still accomplish what I needed. But shrinking a fs is a valid use case… Fair enough, and I'm not suggesting shrink is invalid for that matter. I merely want to understand the actual requirement because there may be better ways to address it. > Given the XFS shrinking issue it might even be nice to not allocate ALL storage, create /, swap and /home without taking up all storage and then let people enlarge what they need… It's an interesting suggestion. But does this really apply to the target audience of users who are a.) using a GUI installer, and b.) choosing to use an automatic/guided partitioning layout? Is that sort of user likely to jump into a resize operation from the command line post-install? Why wouldn't they just use Manual Partitioning? What you suggest might seem plausible for Server. But I don't think that's a good idea for Workstation, to burden the user with an incomplete partition layout that (silently) proposes they complete or customize it post-install.

> >> People do shrink volumes, and this lack of flexibility is an important >> consideration I feel was ignored in the Server WG decision. > > What is the use case for volume shrinking in a server context? Dual boot is a total edge case for servers. I shrink ext4 filesystems on servers pretty frequently. Most recently because: *) Received bad information from an end user which required changing several LVs/FSs. *) An "oops" situation where a filesystem was incorrectly increased by an extra order of magnitude *) Unexpected (e.g. emergency) growth of an application which required increasing a filesystem and shrinking another (lesser) used filesystem. Yes in all three aforementioned cases we had to unmount the ext4 filesystem in order to shrink it, however, we would _not_ have been able to do this with xfs. On a semi related note: I grow/shrink JFS2 filesystems (on AIX) all the time. It would be great if ext4 had online shrink. -Jacob -- devel mailing list devel(a)lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

On Mar 1, 2014, at 1:19 PM, Jon <jdisnard(a)gmail.com&gt; wrote: > The inability to shrink or reduce XFS is rather disappointing. I've > seen a few sarcastic remarks along the lines of (paraphrased): why > would anyone ever want to shrink a volume? In the context of server, and default installs, why is a valid question. > People do shrink volumes, and this lack of flexibility is an important > consideration I feel was ignored in the Server WG decision. What is the use case for volume shrinking in a server context? Dual boot is a total edge case for servers.

> for me > personally, I'm not sure any performance gains are enough to > compensate for the lack of flexibility. Considering that LVM has the > ability to resize volumes, ext4 pairs very well, Unless you use system-storage-manager, you might refresh the steps required to do an ext4 on LVM resize. I don't think the person who understands how to do that is really the target audience for default installs.

> and I'm skeptical > thin provisioning does enough make-up for the lack of XFS shrinking. It even makes up for ext4 shrinking in two ways. 1. Instead of making an LV smaller than possibly needed, make it the size it probably should be from the start. It only consumes extents from the thinpool as needed. 2. Upon significant file deletion, fstrim returns unused extents back to the thinpool. This is faster, more efficient, and less fragile than file system shrink. Again, the problem is that commands are a bit esoteric, but maybe system-storage-manager can help out with this once it support thin provisioning.

> So my question to the Server WG, did anybody consider this aspect of > XFS (lack of shrinking) before making the decision? What were the > highest reasons for NOT staying with EXT4? I realize the thread has well over 100 emails in it, but it was considered. The simple explanation why XFS was chosen is because it was well vetted by Red Hat storage experts for use as the default in RHEL 7 - and I understand that sgallagh is working on a summary of those reasons, which would apply here as well. I'd say top on the list is XFS is scales linearly as more threads are thrown at it, it's very good at parallelism, and it support project quotas which often obviates the need to create separate file systems as a means of constraining storage usage rather than doing it only by user or group.

The shrink/grow thing was clever, but also a bit abusive from a filesystem design point of view.

We no longer release Fedora ARM rootfs tarballs, too hard to educate people to do the right thing with ACL's, xattrs, selinux, etc... Anyhow, it's actually a great way to ship a Fedora rootfs... just shrink the filesystem down to the smallest size, and allow the user to simply resize2fs the image into their storage.

This would not be possible with XFS for the server variant of Fedora-ARM, and I feel represents a significant challenge to the ARM team.

Thin provisioning sounds great, but I'm not sure it replaces the ability to shrink in all situations, but it appears to negate many of the situations I've mentioned, but not all.

I would imagine people turning their ARM dev board into a home NAS or some similar storage kit, and the linear scale-up of XFS is really appealing.

On the rel-eng side we are not using anaconda to compose the ARM images because we cannot put Anaconda into koji tasks, so instead we use appliance-tools for ARM images.

On Sat, 2014-03-01 at 12:04 +0000, Ian Malone wrote: > On 28 February 2014 20:45, Adam Williamson <awilliam(a)redhat.com&gt; wrote: > > On Thu, 2014-02-27 at 23:16 -0700, Chris Murphy wrote:

> As you say they are 'plain' filesystems. Though I now regret not > sending my small datapoint in before the Server WG decision. That's > that a while ago, after using XFS for a long time we started putting > new filesystems onto ext4 and in the past month we moved probably our > largest remaining dataset (1.1TB) from XFS to ext4, the main reason > has been flexibility with resizing. Particularly the XFS 32bit inode > ceiling, (inode64 not working well with NFS). > 1TB doesn't sound very big. These are imaging datasets in a research > environment, files going from small text to images at tens of MB (some > bigger, but not the dominant type). Projects usually get their own FS > (for a variety of reasons including backup, audit and budgeting > reasons). And often it's not known how large a project will eventually > be, so filesystems get extended as appropriate. With XFS we have to > take care to avoid the 32bit inode ceiling, and most recently found a > filesystem that refused to take any more files for some other reason, > even after creating a new clean copy. We didn't get to the bottom of > that, and moved the data to ext4.

As far as I know inode64 is not really a problem on NFS anymore, which is why I did not raise this as an issue at all (I use NFS and I have a 6TB XFS filesystem with inode64).

On 03/01/2014 02:30 PM, Ian Malone wrote: > On 1 March 2014 18:57, Simo Sorce <simo(a)redhat.com&gt; wrote: >> On Sat, 2014-03-01 at 12:04 +0000, Ian Malone wrote: >>> On 28 February 2014 20:45, Adam Williamson <awilliam(a)redhat.com&gt; wrote: >>>> On Thu, 2014-02-27 at 23:16 -0700, Chris Murphy wrote: > >>> As you say they are 'plain' filesystems. Though I now regret not >>> sending my small datapoint in before the Server WG decision. That's >>> that a while ago, after using XFS for a long time we started putting >>> new filesystems onto ext4 and in the past month we moved probably our >>> largest remaining dataset (1.1TB) from XFS to ext4, the main reason >>> has been flexibility with resizing. Particularly the XFS 32bit inode >>> ceiling, (inode64 not working well with NFS). > >> As far as I know inode64 is not really a problem on NFS anymore, which >> is why I did not raise this as an issue at all (I use NFS and I have a >> 6TB XFS filesystem with inode64). >> > > Unless you have legacy systems that must talk to it. Can we get some definition of "legacy" here? kernel/nfs-utils versions?

On 03/02/2014 01:17 PM, Ian Malone wrote: >> >> Can we get some definition of "legacy" here? kernel/nfs-utils versions? >> > > > I'd have to check what I can share. If it helps: not current RHEL or > recent Fedora, until recently some that were over five years old. Also > this comment in the XFS FAQ: "Beware that some old programs might have > problems reading 64bit inodes" which seems to be related to stat vs > stat64, there are some old programs that might require us to modify. Just to be clear, defaults are only important when you install. I don't think anyone is suggesting compiling out the ext4 code from the kernel so this should be of zero impact to someone running a file year old system who wants to upgrade to something modern. If you need to stay on that five year old system and not upgrade, I am not sure I understand why it would be a reasonable example. As other people have mentioned, there are ways to create an XFS instance that does not use 64 bit inodes (but that is *not* a sane default since the problems you refer to are not common with modern apps).

I am pretty sure that ext4 is a built-in module in Fedora kernels, as well as in the boot environment; making XFS the default will require also building it in, pretty much forever, while we still need extXX, and whatever comes next (btrfs?). I am OK with that, though.

Once upon a time, Przemek Klosowski <przemek.klosowski(a)nist.gov&gt; said: > I am pretty sure that ext4 is a built-in module in Fedora kernels, > as well as in the boot environment; making XFS the default will > require also building it in, pretty much forever, while we still > need extXX, and whatever comes next (btrfs?). I am OK with that, > though. Not really, it is only recent Fedora releases that have built ext4 into the kernel. We went many years with ext3 being loaded as a module (was it ever built in?); ext4 was added more for convenience. Having the "common stuff" that's going to be loaded on something like 99% of Fedora systems anyway built into the kernel is slightly more efficient and more convenient. If we have a split between filesystems between products, then it probably doesn't make sense to build any of them into the kernel anymore (well, assuming they all use the same kernel RPMs).

On 03/04/2014 11:26 PM, Przemek Klosowski wrote: > On 02/28/2014 03:45 PM, Adam Williamson wrote: >> As a server WG member I voted +1 on XFS as I have no particular objection to XFS as a filesystem, but I do think it seems a bit sub-optimal for us to wind up with server and desktop having defaults that are very similar but slightly different, for no apparently great reason. > This may be a historical bias. XFS is a large code base (*), which means two things: a larger bug surface, and a larger memory footprint that used to be a problem for personal desktop-type machines but less so for better endowed servers. > > I understand that by now XFS got so much exercise that its > robustness is unimpeachable. As to the size, I see that while the > latest XFS kernel module is one of the larger kernel modules > around, it probably is no longer significant on today's multi-GB > systems---the extra megabyte at current memory prices is just a one > cent increase in the system cost, after all. > > Having said that, I don't use XFS nowadays so I don't know how much > more memory it allocates in typical use---can anyone comment on the > actual memory footprint of running XFS? > > I am pretty sure that ext4 is a built-in module in Fedora kernels, > as well as in the boot environment; making XFS the default will > require also building it in, pretty much forever, while we still > need extXX, and whatever comes next (btrfs?). I am OK with that, > though.>> > > (*) 2.9MB of XFS source code vs 1.3MB in ext4 dirs > (**) xfs.ko is 1.3MB > > You need to count the jbd2 code for ext4 as well, Ric

On 02/27/2014 11:20 PM, Chris Murphy wrote: > > In the context of the default ext4+LVM layout the conversion still means separate /boot, /, and /home file systems. A major benefit of the Btrfs layout is these are subvolumes, which instead draw space from one volume pool. And that's lost with a conversion strategy. It also means going from a Fedora "standard" layout to a distinctly non-standard one because our Btrfs layout isn't like the result you'd get from what you're talking about. > > > Chris Murphy > A question I have, forgive me if it seems stupid, is why not just change the standard?

On Feb 26, 2014, at 12:53 PM, Michael Cronenworth <mike(a)cchtml.com&gt; wrote: > Chris Murphy wrote: >> by default we put ext4 on LVM > The tool works in this use-case unless something has broken it recently. It can be done, the convert tool should work, and Btrfs should work on any device mapper instance. However… In the context of the default ext4+LVM layout the conversion still means separate /boot, /, and /home file systems. A major benefit of the Btrfs layout is these are subvolumes, which instead draw space from one volume pool. And that's lost with a conversion strategy. It also means going from a Fedora "standard" layout to a distinctly non-standard one because our Btrfs layout isn't like the result you'd get from what you're talking about. Chris Murphy