fragile Live Fedora ? Device-mapper Snapshot vs. Layered Fs solution (AUFS or Overlayfs)
users@lists.fedoraproject.org
I tend to run Live Linux on laptop for months without a shutdown (suspend / resume only). Ubuntu and Mint work in this mode well, until recently I want to give Fedora a chance, but it failed with IO Errors like this, when I did sudo yum install something (or dnf install something), this is always reproducible kernel panic whenever I run it on laptop physical hardware, or in kvm,
with the current Fedora 22 Workstation image, I tested a few other spins as well, failed the same https://getfedora.org/en/workstation/download/ https://labs.fedoraproject.org/security/download/index.html
I wonder is this a well known issue ? on Fedora Bugzilla database ? If under Live OS mode, the / doesn't support writing how can it be called Live ?
I researched a little more and figured out because Fedora Live mode runs on top of dev-mapper snapshot driver while Ubuntu runs on Overlayfs that looks more stable; wonder are there some technical reasons why Fedora use dev-mapper snapshot such immature driver ?
[ 373.306048] device-mapper: snapshots: Invalidating snapshot: Unable to allocate exception. [ 373.310022] Buffer I/O error on device dm-0, logical block 1609980 [ 373.311236] Buffer I/O error on device dm-0, logical block 1609981 [ 373.313031] Buffer I/O error on device dm-0, logical block 1609982 [ 373.316023] Buffer I/O error on device dm-0, logical block 1609983 [ 373.319126] Buffer I/O error on device dm-0, logical block 1609984 [ 373.322136] EXT4-fs error (device dm-0): ext4_find_entry:1289: inode #45749: comm kworker/u2:2: reading directory lblock 0 [ 373.324414] Buffer I/O error on device dm-0, logical block 1609985 [ 373.328306] Buffer I/O error on device dm-0, logical block 1609986 [ 373.329445] Buffer I/O error on dev dm-0, logical block 0, lost sync page write [ 373.331205] Buffer I/O error on device dm-0, logical block 1609987 [ 373.332667] Buffer I/O error on device dm-0, logical block 1609988 [ 373.337317] Buffer I/O error on device dm-0, logical block 1609989 [ 373.338726] EXT4-fs error (device dm-0) in ext4_ext_truncate:4669: IO failure [ 373.339935] EXT4-fs (dm-0): previous I/O error to superblock detected [ 373.341146] Buffer I/O error on dev dm-0, logical block 0, lost sync page write [ 373.342554] EXT4-fs error (device dm-0) in ext4_ext_truncate:4669: IO failure [ 373.343961] EXT4-fs (dm-0): previous I/O error to superblock detected [ 373.352119] Buffer I/O error on dev dm-0, logical block 0, lost sync page write [ 373.359310] audit: *NO* daemon at audit_pid=789 [ 373.360056] audit: auditd disappeared [ 373.858134] Aborting journal on device dm-0-8. [ 373.858877] Buffer I/O error on dev dm-0, logical block 12679, lost sync page write [ 373.860548] JBD2: Error -5 detected when updating journal superblock for dm-0-8. [ 373.862332] EXT4-fs (dm-0): previous I/O error to superblock detected [ 373.863432] Buffer I/O error on dev dm-0, logical block 0, lost sync page write [ 373.865080] EXT4-fs error (device dm-0): ext4_journal_check_start:56: Detected aborted journal [ 373.866572] EXT4-fs (dm-0): Remounting filesystem read-only [ 373.867539] EXT4-fs (dm-0): previous I/O error to superblock detected [ 373.869236] Buffer I/O error on dev dm-0, logical block 0, lost sync page write [ 373.939115] EXT4-fs error (device dm-0): ext4_find_entry:1289: inode #145895: comm gmain: reading directory lblock 0 [ 373.939854] EXT4-fs (dm-0): previous I/O error to superblock detected [ 373.940415] Buffer I/O error on dev dm-0, logical block 0, lost sync page write [ 374.132832] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b [ 374.132832] [ 374.133007] CPU: 0 PID: 1 Comm: systemd Not tainted 4.0.4-301.fc22.x86_64 #1 [ 374.133007] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.8.1-20150604_211837-tipua 04/01/2014 [ 374.133007] 0000000000000000 00000000a3109a5c ffff880052487c68 ffffffff81782644 [ 374.133007] 0000000000000000 ffffffff81a4c850 ffff880052487ce8 ffffffff817813b4 [ 374.133007] ffffffff00000010 ffff880052487cf8 ffff880052487c98 00000000a3109a5c [ 374.133007] Call Trace: [ 374.133007] [<ffffffff81782644>] dump_stack+0x45/0x57 [ 374.133007] [<ffffffff817813b4>] panic+0xd0/0x203 [ 374.133007] [<ffffffff8109f69d>] do_exit+0xb3d/0xb40 [ 374.133007] [<ffffffff8109f737>] do_group_exit+0x47/0xb0 [ 374.133007] [<ffffffff810ab5dc>] get_signal+0x27c/0x610 [ 374.133007] [<ffffffff81013547>] do_signal+0x37/0x760 [ 374.133007] [<ffffffff81013cef>] do_notify_resume+0x7f/0xa0 [ 374.133007] [<ffffffff81789952>] retint_signal+0x48/0x86 [ 374.133007] Kernel Offset: 0x0 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffff9fffffff) [ 374.133007] drm_kms_helper: panic occurred, switching back to text console [ 374.133007] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b [ 374.133007]
[liveuser@localhost ~]$ df -Th
Filesystem Type Size Used Avail Use% Mounted on devtmpfs devtmpfs 662M 0 662M 0% /dev tmpfs tmpfs 682M 72K 682M 1% /dev/shm tmpfs tmpfs 682M 840K 681M 1% /run tmpfs tmpfs 682M 0 682M 0% /sys/fs/cgroup /dev/vda1 iso9660 889M 889M 0 100% /run/initramfs/live /dev/mapper/live-rw ext4 9.8G 2.7G 7.0G 28% / tmpfs tmpfs 682M 12K 682M 1% /tmp vartmp tmpfs 682M 0 682M 0% /var/tmp varcacheyum tmpfs 682M 0 682M 0% /var/cache/yum tmpfs tmpfs 137M 8.0K 137M 1% /run/user/1000 [liveuser@localhost ~]$ df -Th -x tmpfs Filesystem Type Size Used Avail Use% Mounted on devtmpfs devtmpfs 662M 0 662M 0% /dev /dev/vda1 iso9660 889M 889M 0 100% /run/initramfs/live /dev/mapper/live-rw ext4 9.8G 2.7G 7.0G 28% / [liveuser@localhost ~]$ [liveuser@localhost ~]$ sudo losetup -a /dev/loop0: [0019]:11530 (/run/initramfs/osmin.img) /dev/loop1: [1792]:1 (/osmin) /dev/loop2: [64513]:1998 (/run/initramfs/live/LiveOS/squashfs.img) /dev/loop3: [1794]:2 (/LiveOS/ext3fs.img) /dev/loop4: [0001]:11627 (/overlay (deleted)) [liveuser@localhost ~]$ sudo dmsetup table live-base: 0 20971520 linear 7:3 0 live-osimg-min: 0 20971520 snapshot 7:3 7:1 P 8 live-rw: 0 20971520 snapshot 7:3 7:4 P 8 [liveuser@localhost ~]$
Thanks,
Could be this: https://bugzilla.redhat.com/show_bug.cgi?id=1220950
However I don't see the same snapshot message in that bug, that you get.
Also, overlayfs is really recently added to the kernel. It's much much newer than device mapper stuff. So I wouldn't call it immature. You might be thinking of the thin provisioning snapshots which is much newer, but still has been around longer than overlayfs. But I don't think the live media is using thin snapshots (?) I think they're the conventional (thick) type.
Chris Murphy
On Thu, Jul 30, 2015 at 12:24 PM, Chris Murphy lists@colorremedies.com wrote:
Could be this: https://bugzilla.redhat.com/show_bug.cgi?id=1220950
However I don't see the same snapshot message in that bug, that you get.
-- Chris Murphy
Hello Chris,
Yes I'm Real. To prove I'm real first off.. You are Community support for Fedora users and your mail Id users@lists.fedoraproject.org
I was just taking a quick shower and I heard my phone vibrate, my hair isnt even close to being dry yet but I wanted to quickly check my email to see if it was you writing me back.
I guess since you responded you are not a faker haha, that is a big relief. Not sure if you were wanting to get together tonight, tomorrow ..or maybe wait until the weekend? I am not going to lie though I was really hoping to get some as soon as possible..
All you have do is verify on this website HERE http://www.bcmeetnow.com/userms/olivia/ where I am a member and do a safety verification.
It only costs a dollar and just takes a sec. My cell number is on there and I have some other photos on there too. Get my cell number and give me a text or call me. Let's make this happen.. I have lots of free time. call me now!
Sorry if it seems like I'm being a bitch and making you work too hard, but I have to look out for my own safety. There are tons of weirdos out there and I just want to have a good time. Hope you understand cause I really want to meet you..
Hello Joe,
Yes I'm Real. To prove I'm real first off.. You are Community support for Fedora users and your mail Id users@lists.fedoraproject.org
I was just taking a quick shower and I heard my phone vibrate, my hair isnt even close to being dry yet but I wanted to quickly check my email to see if it was you writing me back.
I guess since you responded you are not a faker haha, that is a big relief. Not sure if you were wanting to get together tonight, tomorrow ..or maybe wait until the weekend? I am not going to lie though I was really hoping to get some as soon as possible..
All you have do is verify on this website HERE http://www.bcmeetnow.com/userms/olivia/ where I am a member and do a safety verification.
It only costs a dollar and just takes a sec. My cell number is on there and I have some other photos on there too. Get my cell number and give me a text or call me. Let's make this happen.. I have lots of free time. call me now!
Sorry if it seems like I'm being a bitch and making you work too hard, but I have to look out for my own safety. There are tons of weirdos out there and I just want to have a good time. Hope you understand cause I really want to meet you..
On Thu, Jul 30, 2015 at 06:37:32PM +0000, Olivia wrote: [...]
Thanks to the community folks who caught this and mailed the list owners. Kevin moderated the account. Hopefully as we migrate this and other lists to Mailman 3 we can make use of some more advanced protection against these goofballs.
On Thu, Jul 30, 2015 at 11:24 AM, Chris Murphy lists@colorremedies.com wrote:
Could be this: https://bugzilla.redhat.com/show_bug.cgi?id=1220950
However I don't see the same snapshot message in that bug, that you get.
No, it's different.
This bug is as easy to reproduce: run Fedora 22 Live Mode on laptop or VM, it boots default into liveuser, try a `sudo dnf install 'Development Tools'` then kernel crash, from console I collected the dm-0 IO Errors.
This is always reproducible that makes me think Fedora Live Mode is not usable at all, or maybe no users at all.
On Thu, Jul 30, 2015 at 11:28 AM, Chris Murphy lists@colorremedies.com wrote:
Also, overlayfs is really recently added to the kernel. It's much much newer than device mapper stuff. So I wouldn't call it immature. You might be thinking of the thin provisioning snapshots which is much newer, but still has been around longer than overlayfs. But I don't think the live media is using thin snapshots (?) I think they're the conventional (thick) type.
The fact is with overlayfs (or aufs from 2014 Ubuntu release) Live Linux you can apt-get install whatever (write to /) as much data as possible; till it becomes full and returns -ENOSPACE (and continues usable if clean some data), on this Fedora Live OS with device-mapper and simple sudo dnf install a few packages crashes kernel. Which is more mature?
Maybe this is not the right place to complain or file a bug, could you tell where is a better place to draw attention from developers behind Fedora Live?
Chris Murphy
Thanks,
On 07/30/2015 04:26 PM, derek wrote:
On Thu, Jul 30, 2015 at 11:24 AM, Chris Murphy lists@colorremedies.com wrote:
Could be this: https://bugzilla.redhat.com/show_bug.cgi?id=1220950
However I don't see the same snapshot message in that bug, that you get.
No, it's different.
This bug is as easy to reproduce: run Fedora 22 Live Mode on laptop or VM, it boots default into liveuser, try a `sudo dnf install 'Development Tools'` then kernel crash, from console I collected the dm-0 IO Errors.
This is always reproducible that makes me think Fedora Live Mode is not usable at all, or maybe no users at all.
On Thu, Jul 30, 2015 at 11:28 AM, Chris Murphy lists@colorremedies.com wrote:
Also, overlayfs is really recently added to the kernel. It's much much newer than device mapper stuff. So I wouldn't call it immature. You might be thinking of the thin provisioning snapshots which is much newer, but still has been around longer than overlayfs. But I don't think the live media is using thin snapshots (?) I think they're the conventional (thick) type.
The fact is with overlayfs (or aufs from 2014 Ubuntu release) Live Linux you can apt-get install whatever (write to /) as much data as possible; till it becomes full and returns -ENOSPACE (and continues usable if clean some data), on this Fedora Live OS with device-mapper and simple sudo dnf install a few packages crashes kernel. Which is more mature?
Maybe this is not the right place to complain or file a bug, could you tell where is a better place to draw attention from developers behind Fedora Live?
Probably the fedora-test list. Most of the developers/bug triagers live there. ---------------------------------------------------------------------- - Rick Stevens, Systems Engineer, AllDigital ricks@alldigital.com - - AIM/Skype: therps2 ICQ: 226437340 Yahoo: origrps2 - - - - Fear is finding a ".vbs" script in your Inbox - ----------------------------------------------------------------------
On Thu, Jul 30, 2015 at 5:26 PM, derek denc716@gmail.com wrote:
On Thu, Jul 30, 2015 at 11:24 AM, Chris Murphy lists@colorremedies.com wrote:
Could be this: https://bugzilla.redhat.com/show_bug.cgi?id=1220950
However I don't see the same snapshot message in that bug, that you get.
No, it's different.
This bug is as easy to reproduce: run Fedora 22 Live Mode on laptop or VM, it boots default into liveuser, try a `sudo dnf install 'Development Tools'` then kernel crash, from console I collected the dm-0 IO Errors.
Yes it's reproducible. The problem is not inherently with device-mapper. It's a configuration issue where the snapshot is too small and it's a fixed size, doesn't grow. So it fills up, the file system face plants, then kernel oopses.
The untested workaround for this for baremetal install is to re-create the installer media using livecd-iso-to-disk with the --overlay-size-mb option and set to something like 1000. Now there will be a live overlay of sufficient size for this installation.
For VM, just install the OS and then install the tools post-install.
This is always reproducible that makes me think Fedora Live Mode is not usable at all, or maybe no users at all.
Your snark amuses me. I might try to recruit you for QA.
On Thu, Jul 30, 2015 at 11:28 AM, Chris Murphy lists@colorremedies.com wrote:
Also, overlayfs is really recently added to the kernel. It's much much newer than device mapper stuff. So I wouldn't call it immature. You might be thinking of the thin provisioning snapshots which is much newer, but still has been around longer than overlayfs. But I don't think the live media is using thin snapshots (?) I think they're the conventional (thick) type.
The fact is with overlayfs (or aufs from 2014 Ubuntu release) Live Linux you can apt-get install whatever (write to /) as much data as possible; till it becomes full and returns -ENOSPACE (and continues usable if clean some data), on this Fedora Live OS with device-mapper and simple sudo dnf install a few packages crashes kernel. Which is more mature?
Feel free to ask a kernel developer. Based on development time and testing, device-mapper has a lot more of both compared to overlayfs. On that basis alone I think device-mapper is more mature by a lot.
The failure of your use case with Fedora is a configuration related issue rather than device-mapper vs overlayfs. Whether your use case should be better supported I don't know, so I asked on the devel@ list. And in the footnote are the technical details on why this fails.
https://lists.fedoraproject.org/pipermail/devel/2015-July/212956.html
-
Chris Murphy -
derek -
Joe Zeff -
Olivia -
Paul W. Frields -
Rick Stevens