2:59 p.m.
I have an easy to reproduce problem I'm experiencing with a Live image
generated from Fedora 18 bases. If too much gets written to the writable
tmpfs overlay, the host goes all wonky in that most all commands will fail
to run and immediately return one of the above error messages. For
example:
root@aos-61:46 # cat /proc/meminfo
MemTotal: 995020 kB
MemFree: 199348 kB
Buffers: 91192 kB
Cached: 637920 kB
SwapCached: 0 kB
Active: 221796 kB
Inactive: 549340 kB
Active(anon): 43120 kB
Inactive(anon): 3220 kB
Active(file): 178676 kB
Inactive(file): 546120 kB
Unevictable: 0 kB
Mlocked: 0 kB
HighTotal: 110344 kB
HighFree: 220 kB
LowTotal: 884676 kB
LowFree: 199128 kB
SwapTotal: 0 kB
SwapFree: 0 kB
Dirty: 0 kB
Writeback: 0 kB
AnonPages: 42036 kB
Mapped: 21392 kB
Shmem: 4324 kB
Slab: 16696 kB
SReclaimable: 11736 kB
SUnreclaim: 4960 kB
KernelStack: 584 kB
PageTables: 612 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 497508 kB
Committed_AS: 92280 kB
VmallocTotal: 122880 kB
VmallocUsed: 35324 kB
VmallocChunk: 87296 kB
HardwareCorrupted: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 4096 kB
DirectMap4k: 12280 kB
DirectMap4M: 892928 kB
root@aos-61:46 # df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 481M 0 481M 0% /dev
tmpfs 486M 0 486M 0% /dev/shm
tmpfs 486M 296K 486M 1% /run
tmpfs 486M 0 486M 0% /sys/fs/cgroup
/dev/sda1 7.4G 1.2G 5.9G 17% /run/initramfs/live
/dev/mapper/live-rw 2.0G 1.3G 658M 67% /
tmpfs 486M 1.5M 485M 1% /tmp
root@aos-61:46 # mount
proc on /proc type proc (rw,relatime)
sysfs on /sys type sysfs (rw,relatime)
devtmpfs on /dev type devtmpfs
(rw,nosuid,size=492160k,nr_inodes=123040,mode=755)
securityfs on /sys/kernel/security type securityfs
(rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,relatime)
devpts on /dev/pts type devpts (rw,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755)
tmpfs on /sys/fs/cgroup type tmpfs (rw,nosuid,nodev,noexec,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup
(rw,nosuid,nodev,noexec,relatime,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)
cgroup on /sys/fs/cgroup/cpuset type cgroup
(rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup
(rw,nosuid,nodev,noexec,relatime,cpuacct,cpu)
cgroup on /sys/fs/cgroup/memory type cgroup
(rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/devices type cgroup
(rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/freezer type cgroup
(rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/net_cls type cgroup
(rw,nosuid,nodev,noexec,relatime,net_cls)
cgroup on /sys/fs/cgroup/blkio type cgroup
(rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/perf_event type cgroup
(rw,nosuid,nodev,noexec,relatime,perf_event)
/dev/sda1 on /run/initramfs/live type ext3 (ro,relatime)
/dev/mapper/live-rw on / type ext3 (rw,noatime,data=ordered)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs
(rw,relatime,fd=29,pgrp=1,timeout=300,minproto=5,maxproto=5,direct)
mqueue on /dev/mqueue type mqueue (rw,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
tmpfs on /tmp type tmpfs (rw)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime)
configfs on /sys/kernel/config type configfs (rw,relatime)
none on /var/lib/stateless/writable type tmpfs (rw,relatime)
none on /var/cache/man type tmpfs (rw,relatime)
none on /var/lib/xkb type tmpfs (rw,relatime)
none on /var/log type tmpfs (rw,relatime)
none on /var/lib/dbus type tmpfs (rw,relatime)
none on /var/lib/nfs type tmpfs (rw,relatime)
none on /tmp type tmpfs (rw,relatime)
none on /var/lib/dhclient type tmpfs (rw,relatime)
none on /var/tmp type tmpfs (rw,relatime)
none on /var/lib/logrotate.status type tmpfs (rw,relatime)
none on /var/lib/random-seed type tmpfs (rw,relatime)
none on /var/spool type tmpfs (rw,relatime)
/dev/sda1 on /mnt/live type ext3 (rw,noatime)
root@aos-61:46 #
root@aos-61:46 #
root@aos-61:46 # # Lets now make it all go wonky:
root@aos-61:46 # time dd if=/dev/zero of=/foo
Bus error
real 1m15.775s
user 0m2.818s
sys 0m24.129s
root@aos-61:46 #
root@aos-61:46 # ls /root
-bash: /bin/ls: Input/output error
root@aos-61:46 # df -h
-bash: /usr/bin/df: Input/output error
root@aos-61:46 # mount
-bash: /usr/bin/mount: Input/output error
root@aos-61:46 # cat /proc/meminfo
-bash: /usr/bin/cat: Input/output error
Is this expected? Is there anything I can do, e.g., configuration-wise,
that can prevent this? Ideally this would fail much like any other full
disk situation. I understand that the overlay consumes space, i.e.,
memory, for this file growth, including file removals, but I'd at least
like to be able to remotely reboot a system when in this state, however I
can't even do that because the reboot command will either return the same
I/O error or it may succeed but get the I/O error when systemd tries to
read /usr/lib/systemd/system/reboot.target.
I dug around in bugzilla, but found nothing there. I can file a bug, but
which package is likely at fault here?
--
John Florian
9:02 p.m.
On Wed, Mar 6, 2013 at 3:59 PM, <John.Florian(a)dart.biz> wrote:
I have an easy to reproduce problem I'm experiencing with a Live
image
generated from Fedora 18 bases. If too much gets written to the writable
tmpfs overlay, the host goes all wonky in that most all commands will fail
to run and immediately return one of the above error messages. For example:
root@aos-61:46 # cat /proc/meminfo
MemTotal: 995020 kB
MemFree: 199348 kB
Buffers: 91192 kB
Cached: 637920 kB
SwapCached: 0 kB
Active: 221796 kB
Inactive: 549340 kB
Active(anon): 43120 kB
Inactive(anon): 3220 kB
Active(file): 178676 kB
Inactive(file): 546120 kB
Unevictable: 0 kB
Mlocked: 0 kB
HighTotal: 110344 kB
HighFree: 220 kB
LowTotal: 884676 kB
LowFree: 199128 kB
SwapTotal: 0 kB
SwapFree: 0 kB
Dirty: 0 kB
Writeback: 0 kB
AnonPages: 42036 kB
Mapped: 21392 kB
Shmem: 4324 kB
Slab: 16696 kB
SReclaimable: 11736 kB
SUnreclaim: 4960 kB
KernelStack: 584 kB
PageTables: 612 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 497508 kB
Committed_AS: 92280 kB
VmallocTotal: 122880 kB
VmallocUsed: 35324 kB
VmallocChunk: 87296 kB
HardwareCorrupted: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 4096 kB
DirectMap4k: 12280 kB
DirectMap4M: 892928 kB
root@aos-61:46 # df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 481M 0 481M 0% /dev
tmpfs 486M 0 486M 0% /dev/shm
tmpfs 486M 296K 486M 1% /run
tmpfs 486M 0 486M 0% /sys/fs/cgroup
/dev/sda1 7.4G 1.2G 5.9G 17% /run/initramfs/live
/dev/mapper/live-rw 2.0G 1.3G 658M 67% /
tmpfs 486M 1.5M 485M 1% /tmp
root@aos-61:46 # mount
proc on /proc type proc (rw,relatime)
sysfs on /sys type sysfs (rw,relatime)
devtmpfs on /dev type devtmpfs
(rw,nosuid,size=492160k,nr_inodes=123040,mode=755)
securityfs on /sys/kernel/security type securityfs
(rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,relatime)
devpts on /dev/pts type devpts (rw,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755)
tmpfs on /sys/fs/cgroup type tmpfs (rw,nosuid,nodev,noexec,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup
(rw,nosuid,nodev,noexec,relatime,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)
cgroup on /sys/fs/cgroup/cpuset type cgroup
(rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup
(rw,nosuid,nodev,noexec,relatime,cpuacct,cpu)
cgroup on /sys/fs/cgroup/memory type cgroup
(rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/devices type cgroup
(rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/freezer type cgroup
(rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/net_cls type cgroup
(rw,nosuid,nodev,noexec,relatime,net_cls)
cgroup on /sys/fs/cgroup/blkio type cgroup
(rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/perf_event type cgroup
(rw,nosuid,nodev,noexec,relatime,perf_event)
/dev/sda1 on /run/initramfs/live type ext3 (ro,relatime)
/dev/mapper/live-rw on / type ext3 (rw,noatime,data=ordered)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs
(rw,relatime,fd=29,pgrp=1,timeout=300,minproto=5,maxproto=5,direct)
mqueue on /dev/mqueue type mqueue (rw,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
tmpfs on /tmp type tmpfs (rw)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime)
configfs on /sys/kernel/config type configfs (rw,relatime)
none on /var/lib/stateless/writable type tmpfs (rw,relatime)
none on /var/cache/man type tmpfs (rw,relatime)
none on /var/lib/xkb type tmpfs (rw,relatime)
none on /var/log type tmpfs (rw,relatime)
none on /var/lib/dbus type tmpfs (rw,relatime)
none on /var/lib/nfs type tmpfs (rw,relatime)
none on /tmp type tmpfs (rw,relatime)
none on /var/lib/dhclient type tmpfs (rw,relatime)
none on /var/tmp type tmpfs (rw,relatime)
none on /var/lib/logrotate.status type tmpfs (rw,relatime)
none on /var/lib/random-seed type tmpfs (rw,relatime)
none on /var/spool type tmpfs (rw,relatime)
/dev/sda1 on /mnt/live type ext3 (rw,noatime)
root@aos-61:46 #
root@aos-61:46 #
root@aos-61:46 # # Lets now make it all go wonky:
root@aos-61:46 # time dd if=/dev/zero of=/foo
Bus error
real 1m15.775s
user 0m2.818s
sys 0m24.129s
root@aos-61:46 #
root@aos-61:46 # ls /root
-bash: /bin/ls: Input/output error
root@aos-61:46 # df -h
-bash: /usr/bin/df: Input/output error
root@aos-61:46 # mount
-bash: /usr/bin/mount: Input/output error
root@aos-61:46 # cat /proc/meminfo
-bash: /usr/bin/cat: Input/output error
Is this expected? Is there anything I can do, e.g., configuration-wise,
that can prevent this? Ideally this would fail much like any other full
disk situation. I understand that the overlay consumes space, i.e.,
memory, for this file growth, including file removals, but I'd at least
like to be able to remotely reboot a system when in this state, however I
can't even do that because the reboot command will either return the same
I/O error or it may succeed but get the I/O error when systemd tries to
read /usr/lib/systemd/system/reboot.target.
I dug around in bugzilla, but found nothing there. I can file a bug, but
which package is likely at fault here?
--
John Florian
See https://fedoraproject.org/wiki/LiveOS_image for some background and
potential workarounds.
--Fred
9:41 a.m.
From: Frederick Grose <fgrose(a)gmail.com>
On Wed, Mar 6, 2013 at 3:59 PM, <John.Florian(a)dart.biz> wrote:
<snip>
root@aos-61:46 # # Lets now make it all go wonky:
root@aos-61:46 # time dd if=/dev/zero of=/foo
Bus error
real 1m15.775s
user 0m2.818s
sys 0m24.129s
root@aos-61:46 #
root@aos-61:46 # ls /root
-bash: /bin/ls: Input/output error
root@aos-61:46 # df -h
-bash: /usr/bin/df: Input/output error
root@aos-61:46 # mount
-bash: /usr/bin/mount: Input/output error
root@aos-61:46 # cat /proc/meminfo
-bash: /usr/bin/cat: Input/output error
Is this expected? Is there anything I can do, e.g., configuration-
wise, that can prevent this? Ideally this would fail much like any
other full disk situation. I understand that the overlay consumes
space, i.e., memory, for this file growth, including file removals,
but I'd at least like to be able to remotely reboot a system when in
this state, however I can't even do that because the reboot command
will either return the same I/O error or it may succeed but get the
I/O error when systemd tries to read
/usr/lib/systemd/system/reboot.target.
I dug around in bugzilla, but found nothing there. I can file a
bug, but which package is likely at fault here?
--
John Florian
See https://fedoraproject.org/wiki/LiveOS_image for some background
and potential workarounds.
--Fred --
There's really not much on that page that helps me here. I'm trying to
use Live images for a mostly-stateless embedded appliance OS deployed to
hundreds or thousands of devices. I realize that the COW design is always
going to be limited, but a more graceful failure mode is really needed,
somehow. For our use, the biggest gain in stability here actually comes
from systemd's journal with its trim-before-write approach instead of the
legacy write now, trim asynchronously approach we used to have. However,
that only covers one specific use case: logging. Writing to proper
persistent storage allows me to avoid the root file system overlay, but
most of these embedded devices use CF or SD cards for storage, which have
limited write cycles that must be respected.
Is there a way to implement an artificial capacity limit that would
prevent processes from exhausting the overlay so that the reserve might be
used for recording the event and rebooting back to a safer state?
At the very least, I think this page could benefit from a little stronger,
more explicit wording of this failure case. While it talks a little about
some work-arounds, it actually says very little about why they are needed.
Only in the "Overlay Recovery" section does it hint at the crash
potential.
--
John Florian
9:50 a.m.
Is there a way to implement an artificial capacity limit that would
prevent processes from exhausting the overlay so that the reserve
might be used for recording the event and rebooting back to a safer
state?
The easiest way is to play with the partition size (set in your
kickstart file). There are two things that can stop a file being
written: either the overlay is full, or the filesystem is full. If you
set the partition size to a smaller value, you'll get filesystem errors,
which are probably going to be less severe.
Of course, if you have a small partition size and a huge overlay, then
most of your overlay will not ever be usable, so you want to play with
the two things together. As a rough idea, the free space in your
partition (after everything's been installed) should be a bit less than
the overlay size.
There are potential pitfalls here, because free filesystem space doesn't
quite equate with free overlay space. I don't know what happens if you
boot up and delete a large file that was installed in the squashfs--it
might increase the free space as far as the filesystem is concerned, but
it obviously won't buy you any extra overlay space.
James
10:47 a.m.
From: James Heather <j.heather(a)surrey.ac.uk>
> Is there a way to implement an artificial capacity limit that would
> prevent processes from exhausting the overlay so that the reserve
> might be used for recording the event and rebooting back to a safer
> state?
The easiest way is to play with the partition size (set in your
kickstart file). There are two things that can stop a file being
written: either the overlay is full, or the filesystem is full. If you
set the partition size to a smaller value, you'll get filesystem errors,
which are probably going to be less severe.
Thanks for that! I was unaware of this aspect of the setup, but it makes
sense.
Of course, if you have a small partition size and a huge overlay,
then
most of your overlay will not ever be usable, so you want to play with
the two things together. As a rough idea, the free space in your
partition (after everything's been installed) should be a bit less than
the overlay size.
Hmmm... presently I only specify the partition size in the kickstart. I
do not use the --overlay-size-mb <size> option with livecd-iso-to-disk. I
don't recall how I made that decision now, but I seem to recall there once
being documentation stating that roughly half the RAM would be used for
the overlay in this case, as a default. IIRC, I went the "default" route
(i.e., didn't specify --overlay-size-mb) because the devices vary
considerably in the amount of RAM they have, from lowly 1GB Geode based
PC/104 systems to 32GB Dell rackmount servers.
Do you have any advice how I should proceed given that variability and the
desire to avoid the ungraceful failure case I've mentioned? If I
understand you correctly, I would want to limit my partition size such
that the "free" space (i.e., partition size minus installed read-only
image size) to be less than my overlay size, which I'm believing is half
of all RAM. Thus I'd probably want to use a fixed --overlay-size-mb value
based on the smallest 1GB hardware. That in turn would mean the larger
devices are "artificially" crippled for file system COW capacity, but of
course that extra RAM would be free for process.
Am I on the right track?
There are potential pitfalls here, because free filesystem space
doesn't
quite equate with free overlay space. I don't know what happens if you
boot up and delete a large file that was installed in the squashfs--it
might increase the free space as far as the filesystem is concerned, but
it obviously won't buy you any extra overlay space.
Good stuff to keep in mind. For now, I think I'll worry about the above
concerns and then come back to this once I've got the "simple" case down
better.
Thanks!
--
John Florian
5:55 p.m.
On Thu, Mar 7, 2013 at 10:41 AM, <John.Florian(a)dart.biz> wrote:
> From: Frederick Grose <fgrose(a)gmail.com>
> On Wed, Mar 6, 2013 at 3:59 PM, <John.Florian(a)dart.biz> wrote:
<snip>
> root@aos-61:46 # # Lets now make it all go wonky:
> root@aos-61:46 # time dd if=/dev/zero of=/foo
> Bus error
>
> real 1m15.775s
> user 0m2.818s
> sys 0m24.129s
> root@aos-61:46 #
> root@aos-61:46 # ls /root
> -bash: /bin/ls: Input/output error
> root@aos-61:46 # df -h
> -bash: /usr/bin/df: Input/output error
>
> root@aos-61:46 # mount
>
> -bash: /usr/bin/mount: Input/output error
>
> root@aos-61:46 # cat /proc/meminfo
>
> -bash: /usr/bin/cat: Input/output error
>
>
> Is this expected? Is there anything I can do, e.g., configuration-
> wise, that can prevent this? Ideally this would fail much like any
> other full disk situation. I understand that the overlay consumes
> space, i.e., memory, for this file growth, including file removals,
> but I'd at least like to be able to remotely reboot a system when in
> this state, however I can't even do that because the reboot command
> will either return the same I/O error or it may succeed but get the
> I/O error when systemd tries to read
/usr/lib/systemd/system/reboot.target.
>
> I dug around in bugzilla, but found nothing there. I can file a
> bug, but which package is likely at fault here?
> --
> John Florian
>
> See https://fedoraproject.org/wiki/LiveOS_image for some background
> and potential workarounds.
>
> --Fred --
There's really not much on that page that helps me here. I'm trying to
use Live images for a mostly-stateless embedded appliance OS deployed to
hundreds or thousands of devices. I realize that the COW design is always
going to be limited, but a more graceful failure mode is really needed,
somehow. For our use, the biggest gain in stability here actually comes
from systemd's journal with its trim-before-write approach instead of the
legacy write now, trim asynchronously approach we used to have. However,
that only covers one specific use case: logging. Writing to proper
persistent storage allows me to avoid the root file system overlay, but
most of these embedded devices use CF or SD cards for storage, which have
limited write cycles that must be respected.
Is there a way to implement an artificial capacity limit that would
prevent processes from exhausting the overlay so that the reserve might be
used for recording the event and rebooting back to a safer state?
At the very least, I think this page could benefit from a little stronger,
more explicit wording of this failure case. While it talks a little about
some work-arounds, it actually says very little about why they are needed.
Only in the "Overlay Recovery" section does it hint at the crash potential.
--
John Florian
Thank you for the review! I've updated the wiki page based on your
comments,
https://fedoraproject.org/wiki/LiveOS_image
Documenting that a temporary overlay is a 0.5 GiB sparse file in a RAM
filesystem gave me the idea to try using an overlay size greater than
available memory, and hope that kernel out-of-memory warnings would
intervene before the device-mapper filesystem invalidation.
I modified /usr/sbin/dmsquash-live-root in the initramfs to create a
temporary 500 GiB sparse overlay:
dd if=/dev/null of=/overlay bs=1024 count=1 seek=$((512*1024*1024)) 2>
/dev/null
Then after booting an updated, Fedora 18 Live desktop, LiveUSB read only
and running your failure demo,
time dd if=/dev/zero of=/foo
I got out-of-memory warnings after a file of about 450 MiB was written and
the command returned--no crash!
Some post test output:
[root@localhost ~]# dmsetup status
live-osimg-min: 0 8388608 snapshot 2584/2584 24
live-rw: 0 8388608 snapshot 921720/1073741824 3600
top - 18:11:53 up 17 min, 3 users, load average: 0.68, 0.75, 0.57
Tasks: 182 total, 2 running, 180 sleeping, 0 stopped, 0 zombie
%Cpu(s): 1.6 us, 1.6 sy, 0.0 ni, 96.5 id, 0.0 wa, 0.2 hi, 0.0 si,
0.0 st
KiB Mem: 3339812 total, 3260284 used, 79528 free, 316384 buffers
KiB Swap: 3341308 total, 0 used, 3341308 free, 1948108 cached
You might test this method in your systems and let us know how it works.
--Fred
12:18 a.m.
On Fri, Mar 8, 2013 at 6:55 PM, Frederick Grose <fgrose(a)gmail.com> wrote:
On Thu, Mar 7, 2013 at 10:41 AM, <John.Florian(a)dart.biz>
wrote:
> > From: Frederick Grose <fgrose(a)gmail.com>
> > On Wed, Mar 6, 2013 at 3:59 PM, <John.Florian(a)dart.biz> wrote:
> <snip>
> > root@aos-61:46 # # Lets now make it all go wonky:
> > root@aos-61:46 # time dd if=/dev/zero of=/foo
> > Bus error
> >
> > real 1m15.775s
> > user 0m2.818s
> > sys 0m24.129s
> > root@aos-61:46 #
> > root@aos-61:46 # ls /root
> > -bash: /bin/ls: Input/output error
> > root@aos-61:46 # df -h
> > -bash: /usr/bin/df: Input/output error
> >
> > root@aos-61:46 # mount
> >
> > -bash: /usr/bin/mount: Input/output error
> >
> > root@aos-61:46 # cat /proc/meminfo
> >
> > -bash: /usr/bin/cat: Input/output error
> >
> >
> > Is this expected? Is there anything I can do, e.g., configuration-
> > wise, that can prevent this? Ideally this would fail much like any
> > other full disk situation. I understand that the overlay consumes
> > space, i.e., memory, for this file growth, including file removals,
> > but I'd at least like to be able to remotely reboot a system when in
> > this state, however I can't even do that because the reboot command
> > will either return the same I/O error or it may succeed but get the
> > I/O error when systemd tries to read
> /usr/lib/systemd/system/reboot.target.
> >
> > I dug around in bugzilla, but found nothing there. I can file a
> > bug, but which package is likely at fault here?
> > --
> > John Florian
> >
> > See https://fedoraproject.org/wiki/LiveOS_image for some background
> > and potential workarounds.
> >
> > --Fred --
>
>
> There's really not much on that page that helps me here. I'm trying to
> use Live images for a mostly-stateless embedded appliance OS deployed to
> hundreds or thousands of devices. I realize that the COW design is always
> going to be limited, but a more graceful failure mode is really needed,
> somehow. For our use, the biggest gain in stability here actually comes
> from systemd's journal with its trim-before-write approach instead of the
> legacy write now, trim asynchronously approach we used to have. However,
> that only covers one specific use case: logging. Writing to proper
> persistent storage allows me to avoid the root file system overlay, but
> most of these embedded devices use CF or SD cards for storage, which have
> limited write cycles that must be respected.
>
> Is there a way to implement an artificial capacity limit that would
> prevent processes from exhausting the overlay so that the reserve might be
> used for recording the event and rebooting back to a safer state?
>
> At the very least, I think this page could benefit from a little
> stronger, more explicit wording of this failure case. While it talks a
> little about some work-arounds, it actually says very little about why they
> are needed. Only in the "Overlay Recovery" section does it hint at the
> crash potential.
>
> --
> John Florian
>
Thank you for the review! I've updated the wiki page based on your
comments,
https://fedoraproject.org/wiki/LiveOS_image
Documenting that a temporary overlay is a 0.5 GiB sparse file in a RAM
filesystem gave me the idea to try using an overlay size greater than
available memory, and hope that kernel out-of-memory warnings would
intervene before the device-mapper filesystem invalidation.
I modified /usr/sbin/dmsquash-live-root in the initramfs to create a
temporary 500 GiB sparse overlay:
dd if=/dev/null of=/overlay bs=1024 count=1 seek=$((512*1024*1024)) 2>
/dev/null
Then after booting an updated, Fedora 18 Live desktop, LiveUSB read only
and running your failure demo,
time dd if=/dev/zero of=/foo
I got out-of-memory warnings after a file of about 450 MiB was written and
the command returned--no crash!
Some post test output:
[root@localhost ~]# dmsetup status
live-osimg-min: 0 8388608 snapshot 2584/2584 24
live-rw: 0 8388608 snapshot 921720/1073741824 3600
top - 18:11:53 up 17 min, 3 users, load average: 0.68, 0.75, 0.57
Tasks: 182 total, 2 running, 180 sleeping, 0 stopped, 0 zombie
%Cpu(s): 1.6 us, 1.6 sy, 0.0 ni, 96.5 id, 0.0 wa, 0.2 hi, 0.0 si,
0.0 st
KiB Mem: 3339812 total, 3260284 used, 79528 free, 316384 buffers
KiB Swap: 3341308 total, 0 used, 3341308 free, 1948108 cached
You might test this method in your systems and let us know how it works.
--Fred
Pardon my bad observations, my above conclusion IS WRONG and unsupported by
the above test.
I deceived myself with an unfamiliar error message, and actually seem to
have tested James Heather's method in my last test.
My root filesystem size was 4 GiB with about 450 MiB free. An
out-of-disc-space warning is what actually popped up and caused the test
command to exit before another failure or crash.
To retest the oversized overlay hypothesis, I resized the LiveUSB root
filesystem to 12 GiB and repeated the test on it as an attached LiveOS
filesystem, /dev/mapper/dm-PCBV6p (mounted at /mnt/a).
[root@localhost a]# dmsetup status
dm-PCBV6p: 0 25165824 snapshot 24/1073741824 16
[root@localhost ~]# dmsetup table
dm-PCBV6p: 0 25165824 snapshot 7:9 7:10 P 8
[root@localhost ~]# losetup -a
/dev/loop8: [2081]:1833 (/run/media/fgrose/LIVE/LiveOS/squashfs.img)
/dev/loop9: [1800]:3 (/run/media/livemnt-squash-mRJNIA/LiveOS/rootfs.img)
/dev/loop10: [0017]:58601 (/run/media/tmpvJjuX7)
/dev/loop11: [2081]:1832 (/run/media/fgrose/LIVE/LiveOS/home.img)
[root@localhost ~]# df -Th
Filesystem Type Size Used Avail Use% Mounted on
devtmpfs devtmpfs 1.6G 0 1.6G 0% /dev
tmpfs tmpfs 1.6G 152K 1.6G 1% /dev/shm
tmpfs tmpfs 1.6G 3.3M 1.6G 1% /run
tmpfs tmpfs 1.6G 0 1.6G 0% /sys/fs/cgroup
/dev/sda1 ext4 18G 8.8G 7.7G 54% /
tmpfs tmpfs 1.6G 28K 1.6G 1% /tmp
/dev/sdc1 vfat 15G 8.8G 6.2G 59% /run/media/fgrose/LIVE
/dev/loop8 squashfs 929M 929M 0 100%
/run/media/livemnt-squash-mRJNIA
/dev/mapper/dm-PCBV6p ext4 12G 3.4G 8.2G 30% /mnt/a
/dev/loop11 ext4 380M 35M 325M 10% /mnt/a/home
[root@localhost ~]# mount
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime,seclabel)
devtmpfs on /dev type devtmpfs
(rw,nosuid,seclabel,size=1652304k,nr_inodes=413076,mode=755)
securityfs on /sys/kernel/security type securityfs
(rw,nosuid,nodev,noexec,relatime)
selinuxfs on /sys/fs/selinux type selinuxfs (rw,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,seclabel)
devpts on /dev/pts type devpts
(rw,nosuid,noexec,relatime,seclabel,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,seclabel,mode=755)
tmpfs on /sys/fs/cgroup type tmpfs
(rw,nosuid,nodev,noexec,seclabel,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup
(rw,nosuid,nodev,noexec,relatime,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)
cgroup on /sys/fs/cgroup/cpuset type cgroup
(rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup
(rw,nosuid,nodev,noexec,relatime,cpuacct,cpu)
cgroup on /sys/fs/cgroup/memory type cgroup
(rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/devices type cgroup
(rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/freezer type cgroup
(rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/net_cls type cgroup
(rw,nosuid,nodev,noexec,relatime,net_cls)
cgroup on /sys/fs/cgroup/blkio type cgroup
(rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/perf_event type cgroup
(rw,nosuid,nodev,noexec,relatime,perf_event)
/dev/sda1 on / type ext4 (rw,relatime,seclabel,data=ordered)
rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs
(rw,relatime,fd=37,pgrp=1,timeout=300,minproto=5,maxproto=5,direct)
configfs on /sys/kernel/config type configfs (rw,relatime)
mqueue on /dev/mqueue type mqueue (rw,relatime,seclabel)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
tmpfs on /tmp type tmpfs (rw,seclabel)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,seclabel)
gvfsd-fuse on /run/user/1000/gvfs type fuse.gvfsd-fuse
(rw,nosuid,nodev,relatime,user_id=1000,group_id=1000)
/dev/sdc1 on /run/media/fgrose/LIVE type vfat
(rw,nosuid,nodev,relatime,uid=1000,gid=1000,fmask=0022,dmask=0077,codepage=437,iocharset=ascii,shortname=mixed,showexec,utf8,flush,errors=remount-ro,uhelper=udisks2)
/dev/sdb2 on /var/cache/yum type ext4 (rw,relatime,seclabel,data=ordered)
/dev/loop8 on /run/media/livemnt-squash-mRJNIA type squashfs
(ro,relatime,seclabel)
/dev/mapper/dm-PCBV6p on /mnt/a type ext4
(rw,relatime,seclabel,data=ordered)
/dev/loop11 on /mnt/a/home type ext4 (rw,relatime,seclabel,data=ordered)
/dev/sdc1 on /mnt/a/run/initramfs/live type vfat
(rw,nosuid,nodev,relatime,uid=1000,gid=1000,fmask=0022,dmask=0077,codepage=437,iocharset=ascii,shortname=mixed,showexec,utf8,flush,errors=remount-ro)
The target filesystem, /dev/mapper/dm-PCBV6p, did go invalid, was changed
to ro, which led to this test command output:
[root@localhost a]# time dd if=/dev/zero of=foo
dd: writing to ‘foo’: Read-only file system
4029694+0 records in
4029693+0 records out
2063202816 bytes (2.1 GB) copied, 40.0696 s, 51.5 MB/s
real 0m40.079s
user 0m3.799s
sys 0m32.422s
In a separate terminal I manually monitored the dmsetup status:
[root@localhost ~]# dmsetup status
dm-PCBV6p: 0 25165824 snapshot 184/1073741824 16
[root@localhost ~]# dmsetup status
dm-PCBV6p: 0 25165824 snapshot 192/1073741824 16
[root@localhost ~]# dmsetup status
dm-PCBV6p: 0 25165824 snapshot 192/1073741824 16
[root@localhost ~]# dmsetup status
dm-PCBV6p: 0 25165824 snapshot 263440/1073741824 1040
[root@localhost ~]# dmsetup status
dm-PCBV6p: 0 25165824 snapshot 366360/1073741824 1440
[root@localhost ~]# dmsetup status
dm-PCBV6p: 0 25165824 snapshot 526608/1073741824 2064
[root@localhost ~]# dmsetup status
dm-PCBV6p: 0 25165824 snapshot 703280/1073741824 2752
[root@localhost ~]# dmsetup status
dm-PCBV6p: 0 25165824 snapshot 904600/1073741824 3528
[root@localhost ~]# dmsetup status
dm-PCBV6p: 0 25165824 snapshot 1131568/1073741824 4416
[root@localhost ~]# dmsetup status
dm-PCBV6p: 0 25165824 snapshot 1383288/1073741824 5392
[root@localhost ~]# dmsetup status
dm-PCBV6p: 0 25165824 snapshot 1579432/1073741824 6160
[root@localhost ~]# dmsetup status
dm-PCBV6p: 0 25165824 snapshot 1579432/1073741824 6160
[root@localhost ~]# dmsetup status
dm-PCBV6p: 0 25165824 snapshot 1731568/1073741824 6752
[root@localhost ~]# dmsetup status
dm-PCBV6p: 0 25165824 snapshot 2191040/1073741824 8536
[root@localhost ~]# dmsetup status
dm-PCBV6p: 0 25165824 snapshot 2420840/1073741824 9432
[root@localhost ~]# dmsetup status
dm-PCBV6p: 0 25165824 snapshot 2632232/1073741824 10256
[root@localhost ~]# dmsetup status
dm-PCBV6p: 0 25165824 snapshot 2632288/1073741824 10256
[root@localhost ~]# dmsetup status
dm-PCBV6p: 0 25165824 snapshot 2964616/1073741824 11544
[root@localhost ~]# dmsetup status
dm-PCBV6p: 0 25165824 snapshot 3208632/1073741824 12496
[root@localhost ~]# dmsetup status
dm-PCBV6p: 0 25165824 snapshot Invalid
[root@localhost ~]# df -Th
Filesystem Type Size Used Avail Use% Mounted on
devtmpfs devtmpfs 1.6G 0 1.6G 0% /dev
tmpfs tmpfs 1.6G 504K 1.6G 1% /dev/shm
tmpfs tmpfs 1.6G 1000M 632M 62% /run
tmpfs tmpfs 1.6G 0 1.6G 0% /sys/fs/cgroup
/dev/sda1 ext4 18G 8.7G 7.7G 54% /
tmpfs tmpfs 1.6G 68K 1.6G 1% /tmp
/dev/sdc1 vfat 15G 8.8G 6.2G 59% /run/media/fgrose/LIVE
/dev/loop8 squashfs 929M 929M 0 100%
/run/media/livemnt-squash-mRJNIA
/dev/mapper/dm-PCBV6p ext4 12G 4.6G 7.1G 40% /mnt/a
/dev/loop11 ext4 380M 35M 325M 10% /mnt/a/home
The invalidation occurred at the 1.6 GB size limit applied to the /run
tmpfs where the overlay, /dev/loop10, was mounted,
[root@localhost ~]# losetup /dev/loop10
/dev/loop10: [0017]:58601 (/run/media/tmpvJjuX7)
[root@localhost ~]# ls /mnt/a
ls: cannot access /mnt/a/.readahead: Input/output error
top - 00:26:06 up 13 min, 4 users, load average: 0.55, 0.68, 0.44
Tasks: 204 total, 2 running, 202 sleeping, 0 stopped, 0 zombie
%Cpu(s): 4.3 us, 1.6 sy, 0.0 ni, 92.4 id, 1.3 wa, 0.3 hi, 0.0 si,
0.0 st
KiB Mem: 3339812 total, 3256176 used, 83636 free, 68956 buffers
KiB Swap: 3341308 total, 0 used, 3341308 free, 2312664 cached
Notice that Swap was not activated, but free memory got down to ~83 MiB.
When I tested the above on the booted LiveUSB, 2-3 GiB of swap was
activated before the fatal crash.
So an oversized overlay DOES NOT prevent device-mapper invalidation by the
above test method.
--Fred
8:31 a.m.
From: Frederick Grose <fgrose(a)gmail.com>
Thank you for the review! I've updated the wiki page based on
your
comments,
That is much improved, but it may still leave a question in the reader's
mind. I'd change the page myself (assuming I can), but I don't want to
embed false information. I mean, I use livecd-tools a LOT, but to say I'm
an expert for all of its use cases would be quite a stretch. So, I'd
propose something like this:
Critical limitation
LiveOS overlays are employed as write-once files, which always consume
space—never releasing space for reuse. Should the overlay storage space,
whether temporary or persistent, be totally consumed, the system will
likely crash with Input/output or Bus errors. If such a crash does occur
while using temporary storage space for the overlay, a simple reboot will
rectify the situation. With persistent storage the situation is more dire
and will require ... (I don't know this case at all). In either case,
achieving the reboot likely will require a hard reset since attempting a
software initiated reboot will probably fail with more Input/output or Bus
errors.
Thank you for improving this page! It would be awesome to get knowledge
unique to this page incorporated back into the appropriate man pages too.
--
John Florian
8:37 a.m.
Critical limitation
LiveOS overlays are employed as write-once files, which always consume
space—never releasing space for reuse. Should the overlay storage
space, whether temporary or persistent, be totally consumed, the
system will likely crash with Input/output or Bus errors. If such a
crash does occur while using temporary storage space for the overlay,
a simple reboot will rectify the situation. With persistent storage
the situation is more dire and will require ... (I don't know this
case at all). In either case, achieving the reboot likely will
require a hard reset since attempting a software initiated reboot will
probably fail with more Input/output or Bus errors.
... will require appending "reset_overlay" (think that's right) to the
kernel command line on boot-up, which will reset the state to its
initial boot (though it won't reset a persistent home, of course).
It's not quite true that a live overlay will never release space. I
think if you created a file that got stored in the overlay, and then you
deleted it, you'd reclaim the space. But what it can't do is reclaim
space that's used by the squashfs image: deleting things from there will
do something to mark them as deleted in the filesystem, but you won't be
able to use the space, since it wasn't ever taking up any overlay space
in the first place.
I think the overlay operates at the sector level, but I'm not certain of
that.
James
9:09 a.m.
From: James Heather <j.heather(a)surrey.ac.uk>
To: <livecd(a)lists.fedoraproject.org>
Date: 03/11/2013 09:37
Subject: Re: [Fedora-livecd-list] "Input/output error" and/or "Bus
error" for most commands
Sent by: livecd-bounces(a)lists.fedoraproject.org
> Critical limitation
> LiveOS overlays are employed as write-once files, which always consume
> space—never releasing space for reuse. Should the overlay storage
> space, whether temporary or persistent, be totally consumed, the
> system will likely crash with Input/output or Bus errors. If such a
> crash does occur while using temporary storage space for the overlay,
> a simple reboot will rectify the situation. With persistent storage
> the situation is more dire and will require ... (I don't know this
> case at all). In either case, achieving the reboot likely will
> require a hard reset since attempting a software initiated reboot will
> probably fail with more Input/output or Bus errors.
... will require appending "reset_overlay" (think that's right) to the
kernel command line on boot-up, which will reset the state to its
initial boot (though it won't reset a persistent home, of course).
I've updated the page with this much and confirmed/noted that
"reset_overlay" is the former (but now deprecated) form of
"rd.live.overlay.reset". Please review/adjust as necessary.
It's not quite true that a live overlay will never release space.
I
think if you created a file that got stored in the overlay, and then you
deleted it, you'd reclaim the space. But what it can't do is reclaim
space that's used by the squashfs image: deleting things from there will
do something to mark them as deleted in the filesystem, but you won't be
able to use the space, since it wasn't ever taking up any overlay space
in the first place.
I think the overlay operates at the sector level, but I'm not certain of
that.
James
I left this bit out. It's good info, but sounds like a bit of certainty
could be injected. ;-)
Also along these lines, a coworker was playing with the 'discard' mount
option to see if the kernel's TRIM support would help here. Testing
showed that it does not and further research discovered kernel mailing
list comments stating clearly that this feature is not yet available for
the COW images.
--
John Florian
4061
days inactive
4066
days old
livecd@lists.fedoraproject.org
9 comments
3 participants
participants (3)
-
Frederick Grose -
James Heather -
John.Florian＠dart.biz