6:49 a.m.
I currently have a pair of external drives configured as ext4 with
RAID1 using mdadm, and mainly used for backup. My / and /home
filesystems are now BTRFS so I'm looking at converting the external
drives to BTRFS with RAID1. My main reason is to take advantage of
BTRFS checksumming as a guard against bitrot, but I'd also like the
flexibility of setting up subvolumes with different properties (the
disks are currently 90% empty).
Any thoughts on this? What would be the simplest conversion strategy if
I go ahead?
poc
8:17 a.m.
On 31/03/2021 19:49, Patrick O'Callaghan wrote:
I currently have a pair of external drives configured as ext4 with
RAID1 using mdadm, and mainly used for backup. My / and /home
filesystems are now BTRFS so I'm looking at converting the external
drives to BTRFS with RAID1. My main reason is to take advantage of
BTRFS checksumming as a guard against bitrot, but I'd also like the
flexibility of setting up subvolumes with different properties (the
disks are currently 90% empty).
Any thoughts on this? What would be the simplest conversion strategy if
I go ahead?
Just a thought.... If the drives are 90% empty and used mainly for backup and there
is sufficient space on the local storage for the remainder I'd take the path of least
resistance.
I'd copy the "non-backedup" stuff to local storage and then blow the rest
away and
create the raid/btrfs volumes from scratch. After that is done, move the stuff back
and do a backup. I'm lazy. And if possible, I'd rather not do something I'm
unlikely to
have to do again in the future. :-) :-)
--
Remind me to ignore comments which aren't germane to the thread.
3:25 p.m.
On Wed, 2021-03-31 at 21:17 +0800, Ed Greshko wrote:
On 31/03/2021 19:49, Patrick O'Callaghan wrote:
> I currently have a pair of external drives configured as ext4 with
> RAID1 using mdadm, and mainly used for backup. My / and /home
> filesystems are now BTRFS so I'm looking at converting the external
> drives to BTRFS with RAID1. My main reason is to take advantage of
> BTRFS checksumming as a guard against bitrot, but I'd also like the
> flexibility of setting up subvolumes with different properties (the
> disks are currently 90% empty).
>
> Any thoughts on this? What would be the simplest conversion
> strategy if
> I go ahead?
Just a thought.... If the drives are 90% empty and used mainly for
backup and there
is sufficient space on the local storage for the remainder I'd take
the path of least resistance.
I'd copy the "non-backedup" stuff to local storage and then blow the
rest away and
create the raid/btrfs volumes from scratch. After that is done, move
the stuff back
and do a backup. I'm lazy. And if possible, I'd rather not do
something I'm unlikely to
have to do again in the future. :-) :-)
Yes, pretty much what I had in mind.
poc
7 p.m.
Nothing to add but the usual caveats:
https://raid.wiki.kernel.org/index.php/Timeout_Mismatch
I use udev for that instead of init scripts. Concept is the same
though, you want SCT ERC time to be shorter than kernel's command
timer.
The btrfsmaintenance package has a scrub.timer that you can configure
and enable. It's sufficient to scrub once a month or two, but doesn't
hurt to do it more often.
There is a nagios btrfs plugin somewhere for monitoring. Otherwise
something that parses 'btrfs device stats' for value changes would
also work.
A rather different thing about Btrfs compared to mdadm raid, is
there's no concept of faulty drives, i.e. btrfs doesn't "kick" drives
out. Since it can unambiguously know if any block is corrupt or not,
it keeps a pesky failing drive in, and just complains (a lot) about
the bad blocks while using the good blocks.
Device replacement should use 'btrfs replace' rather than 'btrfs
device add/remove'.
More info in 'man mkfs.btrfs' and 'man 5 btrfs'.
It's possible to simulate various problems with loop devices or in a
VM. A fun one is setting up dm-flakey for one of the mirrors. A bit
easier to chew off is to compile btrfs-corrupt-block.c from
btrfs-progs source (it is not included in the btrfs-progs package in
Fedora) which has various options for corrupting data and metadata,
ability to choose which copy gets the damage, etc. It can be
surprising how verbose btrfs is with just one data block is corrupt,
when shared among multiple snapshots. It'll tell you about all of the
instances of the files sharing that one bad block.
--
Chris Murphy
6:52 a.m.
On Wed, 2021-03-31 at 18:00 -0600, Chris Murphy wrote:
Nothing to add but the usual caveats:
https://raid.wiki.kernel.org/index.php/Timeout_Mismatch
That´s pretty scary, though the drives I´m using are 1TB units
scavenged from my extinct NAS so are unlikely to be SMR. They´re both
WD model WD10EZEX drives.
I use udev for that instead of init scripts. Concept is the same
though, you want SCT ERC time to be shorter than kernel's command
timer.
I´ve been using MD for a while and haven´t seen any errors so far.
The btrfsmaintenance package has a scrub.timer that you can
configure
and enable. It's sufficient to scrub once a month or two, but doesn't
hurt to do it more often.
Yes, I have that enabled already, running once a month.
Thanks.
poc
12:52 a.m.
On Thu, Apr 1, 2021 at 5:53 AM Patrick O'Callaghan
<pocallaghan(a)gmail.com> wrote:
On Wed, 2021-03-31 at 18:00 -0600, Chris Murphy wrote:
> Nothing to add but the usual caveats:
> https://raid.wiki.kernel.org/index.php/Timeout_Mismatch
That´s pretty scary, though the drives I´m using are 1TB units
scavenged from my extinct NAS so are unlikely to be SMR. They´re both
WD model WD10EZEX drives.
It's not an SMR concern, it's making sure the drive gives up on errors
faster than the kernel tries to reset due to what it thinks is a
hanging drive.
smartctl -l scterc /dev/sdX
That'll tell you the default setting. I'm pretty sure Blues come with
SCT ERC disabled. Some support it. Some don't. If it's supported
you'll want to set it for something like 70-100 deciseconds (the units
SATA drives use for this feature).
And yeah, linux-raid@ list is chock full of such misconfigurations. It
filters out all the lucky people, and the unlucky people end up on the
list with a big problem which generally looks like this: one dead
drive, and one of the surviving drives with one bad sector that was
never fixed up through normal raid bad sector recovery mechanism,
because the kernel's default is to be impatient and do a link reset on
consumer drives that overthink a simple problem. Upon link reset, the
entire command queue in the drive is lost, and now there's no way to
know what sector it was hanging on, and no way for raid to do a fixup.
The fixup mechanism is, the drive reports an uncorrectable read error
with a sector address *only once it gives up*. And then the md raid
(and btrfs and zfs) can go lookup that sector, find out what data is
on it, go find its mirror, read the good data, and overwrite the bad
sector with good data. The overwrite is what fixes the problem.
If the drive doesn't support SCT ERC, we have to get the kernel to be
more patient. That's done via sysfs.
> I use udev for that instead of init scripts. Concept is the same
> though, you want SCT ERC time to be shorter than kernel's command
> timer.
I´ve been using MD for a while and haven´t seen any errors so far.
And you may never see it. Or you may end up being an unlucky person
with a raid who experiences complete loss of the array. When I say
comes up all the time on linux-raid@ list, it's about once every
couple of weeks. It's seen most often with raid5 because it has more
drives, thus more failures, than raid1 setups. And tolerates only one
failure *in a stripe*. Most everyone considers a failure a complete
drive failure, but drives also partially fail. Two drives partially
failing the sectors in the same stripe is pretty astronomical. But if
one drive dies, and *any* of the remaining drives has a bad sector
that can't be read, the entire stripe is lost. And depending on what's
in that stripe, it can bring down the array.
So, what you want is for the drives to report their errors, rather
than the kernel doing link resets.
--
Chris Murphy
5:23 a.m.
On Thu, 2021-04-01 at 23:52 -0600, Chris Murphy wrote:
It's not an SMR concern, it's making sure the drive gives up
on
errors
faster than the kernel tries to reset due to what it thinks is a
hanging drive.
smartctl -l scterc /dev/sdX
That'll tell you the default setting. I'm pretty sure Blues come with
SCT ERC disabled. Some support it. Some don't. If it's supported
you'll want to set it for something like 70-100 deciseconds (the
units
SATA drives use for this feature).
One doesn´t and one does:
# smartctl -l scterc /dev/sdd
smartctl 7.2 2021-01-17 r5171 [x86_64-linux-5.11.10-200.fc33.x86_64]
(local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke,
www.smartmontools.org
SCT Error Recovery Control command not supported
# smartctl -l scterc /dev/sde
smartctl 7.2 2021-01-17 r5171 [x86_64-linux-5.11.10-200.fc33.x86_64]
(local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke,
www.smartmontools.org
SCT Error Recovery Control:
Read: 85 (8.5 seconds)
Write: 85 (8.5 seconds)
So I guess the /dev/sde drive is set correctly, right? Or would you
recommend disabling SCT ERC for this drive?
poc
2:12 p.m.
On Fri, Apr 2, 2021 at 4:23 AM Patrick O'Callaghan
<pocallaghan(a)gmail.com> wrote:
On Thu, 2021-04-01 at 23:52 -0600, Chris Murphy wrote:
> It's not an SMR concern, it's making sure the drive gives up on
> errors
> faster than the kernel tries to reset due to what it thinks is a
> hanging drive.
>
> smartctl -l scterc /dev/sdX
>
> That'll tell you the default setting. I'm pretty sure Blues come with
> SCT ERC disabled. Some support it. Some don't. If it's supported
> you'll want to set it for something like 70-100 deciseconds (the
> units
> SATA drives use for this feature).
One doesn´t and one does:
# smartctl -l scterc /dev/sdd
smartctl 7.2 2021-01-17 r5171 [x86_64-linux-5.11.10-200.fc33.x86_64]
(local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke,
www.smartmontools.org
SCT Error Recovery Control command not supported
# smartctl -l scterc /dev/sde
smartctl 7.2 2021-01-17 r5171 [x86_64-linux-5.11.10-200.fc33.x86_64]
(local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke,
www.smartmontools.org
SCT Error Recovery Control:
Read: 85 (8.5 seconds)
Write: 85 (8.5 seconds)
So I guess the /dev/sde drive is set correctly, right? Or would you
recommend disabling SCT ERC for this drive?
Leave /dev/sde alone, 85 deciseconds is fine.
Not much can be done with /dev/sdd itself directly. But it is possible
to increase the kernel's command timer for this drive. The usual way
of doing this is via sysfs. I think it can be done with a udev rule
as well, but I'm having a bit of a lapse how to do it. Udev needs to
identify the device by serial number or wwn, but changing the timeout
via sysfs requires knowing that the /dev node is - which of course can
change each time you boot or plug the device in. I don't know enough
about udev. But there should be examples on the internet or you can
just fudge it with the linux-raid wiki guide.
The alternatives? Change the timeout for all /dev/ nodes. That's how
things are by default on Windows and macOS, they just wait a long time
before resetting a drive, giving it enough time for it to give up on
its own. The negative side effect is you might get a long delay
without errors, should the device develop marginally bad sectors.
Another alternative is to just leave it alone, and periodically check
(manually or automate it somehow) for the telltale signs of bad
sectors masked by SATA link resets.
Looks like this:
kernel: ata7.00: status: { DRDY }
kernel: ata7.00: failed command: READ FPDMA QUEUED
kernel: ata7.00: cmd
60/40:f0:98:d2:2b/05:00:45:00:00/40 tag 30 ncq dma 688128 in
res
40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
With this interlaced occasionally
kernel: ata7: hard resetting link
If it happens *then* you can increase the timeout manually, and
initiate a scrub. As long as the timeout is set high enough (most
sources suggest 180 seconds which, yes, it's incredible) eventually
the drive will give up, spit out an error, and Btrfs will fix up that
sector by overwriting it with good data. It could be months, years, or
never, before it happens.
--
Chris Murphy
2:29 p.m.
I turn my scterc down as low as the drive will allow. How low I can
go varies by model. I have a loop that starts at 70 and then keeps
going down such that it will end up setting each disk as low is
allowed as far down as 10. My wd reds allow a min of 20, and I have a
seagate that allows 10.
But even set to 10 (1.0 sec) at a slow 15ms/retry that is 66 retries
reading the block, if it has not got it in that many tries then the
drive might as well give up and let mdadm or something rewrite good
data to the disk.
On Fri, Apr 2, 2021 at 2:13 PM Chris Murphy <lists(a)colorremedies.com> wrote:
>
> On Fri, Apr 2, 2021 at 4:23 AM Patrick O'Callaghan
> <pocallaghan(a)gmail.com> wrote:
> >
> > On Thu, 2021-04-01 at 23:52 -0600, Chris Murphy wrote:
> > > It's not an SMR concern, it's making sure the drive gives up on
> > > errors
> > > faster than the kernel tries to reset due to what it thinks is a
> > > hanging drive.
> > >
> > > smartctl -l scterc /dev/sdX
> > >
> > > That'll tell you the default setting. I'm pretty sure Blues come
with
> > > SCT ERC disabled. Some support it. Some don't. If it's supported
> > > you'll want to set it for something like 70-100 deciseconds (the
> > > units
> > > SATA drives use for this feature).
> >
> > One doesn´t and one does:
> >
> > # smartctl -l scterc /dev/sdd
> > smartctl 7.2 2021-01-17 r5171 [x86_64-linux-5.11.10-200.fc33.x86_64]
> > (local build)
> > Copyright (C) 2002-20, Bruce Allen, Christian Franke,
> > www.smartmontools.org
> >
> > SCT Error Recovery Control command not supported
> >
> > # smartctl -l scterc /dev/sde
> > smartctl 7.2 2021-01-17 r5171 [x86_64-linux-5.11.10-200.fc33.x86_64]
> > (local build)
> > Copyright (C) 2002-20, Bruce Allen, Christian Franke,
> > www.smartmontools.org
> >
> > SCT Error Recovery Control:
> > Read: 85 (8.5 seconds)
> > Write: 85 (8.5 seconds)
> >
> > So I guess the /dev/sde drive is set correctly, right? Or would you
> > recommend disabling SCT ERC for this drive?
>
> Leave /dev/sde alone, 85 deciseconds is fine.
>
> Not much can be done with /dev/sdd itself directly. But it is possible
> to increase the kernel's command timer for this drive. The usual way
> of doing this is via sysfs. I think it can be done with a udev rule
> as well, but I'm having a bit of a lapse how to do it. Udev needs to
> identify the device by serial number or wwn, but changing the timeout
> via sysfs requires knowing that the /dev node is - which of course can
> change each time you boot or plug the device in. I don't know enough
> about udev. But there should be examples on the internet or you can
> just fudge it with the linux-raid wiki guide.
>
> The alternatives? Change the timeout for all /dev/ nodes. That's how
> things are by default on Windows and macOS, they just wait a long time
> before resetting a drive, giving it enough time for it to give up on
> its own. The negative side effect is you might get a long delay
> without errors, should the device develop marginally bad sectors.
>
> Another alternative is to just leave it alone, and periodically check
> (manually or automate it somehow) for the telltale signs of bad
> sectors masked by SATA link resets.
>
> Looks like this:
>
> kernel: ata7.00: status: { DRDY }
> kernel: ata7.00: failed command: READ FPDMA QUEUED
> kernel: ata7.00: cmd
> 60/40:f0:98:d2:2b/05:00:45:00:00/40 tag 30 ncq dma 688128 in
> res
> 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
>
> With this interlaced occasionally
>
> kernel: ata7: hard resetting link
>
> If it happens *then* you can increase the timeout manually, and
> initiate a scrub. As long as the timeout is set high enough (most
> sources suggest 180 seconds which, yes, it's incredible) eventually
> the drive will give up, spit out an error, and Btrfs will fix up that
> sector by overwriting it with good data. It could be months, years, or
> never, before it happens.
>
>
> --
> Chris Murphy
> _______________________________________________
> users mailing list -- users(a)lists.fedoraproject.org
> To unsubscribe send an email to users-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org
> Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
4:40 p.m.
On Fri, 2021-04-02 at 13:12 -0600, Chris Murphy wrote:
> # smartctl -l scterc /dev/sdd
> smartctl 7.2 2021-01-17 r5171 [x86_64-linux-5.11.10-200.fc33.x86_64]
> (local build)
> Copyright (C) 2002-20, Bruce Allen, Christian Franke,
> www.smartmontools.org
>
> SCT Error Recovery Control command not supported
>
> # smartctl -l scterc /dev/sde
> smartctl 7.2 2021-01-17 r5171 [x86_64-linux-5.11.10-200.fc33.x86_64]
> (local build)
> Copyright (C) 2002-20, Bruce Allen, Christian Franke,
> www.smartmontools.org
>
> SCT Error Recovery Control:
> Read: 85 (8.5 seconds)
> Write: 85 (8.5 seconds)
>
> So I guess the /dev/sde drive is set correctly, right? Or would you
> recommend disabling SCT ERC for this drive?
Leave /dev/sde alone, 85 deciseconds is fine.
OK.
Not much can be done with /dev/sdd itself directly. But it is
possible
to increase the kernel's command timer for this drive. The usual way
of doing this is via sysfs.
Tried that and it seems to work.
I think it can be done with a udev rule
as well, but I'm having a bit of a lapse how to do it. Udev needs to
identify the device by serial number or wwn, but changing the timeout
via sysfs requires knowing that the /dev node is - which of course can
change each time you boot or plug the device in. I don't know enough
about udev. But there should be examples on the internet or you can
just fudge it with the linux-raid wiki guide.
This came up in a Google search:
https://github.com/jonathanunderwood/mdraid-safe-timeouts
It´s for MD arrays, but I´m guessing it could be adapted for BTRFS as
well.
poc
PS One thing that slightly confused me in setting up the RAID array is
that, unlike with MD, there´s no new pseudo-device, and you mount the
array by just mounting one of its components, right? (I´ve already done
it and it works, just checking if I´ve missed something).
5:28 p.m.
On Fri, 2021-04-02 at 13:12 -0600, Chris Murphy wrote:
[...] changing the timeout via sysfs requires knowing that the /dev
node is - which of course can change each time you boot or plug the
device in.
In fact I've already labelled the devices so identifying them can be
done by filtering the output of 'lsblk'. Certainly simpler than the
udev alternative.
poc
1119
days inactive
1121
days old
10 comments
4 participants
participants (4)
-
Chris Murphy -
Ed Greshko -
Patrick O'Callaghan -
Roger Heflin