12:48 p.m.
Hi!
I tried to set up a system using four 3TB disks as a raid6. The disks
are new, no old partitions laying around. I used a USB stick as install
media (Fedora 20 x86_64 Gnome Live).
I chose manual partitioning to get everything as I wanted. When this was
ready I got the error message that I also needed a BIOS boot partition
(I gather that this due to being rather large disks). I tried to create
one of those, but it seemed to only be created on one of the discs in
the raid array, I wanted it created on all disks, just as the raid
partitions. This to be able to boot from any of the disks in the array
in case of a failure. As it is now, if the main disk dies, I can not
start the system at all.
Actually I ended up having three BIOS boot partition on the first disk,
being 2, 1, and 1 MB large. Which is quite annoying. But that's another
problem...
Before submitting a bug report/RFE on this. Is there anyone out there
that have done this, from anaconda? I.e. creating raid partitions and
BIOS boot partitions on ALL disks? If so, how did you do it?
Lars
--
Lars E. Pettersson <lars(a)homer.se>
http://www.sm6rpz.se/
3:56 a.m.
On 08/31/14 21:27, Chris Murphy wrote:
On Aug 28, 2014, at 12:37 PM, Lars E. Pettersson <lars(a)homer.se> wrote:
...
> Yes, that was my intention, raid1 for /boot and raid6 for /
Accidentally I set /boot also to raid6 :) But that can be fixed.
Or do n-way raid1, which will cause /boot to have as many copies as you add member
devices.
Yes, that was what I ended up doing.
I partitioned /dev/sda as I wanted, with space for BIOS boot partition,
and also an EFI partition if I ever would need one (more or less a setup
as described in <https://bugzilla.redhat.com/show_bug.cgi?id=1061478> I
have not done anything to the EFI partiton yet though, it is just there
at the moment). Raid1 for /boot and raid6 for /. I then copied this
partition layout to all the other disks.
After that I started anaconda.
The trickiest part was to figure out how anaconda worked. It took me
almost 3 hours and a lot of "googeling" to figure out how to use the
"unknown" partitions in ananconda (click on the format). I would say
that anaconda was not very intuitive... :)
Post install you can
grep -i grub2-install /var/log/anaconda/program.log
# grep grub2-install /var/log/anaconda/anaconda.program.log
17:56:30,287 INFO program: Running... grub2-install --no-floppy /dev/sda
17:56:36,970 INFO program: Running... grub2-install --no-floppy /dev/sdb
17:56:43,012 INFO program: Running... grub2-install --no-floppy /dev/sdc
17:56:49,195 INFO program: Running... grub2-install --no-floppy /dev/sdd
Splendid! I actually did this myself manually, but good to know that
anaconda also does it.
To know for sure how big of a BIOS Boot you need:
ls -lh /boot/grub2/i386-pc/core.img
Mine's 26KB, as my boot fs is XFS, without LVM or raid. Yours will be a bit bigger to
include mdraid1x.mod and raid6rec.mod and anything else that gets pulled in. I bet
it's much less than 100K actually.
Yep, mine was 31KB.
Well, my little home server is now fully functional, but not with the
help of anaconda :)
Lars
--
Lars E. Pettersson <lars(a)homer.se>
http://www.sm6rpz.se/
2:02 p.m.
On Aug 28, 2014, at 12:21 PM, Bruno Wolff III <bruno(a)wolff.to> wrote:
I think using raid 1 (with the 1.0 header format) can work well for that. There can still
grub issues with having a boot just work, but at least you have the stuff you need
available.
GRUB2 can locate vmlinuz/initramfs on md/mdadm raid0,1,10,5,6, even degraded. There is a
BIOS specific limitation recognizing a certain number of drives. So you kinda have to
watch out for that.
A possible problem with metadata format 0.9 (deprecated) and 1.0 is that the metadata is
at the end of the partition; so in raid1 configurations, one of the partitions can be
mounted normally as if there's no raid. The problem comes when modifying the volume,
the raid metadata isn't updated. So when the raid is later assembled, md has no idea
that one of the raid members is different than the other. This can quickly cause one or
both members to become corrupted. So it's generally better (and recommended) to use
metadata 1.2 format, which offsets the file system just beyond 4K from the start of the
partition. That way a single raid member isn't mountable on its own, you have to first
assemble the raid, then mount the file system.
Chris Murphy
1:21 p.m.
On Thu, Aug 28, 2014 at 11:10:54 -0700,
Rick Stevens <ricks(a)alldigital.com> wrote:
I think you need to reserve some small space on all the drives (and
with 3TB drives you can afford to sacrifice a few MB), use the remainder
as your RAID, and let the system put the boot partition in that reserved
space on the primary drive (the one the BIOS sees as the boot drive).
I think using raid 1 (with the 1.0 header format) can work well for that.
There can still grub issues with having a boot just work, but at least
you have the stuff you need available.
1:37 p.m.
On 08/28/14 20:21, Bruno Wolff III wrote:
On Thu, Aug 28, 2014 at 11:10:54 -0700,
Rick Stevens <ricks(a)alldigital.com> wrote:
>
> I think you need to reserve some small space on all the drives (and
> with 3TB drives you can afford to sacrifice a few MB), use the remainder
> as your RAID, and let the system put the boot partition in that reserved
> space on the primary drive (the one the BIOS sees as the boot drive).
I think using raid 1 (with the 1.0 header format) can work well for
that. There can still grub issues with having a boot just work, but at
least you have the stuff you need available.
Yes, that was my intention, raid1 for /boot and raid6 for / Accidentally
I set /boot also to raid6 :) But that can be fixed.
On my old system I use 1TB disks, with /boot as raid1, and grub
boot-loader installed on all disks. On that one I can boot the system
from any of disks. Which is quite handy.
The problem here seem to be that due to the disks being large (larger
than 1TB) they are setup as GPT (GUID Partition Table), and they then
also need a BIOS boot partition to work on non UEFI based systems (if I
have understood it correctly).
So, to be able to boot from any of the disks, I need a BIOS boot
partition on all disks, but anaconda seem to only install it on one of
the disks (i.e. I want the exactly identical partition tables on all disks).
Lars
--
Lars E. Pettersson <lars(a)homer.se>
http://www.sm6rpz.se/
1:10 p.m.
On 08/28/2014 10:48 AM, Lars E. Pettersson issued this missive:
Hi!
I tried to set up a system using four 3TB disks as a raid6. The disks
are new, no old partitions laying around. I used a USB stick as install
media (Fedora 20 x86_64 Gnome Live).
I chose manual partitioning to get everything as I wanted. When this was
ready I got the error message that I also needed a BIOS boot partition
(I gather that this due to being rather large disks). I tried to create
one of those, but it seemed to only be created on one of the discs in
the raid array, I wanted it created on all disks, just as the raid
partitions. This to be able to boot from any of the disks in the array
in case of a failure. As it is now, if the main disk dies, I can not
start the system at all.
Actually I ended up having three BIOS boot partition on the first disk,
being 2, 1, and 1 MB large. Which is quite annoying. But that's another
problem...
Before submitting a bug report/RFE on this. Is there anyone out there
that have done this, from anaconda? I.e. creating raid partitions and
BIOS boot partitions on ALL disks? If so, how did you do it?
AFAIK, a BIOS boot partition has to be on the primary disk where it can
be found. At boot time, the BIOS no idea about the RAID since it's a
construct of the operating system. Thus the BIOS can't use a partition
on the RAID to boot.
I think you need to reserve some small space on all the drives (and
with 3TB drives you can afford to sacrifice a few MB), use the remainder
as your RAID, and let the system put the boot partition in that reserved
space on the primary drive (the one the BIOS sees as the boot drive).
I prefer hardware RAID for just this reason (e.g. if you lose the
primary drive, it can still boot since the RAID is visible to the
BIOS). Hardware SATA RAID controllers aren't that expensive ($350 US)
for the peace of mind they bring.
This is just my opinion. Your mileage may vary. Void where prohibited.
Batteries not included. Some assembly required. Yadda, yadda.
----------------------------------------------------------------------
- Rick Stevens, Systems Engineer, AllDigital ricks(a)alldigital.com -
- AIM/Skype: therps2 ICQ: 22643734 Yahoo: origrps2 -
- -
- grep me no patterns and I'll tell you no lines -
----------------------------------------------------------------------
2:27 p.m.
On Aug 28, 2014, at 12:37 PM, Lars E. Pettersson <lars(a)homer.se> wrote:
On 08/28/14 20:21, Bruno Wolff III wrote:
> On Thu, Aug 28, 2014 at 11:10:54 -0700,
> Rick Stevens <ricks(a)alldigital.com> wrote:
>>
>> I think you need to reserve some small space on all the drives (and
>> with 3TB drives you can afford to sacrifice a few MB), use the remainder
>> as your RAID, and let the system put the boot partition in that reserved
>> space on the primary drive (the one the BIOS sees as the boot drive).
>
> I think using raid 1 (with the 1.0 header format) can work well for
> that. There can still grub issues with having a boot just work, but at
> least you have the stuff you need available.
Yes, that was my intention, raid1 for /boot and raid6 for / Accidentally I set /boot also
to raid6 :) But that can be fixed.
Or do n-way raid1, which will cause /boot to have as many copies as you add member
devices.
On my old system I use 1TB disks, with /boot as raid1, and grub boot-loader installed on
all disks. On that one I can boot the system from any of disks. Which is quite handy.
That's still possible whether /boot is raid1 (n-way or mirror), or raid6. Anaconda
will run grub2-install against all member devices that have a /boot member device (or at
least it should, I have recently done 2 drive testing on this).
Post install you can
grep -i grub2-install /var/log/anaconda/program.log
And it will show you the command used. It should list all member devices. So long as each
member drive has a BIOS Boot partition, grub2-install will find it automatically and
insert core.img there, as well as the specific jump code in the first 440 bytes of the
(protective) MBR to find it.
The problem here seem to be that due to the disks being large (larger
than 1TB) they are setup as GPT (GUID Partition Table), and they then also need a BIOS
boot partition to work on non UEFI based systems (if I have understood it correctly).
MBR partition entries are 32-bit so they're limited to 2TB. There's a small
advantage to using GPT in that there's a backup header and table. So long as your BIOS
doesn't get fussy with GPT, it's fine to use it even on 1TB drives, but it's
not necessary.
So, to be able to boot from any of the disks, I need a BIOS boot
partition on all disks, but anaconda seem to only install it on one of the disks (i.e. I
want the exactly identical partition tables on all disks).
It's messy. Again the installer should leave the user out of this entirely. You
don't need a BIOS Boot that's exactly 1MB in size, so if you don't want to
start from scratch but there's at least 100K free space on each member drive, you can
create a BIOS boot there and then rerun grub2-install against all the drives, e.g.:
grub2-install /dev/sd[abcde]
To know for sure how big of a BIOS Boot you need:
ls -lh /boot/grub2/i386-pc/core.img
Mine's 26KB, as my boot fs is XFS, without LVM or raid. Yours will be a bit bigger to
include mdraid1x.mod and raid6rec.mod and anything else that gets pulled in. I bet
it's much less than 100K actually.
Chris Murphy
1:51 p.m.
On Aug 28, 2014, at 11:48 AM, Lars E. Pettersson <lars(a)homer.se> wrote:
Hi!
I tried to set up a system using four 3TB disks as a raid6. The disks are new, no old
partitions laying around. I used a USB stick as install media (Fedora 20 x86_64 Gnome
Live).
I chose manual partitioning to get everything as I wanted. When this was ready I got the
error message that I also needed a BIOS boot partition (I gather that this due to being
rather large disks). I tried to create one of those, but it seemed to only be created on
one of the discs in the raid array, I wanted it created on all disks, just as the raid
partitions. This to be able to boot from any of the disks in the array in case of a
failure. As it is now, if the main disk dies, I can not start the system at all.
Precreate the GPT on each drive using fdisk, parted, or gdisk with a 1MB BIOS Boot
partition. Anaconda won't delete it, and you won't have to worry about manually
creating it, which is not easy to figure out. I've lobbied to make EFI System
partition and BIOS boot always created automatically behind the scenes, even in manual
partitioning. There's just no good reason for any user who uses a GUI installer to be
responsible for creating such things.
https://bugzilla.redhat.com/show_bug.cgi?id=1022316
Actually I ended up having three BIOS boot partition on the first disk, being 2, 1, and 1
MB large. Which is quite annoying. But that's another problem...
Before submitting a bug report/RFE on this. Is there anyone out there that have done
this, from anaconda? I.e. creating raid partitions and BIOS boot partitions on ALL disks?
If so, how did you do it?
Yes. I created the BIOS boot partition on each disk in advance.
Chris Murphy
3535
days inactive
3540
days old
7 comments
4 participants
participants (4)
-
Bruno Wolff III -
Chris Murphy -
Lars E. Pettersson -
Rick Stevens