6:40 p.m.
Hello All
As I am trying to install Fedora 23 on our PPC64LE server. I am running into an issue with
install. It has to be noted that Fedora 22/21 fail with similar errors on this
configuration:
Any pointers on narrowing this issue down would be greatly appreciated.
To give details on the box:
A) Stoarage: Avago RAID controller 9361-8i (with 15 solid state drives on RAID0 config)
Firmware version we used is 4.300.00-4429.
B) Network: Mellanox Connectx 3 pro card in the box.
Here are the details. Install log is attached:
1. Install Process gets stuck at the following point during installation: (Full
install logs are attached)
[ 35.253098] mlx4_en: 0002:01:00.0: Port 1: Initializing port
[ 35.263979] mlx4_en 0002:01:00.0: Activating port:2
[ 35.271034] mlx4_en: eth1: Link Up
[ 35.281464] mlx4_en: 0002:01:00.0: Port 2: Using 256 TX rings
[ 35.285573] mlx4_en: 0002:01:00.0: Port 2: Using 8 RX rings
[ 35.291084] mlx4_en: 0002:01:00.0: Port 2: frag:0 - size:1518 prefix:0 stride:1536
[ 35.298236] mlx4_en: 0002:01:00.0: Port 2: Initializing port
[ 35.309191] <mlx4_ib> mlx4_ib_add: mlx4_ib: Mellanox ConnectX InfiniBand driver
v2.2-1 (Feb 2014)
[ 35.327918] <mlx4_ib> mlx4_ib_add: counter index 2 for port 1 allocated 1
[ 35.333372] <mlx4_ib> mlx4_ib_add: counter index 3 for port 2 allocated 1
[ 35.354383] <mlx4_ib> mlx4_ib_add: counter index 2 for port 1 allocated 1
[ 35.358463] <mlx4_ib> mlx4_ib_add: counter index 3 for port 2 allocated 1
[ 35.363710] mlx4_en: eth3: Link Up
[ 35.376411] mlx4_core 0000:01:00.0 enp1s0: renamed from eth0
[ 35.432571] mlx4_core 0000:01:00.0 enp1s0d1: renamed from eth1
[ 35.603694] mlx4_core 0002:01:00.0 enP2p1s0d1: renamed from eth3
[ 35.692421] mlx4_core 0002:01:00.0 enP2p1s0: renamed from eth2
[ 37.691861] sd 5:2:10:0: [sdl] Attached SCSI disk
[ 37.692054] sd 5:2:6:0: [sdh] Attached SCSI disk
[ 37.692520] sd 5:2:13:0: [sdo] Attached SCSI disk
[ 37.692800] sd 5:2:14:0: [sdp] Attached SCSI disk
[ 43.690839] sd 5:2:12:0: [sdn] Attached SCSI disk
[ 43.691142] sd 5:2:8:0: [sdj] Attached SCSI disk
[ 43.691335] sd 5:2:11:0: [sdm] Attached SCSI disk
[ 43.691633] sd 5:2:9:0: [sdk] Attached SCSI disk
[ 49.685080] sd 5:2:5:0: [sdg] Attached SCSI disk
[ 49.685256] sd 5:2:4:0: [sdf] Attached SCSI disk
[ 49.685755] sd 5:2:7:0: [sdi] Attached SCSI disk
2. After getting stuck for about 3-5 minutes, rescue shell kicks in as below. However
as soon as it loads, serial console doesn't respond anymore, resulting in not being
able to collect logs from rescue shell.
[ 235.757258] ]dracut-initqueue[1776]: Warning: Could not boot.
235.758170] dracut-initqueue[1776]: Warning: /dev/root does not exist
Starting Setup Virtual Console...
[ OK ] Started Setup Virtual Console.
[ 236.203610] audit: type=1130 audit(235.750:12): pid=1 uid=0 auid=4294967295 s
es=4294967295 subj=kernel msg='unit=systemd-vconsole-setup comm="systemd"
exe="/
usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Starting Dracut Emergency Shell...
Warning: /dev/root does not exist
Generating "/run/initramfs/rdsosreport.txt"
Entering emergency mode. Exit the shell to continue.
Type "journalctl" to view system logs.
You might want to save "/run/initramfs/rdsosreport.txt" to a USB stick or /boot
after mounting them and attach it to a bug report.
I can add details here as necessary.
Best
Adi Gangidi
Rackspace
5:50 a.m.
----- Original Message -----
From: "Adi Gangidi" <adi.gangidi(a)rackspace.com>
To: ppc(a)lists.fedoraproject.org
Cc: "John Hincher" <john.hincher(a)rackspace.com>, "Aaron
Sullivan" <aaron.sullivan(a)rackspace.com>, "Antony Messerli"
<amesserl(a)rackspace.com>, "Major Hayden"
<major.hayden(a)rackspace.com>
Sent: Thursday, September 10, 2015 1:40:08 AM
Subject: Fedora 23 Install Issues on PPC64LE Server
Hello All
Hello,
As I am trying to install Fedora 23 on our PPC64LE server. I am running into
an issue with install. It has to be noted that Fedora 22/21 fail with
similar errors on this configuration:
Any pointers on narrowing this issue down would be greatly appreciated.
To give details on the box:
A ) Stoarage: Avago RAID controller 9361-8i (with 15 solid state drives on
RAID0 config) Firmware version we used is 4.300.00-4429 .
B) Network: Mellanox Connectx 3 pro card in the box.
Here are the details. Install log is attached:
1. Install Process gets stuck at the following point during installation:
(Full install logs are attached)
[ 35.253098] mlx4_en: 0002:01:00.0: Port 1: Initializing port
[ 35.263979] mlx4_en 0002:01:00.0: Activating port:2
[ 35.271034] mlx4_en: eth1: Link Up
[ 35.281464] mlx4_en: 0002:01:00.0: Port 2: Using 256 TX rings
[ 35.285573] mlx4_en: 0002:01:00.0: Port 2: Using 8 RX rings
[ 35.291084] mlx4_en: 0002:01:00.0: Port 2: frag:0 - size:1518 prefix:0
stride:1536
[ 35.298236] mlx4_en: 0002:01:00.0: Port 2: Initializing port
[ 35.309191] <mlx4_ib> mlx4_ib_add: mlx4_ib: Mellanox ConnectX InfiniBand
driver v2.2-1 (Feb 2014)
[ 35.327918] <mlx4_ib> mlx4_ib_add: counter index 2 for port 1 allocated 1
[ 35.333372] <mlx4_ib> mlx4_ib_add: counter index 3 for port 2 allocated 1
[ 35.354383] <mlx4_ib> mlx4_ib_add: counter index 2 for port 1 allocated 1
[ 35.358463] <mlx4_ib> mlx4_ib_add: counter index 3 for port 2 allocated 1
[ 35.363710] mlx4_en: eth3: Link Up
[ 35.376411] mlx4_core 0000:01:00.0 enp1s0: renamed from eth0
[ 35.432571] mlx4_core 0000:01:00.0 enp1s0d1: renamed from eth1
[ 35.603694] mlx4_core 0002:01:00.0 enP2p1s0d1: renamed from eth3
[ 35.692421] mlx4_core 0002:01:00.0 enP2p1s0: renamed from eth2
[ 37.691861] sd 5:2:10:0: [sdl] Attached SCSI disk
[ 37.692054] sd 5:2:6:0: [sdh] Attached SCSI disk
[ 37.692520] sd 5:2:13:0: [sdo] Attached SCSI disk
[ 37.692800] sd 5:2:14:0: [sdp] Attached SCSI disk
[ 43.690839] sd 5:2:12:0: [sdn] Attached SCSI disk
[ 43.691142] sd 5:2:8:0: [sdj] Attached SCSI disk
[ 43.691335] sd 5:2:11:0: [sdm] Attached SCSI disk
[ 43.691633] sd 5:2:9:0: [sdk] Attached SCSI disk
[ 49.685080] sd 5:2:5:0: [sdg] Attached SCSI disk
[ 49.685256] sd 5:2:4:0: [sdf] Attached SCSI disk
[ 49.685755] sd 5:2:7:0: [sdi] Attached SCSI disk
2. After getting stuck for about 3-5 minutes, rescue shell kicks in as
below. However as soon as it loads, serial console doesn’t respond
anymore, resulting in not being able to collect logs from rescue shell.
[ 235.757258] ]dracut-initqueue[1776]: Warning: Could not boot. 235.758170]
dracut-initqueue[1776]: Warning: /dev/root does not exist Starting Setup
Virtual Console... [ OK ] Started Setup Virtual Console. [ 236.203610]
audit: type=1130 audit(235.750:12): pid=1 uid=0 auid=4294967295 s
es=4294967295 subj=kernel msg='unit=systemd-vconsole-setup comm="systemd"
exe="/ usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Starting Dracut Emergency Shell... Warning: /dev/root does not exist
Generating "/run/initramfs/rdsosreport.txt" Entering emergency mode. Exit
the shell to continue. Type "journalctl" to view system logs. You might want
to save "/run/initramfs/rdsosreport.txt" to a USB stick or /boot after
mounting them and attach it to a bug report.
The freezing console might be related to the
https://bugzilla.redhat.com/show_bug.cgi?id=1255074 . Which makes debugging really
annoying and painful especially combined with
https://bugzilla.redhat.com/show_bug.cgi?id=1255066 (it should get fixed in next image
iteration).
Do you observe same behavior of console with f21/f22?
What do you use to obtain console(ipmi sol, physical connection,...)?
What are you using as installation source(DVD, PXE, USB, remote repo,....) and what image
are you using(netinst/dvd)?
Are disks already partitioned or are empty?
Have you been successful with booting/installing any other linux distribution(either
LE/BE)?
Thanks for report and testing,
Jakub
I can add details here as necessary.
Best
Adi Gangidi
Rackspace
_______________________________________________
ppc mailing list
ppc(a)lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/ppc
1:06 p.m.
Adding some more info that I had written down for one of our community members:
Hello Jakub
Here is the response to your questions.
Q: Do you observe same behavior of console with f21/f22?
Install gets stuck at the same place: (Attached SCSI) in f21/f22 but
rescue shell does NOT freeze in f21/f22.
Q: What do you use to obtain console(ipmi sol, physical connection,Š)?
‹Physical connection: Serial USB cable attached to debug port on server.
Q: What are you using as installation source(DVD, PXE, USB, remote
repo,....) and what image are you using(netinst/dvd)?
‹ Installation source is a Fedora 23 DVD content put on a USB stick.
Q: Are disks already partitioned or are empty?
‹ Disks where previously partitioned (I had powerkvm installed on there)
but this error occurs before I can get to the partition screen.
Q: Have you been successful with booting/installing any other linux
distribution(either LE/BE)?
‹ Yes Ubuntu 14.04 / 14.10 (PPC64LE) installed smoothly. With Ubuntu 15.04
install I ran into this install error on partitioning Avago raid:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1475166
However we are working around that bug by setting max_sectors of Megaraid
module manually and thus able to finish the install.
We are also able to install powerKVM (IBM¹s PPC64LE KVM)
Best
Adi
From: Rackspace Hosting
<adi.gangidi@rackspace.com<mailto:adi.gangidi@rackspace.com>>
Date: Wednesday, September 9, 2015 at 6:40 PM
To: "ppc@lists.fedoraproject.org<mailto:ppc@lists.fedoraproject.org>"
<ppc@lists.fedoraproject.org<mailto:ppc@lists.fedoraproject.org>>
Cc: Aaron Sullivan
<aaron.sullivan@rackspace.com<mailto:aaron.sullivan@rackspace.com>>, Major
Hayden <major.hayden@rackspace.com<mailto:major.hayden@rackspace.com>>, John
Hincher <john.hincher@rackspace.com<mailto:john.hincher@rackspace.com>>,
Antony Messerli <amesserl@rackspace.com<mailto:amesserl@rackspace.com>>
Subject: Fedora 23 Install Issues on PPC64LE Server
onnectx 3 pro card in the box.
5:26 p.m.
Hello All
I wanted to update on how we overcame below install issue and successfully installed
Fedora 23 on our power server. This install error was caused by Driver bug (Avago RAID
card, our storage controller) for which the fix is process.
This bug manifests itself to the end user who is installing Fedora 23/22 on Avago RAID
based as: Partitioning / mounting of file system as “ext4” failed. Hence the install
process fails.
More details about bug can be found in last few comments of this ubuntu bug (We found this
bug while installing ubuntu 15.04 for the first time):
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1475166
For now following work around can be used to install Fedora 23 on power systems with Avago
raid card:
1) Power on and wait system boots into Petitboot:
1a) Press "e" on "Install Fedora 23 (64-bit kernel)" option and
append "megaraid_sas.max_sectors=2048" in the end of "boot argument"
2) Select modified option and follow the installation process.
3) After install is complete and you are back to Petitboot: Edit booting option by
appending megaraid_sas.max_sectors=2048, and boot with completed install.
***IMPORTANT NOTE*** Step 3 is needed for every boot time, or you will get system going
to failure due to system can't mount file system.
Alternatively we can add this to Grub config to avoid repeating this process.
Please give it a try as possible.
Also any advise on how to best track this on corresponding Fedora board would be
appreciated.
If what I am doing currently is enough (sharing on this mailing list), that’s good to know
also.
Best
Adi
From: Rackspace Hosting
<adi.gangidi@rackspace.com<mailto:adi.gangidi@rackspace.com>>
Date: Friday, September 11, 2015 at 1:06 PM
To: "ppc@lists.fedoraproject.org<mailto:ppc@lists.fedoraproject.org>"
<ppc@lists.fedoraproject.org<mailto:ppc@lists.fedoraproject.org>>
Cc: Aaron Sullivan
<aaron.sullivan@rackspace.com<mailto:aaron.sullivan@rackspace.com>>, Major
Hayden <major.hayden@rackspace.com<mailto:major.hayden@rackspace.com>>, John
Hincher <john.hincher@rackspace.com<mailto:john.hincher@rackspace.com>>,
Antony Messerli <amesserl@rackspace.com<mailto:amesserl@rackspace.com>>
Subject: Re: Fedora 23 Install Issues on PPC64LE Server
Adding some more info that I had written down for one of our community members:
Hello Jakub
Here is the response to your questions.
Q: Do you observe same behavior of console with f21/f22?
Install gets stuck at the same place: (Attached SCSI) in f21/f22 but
rescue shell does NOT freeze in f21/f22.
Q: What do you use to obtain console(ipmi sol, physical connection,Š)?
‹Physical connection: Serial USB cable attached to debug port on server.
Q: What are you using as installation source(DVD, PXE, USB, remote
repo,....) and what image are you using(netinst/dvd)?
‹ Installation source is a Fedora 23 DVD content put on a USB stick.
Q: Are disks already partitioned or are empty?
‹ Disks where previously partitioned (I had powerkvm installed on there)
but this error occurs before I can get to the partition screen.
Q: Have you been successful with booting/installing any other linux
distribution(either LE/BE)?
‹ Yes Ubuntu 14.04 / 14.10 (PPC64LE) installed smoothly. With Ubuntu 15.04
install I ran into this install error on partitioning Avago raid:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1475166
However we are working around that bug by setting max_sectors of Megaraid
module manually and thus able to finish the install.
We are also able to install powerKVM (IBM¹s PPC64LE KVM)
Best
Adi
From: Rackspace Hosting
<adi.gangidi@rackspace.com<mailto:adi.gangidi@rackspace.com>>
Date: Wednesday, September 9, 2015 at 6:40 PM
To: "ppc@lists.fedoraproject.org<mailto:ppc@lists.fedoraproject.org>"
<ppc@lists.fedoraproject.org<mailto:ppc@lists.fedoraproject.org>>
Cc: Aaron Sullivan
<aaron.sullivan@rackspace.com<mailto:aaron.sullivan@rackspace.com>>, Major
Hayden <major.hayden@rackspace.com<mailto:major.hayden@rackspace.com>>, John
Hincher <john.hincher@rackspace.com<mailto:john.hincher@rackspace.com>>,
Antony Messerli <amesserl@rackspace.com<mailto:amesserl@rackspace.com>>
Subject: Fedora 23 Install Issues on PPC64LE Server
onnectx 3 pro card in the box.
8:23 p.m.
Sending this update to the list again. Since my last message is still stuck in review with
Admin.
Hello All
I wanted to update on how we overcame below install issue and successfully installed
Fedora 23 on our power server. This install error was caused by Driver bug (Avago RAID
card, our storage controller) for which the fix is process.
This bug manifests itself to the end user who is installing Fedora 23/22 on Avago RAID
based as: Partitioning / mounting of file system as “ext4” failed. Hence the install
process fails.
More details about bug can be found in last few comments of this ubuntu bug (We found this
bug while installing ubuntu 15.04 for the first time):
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1475166
For now following work around can be used to install Fedora 23 on power systems with Avago
raid card:
1) Power on and wait system boots into Petitboot:
1a) Press "e" on "Install Fedora 23 (64-bit kernel)" option and
append "megaraid_sas.max_sectors=2048" in the end of "boot argument"
2) Select modified option and follow the installation process.
3) After install is complete and you are back to Petitboot: Edit booting option by
appending megaraid_sas.max_sectors=2048, and boot with completed install.
***IMPORTANT NOTE*** Step 3 is needed for every boot time, or you will get system going
to failure due to system can't mount file system.
Alternatively we can add this to Grub config to avoid repeating this process.
Please give it a try as possible.
Also any advise on how to best track this on corresponding Fedora board would be
appreciated.
If what I am doing currently is enough (sharing on this mailing list), that’s good to know
also.
Best
Adi
From: Rackspace Hosting
<adi.gangidi@rackspace.com<mailto:adi.gangidi@rackspace.com>>
Date: Friday, September 11, 2015 at 1:06 PM
To: "ppc@lists.fedoraproject.org<mailto:ppc@lists.fedoraproject.org>"
<ppc@lists.fedoraproject.org<mailto:ppc@lists.fedoraproject.org>>
Cc: Aaron Sullivan
<aaron.sullivan@rackspace.com<mailto:aaron.sullivan@rackspace.com>>, Major
Hayden <major.hayden@rackspace.com<mailto:major.hayden@rackspace.com>>, John
Hincher <john.hincher@rackspace.com<mailto:john.hincher@rackspace.com>>,
Antony Messerli <amesserl@rackspace.com<mailto:amesserl@rackspace.com>>
Subject: Re: Fedora 23 Install Issues on PPC64LE Server
Adding some more info that I had written down for one of our community members:
Hello Jakub
Here is the response to your questions.
Q: Do you observe same behavior of console with f21/f22?
Install gets stuck at the same place: (Attached SCSI) in f21/f22 but
rescue shell does NOT freeze in f21/f22.
Q: What do you use to obtain console(ipmi sol, physical connection,Š)?
‹Physical connection: Serial USB cable attached to debug port on server.
Q: What are you using as installation source(DVD, PXE, USB, remote
repo,....) and what image are you using(netinst/dvd)?
‹ Installation source is a Fedora 23 DVD content put on a USB stick.
Q: Are disks already partitioned or are empty?
‹ Disks where previously partitioned (I had powerkvm installed on there)
but this error occurs before I can get to the partition screen.
Q: Have you been successful with booting/installing any other linux
distribution(either LE/BE)?
‹ Yes Ubuntu 14.04 / 14.10 (PPC64LE) installed smoothly. With Ubuntu 15.04
install I ran into this install error on partitioning Avago raid:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1475166
However we are working around that bug by setting max_sectors of Megaraid
module manually and thus able to finish the install.
We are also able to install powerKVM (IBM¹s PPC64LE KVM)
Best
Adi
From: Rackspace Hosting
<adi.gangidi@rackspace.com<mailto:adi.gangidi@rackspace.com>>
Date: Wednesday, September 9, 2015 at 6:40 PM
To: "ppc@lists.fedoraproject.org<mailto:ppc@lists.fedoraproject.org>"
<ppc@lists.fedoraproject.org<mailto:ppc@lists.fedoraproject.org>>
Cc: Aaron Sullivan
<aaron.sullivan@rackspace.com<mailto:aaron.sullivan@rackspace.com>>, Major
Hayden <major.hayden@rackspace.com<mailto:major.hayden@rackspace.com>>, John
Hincher <john.hincher@rackspace.com<mailto:john.hincher@rackspace.com>>,
Antony Messerli <amesserl@rackspace.com<mailto:amesserl@rackspace.com>>
Subject: Fedora 23 Install Issues on PPC64LE Server
onnectx 3 pro card in the box.
8:23 a.m.
On 10/01/2015 08:23 PM, Adi Gangidi wrote:
Also any advise on how to best track this on corresponding Fedora
board would be appreciated.
If what I am doing currently is enough (sharing on this mailing list), that’s good to
know also.
It might be worthwhile to add a bug to Red Hat's Bugzilla:
https://bugzilla.redhat.com/enter_bug.cgi?product=Fedora
--
Major Hayden
8:24 a.m.
Will do that.
Thanks
‹Adi
On 10/2/15, 8:23 AM, "Major Hayden" <major.hayden(a)rackspace.com> wrote:
On 10/01/2015 08:23 PM, Adi Gangidi wrote:
> Also any advise on how to best track this on corresponding Fedora board
>would be appreciated.
>
> If what I am doing currently is enough (sharing on this mailing list),
>that¹s good to know also.
It might be worthwhile to add a bug to Red Hat's Bugzilla:
https://bugzilla.redhat.com/enter_bug.cgi?product=Fedora
--
Major Hayden
8:44 a.m.
On Fri, 2 Oct 2015 13:24:56 +0000
Adi Gangidi <adi.gangidi(a)rackspace.com> wrote:
Will do that.
thanks, if I understand correctly the real culprit is in the kernel
Avago driver and that way we (and the Fedora kernel team) can track if
the necessary change went into the mainline kernel or LKML.
Dan
Thanks
‹Adi
On 10/2/15, 8:23 AM, "Major Hayden" <major.hayden(a)rackspace.com>
wrote:
>On 10/01/2015 08:23 PM, Adi Gangidi wrote:
>> Also any advise on how to best track this on corresponding Fedora
>> board
>>would be appreciated.
>>
>> If what I am doing currently is enough (sharing on this mailing
>> list),
>>that¹s good to know also.
>
>It might be worthwhile to add a bug to Red Hat's Bugzilla:
>
> https://bugzilla.redhat.com/enter_bug.cgi?product=Fedora
>
>--
>Major Hayden
_______________________________________________
ppc mailing list
ppc(a)lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/ppc
6:16 p.m.
Here is the link to the bug:
https://bugzilla.redhat.com/show_bug.cgi?id=1269300
Will make sure any progress on merging these changes is reflected on the
filed bug and here on the list.
Thanks
Adi
On 10/2/15, 8:44 AM, "Dan Horák" <dan(a)danny.cz> wrote:
the necessary change went into the mainline kernel or LKML.
3134
days inactive
3161
days old
8 comments
4 participants
participants (4)
-
Adi Gangidi -
Dan Horák -
Jakub Cajka -
Major Hayden