FYI all we're working on xen1's issues. They seem to be iscsi related but its still not quite clear. Basically the last thing we see in the logs is an issue with iscsi, then the box reboots. We're seeing similar log issues around the same time on our other boxes... but they don't reboot. See:
https://fedorahosted.org/fedora-infrastructure/ticket/334
-Mike
stupid question from an outsider - you have OMSA on these systems? If so, what does it have to say, if anything? If it were HP hardware, I could help more :)
On Jan 9, 2008 11:23 AM, Mike McGrath mmcgrath@redhat.com wrote:
FYI all we're working on xen1's issues. They seem to be iscsi related but its still not quite clear. Basically the last thing we see in the logs is an issue with iscsi, then the box reboots. We're seeing similar log issues around the same time on our other boxes... but they don't reboot. See:
https://fedorahosted.org/fedora-infrastructure/ticket/334
-Mike
Fedora-infrastructure-list mailing list Fedora-infrastructure-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
On Wed, 9 Jan 2008, Jon Stanley wrote:
stupid question from an outsider - you have OMSA on these systems? If so, what does it have to say, if anything? If it were HP hardware, I could help more :)
We actually don't monitor the hardware hardly at all on our systems. I'd started to set something up a couple months ago but got distracted. Its mostly a matter of getting someone with the time to do it and the unfortunate lack of OSS solutions in that area.
-Mike
On Wed, 9 Jan 2008, Mike McGrath wrote:
FYI all we're working on xen1's issues. They seem to be iscsi related but its still not quite clear. Basically the last thing we see in the logs is an issue with iscsi, then the box reboots. We're seeing similar log issues around the same time on our other boxes... but they don't reboot. See:
xen2 has started doing this too, having recently upgraded to RHEL5 it is a likely suspect as both xen1 and 2 are now on RHEL5. I've updated the ticket for more info:
https://fedorahosted.org/fedora-infrastructure/ticket/334#comment:5
Right now xen2 is basically downgraded to FC6 as far as kernel and xen libs are concerned. we'll see if that helps. I'll be testing the fixes suggested on xen1 in the coming week.
-Mike
why these machines can't run fedora 8 ?
fedora 8 have new xen version, and new xen kernels.
Mike McGrath wrote:
On Wed, 9 Jan 2008, Mike McGrath wrote:
FYI all we're working on xen1's issues. They seem to be iscsi related but its still not quite clear. Basically the last thing we see in the logs is an issue with iscsi, then the box reboots. We're seeing similar log issues around the same time on our other boxes... but they don't reboot. See:
xen2 has started doing this too, having recently upgraded to RHEL5 it is a likely suspect as both xen1 and 2 are now on RHEL5. I've updated the ticket for more info:
https://fedorahosted.org/fedora-infrastructure/ticket/334#comment:5
Right now xen2 is basically downgraded to FC6 as far as kernel and xen libs are concerned. we'll see if that helps. I'll be testing the fixes suggested on xen1 in the coming week.
-Mike
Fedora-infrastructure-list mailing list Fedora-infrastructure-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
Maybe because Fedora is also a moving target.
Kind regards,
Jeroen van Meeuwen -kanarip
Itamar - IspBrasil wrote:
why these machines can't run fedora 8 ?
fedora 8 have new xen version, and new xen kernels.
Mike McGrath wrote:
On Wed, 9 Jan 2008, Mike McGrath wrote:
FYI all we're working on xen1's issues. They seem to be iscsi related but its still not quite clear. Basically the last thing we see in the logs is an issue with iscsi, then the box reboots. We're seeing similar log issues around the same time on our other boxes... but they don't reboot. See:
xen2 has started doing this too, having recently upgraded to RHEL5 it is a likely suspect as both xen1 and 2 are now on RHEL5. I've updated the ticket for more info:
https://fedorahosted.org/fedora-infrastructure/ticket/334#comment:5
Right now xen2 is basically downgraded to FC6 as far as kernel and xen libs are concerned. we'll see if that helps. I'll be testing the fixes suggested on xen1 in the coming week.
-Mike
Fedora-infrastructure-list mailing list Fedora-infrastructure-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
Fedora-infrastructure-list mailing list Fedora-infrastructure-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
I'm out watching football atm, can't really look at my mail archives, but has xen1 exhibited these issues since downgrading to RHEL5 GA kernel? And is there the common signature of iSCSI issue -> crash here? As someone who has been in the hosting business for a long time, I hate to make two separate issues the same when there is not data to support that conclusion. Feel free to tell me to shut up :-)
On 1/20/08, Mike McGrath mmcgrath@redhat.com wrote:
On Sun, 20 Jan 2008, Itamar - IspBrasil wrote:
why these machines can't run fedora 8 ?
fedora 8 have new xen version, and new xen kernels.
Fedora 8 is ill suited to this particular task which, in reality, is just an appliance/abstraction between our hosts and the hardware.
-Mike
Fedora-infrastructure-list mailing list Fedora-infrastructure-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
On Sun, 20 Jan 2008, Jon Stanley wrote:
I'm out watching football atm, can't really look at my mail archives, but has xen1 exhibited these issues since downgrading to RHEL5 GA kernel? And is there the common signature of iSCSI issue -> crash here? As someone who has been in the hosting business for a long time, I hate to make two separate issues the same when there is not data to support that conclusion. Feel free to tell me to shut up :-)
xen1 started exhibiting these issues when we moved from FC6 to RHEL5 GA. It was assumed to be hardware issue because of how sporatic the issues were and because we actually do have RHEL5 on other xen hosts.
The iscsi issue may or may not be a red herring but some of the reports listed in the ticket (and I haven't done exhaustive research on this yet) suggests a kernel / poweredge bug that we may be hitting. We had moved all non-redundant guests off of xen1 onto the more stable xen2 box. After upgrading xen2 to RHEL5 we started seeing the same problems with it. There's a few things we can try, I'll be doing so on xen1 with proxy4 as our test host since its competely redundant and has, in the past, crashed that box.
For now I'm going to see if the FC6 kernel and xen libs keep xen2 happy while we're testing / probing at xen1.
-Mike
On Sun, 20 Jan 2008, Mike McGrath wrote:
xen1 started exhibiting these issues when we moved from FC6 to RHEL5 GA. It was assumed to be hardware issue because of how sporatic the issues were and because we actually do have RHEL5 on other xen hosts.
Are xen1/2 the only RHEL5 Xen hosts with that particular quantity of memory?
Jima
On Mon, 21 Jan 2008, Jima wrote:
On Sun, 20 Jan 2008, Mike McGrath wrote:
xen1 started exhibiting these issues when we moved from FC6 to RHEL5 GA. It was assumed to be hardware issue because of how sporatic the issues were and because we actually do have RHEL5 on other xen hosts.
Are xen1/2 the only RHEL5 Xen hosts with that particular quantity of memory?
Yes, though in the future they should all be in similar specs (1U box, 8 core 16-32G memory)
-Mike
On Sun, 2008-01-20 at 16:43 -0600, Mike McGrath wrote:
On Sun, 20 Jan 2008, Itamar - IspBrasil wrote:
why these machines can't run fedora 8 ? fedora 8 have new xen version, and new xen kernels.
Fedora 8 is ill suited to this particular task which, in reality, is just an appliance/abstraction between our hosts and the hardware.
Please don't see any troll in this question, but why do you say that?
Do you mean Fedora in general since this box is seen as a rarely touched appliance and you don't want to upgrade it often? Or are you referring to Fedora 8 in particular because of some shortcoming in the release?
I'm using F8 for a dom0 and pretty happy with it and libvirt in my particular environment.
2008/1/30 Dale Bewley dale@bewley.net:
Do you mean Fedora in general since this box is seen as a rarely touched appliance and you don't want to upgrade it often?
Generally for this reason. Frequent upgrades are rough in an environment that is supposed to be up most of the time. Especially when that box houses several VMs.
--Jeffrey
On Wed, 30 Jan 2008, Dale Bewley wrote:
On Sun, 2008-01-20 at 16:43 -0600, Mike McGrath wrote:
On Sun, 20 Jan 2008, Itamar - IspBrasil wrote:
why these machines can't run fedora 8 ? fedora 8 have new xen version, and new xen kernels.
Fedora 8 is ill suited to this particular task which, in reality, is just an appliance/abstraction between our hosts and the hardware.
Please don't see any troll in this question, but why do you say that?
Do you mean Fedora in general since this box is seen as a rarely touched appliance and you don't want to upgrade it often? Or are you referring to Fedora 8 in particular because of some shortcoming in the release?
I'm using F8 for a dom0 and pretty happy with it and libvirt in my particular environment.
Just in that we don't like to touch it much. Our teams resources vs what we're trying to provide don't mix well with having to rebuild the machines all the time. We actually do have some Fedora dom0 boxes but in general its nice to setup and forget about them which we have generally been able to do.
-Mike
infrastructure@lists.fedoraproject.org
-
Dale Bewley -
itamar -
Jeffrey Tadlock -
Jeroen van Meeuwen -
Jima -
Jon Stanley -
Mike McGrath