So we're in a bit of trouble. Right now the ext3 filesystem on /mnt/koji is in a sort of funky state. There's a section of the filesystem that is pretty borked. Running fsck.ext3 on it causes a segfault. The core dump is 2G right now. This is likely from the short few hours we ran /mnt/koji from a RHEL4 box (lesson learned there...)
Steps to proceed:
1) Snapshots - I've been doing all of my tests from snapshots and will continue to do so. The share itself doesn't seem to be catastrophically broken, just that one section.
2) Contact some fs experts with the dump and see what is going on.
3) Fix the filesystem.
4) Also, for the last 4 nights at around 3 am my time the filesystem has hit this part of the disk causing it to get remounted ro. I'd like to setup a script to auto correct this issue until we can come up with a permanent fix. Just saves me from having to get up in the middle of the night.
Is anyone against this? I'm mostly worried about step3 then anything. Before we do that I want to make sure we have a proper backup and do plenty of tests from snapshots. It might be best to do 3) after the release.
-Mike
On Mon, 2008-04-21 at 09:04 -0500, Mike McGrath wrote:
So we're in a bit of trouble. Right now the ext3 filesystem on /mnt/koji is in a sort of funky state. There's a section of the filesystem that is pretty borked. Running fsck.ext3 on it causes a segfault. The core dump is 2G right now. This is likely from the short few hours we ran /mnt/koji from a RHEL4 box (lesson learned there...)
Steps to proceed:
- Snapshots - I've been doing all of my tests from snapshots and will
continue to do so. The share itself doesn't seem to be catastrophically broken, just that one section.
Contact some fs experts with the dump and see what is going on.
Fix the filesystem.
Also, for the last 4 nights at around 3 am my time the filesystem has
hit this part of the disk causing it to get remounted ro. I'd like to setup a script to auto correct this issue until we can come up with a permanent fix. Just saves me from having to get up in the middle of the night.
Is anyone against this? I'm mostly worried about step3 then anything. Before we do that I want to make sure we have a proper backup and do plenty of tests from snapshots. It might be best to do 3) after the release.
This gets a +1 from me. Must keep that thing stable, and you sane.
2008/4/21 Jesse Keating jkeating@redhat.com:
On Mon, 2008-04-21 at 09:04 -0500, Mike McGrath wrote:
So we're in a bit of trouble. Right now the ext3 filesystem on /mnt/koji is in a sort of funky state. There's a section of the filesystem that is pretty borked. Running fsck.ext3 on it causes a segfault. The core dump is 2G right now. This is likely from the short few hours we ran /mnt/koji from a RHEL4 box (lesson learned there...)
Steps to proceed:
- Snapshots - I've been doing all of my tests from snapshots and will
continue to do so. The share itself doesn't seem to be catastrophically broken, just that one section.
Contact some fs experts with the dump and see what is going on.
Fix the filesystem.
Also, for the last 4 nights at around 3 am my time the filesystem has
hit this part of the disk causing it to get remounted ro. I'd like to setup a script to auto correct this issue until we can come up with a permanent fix. Just saves me from having to get up in the middle of the night.
Is anyone against this? I'm mostly worried about step3 then anything. Before we do that I want to make sure we have a proper backup and do plenty of tests from snapshots. It might be best to do 3) after the release.
This gets a +1 from me. Must keep that thing stable, and you sane.
Ah but Mike has viewed the depths of filesystem creation and seen the dark Lords who live there dancing forever around the sleeping Nameless One. Its too late.. the best we can do is wall him up in a room next to Seth and Stephen Tweedie and let them chant their prayers to their new Mad Overlords: "Ph'sck mount'slafh Extphree" though Seth's is more "O'yum rp'm" but they do go together as harmony.
On Mon, 2008-04-21 at 09:51 -0600, Stephen John Smoogen wrote:
Ah but Mike has viewed the depths of filesystem creation and seen the dark Lords who live there dancing forever around the sleeping Nameless One. Its too late.. the best we can do is wall him up in a room next to Seth and Stephen Tweedie and let them chant their prayers to their new Mad Overlords: "Ph'sck mount'slafh Extphree" though Seth's is more "O'yum rp'm" but they do go together as harmony.
Stephen, The pills you're taking today are clearly just placebos. Tell the doctors you'd like the 'good stuff' from now on. okay? :)
-sv
On Mon, Apr 21, 2008 at 10:02 AM, seth vidal skvidal@fedoraproject.org wrote:
On Mon, 2008-04-21 at 09:51 -0600, Stephen John Smoogen wrote:
Ah but Mike has viewed the depths of filesystem creation and seen the dark Lords who live there dancing forever around the sleeping Nameless One. Its too late.. the best we can do is wall him up in a room next to Seth and Stephen Tweedie and let them chant their prayers to their new Mad Overlords: "Ph'sck mount'slafh Extphree" though Seth's is more "O'yum rp'm" but they do go together as harmony.
Stephen, The pills you're taking today are clearly just placebos. Tell the doctors you'd like the 'good stuff' from now on. okay? :)
Oh I am not falling for that trick again.. You said last week to tell them to up the voltage.. haha very funny that hurt.
infrastructure@lists.fedoraproject.org
-
Jesse Keating -
Mike McGrath -
Seth Vidal -
Stephen John Smoogen