7:25 p.m.
Ok, it's been a while since I ran into something like this but I have a
process hung that I can't seem to kill. It's a gtar process doing a
backup of my personal files to a large drive I use as a backup device.
The process info:
root 11952 1 0 Jan17 ? 00:00:00 gtar
--use-compress-program=/usr/local/admin/bin/gzip1 --totals -cvl
--exclude core --exclude .thumbnail --exclude xover-cache --exclude cache
Files that are open:
[root@penguin root]# lsof -p 11952
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
gtar 11952 root cwd DIR 22,66 4096 65 /backup/greg
gtar 11952 root rtd DIR 3,2 4096 2 /
gtar 11952 root txt REG 3,2 153992 368163 /bin/tar
gtar 11952 root mem REG 3,2 107724 228281 /lib/ld-2.3.2.so
gtar 11952 root mem REG 3,2 1578228 416151 /lib/tls/libc-2.3.2.so
gtar 11952 root mem REG 3,2 33513072 468527
/usr/lib/locale/locale-archive
gtar 11952 root 0r FIFO 0,5 39450 pipe
gtar 11952 root 1w REG 3,2 82 224452 /var/log/nfs.list
gtar 11952 root 2w REG 3,2 82 224452 /var/log/nfs.list
The script I use saves a list of archived files to /var/log/nfs.list
(used to work over NFS, not backs up to local drive).
I've tried every signal I could think of to use with kill but the
process just won't die. It's not using any CPU but the load average has
been over 1.0 since the process got hung (2 days ago).
All files are accessed on local drives, NFS should not be in the
equation at all.
Any ideas????
...besides a reboot of course.
--
Greg Gulik http://www.gulik.org/greg/
greg @ gulik.org
7:33 p.m.
Gregory Gulik wrote:
Ok, it's been a while since I ran into something like this but I have
a process hung that I can't seem to kill. It's a gtar process doing a
backup of my personal files to a large drive I use as a backup device.
[...]
Any ideas????
...besides a reboot of course.
a) do an strace on it and see where it's blocked
b) attach with gdb and kill it that way
-dg-
7:49 p.m.
Ok:
[root@penguin root]# strace -p 11952
Process 11952 attached - interrupt to quit
Didn't show anything and then I had a hard time killing strace but I did
do it.
Then:
[root@penguin root]# gdb /bin/tar 11952
GNU gdb Red Hat Linux (5.3.90-0.20030710.41rh)
Copyright 2003 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "i386-redhat-linux-gnu"...
(no debugging symbols found)...Using host libthread_db library
"/lib/tls/libthread_db.so.1".
Attaching to program: /bin/tar, process 11952
...then nothing...
Dave Goldblatt wrote:
a) do an strace on it and see where it's blocked
b) attach with gdb and kill it that way
-dg-
--
Greg Gulik http://www.gulik.org/greg/
greg @ gulik.org
10:15 p.m.
F S UID PID PPID C PRI NI ADDR SZ WCHAN TTY TIME CMD
1 D 0 11952 1 0 80 2 - 1341 wait_o ? 00:00:00 gtar
Dave Goldblatt wrote:
Hmm.. what state is the process in? (flag from ps -l)
-dg-
--
Greg Gulik http://www.gulik.org/greg/
greg(a)gulik.org http://www.drivingevents.com/
1:51 a.m.
On Tue, 2004-01-20 at 05:15, Gregory Gulik wrote:
F S UID PID PPID C PRI NI ADDR SZ WCHAN TTY TIME
CMD
1 D 0 11952 1 0 80 2 - 1341 wait_o ? 00:00:00 gtar
This process is blocked waiting for I/O, so I can't be killed until the
I/O request is completed. This is something usual for processes which
use NFS: when the remote NFS dies or stops responding, the client
program stays at D and can't be killed.
10:39 a.m.
Felipe Alfaro Solana said:
On Tue, 2004-01-20 at 05:15, Gregory Gulik wrote:
> F S UID PID PPID C PRI NI ADDR SZ WCHAN TTY TIME CMD
> 1 D 0 11952 1 0 80 2 - 1341 wait_o ? 00:00:00
> gtar
This process is blocked waiting for I/O, so I can't be killed until the
I/O request is completed. This is something usual for processes which
use NFS: when the remote NFS dies or stops responding, the client
program stays at D and can't be killed.
I find the above very interesting, and it reminds me of the purpose of the
"MIH routine" in mainframe systems... MIH = Missing Interrupt Handler. In
mainframe parlance, if the "device end interrupt" got "lost", the
associated IO device would appear hung and no further IO to/from it would
occur. The MIH routine detects a "stalled device queue" and re-initiates
the IO. (Which may fail with an IO error, but that's OK, the error can be
processed.
Maybe Linuc needs an "MIH routine, especially for network devices.
7:07 a.m.
Gregory Gulik wrote:
F S UID PID PPID C PRI NI ADDR SZ WCHAN TTY TIME CMD
1 D 0 11952 1 0 80 2 - 1341 wait_o ? 00:00:00 gtar
That's what I suspected - the app is in an I/O wait (technically,
"uninterruptible sleep"). Usually this is due to an NFS hang, although
it can be caused by other media which has wedged but not timed out. One
trick which sometimes works is to unmount the mount point it's stuck
on.. if that doesn't work you're pretty much stuck.
You can try to force an unmount of a non-responsive NFS server by using
the -f option for umount (and optionally add the -l option).
-dg-
8:32 a.m.
Gregory Gulik wrote:
>
> F S UID PID PPID C PRI NI ADDR SZ WCHAN TTY TIME CMD
> 1 D 0 11952 1 0 80 2 - 1341 wait_o ? 00:00:00
> gtar
That's what I suspected - the app is in an I/O wait (technically,
"uninterruptible sleep"). Usually this is due to an NFS hang, although
it can be caused by other media which has wedged but not timed out. One
trick which sometimes works is to unmount the mount point it's stuck
on.. if that doesn't work you're pretty much stuck.
Are you running a
script that kicks off more than one backup operation?
Such as one for each file system?
In that case you need to kill the parent process first. Otherwise the
parent will just kick off the next child process in line.
David
4:50 p.m.
I did an lsof on that process and all the files it was accessing were
local file systems.
I tried to unmount. As a matter of fact I figured out that process was
stuck because I was trying to unmount my "backup" file system to switch
to a USB based drive.
Anyway, it's moot now, I couldn't wait any longer and ended up rebooting
the system.
Dave Goldblatt wrote:
That's what I suspected - the app is in an I/O wait (technically,
"uninterruptible sleep"). Usually this is due to an NFS hang, although
it can be caused by other media which has wedged but not timed out. One
trick which sometimes works is to unmount the mount point it's stuck
on.. if that doesn't work you're pretty much stuck.
You can try to force an unmount of a non-responsive NFS server by using
the -f option for umount (and optionally add the -l option).
--
Greg Gulik http://www.gulik.org/greg/
greg @ gulik.org
5:24 p.m.
Em Ter, 2004-01-20 às 19:50, Gregory Gulik escreveu:
I did an lsof on that process and all the files it was accessing were
local file systems.
I tried to unmount. As a matter of fact I figured out that process was
stuck because I was trying to unmount my "backup" file system to switch
to a USB based drive.
Anyway, it's moot now, I couldn't wait any longer and ended up rebooting
the system.
This can happen with bad hd and bad cables (!)
--
[]s
Alexandre Ganso
500 FOUR vermelha - Diretor Steel Goose Moto Group
5:58 p.m.
On Tue, Jan 20, 2004 at 08:07:31AM -0500, Dave Goldblatt wrote:
Gregory Gulik wrote:
>
>F S UID PID PPID C PRI NI ADDR SZ WCHAN TTY TIME CMD
>1 D 0 11952 1 0 80 2 - 1341 wait_o ? 00:00:00 gtar
That's what I suspected - the app is in an I/O wait (technically,
"uninterruptible sleep"). Usually this is due to an NFS hang, although
it can be caused by other media which has wedged but not timed out.
What does "lsof" tell you? What IO is the process waiting on?
For NFS the mount flag "intr" can help (next time). Umount -k may
also kill it this time.
You may want to look at a back trace for the process and see what the last
IO request was. There are two stacks of interest, the user space
side and the kernel space side of the system call.
Depending on library support and the precise action inside system
calls a read() of ten characters will not return until ten characters
are present. A read of a line will not return until a new line
marker.... And for some things the end of file is important.
Make sure you understand what "gtar" is being asked to do. It's
standard input file descriptor may be hung (no new line/eof). It may
be hung on IO for a file read or write or scratch file. Has anyone
attempted to backup /dev/random, /dev/zero or a named pipe? Devzero
took weeks on my old infinity computer and check sums on devrandom
never matched.
Once a system call dives down deep into the OS the handling of user
space signals (kill -N) gets murky. As I noted above NFS has a flag
"intr" to permit the interrupt in user space to end the file IO. For
NFS this may be possible at multiple levels because timers for network
IO trigger and wake NFS up. For other IO devices there may not be a
failsafe timer to wake up the driver and return an error.
Because "stuck in IO" often translates into stuck in a hardware driver
it is important to report the specifics of the device involved and the
specifics of the driver. Note that RAID devices are layered. The
pseudo hardware that is the raid depends on real hardware. Thus there
are multiple places where a 'hang' could be generated.
For some IO, make sure things have not gotten so slow that no useful
progress is being made. Error recovery, Raid recovery or swap IO can
make things look broken. The machine is working, but no progress is
being made and each time you "look" the process is stuck in IO. You
may be missing baby step IO (but now you should be able to kill it).
--
T o m M i t c h e l l
mitch48-at-sbcglobal-dot-net
7:50 p.m.
I've tried every signal from 1 through 31 numerous times and the process
is still there.
don wrote:
kill -9 <pid>
--
Greg Gulik http://www.gulik.org/greg/
greg @ gulik.org
8:28 p.m.
More times than I can count.
Benno Goedhart wrote:
Have you also tried kill -HUP <pid>
--
Greg Gulik http://www.gulik.org/greg/
greg @ gulik.org
8:34 p.m.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Gregory Gulik wrote:
|
| More times than I can count.
|
|
| Benno Goedhart wrote:
|
|>
|> Have you also tried kill -HUP <pid>
|>
|>
|>
|
This is a suggestion coming from a complete idiot, but why not try
logging out? That might get rid of it.
But then again, it might not.
- -NIE
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.3 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iD8DBQFADJO3HBMJeYrn+U8RAkqYAJ9xym9uj2qPrBHg9KKxjB1uf4DOagCcDpJq
4ysb2bqrdTeTo9/5rX6etjo=
=5HEc
-----END PGP SIGNATURE-----
8:39 p.m.
Since the process was started by cron I doubt it would have any effect.
Heck, I restarted crond and atd just to be sure as well.
At this point, I'm almost frustrated enough to reboot the machine. I
unfortunately have to wait until several tasks on that machine are done
before I can do that.
Nicholas Evans wrote:
This is a suggestion coming from a complete idiot, but why not try
logging out? That might get rid of it.
But then again, it might not.
--
Greg Gulik http://www.gulik.org/greg/
greg @ gulik.org
7:55 p.m.
It does spawn a script that runs "gzip -1"
That process is no longer around.
I've looked for any child or parent processes but those have been
killed, it's just this stubborn process that just won't go away.
Bevan C. Bennett wrote:
Are there any child processes or otherwise associated other programs
that it might be waiting for the death of?
What was the command-line/script that spawned this job?
Are there any other hung processes?
--
Greg Gulik http://www.gulik.org/greg/
greg @ gulik.org
8:23 p.m.
Gregory Gulik wrote:
It does spawn a script that runs "gzip -1"
That process is no longer around.
I've looked for any child or parent processes but those have been
killed, it's just this stubborn process that just won't go away.
The dread "zombie" process. Attach to it using gdb, then "kill"
followed by "quit". This should kill it off. If not, the only way to
get rid of zombies is to reboot or have the "magic elixir".
----------------------------------------------------------------------
- Rick Stevens, Senior Systems Engineer rstevens(a)vitalstream.com -
- VitalStream, Inc. http://www.vitalstream.com -
- -
- Do you know how to save five drowning lawyers? No? GOOD! -
----------------------------------------------------------------------
7382
days inactive
7409
days old
20 comments
11 participants
participants (11)
-
Alexandre Strube -
Benno Goedhart -
Bevan C. Bennett -
Dave Goldblatt -
David Jackson -
Don Russell -
Felipe Alfaro Solana -
Gregory Gulik -
Nicholas Evans -
Rick Stevens -
Tom Mitchell