On Friday, November 12, 2010 22:52:12 Patrick Boutilier wrote:
On 11/12/2010 06:43 PM, Marko Vojinovic wrote:
On Friday, November 12, 2010 22:14:34 you wrote:
On Fri, Nov 12, 2010 at 3:55 PM, Marko Vojinovicvvmarko@gmail.com
wrote:
I traced the issue down to swap usage --- after a couple of days of running, *something* seems to gradually fill the memory with junk
Run top and type "M". That will sort process by their resident memory usage. It won't had to spot the offender.
top - 22:16:38 up 10 days, 5:11, 5 users, load average: 0.38, 0.45, 0.37 Tasks: 200 total, 1 running, 199 sleeping, 0 stopped, 0 zombie Cpu(s): 10.4%us, 8.5%sy, 0.0%ni, 79.9%id, 1.0%wa, 0.0%hi, 0.2%si, 0.0%st Mem: 2048088k total, 1657424k used, 390664k free, 18640k buffers Swap: 4192956k total, 1038432k used, 3154524k free, 566580k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2528 vmarko 20 0 2029m 171m 19m S 1.3 8.6 564:19.55 firefox 2397 vmarko 20 0 366m 148m 7460 S 1.3 7.4 230:40.39 skype 2366 vmarko 20 0 565m 92m 3044 S 0.0 4.6 12:10.68 emerald 5723 vmarko 20 0 989m 82m 17m S 0.0 4.1 10:29.49 kmail 1739 root 20 0 471m 59m 11m S 19.9 3.0 3780:01 X 2021 vmarko 20 0 1380m 57m 9476 S 6.3 2.9 780:37.22 ktorrent 2016 vmarko 20 0 293m 42m 8176 S 0.7 2.1 137:01.00 skype 2222 vmarko 20 0 789m 28m 5436 S 0.0 1.4 11:05.85 cairo-dock 1998 vmarko 20 0 937m 20m 8824 S 4.3 1.0 804:03.95 plasma-desktop 2603 vmarko 20 0 546m 18m 7088 S 0.0 0.9 1:39.64 kile
22635 vmarko 20 0 480m 12m 6452 S 0.7 0.6 43:06.53 konsole
2053 vmarko 9 -11 501m 12m 11m S 0.3 0.6 259:54.25 pulseaudio
10764 vmarko 20 0 455m 8572 4608 S 0.0 0.4 1:07.09 okular 32679 vmarko 20 0 285m 8528 5616 S 0.0 0.4 0:00.11 kio_pop3
So what do you make of it?
You have almost 400M free and almost 600M cached. Not sure why so much goes into swap. What are the results of the free command after running this command?
swapoff -a && swapon -a
Oh, if I try to turn off the swap now (just swapoff -a) the machine will grind to a halt for some time (half an hour or so), then the command will fail and the swap will get back in use (I mean without running the swapon -a part). Tried it several times before. Apparently the kernel eventually figures out there is not enough RAM for the swapoff command to complete, and instead the command gets killed before swap gets off. That's at least how I explain this behavior to myself. But the system is in a very bad state after that, and basically the only thing that makes sense at that point is to reboot.
I'll try again, just to make sure, but I'll probably end up having to reboot the machine. But then we'd have to wait for a week or so for the system to reach this state again, so I'd rather postpone this step for later, when all other ideas run out.
Speaking of turning off swap, I tried also to do a fresh boot, then do a swapoff -a, and keep the system running without swap. After a while (several days) the machine would start to choke, processes randomly getting killed, etc., all symptoms of "not enough memory" problem. Eventually it would end up with a complete lockup, and I'd have to reboot.
So any ideas how to troubleshoot this memory leakage, before I do something that risks requiring a reboot?
Best, :-) Marko
P.S. Apparently I inadvertently stole the thread from the OP. But if he is having the same issue, I guess it doesn't hurt... :-)