OK, I tested it more thoroughly and I was right all along (as usual)
memory is shared and RSS includes the shared pages.
I attached a python script I wrote to compare actual pages between
processes.
fork() acts as expected even in python. The problem is that after the
fork VDSM keeps changing the memory and the children keep a copy of the
memory.
This means that even though they'll never use it a copy will exists.
Because of how we spawn processes. child processes can have different
versions of VDSM thus actually ending up taking all the memory not
because they rewrote it but because VDSM did.
Comparing OOPs that spawned at almost the same time they share a lot of
memory
194 (776 KB) only in A, 2146 (8584 KB) shared, 194 (776 KB) only in B
OOP processes that spawned a few hours apart have a lot less in common
2336 (9344 KB) only in A, 4 (16 KB) shared, 2521 (10084 KB) only in B
So spawning all the processes at once instead of gradually would solve
this memory problem.
I added the script if you want to test it yourselves on your hosts and
test progs
On 07/21/2011 09:52 AM, Dan Kenigsberg wrote:
On Wed, Jul 20, 2011 at 11:41:33AM -0400, David Malcolm wrote:
> On Wed, 2011-07-20 at 13:39 +0300, Moran Goldboim wrote:
>> Hi Dave,
>>
>> We are suffering high memory usage on vdsm as a result of using
>> threading in python, child process is taking most of the father memory.
>> for example:
>>
>> import multiprocessing
>> import time
>> l = [1000] * 2**20
>> def run():
>> time.sleep(10000)
>> multiprocessing.Process(target=run).start()
> Multiprocessing is one of my least favorite modules. It doesn't use
> threads so much as it forks child processes, without calling exec:
> each child process within multiprocessing has an entire copy of the
> parent, which gradually leads to copy-on-write differences, even for
> supposedly read-only operations, given that python uses reference
> counting (since all of the refcounts are constantly going through
> inc/dec operations, so pages rapidly get written to).
Ouch. Though I'm not sure this explains the issue with the example
script: the child process does nothing, so it states should not change
with time. It appeared to me that the pages became dirty
instantaneously. Am I missing smething?
>
>> grep Shared_Dirty /proc/22986/smaps | awk '{s=s+$2} END {print s}'
>> 11272
>>
>> top:
>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
>> 22985 mgoldboi 20 0 184m 14m 2908 S 0.0 0.4 0:00.04 python
>> 22986 mgoldboi 20 0 184m 12m 616 S 0.0 0.3 0:00.00 python
>>
>> is it a known issue? can we do something about it?
> How much do you need the child to inherit state from the parent? How
> long-lived is the child? In theory, I guess you could start "del"-ing
> things within the child, and run a garbage collection first. But if you
> merely need a subprocess, could you use the "subprocess" module instead?
> (which does an "exec" after the fork, so you're starting with a clean
> slate).
We used to have our own inter-process messaging with Python 2.4, and
where very happy to ditch it in favor of multiprocessing... I am not
sure we can go back to that. Saggi, can you find a way around this?
> FWIW I wrote a tool to analyse python memory usage: gdb-heap; see
>
http://fedoraproject.org/wiki/Features/MemoryDebuggingTools
> However, it is unpolished and (ironically) very memory-hungry itself.
>
> Hope this is helpful
It is, in an unhappy way.
Thanks,
Dan.