RE: How to minimize (unshareable) memory usage?

["Boehm, Hans" <hans.boehm@hp.com>]

I think the current code fails the heap expansion if it would have resulted in a heap larger than specified; it doesn't reduce the size of the heap expansion so that it would barely still fit.  This means it might be failing sooner than you intended.   Also setting GC_INITIAL_HEAP_SIZE may help.

I would put a breakpoint in the failing GC_expand_hp_inner call and then call GC_dump(),which will tell you more about the collector's view of the heap.  Since this collector doesn't move objects, it's quite possible that there just wasn't a hole large enough for the requested object, particularly if you allocate some really large objects.  If that is indeed the case, you may be able to circumvent the issue by allocating multiple smaller objects instead.

The other, sometimes significant, source of space overhead is the fact that the collector splits the heap into chunks whose size is the smallest possible multiple of 4K or 8K, depending on the paltform, and then uses each chunk for a fixed object size.  Thus objects of size 2049 or 4097 bytes (including the Java header) tend to waste a lot of space.

Hans

> -----Original Message-----
> From: Justin Santa Barbara [mailto:justin@fathomdb.com] 
> Sent: Tuesday, January 19, 2010 2:34 AM
> To: java@gcc.gnu.org
> Cc: Boehm, Hans; Andrew Haley; David Daney
> Subject: Re: How to minimize (unshareable) memory usage?
> 
> Thanks to everyone for the great suggestions.  I've tracked 
> down the mysterious 8MB segments: they are the stacks for my 
> threads (by putting a breakpoint on mmap)  Though reserved in 
> virtual memory space, pages shouldn't be allocated unless 
> they're actually touched, so this is good news for physical 
> memory.  I verified this by dumping /proc/<pid>/smaps; the 
> output of one of my 8MB segments shows that only 16KB or so 
> is really used.  (Output attached at the end)
> 
> So the next biggest culprit is my heap memory usage ... I 
> tried using GCInfo.setOOMDump, and did get a dump.  But all 
> is not entirely well:
> my GC_MAXIMUM_HEAP_SIZE was set to 8,000,000 bytes, which 
> results in an out of memory dump, but gc-analyze reports only 
> 5.5MB usage, and of that half is listed under the 'free' 
> column in the "Used Blocks"
> section of the dump.
> 
> *** Used Blocks ***
> 
>   Size     Kind            Blocks     Used       Free       Wasted
> -------  -------------    ------- ---------- ----------    -------
> <...snipped...>
> -------  -------------    ------- ---------- ----------    -------
>                             1,352  2,700,672  2,758,624     78,496
> Total bytes =  5,537,792
> 
> The percentages elsewhere also support the idea that about 
> 3MB of data is actually in use, so I'm not sure exactly why 
> we're failing memory requests when we don't seem to be close 
> to the limit.
> 
> As for the 6MB of writeable data that shows against libgcj, I 
> believe this is the 'normal' data segment of the shared 
> library.  Cutting this down would probably be difficult.  The 
> easiest approach for me would be modularizing gcj so I don't 
> have to pay for swing or awt.  Again, suggestions are very 
> welcome, but I think that this particular segment will be 
> much more complicated than figuring out why the heap is not 
> being used as efficiently as I hope it could be.
> 
> Thanks
> Justin
> 
> ---
> 
> The smap output:
> 
> 8MB of stack, but only 16KB actually used:
> 
> b4b05000-b5305000 rw-p 00000000 00:00 0
> Size:               8192 kB
> Rss:                  16 kB
> Pss:                  16 kB
> Shared_Clean:          0 kB
> Shared_Dirty:          0 kB
> Private_Clean:         0 kB
> Private_Dirty:        16 kB
> Referenced:           16 kB
> Swap:                  0 kB
> KernelPageSize:        4 kB
> MMUPageSize:           4 kB
> 
> The heap segment (with a 10,000,000 byte limit to avoid 
> crashing, and captured on a different machine from the other two):
> 7f43a12b0000-7f43a1bc0000 rw-p 7f43a12b0000 00:00 0
> Size:               9280 kB
> Rss:                9280 kB
> Shared_Clean:          0 kB
> Shared_Dirty:          0 kB
> Private_Clean:         0 kB
> Private_Dirty:      9280 kB
> Referenced:         9280 kB
> 
> 
> The data segment of libgcj (?), mostly dirty:
> b7122000-b7739000 rw-p 01ae0000 08:01 28534016   
> /usr/local/lib/libgcj.so.10.0.0
> Size:               6236 kB
> Rss:                6236 kB
> Pss:                6236 kB
> Shared_Clean:          0 kB
> Shared_Dirty:          0 kB
> Private_Clean:      1644 kB
> Private_Dirty:      4592 kB
> Referenced:         6236 kB
> Swap:                  0 kB
> KernelPageSize:        4 kB
> MMUPageSize:           4 kB
>