* How to minimize (unshareable) memory usage? [not found] <a3fa9dad1001180133h252738bfl5d887cb2670475e3@mail.gmail.com> @ 2010-01-18 9:40 ` Justin Santa Barbara 2010-01-18 9:50 ` Andrew Haley 2010-03-18 15:36 ` hari001 0 siblings, 2 replies; 7+ messages in thread From: Justin Santa Barbara @ 2010-01-18 9:40 UTC (permalink / raw) To: java I have some simple Java utilities that I've written that need to run on machines with limited free memory, and I'm looking at GCJ for this. Using the full Sun JVM we run into problems with memory usage - we're able to limit the heap to 8MB, but we need to allow at least 16MB for PermGen (despite not really having much application code), and at least 8MB for ReservedCodeCacheSize; even a simple utility therefore needs at least 32MB of unshareable memory. Running under GCJ the results are encouraging, but still not as good as I'd hope. The output from pmap is included below. The interesting ones are these four: 00010000 10404K rw--- [ anon ] b4125000 8192K rw--- [ anon ] b49d2000 8192K rw--- [ anon ] b7096000 6324K rw--- /usr/lib/libgcj.so.10.0.0 I've limited GCJ to 10MB of heap (it seems to need a little more than the Sun JVM for some reason), so that's the first entry. But what are the two 8MB anonymous allocations that aren't the 10MB heap? What is the 6MB of read/write libgcj mappings? Any hints on how to minimize unshareable memory usage would be greatly appreciated. I'm using the default gcj from Ubuntu Karmic (gcj -v output also included below) Thanks, Justin 00010000 10404K rw--- [ anon ] 08048000 244K r-x-- /tmp/a.out 08085000 56K rw--- /tmp/a.out 08093000 12K rw--- [ anon ] 09719000 404K rw--- [ anon ] b4124000 4K ----- [ anon ] b4125000 8192K rw--- [ anon ] b4925000 64K r-x-- /lib/tls/i686/cmov/libresolv-2.10.1.so b4935000 4K r---- /lib/tls/i686/cmov/libresolv-2.10.1.so b4936000 4K rw--- /lib/tls/i686/cmov/libresolv-2.10.1.so b4937000 8K rw--- [ anon ] b4939000 20K r-x-- /lib/tls/i686/cmov/libnss_dns-2.10.1.so b493e000 4K r---- /lib/tls/i686/cmov/libnss_dns-2.10.1.so b493f000 4K rw--- /lib/tls/i686/cmov/libnss_dns-2.10.1.so b4940000 380K r-x-- /usr/lib/libgmp.so.3.5.0 b499f000 4K r---- /usr/lib/libgmp.so.3.5.0 b49a0000 4K rw--- /usr/lib/libgmp.so.3.5.0 b49bc000 84K r---- /var/lib/gcj-4.4/classmap.db b49d1000 4K ----- [ anon ] b49d2000 8192K rw--- [ anon ] b51d2000 948K r---- /usr/lib/locale/en_US.utf8/LC_COLLATE b52bf000 40K r-x-- /lib/tls/i686/cmov/libnss_files-2.10.1.so b52c9000 4K r---- /lib/tls/i686/cmov/libnss_files-2.10.1.so b52ca000 4K rw--- /lib/tls/i686/cmov/libnss_files-2.10.1.so b52cb000 76K r-x-- /lib/tls/i686/cmov/libnsl-2.10.1.so b52de000 4K r---- /lib/tls/i686/cmov/libnsl-2.10.1.so b52df000 4K rw--- /lib/tls/i686/cmov/libnsl-2.10.1.so b52e0000 8K rw--- [ anon ] b52e4000 8K r-x-- /lib/libnss_mdns4.so.2 b52e6000 4K r---- /lib/libnss_mdns4.so.2 b52e7000 4K rw--- /lib/libnss_mdns4.so.2 b52e8000 8K r-x-- /lib/libnss_mdns4_minimal.so.2 b52ea000 4K r---- /lib/libnss_mdns4_minimal.so.2 b52eb000 4K rw--- /lib/libnss_mdns4_minimal.so.2 b52ec000 20K r-x-- /usr/lib/gcj-4.4-10/libjavamath.so b52f1000 4K rw--- /usr/lib/gcj-4.4-10/libjavamath.so b52f2000 4K rw-s- /tmp/ffiTjScy8 (deleted) b52f3000 4K r-xs- /tmp/ffiTjScy8 (deleted) b52f4000 4K r---- /usr/lib/locale/en_US.utf8/LC_NUMERIC b52f5000 4K r---- /usr/lib/locale/en_US.utf8/LC_TIME b52f6000 4K r---- /usr/lib/locale/en_US.utf8/LC_MONETARY b52f7000 4K r---- /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES b52f8000 4K r---- /usr/lib/locale/en_US.utf8/LC_PAPER b52f9000 4K r---- /usr/lib/locale/en_US.utf8/LC_NAME b52fa000 4K r---- /usr/lib/locale/en_US.utf8/LC_ADDRESS b52fb000 4K r---- /usr/lib/locale/en_US.utf8/LC_TELEPHONE b52fc000 4K r---- /usr/lib/locale/en_US.utf8/LC_MEASUREMENT b52fd000 252K r---- /usr/lib/locale/en_US.utf8/LC_CTYPE b533c000 8K rw--- [ anon ] b533e000 1272K r-x-- /lib/tls/i686/cmov/libc-2.10.1.so b547c000 8K r---- /lib/tls/i686/cmov/libc-2.10.1.so b547e000 4K rw--- /lib/tls/i686/cmov/libc-2.10.1.so b547f000 12K rw--- [ anon ] b5482000 8K r-x-- /lib/tls/i686/cmov/libdl-2.10.1.so b5484000 4K r---- /lib/tls/i686/cmov/libdl-2.10.1.so b5485000 4K rw--- /lib/tls/i686/cmov/libdl-2.10.1.so b5486000 80K r-x-- /lib/libz.so.1.2.3.3 b549a000 4K r---- /lib/libz.so.1.2.3.3 b549b000 4K rw--- /lib/libz.so.1.2.3.3 b549c000 28K r-x-- /lib/tls/i686/cmov/librt-2.10.1.so b54a3000 4K r---- /lib/tls/i686/cmov/librt-2.10.1.so b54a4000 4K rw--- /lib/tls/i686/cmov/librt-2.10.1.so b54a5000 84K r-x-- /lib/tls/i686/cmov/libpthread-2.10.1.so b54ba000 4K r---- /lib/tls/i686/cmov/libpthread-2.10.1.so b54bb000 4K rw--- /lib/tls/i686/cmov/libpthread-2.10.1.so b54bc000 8K rw--- [ anon ] b54be000 144K r-x-- /lib/tls/i686/cmov/libm-2.10.1.so b54e2000 4K r---- /lib/tls/i686/cmov/libm-2.10.1.so b54e3000 4K rw--- /lib/tls/i686/cmov/libm-2.10.1.so b54e4000 4K rw--- [ anon ] b54e5000 28356K r-x-- /usr/lib/libgcj.so.10.0.0 b7096000 6324K rw--- /usr/lib/libgcj.so.10.0.0 b76c3000 280K rw--- [ anon ] b7709000 112K r-x-- /lib/libgcc_s.so.1 b7725000 4K r---- /lib/libgcc_s.so.1 b7726000 4K rw--- /lib/libgcc_s.so.1 b7727000 4K r---- /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION b7728000 36K r-x-- /lib/tls/i686/cmov/libnss_nis-2.10.1.so b7731000 4K r---- /lib/tls/i686/cmov/libnss_nis-2.10.1.so b7732000 4K rw--- /lib/tls/i686/cmov/libnss_nis-2.10.1.so b7733000 24K r-x-- /lib/tls/i686/cmov/libnss_compat-2.10.1.so b7739000 4K r---- /lib/tls/i686/cmov/libnss_compat-2.10.1.so b773a000 4K rw--- /lib/tls/i686/cmov/libnss_compat-2.10.1.so b773b000 28K r--s- /usr/lib/gconv/gconv-modules.cache b7742000 8K rw--- [ anon ] b7744000 4K r-x-- [ anon ] b7745000 108K r-x-- /lib/ld-2.10.1.so b7760000 4K r---- /lib/ld-2.10.1.so b7761000 4K rw--- /lib/ld-2.10.1.so bff3e000 84K rw--- [ stack ] total 66632K gcj -v Using built-in specs. Reading specs from /usr/lib/gcc/i486-linux-gnu/4.4.1/libgcj.spec rename spec startfile to startfileorig rename spec lib to liborig Target: i486-linux-gnu Configured with: ../src/configure -v --with-pkgversion='Ubuntu 4.4.1-5ubuntu2' --with-bugurl=file:///usr/share/doc/gcj-4.4/README.Bugs --enable-languages=c,c++,java --prefix=/usr --enable-shared --enable-multiarch --enable-linker-build-id --with-system-zlib --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --with-gxx-include-dir=/usr/include/c++/4.4 --program-suffix=-4.4 --enable-nls --enable-clocale=gnu --disable-libmudflap --enable-java-awt=gtk --enable-gtk-cairo --disable-plugin --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-4.4/jre --enable-java-home --with-jvm-root-dir=/usr/lib/jvm/java-1.5.0-gcj-4.4 --with-jvm-jar-dir=/usr/lib/jvm-exports/java-1.5.0-gcj-4.4 --with-arch-directory=i386 --with-ecj-jar=/usr/share/java/eclipse-ecj.jar --enable-targets=all --disable-werror --with-arch-32=i486 --with-tune=generic --enable-checking=release --build=i486-linux-gnu --host=i486-linux-gnu --target=i486-linux-gnu Thread model: posix gcc version 4.4.1 (Ubuntu 4.4.1-5ubuntu2) ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: How to minimize (unshareable) memory usage? 2010-01-18 9:40 ` How to minimize (unshareable) memory usage? Justin Santa Barbara @ 2010-01-18 9:50 ` Andrew Haley 2010-01-18 17:11 ` Hans Boehm 2010-03-18 15:36 ` hari001 1 sibling, 1 reply; 7+ messages in thread From: Andrew Haley @ 2010-01-18 9:50 UTC (permalink / raw) To: Justin Santa Barbara; +Cc: java On 01/18/2010 09:40 AM, Justin Santa Barbara wrote: > I have some simple Java utilities that I've written that need to run > on machines with limited free memory, and I'm looking at GCJ for this. > Using the full Sun JVM we run into problems with memory usage - we're > able to limit the heap to 8MB, but we need to allow at least 16MB for > PermGen (despite not really having much application code), and at > least 8MB for ReservedCodeCacheSize; even a simple utility therefore > needs at least 32MB of unshareable memory. > > Running under GCJ the results are encouraging, but still not as good > as I'd hope. The output from pmap is included below. The interesting > ones are these four: > 00010000 10404K rw--- [ anon ] > b4125000 8192K rw--- [ anon ] > b49d2000 8192K rw--- [ anon ] > b7096000 6324K rw--- /usr/lib/libgcj.so.10.0.0 > > I've limited GCJ to 10MB of heap (it seems to need a little more than > the Sun JVM for some reason), so that's the first entry. But what are > the two 8MB anonymous allocations that aren't the 10MB heap? What is > the 6MB of read/write libgcj mappings? Hard to be sure, but propbably more heap. To find out for sure I'd put a breakpoint on mmap(). Andrew. ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: How to minimize (unshareable) memory usage? 2010-01-18 9:50 ` Andrew Haley @ 2010-01-18 17:11 ` Hans Boehm 2010-01-18 17:56 ` David Daney 0 siblings, 1 reply; 7+ messages in thread From: Hans Boehm @ 2010-01-18 17:11 UTC (permalink / raw) To: Andrew Haley; +Cc: Justin Santa Barbara, java The output of GC_dump() might also tell you something. It won't list heap sections associated with the system malloc. Note that the collector doesn't insist on a contiguous heap. Hans ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: How to minimize (unshareable) memory usage? 2010-01-18 17:11 ` Hans Boehm @ 2010-01-18 17:56 ` David Daney 2010-01-19 10:34 ` Justin Santa Barbara 0 siblings, 1 reply; 7+ messages in thread From: David Daney @ 2010-01-18 17:56 UTC (permalink / raw) To: Hans Boehm; +Cc: Andrew Haley, Justin Santa Barbara, java Hans Boehm wrote: > > The output of GC_dump() might also tell you something. It won't > list heap sections associated with the system malloc. Note that the > collector doesn't insist on a contiguous heap. > And if you want even more details about how memory is being used, there is always the gnu.gcj.util.GCInfo class. That and the gnu.gcj.tools.gc_analyze.MemoryAnalyze post processor give quite a bit of information. These facilities are built on top of GC_dump() but provide a more detailed analysis of the contents of the memory. David Daney ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: How to minimize (unshareable) memory usage? 2010-01-18 17:56 ` David Daney @ 2010-01-19 10:34 ` Justin Santa Barbara 2010-01-19 22:52 ` Boehm, Hans 0 siblings, 1 reply; 7+ messages in thread From: Justin Santa Barbara @ 2010-01-19 10:34 UTC (permalink / raw) To: java; +Cc: Hans Boehm, Andrew Haley, David Daney Thanks to everyone for the great suggestions. I've tracked down the mysterious 8MB segments: they are the stacks for my threads (by putting a breakpoint on mmap) Though reserved in virtual memory space, pages shouldn't be allocated unless they're actually touched, so this is good news for physical memory. I verified this by dumping /proc/<pid>/smaps; the output of one of my 8MB segments shows that only 16KB or so is really used. (Output attached at the end) So the next biggest culprit is my heap memory usage ... I tried using GCInfo.setOOMDump, and did get a dump. But all is not entirely well: my GC_MAXIMUM_HEAP_SIZE was set to 8,000,000 bytes, which results in an out of memory dump, but gc-analyze reports only 5.5MB usage, and of that half is listed under the 'free' column in the "Used Blocks" section of the dump. *** Used Blocks *** Size Kind Blocks Used Free Wasted ------- ------------- ------- ---------- ---------- ------- <...snipped...> ------- ------------- ------- ---------- ---------- ------- 1,352 2,700,672 2,758,624 78,496 Total bytes = 5,537,792 The percentages elsewhere also support the idea that about 3MB of data is actually in use, so I'm not sure exactly why we're failing memory requests when we don't seem to be close to the limit. As for the 6MB of writeable data that shows against libgcj, I believe this is the 'normal' data segment of the shared library. Cutting this down would probably be difficult. The easiest approach for me would be modularizing gcj so I don't have to pay for swing or awt. Again, suggestions are very welcome, but I think that this particular segment will be much more complicated than figuring out why the heap is not being used as efficiently as I hope it could be. Thanks Justin --- The smap output: 8MB of stack, but only 16KB actually used: b4b05000-b5305000 rw-p 00000000 00:00 0 Size: 8192 kB Rss: 16 kB Pss: 16 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 16 kB Referenced: 16 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB The heap segment (with a 10,000,000 byte limit to avoid crashing, and captured on a different machine from the other two): 7f43a12b0000-7f43a1bc0000 rw-p 7f43a12b0000 00:00 0 Size: 9280 kB Rss: 9280 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 9280 kB Referenced: 9280 kB The data segment of libgcj (?), mostly dirty: b7122000-b7739000 rw-p 01ae0000 08:01 28534016 /usr/local/lib/libgcj.so.10.0.0 Size: 6236 kB Rss: 6236 kB Pss: 6236 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 1644 kB Private_Dirty: 4592 kB Referenced: 6236 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB ^ permalink raw reply [flat|nested] 7+ messages in thread
* RE: How to minimize (unshareable) memory usage? 2010-01-19 10:34 ` Justin Santa Barbara @ 2010-01-19 22:52 ` Boehm, Hans 0 siblings, 0 replies; 7+ messages in thread From: Boehm, Hans @ 2010-01-19 22:52 UTC (permalink / raw) To: Justin Santa Barbara, java; +Cc: Andrew Haley, David Daney I think the current code fails the heap expansion if it would have resulted in a heap larger than specified; it doesn't reduce the size of the heap expansion so that it would barely still fit. This means it might be failing sooner than you intended. Also setting GC_INITIAL_HEAP_SIZE may help. I would put a breakpoint in the failing GC_expand_hp_inner call and then call GC_dump(),which will tell you more about the collector's view of the heap. Since this collector doesn't move objects, it's quite possible that there just wasn't a hole large enough for the requested object, particularly if you allocate some really large objects. If that is indeed the case, you may be able to circumvent the issue by allocating multiple smaller objects instead. The other, sometimes significant, source of space overhead is the fact that the collector splits the heap into chunks whose size is the smallest possible multiple of 4K or 8K, depending on the paltform, and then uses each chunk for a fixed object size. Thus objects of size 2049 or 4097 bytes (including the Java header) tend to waste a lot of space. Hans > -----Original Message----- > From: Justin Santa Barbara [mailto:justin@fathomdb.com] > Sent: Tuesday, January 19, 2010 2:34 AM > To: java@gcc.gnu.org > Cc: Boehm, Hans; Andrew Haley; David Daney > Subject: Re: How to minimize (unshareable) memory usage? > > Thanks to everyone for the great suggestions. I've tracked > down the mysterious 8MB segments: they are the stacks for my > threads (by putting a breakpoint on mmap) Though reserved in > virtual memory space, pages shouldn't be allocated unless > they're actually touched, so this is good news for physical > memory. I verified this by dumping /proc/<pid>/smaps; the > output of one of my 8MB segments shows that only 16KB or so > is really used. (Output attached at the end) > > So the next biggest culprit is my heap memory usage ... I > tried using GCInfo.setOOMDump, and did get a dump. But all > is not entirely well: > my GC_MAXIMUM_HEAP_SIZE was set to 8,000,000 bytes, which > results in an out of memory dump, but gc-analyze reports only > 5.5MB usage, and of that half is listed under the 'free' > column in the "Used Blocks" > section of the dump. > > *** Used Blocks *** > > Size Kind Blocks Used Free Wasted > ------- ------------- ------- ---------- ---------- ------- > <...snipped...> > ------- ------------- ------- ---------- ---------- ------- > 1,352 2,700,672 2,758,624 78,496 > Total bytes = 5,537,792 > > The percentages elsewhere also support the idea that about > 3MB of data is actually in use, so I'm not sure exactly why > we're failing memory requests when we don't seem to be close > to the limit. > > As for the 6MB of writeable data that shows against libgcj, I > believe this is the 'normal' data segment of the shared > library. Cutting this down would probably be difficult. The > easiest approach for me would be modularizing gcj so I don't > have to pay for swing or awt. Again, suggestions are very > welcome, but I think that this particular segment will be > much more complicated than figuring out why the heap is not > being used as efficiently as I hope it could be. > > Thanks > Justin > > --- > > The smap output: > > 8MB of stack, but only 16KB actually used: > > b4b05000-b5305000 rw-p 00000000 00:00 0 > Size: 8192 kB > Rss: 16 kB > Pss: 16 kB > Shared_Clean: 0 kB > Shared_Dirty: 0 kB > Private_Clean: 0 kB > Private_Dirty: 16 kB > Referenced: 16 kB > Swap: 0 kB > KernelPageSize: 4 kB > MMUPageSize: 4 kB > > The heap segment (with a 10,000,000 byte limit to avoid > crashing, and captured on a different machine from the other two): > 7f43a12b0000-7f43a1bc0000 rw-p 7f43a12b0000 00:00 0 > Size: 9280 kB > Rss: 9280 kB > Shared_Clean: 0 kB > Shared_Dirty: 0 kB > Private_Clean: 0 kB > Private_Dirty: 9280 kB > Referenced: 9280 kB > > > The data segment of libgcj (?), mostly dirty: > b7122000-b7739000 rw-p 01ae0000 08:01 28534016 > /usr/local/lib/libgcj.so.10.0.0 > Size: 6236 kB > Rss: 6236 kB > Pss: 6236 kB > Shared_Clean: 0 kB > Shared_Dirty: 0 kB > Private_Clean: 1644 kB > Private_Dirty: 4592 kB > Referenced: 6236 kB > Swap: 0 kB > KernelPageSize: 4 kB > MMUPageSize: 4 kB > ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: How to minimize (unshareable) memory usage? 2010-01-18 9:40 ` How to minimize (unshareable) memory usage? Justin Santa Barbara 2010-01-18 9:50 ` Andrew Haley @ 2010-03-18 15:36 ` hari001 1 sibling, 0 replies; 7+ messages in thread From: hari001 @ 2010-03-18 15:36 UTC (permalink / raw) To: java Hi, Johnny this side from Canada. As i have seen your comment regarding java & I really appreciate this from begun to end. Thank You Johnny http://www.baldheadcabinets.com/garage-storage-cabinet-systems.html garage storage cabinet -- View this message in context: http://old.nabble.com/How-to-minimize-%28unshareable%29-memory-usage--tp27208054p27947332.html Sent from the gcc - java mailing list archive at Nabble.com. ^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2010-03-18 15:36 UTC | newest] Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- [not found] <a3fa9dad1001180133h252738bfl5d887cb2670475e3@mail.gmail.com> 2010-01-18 9:40 ` How to minimize (unshareable) memory usage? Justin Santa Barbara 2010-01-18 9:50 ` Andrew Haley 2010-01-18 17:11 ` Hans Boehm 2010-01-18 17:56 ` David Daney 2010-01-19 10:34 ` Justin Santa Barbara 2010-01-19 22:52 ` Boehm, Hans 2010-03-18 15:36 ` hari001
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).