problem when mapping malloc to GC_malloc.

problem when mapping malloc to GC_malloc.

[abhishek desai]

Hi,

My JNI code includes redefinitions to the malloc, free and realloc
functions (shown below). These functions call GC_malloc, GC_free and
GC_realloc respectively. This is done so that any calls to the malloc
get allocated through the garbage collector. However this is failing
with segfault. Any clues why this does not work ?
I am using this code along with the libgcj library linked dynamically
with my application.

void *malloc(size_t size)
{
        return GC_malloc(size);
}

void *realloc(void *ptr, size_t size)
{
        return GC_realloc(ptr, size);
}

void free(void *ptr)
{
        GC_free(ptr);
}


regards
abhishek

Re: problem when mapping malloc to GC_malloc.

[Christian Nolte]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


Hi Abhishek,

abhishek desai wrote:
> My JNI code includes redefinitions to the malloc, free and realloc
> functions (shown below). These functions call GC_malloc, GC_free and
> GC_realloc respectively. This is done so that any calls to the malloc
> get allocated through the garbage collector. 

I presume you are using Hans Boehm's garbage collector? If so, did you
follow the instructions on:

http://www.hpl.hp.com/personal/Hans_Boehm/gc/simple_example.html

What platform are you developing for? If it is not Linux, you have to
make sure that GC_INIT() is being called before using any of the GC_xxx
functions.


> However this is failing
> with segfault. Any clues why this does not work ?
> I am using this code along with the libgcj library linked dynamically
> with my application.
> 
> void *malloc(size_t size)
> {
>         return GC_malloc(size);
> }
> 
> void *realloc(void *ptr, size_t size)
> {
>         return GC_realloc(ptr, size);
> }
> 
> void free(void *ptr)
> {
>         GC_free(ptr);
> }

Perhaps you can give us some more context. A simple, compilable example
would be great to see where and how exactly you are using your
redefinitions.

Best regards,
Christian


- --
"He's an unconventional pirate paramedic with a mysterious suitcase
 handcuffed to his arm. She's a pregnant gypsy widow from aristocratic
 European stock. They fight crime!"

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org

iEYEARECAAYFAko6WboACgkQCNjA0nfhW7xDYwCgpI3ntEEScbqbQOpCkfi1Z3KG
lrQAoI8+xh/swyVSgREAMmjX1B8rIWFl
=A/lO
-----END PGP SIGNATURE-----

Re: problem when mapping malloc to GC_malloc.

[Hans Boehm]

On Thu, 18 Jun 2009, abhishek desai wrote:

> Hi,
>
> My JNI code includes redefinitions to the malloc, free and realloc
> functions (shown below). These functions call GC_malloc, GC_free and
> GC_realloc respectively. This is done so that any calls to the malloc
> get allocated through the garbage collector. However this is failing
> with segfault. Any clues why this does not work ?
> I am using this code along with the libgcj library linked dynamically
> with my application.
>
> void *malloc(size_t size)
> {
>        return GC_malloc(size);
> }
>
> void *realloc(void *ptr, size_t size)
> {
>        return GC_realloc(ptr, size);
> }
>
> void free(void *ptr)
> {
>        GC_free(ptr);
> }
>
>
> regards
> abhishek
>
The collector itself supports a REDIRECT_MALLOC option that might get you 
closer.  In general, this is very hard.

There are other functions (calloc, memalign, etc.) that you would also 
have to replace, so that their clients don't end up using the original 
malloc with GC_free.  This is the easy part.

The hard part is that if you replace malloc, low level parts of the system 
will also end up using GC_malloc, and sometimes squirrel away pointers to 
the results in places the GC doesn't really know about.  Recent versions 
of the GC (7.1+) contains some hacks to try to handle this on Linux.  But 
the multithreaded versions still are sometimes not 100% robust.  Gcj's 
version is unlikely to work in this mode, except possibly in 
single-threaded mode.

A real fix here would probably require some new hooks in glibc and the 
startup and libpthread code.

Hans

Re: problem when mapping malloc to GC_malloc.

[BGB]

----- Original Message ----- 
From: "abhishek desai" <abhi00@gmail.com>
To: <java@gcc.gnu.org>
Sent: Thursday, June 18, 2009 8:00 AM
Subject: problem when mapping malloc to GC_malloc.


> Hi,
>
> My JNI code includes redefinitions to the malloc, free and realloc
> functions (shown below). These functions call GC_malloc, GC_free and
> GC_realloc respectively. This is done so that any calls to the malloc
> get allocated through the garbage collector. However this is failing
> with segfault. Any clues why this does not work ?
> I am using this code along with the libgcj library linked dynamically
> with my application.
>
> void *malloc(size_t size)
> {
>        return GC_malloc(size);
> }
>
> void *realloc(void *ptr, size_t size)
> {
>        return GC_realloc(ptr, size);
> }
>
> void free(void *ptr)
> {
>        GC_free(ptr);
> }
>

IMO, this is a very bad way to try to use a GC, and that it is blowing up 
when something like this is tried is no real surprise...

(very likely you will end up with code calling GC_malloc in some places, and 
the real malloc in others, and in-all creating a horrible mess...).


a much better (although IMO still not very good) way to do this would be to 
instead use macros to rename your functions:
#define malloc(x) GC_malloc(x)
...

which will apply locally to whatever code is in question (and, if used with 
caution, should resist blowing up...).


however, my personal recommendation is to NOT use a GC implicitly in this 
way, rather one should make use of the GC explicit (AKA: actually go and 
rename their function calls, and keep track of how they use pointers so that 
GC memory and malloc memory don't get mixed up, ...).


>
> regards
> abhishek
> 

Re: problem when mapping malloc to GC_malloc.

[abhishek desai]

On Thu, Jun 18, 2009 at 10:09 PM, Hans Boehm<Hans.Boehm@hp.com> wrote:
>
>
> On Thu, 18 Jun 2009, abhishek desai wrote:
>
>> Hi,
>>
>> My JNI code includes redefinitions to the malloc, free and realloc
>> functions (shown below). These functions call GC_malloc, GC_free and
>> GC_realloc respectively. This is done so that any calls to the malloc
>> get allocated through the garbage collector. However this is failing
>> with segfault. Any clues why this does not work ?
>> I am using this code along with the libgcj library linked dynamically
>> with my application.
>>
>> void *malloc(size_t size)
>> {
>>       return GC_malloc(size);
>> }
>>
>> void *realloc(void *ptr, size_t size)
>> {
>>       return GC_realloc(ptr, size);
>> }
>>
>> void free(void *ptr)
>> {
>>       GC_free(ptr);
>> }
>>
>>
>> regards
>> abhishek
>>
> The collector itself supports a REDIRECT_MALLOC option that might get you
> closer.  In general, this is very hard.
>
> There are other functions (calloc, memalign, etc.) that you would also have
> to replace, so that their clients don't end up using the original malloc
> with GC_free.  This is the easy part.
>
> The hard part is that if you replace malloc, low level parts of the system
> will also end up using GC_malloc, and sometimes squirrel away pointers to
> the results in places the GC doesn't really know about.  Recent versions of
> the GC (7.1+) contains some hacks to try to handle this on Linux.  But the
> multithreaded versions still are sometimes not 100% robust.  Gcj's version
> is unlikely to work in this mode, except possibly in single-threaded mode.
>
> A real fix here would probably require some new hooks in glibc and the
> startup and libpthread code.
>
> Hans
>

>if you replace malloc, low level parts of the system
> will also end up using GC_malloc
This is what I am trying to achieve but this is turning out to be a tough task.

Some more background here. I am using the gcc version 3.4.6 and the
libgcj associated with it. I know its ancient but the project demands
it. Cant help it. I am compiling for a mipsel architecture using
uclibc. There are a lot of calls to malloc through the libgcj library
or possibly in the other libraries used by libgcj. The calls to malloc
happen even before the main function begins.

uclibc implementation of memory management (malloc, realloc) relies on
mmap and sbrk to get memory which is also used by the gc for the same
purpose. So is not possible to use the gc instead of the uclibc memory
manager ? Which areas could be causing the problem ? Even if low level
parts of the system make calls to GC_malloc how does it affect the
over all behavior ?

--Abhishek

Re: problem when mapping malloc to GC_malloc.

[BGB]

----- Original Message ----- 
From: "abhishek desai" <abhi00@gmail.com>
To: "Hans Boehm" <Hans.Boehm@hp.com>
Cc: <java@gcc.gnu.org>
Sent: Thursday, June 18, 2009 11:18 AM
Subject: Re: problem when mapping malloc to GC_malloc.


On Thu, Jun 18, 2009 at 10:09 PM, Hans Boehm<Hans.Boehm@hp.com> wrote:
>
>
> On Thu, 18 Jun 2009, abhishek desai wrote:
>
>> Hi,
>>
>> My JNI code includes redefinitions to the malloc, free and realloc
>> functions (shown below). These functions call GC_malloc, GC_free and
>> GC_realloc respectively. This is done so that any calls to the malloc
>> get allocated through the garbage collector. However this is failing
>> with segfault. Any clues why this does not work ?
>> I am using this code along with the libgcj library linked dynamically
>> with my application.
>>
>> void *malloc(size_t size)
>> {
>> return GC_malloc(size);
>> }
>>
>> void *realloc(void *ptr, size_t size)
>> {
>> return GC_realloc(ptr, size);
>> }
>>
>> void free(void *ptr)
>> {
>> GC_free(ptr);
>> }
>>
>>
>> regards
>> abhishek
>>
> The collector itself supports a REDIRECT_MALLOC option that might get you
> closer. In general, this is very hard.
>
> There are other functions (calloc, memalign, etc.) that you would also 
> have
> to replace, so that their clients don't end up using the original malloc
> with GC_free. This is the easy part.
>

I will say it is not so easy...

the primary issue here is DLLs / Shared Objects, which may well continue 
using the original functions even though one has managed to replace them in 
the statically-linked portion of the app...

as such, a complete replacement would likely require eliminating any use of 
shared objects as well...

to make matters worse, C runtimes are typically themselves DLLs or SOs, so 
effectively, one would have to likely replace the entire C runtime with a 
customized, statically-linked version...

granted, it may not always be that bad (after all, some OS's only use static 
linking, and others may well prefer the versions of symbols from the main 
binary rather than depended-on libs, ...), but IMO it is neither safe nor 
portable...


> The hard part is that if you replace malloc, low level parts of the system
> will also end up using GC_malloc, and sometimes squirrel away pointers to
> the results in places the GC doesn't really know about. Recent versions of
> the GC (7.1+) contains some hacks to try to handle this on Linux. But the
> multithreaded versions still are sometimes not 100% robust. Gcj's version
> is unlikely to work in this mode, except possibly in single-threaded mode.
>

agreed...

granted most code does not do such things, there is no real guerantee that 
none of the code does so.
for example, the well known XOR-pointer hacks, or representing pointers in a 
serialized form or in misaligned memory, ...


> A real fix here would probably require some new hooks in glibc and the
> startup and libpthread code.
>

agreed...
then it only necessarily works on Linux, but this is still better than 
nothing I guess...


> Hans
>

replying to "abhishek desai":

>> if you replace malloc, low level parts of the system
>> will also end up using GC_malloc
> This is what I am trying to achieve but this is turning out to be a tough 
> task.
>

the question is: does it actually make sense to do so?...

just because you may want to do something does not mean it is a good idea.


> Some more background here. I am using the gcc version 3.4.6 and the
> libgcj associated with it. I know its ancient but the project demands
> it. Cant help it. I am compiling for a mipsel architecture using
> uclibc. There are a lot of calls to malloc through the libgcj library
> or possibly in the other libraries used by libgcj. The calls to malloc
> happen even before the main function begins.
>

this is to be expected.
now where is the REAL problem?...


> uclibc implementation of memory management (malloc, realloc) relies on
> mmap and sbrk to get memory which is also used by the gc for the same
> purpose. So is not possible to use the gc instead of the uclibc memory
> manager ? Which areas could be causing the problem ? Even if low level
> parts of the system make calls to GC_malloc how does it affect the
> over all behavior ?
>

theoretically, a GC "could" be used as the main malloc, however, doing so is 
not always a good idea:
technically, it violates conformance with the C standard (malloc is defined 
to behave in a certain way, and GC_malloc does not necessarily behave in 
this way, and this variation is allowed to cause existing code to break);
it creates a big ugly issue in trying to actually do so (apart from OS 
developers, this task is not entirely practical);
one ends up additionally weighing down the GC with lots of additional work, 
thus reducing its overall performance;
...

so, my personal suggestion is to just let malloc be...

one can then, not worry about trying to replace malloc, and instead simply 
use the GC in addition to malloc (being careful not to mix them up, 
however...).


one may find that each has unique advantages:
GC is good for letting objects be garbage collected;
malloc is good for quickly grabbing big chunks of memory and then freeing 
them when done (for example, for temporary buffers and similar).

as I see it, there is no real problem in letting both sit around abd each do 
their respective tasks...


> --Abhishek