[Bug libc/12232] New: setrlimit interferes with malloc

[Bug libc/12232] New: setrlimit interferes with malloc

[bkorb at gnu dot org]

http://sourceware.org/bugzilla/show_bug.cgi?id=12232

           Summary: setrlimit interferes with malloc
           Product: glibc
           Version: unspecified
            Status: NEW
          Severity: critical
          Priority: P2
         Component: libc
        AssignedTo: drepper.fsp@gmail.com
        ReportedBy: bkorb@gnu.org


The following code comes from gnulib.  It often but does
not always fail.  The problem may be in the OS or in
glibc but I cannot tell because _int_malloc() is too
twisty for me to readily follow.

int
main (int argc, char *argv[])
{
  struct rlimit limit;
  int arg;
  int repeat;
  void * memory;
  if (getrlimit (RLIMIT_AS, &limit) < 0)
    return 77;
  if (limit.rlim_max == RLIM_INFINITY || limit.rlim_max > MAX_ALLOC_TOTAL)
    limit.rlim_max = MAX_ALLOC_TOTAL;
  limit.rlim_cur = limit.rlim_max;
  if (setrlimit (RLIMIT_AS, &limit) < 0)
    return 77;
  memory = malloc (0x88);

By adding this line of code before the "getrlimit" call:
  free (malloc (0x88));

the problem is not seen.  So, if you use malloc() before the
setrlimit call, you are fine.  There is an interaction issue
with possible problems on both sides of the OS/user space divide.


$ uname -a;/lib64/libc.so.6 |head -1;cat /etc/SuSE-release
Linux beethoven 2.6.34.7-0.4-desktop #1 SMP PREEMPT 2010-10-07 19:07:51 +0200
x86_64 x86_64 x86_64 GNU/Linux
GNU C Library stable release version 2.11.2 (20100531), by Roland McGrath et
al.
openSUSE 11.3 (x86_64)
VERSION = 11.3

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug libc/12232] setrlimit interferes with malloc

[drepper.fsp at gmail dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=12232

Ulrich Drepper <drepper.fsp at gmail dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|                            |WORKSFORME

--- Comment #1 from Ulrich Drepper <drepper.fsp at gmail dot com> 2010-11-18 19:01:50 UTC ---
rlimits are enforced in the kernel.  If you don't like the way it's done argue
with the kernel people.

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug libc/12232] setrlimit interferes with malloc

[bkorb at gnu dot org]

http://sourceware.org/bugzilla/show_bug.cgi?id=12232

Bruce Korb <bkorb at gnu dot org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|RESOLVED                    |REOPENED
         Resolution|WORKSFORME                  |

--- Comment #2 from Bruce Korb <bkorb at gnu dot org> 2010-11-18 20:14:25 UTC ---
You did not read this very well.  I know rlimits are enforced
in the kernel.  The problem is that malloc() works correctly
when you use it before calling setrlimit and fails otherwise.
You have some responsibility for ensuring that you are correctly
using the interface before blowing this off.  Malloc should not
fail when you call setrlimit to 10,000,000 before the first call,
whether the fault lies with your code or the kernel.

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug libc/12232] setrlimit interferes with malloc

[jakub at redhat dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=12232

Jakub Jelinek <jakub at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|REOPENED                    |RESOLVED
                 CC|                            |jakub at redhat dot com
         Resolution|                            |INVALID

--- Comment #3 from Jakub Jelinek <jakub at redhat dot com> 2010-11-18 20:26:54 UTC ---
Why not?  If you call malloc before the setrlimit call and then free, it in
many cases will leave some heap pages (either brk area or mmap) mmapped in the
address space, so following malloc, if it can be satisfied from those pages,
can succeed.
If on the other side there are is not enough space in the current pages to
satisfy the request and new brk extension or mmap will be forbidden by the
kernel because of exceeding address space limit, of course malloc will fail.

There is nothing malloc code can do about it.  Scanning all the time
/proc/maps, calling getrlimit and counting if there is room, in addition to be
terribly slow, wouldn't buy you anything, if the kernel won't allow you to
allocate anything, the userspace can't workaround it.

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug libc/12232] setrlimit interferes with malloc

[drepper.fsp at gmail dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=12232

--- Comment #4 from Ulrich Drepper <drepper.fsp at gmail dot com> 2010-11-18 20:39:50 UTC ---
It doesn't matter one bit whether after a malloc things are different.  It is
still the kernel which makes different decisions.

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug libc/12232] setrlimit interferes with malloc

[bkorb at gnu dot org]

http://sourceware.org/bugzilla/show_bug.cgi?id=12232

Bruce Korb <bkorb at gnu dot org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|RESOLVED                    |REOPENED
         Resolution|INVALID                     |

--- Comment #5 from Bruce Korb <bkorb at gnu dot org> 2010-11-18 20:56:08 UTC ---
So, in other words, if you have a process that uses some
10's of K of address space and calls setrlimit to set
it to 10 million bytes, then it is okay for malloc
to fail for a 136 byte allocation?  I think not.

The failure can either be in malloc code or kernel code
and I, as a user space programmer, have no real way to
know if malloc has misused the sbrk()/mmap()/whatever
interface, or if the kernel is mis-responding to those
system calls.  All I know is that I set my address space
size to 10 million bytes in a process that uses no more
than 1 meg of space.  I call malloc(0x88).  I expect
the _first_ call to succeed.  It does not.  There is
no shortage of RAM or swap space.

By the way, after a successful call to malloc, the available
space can then be filled by many calls to malloc.  Presuming that
the first call for 136 bytes did not create a 10 mb arena,
either the first arena was extended or more arenas were
created.  In other words, the malloc usage of sbrk is then
successful.

The code referenced in the description should not fail,
given sufficient RAM and swap space.  I have the space.

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug libc/12232] setrlimit interferes with malloc

[bkorb at gnu dot org]

http://sourceware.org/bugzilla/show_bug.cgi?id=12232

--- Comment #6 from Bruce Korb <bkorb at gnu dot org> 2010-11-18 20:59:44 UTC ---
Here is a theory that might explain the symptom.
Presume malloc requests 16meg on the first sbrk.
It fails because that exceeds 10meg.  The address
space has been constrained to 10meg.  malloc
should do a little homework before returning NULL,
especially given that this is a first call to malloc.
If that is the problem, then that is a malloc bug.

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug libc/12232] setrlimit interferes with malloc

[roucaries.bastien+bugs at gmail dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=12232

roucaries.bastien+bugs at gmail dot com changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |roucaries.bastien+bugs at
                   |                            |gmail dot com

--- Comment #7 from roucaries.bastien+bugs at gmail dot com 2010-11-22 12:37:51 UTC ---
I use ltrace -S ./a.out on the bruce program and I get this. Dear bruce could
you retest using ltrace ?

Bastien

setrlimit(9, 0x7fff34694fc0, 0x7f06d7cc8df0, 0x40076f, 1 <unfinished ...>
SYS_setrlimit(9, 0x7fff34694fc0)                                               
                                       = 0
<... setrlimit resumed> )                                                      
                                       = 0
malloc(136 <unfinished ...>
SYS_brk(NULL)                                                                  
                                       = 0x020fa000
SYS_brk(0x0211b000)                                                            
                                       = 0x020fa000
SYS_mmap(0, 0x100000, 3, 34, 0xffffffff)                                       
                                       = -12
SYS_mmap(0, 0x8000000, 0, 16418, 0xffffffff)                                   
                                       = -12
SYS_mmap(0, 0x4000000, 0, 16418, 0xffffffff)                                   
                                       = -12
SYS_mmap(0, 0x8000000, 0, 16418, 0xffffffff)                                   
                                       = -12
SYS_mmap(0, 0x4000000, 0, 16418, 0xffffffff)                                   
                                       = -12

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug libc/12232] setrlimit interferes with malloc

[bkorb at gnu dot org]

http://sourceware.org/bugzilla/show_bug.cgi?id=12232

--- Comment #8 from Bruce Korb <bkorb at gnu dot org> 2010-11-23 16:00:12 UTC ---
Next week when I get home, sure.

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug libc/12232] setrlimit interferes with malloc

[bkorb at gnu dot org]

http://sourceware.org/bugzilla/show_bug.cgi?id=12232

--- Comment #9 from Bruce Korb <bkorb at gnu dot org> 2011-01-07 22:26:36 UTC ---
I have now run ltrace a half dozen times and the problem does not recur.
The test cycle is about 20 minutes, so I don't have time to test it
too many ways.  So, it seems, ltrace itself interferes with reproducing
the problem. (The problem recurs when running without ltrace.)
To emphasize:  this problem goes away by itself sometimes.  Once it
does go away, it stays away until I reboot.

Here is an ltrace for a successful run:

$ cat tst/dprintf-posix2-1.ltrace
  0.000000 __libc_start_main(0x400a00, 3, 0x7fff1948c568, 0x400c40, 0x400cd0
<unfinished ...>
  0.000195 getrlimit(2, 0x7fff1948c460, 0x7fff1948c588, 0, 0x2b3ec7a39320) = 0
  0.000193 setrlimit(2, 0x7fff1948c460, 0x7fff1948c588, -1, 0x2b3ec7a39320) = 0
  0.000177 getrlimit(9, 0x7fff1948c460, 0x7fff1948c588, -1, 0x2b3ec7a39320) = 0
  0.000176 setrlimit(9, 0x7fff1948c460, 0x7fff1948c588, -1, 0x2b3ec7a39320) = 0
  0.000177 strtol(0x7fff1948d124, 0, 10, -1, 0x2b3ec7a39320) = 1
  0.000141 rpl_fprintf(0x2b3ec7a38780, 0x400d6a, 17, 1, 0x2b3ec7a38580) =
0xffffffff
  0.000465 __errno_location()                    = 0x2b3ec7c5cac8
  0.001162 +++ exited (status 1) +++

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug libc/12232] setrlimit interferes with malloc

[bkorb at gnu dot org]

http://sourceware.org/bugzilla/show_bug.cgi?id=12232

--- Comment #10 from Bruce Korb <bkorb at gnu dot org> 2011-01-07 23:24:18 UTC ---
Here is a gdb session showing the problem.  It jumps around
a lot because it is at -O2.  What happens is that "getrlimit"
and "setrlimit" are each called twice -- once for RLIMIT_DATA
and once for RLIMIT_AS.  "atoi()" is then called to convert
a the string "1" to binary and that is used to select a "for"
loop.  The first iteration fails:

  for (repeat = 0; repeat < NUM_ROUNDS; repeat++)
    {
      /* This may produce a temporary memory allocation of 11000 bytes.
         but should not result in a permanent memory allocation.  */
      if (dprintf (STDOUT_FILENO, "%011000d\n", 17) == -1
          && errno == ENOMEM)
        return 1;
    }

So it seems the ltrace I sent before is correctly showing the problem
viz., the rpl_fprintf call is failing with ENOMEM.  The test appeared
to succeed because ltrace will exit 0 even if the traced program
exits 1.

The gdb session follows:

Breakpoint 1, main (argc=2, argv=0x7fffffffdc78)
    at ../../tests/test-dprintf-posix2.c:48
48      {
(gdb) n
82        if (getrlimit (RLIMIT_DATA, &limit) < 0)
(gdb) 
48      {
(gdb) 
83          return 77;
(gdb) 
48      {
(gdb) 
82        if (getrlimit (RLIMIT_DATA, &limit) < 0)
(gdb) 
84        if (limit.rlim_max == RLIM_INFINITY || limit.rlim_max >
MAX_ALLOC_TOTAL)
(gdb) 
85          limit.rlim_max = MAX_ALLOC_TOTAL;
(gdb) 
87        if (setrlimit (RLIMIT_DATA, &limit) < 0)
(gdb) 
86        limit.rlim_cur = limit.rlim_max;
(gdb) 
87        if (setrlimit (RLIMIT_DATA, &limit) < 0)
(gdb) 
88          return 77;
(gdb) 
87        if (setrlimit (RLIMIT_DATA, &limit) < 0)
(gdb) 
94        if (getrlimit (RLIMIT_AS, &limit) < 0)
(gdb) 
96        if (limit.rlim_max == RLIM_INFINITY || limit.rlim_max >
MAX_ALLOC_TOTAL)
(gdb) 
97          limit.rlim_max = MAX_ALLOC_TOTAL;
(gdb) 
99        if (setrlimit (RLIMIT_AS, &limit) < 0)
(gdb) 
98        limit.rlim_cur = limit.rlim_max;
(gdb) 
99        if (setrlimit (RLIMIT_AS, &limit) < 0)
(gdb) p limit
$1 = {rlim_cur = 10000000, rlim_max = 10000000}
(gdb) n
100         return 77;
(gdb) 
99        if (setrlimit (RLIMIT_AS, &limit) < 0)
(gdb) 
103       arg = atoi (argv[1]);
(gdb) 
104       if (arg == 0)
(gdb) p arg
$2 = <value optimized out>
(gdb) p argv[1]
$3 = 0x7fffffffe210 "1"
(gdb) s
103       arg = atoi (argv[1]);
(gdb) n
104       if (arg == 0)
(gdb) s
120           if (dprintf (STDOUT_FILENO, "%011000d\n", 17) == -1
(gdb) n
121               && errno == ENOMEM)
(gdb) p errno
$4 = 12
(gdb) n
122             return 1;
(gdb) 
126     }
(gdb) c
Continuing.

Program exited with code 01.

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug libc/12232] setrlimit interferes with malloc

[bkorb at gnu dot org]

http://sourceware.org/bugzilla/show_bug.cgi?id=12232

--- Comment #11 from Bruce Korb <bkorb at gnu dot org> 2011-01-07 23:31:06 UTC ---
One more data point.  The following program does not show
the problem.  So, calling free(malloc(0x88)) clears the issue
and the following code doesn't stumble.

#define NUM_ROUNDS 1000
#define MAX_ALLOC_ROUND 10000
#define MAX_ALLOC_TOTAL (NUM_ROUNDS * MAX_ALLOC_ROUND)

int
main(int argc, char ** argv)
{
    struct rlimit limit;
    if (getrlimit (RLIMIT_AS, &limit) < 0)
        return 77;
    if (limit.rlim_max == RLIM_INFINITY || limit.rlim_max > MAX_ALLOC_TOTAL)
        limit.rlim_max = MAX_ALLOC_TOTAL;
    limit.rlim_cur = limit.rlim_max;
    if (setrlimit (RLIMIT_AS, &limit) < 0)
        return 77;
    if (dprintf (STDOUT_FILENO, "%011000d\n", 17) == -1
        && errno == ENOMEM)
        return 1;

    return 0;
}

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug libc/12232] setrlimit interferes with malloc

[drepper.fsp at gmail dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=12232

Ulrich Drepper <drepper.fsp at gmail dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|REOPENED                    |RESOLVED
         Resolution|                            |INVALID

--- Comment #12 from Ulrich Drepper <drepper.fsp at gmail dot com> 2011-01-08 14:32:03 UTC ---
I already said this is no problem at all.  This is an artifact of the way
limits are implemented.

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug libc/12232] setrlimit interferes with malloc

[roucaries.bastien+bugs at gmail dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=12232

--- Comment #13 from roucaries.bastien+bugs at gmail dot com 2011-01-08 21:32:25 UTC ---
Dear bruce, It seems that the program was tested using ltrace only and not
ltrace -S Thus we do not show the call made by libc to the kernel.

Could you retry ?

bastien

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug libc/12232] setrlimit interferes with malloc

[bkorb at gnu dot org]

http://sourceware.org/bugzilla/show_bug.cgi?id=12232

--- Comment #14 from Bruce Korb <bkorb at gnu dot org> 2011-01-09 17:52:29 UTC ---
Created attachment 5185
  --> http://sourceware.org/bugzilla/attachment.cgi?id=5185
ltrace -S -o /tmp/dprintf-posix2.ltrc ./test-dprintf-posix2 1

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug libc/12232] setrlimit interferes with malloc

[bkorb at gnu dot org]

http://sourceware.org/bugzilla/show_bug.cgi?id=12232

--- Comment #15 from Bruce Korb <bkorb at gnu dot org> 2011-01-09 17:53:22 UTC ---
Created attachment 5186
  --> http://sourceware.org/bugzilla/attachment.cgi?id=5186
ltrace -S -o /tmp/fprintf-posix3.ltrc ./test-fprintf-posix3 1

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug libc/12232] setrlimit interferes with malloc

[bkorb at gnu dot org]

http://sourceware.org/bugzilla/show_bug.cgi?id=12232

--- Comment #16 from Bruce Korb <bkorb at gnu dot org> 2011-01-09 17:54:22 UTC ---
(In reply to comment #13)
> ... the program was tested using ltrace only and not ltrace -S
> ....
> Could you retry ?

Sorry about that.  Full logs are attached.  Representative tail end
is copied here:

__libc_start_main(0x4009c0, 2, 0x7fff27c151e8, 0x4036d0, 0x403760 <unfinished
...>
getrlimit(2, 0x7fff27c150e0, 0x7fff27c15200, 0x7f2965bfe4a8, 0x7f2965bff320
<unfinished ...>
SYS_getrlimit(2, 0x7fff27c150e0)                 = 0
<... getrlimit resumed> )                        = 0
setrlimit(2, 0x7fff27c150e0, 0x7fff27c15200, -1, 0x7f2965bff320 <unfinished
...>
SYS_setrlimit(2, 0x7fff27c150e0)                 = 0
<... setrlimit resumed> )                        = 0
getrlimit(9, 0x7fff27c150e0, 0x7fff27c15200, -1, 0x7f2965bff320 <unfinished
...>
SYS_getrlimit(9, 0x7fff27c150e0)                 = 0
<... getrlimit resumed> )                        = 0
setrlimit(9, 0x7fff27c150e0, 0x7fff27c15200, -1, 0x7f2965bff320 <unfinished
...>
SYS_setrlimit(9, 0x7fff27c150e0)                 = 0
<... setrlimit resumed> )                        = 0
strtol(0x7fff27c17263, 0, 10, -1, 0x7f2965bff320) = 1
malloc(88 <unfinished ...>
SYS_brk(NULL)                                    = 0x00606000
SYS_brk(0x00627000)                              = 0x00606000
SYS_mmap(0, 0x100000, 3, 34, 0xffffffff)         = -12
SYS_mmap(0, 0x8000000, 0, 16418, 0xffffffff)     = -12
SYS_mmap(0, 0x4000000, 0, 16418, 0xffffffff)     = -12
SYS_mmap(0, 0x8000000, 0, 16418, 0xffffffff)     = -12
SYS_mmap(0, 0x4000000, 0, 16418, 0xffffffff)     = -12
<... malloc resumed> )                           = NULL
__errno_location()                               = 0x7f29662506a8
__errno_location()                               = 0x7f29662506a8
SYS_exit_group(1 <no return ...>

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug libc/12232] setrlimit interferes with malloc

[bkorb at gnu dot org]

http://sourceware.org/bugzilla/show_bug.cgi?id=12232

--- Comment #17 from Bruce Korb <bkorb at gnu dot org> 2011-01-09 18:58:53 UTC ---
One more data point:  Replacing "return 1" with "abort()"
yields a 300K image size.  I can send the core, too, if
anyone likes.

$ size core
   text    data     bss     dec     hex filename
  65536  225280       0  290816   47000 core \
(core file invoked as ./test-dprintf-posix2 1)

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug libc/12232] setrlimit interferes with malloc

[bkorb at gnu dot org]

http://sourceware.org/bugzilla/show_bug.cgi?id=12232

Bruce Korb <bkorb at gnu dot org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|RESOLVED                    |REOPENED
         Resolution|INVALID                     |

--- Comment #18 from Bruce Korb <bkorb at gnu dot org> 2011-01-10 01:25:05 UTC ---
RE: core image -- I've uploaded both the core and the binary
here:  http://autogen.sourceforge.net/data

$ ldd test-dprintf-posix2 
        linux-vdso.so.1 =>  (0x00007fff34aeb000)
        librt.so.1 => /lib64/librt.so.1 (0x00007f4783c6d000)
        libm.so.6 => /lib64/libm.so.6 (0x00007f4783a16000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f47836b6000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f4783499000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f4783e76000)

The failing scenario is to run it with one argument: the digit "1".

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug libc/12232] setrlimit interferes with malloc

[drepper.fsp at gmail dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=12232

Ulrich Drepper <drepper.fsp at gmail dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|REOPENED                    |RESOLVED
         Resolution|                            |INVALID

--- Comment #19 from Ulrich Drepper <drepper.fsp at gmail dot com> 2011-01-10 03:19:11 UTC ---
Stop reopening this bug.  There is no bug except in your assumptions.

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug libc/12232] setrlimit interferes with malloc

[bkorb at gnu dot org]

http://sourceware.org/bugzilla/show_bug.cgi?id=12232

--- Comment #20 from Bruce Korb <bkorb at gnu dot org> 2011-01-10 16:00:31 UTC ---
(In reply to comment #19)
> Stop reopening this bug.  There is no bug except in your assumptions.

I could open a new one with a better description, there having been
some progress in understanding in the mean time.  WRT the problem
itself, we know for certain that the process is much smaller than
10Meg and that we set the process address space limit to 10Meg.
We also know that the first tiny allocation fails.  glibc should
support this.  If that is a bad assumption, then glibc *MUST DOCUMENT*
any further constraints, like minimum address space.  If you wish
to require glibc clients to have a minimum of 16 Meg, then
document the constraint, please.  Until then, you have a bug.
(And if you think it is a kernel bug, then please carry it forward.
After all, I cannot go to the kernel folks and say, "You have a bug
because Ulrich said so."  You know much better than I do what goes
on between malloc and the kernel.)

In sum: this _is_ a bug, either against malloc or setrlimit.  You pick.

FWIW, below is a copy of /proc/pid/maps during a live gdb session:

94            if (fprintf (stdout, "%011000d\n", 17) == -1
(gdb) ^Z
[1]+  Stopped(SIGTSTP)        gdb --args ./$f 1
$ cat /proc/8496/maps 
00400000-00404000 r-xp 00000000 09:00 4228629                           
/old-home/gnu/proj/gnulib/libposix/_b/libposix-0.1.libposix-6-03d3-dirty/_build/tests/test-fprintf-posix3
00604000-00605000 r--p 00004000 09:00 4228629                           
/old-home/gnu/proj/gnulib/libposix/_b/libposix-0.1.libposix-6-03d3-dirty/_build/tests/test-fprintf-posix3
00605000-00606000 rw-p 00005000 09:00 4228629                           
/old-home/gnu/proj/gnulib/libposix/_b/libposix-0.1.libposix-6-03d3-dirty/_build/tests/test-fprintf-posix3
7ffff7401000-7ffff7418000 r-xp 00000000 08:06 9043976                   
/lib64/libpthread-2.11.2.so
7ffff7418000-7ffff7618000 ---p 00017000 08:06 9043976                   
/lib64/libpthread-2.11.2.so
7ffff7618000-7ffff7619000 r--p 00017000 08:06 9043976                   
/lib64/libpthread-2.11.2.so
7ffff7619000-7ffff761a000 rw-p 00018000 08:06 9043976                   
/lib64/libpthread-2.11.2.so
7ffff761a000-7ffff761e000 rw-p 00000000 00:00 0 
7ffff761e000-7ffff7774000 r-xp 00000000 08:06 9044083                   
/lib64/libc-2.11.2.so
7ffff7774000-7ffff7974000 ---p 00156000 08:06 9044083                   
/lib64/libc-2.11.2.so
7ffff7974000-7ffff7978000 r--p 00156000 08:06 9044083                   
/lib64/libc-2.11.2.so
7ffff7978000-7ffff7979000 rw-p 0015a000 08:06 9044083                   
/lib64/libc-2.11.2.so
7ffff7979000-7ffff797e000 rw-p 00000000 00:00 0 
7ffff797e000-7ffff79d4000 r-xp 00000000 08:06 9044230                   
/lib64/libm-2.11.2.so
7ffff79d4000-7ffff7bd3000 ---p 00056000 08:06 9044230                   
/lib64/libm-2.11.2.so
7ffff7bd3000-7ffff7bd4000 r--p 00055000 08:06 9044230                   
/lib64/libm-2.11.2.so
7ffff7bd4000-7ffff7bd5000 rw-p 00056000 08:06 9044230                   
/lib64/libm-2.11.2.so
7ffff7bd5000-7ffff7bdd000 r-xp 00000000 08:06 9044289                   
/lib64/librt-2.11.2.so
7ffff7bdd000-7ffff7ddc000 ---p 00008000 08:06 9044289                   
/lib64/librt-2.11.2.so
7ffff7ddc000-7ffff7ddd000 r--p 00007000 08:06 9044289                   
/lib64/librt-2.11.2.so
7ffff7ddd000-7ffff7dde000 rw-p 00008000 08:06 9044289                   
/lib64/librt-2.11.2.so
7ffff7dde000-7ffff7dfd000 r-xp 00000000 08:06 9044076                   
/lib64/ld-2.11.2.so
7ffff7fc8000-7ffff7fcc000 rw-p 00000000 00:00 0 
7ffff7ffa000-7ffff7ffb000 rw-p 00000000 00:00 0 
7ffff7ffb000-7ffff7ffc000 r-xp 00000000 00:00 0                          [vdso]
7ffff7ffc000-7ffff7ffd000 r--p 0001e000 08:06 9044076                   
/lib64/ld-2.11.2.so
7ffff7ffd000-7ffff7ffe000 rw-p 0001f000 08:06 9044076                   
/lib64/ld-2.11.2.so
7ffff7ffe000-7ffff7fff000 rw-p 00000000 00:00 0 
7ffffffde000-7ffffffff000 rw-p 00000000 00:00 0                         
[stack]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                 
[vsyscall]

Compiling -O0 and static linking yielded a larger image:
$ size core-test-fprintf-posix3 
   text    data     bss     dec     hex filename
1687552  225280       0 1912832  1d3000 ....
though still much smaller than 10000000.

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

[Bug libc/12232] setrlimit interferes with malloc

[fweimer at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=12232

Florian Weimer <fweimer at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
              Flags|                            |security-

-- 
You are receiving this mail because:
You are on the CC list for the bug.