using -fsplit-stack

using -fsplit-stack

[mathieu lacage]

hi,

I would like to use -fsplit-stack in a small application of mine so I
tried a simple test program to see if it can be used to build binaries
that can have an unlimited stack space (since the stack is supposed to
be allocated on the heap on demand, I naively expect it to be limited by
the heap allocator) and run and be debugged on a normal linux
installation.

$ cat > test.c << EOF
#include <stdio.h>
int main(int argc, char *argv[])
{
  char buffer[10000000];
  printf("hello %p\n", &buffer);
  return 0;
}
EOF

$ gcc -fsplit-stack ./test.c -o test
$ ./test
hello 0x7fbecfd58960
Segmentation fault (core dumped)

Obviously, the same test will also segfault without -fsplit-stack but
now, it segfaults with an interesting backtrace:

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff7de3af5 in do_lookup_x (new_hash=2090266759,
old_hash=0x7ffff6e7b110, result=0x7ffff6e7b0e0, scope=0x7ffff7ffe5a0,
i=0, flags=1, 
    skip=0x0, undef_map=0x7ffff7ffa9a8) at dl-lookup.c:82
82	dl-lookup.c: No such file or directory.
(gdb) bt
#0  0x00007ffff7de3af5 in do_lookup_x (new_hash=2090266759,
old_hash=0x7ffff6e7b110, result=0x7ffff6e7b0e0, scope=0x7ffff7ffe5a0,
i=0, flags=1, 
    skip=0x0, undef_map=0x7ffff7ffa9a8) at dl-lookup.c:82
#1  0x00007ffff7de4533 in _dl_lookup_symbol_x (undef_name=0x7ffff7816f25
"free", undef_map=0x7ffff7ffa9a8, ref=0x7ffff6e7b170, 
    symbol_scope=0x7ffff7ffad00, version=0x7ffff7fd9180, type_class=1,
flags=1, skip_map=0x0) at dl-lookup.c:739
#2  0x00007ffff7de8784 in _dl_fixup (l=<optimized out>,
reloc_arg=<optimized out>) at ../elf/dl-runtime.c:119
'#3  0x00007ffff7def245 in _dl_runtime_resolve ()
at ../sysdeps/x86_64/dl-trampoline.S:42
#4  0x00007ffff784dac4 in _IO_vfprintf_internal (s=<optimized out>,
format=<optimized out>, ap=<optimized out>) at vfprintf.c:2052
#5  0x00007ffff78588d9 in __printf (format=<optimized out>) at
printf.c:35
#6  0x0000000000400d21 in main ()

It looks like printf cannot find free anymore. Strange.

Because I can imagine the myriad of ways in which this could fail and
lead to this kind of strange hard-to-debug problem, here are a couple of
questions about stack-split itself:

1) I have disassembled the generated code and I see one symbol that
appears to be related to this:__morestack
Is there documentation somewhere on the exact semantics that is supposed
to be provided by this function ? Are there other functions that the
stack-split code will potentially call ?

2) What is the story about interaction with code that is not compiled
with stack-split code ? i.e., can I call non-stack-split code from
stack-split code and vice-versa ?

I read the gcc documentation:

     When code compiled with `-fsplit-stack' calls code compiled
     without `-fsplit-stack', there may not be much stack space
     available for the latter code to run.  If compiling all code,
     including library code, with `-fsplit-stack' is not an option,
     then the linker can fix up these calls so that the code compiled
     without `-fsplit-stack' always has a large stack.  Support for
     this is implemented in the gold linker in GNU binutils release 2.21
     and later.

I think that this means that the gold linker will check that every
library that a binary links to was compiled with split-stack and if not
change the generated code to do something differen. I think that it also
means that GNU linker does not do this and that I need to link my binary
myself with gold. I tried to do this actually but I have not yet been
able to find the right combination of options to make gold find the
binaries that contain __morestack, printf, and __stack_check_fail (yeah,
I am using a dumb ubuntu box)

gold -lc ./test.o -o test
gold: error: ./test.o: could not convert call to '__morestack' to
'__morestack_non_split'
./test.o:test.c:function main: error: undefined reference to
'__morestack'

But, anyway, I can't see how that link-time code change would work for
libraries. Say, binary A links to B and C. A and B  are stack-split. C
is not. What is the linker going to do ?. 

What happens at runtime ? Does the loader perform a similar check ? (if
so, which loader/version ?


3) gdb: Does gcc generate the right kind of dwarf information for gdb to
understand how to unwind a stack-split stack ?

4) Any data on CPU/memory performance for using -fsplit-stack ? i.e.,
what would be the downsides of compiling everything with this feature
on ?

Well, anyway, at this point, I welcome feedback/ideas of any kind :)

thanks for this pretty cool feature,
Mathieu

Re: using -fsplit-stack

[Ian Lance Taylor]

On Fri, Nov 30, 2012 at 6:13 AM, mathieu lacage
<mathieu.lacage@cutebugs.net> wrote:
>
> $ gcc -fsplit-stack ./test.c -o test
> $ ./test
> hello 0x7fbecfd58960
> Segmentation fault (core dumped)

You neglected to say which version of GCC you are using.

I tried your test case with both GCC mainline and the GCC 4.7 branch,
and it worked fine.  But I am using the gold linker, not GNU ld.


> Obviously, the same test will also segfault without -fsplit-stack but
> now, it segfaults with an interesting backtrace:

I don't really see why this test will segfault without using
-fsplit-stack.  For me it works fine without -fsplit-stack, which is
what I would expect.


> Program received signal SIGSEGV, Segmentation fault.
> 0x00007ffff7de3af5 in do_lookup_x (new_hash=2090266759,
> old_hash=0x7ffff6e7b110, result=0x7ffff6e7b0e0, scope=0x7ffff7ffe5a0,
> i=0, flags=1,
>     skip=0x0, undef_map=0x7ffff7ffa9a8) at dl-lookup.c:82
> 82      dl-lookup.c: No such file or directory.
> (gdb) bt
> #0  0x00007ffff7de3af5 in do_lookup_x (new_hash=2090266759,
> old_hash=0x7ffff6e7b110, result=0x7ffff6e7b0e0, scope=0x7ffff7ffe5a0,
> i=0, flags=1,
>     skip=0x0, undef_map=0x7ffff7ffa9a8) at dl-lookup.c:82
> #1  0x00007ffff7de4533 in _dl_lookup_symbol_x (undef_name=0x7ffff7816f25
> "free", undef_map=0x7ffff7ffa9a8, ref=0x7ffff6e7b170,
>     symbol_scope=0x7ffff7ffad00, version=0x7ffff7fd9180, type_class=1,
> flags=1, skip_map=0x0) at dl-lookup.c:739
> #2  0x00007ffff7de8784 in _dl_fixup (l=<optimized out>,
> reloc_arg=<optimized out>) at ../elf/dl-runtime.c:119
> '#3  0x00007ffff7def245 in _dl_runtime_resolve ()
> at ../sysdeps/x86_64/dl-trampoline.S:42
> #4  0x00007ffff784dac4 in _IO_vfprintf_internal (s=<optimized out>,
> format=<optimized out>, ap=<optimized out>) at vfprintf.c:2052
> #5  0x00007ffff78588d9 in __printf (format=<optimized out>) at
> printf.c:35
> #6  0x0000000000400d21 in main ()
>
> It looks like printf cannot find free anymore. Strange.

That is a crash inside the dynamic linker.  I'm not sure why it happened.


> 1) I have disassembled the generated code and I see one symbol that
> appears to be related to this:__morestack
> Is there documentation somewhere on the exact semantics that is supposed
> to be provided by this function ? Are there other functions that the
> stack-split code will potentially call ?

The detailed documentation on that internal function are in the source
code.  libgcc/config/i386/morestack.S.

That function and __morestack_non_split are the only ones that your
application will call directly.  There are various other functions
involved that are called, directly or indirectly, by __morestack.

> 2) What is the story about interaction with code that is not compiled
> with stack-split code ? i.e., can I call non-stack-split code from
> stack-split code and vice-versa ?

Yes, provided you are using gold.

> gold -lc ./test.o -o test
> gold: error: ./test.o: could not convert call to '__morestack' to
> '__morestack_non_split'
> ./test.o:test.c:function main: error: undefined reference to
> '__morestack'

You neglected to say which version of gold you are using.  In any
case, note that you should not invoke gold directly.  You should
arrange for the GCC driver to invoke it on your behalf.  Otherwise you
won't get the right files.  The way to do this is to install gold
under the name "ld" in some directory DIR, and pass the option -BDIR
to gcc.  You can use gcc -Wl,-debug to see precisely which linker gcc
is running, and make sure that it is gold and not GNU ld.

> But, anyway, I can't see how that link-time code change would work for
> libraries. Say, binary A links to B and C. A and B  are stack-split. C
> is not. What is the linker going to do ?.

The linker adjusts the code on a function by function basis, based on
whether the function calls a non-split-stack function or not.  The
general scheme is outlined on the wiki page:
http://gcc.gnu.org/wiki/SplitStacks .

> What happens at runtime ? Does the loader perform a similar check ? (if
> so, which loader/version ?

The loader does not know anything about split stacks, and it doesn't
need to know.

> 3) gdb: Does gcc generate the right kind of dwarf information for gdb to
> understand how to unwind a stack-split stack ?

Yes.


> 4) Any data on CPU/memory performance for using -fsplit-stack ? i.e.,
> what would be the downsides of compiling everything with this feature
> on ?

I don't have any precise measurements.  The cost of code that does not
split the stack is low: a few additional instructions per function
call.  This is unlikely to make any difference except perhaps in a
program that is heavily CPU bound and makes lots of function calls in
tight loops.  However, actually splitting the stack is relatively
expensive because it requires a couple of system calls.  That can add
up pretty quickly if you have a tight loop that winds up needing to
split the stack to an unfortunate coincidence of available stack space
and the stack requirements of the function being called.

Ian

Re: using -fsplit-stack

[mathieu lacage]

On Fri, 2012-11-30 at 07:16 -0800, Ian Lance Taylor wrote:
> On Fri, Nov 30, 2012 at 6:13 AM, mathieu lacage
> <mathieu.lacage@cutebugs.net> wrote:
> >
> > $ gcc -fsplit-stack ./test.c -o test
> > $ ./test
> > hello 0x7fbecfd58960
> > Segmentation fault (core dumped)
> 
> You neglected to say which version of GCC you are using.

Ah.yes, sorry. gcc (Ubuntu/Linaro 4.7.2-2ubuntu1) 4.7.2


> 
> I tried your test case with both GCC mainline and the GCC 4.7 branch,
> and it worked fine.  But I am using the gold linker, not GNU ld.
> 
> 
> > Obviously, the same test will also segfault without -fsplit-stack but
> > now, it segfaults with an interesting backtrace:
> 
> I don't really see why this test will segfault without using
> -fsplit-stack.  For me it works fine without -fsplit-stack, which is
> what I would expect.

Well, it seems to be segfaulting on my box because the default stack
space is smaller that the above (big ?) array.

Program received signal SIGSEGV, Segmentation fault.
0x0000000000400597 in main ()
(gdb) bt
#0  0x0000000000400597 in main ()
(gdb) disas
Dump of assembler code for function main:
   0x000000000040058c <+0>:	push   %rbp
   0x000000000040058d <+1>:	mov    %rsp,%rbp
   0x0000000000400590 <+4>:	sub    $0x9896a0,%rsp
=> 0x0000000000400597 <+11>:	mov    %edi,-0x989694(%rbp)


> > 1) I have disassembled the generated code and I see one symbol that
> > appears to be related to this:__morestack
> > Is there documentation somewhere on the exact semantics that is supposed
> > to be provided by this function ? Are there other functions that the
> > stack-split code will potentially call ?
> 
> The detailed documentation on that internal function are in the source
> code.  libgcc/config/i386/morestack.S.

Thanks a lot for this pointer !


> > gold -lc ./test.o -o test
> > gold: error: ./test.o: could not convert call to '__morestack' to
> > '__morestack_non_split'
> > ./test.o:test.c:function main: error: undefined reference to
> > '__morestack'
> 
> You neglected to say which version of gold you are using.  In any

Right, my bad. GNU gold (GNU Binutils for Ubuntu 2.22.90.20120924) 1.11

> case, note that you should not invoke gold directly.  You should
> arrange for the GCC driver to invoke it on your behalf.  Otherwise you
> won't get the right files.  The way to do this is to install gold
> under the name "ld" in some directory DIR, and pass the option -BDIR
> to gcc.  You can use gcc -Wl,-debug to see precisely which linker gcc
> is running, and make sure that it is gold and not GNU ld.

thanks for the hint: this is especially useful. How is this expected to
fail if I am not using gold ? (at link-time or run-time ?)

> 
> > But, anyway, I can't see how that link-time code change would work for
> > libraries. Say, binary A links to B and C. A and B  are stack-split. C
> > is not. What is the linker going to do ?.
> 
> The linker adjusts the code on a function by function basis, based on
> whether the function calls a non-split-stack function or not.  The
> general scheme is outlined on the wiki page:
> http://gcc.gnu.org/wiki/SplitStacks .

ok

> > 4) Any data on CPU/memory performance for using -fsplit-stack ? i.e.,
> > what would be the downsides of compiling everything with this feature
> > on ?
> 
> I don't have any precise measurements.  The cost of code that does not
> split the stack is low: a few additional instructions per function
> call.  This is unlikely to make any difference except perhaps in a
> program that is heavily CPU bound and makes lots of function calls in
> tight loops.  However, actually splitting the stack is relatively
> expensive because it requires a couple of system calls.  That can add
> up pretty quickly if you have a tight loop that winds up needing to
> split the stack to an unfortunate coincidence of available stack space
> and the stack requirements of the function being called.

Are you aware of anyone who has tried to rebuild a distribution with
this flag and make the system gcc use this flag by default? It would be
nice from a usability perspective to not have to worry about stack size
ever without having to pay too high for this.

Mathieu

Re: using -fsplit-stack

[mathieu lacage]

On Fri, 2012-11-30 at 14:22 -0800, Ian Lance Taylor wrote:
> On Fri, Nov 30, 2012 at 1:09 PM, mathieu lacage
> <mathieu.lacage@cutebugs.net> wrote:
> >
> > Well, it seems to be segfaulting on my box because the default stack
> > space is smaller that the above (big ?) array.
> 
> Please reply to the mailing list, not just to me.  Thanks.

stupid mailer keeps ignoring my "reply all" :/

> 
> I see.  There does seem to be something wrong here; the split stack
> code ought to have allocated a new stack for you.
> 
> > How is this expected to
> > fail if I am not using gold ? (at link-time or run-time ?)
> 
> If you don't use gold your test case will fail at runtime because the
> main function won't realize that it needs to allocate enough stack
> space for printf.
> 
> > Are you aware of anyone who has tried to rebuild a distribution with
> > this flag and make the system gcc use this flag by default? It would be
> > nice from a usability perspective to not have to worry about stack size
> > ever without having to pay too high for this.
> 
> I am not aware of anybody who has done this.  I think it would be a
> significant effort.

Right. I am merely curious. Having -fsplit-stack work out of the box for
new binaries and libraries I compile on normal distributions is good
enough for me. I will report on further experiments, with PIE
executables on other distributions and with other gcc versions.

thanks a lot,
Mathieu