[Bug c/26461] New: liveness of thread local references across function calls

[Bug c/26461] New: liveness of thread local references across function calls

[yichen dot xie at gmail dot com]

I ran into this problem with stock gcc-4.0.2 on Fedora Core 2. The problem
shows when the following code is compiled with -O2: the address of "array"
seems to be cached and reused across calls to function "h". This becomes a
problem when function h saves its execution context in one thread and resumes
in another thread (thus the address of array changes before and after the
call). Is this a feature or a bug? How could I instruct gcc not to reuse thread
local addresses acrosses function calls? Thanks.

__thread int array[1];
int *f()
{
    for (;;)
        h(array);
    return array;
}


-- 
           Summary: liveness of thread local references across function
                    calls
           Product: gcc
           Version: 4.0.2
            Status: UNCONFIRMED
          Severity: critical
          Priority: P3
         Component: c
        AssignedTo: unassigned at gcc dot gnu dot org
        ReportedBy: yichen dot xie at gmail dot com


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=26461

[Bug c/26461] liveness of thread local references across function calls

[pinskia at gcc dot gnu dot org]

------- Comment #1 from pinskia at gcc dot gnu dot org  2006-02-24 21:32 -------
Why do you think this can ever work in general?
Also why do you need this to work this way?  It seems like you are trying to
deal with your own threading system instead of allowing the OS do its work.


-- 

pinskia at gcc dot gnu dot org changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           Severity|critical                    |normal


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=26461

[Bug middle-end/26461] liveness of thread local references across function calls

[pinskia at gcc dot gnu dot org]

-- 

pinskia at gcc dot gnu dot org changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|UNCONFIRMED                 |WAITING
          Component|c                           |middle-end


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=26461

[Bug middle-end/26461] liveness of thread local references across function calls

[yichen dot xie at gmail dot com]

------- Comment #2 from yichen dot xie at gmail dot com  2006-02-24 22:12 -------
(In reply to comment #1)
> It seems like you are trying to
> deal with your own threading system instead of allowing the OS do its work.
> 

This is indeed what I am trying to do, and C seems to be the perfect language
for doing this. I agree it's not common to be switching thread contexts across
function calls, but I don't think it should be prohibited by GCC.

In my case, "h" simply saves its context, put it on the ready queue, and waits
for another thread to pick it up and resume execution with the new thread local
copy of "array". So the question is there a way to force recompuation of
"&array[0]" after h? Is it reasonable to request for a mechanism to force
recomputation of "&array[0]"?

BTW, the solution IMO is simple: either make sure all thread local values and
addresses (the problem seems to exist only with arrays, the compiler is more
conservative dealing with pointers, etc) are dead after a function call, or add
a mechanism (__attribute__((thread_switch))?) to force it.


-- 

yichen dot xie at gmail dot com changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |yichen dot xie at gmail dot
                   |                            |com


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=26461

[Bug middle-end/26461] liveness of thread local references across function calls

[pinskia at gcc dot gnu dot org]

------- Comment #3 from pinskia at gcc dot gnu dot org  2006-02-24 22:38 -------
(In reply to comment #2)
> (In reply to comment #1)
> > It seems like you are trying to
> > deal with your own threading system instead of allowing the OS do its work.
> > 
> 
> This is indeed what I am trying to do, and C seems to be the perfect language
> for doing this. I agree it's not common to be switching thread contexts across
> function calls, but I don't think it should be prohibited by GCC.

Why not let the OS do its job?  I still don't understand that idea.
Actually no it is not responable in general since GCC assumes the address is
invariant which is correct except for your little weird case.  What function
are you using to save/restore the context?  There are no standard C function
which allows for that.  Even get/setcontext are POSIX but I doubt they support
across threads correctly anyways.  I know setjmp/longjmp don't for sure.


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=26461

[Bug middle-end/26461] liveness of thread local references across function calls

[yichen dot xie at gmail dot com]

------- Comment #4 from yichen dot xie at gmail dot com  2006-02-24 23:06 -------
> Why not let the OS do its job?  I still don't understand that idea.

It's a thread library that builds on top of pthreads, so yes, OS is doing its
job, and we're doing more on top of that. C is a natural choice for us. Does it
help if we rename "h" to "reschedule"?

__thread array[1];
for (;;) {
  // do something with array
  reschedule();
}

> Actually no it is not responable in general since GCC assumes the address is
> invariant which is correct except for your little weird case.  What function

Well, it may be a bit weird for any other language, but not C (IMO). It's
definitely not weird if you compare it to the kernel, which is largely written
in C. 

Thread local objects are invariant within a thread, not across threads. I think
it could be dangerous for gcc to assume that function calls preserve thread
context, esp. when the function is written in assembly. At least there should
be a way to tell the compiler not to assume that, given C is a low-level
language where everything should be possible.

> are you using to save/restore the context?  There are no standard C function

Very simple assembly code that stores/restores a few registers, including %esp.
C is a low level language, and it should interoperate well not only with
standard C functions, but also with assembly or any other "weird" functions.
That's what C is good for, isn't it?

> which allows for that.  Even get/setcontext are POSIX but I doubt they support
> across threads correctly anyways.  I know setjmp/longjmp don't for sure.


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=26461

[Bug middle-end/26461] liveness of thread local references across function calls

[pinskia at gcc dot gnu dot org]

------- Comment #5 from pinskia at gcc dot gnu dot org  2006-02-25 00:02 -------
ISO C is not your normal low level language any more.  It actually tries to be
a high level language.

So this is not a bug.


-- 

pinskia at gcc dot gnu dot org changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|WAITING                     |RESOLVED
         Resolution|                            |INVALID


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=26461

[Bug middle-end/26461] liveness of thread local references across function calls

[yichen dot xie at gmail dot com]

------- Comment #6 from yichen dot xie at gmail dot com  2006-02-25 01:55 -------
(In reply to comment #5)
> ISO C is not your normal low level language any more.  It actually tries to be
> a high level language.
> 
> So this is not a bug.
> 

I still don't think it's a good idea to treat thread local array addresses as
invariant. If you look at the implementation of getcontext/swapcontext, they
intentionally left gs segment register out in the context, leaving open the
possibility that a context saved by one thread be resumed by another. What will
gcc do in this case? 

If you don't mind, could you point me to the section of ISO C where it
specifies that function calls must preserve thread contexts? If not, by all
means it's a bug in the optimizer.

Does any one else have an opinion?


-- 

yichen dot xie at gmail dot com changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|RESOLVED                    |UNCONFIRMED
         Resolution|INVALID                     |


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=26461

[Bug middle-end/26461] liveness of thread local references across function calls

[gpderetta at gmail dot com]

------- Comment #7 from gpderetta at gmail dot com  2009-03-19 12:14 -------
Hi, I'm the author of Boost.Coroutine (not yet part of boost, but one day...).

I have the exact same problem: gcc caches the address of TLS variables across
function calls which breaks when coroutines move from one thread to another.

Note that in my case I'm definitely *not* reinventing threads in user space.
Coroutines are for different use cases than threads (i.e. when you do not need
preemption but simply a way to organize event driven code). One use of
boost.coroutine is on top of boost.asio.

Posix has both threads and swapcontext and nowhere it says that swapcontext 
can't be used in threaded applications. In fact is simply states that the saved
context is restored after a call to setcontext, and IMHO any posix compatible
compiler should support this.

FWIW The microsoft c++ compiler has the /GT (fiber safe TLS) flag to prevent
exactly this kind of optimizations. Probably GCC should support something like
that too.

See: 
http://www.crystalclearsoftware.com/soc/coroutine/coroutinecoroutine_thread.html

for details.

Finally I see the problem even with plain pointers and references, not only
arrays, at least with gcc4.3:

#include <ucontext.h>

void bar(int&);

__thread int x = 0;

void foo(ucontext_t&oucp, ucontext_t& ucp) {
    bar(x);
    swapcontext(&oucp, &ucp);
    bar(x);
}

Compiles down to this (with -O3, on x86_64):

:_Z3fooR8ucontextS0_:
        movq    %fs:0, %rax
        movq    %rbp, -16(%rsp)
        movq    %rbx, -24(%rsp)
        movq    %r12, -8(%rsp)
        movq    %rsi, %rbx
        subq    $24, %rsp
        movq    %rdi, %r12
        leaq    x@TPOFF(%rax), %rbp
        movq    %rbp, %rdi
        call    _Z3barRi
        movq    %r12, %rdi
        movq    %rbx, %rsi
        call    swapcontext
        movq    %rbp, %rdi
        movq    (%rsp), %rbx
        movq    8(%rsp), %rbp
        movq    16(%rsp), %r12
        addq    $24, %rsp
        jmp     _Z3barRi


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=26461

[Bug middle-end/26461] liveness of thread local references across function calls

[dwood at sybase dot com]

------- Comment #8 from dwood at sybase dot com  2010-08-23 21:13 -------
I believe this is a bug or a serious oversite in understanding the need for
support of >>>USER<<< thread local storage.  There are two kinds of software
threads; a) kernel threads(AKA LWP's on Solaris) scheduled by the OS; and b)
user threads scheduled by the user and/or threading library.  Databases such as
Informix and Sybase both manage their own user threads(1 per client
connection).  These run on a small pool of "engines" which are either kernel
threads or processes(No more than one per cpu core).  The user threads do
cooperative scheduling by calling their own yield implementation and they never
yield in a critical section.  These products and perhaps others are not little
weird cases.

These user threads can migrate between kernel threads as load balancing occurs.
 Of course, this requires that the implementation of the user level thread
context switch must save/restore USER TLS variables from/to the TLS areas of
the kernel threads involved.  If the model where M user threads are handled
across N kernel threads is valid then the address of "USER" TLS variables can
change across a context switch.

We, Sybase, have run into the same problem on Solaris-Sparc and are evaluating
whether gcc __thread on x86_64 will have the same problem.

Of course, the database folks often hear from the OS folks the common reply
like: Why are you doing this?  Just trust the OS and leave it all to us.  But
for us we have to trust AIX, Linux, Solaris, HP-UX, Windows, etc.  This is the
same whether we are talking about using USER threads or the dreaded Linux
O_DIRECT debate.


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=26461

[Bug middle-end/26461] liveness of thread local references across function calls

[pinskia at gcc dot gnu dot org]

------- Comment #9 from pinskia at gcc dot gnu dot org  2010-08-23 21:19 -------
I think you should read:
http://gcc.gnu.org/onlinedocs/gcc-4.5.0/gcc/C99-Thread_002dLocal-Edits.html#C99-Thread_002dLocal-Edits

--- CUT ---
In GCC's case the thread is only can be created via pthread.

Note C++0x defines a threading interface and such.  I know of no implementation
that will allow the use of "user threads" really.  Really I think it is wrong
to even think about using "user threads" any more.  The main reason why they
existed was to support OS's which don't have threads but those don't really
exist any more.  Not to mention, support for things like TLS is only only for
OS provided threads.


-- 

pinskia at gcc dot gnu dot org changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|UNCONFIRMED                 |RESOLVED
         Resolution|                            |WONTFIX


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=26461

[Bug middle-end/26461] liveness of thread local references across function calls

[dwood at sybase dot com]

------- Comment #10 from dwood at sybase dot com  2010-08-24 03:31 -------
(In reply to comment #9)
I don't disagree with the thread local writeup however it is lacking in
clarity.  A "flow of control" must be associated with an execution context. 
The existance of getcontext/setcontext allows both:

    1) One flow of control to switch to another flow of control within a single
       execution context;
    2) and, a flow of control to move from one execution context to another.

In fact, __thread variables are not actual bound to a "flow of control" but to
a specific execution context, part of which is usually some kind of thread_t
structure associated with a kernel thread.  If they where bound to a "logical
flow of control" then we wouldn't even need this discussion.  Nowhere above did
I refer to user thread.

The above should be consistant with your comment that TLS is supported for OS
threads only.  But the problem isn't the OS but the compiler.

    "When the address-of operator is applied to a thread-local variable, it is
    evaluated at run-time and returns the address of the current thread's
    instance of that variable."

The question is which OS thread is the current OS thread when the address is
obtained to actually fetch the current thread local value.  setcontext() avoids
changing %fs on Intel and %g7 on Sparc as it is really restoring a suspended
flow of control context onto a kernel thread's context as referenced by
%fs/%g7.  Caching these across function calls in a MT program is faultly.

Just as users of compiler's shouldn't depend on implementation defined behavior
neither should compiler writers assume OS implementation defined behavior.  OS
support for threads/setcontext existed prior to C's decision to provide basic
efficient access to TLS variables.  If some OS implementations of setcontext
allow the context to be "pushed" onto a different kernel thread then it was
gotten from then caching the current kernel thread handle or TLS address across
function calls is wrong.


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=26461