Stack unwinding for green threads

Stack unwinding for green threads

[Botond Dénes]

Hi,

I'm working on Scylla [1], an application which is built using the
seastar framework [2]. This framework provides green threads [3] that
have their own stacks. These threads are created with `setcontext()`
and later we switch in/out using `setjmp()`/`longjmp()`.

We have a collection of python scripts [4] to help debug Scylla, among
these we have a utility command which allows switching in/out of these
green threads in gdb. This command basically (tries) to emulate
`setjmp()`/`longjmp()` in python, saving and restoring registers. There
are several problems with this method. For starters it crashes gdb for
some time now, and also it doesn't work in coredumps, where gdb refuses
to write to registers (even after `set write on`). So for these reasons
I started to look for alternative ways to unwind the stacks of our
green threads. However after going through the gdb documentation [5],
reading all I could find about threads, stacks, frames and the Python
API I haven't found an entry point to call which would unwind a stack
located at a custom address (not the current $rsp). Is there such an
API that I haven't found? Is there another way to achive what I want?
Alternatively, how hard would it be to implement such an API?

Thanks,
Botond

[1] https://github.com/scylladb/scylla
[2] https://github.com/scylladb/seastar
[3] http://docs.seastar.io/master/group__thread-module.html
[4] https://github.com/scylladb/scylla/blob/master/scylla-gdb.py
[5] https://sourceware.org/gdb/onlinedocs/gdb/index.html

Re: Stack unwinding for green threads

[Andrew Burgess]

* Botond Dénes <bdenes@scylladb.com> [2020-07-03 12:50:54 +0300]:

> Hi,
> 
> I'm working on Scylla [1], an application which is built using the
> seastar framework [2]. This framework provides green threads [3] that
> have their own stacks. These threads are created with `setcontext()`
> and later we switch in/out using `setjmp()`/`longjmp()`.
> 
> We have a collection of python scripts [4] to help debug Scylla, among
> these we have a utility command which allows switching in/out of these
> green threads in gdb. This command basically (tries) to emulate
> `setjmp()`/`longjmp()` in python, saving and restoring registers. There
> are several problems with this method. For starters it crashes gdb for
> some time now, and also it doesn't work in coredumps, where gdb refuses
> to write to registers (even after `set write on`). So for these reasons
> I started to look for alternative ways to unwind the stacks of our
> green threads. However after going through the gdb documentation [5],
> reading all I could find about threads, stacks, frames and the Python
> API I haven't found an entry point to call which would unwind a stack
> located at a custom address (not the current $rsp). Is there such an
> API that I haven't found? Is there another way to achive what I want?
> Alternatively, how hard would it be to implement such an API?

No, there's currently nothing in GDB for having a look at a different
stack.

There is 'frame view ...' [1] which I'm sure someone might mention, as
it almost, sort-of, kind-of feels like this should solve your problem,
but it wont.

I did do some work on this area, years ago now, and at the time my
plan was to add a set of commands that solved exactly this problem, my
intention was to have some kind of 'stack create' command that would
allow you to supply an entire register set (if you wanted) and then
GDB would unwind assuming that this was the "current" register
values.  Of course, the only "required" registers would be $sp, and
probably $pc too, but you would have been able to provide others if
you wanted.  Anyway I digress, my need for this work disappeared and
this kind of fell of my radar, though I would like to find time to
revisit this ... one day.

However, thinking about the above kind of gave me an idea, it's not
perfect, but it might work.  How about writing a custom Python
Unwinder[2]?

Here's how I think this might help you:

  (1) By default the custom unwinder is "off", it doesn't claim any
  frames, your stack unwinds in the "normal" way.

  (2) Add a command 'magic-stack-unwind <address-of-register-set>',
  this turns "on" (sets a flag) the unwinder, and stores the address
  of the register set you want to unwind with.

  (3) The unwinder, now "on" would always claim frames at depth 0, and
  would return all of the registers from your stored register set.

I think that this would present you with your alternative stack, but
offset by 1 frame, so I think you'd still get the original "current"
frame, but everything below that would be from your alternative
register stack.  Not ideal maybe, but it might work.

It might even be possible to make use of frame filters[3] to hide
frame 0, though again, I've never tried hiding frame 0, only even
frames > 0, so I don't know how that would work for you.

Good luck,
Andrew



[1] https://sourceware.org/gdb/current/onlinedocs/gdb/Selection.html#Selection
[2] https://sourceware.org/gdb/current/onlinedocs/gdb/Unwinding-Frames-in-Python.html#Unwinding-Frames-in-Python
[3] https://sourceware.org/gdb/current/onlinedocs/gdb/Frame-Filter-API.html#Frame-Filter-API
> 
> Thanks,
> Botond
> 
> [1] https://github.com/scylladb/scylla
> [2] https://github.com/scylladb/seastar
> [3] http://docs.seastar.io/master/group__thread-module.html
> [4] https://github.com/scylladb/scylla/blob/master/scylla-gdb.py
> [5] https://sourceware.org/gdb/onlinedocs/gdb/index.html
> 

Re: Stack unwinding for green threads

[Botond Dénes]

On Fri, 2020-07-03 at 11:16 +0100, Andrew Burgess wrote:
> * Botond Dénes <bdenes@scylladb.com> [2020-07-03 12:50:54 +0300]:
> 
> > Hi,
> > 
> > I'm working on Scylla [1], an application which is built using the
> > seastar framework [2]. This framework provides green threads [3]
> > that
> > have their own stacks. These threads are created with
> > `setcontext()`
> > and later we switch in/out using `setjmp()`/`longjmp()`.
> > 
> > We have a collection of python scripts [4] to help debug Scylla,
> > among
> > these we have a utility command which allows switching in/out of
> > these
> > green threads in gdb. This command basically (tries) to emulate
> > `setjmp()`/`longjmp()` in python, saving and restoring registers.
> > There
> > are several problems with this method. For starters it crashes gdb
> > for
> > some time now, and also it doesn't work in coredumps, where gdb
> > refuses
> > to write to registers (even after `set write on`). So for these
> > reasons
> > I started to look for alternative ways to unwind the stacks of our
> > green threads. However after going through the gdb documentation
> > [5],
> > reading all I could find about threads, stacks, frames and the
> > Python
> > API I haven't found an entry point to call which would unwind a
> > stack
> > located at a custom address (not the current $rsp). Is there such
> > an
> > API that I haven't found? Is there another way to achive what I
> > want?
> > Alternatively, how hard would it be to implement such an API?
> 
> No, there's currently nothing in GDB for having a look at a different
> stack.
> 
> There is 'frame view ...' [1] which I'm sure someone might mention,
> as
> it almost, sort-of, kind-of feels like this should solve your
> problem,
> but it wont.
> 
> I did do some work on this area, years ago now, and at the time my
> plan was to add a set of commands that solved exactly this problem,
> my
> intention was to have some kind of 'stack create' command that would
> allow you to supply an entire register set (if you wanted) and then
> GDB would unwind assuming that this was the "current" register
> values.  Of course, the only "required" registers would be $sp, and
> probably $pc too, but you would have been able to provide others if
> you wanted.  Anyway I digress, my need for this work disappeared and
> this kind of fell of my radar, though I would like to find time to
> revisit this ... one day.
> 
> However, thinking about the above kind of gave me an idea, it's not
> perfect, but it might work.  How about writing a custom Python
> Unwinder[2]?
> 
> Here's how I think this might help you:
> 
>   (1) By default the custom unwinder is "off", it doesn't claim any
>   frames, your stack unwinds in the "normal" way.
> 
>   (2) Add a command 'magic-stack-unwind <address-of-register-set>',
>   this turns "on" (sets a flag) the unwinder, and stores the address
>   of the register set you want to unwind with.
> 
>   (3) The unwinder, now "on" would always claim frames at depth 0,
> and
>   would return all of the registers from your stored register set.
> 
> I think that this would present you with your alternative stack, but
> offset by 1 frame, so I think you'd still get the original "current"
> frame, but everything below that would be from your alternative
> register stack.  Not ideal maybe, but it might work.


I tried this, but gdb doesn't seem to like the fact that I start
returing heap addresses (where customs stacks are located) as $rsp, I
get:

Backtrace stopped: previous frame inner to this frame (corrupt stack?)

> 
> It might even be possible to make use of frame filters[3] to hide
> frame 0, though again, I've never tried hiding frame 0, only even
> frames > 0, so I don't know how that would work for you.
> 
> Good luck,
> Andrew
> 
> 
> 
> [1] 
> https://sourceware.org/gdb/current/onlinedocs/gdb/Selection.html#Selection
> [2] 
> https://sourceware.org/gdb/current/onlinedocs/gdb/Unwinding-Frames-in-Python.html#Unwinding-Frames-in-Python
> [3] 
> https://sourceware.org/gdb/current/onlinedocs/gdb/Frame-Filter-API.html#Frame-Filter-API
> > Thanks,
> > Botond
> > 
> > [1] https://github.com/scylladb/scylla
> > [2] https://github.com/scylladb/seastar
> > [3] http://docs.seastar.io/master/group__thread-module.html
> > [4] https://github.com/scylladb/scylla/blob/master/scylla-gdb.py
> > [5] https://sourceware.org/gdb/onlinedocs/gdb/index.html
> > 

Re: Stack unwinding for green threads

[Christian Biesinger]

On Fri, Jul 3, 2020 at 4:51 AM Botond Dénes <bdenes@scylladb.com> wrote:
>
> Hi,
>
> I'm working on Scylla [1], an application which is built using the
> seastar framework [2]. This framework provides green threads [3] that
> have their own stacks. These threads are created with `setcontext()`
> and later we switch in/out using `setjmp()`/`longjmp()`.
>
> We have a collection of python scripts [4] to help debug Scylla, among
> these we have a utility command which allows switching in/out of these
> green threads in gdb. This command basically (tries) to emulate
> `setjmp()`/`longjmp()` in python, saving and restoring registers. There
> are several problems with this method. For starters it crashes gdb for
> some time now

It may be worth filing a bug for that at
https://sourceware.org/bugzilla/, crashing GDB sounds like a bug.

Christian

Re: Stack unwinding for green threads

[Botond Dénes]

On Mon, 2020-07-06 at 12:48 -0500, Christian Biesinger wrote:
> On Fri, Jul 3, 2020 at 4:51 AM Botond Dénes <bdenes@scylladb.com>
> wrote:
> > Hi,
> > 
> > I'm working on Scylla [1], an application which is built using the
> > seastar framework [2]. This framework provides green threads [3]
> > that
> > have their own stacks. These threads are created with
> > `setcontext()`
> > and later we switch in/out using `setjmp()`/`longjmp()`.
> > 
> > We have a collection of python scripts [4] to help debug Scylla,
> > among
> > these we have a utility command which allows switching in/out of
> > these
> > green threads in gdb. This command basically (tries) to emulate
> > `setjmp()`/`longjmp()` in python, saving and restoring registers.
> > There
> > are several problems with this method. For starters it crashes gdb
> > for
> > some time now
> 
> It may be worth filing a bug for that at
> https://sourceware.org/bugzilla/, crashing GDB sounds like a bug.


We see a lot of crashes of GDB at ScyllaDB. Yes, I should get into the
habit of reporting these.

> 
> Christian