* Stack unwinding for green threads @ 2020-07-03 9:50 Botond Dénes 2020-07-03 10:16 ` Andrew Burgess 2020-07-06 17:48 ` Christian Biesinger 0 siblings, 2 replies; 5+ messages in thread From: Botond Dénes @ 2020-07-03 9:50 UTC (permalink / raw) To: gdb Hi, I'm working on Scylla [1], an application which is built using the seastar framework [2]. This framework provides green threads [3] that have their own stacks. These threads are created with `setcontext()` and later we switch in/out using `setjmp()`/`longjmp()`. We have a collection of python scripts [4] to help debug Scylla, among these we have a utility command which allows switching in/out of these green threads in gdb. This command basically (tries) to emulate `setjmp()`/`longjmp()` in python, saving and restoring registers. There are several problems with this method. For starters it crashes gdb for some time now, and also it doesn't work in coredumps, where gdb refuses to write to registers (even after `set write on`). So for these reasons I started to look for alternative ways to unwind the stacks of our green threads. However after going through the gdb documentation [5], reading all I could find about threads, stacks, frames and the Python API I haven't found an entry point to call which would unwind a stack located at a custom address (not the current $rsp). Is there such an API that I haven't found? Is there another way to achive what I want? Alternatively, how hard would it be to implement such an API? Thanks, Botond [1] https://github.com/scylladb/scylla [2] https://github.com/scylladb/seastar [3] http://docs.seastar.io/master/group__thread-module.html [4] https://github.com/scylladb/scylla/blob/master/scylla-gdb.py [5] https://sourceware.org/gdb/onlinedocs/gdb/index.html ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Stack unwinding for green threads 2020-07-03 9:50 Stack unwinding for green threads Botond Dénes @ 2020-07-03 10:16 ` Andrew Burgess 2020-07-03 17:36 ` Botond Dénes 2020-07-06 17:48 ` Christian Biesinger 1 sibling, 1 reply; 5+ messages in thread From: Andrew Burgess @ 2020-07-03 10:16 UTC (permalink / raw) To: Botond Dénes; +Cc: gdb * Botond Dénes <bdenes@scylladb.com> [2020-07-03 12:50:54 +0300]: > Hi, > > I'm working on Scylla [1], an application which is built using the > seastar framework [2]. This framework provides green threads [3] that > have their own stacks. These threads are created with `setcontext()` > and later we switch in/out using `setjmp()`/`longjmp()`. > > We have a collection of python scripts [4] to help debug Scylla, among > these we have a utility command which allows switching in/out of these > green threads in gdb. This command basically (tries) to emulate > `setjmp()`/`longjmp()` in python, saving and restoring registers. There > are several problems with this method. For starters it crashes gdb for > some time now, and also it doesn't work in coredumps, where gdb refuses > to write to registers (even after `set write on`). So for these reasons > I started to look for alternative ways to unwind the stacks of our > green threads. However after going through the gdb documentation [5], > reading all I could find about threads, stacks, frames and the Python > API I haven't found an entry point to call which would unwind a stack > located at a custom address (not the current $rsp). Is there such an > API that I haven't found? Is there another way to achive what I want? > Alternatively, how hard would it be to implement such an API? No, there's currently nothing in GDB for having a look at a different stack. There is 'frame view ...' [1] which I'm sure someone might mention, as it almost, sort-of, kind-of feels like this should solve your problem, but it wont. I did do some work on this area, years ago now, and at the time my plan was to add a set of commands that solved exactly this problem, my intention was to have some kind of 'stack create' command that would allow you to supply an entire register set (if you wanted) and then GDB would unwind assuming that this was the "current" register values. Of course, the only "required" registers would be $sp, and probably $pc too, but you would have been able to provide others if you wanted. Anyway I digress, my need for this work disappeared and this kind of fell of my radar, though I would like to find time to revisit this ... one day. However, thinking about the above kind of gave me an idea, it's not perfect, but it might work. How about writing a custom Python Unwinder[2]? Here's how I think this might help you: (1) By default the custom unwinder is "off", it doesn't claim any frames, your stack unwinds in the "normal" way. (2) Add a command 'magic-stack-unwind <address-of-register-set>', this turns "on" (sets a flag) the unwinder, and stores the address of the register set you want to unwind with. (3) The unwinder, now "on" would always claim frames at depth 0, and would return all of the registers from your stored register set. I think that this would present you with your alternative stack, but offset by 1 frame, so I think you'd still get the original "current" frame, but everything below that would be from your alternative register stack. Not ideal maybe, but it might work. It might even be possible to make use of frame filters[3] to hide frame 0, though again, I've never tried hiding frame 0, only even frames > 0, so I don't know how that would work for you. Good luck, Andrew [1] https://sourceware.org/gdb/current/onlinedocs/gdb/Selection.html#Selection [2] https://sourceware.org/gdb/current/onlinedocs/gdb/Unwinding-Frames-in-Python.html#Unwinding-Frames-in-Python [3] https://sourceware.org/gdb/current/onlinedocs/gdb/Frame-Filter-API.html#Frame-Filter-API > > Thanks, > Botond > > [1] https://github.com/scylladb/scylla > [2] https://github.com/scylladb/seastar > [3] http://docs.seastar.io/master/group__thread-module.html > [4] https://github.com/scylladb/scylla/blob/master/scylla-gdb.py > [5] https://sourceware.org/gdb/onlinedocs/gdb/index.html > ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Stack unwinding for green threads 2020-07-03 10:16 ` Andrew Burgess @ 2020-07-03 17:36 ` Botond Dénes 0 siblings, 0 replies; 5+ messages in thread From: Botond Dénes @ 2020-07-03 17:36 UTC (permalink / raw) To: Andrew Burgess; +Cc: gdb On Fri, 2020-07-03 at 11:16 +0100, Andrew Burgess wrote: > * Botond Dénes <bdenes@scylladb.com> [2020-07-03 12:50:54 +0300]: > > > Hi, > > > > I'm working on Scylla [1], an application which is built using the > > seastar framework [2]. This framework provides green threads [3] > > that > > have their own stacks. These threads are created with > > `setcontext()` > > and later we switch in/out using `setjmp()`/`longjmp()`. > > > > We have a collection of python scripts [4] to help debug Scylla, > > among > > these we have a utility command which allows switching in/out of > > these > > green threads in gdb. This command basically (tries) to emulate > > `setjmp()`/`longjmp()` in python, saving and restoring registers. > > There > > are several problems with this method. For starters it crashes gdb > > for > > some time now, and also it doesn't work in coredumps, where gdb > > refuses > > to write to registers (even after `set write on`). So for these > > reasons > > I started to look for alternative ways to unwind the stacks of our > > green threads. However after going through the gdb documentation > > [5], > > reading all I could find about threads, stacks, frames and the > > Python > > API I haven't found an entry point to call which would unwind a > > stack > > located at a custom address (not the current $rsp). Is there such > > an > > API that I haven't found? Is there another way to achive what I > > want? > > Alternatively, how hard would it be to implement such an API? > > No, there's currently nothing in GDB for having a look at a different > stack. > > There is 'frame view ...' [1] which I'm sure someone might mention, > as > it almost, sort-of, kind-of feels like this should solve your > problem, > but it wont. > > I did do some work on this area, years ago now, and at the time my > plan was to add a set of commands that solved exactly this problem, > my > intention was to have some kind of 'stack create' command that would > allow you to supply an entire register set (if you wanted) and then > GDB would unwind assuming that this was the "current" register > values. Of course, the only "required" registers would be $sp, and > probably $pc too, but you would have been able to provide others if > you wanted. Anyway I digress, my need for this work disappeared and > this kind of fell of my radar, though I would like to find time to > revisit this ... one day. > > However, thinking about the above kind of gave me an idea, it's not > perfect, but it might work. How about writing a custom Python > Unwinder[2]? > > Here's how I think this might help you: > > (1) By default the custom unwinder is "off", it doesn't claim any > frames, your stack unwinds in the "normal" way. > > (2) Add a command 'magic-stack-unwind <address-of-register-set>', > this turns "on" (sets a flag) the unwinder, and stores the address > of the register set you want to unwind with. > > (3) The unwinder, now "on" would always claim frames at depth 0, > and > would return all of the registers from your stored register set. > > I think that this would present you with your alternative stack, but > offset by 1 frame, so I think you'd still get the original "current" > frame, but everything below that would be from your alternative > register stack. Not ideal maybe, but it might work. I tried this, but gdb doesn't seem to like the fact that I start returing heap addresses (where customs stacks are located) as $rsp, I get: Backtrace stopped: previous frame inner to this frame (corrupt stack?) > > It might even be possible to make use of frame filters[3] to hide > frame 0, though again, I've never tried hiding frame 0, only even > frames > 0, so I don't know how that would work for you. > > Good luck, > Andrew > > > > [1] > https://sourceware.org/gdb/current/onlinedocs/gdb/Selection.html#Selection > [2] > https://sourceware.org/gdb/current/onlinedocs/gdb/Unwinding-Frames-in-Python.html#Unwinding-Frames-in-Python > [3] > https://sourceware.org/gdb/current/onlinedocs/gdb/Frame-Filter-API.html#Frame-Filter-API > > Thanks, > > Botond > > > > [1] https://github.com/scylladb/scylla > > [2] https://github.com/scylladb/seastar > > [3] http://docs.seastar.io/master/group__thread-module.html > > [4] https://github.com/scylladb/scylla/blob/master/scylla-gdb.py > > [5] https://sourceware.org/gdb/onlinedocs/gdb/index.html > > ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Stack unwinding for green threads 2020-07-03 9:50 Stack unwinding for green threads Botond Dénes 2020-07-03 10:16 ` Andrew Burgess @ 2020-07-06 17:48 ` Christian Biesinger 2020-07-07 6:18 ` Botond Dénes 1 sibling, 1 reply; 5+ messages in thread From: Christian Biesinger @ 2020-07-06 17:48 UTC (permalink / raw) To: Botond Dénes; +Cc: gdb On Fri, Jul 3, 2020 at 4:51 AM Botond Dénes <bdenes@scylladb.com> wrote: > > Hi, > > I'm working on Scylla [1], an application which is built using the > seastar framework [2]. This framework provides green threads [3] that > have their own stacks. These threads are created with `setcontext()` > and later we switch in/out using `setjmp()`/`longjmp()`. > > We have a collection of python scripts [4] to help debug Scylla, among > these we have a utility command which allows switching in/out of these > green threads in gdb. This command basically (tries) to emulate > `setjmp()`/`longjmp()` in python, saving and restoring registers. There > are several problems with this method. For starters it crashes gdb for > some time now It may be worth filing a bug for that at https://sourceware.org/bugzilla/, crashing GDB sounds like a bug. Christian ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Stack unwinding for green threads 2020-07-06 17:48 ` Christian Biesinger @ 2020-07-07 6:18 ` Botond Dénes 0 siblings, 0 replies; 5+ messages in thread From: Botond Dénes @ 2020-07-07 6:18 UTC (permalink / raw) To: Christian Biesinger; +Cc: gdb On Mon, 2020-07-06 at 12:48 -0500, Christian Biesinger wrote: > On Fri, Jul 3, 2020 at 4:51 AM Botond Dénes <bdenes@scylladb.com> > wrote: > > Hi, > > > > I'm working on Scylla [1], an application which is built using the > > seastar framework [2]. This framework provides green threads [3] > > that > > have their own stacks. These threads are created with > > `setcontext()` > > and later we switch in/out using `setjmp()`/`longjmp()`. > > > > We have a collection of python scripts [4] to help debug Scylla, > > among > > these we have a utility command which allows switching in/out of > > these > > green threads in gdb. This command basically (tries) to emulate > > `setjmp()`/`longjmp()` in python, saving and restoring registers. > > There > > are several problems with this method. For starters it crashes gdb > > for > > some time now > > It may be worth filing a bug for that at > https://sourceware.org/bugzilla/, crashing GDB sounds like a bug. We see a lot of crashes of GDB at ScyllaDB. Yes, I should get into the habit of reporting these. > > Christian ^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2020-07-07 6:18 UTC | newest] Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2020-07-03 9:50 Stack unwinding for green threads Botond Dénes 2020-07-03 10:16 ` Andrew Burgess 2020-07-03 17:36 ` Botond Dénes 2020-07-06 17:48 ` Christian Biesinger 2020-07-07 6:18 ` Botond Dénes
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).