* Best approach for supporting snapshots for QEMU's gdbstub? @ 2021-05-14 16:06 Alex Bennée 2021-05-15 6:12 ` Pavel Dovgalyuk 2021-05-17 16:49 ` Luis Machado 0 siblings, 2 replies; 7+ messages in thread From: Alex Bennée @ 2021-05-14 16:06 UTC (permalink / raw) To: gdb, QEMU Developers, Pavel Dovgalyuk, Luis Machado Cc: Daniel P. Berrangé Hi, I've been playing around with QEMU's reverse debugging support which I have working with Pavel's latest patches for supporting virtio with record/replay. Once you get the right command line it works well enough although currently each step backwards requires replaying the entire execution history until you get to the right point. QEMU can quite easily snapshot the entire VM state so I was looking to see what the best way to integrate this would be. As far as I can tell there are two interfaces gdb supports: bookmarks and checkpoints. As far as I can tell bookmarks where added as part of GDB's reverse debugging support but attempting to use them from the gdbstub reports: (gdb) bookmark You can't do that when your target is `remote' so I guess that would need an extension to the stub protocol to support? The other option I found was checkpoints which seem to predate support for reverse debugging. However: (gdb) checkpoint checkpoint: can't find fork function in inferior. I couldn't tell what feature needs to be negotiated but I suspect it's something like fork-events if the checkpoint mechanism is designed for user space with a fork/freeze approach. We could of course just add a custom monitor command like the qemu.sstep= command which could be used manually. However that would be a QEMU gdbstub specific approach. The other thing would be to be more intelligent on QEMU's side and save snapshots each time we hit an event, for example each time we hit a given breakpoint. However I do worry that might lead to snapshots growing quite quickly. Any thoughts/suggestions? -- Alex Bennée ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Best approach for supporting snapshots for QEMU's gdbstub? 2021-05-14 16:06 Best approach for supporting snapshots for QEMU's gdbstub? Alex Bennée @ 2021-05-15 6:12 ` Pavel Dovgalyuk 2021-05-17 16:49 ` Luis Machado 1 sibling, 0 replies; 7+ messages in thread From: Pavel Dovgalyuk @ 2021-05-15 6:12 UTC (permalink / raw) To: Alex Bennée, gdb, QEMU Developers, Luis Machado Cc: Daniel P. Berrangé On 14.05.2021 19:06, Alex Bennée wrote: > Hi, > > I've been playing around with QEMU's reverse debugging support which > I have working with Pavel's latest patches for supporting virtio with > record/replay. Once you get the right command line it works well enough > although currently each step backwards requires replaying the entire > execution history until you get to the right point. > > QEMU can quite easily snapshot the entire VM state so I was looking to > see what the best way to integrate this would be. As far as I can tell > there are two interfaces gdb supports: bookmarks and checkpoints. > > As far as I can tell bookmarks where added as part of GDB's reverse > debugging support but attempting to use them from the gdbstub reports: > > (gdb) bookmark > You can't do that when your target is `remote' > > so I guess that would need an extension to the stub protocol to support? > > The other option I found was checkpoints which seem to predate support > for reverse debugging. However: > > (gdb) checkpoint > checkpoint: can't find fork function in inferior. > > I couldn't tell what feature needs to be negotiated but I suspect it's > something like fork-events if the checkpoint mechanism is designed for > user space with a fork/freeze approach. > > We could of course just add a custom monitor command like the > qemu.sstep= command which could be used manually. However that would be > a QEMU gdbstub specific approach. For now you can just use 'monitor savevm sn1' in gdb. But something like 'bookmark' seems more convenient. > The other thing would be to be more intelligent on QEMU's side and save > snapshots each time we hit an event, for example each time we hit a > given breakpoint. However I do worry that might lead to snapshots > growing quite quickly. > > Any thoughts/suggestions? > ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Best approach for supporting snapshots for QEMU's gdbstub? 2021-05-14 16:06 Best approach for supporting snapshots for QEMU's gdbstub? Alex Bennée 2021-05-15 6:12 ` Pavel Dovgalyuk @ 2021-05-17 16:49 ` Luis Machado 2021-05-17 17:27 ` Alex Bennée 1 sibling, 1 reply; 7+ messages in thread From: Luis Machado @ 2021-05-17 16:49 UTC (permalink / raw) To: Alex Bennée, gdb, QEMU Developers, Pavel Dovgalyuk Cc: Daniel P. Berrangé Hi, On 5/14/21 1:06 PM, Alex Bennée wrote: > Hi, > > I've been playing around with QEMU's reverse debugging support which > I have working with Pavel's latest patches for supporting virtio with > record/replay. Once you get the right command line it works well enough > although currently each step backwards requires replaying the entire > execution history until you get to the right point. > > QEMU can quite easily snapshot the entire VM state so I was looking to > see what the best way to integrate this would be. As far as I can tell > there are two interfaces gdb supports: bookmarks and checkpoints. > > As far as I can tell bookmarks where added as part of GDB's reverse > debugging support but attempting to use them from the gdbstub reports: > > (gdb) bookmark > You can't do that when your target is `remote' > > so I guess that would need an extension to the stub protocol to support? > Right. We don't support reverse step/next/continue for remote targets. I think this would be the most appropriate way to implement this feature in GDB. But it is not trivial. > The other option I found was checkpoints which seem to predate support > for reverse debugging. However: > > (gdb) checkpoint > checkpoint: can't find fork function in inferior. > > I couldn't tell what feature needs to be negotiated but I suspect it's > something like fork-events if the checkpoint mechanism is designed for > user space with a fork/freeze approach. Checkpoints are an old mechanism for saving a snapshot from a process, but they don't support threaded inferiors and they need to be able to call fork. > > We could of course just add a custom monitor command like the > qemu.sstep= command which could be used manually. However that would be > a QEMU gdbstub specific approach. That would be an easy and quick way to allow GDB to control things in QEMU, but I wouldn't say it is the best. Monitor commands are basically a bypass of the RSP where GDB sends/receives commands to/from the remote target. > > The other thing would be to be more intelligent on QEMU's side and save > snapshots each time we hit an event, for example each time we hit a > given breakpoint. However I do worry that might lead to snapshots > growing quite quickly. GDB would need to be aware of such snapshots for them to be useful. Otherwise GDB wouldn't be able to use them to restore state. > > Any thoughts/suggestions? > ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Best approach for supporting snapshots for QEMU's gdbstub? 2021-05-17 16:49 ` Luis Machado @ 2021-05-17 17:27 ` Alex Bennée 2021-05-17 17:54 ` Luis Machado ` (2 more replies) 0 siblings, 3 replies; 7+ messages in thread From: Alex Bennée @ 2021-05-17 17:27 UTC (permalink / raw) To: Luis Machado Cc: gdb, QEMU Developers, Pavel Dovgalyuk, Daniel P. Berrangé Luis Machado <luis.machado@linaro.org> writes: > Hi, > > On 5/14/21 1:06 PM, Alex Bennée wrote: >> Hi, >> I've been playing around with QEMU's reverse debugging support which >> I have working with Pavel's latest patches for supporting virtio with >> record/replay. Once you get the right command line it works well enough >> although currently each step backwards requires replaying the entire >> execution history until you get to the right point. >> QEMU can quite easily snapshot the entire VM state so I was looking >> to >> see what the best way to integrate this would be. As far as I can tell >> there are two interfaces gdb supports: bookmarks and checkpoints. >> As far as I can tell bookmarks where added as part of GDB's reverse >> debugging support but attempting to use them from the gdbstub reports: >> (gdb) bookmark >> You can't do that when your target is `remote' >> so I guess that would need an extension to the stub protocol to >> support? >> > > Right. We don't support reverse step/next/continue for remote targets. > I think this would be the most appropriate way to implement this > feature in GDB. But it is not trivial. You do because ";ReverseStep+;ReverseContinue+" is part of the gdbstub negotiation handshake. Out of interest how is rr implemented? It presents a gdb interface so I thought it was some implemented using some remote magic. <snip> >> We could of course just add a custom monitor command like the >> qemu.sstep= command which could be used manually. However that would be >> a QEMU gdbstub specific approach. > > That would be an easy and quick way to allow GDB to control things in > QEMU, but I wouldn't say it is the best. Monitor commands are > basically a bypass of the RSP where GDB sends/receives commands > to/from the remote target. We have some underlying commands we can set via the monitor including: monitor info replay monitor replay_seek <N> monitor replay_break <N> > >> The other thing would be to be more intelligent on QEMU's side and >> save >> snapshots each time we hit an event, for example each time we hit a >> given breakpoint. However I do worry that might lead to snapshots >> growing quite quickly. > > GDB would need to be aware of such snapshots for them to be useful. > Otherwise GDB wouldn't be able to use them to restore state. What does GDB need to know about them? Does it include something like the icount at a particular point. I'm curious at how a break and reverse-continue is meant to work if that breakpoint is hit multiple times from the start of a run. You need to know if the last time you hit a particular breakpoint was in fact the last time before where the user was when they hit reverse-continue. > >> Any thoughts/suggestions? >> -- Alex Bennée ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Best approach for supporting snapshots for QEMU's gdbstub? 2021-05-17 17:27 ` Alex Bennée @ 2021-05-17 17:54 ` Luis Machado 2021-05-17 19:01 ` Peter Maydell 2021-05-18 5:26 ` Pavel Dovgalyuk 2 siblings, 0 replies; 7+ messages in thread From: Luis Machado @ 2021-05-17 17:54 UTC (permalink / raw) To: Alex Bennée Cc: gdb, QEMU Developers, Pavel Dovgalyuk, Daniel P. Berrangé On 5/17/21 2:27 PM, Alex Bennée wrote: > > Luis Machado <luis.machado@linaro.org> writes: > >> Hi, >> >> On 5/14/21 1:06 PM, Alex Bennée wrote: >>> Hi, >>> I've been playing around with QEMU's reverse debugging support which >>> I have working with Pavel's latest patches for supporting virtio with >>> record/replay. Once you get the right command line it works well enough >>> although currently each step backwards requires replaying the entire >>> execution history until you get to the right point. >>> QEMU can quite easily snapshot the entire VM state so I was looking >>> to >>> see what the best way to integrate this would be. As far as I can tell >>> there are two interfaces gdb supports: bookmarks and checkpoints. >>> As far as I can tell bookmarks where added as part of GDB's reverse >>> debugging support but attempting to use them from the gdbstub reports: >>> (gdb) bookmark >>> You can't do that when your target is `remote' >>> so I guess that would need an extension to the stub protocol to >>> support? >>> >> >> Right. We don't support reverse step/next/continue for remote targets. >> I think this would be the most appropriate way to implement this >> feature in GDB. But it is not trivial. > > You do because ";ReverseStep+;ReverseContinue+" is part of the gdbstub > negotiation handshake. Interesting... I was looking at the vCont; packets for inferior movement. The regular c/C/s/S packet are deprecated and vCont; equivalents should be used instead. It seems the reverse continue (bc) and reverse step (bs) packets can be used, but they are not vCont packets. That's confusing. I suppose nobody took the time to implement bc/bs equivalents for vCont. > > Out of interest how is rr implemented? It presents a gdb interface so I > thought it was some implemented using some remote magic. I don't know. I have never used rr. > > <snip> > >>> We could of course just add a custom monitor command like the >>> qemu.sstep= command which could be used manually. However that would be >>> a QEMU gdbstub specific approach. >> >> That would be an easy and quick way to allow GDB to control things in >> QEMU, but I wouldn't say it is the best. Monitor commands are >> basically a bypass of the RSP where GDB sends/receives commands >> to/from the remote target. > > We have some underlying commands we can set via the monitor including: > > monitor info replay > monitor replay_seek <N> > monitor replay_break <N> > >> >>> The other thing would be to be more intelligent on QEMU's side and >>> save >>> snapshots each time we hit an event, for example each time we hit a >>> given breakpoint. However I do worry that might lead to snapshots >>> growing quite quickly. >> >> GDB would need to be aware of such snapshots for them to be useful. >> Otherwise GDB wouldn't be able to use them to restore state. > > What does GDB need to know about them? Does it include something like > the icount at a particular point. GDB needs to know they exist so the user can choose to go back to such snapshots. I haven't dealt with remote reverse execution implementations, but if this information can be exposed to > > I'm curious at how a break and reverse-continue is meant to work if that > breakpoint is hit multiple times from the start of a run. You need to > know if the last time you hit a particular breakpoint was in fact the > last time before where the user was when they hit reverse-continue. When you have record/replay on, there is no real "continue". GDB will instruction-step everything and will record register values and memory changes. When you reverse instruction-step, GDB will restore the state for the previous snapshot. When you reverse continue, GDB will do the same and will move the state backwards snapshot by snapshot. It is not very efficient. So, in that sense, GDB will hit all of the breakpoints again. It doesn't keep track of how many times the breakpoint was hit. It only keeps track of how many instructions were recorded and what register/memory changes happened. If you hit an instruction that GDB doesn't know how to calculate register/memory changes for, it will stop dead on its tracks. In that sense, it is also not very easy to maintain and takes a lot of instruction-parsing to work correctly. No wonder there are more performatic solutions out there. :-) > >> >>> Any thoughts/suggestions? >>> > > ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Best approach for supporting snapshots for QEMU's gdbstub? 2021-05-17 17:27 ` Alex Bennée 2021-05-17 17:54 ` Luis Machado @ 2021-05-17 19:01 ` Peter Maydell 2021-05-18 5:26 ` Pavel Dovgalyuk 2 siblings, 0 replies; 7+ messages in thread From: Peter Maydell @ 2021-05-17 19:01 UTC (permalink / raw) To: Alex Bennée Cc: Luis Machado, Daniel P. Berrangé, gdb, Pavel Dovgalyuk, QEMU Developers On Mon, 17 May 2021 at 18:37, Alex Bennée <alex.bennee@linaro.org> wrote: > Luis Machado <luis.machado@linaro.org> writes: > > Right. We don't support reverse step/next/continue for remote targets. > > I think this would be the most appropriate way to implement this > > feature in GDB. But it is not trivial. > > You do because ";ReverseStep+;ReverseContinue+" is part of the gdbstub > negotiation handshake. > > Out of interest how is rr implemented? It presents a gdb interface so I > thought it was some implemented using some remote magic. AIUI rr just implements the reverse-step/reverse-continue parts of the gdb remote protocol. It makes them fast by internally to its implementation saying "ah, you wanted to do a reverse-step, I can do that by starting from the best available checkpoint and going forwards" and by automatically creating checkpoints at points that it thinks will be useful. gdb and the remote protocol know nothing about these checkpoints -- they are purely created and managed under the hood by rr as an optimisation so that reverse-step is decently fast. (Given that it's the rr end that knows best about what checkpoints are available, how expensive it is to create a checkpoint, etc, that seems not unreasonable.) There are also a handful of rr-specific gdb commands kind of like the QEMU-specific ones, which the user can use to say things like "go directly to this point in time T" which the gdb UI doesn't have a concept of. (Also because rr starts the gdb for you it gets a chance to feed it a few gdb macro definitions which I think mostly just make the debugging experience a bit smoother rather than being critical parts of the gdb-to-stub communication.) thanks -- PMM ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Best approach for supporting snapshots for QEMU's gdbstub? 2021-05-17 17:27 ` Alex Bennée 2021-05-17 17:54 ` Luis Machado 2021-05-17 19:01 ` Peter Maydell @ 2021-05-18 5:26 ` Pavel Dovgalyuk 2 siblings, 0 replies; 7+ messages in thread From: Pavel Dovgalyuk @ 2021-05-18 5:26 UTC (permalink / raw) To: Alex Bennée, Luis Machado Cc: gdb, QEMU Developers, Daniel P. Berrangé On 17.05.2021 20:27, Alex Bennée wrote: > > Luis Machado <luis.machado@linaro.org> writes: > >> Hi, >> >> On 5/14/21 1:06 PM, Alex Bennée wrote: >>> Hi, >>> I've been playing around with QEMU's reverse debugging support which >>> I have working with Pavel's latest patches for supporting virtio with >>> record/replay. Once you get the right command line it works well enough >>> although currently each step backwards requires replaying the entire >>> execution history until you get to the right point. >>> QEMU can quite easily snapshot the entire VM state so I was looking >>> to >>> see what the best way to integrate this would be. As far as I can tell >>> there are two interfaces gdb supports: bookmarks and checkpoints. >>> As far as I can tell bookmarks where added as part of GDB's reverse >>> debugging support but attempting to use them from the gdbstub reports: >>> (gdb) bookmark >>> You can't do that when your target is `remote' >>> so I guess that would need an extension to the stub protocol to >>> support? >>> >> >> Right. We don't support reverse step/next/continue for remote targets. >> I think this would be the most appropriate way to implement this >> feature in GDB. But it is not trivial. > > You do because ";ReverseStep+;ReverseContinue+" is part of the gdbstub > negotiation handshake. > > Out of interest how is rr implemented? It presents a gdb interface so I > thought it was some implemented using some remote magic. > > <snip> > >>> We could of course just add a custom monitor command like the >>> qemu.sstep= command which could be used manually. However that would be >>> a QEMU gdbstub specific approach. >> >> That would be an easy and quick way to allow GDB to control things in >> QEMU, but I wouldn't say it is the best. Monitor commands are >> basically a bypass of the RSP where GDB sends/receives commands >> to/from the remote target. > > We have some underlying commands we can set via the monitor including: > > monitor info replay > monitor replay_seek <N> > monitor replay_break <N> > >> >>> The other thing would be to be more intelligent on QEMU's side and >>> save >>> snapshots each time we hit an event, for example each time we hit a >>> given breakpoint. However I do worry that might lead to snapshots >>> growing quite quickly. >> >> GDB would need to be aware of such snapshots for them to be useful. >> Otherwise GDB wouldn't be able to use them to restore state. > > What does GDB need to know about them? Does it include something like > the icount at a particular point. > > I'm curious at how a break and reverse-continue is meant to work if that > breakpoint is hit multiple times from the start of a run. You need to > know if the last time you hit a particular breakpoint was in fact the > last time before where the user was when they hit reverse-continue. QEMU does this seamlessly. For reverse continue it replays the execution, noticing the last breakpoint that was hit. Then it replays again, but to the icount of that hit. Pavel Dovgalyuk ^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2021-05-18 5:26 UTC | newest] Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2021-05-14 16:06 Best approach for supporting snapshots for QEMU's gdbstub? Alex Bennée 2021-05-15 6:12 ` Pavel Dovgalyuk 2021-05-17 16:49 ` Luis Machado 2021-05-17 17:27 ` Alex Bennée 2021-05-17 17:54 ` Luis Machado 2021-05-17 19:01 ` Peter Maydell 2021-05-18 5:26 ` Pavel Dovgalyuk
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).