* meeting 2007-08-15 9:30 us east coast time @ 2007-08-15 1:32 Andrew Cagney 2007-08-15 13:20 ` Andrew Cagney 2007-08-15 15:56 ` meeting 2007-08-15 9:30 us east coast time Andrew Cagney 0 siblings, 2 replies; 6+ messages in thread From: Andrew Cagney @ 2007-08-15 1:32 UTC (permalink / raw) To: frysk [all welcome, contact me on irc for dial in info] This week: -> sam would like to go some use cases involving inlined code and stack frames ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: meeting 2007-08-15 9:30 us east coast time 2007-08-15 1:32 meeting 2007-08-15 9:30 us east coast time Andrew Cagney @ 2007-08-15 13:20 ` Andrew Cagney 2007-08-15 17:52 ` Andrew Cagney 2007-08-21 9:21 ` Roundtable, breakpoints and lots of unwinding (Was: meeting 2007-08-15 9:30 us east coast time) Mark Wielaard 2007-08-15 15:56 ` meeting 2007-08-15 9:30 us east coast time Andrew Cagney 1 sibling, 2 replies; 6+ messages in thread From: Andrew Cagney @ 2007-08-15 13:20 UTC (permalink / raw) To: frysk Since for the last two weeks the round-table fell off the end of the meeting, here's what Red Hat is doing: scox: printing types esp c++ mjw: bug fixes for stepping; support for .debug_frame in libunwind for instance, lesson the need for asynchronous-unwind-tables in .eh_frame by using .debug_frame when available tim: thread commands in hpd for instance: (fhpd) [0,1] print x [0] 10 [1] 20 i.e., prints x in two processes pmuldoon: core file cleanup; low-level h.w watch-point investigation watch-points, like breakpoints, get implemented at two levels: low is the h/w register manipulation and a correspinding abstraction; high is the translation of an expression into something to watch rmoseley: source window crashers (e.g., hover in f7); frysk.proc.dead.Exe* (executable target) this fills in a functional hole, and extends the collection of supported targets, for instance: (fhpd) load /bin/bash (fhpd) disassemble main npremji: hpd option spit 'n' polish a re-implementation of the Command class was added, but unfortunately the old code wasn't upgraded to the new mechanisms mcvet: reporting exit and termination in hpd, ditto for break-points for instance: (fhpd) run ... program exits ... Process 0 exiting with status 1 (fhpd) cmoller: prototype utrace bindings for frysk getting ready for another target frysk.proc.live.Utrace*; would only be enabled with --utrace option. teresa: support for dw_op_piece in location-expressions (a.k.a., frysk.value.Piece) for instance, a value split between registers and memory sami: inline stack backtraces see todays discussion ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: meeting 2007-08-15 9:30 us east coast time 2007-08-15 13:20 ` Andrew Cagney @ 2007-08-15 17:52 ` Andrew Cagney 2007-08-21 9:21 ` Roundtable, breakpoints and lots of unwinding (Was: meeting 2007-08-15 9:30 us east coast time) Mark Wielaard 1 sibling, 0 replies; 6+ messages in thread From: Andrew Cagney @ 2007-08-15 17:52 UTC (permalink / raw) To: frysk also ... peta: has been using frysk's framework to implement a library tracing tool that works multi-threaded and multi-process me: I've been adding a formatting mechanism (used by the register window, and print) ^ permalink raw reply [flat|nested] 6+ messages in thread
* Roundtable, breakpoints and lots of unwinding (Was: meeting 2007-08-15 9:30 us east coast time) 2007-08-15 13:20 ` Andrew Cagney 2007-08-15 17:52 ` Andrew Cagney @ 2007-08-21 9:21 ` Mark Wielaard 2007-08-22 21:43 ` Phil Muldoon 1 sibling, 1 reply; 6+ messages in thread From: Mark Wielaard @ 2007-08-21 9:21 UTC (permalink / raw) To: Andrew Cagney; +Cc: frysk Hi Andrew, That was a nice summary and overview of the things people are working on. Thanks. Maybe we can add a wiki somewhere to keep these overviews up to date. Here some additions and details on my current work. > mjw: bug fixes for stepping; Low level stepping of breakpoints in particular. This all started with the demo of TestUpdatingDisplayValue which in the end produced a reproducer for http://sourceware.org/bugzilla/show_bug.cgi?id=4747 Which was just one way of how breakpoint stepping while a signal becomes pending could break. Something which is especially nasty when doing out of line stepping. There is now a low-level instruction stepping test framework: http://sourceware.org/bugzilla/show_bug.cgi?id=4763 http://sourceware.org/ml/frysk/2007-q3/msg00118.html And there is now a low-level test framework for testing signals being raised while breakpoint stepping (except user installed signal handlers for now though): http://sourceware.org/ml/frysk/2007-q3/msg00301.html The work by Petr on fltrace exposed some extra issues, who also helped adding some extra tests, it did take some more time then I had originally though, but I believe with the above the code is a lot more robust and better supports what Petr is doing. In particular the following bugs have now been closed: http://sourceware.org/bugzilla/show_bug.cgi?id=4889 http://sourceware.org/bugzilla/show_bug.cgi?id=4894 There are still 3 issues I know of with low-level stepping of breakpoints (none of which I am currently actively working on, but I expect fltrace to hit them, so when that work is progressing we might want to go back to these issues): http://sourceware.org/bugzilla/show_bug.cgi?id=4847 Stepping Trap instructions (in particular inside a trap signal handler) is broken (again) on newer kernels. Getting stepping of a trap instruction and stepping of the trap handler needs to be fully special cased. And the kernel doesn't help because it changes behavior every other release it seems. Not working on this till someone finds a real use case for this one. http://sourceware.org/bugzilla/show_bug.cgi?id=4762 We don't have a real Instruction parser (x86/x86_64) for single stepping out of line framework (see IA32InstructionParser) which means that for almost all instructions we are actually using reset breakpoint stepping which misses breakpoints when used with multiple threads. Not currently working on this one either, but I suspect that the new fltrace work that Petr is doing will soon hit this limitation at which time we should either finish the instruction parser or introduce stop-the-world stepping to make things more robust. http://sourceware.org/bugzilla/show_bug.cgi?id=4895 Low level breakpoints are visible to all Tasks, not just the Task that requested it. This seemed to be a good idea back when they were added since low-level breakpoints are essentially Proc based, but this confuses some users because they have to check the Task argument in their updateHit() handler, the workaround is easy, so not very important, but it would be better if this was cleaned up. > support for .debug_frame in libunwind > for instance, lesson the need for asynchronous-unwind-tables in > .eh_frame by using .debug_frame when available The breakpoints took some more time than I had anticipated, so I am still working on this bit. Since there is a lot of layers to unwinding and documentation is scattered all over the place here is a summary of unwinding as I found it. Both to document my own research and to structure the work a bit. If any of the below is wrong, please let me know. Unwinding the call stack used to be something only a debugger would do and relied on the executable having a frame pointer in a dedicated register that points to the bottom of the stack frame for the current function which also contained the return address [1]. Having a frame pointer allows you to quickly walk the call stack and get all the addresses, if you can map those to the names of the relevant functions they are in you have a nice backtrace for the user. If you want to get more of the state then you could rely on each function having a prologue and epilogue that saved and restored the registers [2] of the caller. Given a calling convention for a particular architecture you could use these to reliably find the original registers on the stack, which in turn with some debug info would give you the values of variables and arguments of the functions on the call stack. Unfortunately compilers got smart and optimized code might not keep a frame pointer (frees up one more register) and might reschedule the function prologue and epilogue instructions between the other instructions in the function. All making it pretty hard for an unwinder to reconstruct the previous call frames on the stack. In particular x86_64 does away with a standard frame pointer. You can still get some information back by conservatively approximating the instructions in the function and guessing at the actual way the various registers are stored [3] but this becomes pretty messy pretty quickly. To help debuggers still get all the information needed to unwind a stack and restore all needed registers the debugging information (DWARF) generated by compilers was extended to include Call Frame Information (CFI) [4] that allows a debugger to reconstruct the the calling pc and registers of a function. This information is stored in the .debug_frame section of an elf file. It uses a simplified version of the dwarf instructions (not all operands are relevant for reconstructing the registers). This section is not guaranteed to be available, it is not necessarily loaded into memory and can even be split off into its own debug info file in some distributions. At the same time different languages got constructs (exceptions, continuations, global gotos, asynchronous garbage collectors, etc) which required some sort of reliable unwinding (and in some cases rewinding) of the call stack. Since some optimizations and some newer architectures also did away with a standard frame pointer another way to reliably unwind the stack was needed. This became the exception handler framework (eh_frame) which is based on the DWARF CFI work but which is slightly different. Unfortunately nobody seems to have documented the precise differences between the formats. So you will have to carefully read both the DWARF standard and the LSB core specification Exception Frames [5] side-by-side. Note that a debugger that wants to walk a stack and recover all registers might need more information than some of these language constructs which might only need unwind information for specific call sites. Depending on optimizations, architecture and language compiled (and sometimes specific distribution default choices) no, full or partial exception handler unwind information and/or frame pointers are generated (see the GCC options -funwind-tables and -fasynchronous-unwind-tables [6]). Both the dwarf and the exception handler specs are architecture neutral. But since you do need to a mapping between the actual registers and the specs you also need to consult the relevant architecture abi that defines the actual mapping. Sometimes these architecture abi specs also define some DWARF/EH extensions. See for example the x86_64 abi spec [7]. Note that in practise what gcc generates overrides any of the above specs, and if a discrepancy is found the spec usually gets updated [8]. And that one should be careful about bugs in the old DWARF2 spec [9] and extension of DWARF specified by the LSB [10] (which mostly augment DWARF2 to be like DWARF3, at least for the exception handler sections). If an .eh_frame section is available in an elf file it is guaranteed to be loaded in memory. But depending on architecture and language being compiled might not be available at all (and neither might the frame pointer or the .debug_frame section). So with that background the work having to be done consists of the following: - libunwind officially only supports the .eh_frame format so it will have to be extended to also support the .debug_frame format. Luckily the differences, although very poorly documented, don't seem to be that large. - libunwind has its own CFI EH/DWARF parser but doesn't come with an interface to feed/read the CFI information directly. This is the Gget_unwind_table.c (unw_get_unwind_table) support that Nurdin added, but which isn't upstream yet. Pushing this upstream would be very beneficial since then we could use pure upstream in frysk, but see the next point, maybe the proposed interface should be changed a little. - Currently we hook into this new unw_get_unwind_table through UnwindH.hxx (createProcInfoFromElfImage), this is called indirectly through the libunwind find_proc_info callback which wants to see the unwind_info filled in. The ElfImage used is created in UnwindAddressSpace.findProcInfo() through the private method getElfImage(long address) by getting the MemoryMap of the address from the Task and either mapping the map from the elf file into memory or if the section is the VDSO by creating an anonymous mmap and filling that through reading the address map and then passing it to the libunwind dwarf reader. Directly mmapping these sections seems wrong here since the sections should already be available through the memory buffers of the proc we are inspecting (which might already have mapped in those sections). So it would be better to use the libunwind addressspace accessors that go through the ByteBuffers also for this. This might mean another change in the libunwind interface so all remote memory accesses go through the same hooks (although unw_get_unwind_table already provides an unw_address_space as argument, so I might be missing something). - For the .debug_info we cannot rely on it being available in the target address space (unlike .eh_frame which always gets loaded) and it might not be in the elf file directly, but might be in a separate debuginfo file. So there we need to locate the section first through libdwfl, load it (also through libdwfl?) and feed that to libunwind. Cheers, Mark [1] http://en.wikipedia.org/wiki/Frame_pointer [2] http://en.wikipedia.org/wiki/Function_prologue [3] http://sourceware.org/gdb/current/onlinedocs/gdbint_3.html#SEC9 [4] http://wiki.dwarfstd.org/ (Dwarf 3 - section 6.4) [5] http://refspecs.freestandards.org/LSB_3.1.0/LSB-Core-generic/LSB-Core-generic/ehframechpt.html [6] http://gcc.gnu.org/onlinedocs/gcc/Code-Gen-Options.html [7] http://www.x86-64.org/documentation/abi.pdf (Section 3.6 and 3.7) [8] http://gcc.gnu.org/bugzilla/show_bug.cgi?id=32982 [9] http://wiki.dwarfstd.org/index.php?title=DWARF_FAQ#How_big_is_a_DW_FORM_ref_addr.3F [10] http://refspecs.freestandards.org/LSB_3.1.0/LSB-Core-generic/LSB-Core-generic/dwarfext.html ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Roundtable, breakpoints and lots of unwinding (Was: meeting 2007-08-15 9:30 us east coast time) 2007-08-21 9:21 ` Roundtable, breakpoints and lots of unwinding (Was: meeting 2007-08-15 9:30 us east coast time) Mark Wielaard @ 2007-08-22 21:43 ` Phil Muldoon 0 siblings, 0 replies; 6+ messages in thread From: Phil Muldoon @ 2007-08-22 21:43 UTC (permalink / raw) Cc: Mark Wielaard, frysk Mark Wielaard wrote: >> mjw: bug fixes for stepping; >> > > - Currently we hook into this new unw_get_unwind_table through > UnwindH.hxx (createProcInfoFromElfImage), this is called indirectly > through the libunwind find_proc_info callback which wants to see the > unwind_info filled in. The ElfImage used is created in > UnwindAddressSpace.findProcInfo() through the private method > getElfImage(long address) by getting the MemoryMap of the address from > the Task and either mapping the map from the elf file into memory or if > the section is the VDSO by creating an anonymous mmap and filling that > through reading the address map and then passing it to the libunwind > dwarf reader. Directly mmapping these sections seems wrong here since > the sections should already be available through the memory buffers of > the proc we are inspecting (which might already have mapped in those > sections). So it would be better to use the libunwind addressspace > accessors that go through the ByteBuffers also for this. This might mean > another change in the libunwind interface so all remote memory accesses > go through the same hooks (although unw_get_unwind_table already > provides an unw_address_space as argument, so I might be missing > something). > > This strikes me as especially worry-some in core file unwinding. Why would you want to mmap at all anything? All access to any memroy/maps in a core file case should be done via .getMemory(). It should be left to the CorefileByteBuffer to arbitrate whether it can find the memory address in the core file and return that, or go to the relevant solib and get it for you. It is the only thing that has the sufficient meta-data, and thus smarts to make that decision. Maybe I misunderstood, but other than the initial "beginning" of the unwind, the unwind code might be mmap'ing the library in directly? Nice write up, by the way. This is ideal blogging material. Regards Phil ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: meeting 2007-08-15 9:30 us east coast time 2007-08-15 1:32 meeting 2007-08-15 9:30 us east coast time Andrew Cagney 2007-08-15 13:20 ` Andrew Cagney @ 2007-08-15 15:56 ` Andrew Cagney 1 sibling, 0 replies; 6+ messages in thread From: Andrew Cagney @ 2007-08-15 15:56 UTC (permalink / raw) To: frysk Three topis: sami discussing inlined functions; phil/nurdin and corefiles and arguments; what should <<frysk PID>> do? Sami: would like to go some use cases involving inlined code and stack frames Here's the code being discussed: sami #include <stdlib.h> sami inline void third(int arg3){ sami int var3 = arg3; sami int* a = 0; sami a[0] = var3; sami } sami inline void second(int arg2){ sami int var2 = arg2; sami third(var2+1); sami } sami inline void first(int arg1){ sami int var1 = arg1; sami second(var1+1); sami } sami int main(){ sami int some_int = 1; sami first(some_int); sami return 0; sami } and here's the output from the current fstack -a: sami #0 0x0000000000400466 in third(int arg3 = < ERROR >) /to/scratch/swagiaal/frysks/frysk.testBuild/frysk-core/frysk/pkglibdir/funit-inlined.c#6 sami int var3 = < value unavailable at pc=0x400466> line#< error > sami * a = < value unavailable at pc=0x400466> line#< error > sami #1 0x0000000000400466 in second(int arg2 = < optimized out >) /to/scratch/swagiaal/frysks/frysk.testBuild/frysk-core/frysk/pkglibdir/funit-inlined.c#6 sami int var2 = < value unavailable at pc=0x400466> line#< error > sami int var3 = < value unavailable at pc=0x400466> line#< error > sami * a = < value unavailable at pc=0x400466> line#< error > sami #2 0x0000000000400466 in first(int arg1 = < optimized out >) /to/scratch/swagiaal/frysks/frysk.testBuild/frysk-core/frysk/pkglibdir/funit-inlined.c#6 sami int var1 = < value unavailable at pc=0x400466> line#< error > sami int var2 = < value unavailable at pc=0x400466> line#< error > sami int var3 = < value unavailable at pc=0x400466> line#< error > sami * a = < value unavailable at pc=0x400466> line#< error > sami #3 0x0000000000400466 in main () from /tmp/frysks/frysk.testBuild/frysk-core/frysk/pkglibdir/funit>> -- Discussion 1: What output should each "rich" or "inline" frame contain? From inline frame #1 (second): int var2 = < value unavailable at pc=0x400466> line#< error > int var3 = < value unavailable at pc=0x400466> line#< error > * a = < value unavailable at pc=0x400466> line#< error > and notice how it lists variables from both second(var2) and third(var3,a). Should inline frames display just their local variables (reflecting the source) or both their and nested inline scopes (reflecting the DIE information). Long discussion which identified that both dynamic and static information can be displayed: dynamic: given an inline instance, display that instances variables only static: given a function containing inline instances, display the full inline information including scopes created by inlining and the default should be the dynamic info (i.e., inline function display just their local variables) but there be a way to display the static information with additional mark-ups to identify nested inlined functions and their scopes. Discussion #2: What should <<(fhpd) where>> display, should it follow fstack (Side bar, fstack's default is to display just a raw stack using ELF information only so that it is simple and fast; there isn't a good reason for mimicking it). Discussion concluded (fhpd)where should: + include inline functions (performance isn't an issue, full information is) + include function signatures (e.g., String foo(int,boolean)) (lets cut/paste of that for setting breakpoints and easy identification of parameters, but avoids problems of parameter values being wrong) + include file/line (I guess?) - no local variables - no parameters Discussion 3: Commands / options for accessing stack information. Discussion concluded that "where" should broadly follow fstack's option convention; for instance <<fstack -a>> and <<where -a>>. In addition, "info" be extended to include commands to dump out details, for instance <<info locals -- print local variables>> <<info scope -- print static scope information>> Discussion 4: How should frames be identified. Discussion identified that using simple numbering (#0 #1 #2) for inline frames would lead to frames being identified by different numbers dependant on the presence of inline debug info, and would not help identify inline vs true frames. Its proposed that a Dewey Decimal notation be adopted for instance, using the above: #0.1 third #0.2 second #0.3 first #0 main where there is no nested frame 0 (too confusing over it being either main or third) ---------------------------------- Topic #2 (phil/nurdin) core files needing executables Background: there are times when, to read a corefule, the executable that the core originated from needs to also be supplied. This is due to the core not containing the full path to the executable or that executable path being wrong (e.g., out-of-date). Nurdin explained how the generic command line parser (used by all commands) will recognize <<core>> <<exe>> and "just does the right thing". ------------------------------------ Topic #3: what should <<frysk PID>> do It was concluded that the answer depends largely on what tool you worked on last; it could bring up either the monitor or the source window. Ignoring the "vote" solution, several options were suggested: - jump to the "session assistant" and, when the process-picker window is displayed have it pre-populated with the list of pids - add --debug | --monitor options ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2007-08-22 21:43 UTC | newest] Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2007-08-15 1:32 meeting 2007-08-15 9:30 us east coast time Andrew Cagney 2007-08-15 13:20 ` Andrew Cagney 2007-08-15 17:52 ` Andrew Cagney 2007-08-21 9:21 ` Roundtable, breakpoints and lots of unwinding (Was: meeting 2007-08-15 9:30 us east coast time) Mark Wielaard 2007-08-22 21:43 ` Phil Muldoon 2007-08-15 15:56 ` meeting 2007-08-15 9:30 us east coast time Andrew Cagney
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).