public inbox for cygwin@cygwin.com
 help / color / mirror / Atom feed
* Strange gdb backtraces on 64-bit Cygwin
@ 2014-09-17 22:00 Ken Brown
  2014-09-18  9:23 ` Markus Hoenicka
  2014-09-23 12:47 ` Ken Brown
  0 siblings, 2 replies; 3+ messages in thread
From: Ken Brown @ 2014-09-17 22:00 UTC (permalink / raw)
  To: cygwin; +Cc: Eli Zaretskii, Markus Hoenicka

[-- Attachment #1: Type: text/plain, Size: 1905 bytes --]

There have been many bug reports involving crashes or assertion failures 
in emacs-X11 or emacs-w32 on 64-bit Cygwin.  Many of these reports 
include gdb backtraces that don't make sense.  The one I'm looking at 
right now is emacs bug#17753.  I'll try to make this email 
self-contained, but anyone interested in the context should start at

   http://debbugs.gnu.org/cgi/bugreport.cgi?bug=17753#47

The situation in that bug report is that a gdb backtrace showed a call 
to the emacs function "run_timers" in Thread 2, which shouldn't happen. 
  (It should only be called in the main thread.)  There was speculation 
as to whether this could be the cause of the crash.  An alternative 
theory is that the gdb backtrace is bogus.  I'm pretty sure that the OP 
(Markus) was running gdb-7.6.50-4.  His report was about emacs-X11, but 
I've observed similar backtraces for emacs-w32.

To try to sort this out, I've done the following experiment: I've run 
emacs-w32 under gdb with a breakpoint at run_timers.  I've done this on 
both 32-bit Cygwin and 64-bit Cygwin [*], using both gdb-7.6.50-4 and 
gdb-7.8.  [For the latter I used my own build, since the bugfix we 
discussed in a different thread hasn't yet made it into the Cygwin 
distro.]  Transcripts of the four gdb sessions are attached; the file 
names indicate the gdb version and the platform.

My reading of these transcripts is that gdb-7.6.50-4 on 64-bit Cygwin is 
the outlier, and the strange occurrence of run_timers in Thread 2 is 
therefore likely to be a result of a gdb bug.  But it would be great if 
someone familiar with recent gdb development could shed some light on 
this.  In particular, is it plausible that there was a bug of this 
nature in gdb-7.6 that affected only the 64-bit platform and that has 
since been fixed?

Ken

[*] I was using the 20140905 Cygwin snapshots, but the results were 
similar with cygwin-1.7.32-1.

[-- Attachment #2: gdb-7.6-x86 --]
[-- Type: text/plain, Size: 7736 bytes --]

$ gdb /usr/bin/emacs-w32.exe
GNU gdb (GDB) 7.6.50.20130728-cvs (cygwin-special)
[...]
Reading symbols from /usr/bin/emacs-w32.exe...Reading symbols from /usr/lib/debug/usr/bin/emacs-w32.exe.dbg...done.
done.
(gdb) b run_timers
Breakpoint 1 at 0x5e625e: file /usr/src/debug/emacs-24.3.93-4/src/atimer.c, line 343.
(gdb) r -Q
Starting program: /usr/bin/emacs-w32.exe -Q
[New Thread 3820.0x1594]
[New Thread 3820.0x1540]
[New Thread 3820.0x1a54]
[New Thread 3820.0x1bcc]
[New Thread 3820.0x1bdc]

Breakpoint 1, run_timers () at /usr/src/debug/emacs-24.3.93-4/src/atimer.c:343
343       struct timespec now = current_timespec ();
(gdb) thread apply all bt

Thread 5 (Thread 3820.0x1bdc):
#0  0x756d78d7 in USER32!DispatchMessageW () from /c/windows/syswow64/USER32.dll
#1  0x005ffd9c in w32_msg_pump (msg_buf=0x3b3acd8)
    at /usr/src/debug/emacs-24.3.93-4/src/w32fns.c:2450
#2  0x005fffdf in w32_msg_worker@4 (arg=0x0)
    at /usr/src/debug/emacs-24.3.93-4/src/w32fns.c:2676
#3  0x61005f04 in _cygtls::call2(unsigned long (*)(void*, void*), void*, void*)@16 (this=<optimized out>, func=func@entry=0x5fff3f <w32_msg_worker@4>,
    arg=0x3b3ac70, arg@entry=0x0, buf=buf@entry=0x3b3cdc4)
    at /home/cygnus/vinschen/mknetrel/src/cygwin-snapshot-20140905-1/winsup/cygwin/cygtls.cc:100
#4  0x61006076 in _cygtls::call (func=0x5fff3f <w32_msg_worker@4>, arg=0x0)
    at /home/cygnus/vinschen/mknetrel/src/cygwin-snapshot-20140905-1/winsup/cygwin/cygtls.cc:30
#5  0x6107f1f8 in threadfunc_fe (arg=<optimized out>)
    at /home/cygnus/vinschen/mknetrel/src/cygwin-snapshot-20140905-1/winsup/cygwin/init.cc:30
#6  0x753e338a in KERNEL32!BaseThreadInitThunk ()
   from /c/windows/syswow64/kernel32.dll
#7  0x77759f72 in ntdll!RtlInitializeExceptionChain ()
   from /c/windows/system32/ntdll.dll
#8  0x77759f45 in ntdll!RtlInitializeExceptionChain ()
   from /c/windows/system32/ntdll.dll
#9  0x00000000 in ?? ()

Thread 4 (Thread 3820.0x1bcc):
#0  0x7773f8d1 in ntdll!ZwWaitForSingleObject ()
   from /c/windows/system32/ntdll.dll
#1  0x7773f8d1 in ntdll!ZwWaitForSingleObject ()
   from /c/windows/system32/ntdll.dll
#2  0x758c14ab in WaitForSingleObjectEx ()
   from /c/windows/syswow64/KERNELBASE.dll
#3  0x00000318 in ?? ()
#4  0x00000000 in ?? ()

Thread 3 (Thread 3820.0x1a54):
#0  0x7774015d in ntdll!ZwWaitForMultipleObjects ()
   from /c/windows/system32/ntdll.dll
#1  0x7774015d in ntdll!ZwWaitForMultipleObjects ()
   from /c/windows/system32/ntdll.dll
#2  0x77772f91 in ntdll!RtlMoveMemory () from /c/windows/system32/ntdll.dll
#3  0x00000001 in ?? ()
#4  0x00000001 in ?? ()
#5  0x00000000 in ?? ()

Thread 2 (Thread 3820.0x1540):
#0  0x7773f905 in ntdll!ZwReadFile () from /c/windows/system32/ntdll.dll
#1  0x7773f905 in ntdll!ZwReadFile () from /c/windows/system32/ntdll.dll
#2  0x758bdd62 in ReadFile () from /c/windows/syswow64/KERNELBASE.dll
#3  0x00000090 in ?? ()
---Type <return> to continue, or q <return> to quit---
#4  0x753e3ee7 in ReadFile () from /c/windows/syswow64/kernel32.dll
#5  0x00000090 in ?? ()
#6  0x610e1b8c in wait_sig ()
    at /home/cygnus/vinschen/mknetrel/src/cygwin-snapshot-20140905-1/winsup/cygwin/sigproc.cc:1239
#7  0x61004cc7 in cygthread::callfunc(bool)@8 (
    this=this@entry=0x6119b840 <threads>, issimplestub=issimplestub@entry=false)
    at /home/cygnus/vinschen/mknetrel/src/cygwin-snapshot-20140905-1/winsup/cygwin/cygthread.cc:51
#8  0x6100523a in cygthread::stub(void*)@4 (arg=0x6119b840 <threads>)
    at /home/cygnus/vinschen/mknetrel/src/cygwin-snapshot-20140905-1/winsup/cygwin/cygthread.cc:93
#9  0x61005f04 in _cygtls::call2(unsigned long (*)(void*, void*), void*, void*)@16 (this=<optimized out>, func=func@entry=0x610051f0 <cygthread::stub(void*)@4>,
    arg=arg@entry=0x6119b840 <threads>, buf=buf@entry=0x2b6cdc4)
    at /home/cygnus/vinschen/mknetrel/src/cygwin-snapshot-20140905-1/winsup/cygwin/cygtls.cc:100
#10 0x61006076 in _cygtls::call (func=0x610051f0 <cygthread::stub(void*)@4>,
    arg=0x6119b840 <threads>)
    at /home/cygnus/vinschen/mknetrel/src/cygwin-snapshot-20140905-1/winsup/cygwin/cygtls.cc:30
#11 0x6107f1f8 in threadfunc_fe (arg=<optimized out>)
    at /home/cygnus/vinschen/mknetrel/src/cygwin-snapshot-20140905-1/winsup/cygwin/init.cc:30
#12 0x753e338a in KERNEL32!BaseThreadInitThunk ()
   from /c/windows/syswow64/kernel32.dll
#13 0x77759f72 in ntdll!RtlInitializeExceptionChain ()
   from /c/windows/system32/ntdll.dll
#14 0x77759f45 in ntdll!RtlInitializeExceptionChain ()
   from /c/windows/system32/ntdll.dll
#15 0x00000000 in ?? ()

Thread 1 (Thread 3820.0x1594):
#0  run_timers () at /usr/src/debug/emacs-24.3.93-4/src/atimer.c:343
#1  0x005e634c in do_pending_atimers ()
    at /usr/src/debug/emacs-24.3.93-4/src/atimer.c:385
#2  0x0050d946 in process_pending_signals ()
    at /usr/src/debug/emacs-24.3.93-4/src/keyboard.c:7105
#3  0x0056c13e in Fmake_list (length=0, init=9555994)
    at /usr/src/debug/emacs-24.3.93-4/src/alloc.c:2644
#4  0x00594cba in concat (nargs=1, args=0x28a030, target_type=Lisp_Cons,
    last_special=false) at /usr/src/debug/emacs-24.3.93-4/src/fns.c:588
#5  0x005947cf in Fcopy_sequence (arg=11843918)
    at /usr/src/debug/emacs-24.3.93-4/src/fns.c:456
#6  0x00508e74 in timer_check ()
    at /usr/src/debug/emacs-24.3.93-4/src/keyboard.c:4568
#7  0x00506f54 in readable_events (flags=1)
    at /usr/src/debug/emacs-24.3.93-4/src/keyboard.c:3447
#8  0x0050d473 in get_input_pending (flags=1)
    at /usr/src/debug/emacs-24.3.93-4/src/keyboard.c:6765
#9  0x00513b17 in detect_input_pending_run_timers (do_display=true)
---Type <return> to continue, or q <return> to quit---
    at /usr/src/debug/emacs-24.3.93-4/src/keyboard.c:9894
#10 0x005dafca in wait_reading_process_output (time_limit=0, nsecs=0,
    read_kbd=-1, do_display=true, wait_for_cell=9555994, wait_proc=0x0,
    just_wait_proc=0) at /usr/src/debug/emacs-24.3.93-4/src/process.c:4699
#11 0x00507c76 in kbd_buffer_get_event (kbp=0x28a2e4, used_mouse_menu=0x28a7e3,
    end_time=0x0) at /usr/src/debug/emacs-24.3.93-4/src/keyboard.c:3906
#12 0x00503c64 in read_event_from_main_queue (end_time=0x0,
    local_getcjmp=0x28a5ac, used_mouse_menu=0x28a7e3)
    at /usr/src/debug/emacs-24.3.93-4/src/keyboard.c:2246
#13 0x00503ec0 in read_decoded_event_from_main_queue (end_time=0x0,
    local_getcjmp=0x28a5ac, prev_event=9555994, used_mouse_menu=0x28a7e3)
    at /usr/src/debug/emacs-24.3.93-4/src/keyboard.c:2309
#14 0x005056af in read_char (commandflag=1, map=11275222, prev_event=9555994,
    used_mouse_menu=0x28a7e3, end_time=0x0)
    at /usr/src/debug/emacs-24.3.93-4/src/keyboard.c:2895
#15 0x00512218 in read_key_sequence (keybuf=0x28a900, bufsize=30,
    prompt=9555994, dont_downcase_last=false, can_return_switch_frame=true,
    fix_current_buffer=true, prevent_redisplay=false)
    at /usr/src/debug/emacs-24.3.93-4/src/keyboard.c:9088
#16 0x00502184 in command_loop_1 ()
    at /usr/src/debug/emacs-24.3.93-4/src/keyboard.c:1452
#17 0x0058d2bd in internal_condition_case (bfun=0x501e06 <command_loop_1>,
    handlers=9606530, hfun=0x501633 <cmd_error>)
    at /usr/src/debug/emacs-24.3.93-4/src/eval.c:1354
#18 0x00501ab0 in command_loop_2 (ignore=9555994)
    at /usr/src/debug/emacs-24.3.93-4/src/keyboard.c:1177
#19 0x0058c82a in internal_catch (tag=9608722, func=0x501a8c <command_loop_2>,
    arg=9555994) at /usr/src/debug/emacs-24.3.93-4/src/eval.c:1118
#20 0x00501a66 in command_loop ()
    at /usr/src/debug/emacs-24.3.93-4/src/keyboard.c:1156
#21 0x005011c7 in recursive_edit_1 ()
    at /usr/src/debug/emacs-24.3.93-4/src/keyboard.c:777
#22 0x00501387 in Frecursive_edit ()
    at /usr/src/debug/emacs-24.3.93-4/src/keyboard.c:848
#23 0x004ff777 in main (argc=2, argv=0x28ac3c)
    at /usr/src/debug/emacs-24.3.93-4/src/emacs.c:1647

[-- Attachment #3: gdb-7.6-x86_64 --]
[-- Type: text/plain, Size: 5148 bytes --]

$ gdb /usr/bin/emacs-w32.exe
GNU gdb (GDB) 7.6.50.20130728-cvs (cygwin-special)
[...]
Reading symbols from /usr/bin/emacs-w32.exe...Reading symbols from /usr/lib/debug/usr/bin/emacs-w32.exe.dbg...done.
done.
(gdb) b run_timers
Breakpoint 1 at 0x10064785c: file /usr/src/debug/emacs-24.3.93-4/src/atimer.c, line 342.
(gdb) r -Q
Starting program: /usr/bin/emacs-w32.exe -Q
[New Thread 5864.0x6b4]
[New Thread 5864.0x754]
[New Thread 5864.0x1ac8]
[New Thread 5864.0x1928]
[New Thread 5864.0x63c]

Breakpoint 1, run_timers () at /usr/src/debug/emacs-24.3.93-4/src/atimer.c:342
342     {
(gdb) thread apply all bt

Thread 5 (Thread 5864.0x63c):
#0  0x0000000077339e6a in USER32!SfmDxSetSwapChainStats ()
   from /c/windows/system32/USER32.dll
#1  0x0000000077339e9e in USER32!GetMessageW ()
   from /c/windows/system32/USER32.dll
#2  0x0000000000000000 in ?? ()

Thread 4 (Thread 5864.0x1928):
#0  0x00000000775912fa in ntdll!ZwWaitForSingleObject ()
   from /c/windows/system32/ntdll.dll
#1  0x000007fefd4e10dc in WaitForSingleObjectEx ()
   from /c/windows/system32/KERNELBASE.dll
#2  0x0000000000000000 in ?? ()

Thread 3 (Thread 5864.0x1ac8):
#0  0x000000007759186a in ntdll!ZwWaitForMultipleObjects ()
   from /c/windows/system32/ntdll.dll
#1  0x000000007755b037 in ntdll!TpIsTimerSet ()
   from /c/windows/system32/ntdll.dll
#2  0x0000000000000000 in ?? ()

Thread 2 (Thread 5864.0x754):
#0  0x000000007759131a in ntdll!ZwReadFile () from /c/windows/system32/ntdll.dll
#1  0x000007fefd4e1a7a in ReadFile () from /c/windows/system32/KERNELBASE.dll
#2  0x000000000042ce00 in ?? ()
#3  0x0000000100647983 in run_timers ()
    at /usr/src/debug/emacs-24.3.93-4/src/atimer.c:364
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 1 (Thread 5864.0x6b4):
#0  run_timers () at /usr/src/debug/emacs-24.3.93-4/src/atimer.c:342
#1  0x00000001006479b4 in do_pending_atimers ()
    at /usr/src/debug/emacs-24.3.93-4/src/atimer.c:385
#2  0x000000010053fb87 in process_pending_signals ()
    at /usr/src/debug/emacs-24.3.93-4/src/keyboard.c:7105
#3  0x00000001005b3c53 in Fmake_list (length=0, init=4306501682)
    at /usr/src/debug/emacs-24.3.93-4/src/alloc.c:2644
#4  0x00000001005e4e85 in concat (nargs=1, args=0x429650,
    target_type=Lisp_Cons, last_special=false)
    at /usr/src/debug/emacs-24.3.93-4/src/fns.c:588
#5  0x00000001005e4868 in Fcopy_sequence (arg=25770933814)
    at /usr/src/debug/emacs-24.3.93-4/src/fns.c:456
#6  0x000000010053a111 in timer_check ()
    at /usr/src/debug/emacs-24.3.93-4/src/keyboard.c:4568
#7  0x000000010053789f in readable_events (flags=1)
    at /usr/src/debug/emacs-24.3.93-4/src/keyboard.c:3447
#8  0x000000010053f5dd in get_input_pending (flags=1)
    at /usr/src/debug/emacs-24.3.93-4/src/keyboard.c:6765
#9  0x000000010054707e in detect_input_pending_run_timers (do_display=true)
    at /usr/src/debug/emacs-24.3.93-4/src/keyboard.c:9894
#10 0x000000010063a087 in wait_reading_process_output (time_limit=0, nsecs=0,
    read_kbd=-1, do_display=true, wait_for_cell=4306501682, wait_proc=0x0,
    just_wait_proc=0) at /usr/src/debug/emacs-24.3.93-4/src/process.c:4699
#11 0x00000001005388e4 in kbd_buffer_get_event (kbp=0x429b88,
    used_mouse_menu=0x42a3cf, end_time=0x0)
    at /usr/src/debug/emacs-24.3.93-4/src/keyboard.c:3906
#12 0x00000001005337c3 in read_event_from_main_queue (end_time=0x0,
    local_getcjmp=0x429fb0, used_mouse_menu=0x42a3cf)
    at /usr/src/debug/emacs-24.3.93-4/src/keyboard.c:2246
#13 0x0000000100533b00 in read_decoded_event_from_main_queue (end_time=0x0,
    local_getcjmp=0x429fb0, prev_event=4306501682, used_mouse_menu=0x42a3cf)
    at /usr/src/debug/emacs-24.3.93-4/src/keyboard.c:2309
#14 0x0000000100535ab4 in read_char (commandflag=1, map=25770257174,
    prev_event=4306501682, used_mouse_menu=0x42a3cf, end_time=0x0)
    at /usr/src/debug/emacs-24.3.93-4/src/keyboard.c:2895
#15 0x000000010054507f in read_key_sequence (keybuf=0x42a5e0, bufsize=30,
    prompt=4306501682, dont_downcase_last=false, can_return_switch_frame=true,
    fix_current_buffer=true, prevent_redisplay=false)
    at /usr/src/debug/emacs-24.3.93-4/src/keyboard.c:9088
#16 0x00000001005314d4 in command_loop_1 ()
    at /usr/src/debug/emacs-24.3.93-4/src/keyboard.c:1452
#17 0x00000001005db83c in internal_condition_case (
    bfun=0x100530fbf <command_loop_1>, handlers=4306576130,
    hfun=0x10053054a <cmd_error>)
    at /usr/src/debug/emacs-24.3.93-4/src/eval.c:1354
#18 0x0000000100530b8d in command_loop_2 (ignore=4306501682)
    at /usr/src/debug/emacs-24.3.93-4/src/keyboard.c:1177
#19 0x00000001005dab8d in internal_catch (tag=4306570290,
    func=0x100530b5b <command_loop_2>, arg=4306501682)
    at /usr/src/debug/emacs-24.3.93-4/src/eval.c:1118
#20 0x0000000100530b1c in command_loop ()
    at /usr/src/debug/emacs-24.3.93-4/src/keyboard.c:1156
#21 0x000000010052ffab in recursive_edit_1 ()
    at /usr/src/debug/emacs-24.3.93-4/src/keyboard.c:777
#22 0x00000001005301dd in Frecursive_edit ()
    at /usr/src/debug/emacs-24.3.93-4/src/keyboard.c:848
#23 0x000000010052debe in main (argc=2, argv=0x42ab00)
    at /usr/src/debug/emacs-24.3.93-4/src/emacs.c:1647

[-- Attachment #4: gdb-7.8-x86 --]
[-- Type: text/plain, Size: 7594 bytes --]

$ gdb /usr/bin/emacs-w32.exe
GNU gdb (GDB) 7.8
[...]
Reading symbols from /usr/bin/emacs-w32.exe...Reading symbols from /usr/lib/debug//usr/bin/emacs-w32.exe.dbg...done.
done.
(gdb) b run_timers
Breakpoint 1 at 0x5e625e: file /usr/src/debug/emacs-24.3.93-4/src/atimer.c, line 343.
(gdb) r -Q
Starting program: /usr/bin/emacs-w32.exe -Q
[New Thread 1568.0x1554]
[New Thread 1568.0x1bd0]
[New Thread 1568.0x1420]
[New Thread 1568.0x1088]
[New Thread 1568.0x1130]

Breakpoint 1, run_timers () at /usr/src/debug/emacs-24.3.93-4/src/atimer.c:343
343       struct timespec now = current_timespec ();
(gdb) thread apply all bt

Thread 5 (Thread 1568.0x1130):
#0  0x756d78d7 in USER32!DispatchMessageW () from /c/windows/syswow64/USER32.dll
#1  0x005ffd9c in w32_msg_pump (msg_buf=0x3b9acd8)
    at /usr/src/debug/emacs-24.3.93-4/src/w32fns.c:2450
#2  0x005fffdf in w32_msg_worker@4 (arg=0x0)
    at /usr/src/debug/emacs-24.3.93-4/src/w32fns.c:2676
#3  0x61005f04 in _cygtls::call2(unsigned long (*)(void*, void*), void*, void*)@16 (this=<optimized out>, func=func@entry=0x5fff3f <w32_msg_worker@4>,
    arg=0x3b9ac70, arg@entry=0x0, buf=buf@entry=0x3b9cdc4)
    at /home/cygnus/vinschen/mknetrel/src/cygwin-snapshot-20140905-1/winsup/cygwin/cygtls.cc:100
#4  0x61006076 in _cygtls::call (func=0x5fff3f <w32_msg_worker@4>, arg=0x0)
    at /home/cygnus/vinschen/mknetrel/src/cygwin-snapshot-20140905-1/winsup/cygwin/cygtls.cc:30
#5  0x6107f1f8 in threadfunc_fe (arg=<optimized out>)
    at /home/cygnus/vinschen/mknetrel/src/cygwin-snapshot-20140905-1/winsup/cygwin/init.cc:30
#6  0x753e338a in KERNEL32!BaseThreadInitThunk ()
   from /c/windows/syswow64/kernel32.dll
#7  0x77759f72 in ntdll!RtlInitializeExceptionChain ()
   from /c/windows/SysWOW64/ntdll.dll
#8  0x77759f45 in ntdll!RtlInitializeExceptionChain ()
   from /c/windows/SysWOW64/ntdll.dll
#9  0x00000000 in ?? ()

Thread 4 (Thread 1568.0x1088):
#0  0x7773f8d1 in ntdll!ZwWaitForSingleObject ()
   from /c/windows/SysWOW64/ntdll.dll
#1  0x7773f8d1 in ntdll!ZwWaitForSingleObject ()
   from /c/windows/SysWOW64/ntdll.dll
#2  0x758c14ab in WaitForSingleObjectEx ()
   from /c/windows/syswow64/KERNELBASE.dll
#3  0x00000320 in ?? ()
#4  0x00000000 in ?? ()

Thread 3 (Thread 1568.0x1420):
#0  0x7774015d in ntdll!ZwWaitForMultipleObjects ()
   from /c/windows/SysWOW64/ntdll.dll
#1  0x7774015d in ntdll!ZwWaitForMultipleObjects ()
   from /c/windows/SysWOW64/ntdll.dll
#2  0x77772f91 in ntdll!RtlMoveMemory () from /c/windows/SysWOW64/ntdll.dll
#3  0x00000001 in ?? ()
#4  0x00000001 in ?? ()
#5  0x00000000 in ?? ()

Thread 2 (Thread 1568.0x1bd0):
#0  0x7773f905 in ntdll!ZwReadFile () from /c/windows/SysWOW64/ntdll.dll
#1  0x7773f905 in ntdll!ZwReadFile () from /c/windows/SysWOW64/ntdll.dll
#2  0x758bdd62 in ReadFile () from /c/windows/syswow64/KERNELBASE.dll
#3  0x00000090 in ?? ()
#4  0x753e3ee7 in ReadFile () from /c/windows/syswow64/kernel32.dll
#5  0x00000090 in ?? ()
#6  0x610e1b8c in wait_sig ()
    at /home/cygnus/vinschen/mknetrel/src/cygwin-snapshot-20140905-1/winsup/cygwin/sigproc.cc:1239
#7  0x61004cc7 in cygthread::callfunc(bool)@8 (
    this=this@entry=0x6119b840 <threads>, issimplestub=issimplestub@entry=false)
    at /home/cygnus/vinschen/mknetrel/src/cygwin-snapshot-20140905-1/winsup/cygwin/cygthread.cc:51
#8  0x6100523a in cygthread::stub(void*)@4 (arg=0x6119b840 <threads>)
    at /home/cygnus/vinschen/mknetrel/src/cygwin-snapshot-20140905-1/winsup/cygwin/cygthread.cc:93
#9  0x61005f04 in _cygtls::call2(unsigned long (*)(void*, void*), void*, void*)@16 (this=<optimized out>, func=func@entry=0x610051f0 <cygthread::stub(void*)@4>,
    arg=arg@entry=0x6119b840 <threads>, buf=buf@entry=0x2d1cdc4)
    at /home/cygnus/vinschen/mknetrel/src/cygwin-snapshot-20140905-1/winsup/cygwin/cygtls.cc:100
#10 0x61006076 in _cygtls::call (func=0x610051f0 <cygthread::stub(void*)@4>,
    arg=0x6119b840 <threads>)
    at /home/cygnus/vinschen/mknetrel/src/cygwin-snapshot-20140905-1/winsup/cygwin/cygtls.cc:30
#11 0x6107f1f8 in threadfunc_fe (arg=<optimized out>)
    at /home/cygnus/vinschen/mknetrel/src/cygwin-snapshot-20140905-1/winsup/cygwin/init.cc:30
#12 0x753e338a in KERNEL32!BaseThreadInitThunk ()
   from /c/windows/syswow64/kernel32.dll
#13 0x77759f72 in ntdll!RtlInitializeExceptionChain ()
   from /c/windows/SysWOW64/ntdll.dll
#14 0x77759f45 in ntdll!RtlInitializeExceptionChain ()
   from /c/windows/SysWOW64/ntdll.dll
#15 0x00000000 in ?? ()

Thread 1 (Thread 1568.0x1554):
#0  run_timers () at /usr/src/debug/emacs-24.3.93-4/src/atimer.c:343
#1  0x005e634c in do_pending_atimers ()
    at /usr/src/debug/emacs-24.3.93-4/src/atimer.c:385
#2  0x0050d946 in process_pending_signals ()
    at /usr/src/debug/emacs-24.3.93-4/src/keyboard.c:7105
#3  0x0056c13e in Fmake_list (length=0, init=9555994)
    at /usr/src/debug/emacs-24.3.93-4/src/alloc.c:2644
#4  0x00594cba in concat (nargs=1, args=0x28a030, target_type=Lisp_Cons,
    last_special=false) at /usr/src/debug/emacs-24.3.93-4/src/fns.c:588
#5  0x005947cf in Fcopy_sequence (arg=11843918)
    at /usr/src/debug/emacs-24.3.93-4/src/fns.c:456
#6  0x00508e74 in timer_check ()
    at /usr/src/debug/emacs-24.3.93-4/src/keyboard.c:4568
#7  0x00506f54 in readable_events (flags=1)
    at /usr/src/debug/emacs-24.3.93-4/src/keyboard.c:3447
#8  0x0050d473 in get_input_pending (flags=1)
    at /usr/src/debug/emacs-24.3.93-4/src/keyboard.c:6765
#9  0x00513b17 in detect_input_pending_run_timers (do_display=true)
    at /usr/src/debug/emacs-24.3.93-4/src/keyboard.c:9894
#10 0x005dafca in wait_reading_process_output (time_limit=0, nsecs=0,
    read_kbd=-1, do_display=true, wait_for_cell=9555994, wait_proc=0x0,
    just_wait_proc=0) at /usr/src/debug/emacs-24.3.93-4/src/process.c:4699
#11 0x00507c76 in kbd_buffer_get_event (kbp=0x28a2e4, used_mouse_menu=0x28a7e3,
    end_time=0x0) at /usr/src/debug/emacs-24.3.93-4/src/keyboard.c:3906
#12 0x00503c64 in read_event_from_main_queue (end_time=0x0,
    local_getcjmp=0x28a5ac, used_mouse_menu=0x28a7e3)
    at /usr/src/debug/emacs-24.3.93-4/src/keyboard.c:2246
#13 0x00503ec0 in read_decoded_event_from_main_queue (end_time=0x0,
    local_getcjmp=0x28a5ac, prev_event=9555994, used_mouse_menu=0x28a7e3)
    at /usr/src/debug/emacs-24.3.93-4/src/keyboard.c:2309
#14 0x005056af in read_char (commandflag=1, map=11275222, prev_event=9555994,
    used_mouse_menu=0x28a7e3, end_time=0x0)
    at /usr/src/debug/emacs-24.3.93-4/src/keyboard.c:2895
#15 0x00512218 in read_key_sequence (keybuf=0x28a900, bufsize=30,
    prompt=9555994, dont_downcase_last=false, can_return_switch_frame=true,
    fix_current_buffer=true, prevent_redisplay=false)
    at /usr/src/debug/emacs-24.3.93-4/src/keyboard.c:9088
#16 0x00502184 in command_loop_1 ()
    at /usr/src/debug/emacs-24.3.93-4/src/keyboard.c:1452
#17 0x0058d2bd in internal_condition_case (bfun=0x501e06 <command_loop_1>,
    handlers=9606530, hfun=0x501633 <cmd_error>)
    at /usr/src/debug/emacs-24.3.93-4/src/eval.c:1354
#18 0x00501ab0 in command_loop_2 (ignore=9555994)
    at /usr/src/debug/emacs-24.3.93-4/src/keyboard.c:1177
#19 0x0058c82a in internal_catch (tag=9608722, func=0x501a8c <command_loop_2>,
    arg=9555994) at /usr/src/debug/emacs-24.3.93-4/src/eval.c:1118
#20 0x00501a66 in command_loop ()
    at /usr/src/debug/emacs-24.3.93-4/src/keyboard.c:1156
#21 0x005011c7 in recursive_edit_1 ()
    at /usr/src/debug/emacs-24.3.93-4/src/keyboard.c:777
#22 0x00501387 in Frecursive_edit ()
    at /usr/src/debug/emacs-24.3.93-4/src/keyboard.c:848
#23 0x004ff777 in main (argc=2, argv=0x28ac3c)
    at /usr/src/debug/emacs-24.3.93-4/src/emacs.c:1647

[-- Attachment #5: gdb-7.8-x86_64 --]
[-- Type: text/plain, Size: 8377 bytes --]

$ gdb /usr/bin/emacs-w32.exe
GNU gdb (GDB) 7.8
[...]
Reading symbols from /usr/bin/emacs-w32.exe...Reading symbols from /usr/lib/debug//usr/bin/emacs-w32.exe.dbg...done.
done.
(gdb) b run_timers
Breakpoint 1 at 0x10064786a: file /usr/src/debug/emacs-24.3.93-4/src/atimer.c, line 343.
(gdb) r -Q
Starting program: /usr/bin/emacs-w32.exe -Q
[New Thread 2804.0x17f4]
[New Thread 2804.0x126c]
[New Thread 2804.0x2e4]
[New Thread 2804.0x690]
[New Thread 2804.0x19e8]

Breakpoint 1, run_timers () at /usr/src/debug/emacs-24.3.93-4/src/atimer.c:343
343       struct timespec now = current_timespec ();
(gdb) thread apply all bt

Thread 5 (Thread 2804.0x19e8):
#0  0x0000000077339e6a in USER32!SfmDxSetSwapChainStats ()
   from /c/windows/system32/USER32.dll
#1  0x0000000077339e9e in USER32!GetMessageW ()
   from /c/windows/system32/USER32.dll
#2  0x000000010066598a in w32_msg_pump (msg_buf=0x3a1ac00)
    at /usr/src/debug/emacs-24.3.93-4/src/w32fns.c:2450
#3  0x0000000100665c31 in w32_msg_worker (arg=0x0)
    at /usr/src/debug/emacs-24.3.93-4/src/w32fns.c:2676
#4  0x000000018004613b in _cygtls::call2 (this=0x3a1ce00,
    func=0x100665b90 <w32_msg_worker>, arg=0x0, buf=buf@entry=0x3a1cd50)
    at /home/cygnus/vinschen/mknetrel/src/cygwin-snapshot-20140905-1/winsup/cygwin/cygtls.cc:100
#5  0x0000000180046294 in _cygtls::call (func=<optimized out>,
    arg=<optimized out>)
    at /home/cygnus/vinschen/mknetrel/src/cygwin-snapshot-20140905-1/winsup/cygwin/cygtls.cc:30
#6  0x00000000774359ed in KERNEL32!BaseThreadInitThunk ()
   from /c/windows/system32/kernel32.dll
#7  0x000000007756c541 in ntdll!RtlUserThreadStart ()
   from /c/windows/SYSTEM32/ntdll.dll
#8  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 4 (Thread 2804.0x690):
#0  0x00000000775912fa in ntdll!ZwWaitForSingleObject ()
   from /c/windows/SYSTEM32/ntdll.dll
#1  0x000007fefd4e10dc in WaitForSingleObjectEx ()
   from /c/windows/system32/KERNELBASE.dll
#2  0x0000000180045501 in cygthread::stub (
    arg=arg@entry=0x1801d9558 <threads+88>)
    at /home/cygnus/vinschen/mknetrel/src/cygwin-snapshot-20140905-1/winsup/cygwin/cygthread.cc:114
#3  0x000000018004613b in _cygtls::call2 (this=0x361ce00,
    func=0x180045470 <cygthread::stub(void*)>, arg=0x1801d9558 <threads+88>,
    buf=buf@entry=0x361cd50)
    at /home/cygnus/vinschen/mknetrel/src/cygwin-snapshot-20140905-1/winsup/cygwin/cygtls.cc:100
#4  0x0000000180046294 in _cygtls::call (func=<optimized out>,
    arg=<optimized out>)
    at /home/cygnus/vinschen/mknetrel/src/cygwin-snapshot-20140905-1/winsup/cygwin/cygtls.cc:30
#5  0x00000000774359ed in KERNEL32!BaseThreadInitThunk ()
   from /c/windows/system32/kernel32.dll
#6  0x000000007756c541 in ntdll!RtlUserThreadStart ()
   from /c/windows/SYSTEM32/ntdll.dll
#7  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 3 (Thread 2804.0x2e4):
#0  0x000000007759186a in ntdll!ZwWaitForMultipleObjects ()
   from /c/windows/SYSTEM32/ntdll.dll
#1  0x000000007755b037 in ntdll!TpIsTimerSet ()
   from /c/windows/SYSTEM32/ntdll.dll
#2  0x00000000774359ed in KERNEL32!BaseThreadInitThunk ()
   from /c/windows/system32/kernel32.dll
#3  0x000000007756c541 in ntdll!RtlUserThreadStart ()
   from /c/windows/SYSTEM32/ntdll.dll
#4  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 2 (Thread 2804.0x126c):
#0  0x000000007759131a in ntdll!ZwReadFile () from /c/windows/SYSTEM32/ntdll.dll
#1  0x000007fefd4e1a7a in ReadFile () from /c/windows/system32/KERNELBASE.dll
#2  0x0000000077430a19 in ReadFile () from /c/windows/system32/kernel32.dll
#3  0x000000018011ec62 in wait_sig ()
    at /home/cygnus/vinschen/mknetrel/src/cygwin-snapshot-20140905-1/winsup/cygwin/sigproc.cc:1239
#4  0x0000000180044f65 in cygthread::callfunc (
    this=this@entry=0x1801d9500 <threads>,
    issimplestub=issimplestub@entry=false)
    at /home/cygnus/vinschen/mknetrel/src/cygwin-snapshot-20140905-1/winsup/cygwin/cygthread.cc:51
#5  0x00000001800454ca in cygthread::stub (arg=arg@entry=0x1801d9500 <threads>)
    at /home/cygnus/vinschen/mknetrel/src/cygwin-snapshot-20140905-1/winsup/cygwin/cygthread.cc:93
#6  0x000000018004613b in _cygtls::call2 (this=0x220ce00,
    func=0x180045470 <cygthread::stub(void*)>, arg=0x1801d9500 <threads>,
    buf=buf@entry=0x220cd50)
    at /home/cygnus/vinschen/mknetrel/src/cygwin-snapshot-20140905-1/winsup/cygwin/cygtls.cc:100
#7  0x0000000180046294 in _cygtls::call (func=<optimized out>,
    arg=<optimized out>)
    at /home/cygnus/vinschen/mknetrel/src/cygwin-snapshot-20140905-1/winsup/cygwin/cygtls.cc:30
#8  0x00000000774359ed in KERNEL32!BaseThreadInitThunk ()
   from /c/windows/system32/kernel32.dll
#9  0x000000007756c541 in ntdll!RtlUserThreadStart ()
   from /c/windows/SYSTEM32/ntdll.dll
#10 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 1 (Thread 2804.0x17f4):
#0  run_timers () at /usr/src/debug/emacs-24.3.93-4/src/atimer.c:343
#1  0x00000001006479b4 in do_pending_atimers ()
    at /usr/src/debug/emacs-24.3.93-4/src/atimer.c:385
#2  0x000000010053fb87 in process_pending_signals ()
    at /usr/src/debug/emacs-24.3.93-4/src/keyboard.c:7105
#3  0x00000001005b3c53 in Fmake_list (length=0, init=4306501682)
    at /usr/src/debug/emacs-24.3.93-4/src/alloc.c:2644
#4  0x00000001005e4e85 in concat (nargs=1, args=0x429650,
    target_type=Lisp_Cons, last_special=false)
    at /usr/src/debug/emacs-24.3.93-4/src/fns.c:588
#5  0x00000001005e4868 in Fcopy_sequence (arg=4314362118)
    at /usr/src/debug/emacs-24.3.93-4/src/fns.c:456
#6  0x000000010053a151 in timer_check ()
    at /usr/src/debug/emacs-24.3.93-4/src/keyboard.c:4571
#7  0x000000010053789f in readable_events (flags=1)
    at /usr/src/debug/emacs-24.3.93-4/src/keyboard.c:3447
#8  0x000000010053f5dd in get_input_pending (flags=1)
    at /usr/src/debug/emacs-24.3.93-4/src/keyboard.c:6765
#9  0x000000010054707e in detect_input_pending_run_timers (do_display=true)
    at /usr/src/debug/emacs-24.3.93-4/src/keyboard.c:9894
#10 0x000000010063a087 in wait_reading_process_output (time_limit=0, nsecs=0,
    read_kbd=-1, do_display=true, wait_for_cell=4306501682, wait_proc=0x0,
    just_wait_proc=0) at /usr/src/debug/emacs-24.3.93-4/src/process.c:4699
#11 0x00000001005388e4 in kbd_buffer_get_event (kbp=0x429b88,
    used_mouse_menu=0x42a3cf, end_time=0x0)
    at /usr/src/debug/emacs-24.3.93-4/src/keyboard.c:3906
#12 0x00000001005337c3 in read_event_from_main_queue (end_time=0x0,
    local_getcjmp=0x429fb0, used_mouse_menu=0x42a3cf)
    at /usr/src/debug/emacs-24.3.93-4/src/keyboard.c:2246
#13 0x0000000100533b00 in read_decoded_event_from_main_queue (end_time=0x0,
    local_getcjmp=0x429fb0, prev_event=4306501682, used_mouse_menu=0x42a3cf)
    at /usr/src/debug/emacs-24.3.93-4/src/keyboard.c:2309
#14 0x0000000100535ab4 in read_char (commandflag=1, map=25770257174,
    prev_event=4306501682, used_mouse_menu=0x42a3cf, end_time=0x0)
    at /usr/src/debug/emacs-24.3.93-4/src/keyboard.c:2895
#15 0x000000010054507f in read_key_sequence (keybuf=0x42a5e0, bufsize=30,
    prompt=4306501682, dont_downcase_last=false, can_return_switch_frame=true,
    fix_current_buffer=true, prevent_redisplay=false)
    at /usr/src/debug/emacs-24.3.93-4/src/keyboard.c:9088
#16 0x00000001005314d4 in command_loop_1 ()
    at /usr/src/debug/emacs-24.3.93-4/src/keyboard.c:1452
#17 0x00000001005db83c in internal_condition_case (
    bfun=0x100530fbf <command_loop_1>, handlers=4306576130,
    hfun=0x10053054a <cmd_error>)
    at /usr/src/debug/emacs-24.3.93-4/src/eval.c:1354
#18 0x0000000100530b8d in command_loop_2 (ignore=4306501682)
    at /usr/src/debug/emacs-24.3.93-4/src/keyboard.c:1177
#19 0x00000001005dab8d in internal_catch (tag=4306570290,
    func=0x100530b5b <command_loop_2>, arg=4306501682)
    at /usr/src/debug/emacs-24.3.93-4/src/eval.c:1118
#20 0x0000000100530b1c in command_loop ()
    at /usr/src/debug/emacs-24.3.93-4/src/keyboard.c:1156
#21 0x000000010052ffab in recursive_edit_1 ()
    at /usr/src/debug/emacs-24.3.93-4/src/keyboard.c:777
#22 0x00000001005301dd in Frecursive_edit ()
    at /usr/src/debug/emacs-24.3.93-4/src/keyboard.c:848
#23 0x000000010052debe in main (argc=2, argv=0x42ab00)
    at /usr/src/debug/emacs-24.3.93-4/src/emacs.c:1647


[-- Attachment #6: Type: text/plain, Size: 218 bytes --]

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

^ permalink raw reply	[flat|nested] 3+ messages in thread

* Re: Strange gdb backtraces on 64-bit Cygwin
  2014-09-17 22:00 Strange gdb backtraces on 64-bit Cygwin Ken Brown
@ 2014-09-18  9:23 ` Markus Hoenicka
  2014-09-23 12:47 ` Ken Brown
  1 sibling, 0 replies; 3+ messages in thread
From: Markus Hoenicka @ 2014-09-18  9:23 UTC (permalink / raw)
  To: Ken Brown; +Cc: cygwin, Eli Zaretskii

At 2014-09-17 23:21, Ken Brown was heard to say:
> There have been many bug reports involving crashes or assertion
> failures in emacs-X11 or emacs-w32 on 64-bit Cygwin.  Many of these
> reports include gdb backtraces that don't make sense.  The one I'm
> looking at right now is emacs bug#17753.  I'll try to make this email
> self-contained, but anyone interested in the context should start at
> 
>   http://debbugs.gnu.org/cgi/bugreport.cgi?bug=17753#47
> 
> The situation in that bug report is that a gdb backtrace showed a call
> to the emacs function "run_timers" in Thread 2, which shouldn't
> happen.  (It should only be called in the main thread.)  There was
> speculation as to whether this could be the cause of the crash.  An
> alternative theory is that the gdb backtrace is bogus.  I'm pretty
> sure that the OP (Markus) was running gdb-7.6.50-4.  His report was
> about emacs-X11, but I've observed similar backtraces for emacs-w32.
> 

This is just to confirm the suspected version of gdb on the OP's box:

$ cygcheck -f /usr/bin/gdb
gdb-7.6.50-4

$ gdb -v
GNU gdb (GDB) 7.6.50.20130728-cvs (cygwin-special)
[...]

regards,
Markus

-- 
Markus Hoenicka
http://www.mhoenicka.de
AQ score 38


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

^ permalink raw reply	[flat|nested] 3+ messages in thread

* Re: Strange gdb backtraces on 64-bit Cygwin
  2014-09-17 22:00 Strange gdb backtraces on 64-bit Cygwin Ken Brown
  2014-09-18  9:23 ` Markus Hoenicka
@ 2014-09-23 12:47 ` Ken Brown
  1 sibling, 0 replies; 3+ messages in thread
From: Ken Brown @ 2014-09-23 12:47 UTC (permalink / raw)
  To: cygwin; +Cc: Eli Zaretskii

On 9/17/2014 5:21 PM, Ken Brown wrote:
> There have been many bug reports involving crashes or assertion failures
> in emacs-X11 or emacs-w32 on 64-bit Cygwin.  Many of these reports
> include gdb backtraces that don't make sense.  The one I'm looking at
> right now is emacs bug#17753.  I'll try to make this email
> self-contained, but anyone interested in the context should start at
>
>    http://debbugs.gnu.org/cgi/bugreport.cgi?bug=17753#47
>
> The situation in that bug report is that a gdb backtrace showed a call
> to the emacs function "run_timers" in Thread 2, which shouldn't happen.
>   (It should only be called in the main thread.)  There was speculation
> as to whether this could be the cause of the crash.  An alternative
> theory is that the gdb backtrace is bogus.  I'm pretty sure that the OP
> (Markus) was running gdb-7.6.50-4.  His report was about emacs-X11, but
> I've observed similar backtraces for emacs-w32.
>
> To try to sort this out, I've done the following experiment: I've run
> emacs-w32 under gdb with a breakpoint at run_timers.  I've done this on
> both 32-bit Cygwin and 64-bit Cygwin [*], using both gdb-7.6.50-4 and
> gdb-7.8.  [For the latter I used my own build, since the bugfix we
> discussed in a different thread hasn't yet made it into the Cygwin
> distro.]  Transcripts of the four gdb sessions are attached; the file
> names indicate the gdb version and the platform.
>
> My reading of these transcripts is that gdb-7.6.50-4 on 64-bit Cygwin is
> the outlier, and the strange occurrence of run_timers in Thread 2 is
> therefore likely to be a result of a gdb bug.  But it would be great if
> someone familiar with recent gdb development could shed some light on
> this.  In particular, is it plausible that there was a bug of this
> nature in gdb-7.6 that affected only the 64-bit platform and that has
> since been fixed?

I bisected the binutils-gdb repository and found that the strange 
backtrace stopped after the following commit:

commit 9058cc3a1bbf4c43a484120290e4245622782bb0
Author: Tristan Gingold <gingold@adacore.com>
Date:   Mon Sep 2 09:28:02 2013 +0000

     2013-09-02  Tristan Gingold  <gingold@adacore.com>

         * NEWS: Add entry mentioning support for native Windows x64
         SEH data.

         * amd64-windows-tdep.c: #include "objfiles.h", "frame-unwind.h",
         "coff/internal.h", "coff/i386.h", "coff/pe.h" and "libcoff.h".
         (struct amd64_windows_frame_cache): New struct.
         (amd64_windows_w2gdb_regnum): New global.
         (pc_in_range, amd64_windows_frame_decode_epilogue)
         (amd64_windows_frame_decode_insns, amd64_windows_find_unwind_info)
         (amd64_windows_frame_cache, amd64_windows_frame_prev_register)
         (amd64_windows_frame_this_id): New functions.
         (amd64_windows_frame_unwind): New static global.
         (amd64_windows_skip_prologue): New function.
         (amd64_windows_init_abi): Call frame_unwind_prepend_unwinder
         with amd64_windows_frame_unwind. Call set_gdbarch_skip_prologue
         with amd64_windows_skip_prologue.

So I think it's pretty clear that the strange backtrace I observed with 
gdb-7.6.50-4 on 64-bit Cygwin was indeed due to a deficiency in gdb.

I hope that people who have been experiencing emacs crashes with 
"impossible" backtraces will update to gdb-7.8-2.

Ken

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

^ permalink raw reply	[flat|nested] 3+ messages in thread

end of thread, other threads:[~2014-09-23 12:31 UTC | newest]

Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2014-09-17 22:00 Strange gdb backtraces on 64-bit Cygwin Ken Brown
2014-09-18  9:23 ` Markus Hoenicka
2014-09-23 12:47 ` Ken Brown

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).