Discrepancy in GDB/MI output stream when debugging remote target via pipe to gdbserver

Discrepancy in GDB/MI output stream when debugging remote target via pipe to gdbserver

[Angelo J. Piazza]

[-- Attachment #1: Type: text/plain, Size: 6657 bytes --]

I'm trying to debug a program running on a remote device using gdbserver via stdio pipes. My issue is , when using the MI interpreter, stdout messages (i.e. cout) from the debugee are coming through as log streams (prepended with '&') as opposed to regular text. I illustrate this in detail below. My arch is x86_64.

Though my end-goal is to perform this remote debugging over ssh (security is important), the issue I've run into persists whether I'm piping through ssh or not. Therefore, the examples below will omit the ssh part for ease of reproduction.

Here are the commands I use to initialize my debug session... (I've tried to add red text to the commands I input, hopefully it shows through)

--- Begin piped gdbserver sample ---
gdb --interpreter=mi
=thread-group-added,id="i1"
~"GNU gdb (GDB) 12.0.90\n"
~"Copyright (C) 2022 Free Software Foundation, Inc.\n"
~"License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html\nThis is free software: you are free to change and redistribute it.\nThere is NO WARRANTY, to the extent permitted by law."
~"\nType \"show copying\" and \"show warranty\" for details.\n"
~"This GDB was configured as \"x86_64-pc-linux-gnu\".\n"
~"Type \"show configuration\" for configuration details.\n"
~"For bug reporting instructions, please see:\n"
~"https://www.gnu.org/software/gdb/bugs/.\n"
~"Find the GDB manual and other documentation resources online at:\n    http://www.gnu.org/software/gdb/documentation/."
~"\n\n"
~"For help, type \"help\".\n"
~"Type \"apropos word\" to search for commands related to \"word\".\n"
(gdb)
-target-select extended-remote | gdbserver --multi -
&"Remote debugging using stdio\n"
=tsv-created,name="trace_timestamp",initial="0"
^connected
(gdb)
-interpreter-exec console "set remote exec-file /home/odt/ohalo-olympus/OlympusFAS"
=cmd-param-changed,param="remote exec-file",value="/home/odt/ohalo-olympus/OlympusFAS"
^done
(gdb)
-file-exec-and-symbols /home/odt/Projects/olympus/build/Release/src/OlympusFAS
^done
(gdb)
-exec-run
&"stdin/stdout redirected\n"
&"Process /home/odt/ohalo-olympus/OlympusFAS created; pid = 74077\n"
...
&"Starting up Project\n"
&"Running from /home/odt/ohalo-olympus\n"
&"Loaded Valid Configuration File.\n"
&"\n"
&"Set Default Camera Parameters\n"
&"State File 1 Not Valid.\n"
&"State File 2 Not Valid.\n"
&"State File 3 Not Valid.\n"
&"Unable to Set Startup Time.\n"
&"[0] System Startup Complete But Recover Data Not Valid\n"
&"\n"
&"Logging Started to logger.txt\n"
&"Sync With Network Time.\n"
&"Current Image File Is -1\n"
&"Unable to Find Current Log File.\n"
&"Camera Startup Success.\n"
&"Unable to Find Current Data File."
&"\n"
&"Unable to Find Next Log File."
&"\n"
&"Unable to Find Next Data File."
...
--- End piped gdbserver sample ---

Notice at the end how the print statements are prepended with an ampersand. These are the print outputs from my program, which should just be regular print statements, but are instead marked as log streams from gdb's internal engine and thus get parsed by my front-end GUI as such.

As a comparison, below is the exact same program run locally (not via a piped remote)

--- Begin local debug sample ---
gdb --interpreter=mi
=thread-group-added,id="i1"
~"GNU gdb (GDB) 12.0.90\n"
~"Copyright (C) 2022 Free Software Foundation, Inc.\n"
~"License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html\nThis is free software: you are free to change and redistribute it.\nThere is NO WARRANTY, to the extent permitted by law."
~"\nType \"show copying\" and \"show warranty\" for details.\n"
~"This GDB was configured as \"x86_64-pc-linux-gnu\".\n"
~"Type \"show configuration\" for configuration details.\n"
~"For bug reporting instructions, please see:\n"
~"https://www.gnu.org/software/gdb/bugs/.\n"
~"Find the GDB manual and other documentation resources online at:\n    http://www.gnu.org/software/gdb/documentation/."
~"\n\n"
~"For help, type \"help\".\n"
~"Type \"apropos word\" to search for commands related to \"word\".\n"
(gdb)
-file-exec-and-symbols /home/odt/Projects/olympus/build/Release/src/OlympusFAS
^done
(gdb)
-exec-run
...
Starting up Project
Loaded Valid Configuration File.

Set Default Camera Parameters
State File 1 Not Valid.
State File 2 Not Valid.
State File 3 Not Valid.
Unable to Set Startup Time.
[0] System Startup Complete But Recover Data Not Valid
=thread-created,id="2",group-id="i1"
~"[New Thread 0x7fffec773700 (LWP 74434)]\n"
*running,thread-id="2"

Logging Started to logger.txt
FLIGHT APP VERSION 1060-S-6130-09
Sync With Network Time.
Current Image File Is -1
Unable to Find Current Log File.
Camera Startup Success.
Unable to Find Current Data File.
...
--- End local debug sample ---

Is this a bug? Is there a better way to establish the remote target so that the MI interpreter can properly differentiate between actual gdb log messages and the stdout from a piped remote?

The ultimate consequence of this is mostly aesthetic, as the GUI I am using for debug interprets the amperstand-prepended messages as log messages and therefore does not display them to my console. This is rather inconvenient as I gain the ability to step through code, but lose my print statements.  I want to have my cake and eat it too!

This is my first time posting to this email list, so please forgive me if I've done something improper. Hopefully this is just a trivial thing I am missing!

Respectfully,
Angelo



Angelo J. Piazza
The Bionetics Corporation<http://www.bionetics.com>

Palm Bay, FL
Phone: (321) 261-3784
FAX:

[Bionetics]

Important Notice to Recipient: Please do not read, copy or disseminate this communication or any attachment unless you are the intended addressee. This communication may contain confidential information intended only for the addressee(s). Anyone who receives this communication in error should treat it as confidential and is asked to contact the sender at the e-mail, phone number or fax number listed above. Please do not forward or disseminate this information to any third party.

Important Notice to Recipient: 
Please do not read, copy or disseminate this communication or any attachment unless you are the intended addressee. 
This communication may contain confidential information intended only for the addressee(s). 
Anyone who receives this communication in error should treat it as confidential and is asked to contact the sender at the e-mail, phone number or fax number listed above.
Please do not forward or disseminate this information to any third party.

Re: Discrepancy in GDB/MI output stream when debugging remote target via pipe to gdbserver

[Simon Marchi]

On 11/9/22 21:26, Angelo J. Piazza via Gdb wrote:
> I'm trying to debug a program running on a remote device using gdbserver via stdio pipes. My issue is , when using the MI interpreter, stdout messages (i.e. cout) from the debugee are coming through as log streams (prepended with '&') as opposed to regular text. I illustrate this in detail below. My arch is x86_64.
> 
> Though my end-goal is to perform this remote debugging over ssh (security is important), the issue I've run into persists whether I'm piping through ssh or not. Therefore, the examples below will omit the ssh part for ease of reproduction.
> 
> Here are the commands I use to initialize my debug session... (I've tried to add red text to the commands I input, hopefully it shows through)
> 
> --- Begin piped gdbserver sample ---
> gdb --interpreter=mi
> =thread-group-added,id="i1"
> ~"GNU gdb (GDB) 12.0.90\n"
> ~"Copyright (C) 2022 Free Software Foundation, Inc.\n"
> ~"License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html\nThis is free software: you are free to change and redistribute it.\nThere is NO WARRANTY, to the extent permitted by law."
> ~"\nType \"show copying\" and \"show warranty\" for details.\n"
> ~"This GDB was configured as \"x86_64-pc-linux-gnu\".\n"
> ~"Type \"show configuration\" for configuration details.\n"
> ~"For bug reporting instructions, please see:\n"
> ~"https://www.gnu.org/software/gdb/bugs/.\n"
> ~"Find the GDB manual and other documentation resources online at:\n    http://www.gnu.org/software/gdb/documentation/."
> ~"\n\n"
> ~"For help, type \"help\".\n"
> ~"Type \"apropos word\" to search for commands related to \"word\".\n"
> (gdb)
> -target-select extended-remote | gdbserver --multi -
> &"Remote debugging using stdio\n"
> =tsv-created,name="trace_timestamp",initial="0"
> ^connected
> (gdb)
> -interpreter-exec console "set remote exec-file /home/odt/ohalo-olympus/OlympusFAS"
> =cmd-param-changed,param="remote exec-file",value="/home/odt/ohalo-olympus/OlympusFAS"
> ^done
> (gdb)
> -file-exec-and-symbols /home/odt/Projects/olympus/build/Release/src/OlympusFAS
> ^done
> (gdb)
> -exec-run
> &"stdin/stdout redirected\n"
> &"Process /home/odt/ohalo-olympus/OlympusFAS created; pid = 74077\n"
> ...
> &"Starting up Project\n"
> &"Running from /home/odt/ohalo-olympus\n"
> &"Loaded Valid Configuration File.\n"
> &"\n"
> &"Set Default Camera Parameters\n"
> &"State File 1 Not Valid.\n"
> &"State File 2 Not Valid.\n"
> &"State File 3 Not Valid.\n"
> &"Unable to Set Startup Time.\n"
> &"[0] System Startup Complete But Recover Data Not Valid\n"
> &"\n"
> &"Logging Started to logger.txt\n"
> &"Sync With Network Time.\n"
> &"Current Image File Is -1\n"
> &"Unable to Find Current Log File.\n"
> &"Camera Startup Success.\n"
> &"Unable to Find Current Data File."
> &"\n"
> &"Unable to Find Next Log File."
> &"\n"
> &"Unable to Find Next Data File."
> ...
> --- End piped gdbserver sample ---
> 
> Notice at the end how the print statements are prepended with an ampersand. These are the print outputs from my program, which should just be regular print statements, but are instead marked as log streams from gdb's internal engine and thus get parsed by my front-end GUI as such.
> 
> As a comparison, below is the exact same program run locally (not via a piped remote)
> 
> --- Begin local debug sample ---
> gdb --interpreter=mi
> =thread-group-added,id="i1"
> ~"GNU gdb (GDB) 12.0.90\n"
> ~"Copyright (C) 2022 Free Software Foundation, Inc.\n"
> ~"License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html\nThis is free software: you are free to change and redistribute it.\nThere is NO WARRANTY, to the extent permitted by law."
> ~"\nType \"show copying\" and \"show warranty\" for details.\n"
> ~"This GDB was configured as \"x86_64-pc-linux-gnu\".\n"
> ~"Type \"show configuration\" for configuration details.\n"
> ~"For bug reporting instructions, please see:\n"
> ~"https://www.gnu.org/software/gdb/bugs/.\n"
> ~"Find the GDB manual and other documentation resources online at:\n    http://www.gnu.org/software/gdb/documentation/."
> ~"\n\n"
> ~"For help, type \"help\".\n"
> ~"Type \"apropos word\" to search for commands related to \"word\".\n"
> (gdb)
> -file-exec-and-symbols /home/odt/Projects/olympus/build/Release/src/OlympusFAS
> ^done
> (gdb)
> -exec-run
> ...
> Starting up Project
> Loaded Valid Configuration File.
> 
> Set Default Camera Parameters
> State File 1 Not Valid.
> State File 2 Not Valid.
> State File 3 Not Valid.
> Unable to Set Startup Time.
> [0] System Startup Complete But Recover Data Not Valid
> =thread-created,id="2",group-id="i1"
> ~"[New Thread 0x7fffec773700 (LWP 74434)]\n"
> *running,thread-id="2"
> 
> Logging Started to logger.txt
> FLIGHT APP VERSION 1060-S-6130-09
> Sync With Network Time.
> Current Image File Is -1
> Unable to Find Current Log File.
> Camera Startup Success.
> Unable to Find Current Data File.
> ...
> --- End local debug sample ---
> 
> Is this a bug? Is there a better way to establish the remote target so that the MI interpreter can properly differentiate between actual gdb log messages and the stdout from a piped remote?
> 
> The ultimate consequence of this is mostly aesthetic, as the GUI I am using for debug interprets the amperstand-prepended messages as log messages and therefore does not display them to my console. This is rather inconvenient as I gain the ability to step through code, but lose my print statements.  I want to have my cake and eat it too!

I think the difference comes from the fact that you start and connect to
gdbserver via stdin/stdout:

  -target-select extended-remote | gdbserver --multi -

In this case, GDB talks (remote protocol queries) to gdbserver via
gdbserver's stdin.  gdbserver talks (remote protocol responses) to GDB
via its stdout.  And all the rest, that is gdbserver's own output and
your program's output get shoved in gdbserver's stderr.  GDB collects
that and prints it as "internal messages".

It might be better if you were able to start gdbserver from your
frontend directly and make GDB connect to it using a tcp connection.
Your frontend could collect gdbserver and your program's output
directly.  I suppose you could make gdbserver listen just locally (on
127.0.0.1) and connect through an ssh tunnel.

Otherwise, since programs are started through the shell (by default), it
might be possible to append a shell redirection to redirect your
program's output to a file, a named pipe or whatever.

Simon