RFC: File transfer commands

RFC: File transfer commands

[Daniel Jacobowitz]

Two years ago, I worked on a GDB port to SymbianOS.  The SymbianOS
target was interesting in a couple of ways that required changes to
GDB.  One was the target shared library list, which is included in
GDB 6.7.  Another was the need to specify a running process to attach
to over a remote connection, which I'll be posting about in a few days
or a week.  And a third way was the need for file transfer commands.

SymbianOS is a mobile device platform, primarily phones.  So while the
development board I was using had an Ethernet jack, SymbianOS didn't
do much with it; when was the last time you saw a consumer cell phone
with a 100-BaseT connection?  The only ways to get files onto the
target system were via an external flash (MMC card) and over the
serial port.  While running the GDB stub, the serial port was occupied
by the GDB serial protocol.  In order to run the testsuite, we needed
a way to upload files to the target while that serial port was in
use.

Here is the documentation I've written for it.  There are three new
CLI commands (remote put, remote get, remote delete); three new MI
commands (-target-file-put, -target-file-get, -target-file-delete);
and five new remote protocol packets (vFile:open, vFile:close,
vFile:pread, vFile:pwrite, vFile:unlink).  I'm not committed to any
of the command names or packets; alternatives welcome.

Should GDB have this feature?  Are these the right commands for it?

Patch coming up on gdb-patches.

Index: gdb/doc/gdb.texinfo
===================================================================
--- gdb/doc/gdb.texinfo.orig	2007-10-26 09:50:44.000000000 -0400
+++ gdb/doc/gdb.texinfo	2007-10-29 15:18:09.000000000 -0400
@@ -12615,6 +12615,7 @@ configuration of @value{GDBN}; use @code
 
 @menu
 * Connecting::                  Connecting to a remote target
+* File Transfer::               Sending files to a remote system
 * Server::	                Using the gdbserver program
 * Remote Configuration::        Remote configuration
 * Remote Stub::                 Implementing a remote stub
@@ -12762,6 +12763,37 @@ can add new commands that only the exter
 and implement.
 @end table
 
+@node File Transfer
+@section Sending files to a remote system
+@cindex remote target, file transfer
+@cindex file transfer
+
+Some remote targets offer the ability to transfer files over the same
+connection used to communicate with @value{GDBN}.  This is convenient
+for targets accessible through other means, e.g.@: GNU/Linux systems
+running @code{gdbserver} over a network interface.  For other targets,
+e.g.@: embedded devices with only a single serial port, this may be
+the only way to upload or download files.
+
+Not all remote targets support these commands.
+
+@table @code
+@kindex remote put
+@item remote put @var{hostfile} @var{targetfile}
+Copy @var{hostfile} from the host system (the machine running
+@value{GDBN}) to the path @var{targetfile} on the target system.
+
+@kindex remote get
+@item remote get @var{targetfile} @var{hostfile}
+Copy @var{targetfile} from the target system to @var{hostfile}
+on the host system.
+
+@kindex remote delete
+@item remote delete @var{targetfile}
+Delete @var{targetfile} from the target system.
+
+@end table
+
 @node Server
 @section Using the @code{gdbserver} Program
 
@@ -13098,6 +13130,25 @@ are:
 @tab @code{QPassSignals}
 @tab @code{handle @var{signal}}
 
+@item @code{hostio-close-packet}
+@tab @code{vFile:close}
+@tab @code{remote get}, @code{remote put}
+
+@item @code{hostio-open-packet}
+@tab @code{vFile:open}
+@tab @code{remote get}, @code{remote put}
+
+@item @code{hostio-pread-packet}
+@tab @code{vFile:pread}
+@tab @code{remote get}, @code{remote put}
+
+@item @code{hostio-pwrite-packet}
+@tab @code{vFile:pwrite}
+@tab @code{remote get}, @code{remote put}
+
+@item @code{hostio-unlink-packet}
+@tab @code{vFile:unlink}
+@tab @code{remote delete}
 @end multitable
 
 @node Remote Stub
@@ -17248,6 +17299,7 @@ may repeat one or more times.
 * GDB/MI Signal Handling Commands::
 @end ignore
 * GDB/MI Target Manipulation::
+* GDB/MI File Transfer Commands::
 * GDB/MI Miscellaneous Commands::
 @end menu
 
@@ -21195,6 +21247,88 @@ The corresponding @value{GDBN} command i
 @end smallexample
 
 @c %%%%%%%%%%%%%%%%%%%%%%%%%%%% SECTION %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+@node GDB/MI File Transfer Commands
+@section @sc{gdb/mi} File Transfer Commands
+
+
+@subheading The @code{-target-file-put} Command
+@findex -target-file-put
+
+@subsubheading Synopsis
+
+@smallexample
+ -target-file-put @var{hostfile} @var{targetfile}
+@end smallexample
+
+Copy @var{hostfile} from the host system (the machine running
+@value{GDBN}) to the path @var{targetfile} on the target system.
+
+@subsubheading @value{GDBN} Command
+
+The corresponding @value{GDBN} command is @samp{remote put}.
+
+@subsubheading Example
+
+@smallexample
+(gdb)
+-target-file-put localfile remotefile
+^done
+(gdb)
+@end smallexample
+
+
+@subheading The @code{-target-file-put} Command
+@findex -target-file-get
+
+@subsubheading Synopsis
+
+@smallexample
+ -target-file-get @var{targetfile} @var{hostfile}
+@end smallexample
+
+Copy @var{targetfile} from the target system to @var{hostfile}
+on the host system.
+
+@subsubheading @value{GDBN} Command
+
+The corresponding @value{GDBN} command is @samp{remote get}.
+
+@subsubheading Example
+
+@smallexample
+(gdb)
+-target-file-get remotefile localfile
+^done
+(gdb)
+@end smallexample
+
+
+@subheading The @code{-target-file-delete} Command
+@findex -target-file-delete
+
+@subsubheading Synopsis
+
+@smallexample
+ -target-file-delete @var{targetfile}
+@end smallexample
+
+Delete @var{targetfile} from the target system.
+
+@subsubheading @value{GDBN} Command
+
+The corresponding @value{GDBN} command is @samp{remote delete}.
+
+@subsubheading Example
+
+@smallexample
+(gdb)
+-target-file-delete remotefile
+^done
+(gdb)
+@end smallexample
+
+
+@c %%%%%%%%%%%%%%%%%%%%%%%%%%%% SECTION %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 @node GDB/MI Miscellaneous Commands
 @section Miscellaneous @sc{gdb/mi} Commands
 
@@ -22819,6 +22953,7 @@ Show the current setting of the target w
 * General Query Packets::
 * Register Packet Format::
 * Tracepoint Packets::
+* Host I/O Packets::
 * Interrupts::
 * Examples::
 * File-I/O Remote Protocol Extension::
@@ -23322,6 +23457,11 @@ command in the @samp{vCont} packet.
 The @samp{vCont} packet is not supported.
 @end table
 
+@item vFile:@var{operation}:@var{parameter}@dots{}
+@cindex @samp{vFile} packet
+Perform a file operation on the target system.  For details,
+@xref{Host I/O Packets}.
+
 @item vFlashErase:@var{addr},@var{length}
 @cindex @samp{vFlashErase} packet
 Direct the stub to erase @var{length} bytes of flash starting at
@@ -24471,6 +24611,101 @@ There is a trace experiment running.
 @end table
 
 
+@node Host I/O Packets
+@section Host I/O Packets
+@cindex Host I/O, remote protocol
+@cindex file transfer, remote protocol
+
+The @dfn{Host I/O} packets allow @value{GDBN} to perform I/O
+operations on the far side of a remote link.  For example, Host I/O is
+used to upload and download files to a remote target with its own
+filesystem.  The protocol has some common features with the File-I/O
+protocol, e.g.@: all constants and data structures are encoded in the
+same way.  However, the packets are structured differently, because
+requests are initiated by @value{GDBN}, and the target's memory is not
+involved.  @xref{File-I/O Remote Protocol Extension}, for more details
+on the target-initiated protocol.
+
+The Host I/O request packets all encode a single operation along with
+its arguments.  They have this format:
+
+@table @samp
+
+@item vFile:@var{operation}: @var{parameter}@dots{}
+@var{operation} is the name of the particular request; the target
+should compare the entire packet name up to the second colon when checking
+for a supported operation.  The format of @var{parameter} depends on
+the operation.  Numbers are always passed in hexadecimal.  Negative
+numbers have an explicit minus sign (i.e.@: two's complement is not
+used).  Strings (e.g.@: filenames) are encoded as a series of
+hexadecimal bytes.  The last argument to a system call may be a
+buffer of escaped binary data (@pxref{Binary Data}).
+
+@end table
+
+The valid responses to Host I/O packets are:
+
+@table @samp
+
+@item F @var{result} [, @var{errno}] [; @var{attachment}]
+@var{result} is the integer value returned by this operation, usually
+non-negative for success and -1 for errors.  If an error has occured,
+@var{errno} will be included in the result.  @var{errno} will have a
+value defined by the File-I/O protocol (@pxref{Errno Values}).  For
+operations which return data, @var{attachment} will be provided a
+binary buffer.  Binary buffers in response packets are escaped in the
+normal way (@pxref{Binary Data}).  See the individual packet
+documentation for the interpretation of @var{result} and
+@var{attachment}.
+
+@item
+An empty response indicates that this operation is not recognized.
+
+@end table
+
+These are the supported Host I/O operations:
+
+@table @samp
+@item vFile:open: @var{pathname}, @var{flags}, @var{mode}
+Open a file at @var{pathname} and return a file descriptor for it, or
+return -1 if an error occurs.  @var{pathname} is a string,
+@var{flags} is an integer indicating a mask of open flags
+(@pxref{Open Flags}), and @var{mode} is an integer indicating a mask
+of mode bits to use if the file is created (@pxref{mode_t Values}).
+
+@item vFile:close: @var{fd}
+Close the open file corresponding to @var{fd} and return 0, or
+-1 if an error occurs.
+
+@item vFile:pread: @var{fd}, @var{count}, @var{offset}
+Read data from the open file corresponding to @var{fd}.  Up to
+@var{count} bytes will be read from the file, starting at @var{offset}
+relative to the start of the file.  The target may read fewer bytes;
+common reasons include packet size limits and an end-of-file
+condition.  The number of bytes read is returned.  Zero should only be
+returned for a successful read at the end of the file, or if
+@var{count} was zero.
+
+The data read should be returned as a binary attachment on success,
+even if zero bytes were read.  The return value is the number of
+target bytes read; the binary attachment may be longer if some
+characters were escaped.
+
+@item vFile:pwrite: @var{fd}, @var{offset}, @var{data}
+Write @var{data} (a binary buffer) to the open file corresponding
+to @var{fd}.  Start the write at @var{offset} from the start of the
+file.  Unlike many @code{write} system calls, there is no
+separate @var{count} argument; the length of @var{data} in the
+packet is used.  @samp{vFile:write} returns the number of bytes written,
+which may be shorter than the length of @var{data}, or -1 if an
+error occurred.
+
+@item vFile:unlink: @var{pathname}
+Delete the file at @var{pathname} on the target.  Return 0,
+or -1 if an error occurs.  @var{pathname} is a string.
+
+@end table
+
 @node Interrupts
 @section Interrupts
 @cindex interrupts (remote protocol)


-- 
Daniel Jacobowitz
CodeSourcery

Re: RFC: File transfer commands

[Ulrich Weigand]

Daniel Jacobowitz wrote:

> Two years ago, I worked on a GDB port to SymbianOS.  The SymbianOS
> target was interesting in a couple of ways that required changes to
> GDB.  One was the target shared library list, which is included in
> GDB 6.7.  Another was the need to specify a running process to attach
> to over a remote connection, which I'll be posting about in a few days
> or a week.  And a third way was the need for file transfer commands.

That's interesting, as remote file transfer is on my current to-do
list ;-)   I'd like to remove the requirement to have exact copies
of target shared libraries available on the host system when doing
remote debugging via gdbserver; this is an annoyingly frequent cause
of user complaints ...   To fix this, I'm thinking of extending the
shared library layer to directly access the shared library files
available on the target instead of searching for copies on the host.
This would use some sort of file access via the remote protocol.

> Here is the documentation I've written for it.  There are three new
> CLI commands (remote put, remote get, remote delete); three new MI
> commands (-target-file-put, -target-file-get, -target-file-delete);
> and five new remote protocol packets (vFile:open, vFile:close,
> vFile:pread, vFile:pwrite, vFile:unlink).  I'm not committed to any
> of the command names or packets; alternatives welcome.
> 
> Should GDB have this feature?  Are these the right commands for it?

Hmm, I'd have chosen to use a new target object (TARGET_OBJECT_FILE)
and corresponding qXfer packet; the "annex" would obviously be the
remote file name.

However, that would not support unlink, and it would cause the 
target to re-open the file for each access.  That wouldn't matter
for my intended use scenario, not sure if it would in yours ...

Bye,
Ulrich

-- 
  Dr. Ulrich Weigand
  GNU Toolchain for Linux on System z and Cell BE
  Ulrich.Weigand@de.ibm.com

Re: RFC: File transfer commands

[Daniel Jacobowitz]

On Mon, Oct 29, 2007 at 09:29:41PM +0100, Ulrich Weigand wrote:
> Hmm, I'd have chosen to use a new target object (TARGET_OBJECT_FILE)
> and corresponding qXfer packet; the "annex" would obviously be the
> remote file name.
> 
> However, that would not support unlink, and it would cause the 
> target to re-open the file for each access.  That wouldn't matter
> for my intended use scenario, not sure if it would in yours ...

Yeah, I did think about using qXfer.  Having to close and re-open the
file isn't a problem for me either, but that didn't allow for other
useful operations like delete.  I haven't implemented any more
operations than you see here, but it would be a reasonable addition
to implement "remote ls" - and the corresponding MI commands, which
might let us use a GUI file browser to select the target program
to execute.

I'd figured on retrieving remote shared libraries from memory, rather
than from the filesystem; we don't need the contents, just the dynamic
symbol table.  But that wouldn't take advantage of unstripped
libraries on the target with full symbols, so maybe the file would be
useful too.

-- 
Daniel Jacobowitz
CodeSourcery

Re: RFC: File transfer commands

[Pedro Alves]

Daniel Jacobowitz wrote:
> Should GDB have this feature?
Nice!

> Are these the right commands for it?
Did you consider reusing the file-I/O remote protocol extension (with
host/target reversed)?
http://sourceware.org/gdb/current/onlinedocs/gdb_33.html#SEC681

Cheers,
Pedro Alves

Re: RFC: File transfer commands

[Daniel Jacobowitz]

On Mon, Oct 29, 2007 at 11:39:15PM +0000, Pedro Alves wrote:
> > Are these the right commands for it?
> Did you consider reusing the file-I/O remote protocol extension (with
> host/target reversed)?
> http://sourceware.org/gdb/current/onlinedocs/gdb_33.html#SEC681

I stayed as close to it as I could manage.  The differences are
unavoidable, though, because the File I/O bits store pathnames as
address/length pairs in target memory.  It's too silly (and sometimes
too hard) to poke a filename into target memory just so the debug
server can open it.

My first draft used the same Fopen, Fwrite, etc. packets as the
File I/O bits.  Eventually I realized that that assigned two
conflicting meanings to the F packet.  If I'd tried it on a
stub that supported both it would have gotten really hairy.

-- 
Daniel Jacobowitz
CodeSourcery

Re: RFC: File transfer commands

[Jim Blandy]

Daniel Jacobowitz <drow at false.org> writes:
> On Mon, Oct 29, 2007 at 09:29:41PM +0100, Ulrich Weigand wrote:
>> Hmm, I'd have chosen to use a new target object (TARGET_OBJECT_FILE)
>> and corresponding qXfer packet; the "annex" would obviously be the
>> remote file name.
>> 
>> However, that would not support unlink, and it would cause the 
>> target to re-open the file for each access.  That wouldn't matter
>> for my intended use scenario, not sure if it would in yours ...
>
> Yeah, I did think about using qXfer.  Having to close and re-open the
> file isn't a problem for me either, but that didn't allow for other
> useful operations like delete.  I haven't implemented any more
> operations than you see here, but it would be a reasonable addition
> to implement "remote ls" - and the corresponding MI commands, which
> might let us use a GUI file browser to select the target program
> to execute.

Yeah, 'remote ls' seems like the logical missing piece.  Why not use
'vFile:' as the prefix for delete, and qXfer for read and write?

Re: RFC: File transfer commands

[Daniel Jacobowitz]

On Tue, Oct 30, 2007 at 11:29:12AM -0700, Jim Blandy wrote:
> Yeah, 'remote ls' seems like the logical missing piece.  Why not use
> 'vFile:' as the prefix for delete, and qXfer for read and write?

No preference really besides bikeshed.  More consistency with qXfer,
less with other file operations.  Is the annex the filename or an open
file descriptor?

-- 
Daniel Jacobowitz
CodeSourcery

Re: RFC: File transfer commands

[Jim Blandy]

Daniel Jacobowitz <drow at false.org> writes:
> On Tue, Oct 30, 2007 at 11:29:12AM -0700, Jim Blandy wrote:
>> Yeah, 'remote ls' seems like the logical missing piece.  Why not use
>> 'vFile:' as the prefix for delete, and qXfer for read and write?
>
> No preference really besides bikeshed.  More consistency with qXfer,
> less with other file operations.  Is the annex the filename or an open
> file descriptor?

It's not a big deal to me at all.  I think the person hashing out the
details, writing the docs, and doing the implementation gets to make
the call.