[patch, fortran, docs] Unformatted sequential and special files

[patch, fortran, docs] Unformatted sequential and special files

[Thomas Koenig]

[-- Attachment #1: Type: text/plain, Size: 389 bytes --]

Hello world,

the attached patch documents the format for unformatted sequential
files and what is, and is not, supported with special files.

Tested with "make dvi" and "make info".  OK for trunk?

	Thomas

2013-08-30  Thomas Koenig  <tkoenig@gcc.gnu.org>

        PR fortran/30162
        * gfortran.texi:  Document unformatted sequential file format
        and I/O with special files.

[-- Attachment #2: p1.diff --]
[-- Type: text/x-patch, Size: 3029 bytes --]

Index: gfortran.texi
===================================================================
--- gfortran.texi	(Revision 201996)
+++ gfortran.texi	(Arbeitskopie)
@@ -1121,6 +1121,8 @@
 * Internal representation of LOGICAL variables::
 * Thread-safety of the runtime library::
 * Data consistency and durability::
+* Unformatted sequential file format::
+* I/O with special files::
 @end menu
 
 
@@ -1291,7 +1293,75 @@
 releasing @code{fcntl} file locks, if the server supports them, will
 also force cache validation and flushing dirty data and metadata.
 
+@node Unformatted sequential file format
+@section Unformatted sequential file format
+@cindex unformatted sequential files
+@cindex record marker
+@cindex subrecord
 
+Unformatted sequential files are stored using record markers. Each
+full record consists of a leading record marker, the data written
+by the user program, and a trailing record marker.  The record markers
+are four-byte integers by default, and eight-byte integers if the
+@option{-fmax-subrecord-length=8} option is in effect. Each record
+marker contains the number of bytes of data in the record.
+
+The maximum number of bytes of user data in a record is 2147483639 for
+a four-byte record marker. If this is exceeded, a record is split into
+subrecords. Each subrecord also has a leading and a trailing record
+marker. If the leading record marker contains a negative number, the
+number of user data bytes in the subrecord equals the absolute value
+of this number, and another subrecord follows the current one.  If the
+trailing record marker contains a negative number, then the number of
+bytes of user data equals the absolute value of that number, and there
+is a preceding subrecord.
+
+The format for unformatted sequential data can be duplicated using
+unformatted stream, as shown in this example program:
+
+@smallexample
+program main
+  implicit none
+  integer :: i
+  real, dimension(10) :: a, b
+  call random_number(a)
+  open (10,file='test.dat',form='unformatted',access='stream')
+  inquire (iolength=i) a
+  write (10) i, a, i
+  close (10)
+  open (10,file='test.dat',form='unformatted')
+  read (10) b
+  if (all (a == b)) print *,'success!'
+end program main
+@end smallexample
+
+@node I/O with special files
+@section I/O with special files
+@cindex special files
+@cindex pipes
+@cindex FIFO
+@cindex terminal devices
+@cindex BACKSPACE
+
+Special files such as pipes, FIFOs or terminal devices are supported
+only for the following types of file access:
+
+@itemize
+
+@item Formatted sequential
+
+@item Formatted stream
+
+@item Unformatted stream
+
+@end itemize
+
+Unformatted sequential file access is not supported for
+special files.  If necessary, it can be simulated using
+unformatted stream, see @ref{Unformatted sequential file format}.
+
+@code{BACKSPACE} is not supported for special files.
+
 @c ---------------------------------------------------------------------
 @c Extensions
 @c ---------------------------------------------------------------------

Re: [patch, fortran, docs] Unformatted sequential and special files

[Janne Blomqvist]

On Fri, Aug 30, 2013 at 8:18 PM, Thomas Koenig <tkoenig@netcologne.de> wrote:
> Hello world,
>
> the attached patch documents the format for unformatted sequential
> files and what is, and is not, supported with special files.
>
> Tested with "make dvi" and "make info".  OK for trunk?

The unformatted sequential format part looks Ok, but the special files
section is lacking. E.g. what about

- The REWIND statement?
- The ENDFILE statement?
- ACCESS='stream' and the POS= specifier?
- ACCESS='direct'? (I suspect this should work for
pipes/FIFO's/terminals in case REC= numbers are sequential, but is
that a guarantee we want to make?)
- Special files which are special in other ways. E.g. block special
files tend to allow seeking, but IO must be block aligned.

>
>         Thomas
>
> 2013-08-30  Thomas Koenig  <tkoenig@gcc.gnu.org>
>
>         PR fortran/30162
>         * gfortran.texi:  Document unformatted sequential file format
>         and I/O with special files.



-- 
Janne Blomqvist

Re: [patch, fortran, docs] Unformatted sequential and special files

[Thomas Koenig]

[-- Attachment #1: Type: text/plain, Size: 1021 bytes --]

Hi Janne,

I have tried to answer your points in the attached patch.

> The unformatted sequential format part looks Ok, but the special files
> section is lacking. E.g. what about
> 
> - The REWIND statement?

Not supported.

> - The ENDFILE statement?

Not supported.

> - ACCESS='stream' and the POS= specifier?

Only for inpquire.

> - ACCESS='direct'? (I suspect this should work for
> pipes/FIFO's/terminals in case REC= numbers are sequential, but is
> that a guarantee we want to make?)

I have not listed this as supported in the patch.

> - Special files which are special in other ways. E.g. block special
> files tend to allow seeking, but IO must be block aligned.

Also listed as not supported; I suspect buffering could cause grief
there.

I have updated an attached patch.  OK?

	Thomas

2013-08-30  Thomas Koenig  <tkoenig@gcc.gnu.org>

        PR fortran/30162
        * gfortran.texi:  Document unformatted sequential file format
        and I/O with special files.
ig25@linux-fd1f:~/Krempel/Unformatt

[-- Attachment #2: p2.diff --]
[-- Type: text/x-patch, Size: 3332 bytes --]

Index: gfortran.texi
===================================================================
--- gfortran.texi	(Revision 201996)
+++ gfortran.texi	(Arbeitskopie)
@@ -1121,6 +1121,8 @@
 * Internal representation of LOGICAL variables::
 * Thread-safety of the runtime library::
 * Data consistency and durability::
+* Unformatted sequential file format::
+* I/O with special files::
 @end menu
 
 
@@ -1291,7 +1293,85 @@
 releasing @code{fcntl} file locks, if the server supports them, will
 also force cache validation and flushing dirty data and metadata.
 
+@node Unformatted sequential file format
+@section Unformatted sequential file format
+@cindex unformatted sequential files
+@cindex record marker
+@cindex subrecord
 
+Unformatted sequential files are stored using record markers. Each
+full record consists of a leading record marker, the data written
+by the user program, and a trailing record marker.  The record markers
+are four-byte integers by default, and eight-byte integers if the
+@option{-fmax-subrecord-length=8} option is in effect. Each record
+marker contains the number of bytes of data in the record.
+
+The maximum number of bytes of user data in a record is 2147483639 for
+a four-byte record marker. If this is exceeded, a record is split into
+subrecords. Each subrecord also has a leading and a trailing record
+marker. If the leading record marker contains a negative number, the
+number of user data bytes in the subrecord equals the absolute value
+of this number, and another subrecord follows the current one.  If the
+trailing record marker contains a negative number, then the number of
+bytes of user data equals the absolute value of that number, and there
+is a preceding subrecord.
+
+The format for unformatted sequential data can be duplicated using
+unformatted stream, as shown in this example program:
+
+@smallexample
+program main
+  implicit none
+  integer :: i
+  real, dimension(10) :: a, b
+  call random_number(a)
+  open (10,file='test.dat',form='unformatted',access='stream')
+  inquire (iolength=i) a
+  write (10) i, a, i
+  close (10)
+  open (10,file='test.dat',form='unformatted')
+  read (10) b
+  if (all (a == b)) print *,'success!'
+end program main
+@end smallexample
+
+@node I/O with special files
+@section I/O with special files
+@cindex special files
+@cindex pipes
+@cindex FIFO
+@cindex terminal devices
+@cindex block devices
+@cindex sockets
+@cindex BACKSPACE
+@cindex REWIND
+@cindex ENDFILE
+
+Special character-oriented files such as pipes, FIFOs or terminal
+devices are supported only for the following types of file access:
+
+@itemize
+
+@item Formatted sequential
+
+@item Formatted stream
+
+@item Unformatted stream
+
+@end itemize
+
+For special files, the @code{POS=} specifier for stream I/O can only
+be used in @code{INQUIRE} statements.
+
+Unformatted sequential file access is @emph{not} supported for special
+files.  If necessary, it can be simulated using unformatted stream,
+see @ref{Unformatted sequential file format}.
+
+I/O to and from block devices are also not supported.
+
+@code{BACKSPACE}, @code{REWIND} and @code{ENDFILE} are not supported
+for special files.
+
 @c ---------------------------------------------------------------------
 @c Extensions
 @c ---------------------------------------------------------------------

Re: [patch, fortran, docs] Unformatted sequential and special files

[Tobias Burnus]

Thomas Koenig wrote:
> +Unformatted sequential files are stored using record markers. Each
> +full record consists of a leading record marker, the data written
> +by the user program, and a trailing record marker.  The record markers
> +are four-byte integers by default, and eight-byte integers if the
> +@option{-fmax-subrecord-length=8} option is in effect. Each record
> +marker contains the number of bytes of data in the record.

I wonder whether one should discourage the use of 
-fmax-subrecord-length=8 more explicitly here - when reading until here 
one might think that there is a 2GB limit for =4.

> +The maximum number of bytes of user data in a record is 2147483639 for
> +a four-byte record marker.

Why this number? I had expected it to be exactly  2 GiB (minus 1 Byte) = 
2**31-1 = 2147483647. Your number is one byte shorter. Why?

Actually, I had also mentioned GiB as those are simpler to remember, e.g.
"The maximal user data per record is 2 gigabytes (2147483647/2147483639 
bytes) for the four-byte record marker.

> If this is exceeded, a record is split into
> +subrecords. Each subrecord also has a leading and a trailing record
> +marker. If the leading record marker contains a negative number, the
> +number of user data bytes in the subrecord equals the absolute value
> +of this number, and another subrecord follows the current one.  If the
> +trailing record marker contains a negative number, then the number of
> +bytes of user data equals the absolute value of that number, and there
> +is a preceding subrecord.
> +
> +The format for unformatted sequential data can be duplicated using
> +unformatted stream, as shown in this example program:

(which assumes records shorter than 2 GiB)

> +
> +@smallexample
> +program main
> +  implicit none
> +  integer :: i

I wonder whether one should make this more explicit by using

use iso_fortran_env, only: int32
integer(int32) :: i

> +Unformatted sequential file access is @emph{not} supported for special
> +files.  If necessary, it can be simulated using unformatted stream,
> +see @ref{Unformatted sequential file format}.
> +
> +I/O to and from block devices are also not supported.
> +
> +@code{BACKSPACE}, @code{REWIND} and @code{ENDFILE} are not supported
> +for special files.

I think I understand what you mean but  "I/O to and from block devices 
are also not supported." sounds as if it also could apply to all I/O, 
including stream I/O.

How about adding "block devices" to the list of special files in the 
intro? You could also mention sockets, which behave similarly. The 
BACKSPACE/REWIND/ENDFILE could be moved directly after the itemize in 
the same paragraph as the POS=.

And instead of "unformatted sequential", you could write "unformatted 
sequential and directed access", which reminds me that we should also 
document that file format for the sake of completeness.

Tobias

Re: [patch, fortran, docs] Unformatted sequential and special files

[Thomas Koenig]

[-- Attachment #1: Type: text/plain, Size: 350 bytes --]

Hello world,

here is a rewrite, which I hope make things more clear.
Unformatted sequential files are now made up of subrecords, where
a logical record may only have one.

Regarding block devices: I don't know anybody who ever used them
from gfortran, so I tried to be as vague as possible.

Any more suggestions?  OK for trunk otherwise?

	Thomas


[-- Attachment #2: p3.diff --]
[-- Type: text/x-patch, Size: 4146 bytes --]

Index: gfortran.texi
===================================================================
--- gfortran.texi	(Revision 201996)
+++ gfortran.texi	(Arbeitskopie)
@@ -1121,6 +1121,8 @@
 * Internal representation of LOGICAL variables::
 * Thread-safety of the runtime library::
 * Data consistency and durability::
+* Unformatted sequential file format::
+* I/O with special files::
 @end menu
 
 
@@ -1291,7 +1293,109 @@
 releasing @code{fcntl} file locks, if the server supports them, will
 also force cache validation and flushing dirty data and metadata.
 
+@node Unformatted sequential file format
+@section Unformatted sequential file format
+@cindex unformatted sequential
+@cindex sequential, unformatted
+@cindex record marker
+@cindex subrecord
 
+Unformatted sequential files are stored as logical records using
+record markers.  Each logical record consists of one of more subrecords.
+
+Each subrecord consists of a leading record marker, the data written
+by the user program, and a trailing record marker.  The record markers
+are four-byte integers by default, and eight-byte integers if the
+@option{-fmax-subrecord-length=8} option (which exists for backwards
+compability reasons only) is in effect.
+
+The maximum number of bytes of user data in a subrecord is 2147483639
+(2 GiB - 9) for a four-byte record marker.  If a logical record
+contains more data, the data is distributed among several subrecords.
+
+The absolute of the number stored in the record markers is the number
+of bytes of user data in the corresponding subrecord.  If the leading
+record marker of a subrecord contains a negative number, another
+subrecord follows the current one.  If the trailing record marker
+contains a negative number, then there is a preceding subrecord.
+
+In the most simple case, with only one subrecord per logical record,
+both record markers contain the number of bytes of user data in the
+record,
+
+The format for unformatted sequential data can be duplicated using
+unformatted stream, as shown in the example program for a single
+subrecord only:
+
+@smallexample
+program main
+  use iso_fortran_env, only: int32
+  implicit none
+  integer(int32) :: i 
+  real, dimension(10) :: a, b
+  call random_number(a)
+  open (10,file='test.dat',form='unformatted',access='stream')
+  inquire (iolength=i) a
+  write (10) i, a, i
+  close (10)
+  open (10,file='test.dat',form='unformatted')
+  read (10) b
+  if (all (a == b)) print *,'success!'
+end program main
+@end smallexample
+
+@node I/O with special files
+@section I/O with special files
+@cindex special files
+@cindex character devices
+@cindex devices, character
+@cindex block devices
+@cindex devices, block
+@cindex sockets
+@cindex special files, character
+@cindex BACKSPACE
+@cindex REWIND
+@cindex ENDFILE
+@cindex pipes
+@cindex FIFO
+@cindex terminal devices
+@cindex unformatted sequential
+@cindex sequential, unformatted
+@cindex unformatted stream
+@cindex stream, unformatted
+@cindex formatted stream
+@cindex stream, formatted
+@cindex direct access
+
+Special character files such as pipes, FIFOs, sockets or terminal
+devices are supported only for the following types of file access:
+
+@itemize
+
+@item Formatted sequential
+
+@item Formatted stream
+
+@item Unformatted stream
+
+@end itemize
+
+For special character files, the @code{POS=} specifier for stream I/O
+can only be used in @code{INQUIRE} statements.
+
+@code{BACKSPACE}, @code{REWIND} and @code{ENDFILE} are not supported
+for character special files.
+
+Unformatted sequential and direct file access are @emph{not} supported
+for character special files.  If necessary, unformatted sequential
+access can be simulated using unformatted stream, see @ref{Unformatted
+sequential file format}.
+
+Block devices have not been tested.  It is likely that only
+unformatted stream and direct access will work for those.  Some
+restrictions specific to the operating system regarding sizes and
+alignment of data may apply.
+
 @c ---------------------------------------------------------------------
 @c Extensions
 @c ---------------------------------------------------------------------

Re: [patch, fortran, docs] Unformatted sequential and special files

[Janne Blomqvist]

On Tue, Sep 3, 2013 at 3:04 PM, Thomas Koenig <tkoenig@netcologne.de> wrote:
> Hello world,
>
> here is a rewrite, which I hope make things more clear.
> Unformatted sequential files are now made up of subrecords, where
> a logical record may only have one.

Looks ok.

> Regarding block devices: I don't know anybody who ever used them
> from gfortran, so I tried to be as vague as possible.
>
> Any more suggestions?  OK for trunk otherwise?

I'm still not comfortable with the wording. As I've argued before,
special files are special in different ways, and can behave
differently on different systems, so it's difficult to say anything
definitive about their behavior in general. Maybe being more explicit
about what is supported for a limited subset would help. E.g. starting
the section with something like

"For terminal devices, pipes, FIFO's and sockets the following file
modes are supported:
- ...

For other special files and other file modes, the result is undefined."

("undefined" rather than "not supported", as we're not going out of
our way to prevent it if somebody wants to do it either)

- Wrt the POS= specifier with INQUIRE, even it it "works" (as in, does
not generate an error), there is no sensible concept of file position
for a stream file anyway, so perhaps we shouldn't explicitly say it's
supported either.

- Wrt. block devices, perhaps remove that section and cover it just
with the "...implementation defined" sentence above.



-- 
Janne Blomqvist

Re: [patch, fortran, docs] Unformatted sequential and special files

[Tobias Burnus]

Thomas, what's actually the status of this patch? I think it was 
half-complete in September.

Tobias

On September 4, 2013 12:40, Janne Blomqvist wrote:
> On Tue, Sep 3, 2013 at 3:04 PM, Thomas Koenig <tkoenig@netcologne.de> wrote:
>> Hello world,
>>
>> here is a rewrite, which I hope make things more clear.
>> Unformatted sequential files are now made up of subrecords, where
>> a logical record may only have one.
> Looks ok.
>
>> Regarding block devices: I don't know anybody who ever used them
>> from gfortran, so I tried to be as vague as possible.
>>
>> Any more suggestions?  OK for trunk otherwise?
> I'm still not comfortable with the wording. As I've argued before,
> special files are special in different ways, and can behave
> differently on different systems, so it's difficult to say anything
> definitive about their behavior in general. Maybe being more explicit
> about what is supported for a limited subset would help. E.g. starting
> the section with something like
>
> "For terminal devices, pipes, FIFO's and sockets the following file
> modes are supported:
> - ...
>
> For other special files and other file modes, the result is undefined."
>
> ("undefined" rather than "not supported", as we're not going out of
> our way to prevent it if somebody wants to do it either)
>
> - Wrt the POS= specifier with INQUIRE, even it it "works" (as in, does
> not generate an error), there is no sensible concept of file position
> for a stream file anyway, so perhaps we shouldn't explicitly say it's
> supported either.
>
> - Wrt. block devices, perhaps remove that section and cover it just
> with the "...implementation defined" sentence above.
>
>
>