[Bug libfortran/30009] New: Unformatted reads exceeding one record should use the next record not give EOF

[Bug libfortran/30009] New: Unformatted reads exceeding one record should use the next record not give EOF

[burnus at gcc dot gnu dot org]

See http://gcc.gnu.org/ml/fortran/2006-11/msg00708.html
Reported by Chris Talley.

Tested with current 4.3 and also with 4.1.

Testcase, see URL; essential parts of the test case (form="unformatted"):
       WRITE(1) 1
       WRITE(1) 2, 3, 4
       WRITE(1) 5, 6, 7
       REWIND(1)
       READ(1,ERR=10) I,J,K
The read should read "1" of the first record and then "2" and "3" of the second
record. gfortran however reads "1", hits then the end of the record and gives
an end-of-file error.

Note that none of the end-of-file conditions given in 9.10 of Fortran 2003 (cf.
also 9.4.1.5f. of Fortran 95) is met. Note further that an end-of-record
condition cannot occure for unformatted files.


-- 
           Summary: Unformatted reads exceeding one record should use the
                    next record not give EOF
           Product: gcc
           Version: 4.3.0
            Status: UNCONFIRMED
          Keywords: wrong-code
          Severity: normal
          Priority: P3
         Component: libfortran
        AssignedTo: unassigned at gcc dot gnu dot org
        ReportedBy: burnus at gcc dot gnu dot org


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=30009

[Bug libfortran/30009] Unformatted reads exceeding one record should use the next record not give EOF

[brooks at gcc dot gnu dot org]

-- 

brooks at gcc dot gnu dot org changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|UNCONFIRMED                 |NEW
     Ever Confirmed|0                           |1
      Known to fail|                            |4.2.0 4.3.0
   Last reconfirmed|0000-00-00 00:00:00         |2006-11-28 21:49:15
               date|                            |


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=30009

[Bug libfortran/30009] Unformatted reads exceeding one record should use the next record not give EOF

[brooks at gcc dot gnu dot org]

------- Comment #1 from brooks at gcc dot gnu dot org  2006-11-28 21:49 -------
*** Bug 30010 has been marked as a duplicate of this bug. ***


-- 

brooks at gcc dot gnu dot org changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |brooks at gcc dot gnu dot
                   |                            |org


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=30009

[Bug libfortran/30009] Unformatted reads exceeding one record should use the next record not give EOF

[tkoenig at gcc dot gnu dot org]

------- Comment #2 from tkoenig at gcc dot gnu dot org  2006-11-28 22:23 -------
(In reply to comment #0)
> See http://gcc.gnu.org/ml/fortran/2006-11/msg00708.html
> Reported by Chris Talley.
> 
> Tested with current 4.3 and also with 4.1.
> 
> Testcase, see URL; essential parts of the test case (form="unformatted"):
>        WRITE(1) 1
>        WRITE(1) 2, 3, 4
>        WRITE(1) 5, 6, 7
>        REWIND(1)
>        READ(1,ERR=10) I,J,K
> The read should read "1" of the first record and then "2" and "3" of the second
> record.

I don't think so.

F 2003, 9.5.3.4.1 (Unformatted data transfer):

# On input from a file connected for sequential or direct access,
# the number of file storage units required by the input list shall
# be less than or equal to the number of file storage units in the record.

This is not a constraint, so the burden is on the programmer.
In other words, the program is non-conforming.  We are definitely
not required to read past the end of the current record.

> gfortran however reads "1", hits then the end of the record and gives
> an end-of-file error.

End of file is misleading, true.  In the "Intel subrecord" patch
(PR 29568), an error condition is raised for this case.  I have

Changing keyword to "diagnostic", and assigning to myself.


-- 

tkoenig at gcc dot gnu dot org changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |tkoenig at gcc dot gnu dot
                   |                            |org
  BugsThisDependsOn|                            |29568
         AssignedTo|unassigned at gcc dot gnu   |tkoenig at gcc dot gnu dot
                   |dot org                     |org
             Status|NEW                         |ASSIGNED
           Keywords|wrong-code                  |diagnostic
   Last reconfirmed|2006-11-28 21:49:15         |2006-11-28 22:23:23
               date|                            |


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=30009

[Bug libfortran/30009] Unformatted reads exceeding storage units gives EOF instead of ERR

[burnus at gcc dot gnu dot org]

------- Comment #3 from burnus at gcc dot gnu dot org  2006-11-28 22:58 -------
(In reply to comment #2)
> > The read should read "1" of the first record and then "2" and "3" of the
> > second record.
> I don't think so.

You are of cause right. I should have really read

> F 2003, 9.5.3.4.1 (Unformatted data transfer):

more carefully (I did twice and missed that part). The bug is really that if
one reads too much data END= and not ERR= is triggered.

A quick test with ifort shows that it outputs the message "input statement
requires too much data" if one removes ERR=.


-- 

burnus at gcc dot gnu dot org changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
      Known to fail|4.2.0 4.3.0                 |4.1.0 4.2.0 4.3.0
            Summary|Unformatted reads exceeding |Unformatted reads exceeding
                   |one record should use the   |storage units gives EOF
                   |next record not give EOF    |instead of ERR


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=30009

[Bug libfortran/30009] Unformatted reads exceeding storage units gives EOF instead of ERR

[tkoenig at gcc dot gnu dot org]

------- Comment #4 from tkoenig at gcc dot gnu dot org  2006-12-02 21:56 -------
g77 gets this right:

$ cat eor2.f
      program eor
      WRITE(1) 1
      REWIND(1)
      READ(1,ERR=10) I,J,K
      print *,"no error"
      stop
 10   print *,"error value"
      end
$ g77 eor2.f && ./a.out
 error value


-- 

tkoenig at gcc dot gnu dot org changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
OtherBugsDependingO|                            |19292
              nThis|                            |


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=30009

[Bug libfortran/30009] Unformatted reads exceeding storage units gives EOF instead of ERR

[jvdelisle at gcc dot gnu dot org]

------- Comment #5 from jvdelisle at gcc dot gnu dot org  2006-12-02 23:31 -------
Please note the subtle difference with this case while working this:

       WRITE(1) 1
       REWIND(1)
       READ(1,ERR=10) I
       READ(1,END=10) J
       print *,"no error"
       stop
 10    print *,"error value"
       end

This is a true EOF condition and gfortran gets that right.  Ifort and gfortran
agree on this behavior as well.

Also note that EOF and EOR are not the same as ERR.  ERR would imply something
more serious then EOF or EOR which would be considered normal things to
encounter, like reading until you find the end of a file and then stopping as
opposed to a bad sector on a disk.


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=30009

[Bug libfortran/30009] Unformatted reads exceeding storage units gives EOF instead of ERR

[burnus at gcc dot gnu dot org]

------- Comment #6 from burnus at gcc dot gnu dot org  2006-12-03 08:32 -------
For
       READ(1,ERR=10) J ! Read beyond EOF
there are two possible implementations one finds:
  ifort: forrtl: severe (24): end-of-file during read
  gfortran: Fortran runtime error: End of file
  g95: Internal Error: EOF condition not handled-- END= tag needed
  sunf95: End-of-file:  -1
or
  NAG f95: error value
  g77:  error value


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=30009

[Bug libfortran/30009] Unformatted reads exceeding storage units gives EOF instead of ERR

[tkoenig at gcc dot gnu dot org]

------- Comment #7 from tkoenig at gcc dot gnu dot org  2006-12-03 14:06 -------
(In reply to comment #6)
> For
>        READ(1,ERR=10) J ! Read beyond EOF
> there are two possible implementations one finds:
>   ifort: forrtl: severe (24): end-of-file during read

Really?  With ifort 8.1, I get

$ cat end-of-file.f
       WRITE(1) 1
       REWIND(1)
       READ(1) I
       READ(1,END=10) J
       print *,"no error"
       stop
 10    print *,"error value"
       end

$ ifort -v && ifort end-of-file.f && ./a.out
Version 8.1
 error value

I do think this is the correct behavior, and I'll take care
to maintain this behavior.


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=30009

[Bug libfortran/30009] Unformatted reads exceeding storage units gives EOF instead of ERR

[burnus at gcc dot gnu dot org]

------- Comment #8 from burnus at gcc dot gnu dot org  2006-12-03 14:17 -------
> >        READ(1,ERR=10) J ! Read beyond EOF
> >   ifort: forrtl: severe (24): end-of-file during read
> 
> Really?
Yes, really. Note: END /= ERR in the two examples.

> With ifort 8.1, I get
>        READ(1,END=10) J
>  error value


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=30009

[Bug libfortran/30009] Unformatted reads exceeding storage units gives EOF instead of ERR

[tkoenig at gcc dot gnu dot org]

------- Comment #9 from tkoenig at gcc dot gnu dot org  2006-12-03 18:56 -------
I've looked at the F 2003 standard, and at least there
the wording is clear:

9.10:

# An end-of-file condition occurs in the following cases:
# 
# (1) When an endfile record is encountered during reading of a file
#     for sequential access.

This is not the case here, when we try to read too many items from a record.

# (2) When an attempt is made to read beyond the end of an internal file.
# 
# (3) When an attempt is made to read beyond the end of a stream file.

So we need an error condition here.


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=30009

[Bug libfortran/30009] Unformatted reads exceeding storage units gives EOF instead of ERR

[tkoenig at gcc dot gnu dot org]

------- Comment #10 from tkoenig at gcc dot gnu dot org  2006-12-03 21:02 -------
Created an attachment (id=12734)
 --> (http://gcc.gnu.org/bugzilla/attachment.cgi?id=12734&action=view)
first attempt at a patch

This should fix this PR, and PR 30056 as well.


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=30009

[Bug libfortran/30009] Unformatted reads exceeding storage units gives EOF instead of ERR

[jvdelisle at gcc dot gnu dot org]

------- Comment #11 from jvdelisle at gcc dot gnu dot org  2006-12-04 02:18 -------
Patch reviewed and tested.  Looks good to go.  I assume you tested the part of
the corrupted file somehow.  I suppose we could right a test case uses stream
I/O to build a bogus file and then try to read it and confirm that error.  That
could be another testcase.


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=30009

[Bug libfortran/30009] Unformatted reads exceeding storage units gives EOF instead of ERR

[patchapp at dberlin dot org]

------- Comment #12 from patchapp at dberlin dot org  2006-12-05 19:01 -------
Subject: Bug number PR 30009

A patch for this bug has been added to the patch tracker.
The mailing list url for the patch is
http://gcc.gnu.org/ml/gcc-patches/2006-12/msg00334.html


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=30009

[Bug libfortran/30009] Unformatted reads exceeding storage units gives EOF instead of ERR

[tkoenig at gcc dot gnu dot org]

------- Comment #13 from tkoenig at gcc dot gnu dot org  2006-12-06 19:26 -------
Subject: Bug 30009

Author: tkoenig
Date: Wed Dec  6 19:25:44 2006
New Revision: 119592

URL: http://gcc.gnu.org/viewcvs?root=gcc&view=rev&rev=119592
Log:
2006-12-06  Thomas Koenig  <Thomas.Koenig@online.de>

        PR libfortran/30009
        PR libfortran/30056
        * gfortran.dg/read_eof_4.f90:  Add tests.
        * gfortran.dg/readwrite_unf_direct_eor_1.f90:  New test.
        * gfortran.dg/unf_read_corrupted_1.f90: New test.

2006-12-06  Thomas Koenig  <Thomas.Koenig@online.de>

        PR libfortran/30009
        PR libfortran/30056
        * libgfortran.h:  Add ERROR_CORRUPT_FILE to error_codes.
        * runtime/error.c (translate_error):  Add handling for
        ERROR_CORRUPT_FILE.
        * io/transfer.c (read_block_direct):  Add comment about
        EOR for stream files.
        Remove test for no bytes left for direct access files.
        Generate an ERROR_SHORT_RECORD if the read was short.
        For unformatted sequential files:  Check endfile condition.
        Remove test for no bytes left.  End of file here means
        that the file structure has been corrupted.  Pre-position
        the file for the next record in case of error.
        (write_buf):  Whitespace fix.  Subtract the number of bytes
        written from bytes_left.


Added:
    trunk/gcc/testsuite/gfortran.dg/readwrite_unf_direct_eor_1.f90
    trunk/gcc/testsuite/gfortran.dg/unf_read_corrupted_1.f90
Modified:
    trunk/gcc/testsuite/ChangeLog
    trunk/gcc/testsuite/gfortran.dg/read_eof_4.f90
    trunk/libgfortran/ChangeLog
    trunk/libgfortran/io/transfer.c
    trunk/libgfortran/libgfortran.h
    trunk/libgfortran/runtime/error.c


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=30009

[Bug libfortran/30009] Unformatted reads exceeding storage units gives EOF instead of ERR

[tkoenig at gcc dot gnu dot org]

------- Comment #14 from tkoenig at gcc dot gnu dot org  2006-12-10 22:16 -------
Subject: Bug 30009

Author: tkoenig
Date: Sun Dec 10 22:16:14 2006
New Revision: 119712

URL: http://gcc.gnu.org/viewcvs?root=gcc&view=rev&rev=119712
Log:
2006-12-10  Thomas Koenig  <Thomas.Koenig@online.de>

        Backport from mainline
        PR libfortran/29568
        * gfortran.dg/convert_implied_open.f90:  Change to
        new default record length.
        * gfortran.dg/unf_short_record_1.f90:  Adapt to
        new error message.
        * gfortran.dg/unformatted_subrecords_1.f90:  New test.
        PR libfortran/30009
        PR libfortran/30056
        * gfortran.dg/read_eof_4.f90:  Add tests.
        * gfortran.dg/readwrite_unf_direct_eor_1.f90:  New test.
        * gfortran.dg/unf_read_corrupted_1.f90: New test.

2006-12-10  Thomas Koenig  <Thomas.Koenig@online.de>

        PR libfortran/29568
        * gfortran.h (gfc_option_t):  Add max_subrecord_length.
        (top level): Define MAX_SUBRECORD_LENGTH.
        * lang.opt:  Add option -fmax-subrecord-length=.
        * trans-decl.c:  Add new function set_max_subrecord_length.
        (gfc_generate_function_code): If we are within the main
        program and max_subrecord_length has been set, call
        set_max_subrecord_length.
        * options.c (gfc_init_options):  Add defaults for
        max_subrecord_lenght, convert and record_marker.
        (gfc_handle_option):  Add handling for
        -fmax_subrecord_length.
        * invoke.texi:  Document the new default for
        -frecord-marker=<n>.

2006-12-10  Thomas Koenig  <Thomas.Koenig@online.de>

        PR libfortran/29568
        * libgfortran/libgfortran.h (compile_options_t):  Add
        record_marker. (top level):  Define GFC_MAX_SUBRECORD_LENGTH.
        * runtime/compile_options.c (set_record_marker):  Change
        default to four-byte record marker.
        (set_max_subrecord_length):  New function.
        * runtime/error.c (translate_error):  Change error message
        for short record on unformatted read.
        * io/io.h (gfc_unit):  Add recl_subrecord, bytes_left_subrecord
        and continued.
        * io/file_pos.c (unformatted_backspace):  Change default of record
        marker size to four bytes.  Loop over subrecords.
        * io/open.c:  Default recl is max_offset.  If
        compile_options.max_subrecord_length has been set, set set
        u->recl_subrecord to its value, to the maximum value otherwise.
        * io/transfer.c (top level):  Add prototypes for us_read, us_write,
        next_record_r_unf and next_record_w_unf.
        (read_block_direct):  Separate codepaths for unformatted direct
        and unformatted sequential.  If a recl has been set by the
        user, use the number of bytes left for the record if it is smaller
        than the read request.  Loop over subrecords.  Set an error if the
        user has set a recl and the read was short.
        (write_buf):  Separate codepaths for unformatted direct and
        unformatted sequential. If a recl has been set by the
        user, use the number of bytes left for the record if it is smaller
        than the read request.  Loop over subrecords.  Set an error if the
        user has set a recl and the read was short.
        (us_read):  Add parameter continued (to indicate that bytes_left
        should not be intialized).  Change default of record marker size
        to four bytes. Use subrecord.  If the subrecord length is smaller than
        zero, this indicates a continuation.
        (us_write):  Add parameter continued (to indicate that the continued
        flag should be set).  Use subrecord.
        (pre_position):  Use 0 for continued on us_write and us_read calls.
        (skip_record):  New function.
        (next_record_r_unf):  New function.
        (next_record_r):  Use next_record_r_unf.
        (write_us_marker):  Default size for record markers is four bytes.
        (next_record_w_unf):  New function.
        (next_record_w):  Use next_record_w_unf.
        PR libfortran/30009
        PR libfortran/30056
        * libgfortran.h:  Add ERROR_CORRUPT_FILE to error_codes.
        * runtime/error.c (translate_error):  Add handling for
        ERROR_CORRUPT_FILE.
        * io/transfer.c (read_block_direct):  Add comment about
        EOR for stream files.
        Remove test for no bytes left for direct access files.
        Generate an ERROR_SHORT_RECORD if the read was short.
        For unformatted sequential files:  Check endfile condition.
        Remove test for no bytes left.  End of file here means
        that the file structure has been corrupted.  Pre-position
        the file for the next record in case of error.
        (write_buf):  Whitespace fix.  Subtract the number of bytes
        written from bytes_left.


Added:
   
branches/gcc-4_2-branch/gcc/testsuite/gfortran.dg/readwrite_unf_direct_eor_1.f90
    branches/gcc-4_2-branch/gcc/testsuite/gfortran.dg/unf_read_corrupted_1.f90
   
branches/gcc-4_2-branch/gcc/testsuite/gfortran.dg/unformatted_subrecord_1.f90
Modified:
    branches/gcc-4_2-branch/gcc/fortran/ChangeLog
    branches/gcc-4_2-branch/gcc/fortran/gfortran.h
    branches/gcc-4_2-branch/gcc/fortran/lang.opt
    branches/gcc-4_2-branch/gcc/fortran/options.c
    branches/gcc-4_2-branch/gcc/fortran/trans-decl.c
    branches/gcc-4_2-branch/gcc/testsuite/ChangeLog
    branches/gcc-4_2-branch/gcc/testsuite/gfortran.dg/convert_implied_open.f90
    branches/gcc-4_2-branch/gcc/testsuite/gfortran.dg/read_eof_4.f90
    branches/gcc-4_2-branch/gcc/testsuite/gfortran.dg/unf_short_record_1.f90
    branches/gcc-4_2-branch/libgfortran/ChangeLog
    branches/gcc-4_2-branch/libgfortran/io/file_pos.c
    branches/gcc-4_2-branch/libgfortran/io/io.h
    branches/gcc-4_2-branch/libgfortran/io/open.c
    branches/gcc-4_2-branch/libgfortran/io/transfer.c
    branches/gcc-4_2-branch/libgfortran/libgfortran.h
    branches/gcc-4_2-branch/libgfortran/runtime/compile_options.c
    branches/gcc-4_2-branch/libgfortran/runtime/error.c


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=30009

[Bug libfortran/30009] [4.1 only] Unformatted reads exceeding storage units gives EOF instead of ERR

[tkoenig at gcc dot gnu dot org]

------- Comment #15 from tkoenig at gcc dot gnu dot org  2006-12-15 22:26 -------
I have had a look at this for 4.1, and the effort
to backport this patch and the one for PR 30056 will
be fairly large.

Because 4.1 has neither stream I/O nor the subrecod
patch, fixing would require a substantial rewrite
of the patch.

Comments?  Does anybody feel this is important?


-- 

tkoenig at gcc dot gnu dot org changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
      Known to fail|4.1.0 4.2.0 4.3.0           |4.1.0
      Known to work|                            |4.2.0 4.3.0
            Summary|Unformatted reads exceeding |[4.1 only] Unformatted reads
                   |storage units gives EOF     |exceeding storage units
                   |instead of ERR              |gives EOF instead of ERR


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=30009

[Bug libfortran/30009] [4.1 only] Unformatted reads exceeding storage units gives EOF instead of ERR

[jvdelisle at gcc dot gnu dot org]

------- Comment #16 from jvdelisle at gcc dot gnu dot org  2006-12-16 00:34 -------
I think I would not risk potential further breakage and leave 4.1 alone.  4.2
will be coming along soon enough.


-- 

jvdelisle at gcc dot gnu dot org changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
         AssignedTo|tkoenig at gcc dot gnu dot  |unassigned at gcc dot gnu
                   |org                         |dot org
             Status|ASSIGNED                    |NEW


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=30009

[Bug libfortran/30009] [4.1 only] Unformatted reads exceeding storage units gives EOF instead of ERR

[tkoenig at gcc dot gnu dot org]

------- Comment #17 from tkoenig at gcc dot gnu dot org  2006-12-16 08:18 -------
I'm leaving this open for now, but
unassigning myself.  If anybody wants
to tackle this for 4.1, feel free.


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=30009

[Bug libfortran/30009] [4.1 only] Unformatted reads exceeding storage units gives EOF instead of ERR

[jvdelisle at gcc dot gnu dot org]

------- Comment #18 from jvdelisle at gcc dot gnu dot org  2007-01-02 04:48 -------
Closing, we need to focus resources on current trunk if we can.


-- 

jvdelisle at gcc dot gnu dot org changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|                            |FIXED
   Target Milestone|---                         |4.2.0


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=30009