[Bug fortran/109358] New: Wrong formatting with T-descriptor during stream output

[Bug fortran/109358] New: Wrong formatting with T-descriptor during stream output

[baradi09 at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=109358

            Bug ID: 109358
           Summary: Wrong formatting with T-descriptor during stream
                    output
           Product: gcc
           Version: 12.2.0
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: fortran
          Assignee: unassigned at gcc dot gnu.org
          Reporter: baradi09 at gmail dot com
  Target Milestone: ---

This is a very minor bug, but probably easy to fix. It seems, that the T data
edit descriptor handling is non-standard conforming when using formatted stream
I/O:

Given the following MWE:

program tabformat
  implicit none

  integer :: fd

  open(newunit=fd, file="test.stream.txt", access="stream", form="formatted")  
  write(fd, "(a)") "1234567890123"
  write(fd, "(a, t10, a)") "1234", "0123"  ! A-descriptor, file positioning
allowed                                 
  write(fd, "(i4, t10, i4.4)") 1234, 0123  !  I-descriptor, file positioning
not allowed                               
  close(fd)

end program tabformat

The resulting file contains

1234567890123
1234         0123   # 9 spaces between blocks
1234         0123   # 9 spaces between blocks

Apparently, a file positioning takes place during the execution of the write
statement. 

However, if I understand 13.7.1 §1 correctly, file positioning may only happen
with the A-descriptor (and it is optional even there). So the standard
conforming output would be either (if file positioning happens after the
A-descriptor)

1234567890123
1234         0123    # 9 spaces
1234     0123        # 5 spaces

or (if no file positioning happens after the A-descriptor)

1234567890123
1234     0123        # 5 spaces
1234     0123        # 5 spaces

I personally would prefer latter, and it would be also equivalent to the
behavior of the Intel and NAG compilers.

[Bug fortran/109358] Wrong formatting with T-descriptor during stream output

[jvdelisle at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=109358

Jerry DeLisle <jvdelisle at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |jvdelisle at gcc dot gnu.org

--- Comment #1 from Jerry DeLisle <jvdelisle at gcc dot gnu.org> ---
I will have to investigate further. I was involved in implementing a lot of
this.

[Bug fortran/109358] Wrong formatting with T-descriptor during stream output

[jvdelisle at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=109358

Jerry DeLisle <jvdelisle at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|UNCONFIRMED                 |NEW
     Ever confirmed|0                           |1
   Last reconfirmed|                            |2023-04-06

--- Comment #2 from Jerry DeLisle <jvdelisle at gcc dot gnu.org> ---
I am using my copy of the F2018 standard.

"13.8.1.2 The Tn edit descriptor indicates that the transmission of the next
character to or from a record is to occur at the nth character position of the
record, relative to the left tab limit. This position can be in either
direction from the current position."

Currently we do this:

1234567890123
1234         0123
1234         0123

We should do this:

1234567890123
1234     0123
1234     0123

"13.5 (3) During formatted stream output, processing of an A edit descriptor
can cause file positioning to occur (13.7.4)."

I am not finding anything that says file positioning is not allowed.  The above
is just elaborating different ways that an A edit descriptor affects the
positioning.  Positioning is always done no matter what with the T specifier.

Regardless, I do need to fix this.

[Bug fortran/109358] Wrong formatting with T-descriptor during stream output

[kargl at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=109358

kargl at gcc dot gnu.org changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |kargl at gcc dot gnu.org

--- Comment #3 from kargl at gcc dot gnu.org ---
(In reply to Jerry DeLisle from comment #2)
> I am using my copy of the F2018 standard.
> 
> "13.8.1.2 The Tn edit descriptor indicates that the transmission of the next
> character to or from a record is to occur at the nth character position of
> the record, relative to the left tab limit. This position can be in either
> direction from the current position."
> 
> Currently we do this:
> 
> 1234567890123
> 1234         0123
> 1234         0123
> 
> We should do this:
> 
> 1234567890123
> 1234     0123
> 1234     0123
> 
> "13.5 (3) During formatted stream output, processing of an A edit descriptor
> can cause file positioning to occur (13.7.4)."
> 
> I am not finding anything that says file positioning is not allowed.  The
> above is just elaborating different ways that an A edit descriptor affects
> the positioning.  Positioning is always done no matter what with the T
> specifier.
> 
> Regardless, I do need to fix this.

Yes, there is a bug.  The A edit descriptor and whether or not it does
file positioning is not relevant.  The relevant words from F2018 are

   13.8.1.2 T, TL, and TR editing

   The left tab limit affects file positioning by the T and TL edit
   descriptors. Immediately prior to nonchild data transfer (12.6.4.8.3),
   the left tab limit becomes defined as the character position of the
   current record or the current position of the stream file.  If, during
   data transfer, the file is positioned to another record, the left tab
   limit becomes defined as character position one of that record.

with

  write(fd, "(a)") "1234567890123"
  write(fd, "(a, t10, a)") "1234", "0123"  ! A-descriptor, file positioning 

the left tab limit for the 2nd write is established before data
transfer to position 1.  The 'T10' edit descriptor is relative to
that position.  The write of "1234" in the 2nd write might place
the character position to 5, but it does not update the left tab
limit because the write() is not positioned to a new record.

[Bug fortran/109358] Wrong formatting with T-descriptor during stream output

[jvdelisle at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=109358

--- Comment #4 from Jerry DeLisle <jvdelisle at gcc dot gnu.org> ---
Well this is getting quite interesting. There is a bit of discussion going on
the Fortran Discourse about this.

https://fortran-lang.discourse.group/t/tab-formatting-with-stream-access/5466/47

After thinking about this a lot and going back in my mind to the beginning, we
had no concept of a "STREAM" file having a "record". In fact we never even try
to track where the end of a record may be. So my thoughts are is when doing
formatted "STREAM" writes I can introduce a variable in the gfc_unit structure
to keep track in the stream where the end of the last "record" occurred. Now by
"record" this would be either when a /n or /n/r ocurred.  You can think of a
complication where someone just decides to write out a /n or a /n/r explicitly
not using NEWLINE and not using the implicit EOR that happens with every
formatted write statement.

So I begin to believe this is a conceptual error in the standard. The fact that
there is such discussion about it implies that it is a conceptual error.

Regardless, I think I can handle the implicit EOR that occurs and track this,
but I do not want to waste my time with explicit things. Why? The real purpose
of STREAM was suppose to be, in my mind, a way to write a binary stream
irrespective of formatting.

(sigh)

More as I proceed.

[Bug fortran/109358] Wrong formatting with T-descriptor during stream output

[baradi09 at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=109358

--- Comment #5 from Bálint Aradi <baradi09 at gmail dot com> ---
I also think that by allowing for explicit EORs caused by achar(10) characters
in the variable being written or by explicit new_line() calls, the standard
made the formatted stream output probably more complicated then it is worth of.
And the fact that apparently none of the widely used compilers handle those
cases correctly, also indicates over-complication.

However, I still think, that formatted stream output has its justification, as
it allows you to become independent from record length limits, which are set up
when a file is opened and which can not be adapted afterwards. We had been hit
by run-time errors a few times, when the character variable being written
turned out to be too big. (Of course, one could do unformatted stream output
instead, but then one always have to think about adding the newlines manually
at the end of each line...)

[Bug fortran/109358] Wrong formatting with T-descriptor during stream output

[jvdelisle at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=109358

Jerry DeLisle <jvdelisle at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |ASSIGNED

--- Comment #6 from Jerry DeLisle <jvdelisle at gcc dot gnu.org> ---
I have been doing some additional checking and tried this. It is a variation
which is not STREAM related:

program tabs
  implicit none
  integer :: fd
  open(newunit=fd, file="test.txt", form="formatted")
  write(fd, "(a)") "12345678901234567890123456789"
  write(fd, "(i4, t25, t2, i4.4)") 1234, 0123
  close(fd)
end program tabs

With gfortran gcc 10.4.1 20230121 (GCC)

$ gfc10 -static -o tabs.x tabs.f90
$ ./tabs.x 
$ cat test.txt 
12345678901234567890123456789
1234                0123

It seems to me this should be:

10123

Am I wrong?

[Bug fortran/109358] Wrong formatting with T-descriptor during stream output

[sgk at troutmask dot apl.washington.edu]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=109358

--- Comment #7 from Steve Kargl <sgk at troutmask dot apl.washington.edu> ---
On Fri, Jun 02, 2023 at 01:51:02AM +0000, jvdelisle at gcc dot gnu.org wrote:
> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=109358
> 
> Jerry DeLisle <jvdelisle at gcc dot gnu.org> changed:
> 
>            What    |Removed                     |Added
> ----------------------------------------------------------------------------
>              Status|NEW                         |ASSIGNED
> 
> --- Comment #6 from Jerry DeLisle <jvdelisle at gcc dot gnu.org> ---
> I have been doing some additional checking and tried this. It is a variation
> which is not STREAM related:
> 
> program tabs
>   implicit none
>   integer :: fd
>   open(newunit=fd, file="test.txt", form="formatted")
>   write(fd, "(a)") "12345678901234567890123456789"
>   write(fd, "(i4, t25, t2, i4.4)") 1234, 0123
>   close(fd)
> end program tabs
> 
> With gfortran gcc 10.4.1 20230121 (GCC)
> 
> $ gfc10 -static -o tabs.x tabs.f90
> $ ./tabs.x 
> $ cat test.txt 
> 12345678901234567890123456789
> 1234                0123
> 
> It seems to me this should be:
> 
> 10123
> 
> Am I wrong?
> 

I think you're right.  t25 will position you 25 spaces
from the left tab limit and then you get repositioned
with t2 by 2 spaces from the left tab limit.

[Bug fortran/109358] Wrong formatting with T-descriptor during stream output

[jvdelisle at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=109358

Jerry DeLisle <jvdelisle at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |nmm1 at cam dot ac.uk

--- Comment #8 from Jerry DeLisle <jvdelisle at gcc dot gnu.org> ---
*** Bug 53962 has been marked as a duplicate of this bug. ***

[Bug fortran/109358] Wrong formatting with T-descriptor during stream output

[jvdelisle at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=109358

--- Comment #9 from Jerry DeLisle <jvdelisle at gcc dot gnu.org> ---
Created attachment 57365
  --> https://gcc.gnu.org/bugzilla/attachment.cgi?id=57365&action=edit
Preliminary patch

The attached patch fixes the stream I/O related tabbing. This regression tests
fine.  There is another side issue I discovered not related to stream I am
investigating still. I will also work up a test case stream issue.

[Bug fortran/109358] Wrong formatting with T-descriptor during stream output

[jvdelisle at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=109358

--- Comment #10 from Jerry DeLisle <jvdelisle at gcc dot gnu.org> ---
To clarify. The following is the remaining issue that is not related to stream
I/O:

> program tabs
>   implicit none
>   integer :: fd
>   open(newunit=fd, file="test.txt", form="formatted")
>   write(fd, "(a)") "12345678901234567890123456789"
>   write(fd, "(i4, t25, t2, i4.4)") 1234, 0123
>   close(fd)
> end program tabs

Posted earlier in comment #6.

[Bug fortran/109358] Wrong formatting with T-descriptor during stream output

[jvdelisle at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=109358

--- Comment #11 from Jerry DeLisle <jvdelisle at gcc dot gnu.org> ---
I am going to submit the attached patch to close this PR and open a new PR for
the lingering multiple tab edits in a row. The problem there is we have a
special case if we have different T, TL, and TR following each other, a
nonsensical situation. However, we should handle it.  It will require "looking
ahead" or "looking back" in the format statement to know that this is
happening.

[Bug fortran/109358] Wrong formatting with T-descriptor during stream output

[cvs-commit at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=109358

--- Comment #12 from GCC Commits <cvs-commit at gcc dot gnu.org> ---
The master branch has been updated by Jerry DeLisle <jvdelisle@gcc.gnu.org>:

https://gcc.gnu.org/g:153ce7a78edbe339fa0b1cd314dea0554f59faf0

commit r14-8944-g153ce7a78edbe339fa0b1cd314dea0554f59faf0
Author: Jerry DeLisle <jvdelisle@gcc.gnu.org>
Date:   Mon Feb 12 13:12:08 2024 -0800

    libgfortran: Adjust bytes_left and pos for access="STREAM".

    During tab edits, the pos (position) and bytes_used
    Variables were not being set correctly for stream I/O.
    Since stream I/O does not have 'real' records, the
    format buffer active length must be used instead of
    the record length variable.

           PR libgfortran/109358

    libgfortran/ChangeLog:

            * io/transfer.c (formatted_transfer_scalar_write): Adjust
            bytes_used and pos variable for stream access.

    gcc/testsuite/ChangeLog:

            * gfortran.dg/pr109358.f90: New test.

[Bug fortran/109358] Wrong formatting with T-descriptor during stream output

[cvs-commit at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=109358

--- Comment #13 from GCC Commits <cvs-commit at gcc dot gnu.org> ---
The releases/gcc-13 branch has been updated by Jerry DeLisle
<jvdelisle@gcc.gnu.org>:

https://gcc.gnu.org/g:d85a658402d5433ff223ce8f4e70174ed8a20428

commit r13-8322-gd85a658402d5433ff223ce8f4e70174ed8a20428
Author: Jerry DeLisle <jvdelisle@gcc.gnu.org>
Date:   Mon Feb 12 13:12:08 2024 -0800

    libgfortran: Adjust bytes_left and pos for access="STREAM".

    During tab edits, the pos (position) and bytes_used
    Variables were not being set correctly for stream I/O.
    Since stream I/O does not have 'real' records, the
    format buffer active length must be used instead of
    the record length variable.

           PR libgfortran/109358

    libgfortran/ChangeLog:

            * io/transfer.c (formatted_transfer_scalar_write): Adjust
            bytes_used and pos variable for stream access.

    gcc/testsuite/ChangeLog:

            * gfortran.dg/pr109358.f90: New test.

    (cherry picked from commit 153ce7a78edbe339fa0b1cd314dea0554f59faf0)

[Bug fortran/109358] Wrong formatting with T-descriptor during stream output

[jvdelisle at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=109358

Jerry DeLisle <jvdelisle at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|ASSIGNED                    |RESOLVED
         Resolution|---                         |FIXED

--- Comment #14 from Jerry DeLisle <jvdelisle at gcc dot gnu.org> ---
Fixed on trunk and 13

[Bug fortran/109358] Wrong formatting with T-descriptor during stream output

[anlauf at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=109358

anlauf at gcc dot gnu.org changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |anlauf at gcc dot gnu.org
             Status|RESOLVED                    |REOPENED
         Resolution|FIXED                       |---

--- Comment #15 from anlauf at gcc dot gnu.org ---
Jerry, I tend to think the fix does not cover loopholes like multiple
tabbing or nX without printing anything in between.

Consider the enhanced test:

program tabs
  implicit none
  integer :: fd
  open(newunit=fd, file="test-stream.txt", form="formatted", access="stream")
  write(fd, "(a)") "12345678901234567890123456789"
  write(fd, "(i4, t25, t2, i4.4)") 1234, 0123             ! wrong
  write(fd, "(i4, t25, 1x, t2, i4.4)") 1234, 0123         ! wrong
  write(fd, "(i4, t21, a,  t2, i4.4)") 1234, "good", 0123 ! OK
  write(fd, "(i4, t40, i4, t20, i5.5)") 1234, 5555, 67890 ! OK
  close(fd)
  open(newunit=fd, file="test.txt", form="formatted")
  write(fd, "(a)") "12345678901234567890123456789"
  write(fd, "(i4, t25, t2, i4.4)") 1234, 0123             ! wrong
  write(fd, "(i4, t25, 1x, t2, i4.4)") 1234, 0123         ! wrong
  write(fd, "(i4, t21, a,  t2, i4.4)") 1234, "good", 0123 ! OK
  write(fd, "(i4, t40, i4, t20, i5.5)") 1234, 5555, 67890 ! OK
  close(fd)
end program tabs

Expected:

==> test-stream.txt <==
12345678901234567890123456789
10123
10123
10123               good
1234               67890               5555

==> test.txt <==
12345678901234567890123456789
10123
10123
10123               good
1234               67890               5555

With gcc-13/gcc-14 after your patch I get:

==> test-stream.txt <==
12345678901234567890123456789
1234                0123
1234                 0123
10123               good
1234               67890               5555

==> test.txt <==
12345678901234567890123456789
1234                0123
1234                 0123
10123               good
1234               67890               5555

Can you have a further look?

[Bug fortran/109358] Wrong formatting with T-descriptor during stream output

[jvdelisle at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=109358

--- Comment #16 from Jerry DeLisle <jvdelisle at gcc dot gnu.org> ---
I think I was not being very clear when I opened pr113897. nX descriptors are
very similar to TR code and I meeantt to take care of those in the 113897.  The
reason for the separate PR is that the problem is completely different than
this one which is relates to the use of stream I/O.

The consecutive tabs or spaces require looking ahead in the format string and
effectively squashing these before writing the next actual data. This was my
plan for PR113897.