[Bug libfortran/30162] New: I/O with named pipes does not work

[Bug libfortran/30162] New: I/O with named pipes does not work

[jvdelisle at gcc dot gnu dot org]

Reduced test case frim Bud Davis

$ cat a.f
      integer status
      integer i
      open(unit=20,file='np',action='write',
     &     form='unformatted',iostat=status);
      print*,'status from open is ',status
      do i = 1,5
         write(20)i
      end do
      end



$ cat b.f
      integer status
      integer i
      integer val
      open(unit=21,file='np',action='read',
     &     form='unformatted',iostat=status);
      print*,'status from open is ',status
      do i = 1,5
         read(21)val
         print*,val
      end do



Here is what I get:

$ mkfifo np
$ ./b &
[1] 18730
$ ./a
 status from open is            0
At line 7 of file a.f
Fortran runtime error: Illegal seek
 status from open is            0
At line 8 of file b.f
Fortran runtime error: End of file


-- 
           Summary: I/O with named pipes does not work
           Product: gcc
           Version: 4.3.0
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: libfortran
        AssignedTo: unassigned at gcc dot gnu dot org
        ReportedBy: jvdelisle at gcc dot gnu dot org


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=30162

[Bug libfortran/30162] I/O with named pipes does not work

[tkoenig at gcc dot gnu dot org]

------- Comment #1 from tkoenig at gcc dot gnu dot org  2006-12-15 21:14 -------
For writing, I think this fails because we

- write a bogus record marker with value 0
- write out the data
- write out the trailing record marker
- seek to the first record marker
- write out its value
- seek past the end

These seeks actually translate to OS calls, which of course fails
for pipes.

For reading, there is another problem:  If I

- redirect the output from a to a file
- cat it into the named pipe
- read it using b

it gives a bogus "corrupt unformatted sequential file" error,
for a reason that I don't understand at the moment.

Ouch.  This one is nasty.


-- 

tkoenig at gcc dot gnu dot org changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |tkoenig at gcc dot gnu dot
                   |                            |org
             Status|UNCONFIRMED                 |NEW
     Ever Confirmed|0                           |1
   Last reconfirmed|0000-00-00 00:00:00         |2006-12-15 21:14:51
               date|                            |


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=30162

[Bug libfortran/30162] I/O with named pipes does not work

[tkoenig at gcc dot gnu dot org]

------- Comment #2 from tkoenig at gcc dot gnu dot org  2006-12-16 11:44 -------
(In reply to comment #1)

> These seeks actually translate to OS calls, which of course fails
> for pipes.

We could get by for writing if we

- eliminated the seeks as long as we don't actually go past our buffer
- flushed after every write statement if we have a non-seekable file
- set the subrecord size to the buffer length - 2*length of the record marker.


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=30162

[Bug libfortran/30162] I/O with named pipes does not work

[tkoenig at gcc dot gnu dot org]

-- 

tkoenig at gcc dot gnu dot org changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
         AssignedTo|unassigned at gcc dot gnu   |tkoenig at gcc dot gnu dot
                   |dot org                     |org
             Status|NEW                         |ASSIGNED
   Last reconfirmed|2006-12-15 21:14:51         |2006-12-16 21:34:27
               date|                            |


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=30162

[Bug libfortran/30162] I/O with named pipes does not work

[tkoenig at gcc dot gnu dot org]

------- Comment #3 from tkoenig at gcc dot gnu dot org  2006-12-28 21:22 -------
This is deep regression country; in the time
I have to devote to this, I couldn't work it out.

Unassigning myself (for now).


-- 

tkoenig at gcc dot gnu dot org changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
         AssignedTo|tkoenig at gcc dot gnu dot  |unassigned at gcc dot gnu
                   |org                         |dot org
             Status|ASSIGNED                    |NEW


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=30162

[Bug libfortran/30162] I/O with named pipes does not work

[tkoenig at gcc dot gnu dot org]

------- Comment #4 from tkoenig at gcc dot gnu dot org  2006-12-28 21:39 -------
Additional data points:

Reading/writing unformatted sequential doesn't work with g77
(fails with an error message on open), and it fails with ifort 8
for records longer than 2**18 bytes (ifort's default buffer size).

Because of the complications with record markers, this might be
much easier to implement for stream access.




-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=30162

[Bug libfortran/30162] I/O with named pipes does not work

[jvdelisle at gcc dot gnu dot org]

------- Comment #5 from jvdelisle at gcc dot gnu dot org  2006-12-28 23:11 -------
I will work at it.


-- 

jvdelisle at gcc dot gnu dot org changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
         AssignedTo|unassigned at gcc dot gnu   |jvdelisle at gcc dot gnu dot
                   |dot org                     |org
             Status|NEW                         |ASSIGNED
   Last reconfirmed|2006-12-16 21:34:27         |2006-12-28 23:11:10
               date|                            |


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=30162

[Bug libfortran/30162] I/O with named pipes does not work

[tkoenig at gcc dot gnu dot org]

------- Comment #6 from tkoenig at gcc dot gnu dot org  2006-12-29 09:51 -------
(In reply to comment #5)
> I will work at it.

Thanks, I'll be happy to assist with discussions and review.

(Those who can, fix; those who can't, review :-)


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=30162

[Bug libfortran/30162] I/O with named pipes does not work

[jvdelisle at gcc dot gnu dot org]

------- Comment #7 from jvdelisle at gcc dot gnu dot org  2007-01-01 04:58 -------
I have formatted named pipe I/O working, at least for the equivalent test cases
given here.


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=30162

[Bug libfortran/30162] I/O with named pipes does not work

[tkoenig at gcc dot gnu dot org]

------- Comment #8 from tkoenig at gcc dot gnu dot org  2007-01-01 15:17 -------
(In reply to comment #7)
> I have formatted named pipe I/O working, at least for the equivalent test cases
> given here.

Great!

If you want me to, I'll be willing to test your patch.

Thomas


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=30162

[Bug libfortran/30162] I/O with named pipes does not work

[jvdelisle at gcc dot gnu dot org]

------- Comment #9 from jvdelisle at gcc dot gnu dot org  2007-01-01 17:51 -------
Preliminary patch for formatted only.

Index: io/unix.c
===================================================================
*** io/unix.c   (revision 120301)
--- io/unix.c   (working copy)
*************** fd_flush (unix_stream * s)
*** 349,355 ****
    size_t writelen;

    if (s->ndirty == 0)
!     return SUCCESS;;

    if (s->physical_offset != s->dirty_offset &&
        lseek (s->fd, s->dirty_offset, SEEK_SET) < 0)
--- 349,358 ----
    size_t writelen;

    if (s->ndirty == 0)
!     return SUCCESS;
!   
!   if (s->file_length == -1)
!     return SUCCESS;

    if (s->physical_offset != s->dirty_offset &&
        lseek (s->fd, s->dirty_offset, SEEK_SET) < 0)
*************** fd_sfree (unix_stream * s)
*** 562,567 ****
--- 565,574 ----
  static try
  fd_seek (unix_stream * s, gfc_offset offset)
  {
+ 
+   if (s->file_length == -1)
+     return SUCCESS;
+ 
    if (s->physical_offset == offset) /* Are we lucky and avoid syscall?  */
      {
        s->logical_offset = offset;
*************** static try
*** 583,589 ****
  fd_truncate (unix_stream * s)
  {
    if (lseek (s->fd, s->logical_offset, SEEK_SET) == -1)
!     return FAILURE;

    /* non-seekable files, like terminals and fifo's fail the lseek.
       Using ftruncate on a seekable special file (like /dev/null)
--- 590,596 ----
  fd_truncate (unix_stream * s)
  {
    if (lseek (s->fd, s->logical_offset, SEEK_SET) == -1)
!     return SUCCESS;

    /* non-seekable files, like terminals and fifo's fail the lseek.
       Using ftruncate on a seekable special file (like /dev/null)
*************** fd_to_stream (int fd, int prot)
*** 1009,1015 ****
    /* Get the current length of the file. */

    fstat (fd, &statbuf);
!   s->file_length = S_ISREG (statbuf.st_mode) ? statbuf.st_size : -1;
    s->special_file = !S_ISREG (statbuf.st_mode);

    fd_open (s);
--- 1016,1027 ----
    /* Get the current length of the file. */

    fstat (fd, &statbuf);
! 
!   if (lseek (fd, 0, SEEK_CUR) == (off_t) -1)
!     s->file_length = -1;
!   else
!     s->file_length = S_ISREG (statbuf.st_mode) ? statbuf.st_size : -1;
! 
    s->special_file = !S_ISREG (statbuf.st_mode);

    fd_open (s);


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=30162

[Bug libfortran/30162] I/O with named pipes does not work

[tkoenig at gcc dot gnu dot org]

------- Comment #10 from tkoenig at gcc dot gnu dot org  2007-01-02 22:08 -------
(In reply to comment #9)

> Preliminary patch for formatted only.

Looks pretty good, at least the tests pass.


> Index: io/unix.c
> ===================================================================
> *** io/unix.c   (revision 120301)
> --- io/unix.c   (working copy)
> *************** fd_flush (unix_stream * s)
> *** 349,355 ****
>     size_t writelen;
> 
>     if (s->ndirty == 0)
> !     return SUCCESS;;
> 
>     if (s->physical_offset != s->dirty_offset &&
>         lseek (s->fd, s->dirty_offset, SEEK_SET) < 0)
> --- 349,358 ----
>     size_t writelen;
> 
>     if (s->ndirty == 0)
> !     return SUCCESS;
> !   
> !   if (s->file_length == -1)
> !     return SUCCESS;

I don't understand this part.  Shouldn't we flush the
buffer anyway, even without a seek?


>   fd_truncate (unix_stream * s)
>   {
>     if (lseek (s->fd, s->logical_offset, SEEK_SET) == -1)
> !     return SUCCESS;

A small matter, we should test for seekable files first:

      if (s->file_length == -1)
        return SUCCESS;

      if (lseek(s->fd, s->logical_offset, SEEK_SET) == -1)
        return FAILURE;

If this fails, something is really wrong.

As an aside, this already works for ACCESS="stream" and unformatted
files, so I think we should think about committing this.


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=30162

[Bug libfortran/30162] I/O with named pipes does not work

[patchapp at dberlin dot org]

------- Comment #11 from patchapp at dberlin dot org  2007-01-05 05:00 -------
Subject: Bug number PR30162

A patch for this bug has been added to the patch tracker.
The mailing list url for the patch is
http://gcc.gnu.org/ml/gcc-patches/2007-01/msg00325.html


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=30162

[Bug libfortran/30162] I/O with named pipes does not work

[jvdelisle at gcc dot gnu dot org]

------- Comment #12 from jvdelisle at gcc dot gnu dot org  2007-01-06 00:14 -------
Subject: Bug 30162

Author: jvdelisle
Date: Sat Jan  6 00:14:38 2007
New Revision: 120512

URL: http://gcc.gnu.org/viewcvs?root=gcc&view=rev&rev=120512
Log:
2007-01-05  Jerry DeLisle  <jvdelisle@gcc.gnu.org>

        PR libgfortran/30162
        * io/unix.c (fd_flush): Don't seek if file is not seekable, defined as
        s->file_length == -1.
        (fd_alloc_w_at): Do not adjust file_length if file is not seekable.
        (fd_seek): If not seekable, just return success.
        (fd_truncate): If not seekable, no need to truncate.  Return failure if
        seek fails and the stream is not a pipe.
        (fd_to_stream): Make test for non-seekable file more robust.

Modified:
    trunk/libgfortran/ChangeLog
    trunk/libgfortran/io/unix.c


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=30162

[Bug libfortran/30162] I/O with named pipes does not work

[jvdelisle at gcc dot gnu dot org]

------- Comment #13 from jvdelisle at gcc dot gnu dot org  2007-01-06 21:07 -------
A difficult case to deal with is:

program main
  print *,foo(0)
contains
  function foo (n) result(res)
    integer, intent(in) :: n
    integer, allocatable :: res(:)
    logical :: init = .false.

    if (.not. init) then
       allocate (res(1))
       init = .true.
    else
       allocate (res(2))
    end if

    res = n
  end function foo
end program main


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=30162

[Bug libfortran/30162] I/O with named pipes does not work

[tkoenig at gcc dot gnu dot org]

------- Comment #14 from tkoenig at gcc dot gnu dot org  2007-01-06 22:26 -------
Another difficult case is:

program main
  character(len=4) c
  c = 'ab  '
  write (10) trim(c)
end program main


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=30162

[Bug libfortran/30162] I/O with named pipes does not work

[jvdelisle at gcc dot gnu dot org]

------- Comment #15 from jvdelisle at gcc dot gnu dot org  2007-01-07 01:34 -------
Subject: Bug 30162

Author: jvdelisle
Date: Sun Jan  7 01:34:03 2007
New Revision: 120544

URL: http://gcc.gnu.org/viewcvs?root=gcc&view=rev&rev=120544
Log:
2007-01-06  Jerry DeLisle  <jvdelisle@gcc.gnu.org>

        PR libgfortran/30162
        * io/unix.c (fd_flush): Don't seek if file is not seekable, defined as
        s->file_length == -1.
        (fd_alloc_w_at): Do not adjust file_length if file is not seekable.
        (fd_seek): If not seekable, just return success.
        (fd_truncate): If not seekable, no need to truncate.  Return failure if
        seek fails and the stream is not a pipe.
        (fd_to_stream): Make test for non-seekable file more robust.

2007-01-06  Jerry DeLisle  <jvdelisle@gcc.gnu.org>

        * ChangeLog: Copied to ...
        * ChangeLog-2006: here.


Added:
    branches/gcc-4_2-branch/libgfortran/ChangeLog-2006
Modified:
    branches/gcc-4_2-branch/libgfortran/ChangeLog
    branches/gcc-4_2-branch/libgfortran/io/unix.c


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=30162

[Bug libfortran/30162] I/O with named pipes does not work

[jvdelisle at gcc dot gnu dot org]

-- 

jvdelisle at gcc dot gnu dot org changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
         AssignedTo|jvdelisle at gcc dot gnu dot|unassigned at gcc dot gnu
                   |org                         |dot org
             Status|ASSIGNED                    |NEW


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=30162

[Bug libfortran/30162] I/O with named pipes does not work

[burnus at gcc dot gnu dot org]

------- Comment #16 from burnus at gcc dot gnu dot org  2007-02-01 09:27 -------
Is this bug fixed or not? I see a 4.3 and a 4.2 check in.
Or is something missing, if yes, what?


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=30162

[Bug libfortran/30162] I/O with named pipes does not work

[jvdelisle at gcc dot gnu dot org]

------- Comment #17 from jvdelisle at gcc dot gnu dot org  2007-02-02 02:00 -------
We do not have this working for unformatted I/O.  It will require frontend work
as well as library to implement.  Specifically, the size of records, determined
from the output list, must be determined prior to performing transfers.

On the library side, once the record size is known, seeking backwards in named
pipes must be avoided.  Right now we seek backward to save the beginning record
marker after the length of the record is determined by writing the record. 
This backward seek is costly in performance.

I am convinced that these changes to frontend and library can be done.  I just
do not have time right now to pursue it.  I will be studying the problem in the
meantime, but I do not want to hold back anyone else from making the attempt,
thinking I may be working it.


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=30162

[Bug libfortran/30162] I/O with named pipes does not work

[dagurasu15 at hotmail dot com]

------- Comment #18 from dagurasu15 at hotmail dot com  2007-08-18 19:06 -------
I think this is the same bug.  If not let me know and I'll repost it.
This is a real bummer.  I use this to turn off noise from routines I don't have
control of (and I mean ALOT of noise).  Maybe there is another way to do what I
want though.  Anyway, the following produces a seek error on the write line. 
It worked well in f77.  Now using gcc-4.1.2

       program bug
        close(6)
        open(6,file='/dev/null',status='old')
        close(6)
c  would normally do some noisy stuff here 
c    but not needed to make the bug.
        open(6,file='/dev/stdout',status='old')
        write (*,*)"Hello"
       end


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=30162

[Bug libfortran/30162] I/O with named pipes does not work

[dagurasu15 at hotmail dot com]

------- Comment #19 from dagurasu15 at hotmail dot com  2007-08-18 19:08 -------
(In reply to comment #18)
> I think this is the same bug.  If not let me know and I'll repost it.
> This is a real bummer.  I use this to turn off noise from routines I don't have
> control of (and I mean ALOT of noise).  Maybe there is another way to do what I
> want though.  Anyway, the following produces a seek error on the write line. 
> It worked well in f77.  Now using gcc-4.1.2
> 
>        program bug
>         close(6)
>         open(6,file='/dev/null',status='old')
>         close(6)
> c  would normally do some noisy stuff here 
> c    but not needed to make the bug.
>         open(6,file='/dev/stdout',status='old')
>         write (*,*)"Hello"
>        end
> 

of course the comment should be one line higher.


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=30162

[Bug libfortran/30162] I/O with named pipes does not work

[jvdelisle at gcc dot gnu dot org]

------- Comment #20 from jvdelisle at gcc dot gnu dot org  2007-08-18 19:39 -------
I will try to have a look in the next few days.


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=30162

[Bug libfortran/30162] I/O with named pipes does not work

[jvdelisle at gcc dot gnu dot org]

------- Comment #21 from jvdelisle at gcc dot gnu dot org  2007-08-19 19:45 -------
This works on 4.2 and 4.3.  Not planning to backport to 4.1.  See if you can
get a later version of gfortran.  If you have problems doing so, check the
gfortran wiki binaries.  If that fails, let me know.


-- 

jvdelisle at gcc dot gnu dot org changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|                            |WONTFIX


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=30162

[Bug libfortran/30162] I/O with named pipes does not work

[burnus at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=30162

Tobias Burnus <burnus at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|RESOLVED                    |REOPENED
         Resolution|WONTFIX                     |

--- Comment #24 from Tobias Burnus <burnus at gcc dot gnu.org> 2012-08-06 13:43:05 UTC ---
(In reply to comment #23)
> The real bug is thus that we shouldn't try to seek or truncate the fd at all,
> as it's not necessary in this case.

I think the issue occurs for:

st_write_done (st_parameter_dt *dtp)
...
  if (dtp->u.p.current_unit != NULL
      && dtp->u.p.current_unit->flags.access == ACCESS_SEQUENTIAL)
    switch (dtp->u.p.current_unit->endfile)
...
      case NO_ENDFILE:
        /* Get rid of whatever is after this record.  */
        if (!is_internal_unit (dtp))
          unit_truncate (dtp->u.p.current_unit,
                         stell (dtp->u.p.current_unit->s),
                         &dtp->common);
        dtp->u.p.current_unit->endfile = AT_ENDFILE;

with a file as simple as

  open (99, file="foo.dat")
  write (99, '(a)') 'Hello world'
  end


$ mkfifo foo.dat
$ tail -f foo.dat &
[2] 342
$ gfortran test.f90 && ./a.out
At line 2 of file test.f90 (unit = 99, file = 'foo.dat')
Fortran runtime error: Invalid argument
Hello world
$ fg
tail -f foo.dat
^C

[Bug libfortran/30162] I/O with named pipes does not work

[jb at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=30162

--- Comment #23 from Janne Blomqvist <jb at gcc dot gnu.org> 2012-08-03 19:45:10 UTC ---
(In reply to comment #22)
> Revision 180701 removed all checks for special files in unit.c:unit_truncate().

Yes. As the email message introducing the patch explained, special files are
special in many ways, and trying to impose some kind of uniform behavior on
them is bound to fail in more or less surprising and inconvenient ways. For the
same reason, the code for handling special files were removed from the buffered
IO functions, and all non-regular files now just use the raw IO functions
instead. The only sane way is to seek/truncate only when the user code requires
such a behavior, and if that then fails, return an error and let the user
handle it.

The real bug is thus that we shouldn't try to seek or truncate the fd at all,
as it's not necessary in this case.

I recall that I tried some simple programs with named pipes when I implemented
the patch, but maybe some subsequent patch then broke it again.

[Bug libfortran/30162] I/O with named pipes does not work

[iliev at rz dot rwth-aachen.de]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=30162

Hristo Iliev <iliev at rz dot rwth-aachen.de> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |iliev at rz dot
                   |                            |rwth-aachen.de

--- Comment #22 from Hristo Iliev <iliev at rz dot rwth-aachen.de> 2012-08-02 17:34:03 UTC ---
Revision 180701 removed all checks for special files in unit.c:unit_truncate().
As a consequence programs compiled with gfortran 4.6.3 and newer cannot to
write (formatted) to Unix named pipes as ftruncate() fails with EINVAL:

(64-bit code)

lseek(3, 0, SEEK_CUR)                   = -1 ESPIPE (Illegal seek)
write(3, " Hello, world!\n", 15)        = 15
lseek(3, 0, SEEK_CUR)                   = -1 ESPIPE (Illegal seek)
ftruncate(3, 18446744073709551615)      = -1 EINVAL (Invalid argument)

(32-bit code)

_llseek(3, 0, 0xffba8fa0, SEEK_CUR)     = -1 ESPIPE (Illegal seek)
write(3, " Hello, world!\n", 15)        = 15
_llseek(3, 0, 0xffba9080, SEEK_CUR)     = -1 ESPIPE (Illegal seek)
ftruncate64(3, 18446744073709551615)    = -1 EINVAL (Invalid argument)