Consistent mode of file-opens

Consistent mode of file-opens

[Kimon Kontovassilis]

Hello, fellows!

Although it's several months I 've joined the list, this is my first
stand-up-and-talk time; so, I guess, before anything else, I should send
a couple of sincere THANK-YOU words both to the CYGNUS people 
and to the many persons on this list that keep on improving gnu-win32.

The purpose of this message is the expression of my worries on the 
consistency of binary/text modes of opened files. Extensive (maybe too
extensive) discussions have taken place on this subject and I definitely
wouldn't want this message to regurgitate things already said.
Thus, I consider all past discussions on default binary vs text translations
an already known background and I 'm going to build on the (apparently now
permanently adopted) proposition that the default file-open mode is determined
by the mount-table flags in the registry.

Consider now a system setup where all mount points have been flagged as
binary. (As previous discussions on this list have fully explained, such a
setting is virtually necessary, if binary-file-oriented applications are to
behave properly without being individually modified.)
The problem is that stdin, stdout and stderr are still opened in text mode.
This breaks redirection and pipe operations dealing with binary data.
Things like

	gunzip -c file.tar.gz | tar xvf -
or
	tr < binfile -d some-funny-char | split
or
	cat pieces of a big bin file > wholefile

do no work properly, as is well known to most participants in this list.
I find this a major flaw: redirections and pipes should be transparent to
the operation of programs (and this includes programs operating on binary
files).

Why don't std{in,out,err} obey the default translation mode? Well, I haven't
heard of an explicit reason for this, but I suppose a good reason is that
these streams are by default connected to the console, and the console is
inherently text-oriented. The default keyboard drivers forward lines ended
by <cr>/<lf> and ^Z is valuable for signifying EOF from the keyboard.
Furthermore, the default console driver requires <cr>/<lf> pairs to
properly display lines on the screen.

So, in effect, opening in text mode is the right thing to do, as long as
the standard streams are connected to the console. However, if these
streams are redirected, the streams should be opened in binary mode, if the
rest of file opens have been setup so.

There is some time that it has occurred to me that the right mode is not
hard to find out, given the function isatty(). Try it out for yourselves;
save this little program:

	#include <stdio.h>
	#include <string.h>
	#include <unistd.h>
	
	void check_fd(int fd, FILE *ofp)
	{
		if (isatty(fd))
			fprintf(ofp, "fd %d IS a tty:\t%s\n", fd, ttyname(fd));
		else
			fprintf(ofp, "fd %d is NOT a tty\n", fd);
	}
	
	int main(int argc, char *argv[])
	{
	 	/* where to report */
		FILE *ofp =
			(argc > 1 && strcmp(argv[1], "-1") == 0)
			? stdout : stderr;
	
		check_fd(0, ofp);
		check_fd(1, ofp);
		check_fd(2, ofp);
	}

compile it to produce, say, test4tty, then try:

	test4tty
	test4tty > file
	test4tty < file
	echo | test4tty
	test4tty | cat

with same results in all of COMMAND.COM, 4DOS and bash. In bash you may
further wish to try things like

	test4tty < /dev/null
	test4tty -1 2>file
	test4tty < /dev/null 2>&1 | cat
	echo `test4tty`

and you will see that isatty() does a pretty good job. Similarly, things
work consistently with popen()/pclose().

So, this brings up the following question: couldn't the start-up code
(maybe just before handing over to main, or at any other appropriate point)
do things like:

	if (!isatty(0))
		_setmode(0, O_BINARY);

(and similarly for descriptors 1 and 2) if the default mode for (say) the root
drive is binary? Or, if you don't like the "root" drive convention, add
provision for a new registry key, governing the standard streams when they are
!isatty(), this registry key being settable by some extended notation in the
mount command.

It seems so obvious to me that I 'm quite surprised that no-one else saw it
before me. The feasibility of my solution, however, is purely speculative,
as I haven't access to the source code.
Maybe I 'm foolishly ignorant of some other major issue that prevents this
solution :-) If so, I would definitely want to hear about it.

Best regards,
Kimon
-----------------
Dr. Kimon Kontovasilis,
Institute for Informatics & Telecommunications,
National Center for Scientific Research Demokritos,
GR-15310 AG. PARASKEVI ATTIKIS, PoB 60228, Greece
-
For help on using this list (especially unsubscribing), send a message to
"gnu-win32-request@cygnus.com" with one line of text: "help".

Re: Consistent mode of file-opens

[Kimon Kontovasilis]

> >From: kkont@estia.iit.nrcps.ariadne-t.gr (Kimon Kontovasilis)
> >Subject: Re: Consistent mode of file-opens
> >To: earnie_boyd@hotmail.com (Earnie Boyd)
> >Date: Thu, 9 Oct 1997 17:09:16 +0200 (EET)
> >Cc: gnu-win32@cygnus.com
> ---snip---
> >I guess this renders obsolete most of my original message. However, 
> since
> >all individual utilities work adequately well with the other 
> dos-oriented
> >shells and since all the shells that I know of support redirection and
> >(simulated) pipes, it would be nice if the proper file mode selection 
> worked
> >on these shells too.
> >
> 
> I believe that the redirection file modes are controlled by the shells 
> themselves.
> 
> -        \\||//
> ---o0O0--Earnie--0O0o----
> -earnie_boyd@hotmail.com-
> ------ooo0O--O0ooo-------
> 

Well, that's not true, at least for the "standard" MS-DOS based shells.
The following are happening on a drive mounted -b (so that e.g., od reports
all bytes in files). The shell in use is 4DOS.COM (same results occur
with COMMAND.COM), on a Win95 machine.

c:\tmp\gcc-trials> cat chkdos.c
#include <stdio.h>
#include <string.h>
#include <fcntl.h>

int main(int argc, char *argv[])
{
	char *test_str = "A test line\nAnother line\n";

	if (argc > 1 && strcmp(argv[1], "-b") == 0)
		_setmode(1, O_BINARY);
	printf("%s", test_str);
	exit(0);
}

c:\tmp\gcc-trials> gcc chkdos.c -o chkdos

c:\tmp\gcc-trials> chkdos 
A test line
Another line

c:\tmp\gcc-trials> chkdos -b
A test line
           Another line

c:\tmp\gcc-trials> chkdos > a

c:\tmp\gcc-trials> od -c a
0000000   A       t   e   s   t       l   i   n   e  \r  \n   A   n   o
0000020   t   h   e   r       l   i   n   e  \r  \n
0000033

c:\tmp\gcc-trials> chkdos -b > a

c:\tmp\gcc-trials> od -c a
0000000   A       t   e   s   t       l   i   n   e  \n   A   n   o   t
0000020   h   e   r       l   i   n   e  \n
0000031

The above show that 4DOS and COMMAND do not alter the file mode when
redirecting. In contrast, as you remarked, bash does change things, so as
to assure the right mode.

c:\tmp\gcc-trials> bash -c "chkdos"
A test line
Another line

c:\tmp\gcc-trials> bash -c "chkdos -b"
A test line
           Another line

c:\tmp\gcc-trials> bash -c "chkdos > a; od -c a"
0000000   A       t   e   s   t       l   i   n   e  \n   A   n   o   t
0000020   h   e   r       l   i   n   e  \n
0000031

c:\tmp\gcc-trials> bash -c "chkdos -b > a; od -c a"
0000000   A       t   e   s   t       l   i   n   e  \n   A   n   o   t
0000020   h   e   r       l   i   n   e  \n
0000031

It appeards, that the DOS-based shells take an approach similar to globbing:
"let the program do it itself". Ugly and silly, I know, but a fact of life.
That's why I proposed the startup code hack, in an attempt to try to remedy the
problem in the same way it is done with globbing.

Best regards,

Kimon Kontovasilis,
NCSR "Demokritos"
-
For help on using this list (especially unsubscribing), send a message to
"gnu-win32-request@cygnus.com" with one line of text: "help".

Re: Consistent mode of file-opens

[Larry Hall]

At 06:30 AM 10/9/97 PDT, Earnie Boyd wrote:
>
>
>>From: kkont@estia.iit.nrcps.ariadne-t.gr (Kimon Kontovassilis)
>>Subject: Consistent mode of file-opens
>>To: gnu-win32@cygnus.com
>>Date: Wed, 8 Oct 1997 18:31:19 +0200 (EET)
>>Cc: kkont@estia.iit.nrcps.ariadne-t.gr (Kimon Kontovassilis)
>>
>>Hello, fellows!
>>
>---snip---
>>
>>Consider now a system setup where all mount points have been flagged as
>>binary. (As previous discussions on this list have fully explained, 
>such a
>>setting is virtually necessary, if binary-file-oriented applications 
>are to
>>behave properly without being individually modified.)
>>The problem is that stdin, stdout and stderr are still opened in text 
>mode.
>>This breaks redirection and pipe operations dealing with binary data.
>>Things like
>>
>>	gunzip -c file.tar.gz | tar xvf -
>
>You must be executing from the MSDosshell.  I've done this sort of 
>operation many times both on WNT3.51 and W95 within BASH without any 
>problems.
>

Nope, Kimon is right on this count.  Pipes, stdin, stdout, and stderr are
"text" mode in gnu-win32 regardless of the mounting option.  A code change 
is necessary to modify this.  I forget exactly where this needs to be done
but its either in bash or winsup code....



Larry

-
For help on using this list (especially unsubscribing), send a message to
"gnu-win32-request@cygnus.com" with one line of text: "help".

Re: Consistent mode of file-opens

[Kimon Kontovasilis]

> 
> Hi Kimon, you wrote:
> 
> : default file-open mode is determined by the mount-table flags in the registry.
> 
> : The problem is that stdin, stdout and stderr are still opened in text mode.
> : This breaks redirection and pipe operations dealing with binary data.
> 
> What about a mount flag in the registry for pipes, too?
> 
> 
> Bye, Heribert (dahms@ifk20.mach.uni-karlsruhe.de)
> 

As you might already have seen in the reply of Earnie Boyd to my original
mail, things work correctly under bash, so in this environment nothing
more needs to be done. Things are different with the "standard" MS-DOS
based shells. For these, the scheme I proposed in my original post,
as ammended in my reply to Earnie's comment, may still be applicable.

Best regards,

Kimon Kontovasilis,
NCSR "Demokritos"
-
For help on using this list (especially unsubscribing), send a message to
"gnu-win32-request@cygnus.com" with one line of text: "help".

Re: Consistent mode of file-opens

[Kimon Kontovasilis]

Dear Earnie,

> 
> >From: kkont@estia.iit.nrcps.ariadne-t.gr (Kimon Kontovassilis)
> >Subject: Consistent mode of file-opens
> >To: gnu-win32@cygnus.com
> >Date: Wed, 8 Oct 1997 18:31:19 +0200 (EET)
> >Cc: kkont@estia.iit.nrcps.ariadne-t.gr (Kimon Kontovassilis)
> >
> >Hello, fellows!
> >
> ---snip---
> >
> >Consider now a system setup where all mount points have been flagged as
> >binary. (As previous discussions on this list have fully explained, 
> such a
> >setting is virtually necessary, if binary-file-oriented applications 
> are to
> >behave properly without being individually modified.)
> >The problem is that stdin, stdout and stderr are still opened in text 
> mode.
> >This breaks redirection and pipe operations dealing with binary data.
> >Things like
> >
> >	gunzip -c file.tar.gz | tar xvf -
> 
> You must be executing from the MSDosshell.  I've done this sort of 
> operation many times both on WNT3.51 and W95 within BASH without any 
> problems.
> 
> -        \\||//
> ---o0O0--Earnie--0O0o----
> -earnie_boyd@hotmail.com-
> ------ooo0O--O0ooo-------
> 

you 're indeed true. Thanks for your hint. I just checked and indeed verified
that redirections and pipes, from within bash, indeed behave properly.
(I must have been remarkably stupid not to try the test in the first place.)

I guess this renders obsolete most of my original message. However, since
all individual utilities work adequately well with the other dos-oriented
shells and since all the shells that I know of support redirection and
(simulated) pipes, it would be nice if the proper file mode selection worked
on these shells too.

Could it be that the start up code would check for a set PID variable
and, if not, perform the isatty()/_setmode() trick I was talking about?
(If I recall correctly, the PID check-trick is the mechanism used for invoking
the globber, when a gnuwin32-based app. runs outside of bash.)

Thanks again for your remark.

Kimon Kontovasilis,
NCSR "Demokritos".
-
For help on using this list (especially unsubscribing), send a message to
"gnu-win32-request@cygnus.com" with one line of text: "help".

RE: Consistent mode of file-opens

[dahms]

Hi Kimon, you wrote:

: default file-open mode is determined by the mount-table flags in the registry.

: The problem is that stdin, stdout and stderr are still opened in text mode.
: This breaks redirection and pipe operations dealing with binary data.

What about a mount flag in the registry for pipes, too?


Bye, Heribert (dahms@ifk20.mach.uni-karlsruhe.de)
-
For help on using this list (especially unsubscribing), send a message to
"gnu-win32-request@cygnus.com" with one line of text: "help".

Re: Consistent mode of file-opens

[Earnie Boyd]

>From: kkont@estia.iit.nrcps.ariadne-t.gr (Kimon Kontovassilis)
>Subject: Consistent mode of file-opens
>To: gnu-win32@cygnus.com
>Date: Wed, 8 Oct 1997 18:31:19 +0200 (EET)
>Cc: kkont@estia.iit.nrcps.ariadne-t.gr (Kimon Kontovassilis)
>
>Hello, fellows!
>
---snip---
>
>Consider now a system setup where all mount points have been flagged as
>binary. (As previous discussions on this list have fully explained, 
such a
>setting is virtually necessary, if binary-file-oriented applications 
are to
>behave properly without being individually modified.)
>The problem is that stdin, stdout and stderr are still opened in text 
mode.
>This breaks redirection and pipe operations dealing with binary data.
>Things like
>
>	gunzip -c file.tar.gz | tar xvf -

You must be executing from the MSDosshell.  I've done this sort of 
operation many times both on WNT3.51 and W95 within BASH without any 
problems.

-        \\||//
---o0O0--Earnie--0O0o----
-earnie_boyd@hotmail.com-
------ooo0O--O0ooo-------


______________________________________________________
Get Your Private, Free Email at http://www.hotmail.com
-
For help on using this list (especially unsubscribing), send a message to
"gnu-win32-request@cygnus.com" with one line of text: "help".

Re: Consistent mode of file-opens

[Earnie Boyd]

>From: kkont@estia.iit.nrcps.ariadne-t.gr (Kimon Kontovasilis)
>Subject: Re: Consistent mode of file-opens
>To: earnie_boyd@hotmail.com (Earnie Boyd)
>Date: Thu, 9 Oct 1997 17:09:16 +0200 (EET)
>Cc: gnu-win32@cygnus.com
---snip---
>I guess this renders obsolete most of my original message. However, 
since
>all individual utilities work adequately well with the other 
dos-oriented
>shells and since all the shells that I know of support redirection and
>(simulated) pipes, it would be nice if the proper file mode selection 
worked
>on these shells too.
>

I believe that the redirection file modes are controlled by the shells 
themselves.

-        \\||//
---o0O0--Earnie--0O0o----
-earnie_boyd@hotmail.com-
------ooo0O--O0ooo-------


______________________________________________________
Get Your Private, Free Email at http://www.hotmail.com
-
For help on using this list (especially unsubscribing), send a message to
"gnu-win32-request@cygnus.com" with one line of text: "help".