textmode for stdout, what is "correct" now?

textmode for stdout, what is "correct" now?

[Michael Haubenwallner]

Hi,

so I find myself struggling with textmode versus binmode for stdio again.

Running the openssl command (from within the apps/ build directory here) does
yield different results regarding carriage return depending on the version:

$ ./apps/openssl version
OpenSSL 1.0.2p  14 Aug 2018
$ ./apps/openssl x509 -hash -noout -in /etc/pki/tls/cert.pem | xxd
00000000: 6139 3464 3039 6535 0a                   a94d09e5.


$ ./apps/openssl version
OpenSSL 1.1.0j  20 Nov 2018
$ ./apps/openssl x509 -hash -noout -in /etc/pki/tls/cert.pem | xxd
00000000: 6139 3464 3039 6535 0d0a                 a94d09e5..

Some subsequent shell script does create wrong symlink filenames
(with embedded CR) when used with openssl-1.1.x.

The commit that changed this behaviour in openssl-1.1 is:
https://github.com/openssl/openssl/commit/bdd58d98467e9f0f6635c1628e1eae304383afb1

From an openssl developer's point of view, I can understand to set
textmode when the intent is to output some text, and to set
binmode when the intent is to output some binary data.

Question now is: These days, what is the correct way to handle this?

Is it up to openssl to not use textmode at all with Cygwin (what if
used from within some cmd.exe?), or is it up to the shell script to
explicitly drop any carriage return here?

Thanks!
/haubi/

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: textmode for stdout, what is "correct" now?

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 1848 bytes --]

On Feb 14 16:23, Michael Haubenwallner wrote:
> Hi,
> 
> so I find myself struggling with textmode versus binmode for stdio again.
> 
> Running the openssl command (from within the apps/ build directory here) does
> yield different results regarding carriage return depending on the version:
> 
> $ ./apps/openssl version
> OpenSSL 1.0.2p  14 Aug 2018
> $ ./apps/openssl x509 -hash -noout -in /etc/pki/tls/cert.pem | xxd
> 00000000: 6139 3464 3039 6535 0a                   a94d09e5.
> 
> 
> $ ./apps/openssl version
> OpenSSL 1.1.0j  20 Nov 2018
> $ ./apps/openssl x509 -hash -noout -in /etc/pki/tls/cert.pem | xxd
> 00000000: 6139 3464 3039 6535 0d0a                 a94d09e5..
> 
> Some subsequent shell script does create wrong symlink filenames
> (with embedded CR) when used with openssl-1.1.x.
> 
> The commit that changed this behaviour in openssl-1.1 is:
> https://github.com/openssl/openssl/commit/bdd58d98467e9f0f6635c1628e1eae304383afb1
> 
> >From an openssl developer's point of view, I can understand to set
> textmode when the intent is to output some text, and to set
> binmode when the intent is to output some binary data.

How do you create \r\n in this case?  The upstream patch never
adds the explicit 't' flag.  It only adds 'b' or nothing.  So
the output should be \n only unless you write to a file on a
text mode mount.  What am I missing?

> Question now is: These days, what is the correct way to handle this?
> 
> Is it up to openssl to not use textmode at all with Cygwin (what if
> used from within some cmd.exe?), or is it up to the shell script to
> explicitly drop any carriage return here?

They way openssl does it looks right.  Off the top of my head I don't 
grok where you get the \r from in the example above.


Corinna

-- 
Corinna Vinschen
Cygwin Maintainer

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: textmode for stdout, what is "correct" now?

[Michael Haubenwallner]

On 2/14/19 5:20 PM, Corinna Vinschen wrote:
> On Feb 14 16:23, Michael Haubenwallner wrote:
>> Hi,
>>
>> so I find myself struggling with textmode versus binmode for stdio again.
>>
>> Running the openssl command (from within the apps/ build directory here) does
>> yield different results regarding carriage return depending on the version:
>>
>> $ ./apps/openssl version
>> OpenSSL 1.0.2p  14 Aug 2018
>> $ ./apps/openssl x509 -hash -noout -in /etc/pki/tls/cert.pem | xxd
>> 00000000: 6139 3464 3039 6535 0a                   a94d09e5.
>>
>>
>> $ ./apps/openssl version
>> OpenSSL 1.1.0j  20 Nov 2018
>> $ ./apps/openssl x509 -hash -noout -in /etc/pki/tls/cert.pem | xxd
>> 00000000: 6139 3464 3039 6535 0d0a                 a94d09e5..
>>
>> Some subsequent shell script does create wrong symlink filenames
>> (with embedded CR) when used with openssl-1.1.x.
>>
>> The commit that changed this behaviour in openssl-1.1 is:
>> https://github.com/openssl/openssl/commit/bdd58d98467e9f0f6635c1628e1eae304383afb1
>>
>> >From an openssl developer's point of view, I can understand to set
>> textmode when the intent is to output some text, and to set
>> binmode when the intent is to output some binary data.
> 
> How do you create \r\n in this case?  The upstream patch never
> adds the explicit 't' flag.  It only adds 'b' or nothing.  So
> the output should be \n only unless you write to a file on a
> text mode mount.  What am I missing?

Down the line in their BIO module they do use setmode(fd, O_TEXT),
which is the one that does introduce the \r, as far as I know.

The backtrace in openssl-1.1.1a in this use case is:

https://github.com/openssl/openssl/blob/OpenSSL_1_1_1-stable/crypto/bio/bss_file.c#L257 in file_ctrl()
https://github.com/openssl/openssl/blob/OpenSSL_1_1_1-stable/crypto/bio/bio_lib.c#L528 in BIO_ctrl()
https://github.com/openssl/openssl/blob/OpenSSL_1_1_1-stable/crypto/bio/bss_file.c#L104 in BIO_new_fp()
https://github.com/openssl/openssl/blob/OpenSSL_1_1_1-stable/apps/apps.c#L2471 in dup_bio_out()
https://github.com/openssl/openssl/blob/OpenSSL_1_1_1-stable/apps/apps.c#L2624 in bio_open_default_()
https://github.com/openssl/openssl/blob/OpenSSL_1_1_1-stable/apps/apps.c#L2652 in bio_open_default()
https://github.com/openssl/openssl/blob/OpenSSL_1_1_1-stable/apps/x509.c#L595 in x509_main()
https://github.com/openssl/openssl/blob/OpenSSL_1_1_1-stable/apps/openssl.c#L564 in do_cmd()
https://github.com/openssl/openssl/blob/OpenSSL_1_1_1-stable/apps/openssl.c#L183 in main()

> 
>> Question now is: These days, what is the correct way to handle this?
>>
>> Is it up to openssl to not use textmode at all with Cygwin (what if
>> used from within some cmd.exe?), or is it up to the shell script to
>> explicitly drop any carriage return here?
> 
> They way openssl does it looks right.  Off the top of my head I don't 
> grok where you get the \r from in the example above.

So a "correct" and portable way in shell scripts really is something like this?
$ var=`openssl ... | tr -d "\\r"`
or even (with bash or similar):
$ var=$(openssl ...); var=${var%$'\r'}

Thanks!
/haubi/

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: textmode for stdout, what is "correct" now?

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 2167 bytes --]

On Feb 15 08:56, Michael Haubenwallner wrote:
> On 2/14/19 5:20 PM, Corinna Vinschen wrote:
> > On Feb 14 16:23, Michael Haubenwallner wrote:
> >> Hi,
> >>
> >> so I find myself struggling with textmode versus binmode for stdio again.
> >>
> >> Running the openssl command (from within the apps/ build directory here) does
> >> yield different results regarding carriage return depending on the version:
> >>
> >> $ ./apps/openssl version
> >> OpenSSL 1.0.2p  14 Aug 2018
> >> $ ./apps/openssl x509 -hash -noout -in /etc/pki/tls/cert.pem | xxd
> >> 00000000: 6139 3464 3039 6535 0a                   a94d09e5.
> >>
> >>
> >> $ ./apps/openssl version
> >> OpenSSL 1.1.0j  20 Nov 2018
> >> $ ./apps/openssl x509 -hash -noout -in /etc/pki/tls/cert.pem | xxd
> >> 00000000: 6139 3464 3039 6535 0d0a                 a94d09e5..
> >>
> >> Some subsequent shell script does create wrong symlink filenames
> >> (with embedded CR) when used with openssl-1.1.x.
> >>
> >> The commit that changed this behaviour in openssl-1.1 is:
> >> https://github.com/openssl/openssl/commit/bdd58d98467e9f0f6635c1628e1eae304383afb1
> >>
> >> >From an openssl developer's point of view, I can understand to set
> >> textmode when the intent is to output some text, and to set
> >> binmode when the intent is to output some binary data.
> > 
> > How do you create \r\n in this case?  The upstream patch never
> > adds the explicit 't' flag.  It only adds 'b' or nothing.  So
> > the output should be \n only unless you write to a file on a
> > text mode mount.  What am I missing?
> 
> Down the line in their BIO module they do use setmode(fd, O_TEXT),
> which is the one that does introduce the \r, as far as I know.

This one is not so nice.  Somebody should tell upstream we only
want explicit O_BINARY these days, but no explicit O_TEXT.

> The backtrace in openssl-1.1.1a in this use case is:
> [...]
> >> Question now is: These days, what is the correct way to handle this?

Telling upstream not to use O_TEXT on Cygwin in the first place, I think.

For scripting, d2u should help.


Corinna

-- 
Corinna Vinschen
Cygwin Maintainer

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: textmode for stdout, what is "correct" now?

[Michael Haubenwallner]

On 2/15/19 11:22 AM, Corinna Vinschen wrote:
> On Feb 15 08:56, Michael Haubenwallner wrote:
>> On 2/14/19 5:20 PM, Corinna Vinschen wrote:
>>> On Feb 14 16:23, Michael Haubenwallner wrote:
>>>> Hi,
>>>>
>>>> so I find myself struggling with textmode versus binmode for stdio again.
>>>>
>>>> Running the openssl command (from within the apps/ build directory here) does
>>>> yield different results regarding carriage return depending on the version:
>>>>
>>>> $ ./apps/openssl version
>>>> OpenSSL 1.0.2p  14 Aug 2018
>>>> $ ./apps/openssl x509 -hash -noout -in /etc/pki/tls/cert.pem | xxd
>>>> 00000000: 6139 3464 3039 6535 0a                   a94d09e5.
>>>>
>>>>
>>>> $ ./apps/openssl version
>>>> OpenSSL 1.1.0j  20 Nov 2018
>>>> $ ./apps/openssl x509 -hash -noout -in /etc/pki/tls/cert.pem | xxd
>>>> 00000000: 6139 3464 3039 6535 0d0a                 a94d09e5..
>>>>
>>>> Some subsequent shell script does create wrong symlink filenames
>>>> (with embedded CR) when used with openssl-1.1.x.
>>>>
>>>> The commit that changed this behaviour in openssl-1.1 is:
>>>> https://github.com/openssl/openssl/commit/bdd58d98467e9f0f6635c1628e1eae304383afb1
>>>>
>>>> >From an openssl developer's point of view, I can understand to set
>>>> textmode when the intent is to output some text, and to set
>>>> binmode when the intent is to output some binary data.
>>>
>>> How do you create \r\n in this case?  The upstream patch never
>>> adds the explicit 't' flag.  It only adds 'b' or nothing.  So
>>> the output should be \n only unless you write to a file on a
>>> text mode mount.  What am I missing?
>>
>> Down the line in their BIO module they do use setmode(fd, O_TEXT),
>> which is the one that does introduce the \r, as far as I know.
> 
> This one is not so nice.  Somebody should tell upstream we only
> want explicit O_BINARY these days, but no explicit O_TEXT.

Is this correct even for situations where the cygwin1.dll is used
outside the Cygwin distribution, like git-bash, MSYS or similar,
where cygwin-based executables eventually are used from within some
CMD or PowerShell script? Or should they use unix2dos/dos2unix then?

OTOH, would it make sense to ignore the O_TEXT flag in cygwin1.dll?

> 
>> The backtrace in openssl-1.1.1a in this use case is:
>> [...]
>>>> Question now is: These days, what is the correct way to handle this?
> 
> Telling upstream not to use O_TEXT on Cygwin in the first place, I think.

I can do that, but if I were an upstream developer I would ask questions
like above...

> 
> For scripting, d2u should help.

Plus, to be portable: type d2u >/dev/null 2>&1 || d2u() { cat; }

So, firsthand I do prefer to avoid that need.

/haubi/

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: textmode for stdout, what is "correct" now?

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 2032 bytes --]

On Feb 15 13:03, Michael Haubenwallner wrote:
> On 2/15/19 11:22 AM, Corinna Vinschen wrote:
> > On Feb 15 08:56, Michael Haubenwallner wrote:
> >> On 2/14/19 5:20 PM, Corinna Vinschen wrote:
> >>> On Feb 14 16:23, Michael Haubenwallner wrote:
> >>>> Hi,
> >>>> [SNIP]
> >> Down the line in their BIO module they do use setmode(fd, O_TEXT),
> >> which is the one that does introduce the \r, as far as I know.
> > 
> > This one is not so nice.  Somebody should tell upstream we only
> > want explicit O_BINARY these days, but no explicit O_TEXT.
> 
> Is this correct even for situations where the cygwin1.dll is used
> outside the Cygwin distribution, like git-bash, MSYS or similar,

This is OpenSSL, not the Cygwin DLL.

> where cygwin-based executables eventually are used from within some
> CMD or PowerShell script? Or should they use unix2dos/dos2unix then?

Only if the \r is really required.  Typically it isn't.

> OTOH, would it make sense to ignore the O_TEXT flag in cygwin1.dll?

That's an interesting idea.  The O_TEXT flag is already ignored in a lot
of cases, e.g. for pipes.  Only when opening files does it have an
effect, mostly.  I'm not sure we should really switch it off.  Maybe we
can consider a CYGWIN env var setting at one point.

> >> The backtrace in openssl-1.1.1a in this use case is:
> >> [...]
> >>>> Question now is: These days, what is the correct way to handle this?
> > 
> > Telling upstream not to use O_TEXT on Cygwin in the first place, I think.
> 
> I can do that, but if I were an upstream developer I would ask questions
> like above...

I sent a patch upstream and questions got asked.  But this is not
a native openssl lib, this is *Cygwin's* openssl lib, and it should
behave like a Cygwin lib.

> > For scripting, d2u should help.
> 
> Plus, to be portable: type d2u >/dev/null 2>&1 || d2u() { cat; }

For portability, use `dos2unix'.  It's packed with most (all?) Linux
distros.


Corinna

-- 
Corinna Vinschen
Cygwin Maintainer

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: textmode for stdout, what is "correct" now?

[Andrey Repin]

Greetings, Michael Haubenwallner!

>>
>> For scripting, d2u should help.

> Plus, to be portable: type d2u >/dev/null 2>&1 || d2u() { cat; }

_d2u="$( which d2u 2> /dev/null || echo cat )"

if you want to be THAT nice.

> So, firsthand I do prefer to avoid that need.

+1


-- 
With best regards,
Andrey Repin
Friday, February 15, 2019 15:41:48

Sorry for my terrible english...


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: textmode for stdout, what is "correct" now?

[Michael Haubenwallner]

On 2/15/19 1:48 PM, Corinna Vinschen wrote:
> On Feb 15 13:03, Michael Haubenwallner wrote:
>> On 2/15/19 11:22 AM, Corinna Vinschen wrote:
>>> On Feb 15 08:56, Michael Haubenwallner wrote:
>>>> On 2/14/19 5:20 PM, Corinna Vinschen wrote:
>>>>> On Feb 14 16:23, Michael Haubenwallner wrote:
>>>>>> Hi,
>>>>>> [SNIP]
>>>> Down the line in their BIO module they do use setmode(fd, O_TEXT),
>>>> which is the one that does introduce the \r, as far as I know.
>>>
>>> This one is not so nice.  Somebody should tell upstream we only
>>> want explicit O_BINARY these days, but no explicit O_TEXT.

To me it sounds strange to use the one but not the other:

If we don't want O_TEXT at all, isn't O_BINARY obsolete as well,
so the advise should be to use neither - just like real *nix?

A consequence then might be to deprecate (or even remove) them
from the public API header files.

>> Is this correct even for situations where the cygwin1.dll is used
>> outside the Cygwin distribution, like git-bash, MSYS or similar,
> 
> This is OpenSSL, not the Cygwin DLL.

Actually I meant executables linked against the Cygwin DLL being
executed by non-Cygwin (native Win32) programs.

>> where cygwin-based executables eventually are used from within some
>> CMD or PowerShell script? Or should they use unix2dos/dos2unix then?
> 
> Only if the \r is really required.  Typically it isn't.

Ok then.

>> OTOH, would it make sense to ignore the O_TEXT flag in cygwin1.dll?
> 
> That's an interesting idea.  The O_TEXT flag is already ignored in a lot
> of cases, e.g. for pipes.  Only when opening files does it have an
> effect, mostly.  I'm not sure we should really switch it off.  Maybe we
> can consider a CYGWIN env var setting at one point.

I've thought of the CYGWIN env var too whether to ignore O_TEXT, but the
more I think of it, the more I can think of multiple troubles as well...

>>>> The backtrace in openssl-1.1.1a in this use case is:
>>>> [...]
>>>>>> Question now is: These days, what is the correct way to handle this?
>>>
>>> Telling upstream not to use O_TEXT on Cygwin in the first place, I think.
>>
>> I can do that, but if I were an upstream developer I would ask questions
>> like above...
> 
> I sent a patch upstream and questions got asked.  But this is not
> a native openssl lib, this is *Cygwin's* openssl lib, and it should
> behave like a Cygwin lib.

Agreed.

Thanks!
/haubi/

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

portable use of dos2unix WAS: Re: textmode for stdout, what is "correct" now?

[Michael Haubenwallner]

Hi Andrey,

On 2/15/19 1:45 PM, Andrey Repin wrote:
> Greetings, Michael Haubenwallner!
> 
>>>
>>> For scripting, d2u should help.
> 
>> Plus, to be portable: type d2u >/dev/null 2>&1 || d2u() { cat; }
> 
> _d2u="$( which d2u 2> /dev/null || echo cat )"

To be honest, this is less portable for some reasons:

* "$(...)" is not Bourne Shell (/bin/sh) but POSIX Shell (ksh, bash, ...)
* "which" is C Shell and has it's own troubles (may load ~/.cshrc first)

On 2/15/19 1:48 PM, Corinna Vinschen wrote:
> 
> For portability, use `dos2unix'.  It's packed with most (all?) Linux
> distros.

Well... available yes, but not installed by default (any more).

> 
> if you want to be THAT nice.
> 
>> So, firsthand I do prefer to avoid that need.
> 
> +1

/haubi/

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: portable use of dos2unix WAS: Re: textmode for stdout, what is "correct" now?

[Andrey Repin]

Greetings, Michael Haubenwallner!

> On 2/15/19 1:45 PM, Andrey Repin wrote:
>> Greetings, Michael Haubenwallner!
>> 
>>>>
>>>> For scripting, d2u should help.
>> 
>>> Plus, to be portable: type d2u >/dev/null 2>&1 || d2u() { cat; }
>> 
>> _d2u="$( which d2u 2> /dev/null || echo cat )"

> To be honest, this is less portable for some reasons:

> * "$(...)" is not Bourne Shell (/bin/sh) but POSIX Shell (ksh, bash, ...)

And /bin/sh is a POSIX shell. I don't know what you are trying to say here.
http://pubs.opengroup.org/onlinepubs/9699919799/utilities/sh.html#tag_20_117
http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_06_03

> * "which" is C Shell and has it's own troubles (may load ~/.cshrc first)

$ which which
/bin/which

It's a separate executable, unless shell provides an override.
How it could possible load foreign RC file is beyond me.

----

I have a different question though. The commands you presented in the OP
look like you aren't using Cygwin-suppied OpenSSL binary.


-- 
With best regards,
Andrey Repin
Friday, February 15, 2019 21:28:57

Sorry for my terrible english...


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: portable use of dos2unix WAS: Re: textmode for stdout, what is "correct" now?

[Michael Haubenwallner]

Hi Andrey,

On 2/15/19 7:35 PM, Andrey Repin wrote:
> Greetings, Michael Haubenwallner!
> 
>> On 2/15/19 1:45 PM, Andrey Repin wrote:
>>> Greetings, Michael Haubenwallner!
>>>
>>>>>
>>>>> For scripting, d2u should help.
>>>
>>>> Plus, to be portable: type d2u >/dev/null 2>&1 || d2u() { cat; }
>>>
>>> _d2u="$( which d2u 2> /dev/null || echo cat )"
> 
>> To be honest, this is less portable for some reasons:
> 
>> * "$(...)" is not Bourne Shell (/bin/sh) but POSIX Shell (ksh, bash, ...)
> 
> And /bin/sh is a POSIX shell. I don't know what you are trying to say here.
> http://pubs.opengroup.org/onlinepubs/9699919799/utilities/sh.html#tag_20_117
> http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_06_03

Well, /bin/sh *shall* be a POSIX shell. But the reality is not there yet:
https://www.gnu.org/software/autoconf/manual/autoconf.html#Portable-Shell

>> * "which" is C Shell and has it's own troubles (may load ~/.cshrc first)
> 
> $ which which
> /bin/which
> 
> It's a separate executable, unless shell provides an override.
> How it could possible load foreign RC file is beyond me.

On AIX for example, /usr/bin/which is a csh script by itself:

$ type which
which is /usr/bin/which

$ file /usr/bin/which
/usr/bin/which: shell script  - csh (C shell)

$ head -n5 /usr/bin/which
#!/usr/bin/csh -f
set prompt = "% "
if ( -f ~/.cshrc) then
        source ~/.cshrc
endif

Now imagine what happens when ~/.cshrc does mess with PATH.

Anyway: To find something along PATH, I do prefer some shell builtin rather
than something to be found along PATH.

> 
> ----
> 
> I have a different question though. The commands you presented in the OP
> look like you aren't using Cygwin-suppied OpenSSL binary.

This is true: The Cygwin distro does provide openssl-1.0.2p, but I'm building
the *portable* Gentoo Prefix distro on Cygwin, with openssl-1.1.1a already.
But whenever the Cygwin distro would bump to openssl-1.1, the same problem
would arise there as well, just becomes noticed much later probably.

/haubi/

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: textmode for stdout, what is "correct" now?

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 1176 bytes --]

On Feb 15 19:07, Michael Haubenwallner wrote:
> On 2/15/19 1:48 PM, Corinna Vinschen wrote:
> > On Feb 15 13:03, Michael Haubenwallner wrote:
> >> On 2/15/19 11:22 AM, Corinna Vinschen wrote:
> >>> On Feb 15 08:56, Michael Haubenwallner wrote:
> >>>> On 2/14/19 5:20 PM, Corinna Vinschen wrote:
> >>>>> On Feb 14 16:23, Michael Haubenwallner wrote:
> >>>>>> Hi,
> >>>>>> [SNIP]
> >>>> Down the line in their BIO module they do use setmode(fd, O_TEXT),
> >>>> which is the one that does introduce the \r, as far as I know.
> >>>
> >>> This one is not so nice.  Somebody should tell upstream we only
> >>> want explicit O_BINARY these days, but no explicit O_TEXT.
> 
> To me it sounds strange to use the one but not the other:
> 
> If we don't want O_TEXT at all, isn't O_BINARY obsolete as well,
> so the advise should be to use neither - just like real *nix?

No, on text mode mounts O_BINARY makes sure that the result is
actually a binary file.

> A consequence then might be to deprecate (or even remove) them
> from the public API header files.

As long as we have text mode mounts, no.


Corinna

-- 
Corinna Vinschen
Cygwin Maintainer

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: portable use of dos2unix WAS: Re: textmode for stdout, what is "correct" now?

[Andrey Repin]

Greetings, Michael Haubenwallner!

>> I have a different question though. The commands you presented in the OP
>> look like you aren't using Cygwin-suppied OpenSSL binary.

> This is true: The Cygwin distro does provide openssl-1.0.2p, but I'm building
> the *portable* Gentoo Prefix distro on Cygwin, with openssl-1.1.1a already.
> But whenever the Cygwin distro would bump to openssl-1.1, the same problem
> would arise there as well, just becomes noticed much later probably.

Thanks, that was what I was missing all this time.


-- 
With best regards,
Andrey Repin
Friday, February 15, 2019 23:37:18

Sorry for my terrible english...


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: portable use of dos2unix WAS: Re: textmode for stdout, what is "correct" now?

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 800 bytes --]

On Feb 15 20:33, Michael Haubenwallner wrote:
> On 2/15/19 7:35 PM, Andrey Repin wrote:
> > I have a different question though. The commands you presented in the OP
> > look like you aren't using Cygwin-suppied OpenSSL binary.
> 
> This is true: The Cygwin distro does provide openssl-1.0.2p, but I'm building
> the *portable* Gentoo Prefix distro on Cygwin, with openssl-1.1.1a already.
> But whenever the Cygwin distro would bump to openssl-1.1, the same problem
> would arise there as well, just becomes noticed much later probably.

Does OpenSSH build with OpenSSL 1.1.1a these days?  If so, we might
want to switch OpenSSL finally, too.

Btw., Michael, maybe you're interested to become Cygwin OpenSSL
maintainer?


Thanks,
Corinna

-- 
Corinna Vinschen
Cygwin Maintainer

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: portable use of dos2unix WAS: Re: textmode for stdout, what is "correct" now?

[Brian Inglis]

On 2019-02-15 12:33, Michael Haubenwallner wrote:
> This is true: The Cygwin distro does provide openssl-1.0.2p, but I'm
> building the *portable* Gentoo Prefix distro on Cygwin, with openssl-1.1.1a
> already.
> But whenever the Cygwin distro would bump to openssl-1.1, the same problem 
> would arise there as well, just becomes noticed much later probably.

When you are building another version of a Cygwin package, if you have not done
so already, it is worth installing cygport, all its dependencies, other build
tools suggested in the package maintainer docs, and use those to download the
latest Cygwin package source, in case there are patches provided which make the
package build or run better under Cygwin.
It is then easy to create another package-version directory into which you can
copy the .cygport file and modify it to download, build, test, and install the
other version of the package, optionally with any provided patches if still
required.

-- 
Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada

This email may be disturbing to some readers as it contains
too much technical detail. Reader discretion is advised.

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: textmode for stdout, what is "correct" now?

[Jeffrey Walton]

On Thu, Feb 14, 2019 at 10:23 AM Michael Haubenwallner
<michael.haubenwallner@ssi-schaefer.com> wrote:
>
> so I find myself struggling with textmode versus binmode for stdio again.
>
> Running the openssl command (from within the apps/ build directory here) does
> yield different results regarding carriage return depending on the version:
>
> $ ./apps/openssl version
> OpenSSL 1.0.2p  14 Aug 2018
> $ ./apps/openssl x509 -hash -noout -in /etc/pki/tls/cert.pem | xxd
> 00000000: 6139 3464 3039 6535 0a                   a94d09e5.
>
>
> $ ./apps/openssl version
> OpenSSL 1.1.0j  20 Nov 2018
> $ ./apps/openssl x509 -hash -noout -in /etc/pki/tls/cert.pem | xxd
> 00000000: 6139 3464 3039 6535 0d0a                 a94d09e5..
>
> Some subsequent shell script does create wrong symlink filenames
> (with embedded CR) when used with openssl-1.1.x.
>
> The commit that changed this behaviour in openssl-1.1 is:
> https://github.com/openssl/openssl/commit/bdd58d98467e9f0f6635c1628e1eae304383afb1
>
> From an openssl developer's point of view, I can understand to set
> textmode when the intent is to output some text, and to set
> binmode when the intent is to output some binary data.
>
> Question now is: These days, what is the correct way to handle this?


RFC 1421, Privacy Enhanced Mail (PEM)
(https://www.ietf.org/rfc/rfc1421.txt), is the controlling document.
Lines should break at 64 characters, and EOL is CRLF pairs.

 * 64-char lines is covered in section 4.3.2.4, Printable Encodings
 * CRLF is covered in section 4.3.1  Constraints

Line breaks at 64 chars is an anachronism (in my opinion), but some
software still enforces it. Some software will reject keys and
certificates if not properly broken.

Jeff

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: textmode for stdout, what is "correct" now?

[Michael Haubenwallner]

On 2/15/19 9:31 PM, Corinna Vinschen wrote:
> On Feb 15 19:07, Michael Haubenwallner wrote:
>> On 2/15/19 1:48 PM, Corinna Vinschen wrote:
>>> On Feb 15 13:03, Michael Haubenwallner wrote:
>>>> On 2/15/19 11:22 AM, Corinna Vinschen wrote:
>>>>> On Feb 15 08:56, Michael Haubenwallner wrote:
>>>>>> On 2/14/19 5:20 PM, Corinna Vinschen wrote:
>>>>>>> On Feb 14 16:23, Michael Haubenwallner wrote:
>>>>>>>> Hi,
>>>>>>>> [SNIP]
>>>>>> Down the line in their BIO module they do use setmode(fd, O_TEXT),
>>>>>> which is the one that does introduce the \r, as far as I know.
>>>>>
>>>>> This one is not so nice.  Somebody should tell upstream we only
>>>>> want explicit O_BINARY these days, but no explicit O_TEXT.
>>
>> To me it sounds strange to use the one but not the other:
>>
>> If we don't want O_TEXT at all, isn't O_BINARY obsolete as well,
>> so the advise should be to use neither - just like real *nix?
> 
> No, on text mode mounts O_BINARY makes sure that the result is
> actually a binary file.
> 
>> A consequence then might be to deprecate (or even remove) them
>> from the public API header files.
> 
> As long as we have text mode mounts, no.

Ah... So this feels like a semantic confusion around setmode(O_TEXT):

For the fopen() API family, POSIX says that conforming platforms should
accept (and eventually ignore) the 'b' character in the mode argument.
But note that there is no 't' character specified here.

So with the fopen() API family, application developers are able to
enable or disable binary mode, but are _not_ able to explicitly
enable or disable text mode.

Unfortunately, the setmode() API is semantically different, because it
does not allow to disable binary mode, but to "enable text mode" instead.

So with the setmode() API, when application developers want to disable
binary mode, their only option is to "enable text mode".

Hence, for Cygwin, setmode("enable text mode") should disable binary
mode only, rather than switch to DOS text mode at the same time.

/haubi/

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: textmode for stdout, what is "correct" now?

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 2141 bytes --]

On Feb 16 09:19, Michael Haubenwallner wrote:
> On 2/15/19 9:31 PM, Corinna Vinschen wrote:
> > On Feb 15 19:07, Michael Haubenwallner wrote:
> >> On 2/15/19 1:48 PM, Corinna Vinschen wrote:
> >>> On Feb 15 13:03, Michael Haubenwallner wrote:
> >>>> On 2/15/19 11:22 AM, Corinna Vinschen wrote:
> >>>>> On Feb 15 08:56, Michael Haubenwallner wrote:
> >>>>>> On 2/14/19 5:20 PM, Corinna Vinschen wrote:
> >>>>>>> On Feb 14 16:23, Michael Haubenwallner wrote:
> >>>>>>>> Hi,
> >>>>>>>> [SNIP]
> >>>>>> Down the line in their BIO module they do use setmode(fd, O_TEXT),
> >>>>>> which is the one that does introduce the \r, as far as I know.
> >>>>>
> >>>>> This one is not so nice.  Somebody should tell upstream we only
> >>>>> want explicit O_BINARY these days, but no explicit O_TEXT.
> >>
> >> To me it sounds strange to use the one but not the other:
> >>
> >> If we don't want O_TEXT at all, isn't O_BINARY obsolete as well,
> >> so the advise should be to use neither - just like real *nix?
> > 
> > No, on text mode mounts O_BINARY makes sure that the result is
> > actually a binary file.
> > 
> >> A consequence then might be to deprecate (or even remove) them
> >> from the public API header files.
> > 
> > As long as we have text mode mounts, no.
> 
> Ah... So this feels like a semantic confusion around setmode(O_TEXT):
> 
> For the fopen() API family, POSIX says that conforming platforms should
> accept (and eventually ignore) the 'b' character in the mode argument.
> But note that there is no 't' character specified here.
> 
> So with the fopen() API family, application developers are able to
> enable or disable binary mode, but are _not_ able to explicitly
> enable or disable text mode.
> 
> Unfortunately, the setmode() API is semantically different, because it
> does not allow to disable binary mode, but to "enable text mode" instead.
> 
> So with the setmode() API, when application developers want to disable
> binary mode, their only option is to "enable text mode".

setmode(fd, O_BINARY) works on Cygwin.


Corinna

-- 
Corinna Vinschen
Cygwin Maintainer

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: textmode for stdout, what is "correct" now?

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 2023 bytes --]

On Feb 15 20:26, Jeffrey Walton wrote:
> On Thu, Feb 14, 2019 at 10:23 AM Michael Haubenwallner
> <michael.haubenwallner@ssi-schaefer.com> wrote:
> >
> > so I find myself struggling with textmode versus binmode for stdio again.
> >
> > Running the openssl command (from within the apps/ build directory here) does
> > yield different results regarding carriage return depending on the version:
> >
> > $ ./apps/openssl version
> > OpenSSL 1.0.2p  14 Aug 2018
> > $ ./apps/openssl x509 -hash -noout -in /etc/pki/tls/cert.pem | xxd
> > 00000000: 6139 3464 3039 6535 0a                   a94d09e5.
> >
> >
> > $ ./apps/openssl version
> > OpenSSL 1.1.0j  20 Nov 2018
> > $ ./apps/openssl x509 -hash -noout -in /etc/pki/tls/cert.pem | xxd
> > 00000000: 6139 3464 3039 6535 0d0a                 a94d09e5..
> >
> > Some subsequent shell script does create wrong symlink filenames
> > (with embedded CR) when used with openssl-1.1.x.
> >
> > The commit that changed this behaviour in openssl-1.1 is:
> > https://github.com/openssl/openssl/commit/bdd58d98467e9f0f6635c1628e1eae304383afb1
> >
> > From an openssl developer's point of view, I can understand to set
> > textmode when the intent is to output some text, and to set
> > binmode when the intent is to output some binary data.
> >
> > Question now is: These days, what is the correct way to handle this?
> 
> 
> RFC 1421, Privacy Enhanced Mail (PEM)
> (https://www.ietf.org/rfc/rfc1421.txt), is the controlling document.
> Lines should break at 64 characters, and EOL is CRLF pairs.
> 
>  * 64-char lines is covered in section 4.3.2.4, Printable Encodings
>  * CRLF is covered in section 4.3.1  Constraints
> 
> Line breaks at 64 chars is an anachronism (in my opinion), but some
> software still enforces it. Some software will reject keys and
> certificates if not properly broken.

POSIX software has to write the \r explicitely since it can't rely
on DOS textmode.


Corinna

-- 
Corinna Vinschen
Cygwin Maintainer

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: textmode for stdout, what is "correct" now?

[Michael Haubenwallner]

On 2/16/19 10:38 AM, Corinna Vinschen wrote:
> On Feb 16 09:19, Michael Haubenwallner wrote:
>> On 2/15/19 9:31 PM, Corinna Vinschen wrote:
>>> On Feb 15 19:07, Michael Haubenwallner wrote:
>>>> On 2/15/19 1:48 PM, Corinna Vinschen wrote:
>>>>> On Feb 15 13:03, Michael Haubenwallner wrote:
>>>>>> On 2/15/19 11:22 AM, Corinna Vinschen wrote:
>>>>>>> On Feb 15 08:56, Michael Haubenwallner wrote:
>>>>>>>> On 2/14/19 5:20 PM, Corinna Vinschen wrote:
>>>>>>>>> On Feb 14 16:23, Michael Haubenwallner wrote:
>>>>>>>>>> Hi,
>>>>>>>>>> [SNIP]
>>>>>>>> Down the line in their BIO module they do use setmode(fd, O_TEXT),
>>>>>>>> which is the one that does introduce the \r, as far as I know.
>>>>>>>
>>>>>>> This one is not so nice.  Somebody should tell upstream we only
>>>>>>> want explicit O_BINARY these days, but no explicit O_TEXT.
>>>>
>>>> To me it sounds strange to use the one but not the other:
>>>>
>>>> If we don't want O_TEXT at all, isn't O_BINARY obsolete as well,
>>>> so the advise should be to use neither - just like real *nix?
>>>
>>> No, on text mode mounts O_BINARY makes sure that the result is
>>> actually a binary file.
>>>
>>>> A consequence then might be to deprecate (or even remove) them
>>>> from the public API header files.
>>>
>>> As long as we have text mode mounts, no.
>>
>> Ah... So this feels like a semantic confusion around setmode(O_TEXT):
>>
>> For the fopen() API family, POSIX says that conforming platforms should
>> accept (and eventually ignore) the 'b' character in the mode argument.
>> But note that there is no 't' character specified here.
>>
>> So with the fopen() API family, application developers are able to
>> enable or disable binary mode, but are _not_ able to explicitly
>> enable or disable text mode.
>>
>> Unfortunately, the setmode() API is semantically different, because it
>> does not allow to disable binary mode, but to "enable text mode" instead.
>>
>> So with the setmode() API, when application developers want to disable
>> binary mode, their only option is to "enable text mode".
> 
> setmode(fd, O_BINARY) works on Cygwin.

Well... true.

What I'm trying to say is that restoring the *default* mode is currently 
broken, because traditionally the default mode simply is called "text
mode" - which is enough for DOS only systems, but confusing in Cygwin.

As for Cygwin the default mode is more complex, we may want to call it
something like the "Cygwin text mode" instead.  Consequently, the mode
that does force \r should be called the "DOS text mode".

Whether the "Cygwin text mode" is with or without \r depends on some
other property like the text mount.

However, after setting the "binary mode", libraries like openssl BIO
want to be able to restore the "non-binary mode" - whatever name it
may have.  So they use the only available API: setmode(fd, O_TEXT).
And IMO they're right to expect the "Cygwin text mode" on Cygwin.

If they really need the "DOS text mode", they either should handle the
\r in binary mode - or we provide some new setmode(fd, O_DOSTEXT) API.

In hope to be more clear now,
/haubi/

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: textmode for stdout, what is "correct" now?

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 1451 bytes --]

On Feb 16 18:13, Michael Haubenwallner wrote:
> On 2/16/19 10:38 AM, Corinna Vinschen wrote:
> > setmode(fd, O_BINARY) works on Cygwin.
> 
> Well... true.
> 
> What I'm trying to say is that restoring the *default* mode is currently 
> broken, because traditionally the default mode simply is called "text
> mode" - which is enough for DOS only systems, but confusing in Cygwin.
> 
> As for Cygwin the default mode is more complex, we may want to call it
> something like the "Cygwin text mode" instead.  Consequently, the mode
> that does force \r should be called the "DOS text mode".
> 
> Whether the "Cygwin text mode" is with or without \r depends on some
> other property like the text mount.
> 
> However, after setting the "binary mode", libraries like openssl BIO
> want to be able to restore the "non-binary mode" - whatever name it
> may have.  So they use the only available API: setmode(fd, O_TEXT).
> And IMO they're right to expect the "Cygwin text mode" on Cygwin.
> 
> If they really need the "DOS text mode", they either should handle the
> \r in binary mode - or we provide some new setmode(fd, O_DOSTEXT) API.
> 
> In hope to be more clear now,
> /haubi/

I really miss the problem you're trying to solve here.  Why should an
application setting O_BINARY explicitely revert this decision on the
same file descriptor?  That doesn't make sense.


Corinna

-- 
Corinna Vinschen
Cygwin Maintainer

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: textmode for stdout, what is "correct" now?

[Michael Haubenwallner]

On 2/16/19 6:43 PM, Corinna Vinschen wrote:
> On Feb 16 18:13, Michael Haubenwallner wrote:
>> On 2/16/19 10:38 AM, Corinna Vinschen wrote:
>>> setmode(fd, O_BINARY) works on Cygwin.
>>
>> Well... true.
>>
>> What I'm trying to say is that restoring the *default* mode is currently 
>> broken, because traditionally the default mode simply is called "text
>> mode" - which is enough for DOS only systems, but confusing in Cygwin.
>>
>> As for Cygwin the default mode is more complex, we may want to call it
>> something like the "Cygwin text mode" instead.  Consequently, the mode
>> that does force \r should be called the "DOS text mode".
>>
>> Whether the "Cygwin text mode" is with or without \r depends on some
>> other property like the text mount.
>>
>> However, after setting the "binary mode", libraries like openssl BIO
>> want to be able to restore the "non-binary mode" - whatever name it
>> may have.  So they use the only available API: setmode(fd, O_TEXT).
>> And IMO they're right to expect the "Cygwin text mode" on Cygwin.
>>
>> If they really need the "DOS text mode", they either should handle the
>> \r in binary mode - or we provide some new setmode(fd, O_DOSTEXT) API.
>>
>> In hope to be more clear now,
>> /haubi/
> 
> I really miss the problem you're trying to solve here.  Why should an
> application setting O_BINARY explicitely revert this decision on the
> same file descriptor?  That doesn't make sense.

Well, it's not necessarily about really switching binary mode on and off,
it's more about avoiding breakage when applications try to intuitively
follow the original API, even if that actually causes the call to
setmode(fd, O_TEXT) to be redundant.

OTOH, this question also would apply to native Win32 applications, so why
do people call setmode(fd, O_TEXT) with any DOS based platform at all?

IMO, unfortunately we're not in a position to modify the intention of the
original API.  And finally, I do want to stop discussions like this one
with application developers like openssl, as soon as we can argue like:
"Cygwin does not use \r internally, but does support text mode mounts,
so we had to invent the Cygwin text mode, which may or may not use \r.
Hence you get the Cygwin text mode with O_TEXT, and if you really are
in some 'unix2dos' position, please use the new O_DOSTEXT mode instead."

However, agreed this does not seem to be trivial to implement.  Yet I
will look into it when there is a chance for a patches to be accepted.

Thanks!
/haubi/

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: portable use of dos2unix WAS: Re: textmode for stdout, what is "correct" now?

[Michael Haubenwallner]

On 2/15/19 10:00 PM, Corinna Vinschen wrote:
> On Feb 15 20:33, Michael Haubenwallner wrote:
>> On 2/15/19 7:35 PM, Andrey Repin wrote:
>>> I have a different question though. The commands you presented in the OP
>>> look like you aren't using Cygwin-suppied OpenSSL binary.
>>
>> This is true: The Cygwin distro does provide openssl-1.0.2p, but I'm building
>> the *portable* Gentoo Prefix distro on Cygwin, with openssl-1.1.1a already.
>> But whenever the Cygwin distro would bump to openssl-1.1, the same problem
>> would arise there as well, just becomes noticed much later probably.
> 
> Does OpenSSH build with OpenSSL 1.1.1a these days?  If so, we might
> want to switch OpenSSL finally, too.

Well, OpenSSH-7.9_p1 maybe does, according to https://bugs.gentoo.org/592578

But there's a bunch of other packages that still don't, at least within Gentoo:
https://bugs.gentoo.org/showdependencytree.cgi?id=592438&maxdepth=1

> Btw., Michael, maybe you're interested to become Cygwin OpenSSL
> maintainer?

Uhm, here I have to decline, as I don't really care about the openssl version
in the Cygwin distro.  Providing headsup here and there upon breakage really
is something different.

Sorry, but thank you for the confidence,
/haubi/

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: portable use of dos2unix WAS: Re: textmode for stdout, what is "correct" now?

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 1458 bytes --]

On Feb 18 11:01, Michael Haubenwallner wrote:
> On 2/15/19 10:00 PM, Corinna Vinschen wrote:
> > On Feb 15 20:33, Michael Haubenwallner wrote:
> >> On 2/15/19 7:35 PM, Andrey Repin wrote:
> >>> I have a different question though. The commands you presented in the OP
> >>> look like you aren't using Cygwin-suppied OpenSSL binary.
> >>
> >> This is true: The Cygwin distro does provide openssl-1.0.2p, but I'm building
> >> the *portable* Gentoo Prefix distro on Cygwin, with openssl-1.1.1a already.
> >> But whenever the Cygwin distro would bump to openssl-1.1, the same problem
> >> would arise there as well, just becomes noticed much later probably.
> > 
> > Does OpenSSH build with OpenSSL 1.1.1a these days?  If so, we might
> > want to switch OpenSSL finally, too.
> 
> Well, OpenSSH-7.9_p1 maybe does, according to https://bugs.gentoo.org/592578
> 
> But there's a bunch of other packages that still don't, at least within Gentoo:
> https://bugs.gentoo.org/showdependencytree.cgi?id=592438&maxdepth=1
> 
> > Btw., Michael, maybe you're interested to become Cygwin OpenSSL
> > maintainer?
> 
> Uhm, here I have to decline, as I don't really care about the openssl version
> in the Cygwin distro.  Providing headsup here and there upon breakage really
> is something different.
> 
> Sorry, but thank you for the confidence,
> /haubi/

Too bad, but I had to try ;)


Corinna

-- 
Corinna Vinschen
Cygwin Maintainer

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: textmode for stdout, what is "correct" now?

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 2432 bytes --]

On Feb 18 10:40, Michael Haubenwallner wrote:
> On 2/16/19 6:43 PM, Corinna Vinschen wrote:
> > I really miss the problem you're trying to solve here.  Why should an
> > application setting O_BINARY explicitely revert this decision on the
> > same file descriptor?  That doesn't make sense.
> 
> Well, it's not necessarily about really switching binary mode on and off,
> it's more about avoiding breakage when applications try to intuitively
> follow the original API, even if that actually causes the call to
> setmode(fd, O_TEXT) to be redundant.
> 
> OTOH, this question also would apply to native Win32 applications, so why
> do people call setmode(fd, O_TEXT) with any DOS based platform at all?
> 
> IMO, unfortunately we're not in a position to modify the intention of the
> original API.  And finally, I do want to stop discussions like this one
> with application developers like openssl, as soon as we can argue like:
> "Cygwin does not use \r internally, but does support text mode mounts,
> so we had to invent the Cygwin text mode, which may or may not use \r.
> Hence you get the Cygwin text mode with O_TEXT, and if you really are
> in some 'unix2dos' position, please use the new O_DOSTEXT mode instead."
> 
> However, agreed this does not seem to be trivial to implement.  Yet I
> will look into it when there is a chance for a patches to be accepted.

Bottom line:

- Make O_TEXT equivalent to O_BINARY on the API level so Cygwin
  actually uses binary mode on open(O_TEXT) and setmode(O_TEXT).

- Make O_DOSTEXT equivalent to the former O_TEXT.

Result: we use binary mode even with tools explicitely specifying O_TEXT.

- How do you handle fopen modes "wt", "rt", etc, then?

- What about automode.c, binmode.c, textmode.c, textreadmode.c?

- How do you avoid breakage of existing tools which have been written to
  work explicitely with certain DOS formatted text file and use O_TEXT
  for that?

The answer to the last one could be using a new version check like the
ones already in include/cygwin/version.h.  Existing tools and libs keep
the current behaviour.  Only newly built binaries get the new behaviour.
However, this still may result in breakage if the developer isn't aware
of this subtil change.  As much as I hate O_TEXT mode, there's a
pretty basic expectation how this is supposed to work.


Corinna

-- 
Corinna Vinschen
Cygwin Maintainer

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: textmode for stdout, what is "correct" now?

[Michael Haubenwallner]

On 2/18/19 11:26 AM, Corinna Vinschen wrote:
> On Feb 18 10:40, Michael Haubenwallner wrote:
>> On 2/16/19 6:43 PM, Corinna Vinschen wrote:
>>> I really miss the problem you're trying to solve here.  Why should an
>>> application setting O_BINARY explicitely revert this decision on the
>>> same file descriptor?  That doesn't make sense.
>>
>> Well, it's not necessarily about really switching binary mode on and off,
>> it's more about avoiding breakage when applications try to intuitively
>> follow the original API, even if that actually causes the call to
>> setmode(fd, O_TEXT) to be redundant.
>>
>> OTOH, this question also would apply to native Win32 applications, so why
>> do people call setmode(fd, O_TEXT) with any DOS based platform at all?
>>
>> IMO, unfortunately we're not in a position to modify the intention of the
>> original API.  And finally, I do want to stop discussions like this one
>> with application developers like openssl, as soon as we can argue like:
>> "Cygwin does not use \r internally, but does support text mode mounts,
>> so we had to invent the Cygwin text mode, which may or may not use \r.
>> Hence you get the Cygwin text mode with O_TEXT, and if you really are
>> in some 'unix2dos' position, please use the new O_DOSTEXT mode instead."
>>
>> However, agreed this does not seem to be trivial to implement.  Yet I
>> will look into it when there is a chance for a patches to be accepted.
> 
> Bottom line:
> 
> - Make O_TEXT equivalent to O_BINARY on the API level so Cygwin
>   actually uses binary mode on open(O_TEXT) and setmode(O_TEXT).

No, O_TEXT is neither equal to O_BINARY nor to O_DOSTEXT - it's something
in between.  My first ideas are either (O_BINARY|O_DOSTEXT) or another bit.

> - Make O_DOSTEXT equivalent to the former O_TEXT.

Yes.

> Result: we use binary mode even with tools explicitely specifying O_TEXT.

No, not binary mode. It's text mode with \r being allowed rather than forced.

> - How do you handle fopen modes "wt", "rt", etc, then?

This API feels a little different, because it does allow for with "b" and
without "b", as well as with "t" and without "t", so "t" should translate
to O_DOSTEXT IMO.

While talking about it: What will setmode(fd, 0) do?

> - What about automode.c, binmode.c, textmode.c, textreadmode.c?

Just stumbled over the distinction between readmode and writemode:
What's up with that?
Unless binary mode, reading always could be done in dostext mode.
Here the default is to link without them, and the opposite of binmode.c
is to not use anything, hence the text*mode should be O_DOSTEXT.

> - How do you avoid breakage of existing tools which have been written to
>   work explicitely with certain DOS formatted text file and use O_TEXT
>   for that?
> 
> The answer to the last one could be using a new version check like the
> ones already in include/cygwin/version.h.  Existing tools and libs keep
> the current behaviour.  Only newly built binaries get the new behaviour.

Exactly. And for the check:
For dostext mode: ifdef O_DOSTEXT: use O_DOSTEXT, otherways use O_TEXT.
For cygtext mode: ifdef O_DOSTEXT: use O_TEXT, otherways avoid setmode.

> However, this still may result in breakage if the developer isn't aware
> of this subtil change.  As much as I hate O_TEXT mode, there's a
> pretty basic expectation how this is supposed to work.

Yes, but I do expect this in corner cases only, with unix2dos/dos2unix as
the specific example.

OTOH, with setmode(fd, 0) coming to my mind: If that would denote the default
(=cygwin text) mode, I can imagine we may convince (openssl) developers to use
zero instead of O_TEXT, and everything could be fine without any Cygwin change.

Heck, this would feel like most obvious - even API wise, no?

Then we may want to add O_NOBINARY defined to zero as the only Cygwin change.

Thanks!
/haubi/

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: portable use of dos2unix WAS: Re: textmode for stdout, what is "correct" now?

[Jeffrey Walton]

On Mon, Feb 18, 2019 at 5:01 AM Michael Haubenwallner
<michael.haubenwallner@ssi-schaefer.com> wrote:
>
> On 2/15/19 10:00 PM, Corinna Vinschen wrote:
> > On Feb 15 20:33, Michael Haubenwallner wrote:
> >> On 2/15/19 7:35 PM, Andrey Repin wrote:
> >>> I have a different question though. The commands you presented in the OP
> >>> look like you aren't using Cygwin-suppied OpenSSL binary.
> >>
> >> This is true: The Cygwin distro does provide openssl-1.0.2p, but I'm building
> >> the *portable* Gentoo Prefix distro on Cygwin, with openssl-1.1.1a already.
> >> But whenever the Cygwin distro would bump to openssl-1.1, the same problem
> >> would arise there as well, just becomes noticed much later probably.
> >
> > Does OpenSSH build with OpenSSL 1.1.1a these days?  If so, we might
> > want to switch OpenSSL finally, too.
>
> Well, OpenSSH-7.9_p1 maybe does, according to https://bugs.gentoo.org/592578

I believe OpenSSH sidestepped OpenSSL. That is, it no longer depends
on it in a way that things matter. I don't recall the details.

As far as building OpenSSH against OpenSSL 1.1, see
https://github.com/openssh/openssh-portable/pull/48 .

Also see https://wiki.openssl.org/index.php/OpenSSL_1.1.0_Changes,
which documents some [former] incompatibilities between other programs
and OpenSSL.

Jeff

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: textmode for stdout, what is "correct" now?

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 5591 bytes --]

On Feb 18 12:47, Michael Haubenwallner wrote:
> On 2/18/19 11:26 AM, Corinna Vinschen wrote:
> > On Feb 18 10:40, Michael Haubenwallner wrote:
> >> On 2/16/19 6:43 PM, Corinna Vinschen wrote:
> >>> I really miss the problem you're trying to solve here.  Why should an
> >>> application setting O_BINARY explicitely revert this decision on the
> >>> same file descriptor?  That doesn't make sense.
> >>
> >> Well, it's not necessarily about really switching binary mode on and off,
> >> it's more about avoiding breakage when applications try to intuitively
> >> follow the original API, even if that actually causes the call to
> >> setmode(fd, O_TEXT) to be redundant.
> >>
> >> OTOH, this question also would apply to native Win32 applications, so why
> >> do people call setmode(fd, O_TEXT) with any DOS based platform at all?
> >>
> >> IMO, unfortunately we're not in a position to modify the intention of the
> >> original API.  And finally, I do want to stop discussions like this one
> >> with application developers like openssl, as soon as we can argue like:
> >> "Cygwin does not use \r internally, but does support text mode mounts,
> >> so we had to invent the Cygwin text mode, which may or may not use \r.
> >> Hence you get the Cygwin text mode with O_TEXT, and if you really are
> >> in some 'unix2dos' position, please use the new O_DOSTEXT mode instead."
> >>
> >> However, agreed this does not seem to be trivial to implement.  Yet I
> >> will look into it when there is a chance for a patches to be accepted.
> > 
> > Bottom line:
> > 
> > - Make O_TEXT equivalent to O_BINARY on the API level so Cygwin
> >   actually uses binary mode on open(O_TEXT) and setmode(O_TEXT).
> 
> No, O_TEXT is neither equal to O_BINARY nor to O_DOSTEXT - it's something
> in between.  My first ideas are either (O_BINARY|O_DOSTEXT) or another bit.
> 
> > - Make O_DOSTEXT equivalent to the former O_TEXT.
> 
> Yes.
> 
> > Result: we use binary mode even with tools explicitely specifying O_TEXT.
> 
> No, not binary mode. It's text mode with \r being allowed rather than forced.

You lost me here.  Reading in O_TEXT mode already does not require \r\n,
it just allows \r\n as well as \n as line ending.  Given that, I don't
see a reason to add O_DOSTEXT.  What would it do differently?  Enforcing
\r\n line endings in input?

> Just stumbled over the distinction between readmode and writemode:
> What's up with that?

automode.o and textreadmode.o are just conveniences.  If you link an
application with them, descriptors the app opened O_RDONLY are in O_TEXT
mode automatically, and descriptors opened O_WRONLY are O_BINARY
(automode.o) or depending on mount mode (textreadmode.o) automatically.
You can't mix O_TEXT and O_BINARY on O_RDWR descriptors.

> Unless binary mode, reading always could be done in dostext mode.
> Here the default is to link without them, and the opposite of binmode.c
> is to not use anything, hence the text*mode should be O_DOSTEXT.

I fail to scan this paragraph, sorry.  Are you still taking about
the *mode.c files?

> > - How do you avoid breakage of existing tools which have been written to
> >   work explicitely with certain DOS formatted text file and use O_TEXT
> >   for that?
> > 
> > The answer to the last one could be using a new version check like the
> > ones already in include/cygwin/version.h.  Existing tools and libs keep
> > the current behaviour.  Only newly built binaries get the new behaviour.
> 
> Exactly. And for the check:
> For dostext mode: ifdef O_DOSTEXT: use O_DOSTEXT, otherways use O_TEXT.
> For cygtext mode: ifdef O_DOSTEXT: use O_TEXT, otherways avoid setmode.
> 
> > However, this still may result in breakage if the developer isn't aware
> > of this subtil change.  As much as I hate O_TEXT mode, there's a
> > pretty basic expectation how this is supposed to work.
> 
> Yes, but I do expect this in corner cases only, with unix2dos/dos2unix as
> the specific example.
> 
> OTOH, with setmode(fd, 0) coming to my mind: If that would denote the default
> (=cygwin text) mode, I can imagine we may convince (openssl) developers to use
> zero instead of O_TEXT, and everything could be fine without any Cygwin change.
> 
> Heck, this would feel like most obvious - even API wise, no?
> 
> Then we may want to add O_NOBINARY defined to zero as the only Cygwin change.

No, wait.  This is getting a bit out of hand.  The fact that we have to
handle two different read modes in Cygwin is already bad enough.  I'm
not really looking forward to add another read mode for which I don't
see an obvious need.  You don't really expect lots of upstream devs to
happily pick up on such a change with two new O_ open flags *just* for
Cygwin, do you?

You have two modes for input and three for output:

- input O_BINARY  -> only \n
- input O_TEXT    -> \n and \r\n

- output 0        -> generates \n or \r\n depending on mount mode
- output O_BINARY -> generates \n
- output O_TEXT   -> generates \r\n

I don't see how O_DOSTEXT comes into this picture.  There's no reason
for an enforcing \r\n input mode, and in output mode it won't differ
from O_TEXT, unless you define O_TEXT to be the same as O_BINARY in
future.  We can also already control per-app default settings by linking
apps against one of the *mode.o files.

What do we really gain by inventing two new Cygwin-only open flags,
other than restarting the old O_TEXT problems with upstream devs?


Corinna

-- 
Corinna Vinschen
Cygwin Maintainer

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: textmode for stdout, what is "correct" now?

[Michael Haubenwallner]

On 2/18/19 2:15 PM, Corinna Vinschen wrote:
> On Feb 18 12:47, Michael Haubenwallner wrote:
>> On 2/18/19 11:26 AM, Corinna Vinschen wrote:
>>> On Feb 18 10:40, Michael Haubenwallner wrote:
>>>> On 2/16/19 6:43 PM, Corinna Vinschen wrote:
>>>>> I really miss the problem you're trying to solve here.  Why should an
>>>>> application setting O_BINARY explicitely revert this decision on the
>>>>> same file descriptor?  That doesn't make sense.
>>>>
>>>> Well, it's not necessarily about really switching binary mode on and off,
>>>> it's more about avoiding breakage when applications try to intuitively
>>>> follow the original API, even if that actually causes the call to
>>>> setmode(fd, O_TEXT) to be redundant.
>>>>
>>>> OTOH, this question also would apply to native Win32 applications, so why
>>>> do people call setmode(fd, O_TEXT) with any DOS based platform at all?
>>>>
>>>> IMO, unfortunately we're not in a position to modify the intention of the
>>>> original API.  And finally, I do want to stop discussions like this one
>>>> with application developers like openssl, as soon as we can argue like:
>>>> "Cygwin does not use \r internally, but does support text mode mounts,
>>>> so we had to invent the Cygwin text mode, which may or may not use \r.
>>>> Hence you get the Cygwin text mode with O_TEXT, and if you really are
>>>> in some 'unix2dos' position, please use the new O_DOSTEXT mode instead."
>>>>
>>>> However, agreed this does not seem to be trivial to implement.  Yet I
>>>> will look into it when there is a chance for a patches to be accepted.

<<snip>>
> 
> No, wait.  This is getting a bit out of hand.  The fact that we have to
> handle two different read modes in Cygwin is already bad enough.  I'm
> not really looking forward to add another read mode for which I don't
> see an obvious need.  You don't really expect lots of upstream devs to
> happily pick up on such a change with two new O_ open flags *just* for
> Cygwin, do you?

Agreed, nope.

It turns out that setmode(fd, 0) is what I'm looking for,
and there is no need to change anything in the API.

So rather than not using setmode(), we should tell upstream to not use
O_TEXT but zero instead, which might be easier to accept for them.

> You have two modes for input and three for output:
> 
> - input O_BINARY  -> only \n
> - input O_TEXT    -> \n and \r\n
> 
> - output 0        -> generates \n or \r\n depending on mount mode
> - output O_BINARY -> generates \n
> - output O_TEXT   -> generates \r\n

Erm, even for input there is mode 0: strips \r depending on mount mode.
This actually is without using setmode() at all.
Shouldn't the default (mode 0) behave like O_TEXT for input?

Thanks!
/haubi/

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: textmode for stdout, what is "correct" now?

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 818 bytes --]

On Feb 18 15:45, Michael Haubenwallner wrote:
> On 2/18/19 2:15 PM, Corinna Vinschen wrote:
> > You have two modes for input and three for output:
> > 
> > - input O_BINARY  -> only \n
> > - input O_TEXT    -> \n and \r\n
> > 
> > - output 0        -> generates \n or \r\n depending on mount mode
> > - output O_BINARY -> generates \n
> > - output O_TEXT   -> generates \r\n
> 
> Erm, even for input there is mode 0: strips \r depending on mount mode.

Right.

> This actually is without using setmode() at all.
> Shouldn't the default (mode 0) behave like O_TEXT for input?

POSIXy binary mode is the only sane default.  Apps reading binary input
would start to misbehave otherwise.  That's why text mount mode should be
used with care :}


Corinna

-- 
Corinna Vinschen
Cygwin Maintainer

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]