raise(-1) has stopped returning an error recently

raise(-1) has stopped returning an error recently

[Duncan Roe]

Hi,

I stumbled across this when running 'make check' in the grep source directory.

test-raise failed when it didn't a few weeks ago. I cut it down to the STC at
the end of this email

Under Linux:
> 16:15:57$ ./test-raise
> raise(-1) returned -1

Under Cygwin:
> 16:16:51$ ./test-raise.exe
> raise(-1) returned 0

Hope this is nothing too complicated,

Cheers ... Duncan.

========================

#include <stdio.h>
#include <signal.h>

int
main (void)
{
  int retcod;

  retcod = raise (-1);
  printf("raise(-1) returned %d\n", retcod);
}

Re: raise(-1) has stopped returning an error recently

[Corinna Vinschen]

On Nov 22 16:20, Duncan Roe wrote:
> Hi,
> 
> I stumbled across this when running 'make check' in the grep source directory.
> 
> test-raise failed when it didn't a few weeks ago.

Pretty unlikely.  This code didn;'t change for ages.  I just checked
against Cygwin 3.2.0 and it also returns success.

> #include <stdio.h>
> #include <signal.h>
> 
> int
> main (void)
> {
>   int retcod;
> 
>   retcod = raise (-1);
>   printf("raise(-1) returned %d\n", retcod);
> }

Thanks for the STC.  There are actually two long-standing problems,
one in raise, calling pthread_kill even for non-threaded processes,
one in pthread_kill, not checking the signal number.

I'll patch them right away.


Thanks,
Corinna

Re: raise(-1) has stopped returning an error recently

[Corinna Vinschen]

On Nov 22 11:25, Corinna Vinschen via Cygwin wrote:
> On Nov 22 16:20, Duncan Roe wrote:
> > Hi,
> > 
> > I stumbled across this when running 'make check' in the grep source directory.
> > 
> > test-raise failed when it didn't a few weeks ago.
> 
> Pretty unlikely.  This code didn;'t change for ages.  I just checked
> against Cygwin 3.2.0 and it also returns success.
> 
> > #include <stdio.h>
> > #include <signal.h>
> > 
> > int
> > main (void)
> > {
> >   int retcod;
> > 
> >   retcod = raise (-1);
> >   printf("raise(-1) returned %d\n", retcod);
> > }
> 
> Thanks for the STC.  There are actually two long-standing problems,
> one in raise, calling pthread_kill even for non-threaded processes,
> one in pthread_kill, not checking the signal number.
> 
> I'll patch them right away.

I created a patchset:
https://sourceware.org/git?p=newlib-cygwin.git;a=commitdiff;h=d54d8f173d66
https://sourceware.org/git?p=newlib-cygwin.git;a=commitdiff;h=24bffff63eb7
https://sourceware.org/git?p=newlib-cygwin.git;a=commitdiff;h=afb7c557d2af

Please test the latest developer snapshot from http://cygwin.com/snapshots/


Thanks,
Corinna

Re: raise(-1) has stopped returning an error recently

[Duncan Roe]

On Mon, Nov 22, 2021 at 02:06:19PM +0100, cygwin wrote:
> On Nov 22 11:25, Corinna Vinschen via Cygwin wrote:
> > On Nov 22 16:20, Duncan Roe wrote:
> > > Hi,
> > >
> > > I stumbled across this when running 'make check' in the grep source directory.
> > >
> > > test-raise failed when it didn't a few weeks ago.
> >
> > Pretty unlikely.  This code didn;'t change for ages.  I just checked
> > against Cygwin 3.2.0 and it also returns success.
> >
> > > #include <stdio.h>
> > > #include <signal.h>
> > >
> > > int
> > > main (void)
> > > {
> > >   int retcod;
> > >
> > >   retcod = raise (-1);
> > >   printf("raise(-1) returned %d\n", retcod);
> > > }
> >
> > Thanks for the STC.  There are actually two long-standing problems,
> > one in raise, calling pthread_kill even for non-threaded processes,
> > one in pthread_kill, not checking the signal number.
> >
> > I'll patch them right away.
>
> I created a patchset:
> https://sourceware.org/git?p=newlib-cygwin.git;a=commitdiff;h=d54d8f173d66
> https://sourceware.org/git?p=newlib-cygwin.git;a=commitdiff;h=24bffff63eb7
> https://sourceware.org/git?p=newlib-cygwin.git;a=commitdiff;h=afb7c557d2af
>
> Please test the latest developer snapshot from http://cygwin.com/snapshots/
>
>
> Thanks,
> Corinna

Thanks, that fixed it.

Btw to whoever maintains grep for cygwin: 'make check' should pass on
next release (I patched out the surrogate-pair failre).

Cheers ... Duncan.

Re: raise(-1) has stopped returning an error recently

[Corinna Vinschen]

On Nov 23 19:27, Duncan Roe wrote:
> On Mon, Nov 22, 2021 at 02:06:19PM +0100, cygwin wrote:
> > On Nov 22 11:25, Corinna Vinschen via Cygwin wrote:
> > > On Nov 22 16:20, Duncan Roe wrote:
> > > > #include <stdio.h>
> > > > #include <signal.h>
> > > >
> > > > int
> > > > main (void)
> > > > {
> > > >   int retcod;
> > > >
> > > >   retcod = raise (-1);
> > > >   printf("raise(-1) returned %d\n", retcod);
> > > > }
> > >
> > > Thanks for the STC.  There are actually two long-standing problems,
> > > one in raise, calling pthread_kill even for non-threaded processes,
> > > one in pthread_kill, not checking the signal number.
> > >
> > > I'll patch them right away.
> >
> > I created a patchset:
> > https://sourceware.org/git?p=newlib-cygwin.git;a=commitdiff;h=d54d8f173d66
> > https://sourceware.org/git?p=newlib-cygwin.git;a=commitdiff;h=24bffff63eb7
> > https://sourceware.org/git?p=newlib-cygwin.git;a=commitdiff;h=afb7c557d2af
> >
> > Please test the latest developer snapshot from http://cygwin.com/snapshots/
> 
> Thanks, that fixed it.

Thanks for testing.

> Btw to whoever maintains grep for cygwin: 'make check' should pass on
> next release (I patched out the surrogate-pair failre).

Great!  Brian?


Thanks,
Corinna

Re: raise(-1) has stopped returning an error recently

[Brian Inglis]

On 2021-11-23 02:50, Corinna Vinschen via Cygwin wrote:
> On Nov 23 19:27, Duncan Roe wrote:
>> On Mon, Nov 22, 2021 at 02:06:19PM +0100, cygwin wrote:
>>> On Nov 22 11:25, Corinna Vinschen via Cygwin wrote:
>>>> On Nov 22 16:20, Duncan Roe wrote:
>>>>> #include <stdio.h>
>>>>> #include <signal.h>
>>>>>
>>>>> int
>>>>> main (void)
>>>>> {
>>>>>    int retcod;
>>>>>
>>>>>    retcod = raise (-1);
>>>>>    printf("raise(-1) returned %d\n", retcod);
>>>>> }
>>>>
>>>> Thanks for the STC.  There are actually two long-standing problems,
>>>> one in raise, calling pthread_kill even for non-threaded processes,
>>>> one in pthread_kill, not checking the signal number.
>>>>
>>>> I'll patch them right away.
>>>
>>> I created a patchset:
>>> https://sourceware.org/git?p=newlib-cygwin.git;a=commitdiff;h=d54d8f173d66
>>> https://sourceware.org/git?p=newlib-cygwin.git;a=commitdiff;h=24bffff63eb7
>>> https://sourceware.org/git?p=newlib-cygwin.git;a=commitdiff;h=afb7c557d2af
>>>
>>> Please test the latest developer snapshot from http://cygwin.com/snapshots/
>>
>> Thanks, that fixed it.
> 
> Thanks for testing.
> 
>> Btw to whoever maintains grep for cygwin: 'make check' should pass on
>> next release (I patched out the surrogate-pair failre).

I had no problems with test-raise last release.
I did with surrogate pairs but after spending too much time on all the 
test infrastructure around that, decided it was a low probability event, 
and wait until anyone complains to refer it upstream.
Do Cygwin and/or Windows support surrogate pairs in UTF-8?

-- 
Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada

This email may be disturbing to some readers as it contains
too much technical detail. Reader discretion is advised.
[Data in binary units and prefixes, physical quantities in SI.]

Re: raise(-1) has stopped returning an error recently

[Duncan Roe]

On Tue, Nov 23, 2021 at 11:18:25AM -0700, Brian Inglis wrote:
> On 2021-11-23 02:50, Corinna Vinschen via Cygwin wrote:
> > On Nov 23 19:27, Duncan Roe wrote:
> > > On Mon, Nov 22, 2021 at 02:06:19PM +0100, cygwin wrote:
> > > > On Nov 22 11:25, Corinna Vinschen via Cygwin wrote:
> > > > > On Nov 22 16:20, Duncan Roe wrote:
> > > > > > #include <stdio.h>
> > > > > > #include <signal.h>
> > > > > >
> > > > > > int
> > > > > > main (void)
> > > > > > {
> > > > > >    int retcod;
> > > > > >
> > > > > >    retcod = raise (-1);
> > > > > >    printf("raise(-1) returned %d\n", retcod);
> > > > > > }
> > > > >
> > > > > Thanks for the STC.  There are actually two long-standing problems,
> > > > > one in raise, calling pthread_kill even for non-threaded processes,
> > > > > one in pthread_kill, not checking the signal number.
> > > > >
> > > > > I'll patch them right away.
> > > >
> > > > I created a patchset:
> > > > https://sourceware.org/git?p=newlib-cygwin.git;a=commitdiff;h=d54d8f173d66
> > > > https://sourceware.org/git?p=newlib-cygwin.git;a=commitdiff;h=24bffff63eb7
> > > > https://sourceware.org/git?p=newlib-cygwin.git;a=commitdiff;h=afb7c557d2af
> > > >
> > > > Please test the latest developer snapshot from http://cygwin.com/snapshots/
> > >
> > > Thanks, that fixed it.
> >
> > Thanks for testing.
> >
> > > Btw to whoever maintains grep for cygwin: 'make check' should pass on
> > > next release (I patched out the surrogate-pair failre).
>
> I had no problems with test-raise last release.

I don't remember having a problem with it even a few weeks ago.

> I did with surrogate pairs but after spending too much time on all the test
> infrastructure around that, decided it was a low probability event, and wait
> until anyone complains to refer it upstream.

I wasted time on that too. That's why I patched surrogate-pair to not do its 3rd
test if 'uname -s' indicates Cygwin.

For the full story, see https://debbugs.gnu.org/cgi/bugreport.cgi?bug=27555#5

> Do Cygwin and/or Windows support surrogate pairs in UTF-8?

There are 3 tests in surrogate-pair and only the 3rd one failed. So I guess
surrogate pairs in UTF-8 "mostly work".

Cheers ... Duncan.

Re: raise(-1) has stopped returning an error recently

[Corinna Vinschen]

On Nov 24 09:36, Duncan Roe wrote:
> On Tue, Nov 23, 2021 at 11:18:25AM -0700, Brian Inglis wrote:
> > > On Nov 23 19:27, Duncan Roe wrote:
> > > > Btw to whoever maintains grep for cygwin: 'make check' should pass on
> > > > next release (I patched out the surrogate-pair failre).
> >
> > I had no problems with test-raise last release.
> 
> I don't remember having a problem with it even a few weeks ago.
> 
> > I did with surrogate pairs but after spending too much time on all the test
> > infrastructure around that, decided it was a low probability event, and wait
> > until anyone complains to refer it upstream.
> 
> I wasted time on that too. That's why I patched surrogate-pair to not do its 3rd
> test if 'uname -s' indicates Cygwin.
> 
> For the full story, see https://debbugs.gnu.org/cgi/bugreport.cgi?bug=27555#5

What is that "permanent restriction" in Cygwin?  Is that something we
could fix or something unfixable?  Did you try to debug Cygwin in terms
of that problem?  If not, could you extract a reduced, very simple
stand-alone testcase for further debugging?

> > Do Cygwin and/or Windows support surrogate pairs in UTF-8?

You mean UTF-16.  UTF-8 doesn't know surrogate pairs, UTF-16 does.
Originally there was UCS-2, 16 bits, with only 65536 code points.
However, Unicode left the BMP already with version 2.0 in 1996, so
UTF-16 and surrogate pairs became necessary.  Windows as well as Cygwin
support them.

> There are 3 tests in surrogate-pair and only the 3rd one failed. So I guess
> surrogate pairs in UTF-8 "mostly work".

UTF-16.  The surrogate stuff is evil at times.  Have a look at the
__utf8_wctomb function in
https://sourceware.org/git/?p=newlib-cygwin.git;a=blob;f=newlib/libc/stdlib/wctomb_r.c
Lone surrogate halfs in an input stream are a problem, for instance.


Corinna

Re: raise(-1) has stopped returning an error recently

[Brian Inglis]

[-- Attachment #1: Type: text/plain, Size: 2897 bytes --]

On 2021-11-24 02:25, Corinna Vinschen via Cygwin wrote:
> On Nov 24 09:36, Duncan Roe wrote:
>> On Tue, Nov 23, 2021 at 11:18:25AM -0700, Brian Inglis wrote:
>>>> On Nov 23 19:27, Duncan Roe wrote:
>>>>> Btw to whoever maintains grep for cygwin: 'make check' should pass on
>>>>> next release (I patched out the surrogate-pair failre).
>>>
>>> I had no problems with test-raise last release.
>>
>> I don't remember having a problem with it even a few weeks ago.
>>
>>> I did with surrogate pairs but after spending too much time on all the test
>>> infrastructure around that, decided it was a low probability event, and wait
>>> until anyone complains to refer it upstream.
>>
>> I wasted time on that too. That's why I patched surrogate-pair to not do its 3rd
>> test if 'uname -s' indicates Cygwin.
>>
>> For the full story, see https://debbugs.gnu.org/cgi/bugreport.cgi?bug=27555#5
> 
> What is that "permanent restriction" in Cygwin?  Is that something we
> could fix or something unfixable?  Did you try to debug Cygwin in terms
> of that problem?  If not, could you extract a reduced, very simple
> stand-alone testcase for further debugging?
> 
>>> Do Cygwin and/or Windows support surrogate pairs in UTF-8?
> 
> You mean UTF-16.  UTF-8 doesn't know surrogate pairs, UTF-16 does.
> Originally there was UCS-2, 16 bits, with only 65536 code points.
> However, Unicode left the BMP already with version 2.0 in 1996, so
> UTF-16 and surrogate pairs became necessary.  Windows as well as Cygwin
> support them.

How does Cygwin support UTF-16 locales with surrogate pairs?
Are they the "native" locales inherited from Windows if others are not 
specified e.g. UTF-8, some OEM SBCS or MBCS?

>> There are 3 tests in surrogate-pair and only the 3rd one failed. So I guess
>> surrogate pairs in UTF-8 "mostly work".
> 
> UTF-16.  The surrogate stuff is evil at times.  Have a look at the
> __utf8_wctomb function in
> https://sourceware.org/git/?p=newlib-cygwin.git;a=blob;f=newlib/libc/stdlib/wctomb_r.c
> Lone surrogate halfs in an input stream are a problem, for instance.

Thus the confusion with grep surrogate pair tests which appear to be 
running under a UTF-8 locale: see attached surrogate pair extract from 
cygport --debug grep.cygport check.

Trying to rerun cygport build most tests are now "skipped test: failed 
to find an adequate shell SKIP ... (exit status: 77)"! Something more 
may have changed (in gnulib?) to invalidate Cygwin shell(s) in something 
updated since that grep release in August, as I am getting the same 
skipped tests under GitHub CI, although it could just be that something 
expects say bash > 4.4 or even >= 5!

-- 
Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada

This email may be disturbing to some readers as it contains
too much technical detail. Reader discretion is advised.
[Data in binary units and prefixes, physical quantities in SI.]

[-- Attachment #2: grep-check-surrogate-pair.log --]
[-- Type: text/plain, Size: 5657 bytes --]

FAIL: surrogate-pair
FAIL: surrogate-pair
====================

+ initial_cwd_=$HOME/src/cygwin/grep/grep-3.7-2.x86_64/build/tests
+ testdir_prefix_
+ printf gt
+ pfx_=gt
+ mktempd_ $HOME/src/cygwin/grep/grep-3.7-2.x86_64/build/tests gt-surrogate-pair.XXXX
+ destdir_=$HOME/src/cygwin/grep/grep-3.7-2.x86_64/build/tests
+ template_=gt-surrogate-pair.XXXX
+ MAX_TRIES_=4
+ destdir_slash_=$HOME/src/cygwin/grep/grep-3.7-2.x86_64/build/tests/
+ unset TMPDIR
+ d=$HOME/src/cygwin/grep/grep-3.7-2.x86_64/build/tests/gt-surrogate-pair.QL4K
+ :
+ test -d $HOME/src/cygwin/grep/grep-3.7-2.x86_64/build/tests/gt-surrogate-pair.QL4K
+ ls -dgo $HOME/src/cygwin/grep/grep-3.7-2.x86_64/build/tests/gt-surrogate-pair.QL4K
+ perms=drwx------+ 1 0 Aug 31 01:38 $HOME/src/cygwin/grep/grep-3.7-2.x86_64/build/tests/gt-surrogate-pair.QL4K
+ :
+ echo $HOME/src/cygwin/grep/grep-3.7-2.x86_64/build/tests/gt-surrogate-pair.QL4K
+ return
+ test_dir_=$HOME/src/cygwin/grep/grep-3.7-2.x86_64/build/tests/gt-surrogate-pair.QL4K
+ cd $HOME/src/cygwin/grep/grep-3.7-2.x86_64/build/tests/gt-surrogate-pair.QL4K
+ builddir=..
+ export srcdir builddir
+ gl_init_sh_nl_=

+ IFS= 	

+ expr 1 + 128
+ eval trap 'Exit 129' 1
+ trap Exit 129 1
+ expr 2 + 128
+ eval trap 'Exit 130' 2
+ trap Exit 130 2
+ expr 3 + 128
+ eval trap 'Exit 131' 3
+ trap Exit 131 3
+ expr 13 + 128
+ eval trap 'Exit 141' 13
+ trap Exit 141 13
+ expr 15 + 128
+ eval trap 'Exit 143' 15
+ trap Exit 143 15
+ trap remove_tmp_ 0
+ path_prepend_ ../src
+ test 1 != 0
+ path_dir_=../src
+ abs_path_dir_=$HOME/src/cygwin/grep/grep-3.7-2.x86_64/build/tests/../src
+ PATH=$HOME/src/cygwin/grep/grep-3.7-2.x86_64/build/tests/../src:$HOME/src/cygwin/grep/grep-3.7-2.x86_64/build/src:./src:$HOME/bin:$HOME/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/sbin:...
+ create_exe_shims_ $HOME/src/cygwin/grep/grep-3.7-2.x86_64/build/tests/../src
+ return 0
+ shift
+ test 0 != 0
+ export PATH
+ require_en_utf8_locale_
+ path_prepend_ .
+ test 1 != 0
+ path_dir_=.
+ abs_path_dir_=$HOME/src/cygwin/grep/grep-3.7-2.x86_64/build/tests/.
+ PATH=$HOME/src/cygwin/grep/grep-3.7-2.x86_64/build/tests/.:$HOME/src/cygwin/grep/grep-3.7-2.x86_64/build/tests/../src:$HOME/src/cygwin/grep/grep-3.7-2.x86_64/build/src:./src:$HOME/bin:$HOME/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/sbin:...
+ create_exe_shims_ $HOME/src/cygwin/grep/grep-3.7-2.x86_64/build/tests/.
+ return 0
+ shift
+ test 0 != 0
+ export PATH
+ get-mb-cur-max en_US.UTF-8
+ require_compiled_in_MB_support
+ require_en_utf8_locale_
+ path_prepend_ .
+ test 1 != 0
+ path_dir_=.
+ abs_path_dir_=$HOME/src/cygwin/grep/grep-3.7-2.x86_64/build/tests/.
+ PATH=$HOME/src/cygwin/grep/grep-3.7-2.x86_64/build/tests/.:$HOME/src/cygwin/grep/grep-3.7-2.x86_64/build/tests/.:$HOME/src/cygwin/grep/grep-3.7-2.x86_64/build/tests/../src:$HOME/src/cygwin/grep/grep-3.7-2.x86_64/build/src:./src:$HOME/bin:$HOME/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/sbin:...
+ create_exe_shims_ $HOME/src/cygwin/grep/grep-3.7-2.x86_64/build/tests/.
+ return 0
+ shift
+ test 0 != 0
+ export PATH
+ get-mb-cur-max en_US.UTF-8
+ printf é
+ LC_ALL=en_US.UTF-8 grep [[:lower:]]
é
+ fail=0
+ printf /360/220/220/205
+ s_pair=𐐅
+ printf %s/n 𐐅
+ LC_ALL=en_US.UTF-8
+ export LC_ALL
+ returns_ 1 grep -i anything-else in
+ compare /dev/null out
+ compare_dev_null_ /dev/null out
+ test 2 = 2
+ test x/dev/null = x/dev/null
+ test -s out
+ return 0
+ return 0
+ grep . in
+ cat out
+ cat err
+ io_pair=𐐅:
+ grep --file=in in
+ fail=1
+ compare out in
+ compare_dev_null_ out in
+ test 2 = 2
+ test xout = x/dev/null
+ test xin = x/dev/null
+ return 2
+ compare_ out in
+ diff -u out in
--- out	2021-08-31 01:38:22.950752100 -0600
+++ in	2021-08-31 01:38:22.591357000 -0600
@@ -0,0 +1 @@
+𐐅
+ fail=1
+ grep --file=in -i in
+ fail=1
+ compare out in
+ compare_dev_null_ out in
+ test 2 = 2
+ test xout = x/dev/null
+ test xin = x/dev/null
+ return 2
+ compare_ out in
+ diff -u out in
--- out	2021-08-31 01:38:23.138262500 -0600
+++ in	2021-08-31 01:38:22.591357000 -0600
@@ -0,0 +1 @@
+𐐅
+ fail=1
+ grep --file=in -E in
+ fail=1
+ compare out in
+ compare_dev_null_ out in
+ test 2 = 2
+ test xout = x/dev/null
+ test xin = x/dev/null
+ return 2
+ compare_ out in
+ diff -u out in
--- out	2021-08-31 01:38:23.325773300 -0600
+++ in	2021-08-31 01:38:22.591357000 -0600
@@ -0,0 +1 @@
+𐐅
+ fail=1
+ grep --file=in -F in
+ fail=1
+ compare out in
+ compare_dev_null_ out in
+ test 2 = 2
+ test xout = x/dev/null
+ test xin = x/dev/null
+ return 2
+ compare_ out in
+ diff -u out in
--- out	2021-08-31 01:38:23.528911300 -0600
+++ in	2021-08-31 01:38:22.591357000 -0600
@@ -0,0 +1 @@
+𐐅
+ fail=1
+ grep --file=in -iE in
+ fail=1
+ compare out in
+ compare_dev_null_ out in
+ test 2 = 2
+ test xout = x/dev/null
+ test xin = x/dev/null
+ return 2
+ compare_ out in
+ diff -u out in
--- out	2021-08-31 01:38:23.716423100 -0600
+++ in	2021-08-31 01:38:22.591357000 -0600
@@ -0,0 +1 @@
+𐐅
+ fail=1
+ grep --file=in -iF in
+ fail=1
+ compare out in
+ compare_dev_null_ out in
+ test 2 = 2
+ test xout = x/dev/null
+ test xin = x/dev/null
+ return 2
+ compare_ out in
+ diff -u out in
--- out	2021-08-31 01:38:23.919559100 -0600
+++ in	2021-08-31 01:38:22.591357000 -0600
@@ -0,0 +1 @@
+𐐅
+ fail=1
+ Exit 1
+ set +e
+ exit 1
+ exit 1
+ remove_tmp_
+ __st=1
+ cleanup_
+ :
+ test  = yes
+ cd $HOME/src/cygwin/grep/grep-3.7-2.x86_64/build/tests
+ chmod -R u+rwx $HOME/src/cygwin/grep/grep-3.7-2.x86_64/build/tests/gt-surrogate-pair.QL4K
+ rm -rf $HOME/src/cygwin/grep/grep-3.7-2.x86_64/build/tests/gt-surrogate-pair.QL4K
+ exit 1
FAIL surrogate-pair (exit status: 1)

Re: raise(-1) has stopped returning an error recently

[Corinna Vinschen]

On Nov 24 11:01, Brian Inglis via Cygwin wrote:
> On 2021-11-24 02:25, Corinna Vinschen via Cygwin wrote:
> > > On Tue, Nov 23, 2021 at 11:18:25AM -0700, Brian Inglis wrote:
> > > > Do Cygwin and/or Windows support surrogate pairs in UTF-8?
> > 
> > You mean UTF-16.  UTF-8 doesn't know surrogate pairs, UTF-16 does.
> > Originally there was UCS-2, 16 bits, with only 65536 code points.
> > However, Unicode left the BMP already with version 2.0 in 1996, so
> > UTF-16 and surrogate pairs became necessary.  Windows as well as Cygwin
> > support them.
> 
> How does Cygwin support UTF-16 locales with surrogate pairs?

UTF-16 locales?  There's no such thing.  UTF-16 is just the 16 bit
representation for Unicode, and as such, is independent of the locale.
On the user side, Cygwin only supports UTF-8 as Unicode representation.
Internally you can then convert them to wchar_t which is UTF-16.

> Are they the "native" locales inherited from Windows if others are not
> specified e.g. UTF-8, some OEM SBCS or MBCS?

Just try `locale -av' and you'll see all supported locales and their
respective default codeset.  All of them can be used with .utf8
specifier to use UTF-8 instead of the default codeset.  Some of them
use UTF-8 as default codeset anyway, e. g., fa_IR or yo_NG.

> > > There are 3 tests in surrogate-pair and only the 3rd one failed. So I guess
> > > surrogate pairs in UTF-8 "mostly work".
> > 
> > UTF-16.  The surrogate stuff is evil at times.  Have a look at the
> > __utf8_wctomb function in
> > https://sourceware.org/git/?p=newlib-cygwin.git;a=blob;f=newlib/libc/stdlib/wctomb_r.c
> > Lone surrogate halfs in an input stream are a problem, for instance.
> 
> Thus the confusion with grep surrogate pair tests which appear to be running
> under a UTF-8 locale: see attached surrogate pair extract from cygport
> --debug grep.cygport check.

An STC in plain C might be helpful.


Corinna

Re: raise(-1) has stopped returning an error recently

[Brian Inglis]

On 2021-11-25 05:54, Corinna Vinschen via Cygwin wrote:
> On Nov 24 11:01, Brian Inglis via Cygwin wrote:
>> On 2021-11-24 02:25, Corinna Vinschen via Cygwin wrote:
>>>> On Tue, Nov 23, 2021 at 11:18:25AM -0700, Brian Inglis wrote:
>>>>> Do Cygwin and/or Windows support surrogate pairs in UTF-8?
>>>
>>> You mean UTF-16.  UTF-8 doesn't know surrogate pairs, UTF-16 does.
>>> Originally there was UCS-2, 16 bits, with only 65536 code points.
>>> However, Unicode left the BMP already with version 2.0 in 1996, so
>>> UTF-16 and surrogate pairs became necessary.  Windows as well as Cygwin
>>> support them.
>>
>> How does Cygwin support UTF-16 locales with surrogate pairs?
> 
> UTF-16 locales?  There's no such thing.  UTF-16 is just the 16 bit
> representation for Unicode, and as such, is independent of the locale.
> On the user side, Cygwin only supports UTF-8 as Unicode representation.
> Internally you can then convert them to wchar_t which is UTF-16.
> 
>> Are they the "native" locales inherited from Windows if others are not
>> specified e.g. UTF-8, some OEM SBCS or MBCS?
> 
> Just try `locale -av' and you'll see all supported locales and their
> respective default codeset.  All of them can be used with .utf8
> specifier to use UTF-8 instead of the default codeset.  Some of them
> use UTF-8 as default codeset anyway, e. g., fa_IR or yo_NG.
> 
>>>> There are 3 tests in surrogate-pair and only the 3rd one failed. So I guess
>>>> surrogate pairs in UTF-8 "mostly work".
>>>
>>> UTF-16.  The surrogate stuff is evil at times.  Have a look at the
>>> __utf8_wctomb function in
>>> https://sourceware.org/git/?p=newlib-cygwin.git;a=blob;f=newlib/libc/stdlib/wctomb_r.c
>>> Lone surrogate halfs in an input stream are a problem, for instance.
>>
>> Thus the confusion with grep surrogate pair tests which appear to be running
>> under a UTF-8 locale: see attached surrogate pair extract from cygport
>> --debug grep.cygport check.
> 
> An STC in plain C might be helpful.

I think I might finally have got the point of the test, not knowing much 
about legacy UTF-16 UCS encoding nor surrogate pairs.

 From what I can see:

𐐅  U+010405  f0 90 90 85  DESERET CAPITAL LETTER LONG OO

fails to match itself, presumably others do also.

Presumably this is converted internally on some platforms, including 
Cygwin, to a UTF-16 surrogate pair, and a grep comparison fails, 
although a bash comparison succeeds.

$ printf '\U10405\n' | iconv -f utf-8 -t utf-16be | xxd -g2
00000000: d801 dc05 000a
$ printf '\U10405\n' > t
$ grep -f t t; echo $?
1
$ oo=`printf '\U10405\n'`; [ $oo = $oo ] && echo same || echo diff
same

-- 
Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada

This email may be disturbing to some readers as it contains
too much technical detail. Reader discretion is advised.
[Data in binary units and prefixes, physical quantities in SI.]

Re: raise(-1) has stopped returning an error recently

[Duncan Roe]

On Wed, Nov 24, 2021 at 11:01:46AM -0700, cygwin wrote: [...]
> Trying to rerun cygport build most tests are now "skipped test: failed to
> find an adequate shell SKIP ... (exit status: 77)"! Something more may have
> changed (in gnulib?) to invalidate Cygwin shell(s) in something updated
> since that grep release in August, as I am getting the same skipped tests
> under GitHub CI, although it could just be that something expects say bash >
> 4.4 or even >= 5!
> 
They all pass for me in a directory tree created by 'make dist' under Linux.
Maybe something changed in your test environment?

N.B. grep maintainer replaced my patch with a fancier one that creates new
surrogate-search test that Cygwin skips.

Cheers ... Duncan.

PASS: backref
PASS: backref-alt
PASS: backref-multibyte-slow
PASS: backref-word
PASS: backslash-dot
PASS: backslash-s-and-repetition-operators
PASS: backslash-s-vs-invalid-multibyte
big-hole: skipped test: expensive: disabled by default
SKIP: big-hole
big-match: skipped test: expensive: disabled by default
SKIP: big-match
PASS: binary-file-matches
PASS: bogus-wctob
PASS: bre
PASS: c-locale
PASS: case-fold-backref
PASS: case-fold-backslash-w
PASS: case-fold-char-class
PASS: case-fold-char-range
PASS: case-fold-char-type
PASS: case-fold-titlecase
PASS: char-class-multibyte
PASS: char-class-multibyte2
PASS: context-0
PASS: count-newline
PASS: dfa-coverage
PASS: dfa-heap-overrun
PASS: dfa-infloop
PASS: dfa-invalid-utf8
PASS: dfaexec-multibyte
PASS: empty
PASS: empty-line
PASS: empty-line-mb
PASS: encoding-error
PASS: epipe
XFAIL: equiv-classes
PASS: ere
PASS: euc-mb
false-match-mb-non-utf8: skipped test: no support for the zh_CN.gb18030 locale
SKIP: false-match-mb-non-utf8
PASS: fedora
PASS: fgrep-infloop
PASS: fgrep-longest
PASS: file
PASS: filename-lineno.pl
PASS: fmbtest
PASS: foad1
PASS: grep-dev-null
PASS: grep-dev-null-out
PASS: grep-dir
ratio=0
PASS: hash-collision-perf
PASS: help-version
PASS: high-bit-range
PASS: in-eq-out-infloop
PASS: include-exclude
PASS: inconsistent-range
PASS: initial-tab
PASS: invalid-multibyte-infloop
PASS: khadafy
PASS: kwset-abuse
long-line-vs-2GiB-read: skipped test: expensive: disabled by default
SKIP: long-line-vs-2GiB-read
long-pattern-perf: skipped test: expensive: disabled by default
SKIP: long-pattern-perf
many-regex-performance: skipped test: expensive: disabled by default
SKIP: many-regex-performance
PASS: match-lines
PASS: max-count-overread
PASS: max-count-vs-context
PASS: mb-dot-newline
PASS: mb-non-UTF8-overrun
PASS: mb-non-UTF8-perf-Fw
mb-non-UTF8-performance: skipped test: expensive: disabled by default
SKIP: mb-non-UTF8-performance
PASS: mb-non-UTF8-word-boundary
 warning: \s failed to match \x0a in the en_US.UTF-8 locale
 warning: \s failed to match \x85 in the en_US.UTF-8 locale
 warning: \s failed to match \xe2\x80\x8b in the en_US.UTF-8 locale
 warning: \S mistakenly matched \xe2\x80\x8b in the en_US.UTF-8 locale
PASS: multibyte-white-space
PASS: multiple-begin-or-end-line
PASS: null-byte
PASS: options
pcre: skipped test: no PCRE support
SKIP: pcre
pcre-abort: skipped test: no PCRE support
SKIP: pcre-abort
pcre-context: skipped test: no PCRE support
SKIP: pcre-context
pcre-count: skipped test: no PCRE support
SKIP: pcre-count
pcre-infloop: skipped test: no PCRE support
SKIP: pcre-infloop
pcre-invalid-utf8-infloop: skipped test: no PCRE support
SKIP: pcre-invalid-utf8-infloop
pcre-invalid-utf8-input: skipped test: no PCRE support
SKIP: pcre-invalid-utf8-input
pcre-jitstack: skipped test: no PCRE support
SKIP: pcre-jitstack
pcre-o: skipped test: no PCRE support
SKIP: pcre-o
pcre-utf8: skipped test: no PCRE support
SKIP: pcre-utf8
pcre-w: skipped test: no PCRE support
SKIP: pcre-w
pcre-wx-backref: skipped test: no PCRE support
SKIP: pcre-wx-backref
pcre-z: skipped test: no PCRE support
SKIP: pcre-z
PASS: posix-bracket
PASS: prefix-of-multibyte
PASS: proc
PASS: r-dot
PASS: repetition-overflow
PASS: reversed-range-endpoints
PASS: sjis-mb
PASS: skip-device
PASS: skip-read
PASS: spencer1
PASS: spencer1-locale
PASS: stack-overflow
PASS: status
PASS: surrogate-pair
surrogate-search: skipped test: surrogate-pair search string on Cygwin, where it cannot work
SKIP: surrogate-search
PASS: symlink
triple-backref: expect malfunction on glibc systems due to https://sourceware.org/bugzilla/show_bug.cgi?id=11053
XFAIL: triple-backref
PASS: turkish-I
PASS: turkish-I-without-dot
turkish-eyes: skipped test: your tr_TR.UTF-8 locale appears to be broken
SKIP: turkish-eyes
PASS: two-chars
PASS: two-files
PASS: unibyte-binary
PASS: unibyte-bracket-expr
PASS: unibyte-negated-circumflex
PASS: utf8-bracket
PASS: warn-char-classes
PASS: word-delim-multibyte
PASS: word-multi-file
PASS: word-multibyte
PASS: write-error-msg
PASS: yesno
PASS: z-anchor-newline
============================================================================
Testsuite summary for GNU grep 3.7.42-9c15
============================================================================
# TOTAL: 118
# PASS:  94
# SKIP:  22
# XFAIL: 2
# FAIL:  0
# XPASS: 0
# ERROR: 0
============================================================================
## ---------------------------- Gnulib tests ---------------------------- ##
PASS: test-alignof.exe
PASS: test-alloca-opt.exe
PASS: test-argmatch.exe
PASS: test-arpa_inet.exe
PASS: test-binary-io.sh
PASS: test-bind.exe
PASS: test-bitrotate.exe
PASS: test-btowc1.sh
PASS: test-btowc2.sh
PASS: test-c-ctype.exe
PASS: test-c-stack.sh
PASS: test-c-stack2.sh
PASS: test-c-strcase.sh
PASS: test-calloc-gnu.exe
PASS: test-chdir.exe
PASS: test-cloexec.exe
PASS: test-close.exe
PASS: test-connect.exe
PASS: test-ctype.exe
PASS: test-dfa-invalid-char-class.sh
PASS: test-dfa-invalid-merge.sh
PASS: test-dfa-match.sh
PASS: test-dirent.exe
PASS: test-dup.exe
PASS: test-dup2.exe
PASS: test-dynarray.exe
PASS: test-environ.exe
PASS: test-errno.exe
PASS: test-exclude1.sh
PASS: test-exclude2.sh
PASS: test-exclude3.sh
PASS: test-exclude4.sh
PASS: test-exclude5.sh
PASS: test-exclude6.sh
PASS: test-exclude7.sh
PASS: test-exclude8.sh
PASS: test-fchdir.exe
PASS: test-fcntl-h.exe
PASS: test-fcntl-safer.exe
PASS: test-fcntl.exe
PASS: test-fdopen.exe
PASS: test-fdopendir.exe
PASS: test-fgetc.exe
PASS: test-float.exe
PASS: test-fnmatch-h.exe
PASS: test-fnmatch.exe
PASS: test-fopen-gnu.exe
PASS: test-fopen.exe
PASS: test-fpending.sh
PASS: test-fputc.exe
PASS: test-fread.exe
PASS: test-free.exe
PASS: test-fstat.exe
PASS: test-fstatat.exe
PASS: test-ftruncate.sh
PASS: test-fwrite.exe
PASS: test-getcwd-lgpl.exe
PASS: test-getdtablesize.exe
PASS: test-getopt-gnu.exe
PASS: test-getopt-posix.exe
PASS: test-getprogname.exe
PASS: test-gettimeofday.exe
PASS: test-hard-locale.exe
PASS: test-hash.exe
PASS: test-i-ring.exe
PASS: test-iconv-h.exe
PASS: test-iconv.exe
PASS: test-ignore-value.exe
PASS: test-inet_pton.exe
PASS: test-intprops.exe
PASS: test-inttostr.exe
PASS: test-inttypes.exe
PASS: test-ioctl.exe
PASS: test-isatty.exe
PASS: test-isblank.exe
PASS: test-iswblank.exe
PASS: test-iswdigit.sh
PASS: test-iswxdigit.sh
PASS: test-langinfo.exe
PASS: test-limits-h.exe
PASS: test-listen.exe
PASS: test-locale.exe
PASS: test-localeconv.exe
PASS: test-localename.exe
PASS: test-lseek.sh
PASS: test-lstat.exe
PASS: test-malloc-gnu.exe
PASS: test-malloca.exe
SKIP: test-mbscasecmp.sh
PASS: test-mbsinit.sh
PASS: test-mbsrtowcs1.sh
PASS: test-mbsrtowcs2.sh
PASS: test-mbsrtowcs3.sh
SKIP: test-mbsrtowcs4.sh
PASS: test-mbsstr1.exe
PASS: test-mbsstr2.sh
SKIP: test-mbsstr3.sh
PASS: test-memchr.exe
PASS: test-memchr2.exe
PASS: test-memrchr.exe
PASS: test-nanosleep.exe
PASS: test-netinet_in.exe
PASS: test-nl_langinfo.sh
PASS: test-nl_langinfo-mt.exe
PASS: test-open.exe
PASS: test-openat-safer.exe
PASS: test-openat.exe
PASS: test-pathmax.exe
PASS: test-perror.sh
PASS: test-perror2.exe
PASS: test-pipe.exe
PASS: test-pthread.exe
PASS: test-pthread-thread.exe
PASS: test-pthread_sigmask1.exe
PASS: test-pthread_sigmask2.exe
PASS: test-quotearg-simple.exe
PASS: test-raise.exe
PASS: test-rawmemchr.exe
PASS: test-read.exe
PASS: test-realloc-gnu.exe
PASS: test-reallocarray.exe
PASS: test-regex.exe
PASS: test-sched.exe
PASS: test-select.exe
PASS: test-select-in.sh
PASS: test-select-out.sh
PASS: test-setenv.exe
PASS: test-setlocale_null.exe
PASS: test-setlocale_null-mt-one.exe
PASS: test-setlocale_null-mt-all.exe
PASS: test-setlocale1.sh
PASS: test-setlocale2.sh
PASS: test-setsockopt.exe
PASS: test-sigaction.exe
PASS: test-signal-h.exe
PASS: test-sigprocmask.exe
PASS: test-sigsegv-catch-segv1.exe
PASS: test-sigsegv-catch-segv2.exe
PASS: test-sigsegv-catch-stackoverflow1.exe
PASS: test-sigsegv-catch-stackoverflow2.exe
PASS: test-sleep.exe
PASS: test-snprintf.exe
PASS: test-sockets.exe
PASS: test-stat.exe
PASS: test-stat-time.exe
PASS: test-stdalign.exe
PASS: test-stdbool.exe
PASS: test-stddef.exe
PASS: test-stdint.exe
PASS: test-stdio.exe
PASS: test-stdlib.exe
PASS: test-strerror.exe
PASS: test-strerror_r.exe
PASS: test-striconv.exe
PASS: test-string.exe
PASS: test-strnlen.exe
PASS: test-strstr.exe
PASS: test-strtoimax.exe
PASS: test-strtoll.exe
PASS: test-strtoull.exe
PASS: test-strtoumax.exe
PASS: test-symlink.exe
PASS: test-sys_ioctl.exe
PASS: test-sys_select.exe
PASS: test-sys_socket.exe
PASS: test-sys_stat.exe
PASS: test-sys_time.exe
PASS: test-sys_types.exe
PASS: test-sys_uio.exe
PASS: test-init.sh
PASS: test-thread_self.exe
PASS: test-thread_create.exe
PASS: test-time.exe
PASS: test-dup-safer.exe
PASS: test-unistd.exe
PASS: test-u8-mbtoucr.exe
PASS: test-u8-uctomb.exe
PASS: test-uc_width.exe
PASS: uniwidth/test-uc_width2.sh
PASS: test-unsetenv.exe
PASS: test-vasnprintf.exe
PASS: test-vc-list-files-git.sh
SKIP: test-vc-list-files-cvs.sh
PASS: test-verify.exe
PASS: test-verify.sh
PASS: test-version-etc.sh
PASS: test-wchar.exe
PASS: test-wcrtomb.sh
SKIP: test-wcrtomb-w32-1.sh
SKIP: test-wcrtomb-w32-2.sh
SKIP: test-wcrtomb-w32-3.sh
SKIP: test-wcrtomb-w32-4.sh
SKIP: test-wcrtomb-w32-5.sh
SKIP: test-wcrtomb-w32-6.sh
SKIP: test-wcrtomb-w32-7.sh
PASS: test-wctype-h.exe
PASS: test-wcwidth.exe
PASS: test-xalloc-die.sh
PASS: test-xstrtoimax.sh
PASS: test-xstrtol.sh
============================================================================
Testsuite summary for GNU grep 3.7.42-9c15
============================================================================
# TOTAL: 201
# PASS:  190
# SKIP:  11
# XFAIL: 0
# FAIL:  0
# XPASS: 0
# ERROR: 0
============================================================================

Re: raise(-1) has stopped returning an error recently

[Duncan Roe]

On Wed, Nov 24, 2021 at 10:25:46AM +0100, cygwin wrote: [...]
>
> What is that "permanent restriction" in Cygwin?  Is that something we
> could fix or something unfixable?  Did you try to debug Cygwin in terms
> of that problem?  If not, could you extract a reduced, very simple
> stand-alone testcase for further debugging?
>
The restriction is in grep/gnulib source. From
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=27555#5

>> I've determined the cause, at least. The sole caller of mbstoupper
>> (which contained the relevant __CYGWIN__-specific
>> surrogate-pair-handling code) was removed by the improvements of
>> v2.21-63-g8a33cde, and not long after, I noticed that mbstoupper was
>> no longer used, so removed it altogether.

I submitted a patch to save anyone else wasting time over this. So please don't
you,

Cheers ... Duncan.

Re: raise(-1) has stopped returning an error recently

[Corinna Vinschen]

On Nov 27 10:43, Duncan Roe wrote:
> On Wed, Nov 24, 2021 at 10:25:46AM +0100, cygwin wrote: [...]
> >
> > What is that "permanent restriction" in Cygwin?  Is that something we
> > could fix or something unfixable?  Did you try to debug Cygwin in terms
> > of that problem?  If not, could you extract a reduced, very simple
> > stand-alone testcase for further debugging?
> >
> The restriction is in grep/gnulib source. From
> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=27555#5
> 
> >> I've determined the cause, at least. The sole caller of mbstoupper
> >> (which contained the relevant __CYGWIN__-specific
> >> surrogate-pair-handling code) was removed by the improvements of
> >> v2.21-63-g8a33cde, and not long after, I noticed that mbstoupper was
> >> no longer used, so removed it altogether.
> 
> I submitted a patch to save anyone else wasting time over this. So please don't
> you,

Great, thanks!


Corinna