* raise(-1) has stopped returning an error recently @ 2021-11-22 5:20 Duncan Roe 2021-11-22 10:25 ` Corinna Vinschen 0 siblings, 1 reply; 14+ messages in thread From: Duncan Roe @ 2021-11-22 5:20 UTC (permalink / raw) To: cygwin Hi, I stumbled across this when running 'make check' in the grep source directory. test-raise failed when it didn't a few weeks ago. I cut it down to the STC at the end of this email Under Linux: > 16:15:57$ ./test-raise > raise(-1) returned -1 Under Cygwin: > 16:16:51$ ./test-raise.exe > raise(-1) returned 0 Hope this is nothing too complicated, Cheers ... Duncan. ======================== #include <stdio.h> #include <signal.h> int main (void) { int retcod; retcod = raise (-1); printf("raise(-1) returned %d\n", retcod); } ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: raise(-1) has stopped returning an error recently 2021-11-22 5:20 raise(-1) has stopped returning an error recently Duncan Roe @ 2021-11-22 10:25 ` Corinna Vinschen 2021-11-22 13:06 ` Corinna Vinschen 0 siblings, 1 reply; 14+ messages in thread From: Corinna Vinschen @ 2021-11-22 10:25 UTC (permalink / raw) To: cygwin On Nov 22 16:20, Duncan Roe wrote: > Hi, > > I stumbled across this when running 'make check' in the grep source directory. > > test-raise failed when it didn't a few weeks ago. Pretty unlikely. This code didn;'t change for ages. I just checked against Cygwin 3.2.0 and it also returns success. > #include <stdio.h> > #include <signal.h> > > int > main (void) > { > int retcod; > > retcod = raise (-1); > printf("raise(-1) returned %d\n", retcod); > } Thanks for the STC. There are actually two long-standing problems, one in raise, calling pthread_kill even for non-threaded processes, one in pthread_kill, not checking the signal number. I'll patch them right away. Thanks, Corinna ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: raise(-1) has stopped returning an error recently 2021-11-22 10:25 ` Corinna Vinschen @ 2021-11-22 13:06 ` Corinna Vinschen 2021-11-23 8:27 ` Duncan Roe 0 siblings, 1 reply; 14+ messages in thread From: Corinna Vinschen @ 2021-11-22 13:06 UTC (permalink / raw) To: cygwin On Nov 22 11:25, Corinna Vinschen via Cygwin wrote: > On Nov 22 16:20, Duncan Roe wrote: > > Hi, > > > > I stumbled across this when running 'make check' in the grep source directory. > > > > test-raise failed when it didn't a few weeks ago. > > Pretty unlikely. This code didn;'t change for ages. I just checked > against Cygwin 3.2.0 and it also returns success. > > > #include <stdio.h> > > #include <signal.h> > > > > int > > main (void) > > { > > int retcod; > > > > retcod = raise (-1); > > printf("raise(-1) returned %d\n", retcod); > > } > > Thanks for the STC. There are actually two long-standing problems, > one in raise, calling pthread_kill even for non-threaded processes, > one in pthread_kill, not checking the signal number. > > I'll patch them right away. I created a patchset: https://sourceware.org/git?p=newlib-cygwin.git;a=commitdiff;h=d54d8f173d66 https://sourceware.org/git?p=newlib-cygwin.git;a=commitdiff;h=24bffff63eb7 https://sourceware.org/git?p=newlib-cygwin.git;a=commitdiff;h=afb7c557d2af Please test the latest developer snapshot from http://cygwin.com/snapshots/ Thanks, Corinna ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: raise(-1) has stopped returning an error recently 2021-11-22 13:06 ` Corinna Vinschen @ 2021-11-23 8:27 ` Duncan Roe 2021-11-23 9:50 ` Corinna Vinschen 0 siblings, 1 reply; 14+ messages in thread From: Duncan Roe @ 2021-11-23 8:27 UTC (permalink / raw) To: Corinna Vinschen via Cygwin; +Cc: Corinna Vinschen On Mon, Nov 22, 2021 at 02:06:19PM +0100, cygwin wrote: > On Nov 22 11:25, Corinna Vinschen via Cygwin wrote: > > On Nov 22 16:20, Duncan Roe wrote: > > > Hi, > > > > > > I stumbled across this when running 'make check' in the grep source directory. > > > > > > test-raise failed when it didn't a few weeks ago. > > > > Pretty unlikely. This code didn;'t change for ages. I just checked > > against Cygwin 3.2.0 and it also returns success. > > > > > #include <stdio.h> > > > #include <signal.h> > > > > > > int > > > main (void) > > > { > > > int retcod; > > > > > > retcod = raise (-1); > > > printf("raise(-1) returned %d\n", retcod); > > > } > > > > Thanks for the STC. There are actually two long-standing problems, > > one in raise, calling pthread_kill even for non-threaded processes, > > one in pthread_kill, not checking the signal number. > > > > I'll patch them right away. > > I created a patchset: > https://sourceware.org/git?p=newlib-cygwin.git;a=commitdiff;h=d54d8f173d66 > https://sourceware.org/git?p=newlib-cygwin.git;a=commitdiff;h=24bffff63eb7 > https://sourceware.org/git?p=newlib-cygwin.git;a=commitdiff;h=afb7c557d2af > > Please test the latest developer snapshot from http://cygwin.com/snapshots/ > > > Thanks, > Corinna Thanks, that fixed it. Btw to whoever maintains grep for cygwin: 'make check' should pass on next release (I patched out the surrogate-pair failre). Cheers ... Duncan. ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: raise(-1) has stopped returning an error recently 2021-11-23 8:27 ` Duncan Roe @ 2021-11-23 9:50 ` Corinna Vinschen 2021-11-23 18:18 ` Brian Inglis 0 siblings, 1 reply; 14+ messages in thread From: Corinna Vinschen @ 2021-11-23 9:50 UTC (permalink / raw) To: cygwin On Nov 23 19:27, Duncan Roe wrote: > On Mon, Nov 22, 2021 at 02:06:19PM +0100, cygwin wrote: > > On Nov 22 11:25, Corinna Vinschen via Cygwin wrote: > > > On Nov 22 16:20, Duncan Roe wrote: > > > > #include <stdio.h> > > > > #include <signal.h> > > > > > > > > int > > > > main (void) > > > > { > > > > int retcod; > > > > > > > > retcod = raise (-1); > > > > printf("raise(-1) returned %d\n", retcod); > > > > } > > > > > > Thanks for the STC. There are actually two long-standing problems, > > > one in raise, calling pthread_kill even for non-threaded processes, > > > one in pthread_kill, not checking the signal number. > > > > > > I'll patch them right away. > > > > I created a patchset: > > https://sourceware.org/git?p=newlib-cygwin.git;a=commitdiff;h=d54d8f173d66 > > https://sourceware.org/git?p=newlib-cygwin.git;a=commitdiff;h=24bffff63eb7 > > https://sourceware.org/git?p=newlib-cygwin.git;a=commitdiff;h=afb7c557d2af > > > > Please test the latest developer snapshot from http://cygwin.com/snapshots/ > > Thanks, that fixed it. Thanks for testing. > Btw to whoever maintains grep for cygwin: 'make check' should pass on > next release (I patched out the surrogate-pair failre). Great! Brian? Thanks, Corinna ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: raise(-1) has stopped returning an error recently 2021-11-23 9:50 ` Corinna Vinschen @ 2021-11-23 18:18 ` Brian Inglis 2021-11-23 22:36 ` Duncan Roe 0 siblings, 1 reply; 14+ messages in thread From: Brian Inglis @ 2021-11-23 18:18 UTC (permalink / raw) To: cygwin On 2021-11-23 02:50, Corinna Vinschen via Cygwin wrote: > On Nov 23 19:27, Duncan Roe wrote: >> On Mon, Nov 22, 2021 at 02:06:19PM +0100, cygwin wrote: >>> On Nov 22 11:25, Corinna Vinschen via Cygwin wrote: >>>> On Nov 22 16:20, Duncan Roe wrote: >>>>> #include <stdio.h> >>>>> #include <signal.h> >>>>> >>>>> int >>>>> main (void) >>>>> { >>>>> int retcod; >>>>> >>>>> retcod = raise (-1); >>>>> printf("raise(-1) returned %d\n", retcod); >>>>> } >>>> >>>> Thanks for the STC. There are actually two long-standing problems, >>>> one in raise, calling pthread_kill even for non-threaded processes, >>>> one in pthread_kill, not checking the signal number. >>>> >>>> I'll patch them right away. >>> >>> I created a patchset: >>> https://sourceware.org/git?p=newlib-cygwin.git;a=commitdiff;h=d54d8f173d66 >>> https://sourceware.org/git?p=newlib-cygwin.git;a=commitdiff;h=24bffff63eb7 >>> https://sourceware.org/git?p=newlib-cygwin.git;a=commitdiff;h=afb7c557d2af >>> >>> Please test the latest developer snapshot from http://cygwin.com/snapshots/ >> >> Thanks, that fixed it. > > Thanks for testing. > >> Btw to whoever maintains grep for cygwin: 'make check' should pass on >> next release (I patched out the surrogate-pair failre). I had no problems with test-raise last release. I did with surrogate pairs but after spending too much time on all the test infrastructure around that, decided it was a low probability event, and wait until anyone complains to refer it upstream. Do Cygwin and/or Windows support surrogate pairs in UTF-8? -- Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada This email may be disturbing to some readers as it contains too much technical detail. Reader discretion is advised. [Data in binary units and prefixes, physical quantities in SI.] ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: raise(-1) has stopped returning an error recently 2021-11-23 18:18 ` Brian Inglis @ 2021-11-23 22:36 ` Duncan Roe 2021-11-24 9:25 ` Corinna Vinschen 0 siblings, 1 reply; 14+ messages in thread From: Duncan Roe @ 2021-11-23 22:36 UTC (permalink / raw) To: cygwin On Tue, Nov 23, 2021 at 11:18:25AM -0700, Brian Inglis wrote: > On 2021-11-23 02:50, Corinna Vinschen via Cygwin wrote: > > On Nov 23 19:27, Duncan Roe wrote: > > > On Mon, Nov 22, 2021 at 02:06:19PM +0100, cygwin wrote: > > > > On Nov 22 11:25, Corinna Vinschen via Cygwin wrote: > > > > > On Nov 22 16:20, Duncan Roe wrote: > > > > > > #include <stdio.h> > > > > > > #include <signal.h> > > > > > > > > > > > > int > > > > > > main (void) > > > > > > { > > > > > > int retcod; > > > > > > > > > > > > retcod = raise (-1); > > > > > > printf("raise(-1) returned %d\n", retcod); > > > > > > } > > > > > > > > > > Thanks for the STC. There are actually two long-standing problems, > > > > > one in raise, calling pthread_kill even for non-threaded processes, > > > > > one in pthread_kill, not checking the signal number. > > > > > > > > > > I'll patch them right away. > > > > > > > > I created a patchset: > > > > https://sourceware.org/git?p=newlib-cygwin.git;a=commitdiff;h=d54d8f173d66 > > > > https://sourceware.org/git?p=newlib-cygwin.git;a=commitdiff;h=24bffff63eb7 > > > > https://sourceware.org/git?p=newlib-cygwin.git;a=commitdiff;h=afb7c557d2af > > > > > > > > Please test the latest developer snapshot from http://cygwin.com/snapshots/ > > > > > > Thanks, that fixed it. > > > > Thanks for testing. > > > > > Btw to whoever maintains grep for cygwin: 'make check' should pass on > > > next release (I patched out the surrogate-pair failre). > > I had no problems with test-raise last release. I don't remember having a problem with it even a few weeks ago. > I did with surrogate pairs but after spending too much time on all the test > infrastructure around that, decided it was a low probability event, and wait > until anyone complains to refer it upstream. I wasted time on that too. That's why I patched surrogate-pair to not do its 3rd test if 'uname -s' indicates Cygwin. For the full story, see https://debbugs.gnu.org/cgi/bugreport.cgi?bug=27555#5 > Do Cygwin and/or Windows support surrogate pairs in UTF-8? There are 3 tests in surrogate-pair and only the 3rd one failed. So I guess surrogate pairs in UTF-8 "mostly work". Cheers ... Duncan. ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: raise(-1) has stopped returning an error recently 2021-11-23 22:36 ` Duncan Roe @ 2021-11-24 9:25 ` Corinna Vinschen 2021-11-24 18:01 ` Brian Inglis 2021-11-26 23:43 ` Duncan Roe 0 siblings, 2 replies; 14+ messages in thread From: Corinna Vinschen @ 2021-11-24 9:25 UTC (permalink / raw) To: cygwin On Nov 24 09:36, Duncan Roe wrote: > On Tue, Nov 23, 2021 at 11:18:25AM -0700, Brian Inglis wrote: > > > On Nov 23 19:27, Duncan Roe wrote: > > > > Btw to whoever maintains grep for cygwin: 'make check' should pass on > > > > next release (I patched out the surrogate-pair failre). > > > > I had no problems with test-raise last release. > > I don't remember having a problem with it even a few weeks ago. > > > I did with surrogate pairs but after spending too much time on all the test > > infrastructure around that, decided it was a low probability event, and wait > > until anyone complains to refer it upstream. > > I wasted time on that too. That's why I patched surrogate-pair to not do its 3rd > test if 'uname -s' indicates Cygwin. > > For the full story, see https://debbugs.gnu.org/cgi/bugreport.cgi?bug=27555#5 What is that "permanent restriction" in Cygwin? Is that something we could fix or something unfixable? Did you try to debug Cygwin in terms of that problem? If not, could you extract a reduced, very simple stand-alone testcase for further debugging? > > Do Cygwin and/or Windows support surrogate pairs in UTF-8? You mean UTF-16. UTF-8 doesn't know surrogate pairs, UTF-16 does. Originally there was UCS-2, 16 bits, with only 65536 code points. However, Unicode left the BMP already with version 2.0 in 1996, so UTF-16 and surrogate pairs became necessary. Windows as well as Cygwin support them. > There are 3 tests in surrogate-pair and only the 3rd one failed. So I guess > surrogate pairs in UTF-8 "mostly work". UTF-16. The surrogate stuff is evil at times. Have a look at the __utf8_wctomb function in https://sourceware.org/git/?p=newlib-cygwin.git;a=blob;f=newlib/libc/stdlib/wctomb_r.c Lone surrogate halfs in an input stream are a problem, for instance. Corinna ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: raise(-1) has stopped returning an error recently 2021-11-24 9:25 ` Corinna Vinschen @ 2021-11-24 18:01 ` Brian Inglis 2021-11-25 12:54 ` Corinna Vinschen 2021-11-28 3:04 ` Duncan Roe 2021-11-26 23:43 ` Duncan Roe 1 sibling, 2 replies; 14+ messages in thread From: Brian Inglis @ 2021-11-24 18:01 UTC (permalink / raw) To: cygwin [-- Attachment #1: Type: text/plain, Size: 2897 bytes --] On 2021-11-24 02:25, Corinna Vinschen via Cygwin wrote: > On Nov 24 09:36, Duncan Roe wrote: >> On Tue, Nov 23, 2021 at 11:18:25AM -0700, Brian Inglis wrote: >>>> On Nov 23 19:27, Duncan Roe wrote: >>>>> Btw to whoever maintains grep for cygwin: 'make check' should pass on >>>>> next release (I patched out the surrogate-pair failre). >>> >>> I had no problems with test-raise last release. >> >> I don't remember having a problem with it even a few weeks ago. >> >>> I did with surrogate pairs but after spending too much time on all the test >>> infrastructure around that, decided it was a low probability event, and wait >>> until anyone complains to refer it upstream. >> >> I wasted time on that too. That's why I patched surrogate-pair to not do its 3rd >> test if 'uname -s' indicates Cygwin. >> >> For the full story, see https://debbugs.gnu.org/cgi/bugreport.cgi?bug=27555#5 > > What is that "permanent restriction" in Cygwin? Is that something we > could fix or something unfixable? Did you try to debug Cygwin in terms > of that problem? If not, could you extract a reduced, very simple > stand-alone testcase for further debugging? > >>> Do Cygwin and/or Windows support surrogate pairs in UTF-8? > > You mean UTF-16. UTF-8 doesn't know surrogate pairs, UTF-16 does. > Originally there was UCS-2, 16 bits, with only 65536 code points. > However, Unicode left the BMP already with version 2.0 in 1996, so > UTF-16 and surrogate pairs became necessary. Windows as well as Cygwin > support them. How does Cygwin support UTF-16 locales with surrogate pairs? Are they the "native" locales inherited from Windows if others are not specified e.g. UTF-8, some OEM SBCS or MBCS? >> There are 3 tests in surrogate-pair and only the 3rd one failed. So I guess >> surrogate pairs in UTF-8 "mostly work". > > UTF-16. The surrogate stuff is evil at times. Have a look at the > __utf8_wctomb function in > https://sourceware.org/git/?p=newlib-cygwin.git;a=blob;f=newlib/libc/stdlib/wctomb_r.c > Lone surrogate halfs in an input stream are a problem, for instance. Thus the confusion with grep surrogate pair tests which appear to be running under a UTF-8 locale: see attached surrogate pair extract from cygport --debug grep.cygport check. Trying to rerun cygport build most tests are now "skipped test: failed to find an adequate shell SKIP ... (exit status: 77)"! Something more may have changed (in gnulib?) to invalidate Cygwin shell(s) in something updated since that grep release in August, as I am getting the same skipped tests under GitHub CI, although it could just be that something expects say bash > 4.4 or even >= 5! -- Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada This email may be disturbing to some readers as it contains too much technical detail. Reader discretion is advised. [Data in binary units and prefixes, physical quantities in SI.] [-- Attachment #2: grep-check-surrogate-pair.log --] [-- Type: text/plain, Size: 5657 bytes --] FAIL: surrogate-pair FAIL: surrogate-pair ==================== + initial_cwd_=$HOME/src/cygwin/grep/grep-3.7-2.x86_64/build/tests + testdir_prefix_ + printf gt + pfx_=gt + mktempd_ $HOME/src/cygwin/grep/grep-3.7-2.x86_64/build/tests gt-surrogate-pair.XXXX + destdir_=$HOME/src/cygwin/grep/grep-3.7-2.x86_64/build/tests + template_=gt-surrogate-pair.XXXX + MAX_TRIES_=4 + destdir_slash_=$HOME/src/cygwin/grep/grep-3.7-2.x86_64/build/tests/ + unset TMPDIR + d=$HOME/src/cygwin/grep/grep-3.7-2.x86_64/build/tests/gt-surrogate-pair.QL4K + : + test -d $HOME/src/cygwin/grep/grep-3.7-2.x86_64/build/tests/gt-surrogate-pair.QL4K + ls -dgo $HOME/src/cygwin/grep/grep-3.7-2.x86_64/build/tests/gt-surrogate-pair.QL4K + perms=drwx------+ 1 0 Aug 31 01:38 $HOME/src/cygwin/grep/grep-3.7-2.x86_64/build/tests/gt-surrogate-pair.QL4K + : + echo $HOME/src/cygwin/grep/grep-3.7-2.x86_64/build/tests/gt-surrogate-pair.QL4K + return + test_dir_=$HOME/src/cygwin/grep/grep-3.7-2.x86_64/build/tests/gt-surrogate-pair.QL4K + cd $HOME/src/cygwin/grep/grep-3.7-2.x86_64/build/tests/gt-surrogate-pair.QL4K + builddir=.. + export srcdir builddir + gl_init_sh_nl_= + IFS= + expr 1 + 128 + eval trap 'Exit 129' 1 + trap Exit 129 1 + expr 2 + 128 + eval trap 'Exit 130' 2 + trap Exit 130 2 + expr 3 + 128 + eval trap 'Exit 131' 3 + trap Exit 131 3 + expr 13 + 128 + eval trap 'Exit 141' 13 + trap Exit 141 13 + expr 15 + 128 + eval trap 'Exit 143' 15 + trap Exit 143 15 + trap remove_tmp_ 0 + path_prepend_ ../src + test 1 != 0 + path_dir_=../src + abs_path_dir_=$HOME/src/cygwin/grep/grep-3.7-2.x86_64/build/tests/../src + PATH=$HOME/src/cygwin/grep/grep-3.7-2.x86_64/build/tests/../src:$HOME/src/cygwin/grep/grep-3.7-2.x86_64/build/src:./src:$HOME/bin:$HOME/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/sbin:... + create_exe_shims_ $HOME/src/cygwin/grep/grep-3.7-2.x86_64/build/tests/../src + return 0 + shift + test 0 != 0 + export PATH + require_en_utf8_locale_ + path_prepend_ . + test 1 != 0 + path_dir_=. + abs_path_dir_=$HOME/src/cygwin/grep/grep-3.7-2.x86_64/build/tests/. + PATH=$HOME/src/cygwin/grep/grep-3.7-2.x86_64/build/tests/.:$HOME/src/cygwin/grep/grep-3.7-2.x86_64/build/tests/../src:$HOME/src/cygwin/grep/grep-3.7-2.x86_64/build/src:./src:$HOME/bin:$HOME/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/sbin:... + create_exe_shims_ $HOME/src/cygwin/grep/grep-3.7-2.x86_64/build/tests/. + return 0 + shift + test 0 != 0 + export PATH + get-mb-cur-max en_US.UTF-8 + require_compiled_in_MB_support + require_en_utf8_locale_ + path_prepend_ . + test 1 != 0 + path_dir_=. + abs_path_dir_=$HOME/src/cygwin/grep/grep-3.7-2.x86_64/build/tests/. + PATH=$HOME/src/cygwin/grep/grep-3.7-2.x86_64/build/tests/.:$HOME/src/cygwin/grep/grep-3.7-2.x86_64/build/tests/.:$HOME/src/cygwin/grep/grep-3.7-2.x86_64/build/tests/../src:$HOME/src/cygwin/grep/grep-3.7-2.x86_64/build/src:./src:$HOME/bin:$HOME/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/sbin:... + create_exe_shims_ $HOME/src/cygwin/grep/grep-3.7-2.x86_64/build/tests/. + return 0 + shift + test 0 != 0 + export PATH + get-mb-cur-max en_US.UTF-8 + printf é + LC_ALL=en_US.UTF-8 grep [[:lower:]] é + fail=0 + printf /360/220/220/205 + s_pair=𐐅 + printf %s/n 𐐅 + LC_ALL=en_US.UTF-8 + export LC_ALL + returns_ 1 grep -i anything-else in + compare /dev/null out + compare_dev_null_ /dev/null out + test 2 = 2 + test x/dev/null = x/dev/null + test -s out + return 0 + return 0 + grep . in + cat out + cat err + io_pair=𐐅: + grep --file=in in + fail=1 + compare out in + compare_dev_null_ out in + test 2 = 2 + test xout = x/dev/null + test xin = x/dev/null + return 2 + compare_ out in + diff -u out in --- out 2021-08-31 01:38:22.950752100 -0600 +++ in 2021-08-31 01:38:22.591357000 -0600 @@ -0,0 +1 @@ +𐐅 + fail=1 + grep --file=in -i in + fail=1 + compare out in + compare_dev_null_ out in + test 2 = 2 + test xout = x/dev/null + test xin = x/dev/null + return 2 + compare_ out in + diff -u out in --- out 2021-08-31 01:38:23.138262500 -0600 +++ in 2021-08-31 01:38:22.591357000 -0600 @@ -0,0 +1 @@ +𐐅 + fail=1 + grep --file=in -E in + fail=1 + compare out in + compare_dev_null_ out in + test 2 = 2 + test xout = x/dev/null + test xin = x/dev/null + return 2 + compare_ out in + diff -u out in --- out 2021-08-31 01:38:23.325773300 -0600 +++ in 2021-08-31 01:38:22.591357000 -0600 @@ -0,0 +1 @@ +𐐅 + fail=1 + grep --file=in -F in + fail=1 + compare out in + compare_dev_null_ out in + test 2 = 2 + test xout = x/dev/null + test xin = x/dev/null + return 2 + compare_ out in + diff -u out in --- out 2021-08-31 01:38:23.528911300 -0600 +++ in 2021-08-31 01:38:22.591357000 -0600 @@ -0,0 +1 @@ +𐐅 + fail=1 + grep --file=in -iE in + fail=1 + compare out in + compare_dev_null_ out in + test 2 = 2 + test xout = x/dev/null + test xin = x/dev/null + return 2 + compare_ out in + diff -u out in --- out 2021-08-31 01:38:23.716423100 -0600 +++ in 2021-08-31 01:38:22.591357000 -0600 @@ -0,0 +1 @@ +𐐅 + fail=1 + grep --file=in -iF in + fail=1 + compare out in + compare_dev_null_ out in + test 2 = 2 + test xout = x/dev/null + test xin = x/dev/null + return 2 + compare_ out in + diff -u out in --- out 2021-08-31 01:38:23.919559100 -0600 +++ in 2021-08-31 01:38:22.591357000 -0600 @@ -0,0 +1 @@ +𐐅 + fail=1 + Exit 1 + set +e + exit 1 + exit 1 + remove_tmp_ + __st=1 + cleanup_ + : + test = yes + cd $HOME/src/cygwin/grep/grep-3.7-2.x86_64/build/tests + chmod -R u+rwx $HOME/src/cygwin/grep/grep-3.7-2.x86_64/build/tests/gt-surrogate-pair.QL4K + rm -rf $HOME/src/cygwin/grep/grep-3.7-2.x86_64/build/tests/gt-surrogate-pair.QL4K + exit 1 FAIL surrogate-pair (exit status: 1) ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: raise(-1) has stopped returning an error recently 2021-11-24 18:01 ` Brian Inglis @ 2021-11-25 12:54 ` Corinna Vinschen 2021-11-27 7:24 ` Brian Inglis 2021-11-28 3:04 ` Duncan Roe 1 sibling, 1 reply; 14+ messages in thread From: Corinna Vinschen @ 2021-11-25 12:54 UTC (permalink / raw) To: cygwin On Nov 24 11:01, Brian Inglis via Cygwin wrote: > On 2021-11-24 02:25, Corinna Vinschen via Cygwin wrote: > > > On Tue, Nov 23, 2021 at 11:18:25AM -0700, Brian Inglis wrote: > > > > Do Cygwin and/or Windows support surrogate pairs in UTF-8? > > > > You mean UTF-16. UTF-8 doesn't know surrogate pairs, UTF-16 does. > > Originally there was UCS-2, 16 bits, with only 65536 code points. > > However, Unicode left the BMP already with version 2.0 in 1996, so > > UTF-16 and surrogate pairs became necessary. Windows as well as Cygwin > > support them. > > How does Cygwin support UTF-16 locales with surrogate pairs? UTF-16 locales? There's no such thing. UTF-16 is just the 16 bit representation for Unicode, and as such, is independent of the locale. On the user side, Cygwin only supports UTF-8 as Unicode representation. Internally you can then convert them to wchar_t which is UTF-16. > Are they the "native" locales inherited from Windows if others are not > specified e.g. UTF-8, some OEM SBCS or MBCS? Just try `locale -av' and you'll see all supported locales and their respective default codeset. All of them can be used with .utf8 specifier to use UTF-8 instead of the default codeset. Some of them use UTF-8 as default codeset anyway, e. g., fa_IR or yo_NG. > > > There are 3 tests in surrogate-pair and only the 3rd one failed. So I guess > > > surrogate pairs in UTF-8 "mostly work". > > > > UTF-16. The surrogate stuff is evil at times. Have a look at the > > __utf8_wctomb function in > > https://sourceware.org/git/?p=newlib-cygwin.git;a=blob;f=newlib/libc/stdlib/wctomb_r.c > > Lone surrogate halfs in an input stream are a problem, for instance. > > Thus the confusion with grep surrogate pair tests which appear to be running > under a UTF-8 locale: see attached surrogate pair extract from cygport > --debug grep.cygport check. An STC in plain C might be helpful. Corinna ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: raise(-1) has stopped returning an error recently 2021-11-25 12:54 ` Corinna Vinschen @ 2021-11-27 7:24 ` Brian Inglis 0 siblings, 0 replies; 14+ messages in thread From: Brian Inglis @ 2021-11-27 7:24 UTC (permalink / raw) To: cygwin On 2021-11-25 05:54, Corinna Vinschen via Cygwin wrote: > On Nov 24 11:01, Brian Inglis via Cygwin wrote: >> On 2021-11-24 02:25, Corinna Vinschen via Cygwin wrote: >>>> On Tue, Nov 23, 2021 at 11:18:25AM -0700, Brian Inglis wrote: >>>>> Do Cygwin and/or Windows support surrogate pairs in UTF-8? >>> >>> You mean UTF-16. UTF-8 doesn't know surrogate pairs, UTF-16 does. >>> Originally there was UCS-2, 16 bits, with only 65536 code points. >>> However, Unicode left the BMP already with version 2.0 in 1996, so >>> UTF-16 and surrogate pairs became necessary. Windows as well as Cygwin >>> support them. >> >> How does Cygwin support UTF-16 locales with surrogate pairs? > > UTF-16 locales? There's no such thing. UTF-16 is just the 16 bit > representation for Unicode, and as such, is independent of the locale. > On the user side, Cygwin only supports UTF-8 as Unicode representation. > Internally you can then convert them to wchar_t which is UTF-16. > >> Are they the "native" locales inherited from Windows if others are not >> specified e.g. UTF-8, some OEM SBCS or MBCS? > > Just try `locale -av' and you'll see all supported locales and their > respective default codeset. All of them can be used with .utf8 > specifier to use UTF-8 instead of the default codeset. Some of them > use UTF-8 as default codeset anyway, e. g., fa_IR or yo_NG. > >>>> There are 3 tests in surrogate-pair and only the 3rd one failed. So I guess >>>> surrogate pairs in UTF-8 "mostly work". >>> >>> UTF-16. The surrogate stuff is evil at times. Have a look at the >>> __utf8_wctomb function in >>> https://sourceware.org/git/?p=newlib-cygwin.git;a=blob;f=newlib/libc/stdlib/wctomb_r.c >>> Lone surrogate halfs in an input stream are a problem, for instance. >> >> Thus the confusion with grep surrogate pair tests which appear to be running >> under a UTF-8 locale: see attached surrogate pair extract from cygport >> --debug grep.cygport check. > > An STC in plain C might be helpful. I think I might finally have got the point of the test, not knowing much about legacy UTF-16 UCS encoding nor surrogate pairs. From what I can see: 𐐅 U+010405 f0 90 90 85 DESERET CAPITAL LETTER LONG OO fails to match itself, presumably others do also. Presumably this is converted internally on some platforms, including Cygwin, to a UTF-16 surrogate pair, and a grep comparison fails, although a bash comparison succeeds. $ printf '\U10405\n' | iconv -f utf-8 -t utf-16be | xxd -g2 00000000: d801 dc05 000a $ printf '\U10405\n' > t $ grep -f t t; echo $? 1 $ oo=`printf '\U10405\n'`; [ $oo = $oo ] && echo same || echo diff same -- Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada This email may be disturbing to some readers as it contains too much technical detail. Reader discretion is advised. [Data in binary units and prefixes, physical quantities in SI.] ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: raise(-1) has stopped returning an error recently 2021-11-24 18:01 ` Brian Inglis 2021-11-25 12:54 ` Corinna Vinschen @ 2021-11-28 3:04 ` Duncan Roe 1 sibling, 0 replies; 14+ messages in thread From: Duncan Roe @ 2021-11-28 3:04 UTC (permalink / raw) To: Brian Inglis via Cygwin On Wed, Nov 24, 2021 at 11:01:46AM -0700, cygwin wrote: [...] > Trying to rerun cygport build most tests are now "skipped test: failed to > find an adequate shell SKIP ... (exit status: 77)"! Something more may have > changed (in gnulib?) to invalidate Cygwin shell(s) in something updated > since that grep release in August, as I am getting the same skipped tests > under GitHub CI, although it could just be that something expects say bash > > 4.4 or even >= 5! > They all pass for me in a directory tree created by 'make dist' under Linux. Maybe something changed in your test environment? N.B. grep maintainer replaced my patch with a fancier one that creates new surrogate-search test that Cygwin skips. Cheers ... Duncan. PASS: backref PASS: backref-alt PASS: backref-multibyte-slow PASS: backref-word PASS: backslash-dot PASS: backslash-s-and-repetition-operators PASS: backslash-s-vs-invalid-multibyte big-hole: skipped test: expensive: disabled by default SKIP: big-hole big-match: skipped test: expensive: disabled by default SKIP: big-match PASS: binary-file-matches PASS: bogus-wctob PASS: bre PASS: c-locale PASS: case-fold-backref PASS: case-fold-backslash-w PASS: case-fold-char-class PASS: case-fold-char-range PASS: case-fold-char-type PASS: case-fold-titlecase PASS: char-class-multibyte PASS: char-class-multibyte2 PASS: context-0 PASS: count-newline PASS: dfa-coverage PASS: dfa-heap-overrun PASS: dfa-infloop PASS: dfa-invalid-utf8 PASS: dfaexec-multibyte PASS: empty PASS: empty-line PASS: empty-line-mb PASS: encoding-error PASS: epipe XFAIL: equiv-classes PASS: ere PASS: euc-mb false-match-mb-non-utf8: skipped test: no support for the zh_CN.gb18030 locale SKIP: false-match-mb-non-utf8 PASS: fedora PASS: fgrep-infloop PASS: fgrep-longest PASS: file PASS: filename-lineno.pl PASS: fmbtest PASS: foad1 PASS: grep-dev-null PASS: grep-dev-null-out PASS: grep-dir ratio=0 PASS: hash-collision-perf PASS: help-version PASS: high-bit-range PASS: in-eq-out-infloop PASS: include-exclude PASS: inconsistent-range PASS: initial-tab PASS: invalid-multibyte-infloop PASS: khadafy PASS: kwset-abuse long-line-vs-2GiB-read: skipped test: expensive: disabled by default SKIP: long-line-vs-2GiB-read long-pattern-perf: skipped test: expensive: disabled by default SKIP: long-pattern-perf many-regex-performance: skipped test: expensive: disabled by default SKIP: many-regex-performance PASS: match-lines PASS: max-count-overread PASS: max-count-vs-context PASS: mb-dot-newline PASS: mb-non-UTF8-overrun PASS: mb-non-UTF8-perf-Fw mb-non-UTF8-performance: skipped test: expensive: disabled by default SKIP: mb-non-UTF8-performance PASS: mb-non-UTF8-word-boundary warning: \s failed to match \x0a in the en_US.UTF-8 locale warning: \s failed to match \x85 in the en_US.UTF-8 locale warning: \s failed to match \xe2\x80\x8b in the en_US.UTF-8 locale warning: \S mistakenly matched \xe2\x80\x8b in the en_US.UTF-8 locale PASS: multibyte-white-space PASS: multiple-begin-or-end-line PASS: null-byte PASS: options pcre: skipped test: no PCRE support SKIP: pcre pcre-abort: skipped test: no PCRE support SKIP: pcre-abort pcre-context: skipped test: no PCRE support SKIP: pcre-context pcre-count: skipped test: no PCRE support SKIP: pcre-count pcre-infloop: skipped test: no PCRE support SKIP: pcre-infloop pcre-invalid-utf8-infloop: skipped test: no PCRE support SKIP: pcre-invalid-utf8-infloop pcre-invalid-utf8-input: skipped test: no PCRE support SKIP: pcre-invalid-utf8-input pcre-jitstack: skipped test: no PCRE support SKIP: pcre-jitstack pcre-o: skipped test: no PCRE support SKIP: pcre-o pcre-utf8: skipped test: no PCRE support SKIP: pcre-utf8 pcre-w: skipped test: no PCRE support SKIP: pcre-w pcre-wx-backref: skipped test: no PCRE support SKIP: pcre-wx-backref pcre-z: skipped test: no PCRE support SKIP: pcre-z PASS: posix-bracket PASS: prefix-of-multibyte PASS: proc PASS: r-dot PASS: repetition-overflow PASS: reversed-range-endpoints PASS: sjis-mb PASS: skip-device PASS: skip-read PASS: spencer1 PASS: spencer1-locale PASS: stack-overflow PASS: status PASS: surrogate-pair surrogate-search: skipped test: surrogate-pair search string on Cygwin, where it cannot work SKIP: surrogate-search PASS: symlink triple-backref: expect malfunction on glibc systems due to https://sourceware.org/bugzilla/show_bug.cgi?id=11053 XFAIL: triple-backref PASS: turkish-I PASS: turkish-I-without-dot turkish-eyes: skipped test: your tr_TR.UTF-8 locale appears to be broken SKIP: turkish-eyes PASS: two-chars PASS: two-files PASS: unibyte-binary PASS: unibyte-bracket-expr PASS: unibyte-negated-circumflex PASS: utf8-bracket PASS: warn-char-classes PASS: word-delim-multibyte PASS: word-multi-file PASS: word-multibyte PASS: write-error-msg PASS: yesno PASS: z-anchor-newline ============================================================================ Testsuite summary for GNU grep 3.7.42-9c15 ============================================================================ # TOTAL: 118 # PASS: 94 # SKIP: 22 # XFAIL: 2 # FAIL: 0 # XPASS: 0 # ERROR: 0 ============================================================================ ## ---------------------------- Gnulib tests ---------------------------- ## PASS: test-alignof.exe PASS: test-alloca-opt.exe PASS: test-argmatch.exe PASS: test-arpa_inet.exe PASS: test-binary-io.sh PASS: test-bind.exe PASS: test-bitrotate.exe PASS: test-btowc1.sh PASS: test-btowc2.sh PASS: test-c-ctype.exe PASS: test-c-stack.sh PASS: test-c-stack2.sh PASS: test-c-strcase.sh PASS: test-calloc-gnu.exe PASS: test-chdir.exe PASS: test-cloexec.exe PASS: test-close.exe PASS: test-connect.exe PASS: test-ctype.exe PASS: test-dfa-invalid-char-class.sh PASS: test-dfa-invalid-merge.sh PASS: test-dfa-match.sh PASS: test-dirent.exe PASS: test-dup.exe PASS: test-dup2.exe PASS: test-dynarray.exe PASS: test-environ.exe PASS: test-errno.exe PASS: test-exclude1.sh PASS: test-exclude2.sh PASS: test-exclude3.sh PASS: test-exclude4.sh PASS: test-exclude5.sh PASS: test-exclude6.sh PASS: test-exclude7.sh PASS: test-exclude8.sh PASS: test-fchdir.exe PASS: test-fcntl-h.exe PASS: test-fcntl-safer.exe PASS: test-fcntl.exe PASS: test-fdopen.exe PASS: test-fdopendir.exe PASS: test-fgetc.exe PASS: test-float.exe PASS: test-fnmatch-h.exe PASS: test-fnmatch.exe PASS: test-fopen-gnu.exe PASS: test-fopen.exe PASS: test-fpending.sh PASS: test-fputc.exe PASS: test-fread.exe PASS: test-free.exe PASS: test-fstat.exe PASS: test-fstatat.exe PASS: test-ftruncate.sh PASS: test-fwrite.exe PASS: test-getcwd-lgpl.exe PASS: test-getdtablesize.exe PASS: test-getopt-gnu.exe PASS: test-getopt-posix.exe PASS: test-getprogname.exe PASS: test-gettimeofday.exe PASS: test-hard-locale.exe PASS: test-hash.exe PASS: test-i-ring.exe PASS: test-iconv-h.exe PASS: test-iconv.exe PASS: test-ignore-value.exe PASS: test-inet_pton.exe PASS: test-intprops.exe PASS: test-inttostr.exe PASS: test-inttypes.exe PASS: test-ioctl.exe PASS: test-isatty.exe PASS: test-isblank.exe PASS: test-iswblank.exe PASS: test-iswdigit.sh PASS: test-iswxdigit.sh PASS: test-langinfo.exe PASS: test-limits-h.exe PASS: test-listen.exe PASS: test-locale.exe PASS: test-localeconv.exe PASS: test-localename.exe PASS: test-lseek.sh PASS: test-lstat.exe PASS: test-malloc-gnu.exe PASS: test-malloca.exe SKIP: test-mbscasecmp.sh PASS: test-mbsinit.sh PASS: test-mbsrtowcs1.sh PASS: test-mbsrtowcs2.sh PASS: test-mbsrtowcs3.sh SKIP: test-mbsrtowcs4.sh PASS: test-mbsstr1.exe PASS: test-mbsstr2.sh SKIP: test-mbsstr3.sh PASS: test-memchr.exe PASS: test-memchr2.exe PASS: test-memrchr.exe PASS: test-nanosleep.exe PASS: test-netinet_in.exe PASS: test-nl_langinfo.sh PASS: test-nl_langinfo-mt.exe PASS: test-open.exe PASS: test-openat-safer.exe PASS: test-openat.exe PASS: test-pathmax.exe PASS: test-perror.sh PASS: test-perror2.exe PASS: test-pipe.exe PASS: test-pthread.exe PASS: test-pthread-thread.exe PASS: test-pthread_sigmask1.exe PASS: test-pthread_sigmask2.exe PASS: test-quotearg-simple.exe PASS: test-raise.exe PASS: test-rawmemchr.exe PASS: test-read.exe PASS: test-realloc-gnu.exe PASS: test-reallocarray.exe PASS: test-regex.exe PASS: test-sched.exe PASS: test-select.exe PASS: test-select-in.sh PASS: test-select-out.sh PASS: test-setenv.exe PASS: test-setlocale_null.exe PASS: test-setlocale_null-mt-one.exe PASS: test-setlocale_null-mt-all.exe PASS: test-setlocale1.sh PASS: test-setlocale2.sh PASS: test-setsockopt.exe PASS: test-sigaction.exe PASS: test-signal-h.exe PASS: test-sigprocmask.exe PASS: test-sigsegv-catch-segv1.exe PASS: test-sigsegv-catch-segv2.exe PASS: test-sigsegv-catch-stackoverflow1.exe PASS: test-sigsegv-catch-stackoverflow2.exe PASS: test-sleep.exe PASS: test-snprintf.exe PASS: test-sockets.exe PASS: test-stat.exe PASS: test-stat-time.exe PASS: test-stdalign.exe PASS: test-stdbool.exe PASS: test-stddef.exe PASS: test-stdint.exe PASS: test-stdio.exe PASS: test-stdlib.exe PASS: test-strerror.exe PASS: test-strerror_r.exe PASS: test-striconv.exe PASS: test-string.exe PASS: test-strnlen.exe PASS: test-strstr.exe PASS: test-strtoimax.exe PASS: test-strtoll.exe PASS: test-strtoull.exe PASS: test-strtoumax.exe PASS: test-symlink.exe PASS: test-sys_ioctl.exe PASS: test-sys_select.exe PASS: test-sys_socket.exe PASS: test-sys_stat.exe PASS: test-sys_time.exe PASS: test-sys_types.exe PASS: test-sys_uio.exe PASS: test-init.sh PASS: test-thread_self.exe PASS: test-thread_create.exe PASS: test-time.exe PASS: test-dup-safer.exe PASS: test-unistd.exe PASS: test-u8-mbtoucr.exe PASS: test-u8-uctomb.exe PASS: test-uc_width.exe PASS: uniwidth/test-uc_width2.sh PASS: test-unsetenv.exe PASS: test-vasnprintf.exe PASS: test-vc-list-files-git.sh SKIP: test-vc-list-files-cvs.sh PASS: test-verify.exe PASS: test-verify.sh PASS: test-version-etc.sh PASS: test-wchar.exe PASS: test-wcrtomb.sh SKIP: test-wcrtomb-w32-1.sh SKIP: test-wcrtomb-w32-2.sh SKIP: test-wcrtomb-w32-3.sh SKIP: test-wcrtomb-w32-4.sh SKIP: test-wcrtomb-w32-5.sh SKIP: test-wcrtomb-w32-6.sh SKIP: test-wcrtomb-w32-7.sh PASS: test-wctype-h.exe PASS: test-wcwidth.exe PASS: test-xalloc-die.sh PASS: test-xstrtoimax.sh PASS: test-xstrtol.sh ============================================================================ Testsuite summary for GNU grep 3.7.42-9c15 ============================================================================ # TOTAL: 201 # PASS: 190 # SKIP: 11 # XFAIL: 0 # FAIL: 0 # XPASS: 0 # ERROR: 0 ============================================================================ ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: raise(-1) has stopped returning an error recently 2021-11-24 9:25 ` Corinna Vinschen 2021-11-24 18:01 ` Brian Inglis @ 2021-11-26 23:43 ` Duncan Roe 2021-11-29 10:41 ` Corinna Vinschen 1 sibling, 1 reply; 14+ messages in thread From: Duncan Roe @ 2021-11-26 23:43 UTC (permalink / raw) To: Corinna Vinschen via Cygwin; +Cc: Corinna Vinschen On Wed, Nov 24, 2021 at 10:25:46AM +0100, cygwin wrote: [...] > > What is that "permanent restriction" in Cygwin? Is that something we > could fix or something unfixable? Did you try to debug Cygwin in terms > of that problem? If not, could you extract a reduced, very simple > stand-alone testcase for further debugging? > The restriction is in grep/gnulib source. From https://debbugs.gnu.org/cgi/bugreport.cgi?bug=27555#5 >> I've determined the cause, at least. The sole caller of mbstoupper >> (which contained the relevant __CYGWIN__-specific >> surrogate-pair-handling code) was removed by the improvements of >> v2.21-63-g8a33cde, and not long after, I noticed that mbstoupper was >> no longer used, so removed it altogether. I submitted a patch to save anyone else wasting time over this. So please don't you, Cheers ... Duncan. ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: raise(-1) has stopped returning an error recently 2021-11-26 23:43 ` Duncan Roe @ 2021-11-29 10:41 ` Corinna Vinschen 0 siblings, 0 replies; 14+ messages in thread From: Corinna Vinschen @ 2021-11-29 10:41 UTC (permalink / raw) To: cygwin On Nov 27 10:43, Duncan Roe wrote: > On Wed, Nov 24, 2021 at 10:25:46AM +0100, cygwin wrote: [...] > > > > What is that "permanent restriction" in Cygwin? Is that something we > > could fix or something unfixable? Did you try to debug Cygwin in terms > > of that problem? If not, could you extract a reduced, very simple > > stand-alone testcase for further debugging? > > > The restriction is in grep/gnulib source. From > https://debbugs.gnu.org/cgi/bugreport.cgi?bug=27555#5 > > >> I've determined the cause, at least. The sole caller of mbstoupper > >> (which contained the relevant __CYGWIN__-specific > >> surrogate-pair-handling code) was removed by the improvements of > >> v2.21-63-g8a33cde, and not long after, I noticed that mbstoupper was > >> no longer used, so removed it altogether. > > I submitted a patch to save anyone else wasting time over this. So please don't > you, Great, thanks! Corinna ^ permalink raw reply [flat|nested] 14+ messages in thread
end of thread, other threads:[~2021-11-29 10:41 UTC | newest] Thread overview: 14+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2021-11-22 5:20 raise(-1) has stopped returning an error recently Duncan Roe 2021-11-22 10:25 ` Corinna Vinschen 2021-11-22 13:06 ` Corinna Vinschen 2021-11-23 8:27 ` Duncan Roe 2021-11-23 9:50 ` Corinna Vinschen 2021-11-23 18:18 ` Brian Inglis 2021-11-23 22:36 ` Duncan Roe 2021-11-24 9:25 ` Corinna Vinschen 2021-11-24 18:01 ` Brian Inglis 2021-11-25 12:54 ` Corinna Vinschen 2021-11-27 7:24 ` Brian Inglis 2021-11-28 3:04 ` Duncan Roe 2021-11-26 23:43 ` Duncan Roe 2021-11-29 10:41 ` Corinna Vinschen
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).