[ANNOUNCEMENT] Updated: dash 0.5.12-2

[ANNOUNCEMENT] Updated: dash 0.5.12-2

[Cygwin dash Co-Maintainer via Cygwin-announce]

The following packages have been upgraded in the Cygwin distribution:

* dash	0.5.12-2

This package is being upgraded to current as the previous release
breaks existing function that may be relied on by some scripts.
This release has been rebuilt disabling libc fnmatch and glob as
Cygwin/winsup/newlib do not support locale dependent named character
classes like glibc.

Thanks to Andrey Repin for testing and bringing this to our attention,
and Harald van Dijk on the dash list for pointing out the commit
responsible and the solution.

DASH is a POSIX-compliant implementation of /bin/sh that aims to be
as small as possible. It does this without sacrificing speed where
possible. In fact, it is significantly faster than bash (the GNU
Bourne-Again SHell) for most tasks.

For more information see the project home page:

	http://gondor.apana.org.au/~herbert/dash/

For changes since the previous release, see below;
for complete details see:

https://git.kernel.org/pub/scm/utils/dash/dash.git/log/?h=v0.5.12&showmsg=1


2022-12-11	0.5.12

error:
	Remove USE_NORETURN ifdef

eval:
	Always set exitstatus in evaltree
	Check eflag after redirection error
	Check nflag in evaltree instead of cmdloop
	Do not cache value of eflag in evaltree
	Prevent recursive PS4 expansion
	Test evalskip before flipping status for NNOT

expand:
	Add ifsfree to expand to fix a logic error that causes a buffer over-read
	Always quote caret when using fnmatch
	Make glob(3) interruptible by SIGINT

input:
	Clear unget on RESET
	Remove special case for unget EOF

jobs:
	Always reset SIGINT/SIGQUIT handlers
	Block signals during tcsetpgrp
	Fix waitcmd busy loop
	Only block in waitcmd on first run

man:
	fix formatting

parser:
	Add VSBIT to ensure subtype is never zero
	Fix VSLENGTH parsing with trailing garbage
	Get rid of PEOA
	
redir:
	Retry open64 on EINTR
	
shell:
	Call CHECK_DECL on stat64
        Disable glob again as it strips trailing slashes
        Enable fnmatch/glob by default
        Fail if building --with-libedit and can't find libedit
        Group readdir64/dirent64 with open64

Re: [ANNOUNCEMENT] Updated: dash 0.5.12-2

[Corinna Vinschen]

On Feb 12 22:03, Cygwin dash Co-Maintainer via Cygwin-announce via Cygwin wrote:
> The following packages have been upgraded in the Cygwin distribution:
> 
> * dash	0.5.12-2
> 
> This package is being upgraded to current as the previous release
> breaks existing function that may be relied on by some scripts.
> This release has been rebuilt disabling libc fnmatch and glob as
> Cygwin/winsup/newlib do not support locale dependent named character
> classes like glibc.

So how do we fix in Cygwin without pulling in GLibc code?


Corinna

Re: [ANNOUNCEMENT] Updated: dash 0.5.12-2

[Corinna Vinschen]

Hi Brian,

On Feb 13 10:22, Corinna Vinschen via Cygwin wrote:
> On Feb 12 22:03, Cygwin dash Co-Maintainer via Cygwin-announce via Cygwin wrote:
> > The following packages have been upgraded in the Cygwin distribution:
> > 
> > * dash	0.5.12-2
> > 
> > This package is being upgraded to current as the previous release
> > breaks existing function that may be relied on by some scripts.
> > This release has been rebuilt disabling libc fnmatch and glob as
> > Cygwin/winsup/newlib do not support locale dependent named character
> > classes like glibc.

Can you give me an example?  I'm a bit puzzled because fnmatch as well
as glob in Cygwin support native characters.


Corinna


> 
> So how do we fix in Cygwin without pulling in GLibc code?
> 
> 
> Corinna

Re: [ANNOUNCEMENT] Updated: dash 0.5.12-2

[ASSI]

Corinna Vinschen via Cygwin writes:
> Can you give me an example?  I'm a bit puzzled because fnmatch as well
> as glob in Cygwin support native characters.

AFAIU, the issue was with locale dependent character classes, not
characters.  Andrey specifically mentioned [[:space:]] in the original
bugreport.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Wavetables for the Waldorf Blofeld:
http://Synth.Stromeko.net/Downloads.html#BlofeldUserWavetables

Re: [ANNOUNCEMENT] Updated: dash 0.5.12-2

[Andrey Repin]

Greetings, Corinna Vinschen via Cygwin!

> On Feb 12 22:03, Cygwin dash Co-Maintainer via Cygwin-announce via Cygwin wrote:
>> The following packages have been upgraded in the Cygwin distribution:
>> 
>> * dash        0.5.12-2
>> 
>> This package is being upgraded to current as the previous release
>> breaks existing function that may be relied on by some scripts.
>> This release has been rebuilt disabling libc fnmatch and glob as
>> Cygwin/winsup/newlib do not support locale dependent named character
>> classes like glibc.

> So how do we fix in Cygwin without pulling in GLibc code?

Depends what you call a "fix".

As for an example,

dash -c '_x="   x"; echo "${_x#${_x%%[![:space:]]*}}"'

Using this construct with 0.5.12-1 will produce an empty result.


-- 
With best regards,
Andrey Repin
Monday, February 13, 2023 20:40:36

Sorry for my terrible english...

Re: [ANNOUNCEMENT] Updated: dash 0.5.12-2

[Brian Inglis]

On 2023-02-13 10:43, ASSI via Cygwin wrote:
> Corinna Vinschen via Cygwin writes:
>> On Feb 12 22:03, Cygwin dash Co-Maintainer wrote:
>>> The following packages have been upgraded in the Cygwin distribution:
>>>
>>> * dash        0.5.12-2
>>>
>>> This package is being upgraded to current as the previous release
>>> breaks existing function that may be relied on by some scripts.
>>> This release has been rebuilt disabling libc fnmatch and glob as
>>> Cygwin/winsup/newlib do not support locale dependent named character
>>> classes like glibc.

>> Can you give me an example?  I'm a bit puzzled because fnmatch as well
>> as glob in Cygwin support native characters.

But not locale dependent named character classes like regexp in paths.

> AFAIU, the issue was with locale dependent character classes, not
> characters. Andrey specifically mentioned [[:space:]] in the original
> bugreport.

Specifically support in fnmatch and glob, now updated in glibc, and their use in 
dash for parameter expansion in place of (gnulib?) regexp support.

>> So how do we fix in Cygwin without pulling in GLibc code?

It's fixed in release 2 which disables 0.5.12 changed default (using 
[Cygwin/winsup/newlib] libc fnmatch and glob, without testing functionality), 
and goes back to using its own previous default (gnulib?) regexp code for 
parameter expansion which supports locale dependent named character classes.

As glibc "recently" fixed their fnmatch and glob to support locale dependent 
named character classes like [[:blank:]], dash decided to use those in parameter 
expansions for consistency, but not exactly clearly documented, although they 
added config options to enable/disable each of libc fnmatch/glob.

Cygwin bash parameter expansion also supports locale dependent named character 
classes.

	$ info bash pattern
	$ info sed classes

-- 
Take care. Thanks, Brian Inglis			Calgary, Alberta, Canada

La perfection est atteinte			Perfection is achieved
non pas lorsqu'il n'y a plus rien à ajouter	not when there is no more to add
mais lorsqu'il n'y a plus rien à retirer	but when there is no more to cut
			-- Antoine de Saint-Exupéry

Re: [ANNOUNCEMENT] Updated: dash 0.5.12-2

[Corinna Vinschen]

On Feb 13 12:03, Brian Inglis via Cygwin wrote:
> On 2023-02-13 10:43, ASSI via Cygwin wrote:
> > Corinna Vinschen via Cygwin writes:
> > > On Feb 12 22:03, Cygwin dash Co-Maintainer wrote:
> > > > The following packages have been upgraded in the Cygwin distribution:
> > > > 
> > > > * dash        0.5.12-2
> > > > 
> > > > This package is being upgraded to current as the previous release
> > > > breaks existing function that may be relied on by some scripts.
> > > > This release has been rebuilt disabling libc fnmatch and glob as
> > > > Cygwin/winsup/newlib do not support locale dependent named character
> > > > classes like glibc.
> 
> > > Can you give me an example?  I'm a bit puzzled because fnmatch as well
> > > as glob in Cygwin support native characters.
> 
> But not locale dependent named character classes like regexp in paths.

I checked the dash code of curent dash git, and while its internal glob
implementation supports character classes, they are no localized, using
standard singlebyte functions isalnum, isalpha, etc. under the hood.

So, yeah, what you say further down this mail... looks like dash
supports locale dependent character classes only with glibc.

> > AFAIU, the issue was with locale dependent character classes, not
> > characters. Andrey specifically mentioned [[:space:]] in the original
> > bugreport.
> 
> Specifically support in fnmatch and glob, now updated in glibc, and their
> use in dash for parameter expansion in place of (gnulib?) regexp support.
> 
> > > So how do we fix in Cygwin without pulling in GLibc code?
> 
> It's fixed in release 2 which disables 0.5.12 changed default (using
> [Cygwin/winsup/newlib] libc fnmatch and glob, without testing
> functionality), and goes back to using its own previous default (gnulib?)
> regexp code for parameter expansion which supports locale dependent named
> character classes.
> 
> As glibc "recently" fixed their fnmatch and glob to support locale dependent
> named character classes like [[:blank:]], dash decided to use those in
> parameter expansions for consistency, but not exactly clearly documented,
> although they added config options to enable/disable each of libc
> fnmatch/glob.
> 
> Cygwin bash parameter expansion also supports locale dependent named
> character classes.
> 
> 	$ info bash pattern
> 	$ info sed classes

These info pages are mostly target-system independent.  There's no
guarantee that this works in a Cygwin bash or sed.

Either way, I don't care much for what a certain application provides by
itself.  I'm talking about our libc, that is Cygwin, and what it
provides to processes calling its implementations of regcomp/regexec,
glob and fnmatch.

All these functions have been taken from FreeBSD and all three suffer
shortcomings:

- regcomp/regexec supports POSIX named character classes, collating
  symbols, and equivalence class expressions, but all of them only work
  for ASCII chars.

- fnmatch and glob support neither of named character classes,
  collating symbols, and equivalence class expressions.

I checked the upstream code in FreeBSD, OpenBSD and NetBSD and none of
these functions are improved to support locales (regcomp) or any of
the character classes stuff (fnmatch/glob).

So, if we want to add this support to Cygwin (and thus, to all
applications calling the libc implementation of these functions),
quite a bit of work is required.

Being able to fetch the implementation from some other source
would reduce the effort enourmously :}


Corinna

Re: [ANNOUNCEMENT] Updated: dash 0.5.12-2

[Corinna Vinschen]

Hi Brian,

On Feb 13 20:37, Corinna Vinschen via Cygwin wrote:
> On Feb 13 12:03, Brian Inglis via Cygwin wrote:
> > On 2023-02-13 10:43, ASSI via Cygwin wrote:
> > > Corinna Vinschen via Cygwin writes:
> > > > Can you give me an example?  I'm a bit puzzled because fnmatch as well
> > > > as glob in Cygwin support native characters.
> > 
> > But not locale dependent named character classes like regexp in paths.
> 
> I checked the dash code of curent dash git, and while its internal glob
> implementation supports character classes, they are no localized, using
> standard singlebyte functions isalnum, isalpha, etc. under the hood.
> 
> So, yeah, what you say further down this mail... looks like dash
> supports locale dependent character classes only with glibc.
> [...]
> Either way, I don't care much for what a certain application provides by
> itself.  I'm talking about our libc, that is Cygwin, and what it
> provides to processes calling its implementations of regcomp/regexec,
> glob and fnmatch.
> 
> All these functions have been taken from FreeBSD and all three suffer
> shortcomings:
> 
> - regcomp/regexec supports POSIX named character classes, collating
>   symbols, and equivalence class expressions, but all of them only work
>   for ASCII chars.
> 
> - fnmatch and glob support neither of named character classes,
>   collating symbols, and equivalence class expressions.
> 
> I checked the upstream code in FreeBSD, OpenBSD and NetBSD and none of
> these functions are improved to support locales (regcomp) or any of
> the character classes stuff (fnmatch/glob).
> 
> So, if we want to add this support to Cygwin (and thus, to all
> applications calling the libc implementation of these functions),
> quite a bit of work is required.
> 
> Being able to fetch the implementation from some other source
> would reduce the effort enourmously :}

I took the liberty to add [:<class>:] support to Cygwin's fnmatch(3) and
glob(3) functions.  They also recognize collating symbols [.<coll.] and
equivalence class expressions [=<equiv>=].  But the latter two are not
implemented yet and fnmatch/glob simply skip them in the pattern.

Given that glob and fnmatch use wide characters internally, the support
for character classes is internationalized by default, albeit in a
slightly differentt way than in glibc.  The classes a unicode character
belongs to is not locale dependent in Cygwin/newlib.  All characters
have their classes assigned all the time, so, for instance, the german
character 'ä' is lower and alpha even in the en_US.utf8 locale.

The currently building cygwin test release 3.5.0-0.174.gd6d4436145b8
contains the new code.  Would you mind to build a dash for testing so we
can see if and how it works?


Thanks,
Corinna

Re: [ANNOUNCEMENT] Updated: dash 0.5.12-2

[Corinna Vinschen]

Oh, I'm an idiot.

On Feb 15 14:52, Corinna Vinschen via Cygwin wrote:
> Hi Brian,
> [...]
> The currently building cygwin test release 3.5.0-0.174.gd6d4436145b8
> contains the new code.  Would you mind to build a dash for testing so we
> can see if and how it works?

You don't have to build a new dash.  Just downgrading to
dash-0.5.12-1 will suffice, of course.  D'oh.

So, as a first test, here's the result of Andrey's expression
with dash-0.5.12-1:

  $ dash -c '_x="   x"; echo "${_x#${_x%%[![:space:]]*}}"'
  x


Corinna

Re: [ANNOUNCEMENT] Updated: dash 0.5.12-2

[Andrey Repin]

Greetings, Corinna Vinschen via Cygwin!

> On Feb 15 14:52, Corinna Vinschen via Cygwin wrote:
>> Hi Brian,
>> [...]
>> The currently building cygwin test release 3.5.0-0.174.gd6d4436145b8
>> contains the new code.  Would you mind to build a dash for testing so we
>> can see if and how it works?

> You don't have to build a new dash.  Just downgrading to
> dash-0.5.12-1 will suffice, of course.  D'oh.

> So, as a first test, here's the result of Andrey's expression
> with dash-0.5.12-1:

>   $ dash -c '_x="   x"; echo "${_x#${_x%%[![:space:]]*}}"'
>   x

:thumbsup:


-- 
With best regards,
Andrey Repin
Wednesday, February 15, 2023 18:56:29

Sorry for my terrible english...

Re: [ANNOUNCEMENT] Updated: dash 0.5.12-2

[Brian Inglis]

On 2023-02-15 07:05, Corinna Vinschen via Cygwin wrote:
> On Feb 15 14:52, Corinna Vinschen via Cygwin wrote:
>> The currently building cygwin test release 3.5.0-0.174.gd6d4436145b8
>> contains the new code.  Would you mind to build a dash for testing so we
>> can see if and how it works?

> You don't have to build a new dash.  Just downgrading to
> dash-0.5.12-1 will suffice, of course.  D'oh.
> So, as a first test, here's the result of Andrey's expression
> with dash-0.5.12-1:
>    $ dash -c '_x="   x"; echo "${_x#${_x%%[![:space:]]*}}"'
>    x

Good and quick work!
I was still looking thru my last Cygwin build for the Unicode tables, that I 
remember being rebuilt some time ago, and to find what used them, and whether 
they were updated to Unicode 15.

I'll install the test Cygwin and old dash, then prep and check a new/old dash 
release 3 in test, for release with untest when the Cygwin changes are released, 
rather than play with replacing versions and going back and forth.

-- 
Take care. Thanks, Brian Inglis			Calgary, Alberta, Canada

La perfection est atteinte			Perfection is achieved
non pas lorsqu'il n'y a plus rien à ajouter	not when there is no more to add
mais lorsqu'il n'y a plus rien à retirer	but when there is no more to cut
			-- Antoine de Saint-Exupéry

Re: [ANNOUNCEMENT] Updated: dash 0.5.12-2

[Corinna Vinschen]

Hi Brian,

On Feb 15 15:31, Brian Inglis via Cygwin wrote:
> On 2023-02-15 07:05, Corinna Vinschen via Cygwin wrote:
> > On Feb 15 14:52, Corinna Vinschen via Cygwin wrote:
> > > The currently building cygwin test release 3.5.0-0.174.gd6d4436145b8
> > > contains the new code.  Would you mind to build a dash for testing so we
> > > can see if and how it works?
> 
> > You don't have to build a new dash.  Just downgrading to
> > dash-0.5.12-1 will suffice, of course.  D'oh.
> > So, as a first test, here's the result of Andrey's expression
> > with dash-0.5.12-1:
> >    $ dash -c '_x="   x"; echo "${_x#${_x%%[![:space:]]*}}"'
> >    x
> 
> Good and quick work!
> I was still looking thru my last Cygwin build for the Unicode tables, that I
> remember being rebuilt some time ago, and to find what used them, and
> whether they were updated to Unicode 15.
> 
> I'll install the test Cygwin and old dash, then prep and check a new/old
> dash release 3 in test, for release with untest when the Cygwin changes are
> released, rather than play with replacing versions and going back and forth.

That *may* be a bit early... The patches are not bugfixes, but
implementing new features.  As such, you won't find them in a 3.4 bugfix
release.  They will hit the light with the first 3.5.0 release only,
probably end of 2023.


Corinna