bash shell script: recently running, now failing

bash shell script: recently running, now failing

[Fergus Daly]

I have a "hash bang" bash shell script i.e. first line
#! /bin/sh
or equivalently
#! /bin/bash
For various reasons I want this file to be identified as binary so its second line
is the single character null \x00 showing up in some editors e.g. nano as
 ^@
This does not prevent the script from running to a successful conclusion.
Or not until recently. Now the script fails with
/home/user/bin/file.old.sh: cannot execute binary file
Q1 - was bash recently updated? Would this explain the changed behaviour?
Q2 - if so, is this newly introduced "glitch" known and presumably intended? Or
an unintended consequence that will be retracted in a later update? 
I then altered the first line to
#! /bin/dash
whilst retaining the null character at line 2 and subsequent content also unaltered..
The altered script file.new.sh runs as previously to a successful conclusion.
Q3 - at 1/8 the size of bash and sh, I am not at all sure of the role and reach of dash.
Should the edit (dash replacing bash/sh) be incorporated elsewhere or would this be a
bad idea (and retained only locally in what is indeed an eccentric and one-off context)?

Re: bash shell script: recently running, now failing

[Corinna Vinschen]

On Apr  6 04:43, Fergus Daly via Cygwin wrote:
> I have a "hash bang" bash shell script i.e. first line
> #! /bin/sh
> or equivalently
> #! /bin/bash
> For various reasons I want this file to be identified as binary so its second line
> is the single character null \x00 showing up in some editors e.g. nano as
>  ^@
> This does not prevent the script from running to a successful conclusion.
> Or not until recently. Now the script fails with
> /home/user/bin/file.old.sh: cannot execute binary file
> Q1 - was bash recently updated? Would this explain the changed behaviour?

bash was recently updated from 4.4.12 to 5.2.15.

The behaviour is the same in bash on Linux.  Take this file with
a \0 in line 2:

  $ cat -v x.sh
  #! /bin/bash
  ^@
  echo foo
  $ bash --version | head -1
  GNU bash, version 5.2.15(1)-release (x86_64-redhat-linux-gnu)
  $ ./x.sh
  ./x.sh: ./x.sh: cannot execute binary file

While dash on Linux runs the script:

  $ sed -i -e 's/bash/dash/' x.sh
  $ ./x.sh
  foo

> Q2 - if so, is this newly introduced "glitch" known and presumably
> intended? Or an unintended consequence that will be retracted in a
> later update? 

Bash follows the POSIX standard:
https://pubs.opengroup.org/onlinepubs/9699919799/utilities/sh.html#tag_20_117_07
https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap03.html#tag_03_403

So I don't expect this will change any time soon.

> Q3 - at 1/8 the size of bash and sh, I am not at all sure of the role and reach of dash.

Dash is a minimal shell with no bells and whistles.  It loads ands runs
slightly faster than bash.  If you only need bare minimum bourne shell
behaviour, it's a good choice for scripts.


Corinna

Re: bash shell script: recently running, now failing

[Andrey Repin]

[-- Attachment #1: Type: text/plain, Size: 760 bytes --]

Greetings, Fergus Daly!

> I have a "hash bang" bash shell script i.e. first line
> #! /bin/sh
> or equivalently
> #! /bin/bash

By default, sh is bash in base Cygwin installation.

> Q3 - at 1/8 the size of bash and sh, I am not at all sure of the role and reach of dash.
> Should the edit (dash replacing bash/sh) be incorporated elsewhere or would this be a
> bad idea (and retained only locally in what is indeed an eccentric and one-off context)?

I'm replacing /bin/sh with dash as I've found that even in POSIX mode, bash
allows for a lot of bash'izms in scripts, which would not otherwise run under
(d?a)?sh.

See the post-install script attached.


-- 
With best regards,
Andrey Repin
Thursday, April 6, 2023 15:18:19

Sorry for my terrible english...

[-- Attachment #2: zp_dash-as-bin-sh.dash --]
[-- Type: application/octet-stream, Size: 279 bytes --]

#!/bin/dash

[ -f /bin/dash.exe ] || exit

_f="$(mktemp /bin/sh.prelink.XXXXXX)"

[ -L /bin/sh.exe ] || mv /bin/sh.exe "$_f"

ln -fsT /bin/dash.exe /bin/sh.exe
if [ $? -gt 0 ]; then
    echo "Got a problem creating link for /bin/sh !" >&2
    exit 1
else
    rm "$_f"
fi

exit 0

Re: bash shell script: recently running, now failing

[Scott Smith]

[-- Attachment #1: Type: text/plain, Size: 2284 bytes --]

Place the nul on the third line. For example:

#!/bin/bash
#
# ^@ identify as a binary file
...



On Thu, Apr 6, 2023 at 4:03 AM Corinna Vinschen via Cygwin <
cygwin@cygwin.com> wrote:

> On Apr  6 04:43, Fergus Daly via Cygwin wrote:
> > I have a "hash bang" bash shell script i.e. first line
> > #! /bin/sh
> > or equivalently
> > #! /bin/bash
> > For various reasons I want this file to be identified as binary so its
> second line
> > is the single character null \x00 showing up in some editors e.g. nano as
> >  ^@
> > This does not prevent the script from running to a successful conclusion.
> > Or not until recently. Now the script fails with
> > /home/user/bin/file.old.sh: cannot execute binary file
> > Q1 - was bash recently updated? Would this explain the changed behaviour?
>
> bash was recently updated from 4.4.12 to 5.2.15.
>
> The behaviour is the same in bash on Linux.  Take this file with
> a \0 in line 2:
>
>   $ cat -v x.sh
>   #! /bin/bash
>   ^@
>   echo foo
>   $ bash --version | head -1
>   GNU bash, version 5.2.15(1)-release (x86_64-redhat-linux-gnu)
>   $ ./x.sh
>   ./x.sh: ./x.sh: cannot execute binary file
>
> While dash on Linux runs the script:
>
>   $ sed -i -e 's/bash/dash/' x.sh
>   $ ./x.sh
>   foo
>
> > Q2 - if so, is this newly introduced "glitch" known and presumably
> > intended? Or an unintended consequence that will be retracted in a
> > later update?
>
> Bash follows the POSIX standard:
>
> https://pubs.opengroup.org/onlinepubs/9699919799/utilities/sh.html#tag_20_117_07
>
> https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap03.html#tag_03_403
>
> So I don't expect this will change any time soon.
>
> > Q3 - at 1/8 the size of bash and sh, I am not at all sure of the role
> and reach of dash.
>
> Dash is a minimal shell with no bells and whistles.  It loads ands runs
> slightly faster than bash.  If you only need bare minimum bourne shell
> behaviour, it's a good choice for scripts.
>
>
> Corinna
>
> --
> Problem reports:      https://cygwin.com/problems.html
> FAQ:                  https://cygwin.com/faq/
> Documentation:        https://cygwin.com/docs.html
> Unsubscribe info:     https://cygwin.com/ml/#unsubscribe-simple
>

Re: bash shell script: recently running, now failing

[Adam Dinwoodie]

On Thu, Apr 06, 2023 at 04:43:51AM +0000, Fergus Daly via Cygwin wrote:
> I have a "hash bang" bash shell script i.e. first line
> #! /bin/sh
> or equivalently
> #! /bin/bash
> For various reasons I want this file to be identified as binary so its second line
> is the single character null \x00 showing up in some editors e.g. nano as
>  ^@
> This does not prevent the script from running to a successful conclusion.
> Or not until recently. Now the script fails with
> /home/user/bin/file.old.sh: cannot execute binary file
> Q1 - was bash recently updated? Would this explain the changed behaviour?
> Q2 - if so, is this newly introduced "glitch" known and presumably intended? Or
> an unintended consequence that will be retracted in a later update? 
> I then altered the first line to
> #! /bin/dash
> whilst retaining the null character at line 2 and subsequent content also unaltered..
> The altered script file.new.sh runs as previously to a successful conclusion.
> Q3 - at 1/8 the size of bash and sh, I am not at all sure of the role and reach of dash.
> Should the edit (dash replacing bash/sh) be incorporated elsewhere or would this be a
> bad idea (and retained only locally in what is indeed an eccentric and one-off context)?

Dash is smaller and much less feature-rich than Bash.  Whether Dash is a
suitable replacement for Bash depends on how much (if at all) you're
relying on Bash-specific functions.  For very simple scripts, the only
difference is likely that Dash will be very slightly faster, but working
out whether your script is using any "Bashisms" isn't always a trivial
job.

(I have previously been involved work in migrating scripts between Ksh
and Bash, which is a similar-but-different problem, and there were *a
lot* of surprises in how the two differed.)

Depending on why you want the file to be identified as a binary, and how
that identification is being done, you could move your null byte later
in the file.  In particular, a pattern I've seen several times in Bash
is to have a normal Bash script, finishing with an explicit `exit`,
followed by an actual binary blob; this can be used to create things
like self-extracting bundles, where the binary blob is a tarball and the
script at the top of the file has the instructions for extracting the
tarball.

Re: bash shell script: recently running, now failing

[Scott Smith]

[-- Attachment #1: Type: text/plain, Size: 3882 bytes --]

Coming from an enterprise and supercomputing background, we were able to
control what shell was available, so bashisms weren't a problem any more
than dashisms, fishisms, kornisms, perl or python versionisms, etc, might
be.

But, when I was in a commercial environment, everything - shell, perl, C,
text tools, etc - had to be tested and usage adjusted accordingly. It
wasn't uncommon to have a starter script that performed compatibility tests
and forked to the appropriate version... of the same code, functionally.

Back to the OP's problem.

I haven't seen the bash source, but if I had to guess, based on behavior,
bash is only checking the line after the she-bang. With a little more
testing, I've concluded that the nul ^@ can appear anywhere other than in
line #2. My guess is that, after finding the she-bang, bash is reading one
more line to make the "is or is not binary" determination. I imagine that
is oodles less overhead than scanning the entire file.

I have not tested the behavior if the first line is not a she-bang, such as
if the script is run via argument to bash.



On Thu, Apr 6, 2023 at 1:19 PM Adam Dinwoodie via Cygwin <cygwin@cygwin.com>
wrote:

> On Thu, Apr 06, 2023 at 04:43:51AM +0000, Fergus Daly via Cygwin wrote:
> > I have a "hash bang" bash shell script i.e. first line
> > #! /bin/sh
> > or equivalently
> > #! /bin/bash
> > For various reasons I want this file to be identified as binary so its
> second line
> > is the single character null \x00 showing up in some editors e.g. nano as
> >  ^@
> > This does not prevent the script from running to a successful conclusion.
> > Or not until recently. Now the script fails with
> > /home/user/bin/file.old.sh: cannot execute binary file
> > Q1 - was bash recently updated? Would this explain the changed behaviour?
> > Q2 - if so, is this newly introduced "glitch" known and presumably
> intended? Or
> > an unintended consequence that will be retracted in a later update?
> > I then altered the first line to
> > #! /bin/dash
> > whilst retaining the null character at line 2 and subsequent content
> also unaltered..
> > The altered script file.new.sh runs as previously to a successful
> conclusion.
> > Q3 - at 1/8 the size of bash and sh, I am not at all sure of the role
> and reach of dash.
> > Should the edit (dash replacing bash/sh) be incorporated elsewhere or
> would this be a
> > bad idea (and retained only locally in what is indeed an eccentric and
> one-off context)?
>
> Dash is smaller and much less feature-rich than Bash.  Whether Dash is a
> suitable replacement for Bash depends on how much (if at all) you're
> relying on Bash-specific functions.  For very simple scripts, the only
> difference is likely that Dash will be very slightly faster, but working
> out whether your script is using any "Bashisms" isn't always a trivial
> job.
>
> (I have previously been involved work in migrating scripts between Ksh
> and Bash, which is a similar-but-different problem, and there were *a
> lot* of surprises in how the two differed.)
>
> Depending on why you want the file to be identified as a binary, and how
> that identification is being done, you could move your null byte later
> in the file.  In particular, a pattern I've seen several times in Bash
> is to have a normal Bash script, finishing with an explicit `exit`,
> followed by an actual binary blob; this can be used to create things
> like self-extracting bundles, where the binary blob is a tarball and the
> script at the top of the file has the instructions for extracting the
> tarball.
>
> --
> Problem reports:      https://cygwin.com/problems.html
> FAQ:                  https://cygwin.com/faq/
> Documentation:        https://cygwin.com/docs.html
> Unsubscribe info:     https://cygwin.com/ml/#unsubscribe-simple
>

Re: bash shell script: recently running, now failing

[Brian Inglis]

On 2023-04-06 06:21, Andrey Repin via Cygwin wrote:
>> I have a "hash bang" bash shell script i.e. first line
>> #! /bin/sh
>> or equivalently
>> #! /bin/bash

> By default, sh is bash in base Cygwin installation.

>> Q3 - at 1/8 the size of bash and sh, I am not at all sure of the role and reach of dash.
>> Should the edit (dash replacing bash/sh) be incorporated elsewhere or would this be a
>> bad idea (and retained only locally in what is indeed an eccentric and one-off context)?

> I'm replacing /bin/sh with dash as I've found that even in POSIX mode, bash
> allows for a lot of bash'izms in scripts, which would not otherwise run under
> (d?a)?sh.

> See the post-install script attached.

You should use either a hard link or copy of d/ash from/to /bin/ to allow it (or 
any shell) to be used from Windows cmd shells or Scheduled Tasks, which do not 
recognise Cygwin /usr mounts or symlinks, but can be run with elevated 
privileges and at sometimes more useful points than user cron jobs.
For example, "manually" restarting Cygwin services after delayed startup, 
stopping Cygwin services and processes before upgrades, and cleaning up cron 
jobs that have not finished.
You should also consider changing the sh man page.

-- 
Take care. Thanks, Brian Inglis              Calgary, Alberta, Canada

La perfection est atteinte                   Perfection is achieved
non pas lorsqu'il n'y a plus rien à ajouter  not when there is no more to add
mais lorsqu'il n'y a plus rien à retirer     but when there is no more to cut
                                 -- Antoine de Saint-Exupéry

Re: bash shell script: recently running, now failing

[Andrey Repin]

Greetings, Brian Inglis via Cygwin!

> On 2023-04-06 06:21, Andrey Repin via Cygwin wrote:
>>> I have a "hash bang" bash shell script i.e. first line
>>> #! /bin/sh
>>> or equivalently
>>> #! /bin/bash

>> By default, sh is bash in base Cygwin installation.

>>> Q3 - at 1/8 the size of bash and sh, I am not at all sure of the role and reach of dash.
>>> Should the edit (dash replacing bash/sh) be incorporated elsewhere or would this be a
>>> bad idea (and retained only locally in what is indeed an eccentric and one-off context)?

>> I'm replacing /bin/sh with dash as I've found that even in POSIX mode, bash
>> allows for a lot of bash'izms in scripts, which would not otherwise run under
>> (d?a)?sh.

>> See the post-install script attached.

> You should use either a hard link or copy of d/ash from/to /bin/ to allow
> it (or any shell) to be used from Windows cmd shells or Scheduled Tasks,

That depends on your setup. /usr/bin/env is a real executable and it is a VERY
good idea to handle Windows associations through it.
But on my end it is covered by CYGWIN=winsymlinks:nativestrict and a more
elaborate TCC-RT wrapper script which is again falling down to /usr/bin/env for
the purposes of actually starting the program, be it a script or binary or…

> which do not recognise Cygwin /usr mounts or symlinks, but can be run with
> elevated privileges and at sometimes more useful points than user cron jobs.
> For example, "manually" restarting Cygwin services after delayed startup,
> stopping Cygwin services and processes before upgrades, and cleaning up cron
> jobs that have not finished.

That part is handled elsewhere already.

> You should also consider changing the sh man page.

That's a good idea, thanks. I never did "man sh" so never stumbled across that
discrepancy.


-- 
With best regards,
Andrey Repin
Saturday, April 8, 2023 11:31:34

Sorry for my terrible english...