cygwin-bash compat/regression bug... startup line prob

cygwin-bash compat/regression bug... startup line prob

[Linda Walsh]

This 'used' to work...but doesn't now: -- running
bash with the "-u" flag:

> t.sh
/usr/bin/bash: - : invalid option
Usage:  /usr/bin/bash [GNU long option] [option] ...
         /usr/bin/bash [GNU long option] [option] script-file ...
GNU long options:
         --debug
         --debugger
         --dump-po-strings
         --dump-strings
         --help
         --init-file
         --login
         --noediting
         --noprofile
         --norc
         --posix
         --protected
         --rcfile
         --restricted
         --verbose
         --version
         --wordexp
Shell options:
         -irsD or -c command or -O shopt_option          (invocation only)
         -abefhkmnptuvxBCHP or -o option
> which t.sh
/Users/law.Bliss/bin/t.sh
> cat bin/t.sh
#!/bin/bash -u -x
echo "Hello World"
>
-----------------------------

It has something to do with the "-u" switch -- the "-x" was
added to try to figure out why a script that had just
#!/bin/bash -u
died w/o executing a single line.

If I run it via:
> bash t.sh
Hello World
>
---works---
Also weird -- the interp line says "/bin/bash" not "/usr/bin/bash"
as the shell, so why does the error come from /usr/bin/bash?

It doesn't when I do it interactively:
> /bin/bash - - 
/bin/bash: -: No such file or directory
---
or 'bash' alone:
> bash - - 
bash: -: No such file or directory


----
Ideas?  would a cygcheck be of any use?

Note, if I combine the switches (-ux), it works -- but I only
added the '-x' to debug why '-u' didn't work.

> uname -a
CYGWIN_NT-6.1 Athenae 1.7.29(0.272/5/3) 2014-04-07 13:46 x86_64 Cygwin
> bash --version
GNU bash, version 4.1.11(2)-release (x86_64-unknown-cygwin)

---
This, also, doesn't work:
> echo t.sh|bash -i     #(with or without '-i')
#note -- first some unrelated output from confused startup scripts:

User Bliss\law's .bashrc called 2nd time
/usr/bin/stty: standard input: Inappropriate ioctl for device
/usr/bin/stty: standard input: Inappropriate ioctl for device
dirname: extra operand `a'
Try `dirname --help' for more information.
## here is the 'echo' being read into bash:

law.Bliss> t.sh
/usr/bin/bash: - : invalid option
Usage:  /usr/bin/bash [GNU long option] [option] ...
         /usr/bin/bash [GNU long option] [option] script-file ...
GNU long options:
...
-----


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: cygwin-bash compat/regression bug... startup line prob

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 776 bytes --]

On 04/28/2014 02:43 PM, Linda Walsh wrote:

>> cat bin/t.sh
> #!/bin/bash -u -x

This is an invalid shebang line.  Historically, you are allowed at most
ONE argument to the program that you will be executing.

#!/bin/bash -ux

is valid,

#!/bin/bash -u -x

is not.


> If I run it via:
>> bash t.sh
> Hello World

When you run it in this manner, you are NOT using the shebang line, but
are instead using the 'bash' found on a PATH search, which explains:

>>
> ---works---
> Also weird -- the interp line says "/bin/bash" not "/usr/bin/bash"
> as the shell, so why does the error come from /usr/bin/bash?

the answer to this question.

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: cygwin-bash compat/regression bug... startup line prob

[Linda Walsh]

Eric Blake wrote:
> On 04/28/2014 02:43 PM, Linda Walsh wrote:
>
> >> cat bin/t.sh
> > #!/bin/bash -u 
Um... it doesn't work with 1 argument either. 
>
> This is an invalid shebang line.  Historically, you are allowed at most
> ONE argument to the program that you will be executing.
?!? 
Historically?.. since when?  Never encountered that problem on linux
nor on cygwin.  This isn't a case of 2 args, just the "-u" by itself
doesn't work.

> > Also weird -- the interp line says "/bin/bash" not "/usr/bin/bash"
> > as the shell, so why does the error come from /usr/bin/bash?
>
> the answer to this question.
---
???
This isn't clear to me.  If I am running /bin/bash, why did the error message
say /usr/bin/bash?




--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: cygwin-bash compat/regression bug... startup line prob

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 4267 bytes --]

On 04/30/2014 11:57 PM, Linda Walsh wrote:
> Eric Blake wrote:
>> On 04/28/2014 02:43 PM, Linda Walsh wrote:
>>
>> >> cat bin/t.sh
>> > #!/bin/bash -u 
> Um... it doesn't work with 1 argument either.

Your context quoting is hard to follow. Here, you are complaining about
a she-bang with only one argument,...

>>
>> This is an invalid shebang line.  Historically, you are allowed at most
>> ONE argument to the program that you will be executing.
> ?!? Historically?.. since when?  Never encountered that problem on linux
> nor on cygwin.  This isn't a case of 2 args, just the "-u" by itself
> doesn't work.

...but here, you are quoting my reply to your original example that
included two arguments, although you managed to snip that original text
[for reference, my comment was about _this_ portion of your mail:

| > cat bin/t.sh
| #!/bin/bash -u -x

which is indeed invalid usage].  I made no comment confirming or denying
whether your use of '-u' by itself works (as I was not on cygwin when I
wrote my first reply, and so had no easy way to confirm or refute if
one-argument -u "works" on cygwin); but was merely pointing out that in
your further investigation of the issue, you were creating an
unsupported 2-argument she-bang line, and thus not testing what you
thought you were testing.

As for history of shebang,
https://en.wikipedia.org/wiki/Shebang_%28Unix%29 states that whether
subsequent arguments are split (Solaris) or provided as one argument
(Linux) has been a long time issue, quoting at least this article from
2008
http://mail-index.netbsd.org/netbsd-users/2008/11/09/msg002388.html,
although I'm sure you could trace it back at least to the 1980s.  The
Linux man pages for execve state:

       A maximum line length of 127 characters is allowed for the  first
 line
       in a #! executable shell script.

       The  semantics  of  the  optional-arg argument of an interpreter
script
       vary across implementations.  On Linux, the entire string
following the
       interpreter name is passed as a single argument to the
interpreter, and
       this string can include white space.  However, behavior differs
on some
       other  systems.   Some  systems  use the first white space to
terminate
       optional-arg.  On some systems, an interpreter script can have
multiple
       arguments,  and  white  spaces  in optional-arg are used to
delimit the
       arguments.

Since cygwin emulates Linux, it makes sense that Cygwin likewise accepts
exactly one argument on the shebang line.

> 
>> > Also weird -- the interp line says "/bin/bash" not "/usr/bin/bash"
>> > as the shell, so why does the error come from /usr/bin/bash?
>>
>> the answer to this question.
> ---
> ???
> This isn't clear to me.  If I am running /bin/bash, why did the error
> message
> say /usr/bin/bash?
> 

Because you weren't running /bin/bash at that point in time, but
/usr/bin/bash.  Again, you snipped the relevant portion of your original
mail:

| > bash t.sh

but that says to run 't.sh' using the 'bash' interpreter found first in
your PATH (which presumably had /usr/bin before /bin), and not as a
standalone executable.  A shebang line is ignored when you run a script
directly as an argument to 'bash' - the only time a shebang line is
honored is when it appears as argv[0] of the execve() family of
functions (your command line had it as argv[1]).  To use the shebang,
you have to execute 't.sh', not 'bash t.sh'.  Remember, when bash
executes a script, the #! line is a comment that does not further impact
how bash behaves.

(If you look at the bash source code, it does have fallback code to
emulate shebang magic on platforms where execve() fails due to lacking
shebang support.  But there are two things to note - this fallback does
NOT kick in on cygwin, since cygwin's execve() DOES have shebang
support; and even with that fallback code in bash, it is still only
triggered in instances where you try to execute a script as argv[0] of
the child process, and NOT when you pass the script name as argv[1] of
the bash process.)

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: cygwin-bash compat/regression bug... startup line prob

[Linda Walsh]

Eric Blake wrote:
> On 04/30/2014 11:57 PM, Linda Walsh wrote:
>> Eric Blake wrote:
>>> On 04/28/2014 02:43 PM, Linda Walsh wrote:
>>>
>>>>> cat bin/t.sh
>>>> #!/bin/bash -u 
>> Um... it doesn't work with 1 argument either.
> 
> Your context quoting is hard to follow. Here, you are complaining about
> a she-bang with only one argument,...
----
I think we are on two different pages.


While I had the example with -u + -x, later on in the same note,

-----
> It has something to do with the "-u" switch -- the "-x" was
> added to try to figure out why a script that had just
> #!/bin/bash -u
> died w/o executing a single line. 
-------------




> | > cat bin/t.sh
> | #!/bin/bash -u -x
> 
> which is indeed invalid usage]. 
----
Not on linux, which you say cygwin follows. *YOU* quoted it
saying:

>  On Linux, the entire string following the
>        interpreter name is passed as a single argument to the
> interpreter, and this string can include white space.
>        A maximum line length of 127 characters is allowed for the  first
>  line  in a #! executable shell script.
----------

On linux, (and, thus on cygwin?), "#!/bin/bash -u -x -a -b -c"
is passed as 1 argument to bash.  I.e. the spaces don't break things
into separate arguments on linux.  So any line up to 127 characters
is allowed and anything after the executable name is a single argument
to the end of the line (or 127 chars total).

I.e. the above is valid as well -- but it was the single "-u" switch
that doesn't work.



>>>> Also weird -- the interp line says "/bin/bash" not "/usr/bin/bash"
>>>> as the shell, so why does the error come from /usr/bin/bash?
>>> the answer to this question.
>> ---
>> ???
>> This isn't clear to me.  If I am running /bin/bash, why did the error
>> message
>> say /usr/bin/bash?
>>
> 
> Because you weren't running /bin/bash at that point in time, but
> /usr/bin/bash.  Again, you snipped the relevant portion of your original
----
No...I was... the output at the top was from "t.sh", which had
#!/bin/bash.

But the error message says /usr/bin/bash.


> mail:
> 
> | > bash t.sh
> 
> but that says to run 't.sh' using the 'bash' interpreter found first in
> your PATH
----
But that is not the first example in the email nor the one that gave
the /usr/bin/bash as the error message source.

I said later, that "bash t.sh" worked.. (as well as /bin/bash t.sh)...
it's only when invoked w/o specifying the interpreter on the command
line.  (i.e. going from the shebang line which says /bin/bash)

So why doesn't a single argument work (-u?)




--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

two execve() inconsistencies [was: cygwin-bash compat/regression bug... startup line prob]

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 5708 bytes --]

On 05/01/2014 12:11 PM, Linda Walsh wrote:

> 
> On linux, (and, thus on cygwin?), "#!/bin/bash -u -x -a -b -c"
> is passed as 1 argument to bash.  I.e. the spaces don't break things
> into separate arguments on linux.

Correct.  So follow it through to it's logical conclusion:

/bin/bash "-u -x -a -b -c"

is trying to invoke bash with one GIANT short option string as argv[1].
 Per getopt() parsing rules, it is as if you were trying to invoke:

/bin/bash "-u" "- " "--" "-x" "- " "--" "-a" "- " "--" "-b" "- " "--" "-c"

But bash does NOT have a "- " short option (that is, the short option
whose name is a space character).  You have formed an illegal command
line, and bash is trying to tell you that.  It would be nicer if bash
quoted the first option it doesn't recognize (/usr/bin/bash: "- ":
invalid option) rather than hoping you are smart enough to detect the
trailing space meant that you were trying to pass a space as the option
name (/usr/bin/bash: - : invalid option); but such is life.

> 
> I.e. the above is valid as well -- but it was the single "-u" switch
> that doesn't work.

But my point remains - you WEREN'T passing a single "-u" switch, but
MUST have had a trailing space on the line, and were passing "-u " which
bash is dutifully trying to parse as "-u" "- " and erroring out on the "- ".

However, now that I have further tested on both Linux and cygwin
systems, I CAN state that you have indeed found an inconsistency between
how Cygwin execve() and Linux execve() behave - so if someone is willing
to write a patch to cygwin1.dll to eliminate that inconsistency, it
would be nice.

On Linux, execve() strips trailing space before forming the command line
that gets substituted during shebang execution.  Sadly, this aspect of
Linux is NOT documented in the man page; I had to learn it by
experimentation.  Observe this Linux transcript:

$ cd /tmp
$ printf '#!/bin/sh -u \necho hi\n' > blah.sh
$ chmod +x blah.sh
$ ./blah.sh
hi
$ printf '#!/bin/sh -u -\necho hi\n' > blah.sh
$ ./blah.sh 2>&1 |head -n1
/bin/sh: - : invalid option

Note that when the shebang had "-u ", the trailing space was stripped,
and bash was run with "-u"; but when it had "-u -", there was no
trailing space to strip, and bash given the literal argument "-u -"
dutifully choked on the attempt to parse "- " as a short option.

Cygwin, on the other hand, does not (yet) strip trailing spaces when
constructing the replacement command line from a shebang.  Patches
thoughtfully considered.  Particularly true since some people use
carriage returns on script files under cygwin (I personally detest the
practice, but have at least patched bash to try to allow others to use
it) - stripping trailing whitespace when forming the argv[1] of a
shebang line would allow the shebang to do the right thing of NOT
passing the carriage return to the actual command line of the
interpreter, even if the script is accidentally executed from a binary
mount rather than a text mount.

>> Because you weren't running /bin/bash at that point in time, but
>> /usr/bin/bash.  Again, you snipped the relevant portion of your original
> ----
> No...I was... the output at the top was from "t.sh", which had
> #!/bin/bash.
> 
> But the error message says /usr/bin/bash.

Okay, now that I have tested on a Cygwin machine, I have indeed
reproduced what you are seeing.  When you invoke './t.sh', which has a
shebang calling out '#!/bin/sh', but in a way that provokes an error,
the error calls out /usr/bin/sh.  That means that the execve() logic for
constructing the alternative command line is passing a different argv[0]
to the replacement interpreter than what you actually specified in your
shebang line.  On cygwin, /bin/bash and /usr/bin/bash are ALWAYS the
same file (two different spellings, but the cygwin mount code treats
/bin/ and /usr/bin/ as mounts to the same single Windows directory).
The execve() code responsible for locating the replacement application
to run is first locating "/bin/bash" as written in your shebang line,
then constructing the argv[] to pass to that application - and somewhere
in that process, it ends up rewriting argv[0] to match what it considers
to be the canonical name.  Observe this cygwin transcript:

$ cygpath -w /bin/bash
C:\cygwin\bin\bash
$ cygpath -w /usr/bin/bash
C:\cygwin\bin\bash
$ cygpath -u "C:\cygwin\bin\bash"
/usr/bin/bash

So, although both Unix names are usable, it is as if the execve code has
run your spelling through a conversion to a windows name, then converted
that windows name into the argv[0] for use, rather than preserving your
original argv[0].  Not the end of the world for bash (although bash DOES
care about the spelling of argv[0], it is mostly on whether the basename
was 'sh' or 'bash', and NOT on what directory name it detected) - but it
COULD be problematic for other apps (remember the recent traffic on this
list about how '/bin/gcc' failed while '/usr/bin/gcc' succeeded, all
because the program used argv[0] to construct relative names to
resources where /usr/bin/../share exists but /bin/../share does not
exist?).  So, a patch to cygwin execve() code to preserve the user's
original argv[0] spelling would be appreciated.

> So why doesn't a single argument work (-u?)

Because you didn't pass a single argument, but left trailing space which
bash tried to interpret as a second argument, and because no one has yet
provided the patch for the two execve() inconsistencies that this thread
has uncovered.

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: two execve() inconsistencies (was cygwin-bash compat/regression bug... startup line prob)

[Linda Walsh]

Eric Blake wrote:
>> So why doesn't a single argument work (-u?)
> 
> Because you didn't pass a single argument, but left trailing space ...
---
But looking at the file, I didn't notice that until
I recreated the file w/o using cut/paste (i.e. happened
in another script that had -u  at the top, and I went
to create a test file from that script by copying it into
the test-name, then whittled down the test file till I just
had the she-bang line -- which *looked* fine...sigh.

Well, I had run into two different problems at the same
time, that's more than my daily limit...;^/


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

cygwin-bash compat/regression bug... startup line prob

[Linda Walsh]

Bob McGowan wrote:

> >> Because you weren't running /bin/bash at that point in time, but
> >> /usr/bin/bash.  Again, you snipped the relevant portion of your original
> > ----
> > No...I was... the output at the top was from "t.sh", which had
> > #!/bin/bash.
> >
> > But the error message says /usr/bin/bash.
----
   That's why I commented that something was bogus.  Here it is
again:
law.Bliss> cat /tmp/t.sh
#!/bin/bash -u
echo -e "Hello World\n"

law.Bliss> t.sh
/usr/bin/bash: - : invalid option
Usage:  /usr/bin/bash [GNU long option] [option] ...
       /usr/bin/bash [GNU long option] [option] script-file ...
GNU long options:
----
The reason the above fails... (I got it to work).. is that if there is a space
on the line after the "-u", that also fails.

I didn't deliberately put one there, but that it no longer handles
separate options is an evolution of software devolution.

But NOTE -- the above error claims it is coming from /usr/bin/bash.
That's not my login shell nor on the shebang line.

That's cygwin-black-magic... converting all /bin paths to /usr/bin...even
shells?...


>
>
> Regarding how your script is executed:
>
> If you do "$ bash t.sh", the she-bang line is *ignored*, it's just a
> comment in this case.
---
I'm aware of that -- I was noting that the alternate ways of invoking it
made a difference.  Not that I didn't understand why.






--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: cygwin-bash compat/regression bug... startup line prob

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 1295 bytes --]

On 05/01/2014 04:11 PM, Linda Walsh wrote:

> The reason the above fails... (I got it to work).. is that if there is a
> space
> on the line after the "-u", that also fails.
> 
> I didn't deliberately put one there, but that it no longer handles
> separate options is an evolution of software devolution.

It NEVER worked to put more than one option.  I don't know why you are
arguing "software devolution" when there has been no regression in
behavior, rather you are just experiencing something that has always
been this way.  And I already explained in the other message that yes
indeed cygwin is different than Linux in that trailing whitespace is
passed through on cygwin but stripped in Linux.

> 
> But NOTE -- the above error claims it is coming from /usr/bin/bash.
> That's not my login shell nor on the shebang line.
> 
> That's cygwin-black-magic... converting all /bin paths to /usr/bin...even
> shells?...

I already explained in the other mail that this is a side-effect of
cygwin converting to windows paths and then back during execve shebang
computation, rather than preserving the argv[0] of the user.  You are
welcome to provide a patch.

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: cygwin-bash compat/regression bug... startup line prob

[Linda Walsh]

Eric Blake wrote:
> On 05/01/2014 04:11 PM, Linda Walsh wrote:
> 
>> The reason the above fails... (I got it to work).. is that if there is a
>> space
>> on the line after the "-u", that also fails.
>>
>> I didn't deliberately put one there, but that it no longer handles
>> separate options is an evolution of software devolution.
> 
> It NEVER worked to put more than one option.  I don't know why you are
> arguing "software devolution" when there has been no regression in
> behavior, rather you are just experiencing something that has always
> been this way.  And I already explained in the other message that yes
> indeed cygwin is different than Linux in that trailing whitespace is
> passed through on cygwin but stripped in Linux.
> 
>> But NOTE -- the above error claims it is coming from /usr/bin/bash.
>> That's not my login shell nor on the shebang line.
>>
>> That's cygwin-black-magic... converting all /bin paths to /usr/bin...even
>> shells?...
> 
> I already explained in the other mail that this is a side-effect of
> cygwin converting to windows paths and then back during execve shebang
> computation, rather than preserving the argv[0] of the user.
---
	Like I said -- cygwin-black-magic.  Converting paths to windows
and back again... only innermost practitioner's of windows-magic would
create such spells! ;-)


>  You are welcome to provide a patch.
----
	Aw, shucks, that's awfully nice of you, and if my understanding
of cygwin was such that contributing such a patch wouldn't take more time
learning how to make cygwin than designing and implementing the patch,
I might do such, but it wouldn't be the most efficient use of my time.

    As it stands I need to work around cygwin ignoring Windows
mount-points and turning them into symlinks too often and would likely
consider fixing that a higher priority task, though since it's a
reverting an "extra hack" put in to treat windows mount-points
(Junctions) as symlinks, I wouldn't think fixing that would be that
difficult,

     Personally, I'd like to see linux running as a subsystem on windows
like the old POSIX sybsystem.  That would solve all sorts of compat
problems.  ;-)    (ya, I know, when pigs fly)...



--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple