Challenge: a VERY strange problem with command substitution in bash

Challenge: a VERY strange problem with command substitution in bash

[Jürgen Wagner]

[-- Attachment #1: Type: text/plain, Size: 2990 bytes --]

Hello,
  this is my first posting here as I do not see any other hope of
getting this resolved. Research in mailing lists and other Cygwin users'
questions on various sites have not proven to be useful.

1. Ubuntu 4.4.0-71-generic x86_64, /bin/bash

$ value="$( date | cat )"; echo "$? <$value>"
0 <Tue Jul 11 20:21:22 CEST 2017>
$ value="$( date )"; echo "$? <$value>"
0 <Tue Jul 11 20:21:59 CEST 2017>
$ date | cat
Tue Jul 11 20:22:22 CEST 2017

Results are the same for normal users and for root. Everything is fine
and as expected.

2. Windows 10, Version 1703, Build 15063.413 on a Dell 64bit platform,
latest 64bit Cygwin (CYGWIN_NT-10.0, 2.8.1(0.312/5/3) 2017-07-03 14:11
x86_64 Cygwin) /bin/bash

$ value="$( date | cat )"; echo "$? <$value>"
127 <>
$ value="$( date )"; echo "$? <$value>"
0 <Tue Jul 11 20:25:01 CEST 2017>
$ date | cat
Tue Jul 11 20:25:17 CEST 2017

Results are the same for normal users and for users with administrative
rights, even if I run this under the TrustedInstaller rights.
Fully-qualified paths (/bin/date and /bin/cat) does not change the results.

3. On the same Windows 10 system as above, the following forms work fine

$ value="$( date | cat )"; echo "$? <$value>"
0 <Tue Jul 11 20:26:12 CEST 2017>
$ value="$( bash -c 'date | cat' )"; echo "$? <$value>"
0 <Tue Jul 11 20:25:29 CEST 2017>
# value="$( date ; pwd )"; echo "$? <$value>"
0 <Tue Jul 11 20:28:08 CEST 2017
/home/juergen>


Using backquotes instead of the command substitution with $(...) does
not change the results. I could swear this did work in an earlier
version of Cygwin on my Windows 7 machine.

I tried this to see if the code in the parentheses is executed at all:

$ value="$( date 2> foo | cat )"; echo "$? <$value>"

The file "foo" was not created, i.e., it seems the commands don't really
get executed.

$ value="$( date && pwd )"; echo "$? <$value>"
0 <Tue Jul 11 20:49:09 CEST 2017
/home/juergen>

$ value="$( date || pwd )"; echo "$? <$value>"
0 <Tue Jul 11 20:32:27 CEST 2017>

both work, so some control structures seem to be permissible... just not
a pipe.

What is going on? Some misconfiguration? A Cygwin bug? Some interaction
with something weird in Windows 10? I am at loss to understand what
could be wrong... and am now most curious whether anybody has an idea
what is causing this. Does it work/not work in the same way in your
Cygwin installation?

I came across this effect because ssh-host-config did not recognize me
as administrator anymore. It's due to a check for a certain user group
that uses a command substitution with a pipe. Replacing this with an
equivalent command works. The original line used "id -G" and then a
"grep -Eq" to check whether a certain group is on that list.

I am VERY curious now! I've rarely been puzzled that much by such a very
elementary shell expression (looking back at 34 years of Unix experience).

Thanks much in advance!
--Jürgen



[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 3986 bytes --]

Re: Challenge: a VERY strange problem with command substitution in bash

[Jürgen Wagner]

[-- Attachment #1: Type: text/plain, Size: 105 bytes --]

Sorry, little omission: the first example under section 3 is not with
bash, it was done with dash.



[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 3986 bytes --]

Re: Challenge: a VERY strange problem with command substitution in bash

[Jürgen Wagner]

[-- Attachment #1: Type: text/plain, Size: 417 bytes --]

And here another little detail: I installed Babun on the Windows 10 machine.

juergen@saraswati ~
$ value="$( date | cat )"; echo $? $value
0 Tue, Jul 11, 2017 10:24:02 PM

juergen@saraswati ~
$ echo $BASH_VERSION
4.3.33(1)-release

juergen@saraswati ~
$

It works.

The BASH_VERSION on the other Cygwin installation is 4.4.12(3)-release.
The command substitution with a pipe doesn't work there.



[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 3986 bytes --]

Re: Challenge: a VERY strange problem with command substitution in bash

[Steven Penny]

On Tue, 11 Jul 2017 21:12:14, =?UTF-8?Q?J=c3=bcrgen_Wagner?= wrote:
> 2. Windows 10, Version 1703, Build 15063.413 on a Dell 64bit platform,
> latest 64bit Cygwin (CYGWIN_NT-10.0, 2.8.1(0.312/5/3) 2017-07-03 14:11
> x86_64 Cygwin) /bin/bash
> 
> $ value=3D"$( date | cat )"; echo "$? <$value>"
> 127 <>

For what it is worth, I cannot repro this with Windows 7:

    $ value="$( date | cat )"; echo "$? <$value>"
    0 <Tue, Jul 11, 2017  6:16:21 PM>

Also, your example is poor. As you well know, "cat" in this case should be a
noop:

    $ date | cat
    Tue, Jul 11, 2017  6:20:53 PM

Perhaps you can provide a sane example that fails? Something like:

    $ date | sed s/M/Z/
    Tue, Jul 11, 2017  6:21:46 PZ


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Challenge: a VERY strange problem with command substitution in bash

[Richard Beels via cygwin]

At 07/11/2017 at 15:12, Shakespearean monkeys 
danced on Jürgen Wagner's keyboard and said:
>...
>Using backquotes instead of the command 
>substitution with $(...) does not change the 
>results. I could swear this did work in an 
>earlier version of Cygwin on my Windows 7 machine.
>
>I tried this to see if the code in the parentheses is executed at all:
>
>$ value="$( date 2> foo | cat )"; echo "$? <$value>"
>
>The file "foo" was not created, i.e., it seems 
>the commands don't really get executed.
>
>$ value="$( date && pwd )"; echo "$? <$value>"
>0 <Tue Jul 11 20:49:09 CEST 2017
>/home/juergen>
>
>$ value="$( date || pwd )"; echo "$? <$value>"
>0 <Tue Jul 11 20:32:27 CEST 2017>
>
>both work, so some control structures seem to be 
>permissible... just not a pipe.
>
>What is going on? Some misconfiguration? A 
>Cygwin bug? Some interaction with something 
>weird in Windows 10? I am at loss to understand 
>what could be wrong... and am now most curious 
>whether anybody has an idea what is causing 
>this. Does it work/not work in the same way in your Cygwin installation?
>
>I came across this effect because 
>ssh-host-config did not recognize me as 
>administrator anymore. It's due to a check for a 
>certain user group that uses a command 
>substitution with a pipe. Replacing this with an 
>equivalent command works. The original line used 
>"id -G" and then a "grep -Eq" to check whether a certain group is on that list.
>
>I am VERY curious now! I've rarely been puzzled 
>that much by such a very elementary shell 
>expression (looking back at 34 years of Unix experience).


Hi Jurgen.

90% chance it's what is called bloda in these 
parts.  It's in the FAQ on cygwin.com.  I'll go 
out on a limb and say you might have just 
installed/changed your AV/Firewall software.

And if I want to be super-psychic, can I guess 
comodo?  Because I just changed to comodo a 
couple weeks ago and had the same 
subshell/command substitution/pipeline errors you're mentioning.

If so, you need to exclude your cygwin folder 
from AV scanning. AND...  if the software does 
whitelisting or host intrusion protection (HIPS) 
or "run unknown executables in a 
container/sandbox" or something similar, you also 
need to trust all the executables, too.  Or 
switch to something else that doesn't trip cygwin's trigger.

After doing that and a rebaseall, I haven't had a 
fork error in a week.  I can't wait to run setup 
and come up with an update process, though...

Cheers!


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Challenge: a VERY strange problem with command substitution in bash

[Wouter van Doorn]

I bet Comodo is the golden tip. They have introduced whitelisting
without telling anyone, and I have had very strange behaviour (strange
until that whitelist explained it) too. Including that subshell thing.
They call it 'auto-containment'. Just disable that, and done.
Wouter

On 12 July 2017 at 00:22, Richard Beels via cygwin <cygwin@cygwin.com> wrote:
> At 07/11/2017 at 15:12, Shakespearean monkeys danced on Jürgen Wagner's
> keyboard and said:
>>
>> ...
>> Using backquotes instead of the command substitution with $(...) does not
>> change the results. I could swear this did work in an earlier version of
>> Cygwin on my Windows 7 machine.
>>
>> I tried this to see if the code in the parentheses is executed at all:
>>
>> $ value="$( date 2> foo | cat )"; echo "$? <$value>"
>>
>> The file "foo" was not created, i.e., it seems the commands don't really
>> get executed.
>>
>> $ value="$( date && pwd )"; echo "$? <$value>"
>> 0 <Tue Jul 11 20:49:09 CEST 2017
>> /home/juergen>
>>
>> $ value="$( date || pwd )"; echo "$? <$value>"
>> 0 <Tue Jul 11 20:32:27 CEST 2017>
>>
>> both work, so some control structures seem to be permissible... just not a
>> pipe.
>>
>> What is going on? Some misconfiguration? A Cygwin bug? Some interaction
>> with something weird in Windows 10? I am at loss to understand what could be
>> wrong... and am now most curious whether anybody has an idea what is causing
>> this. Does it work/not work in the same way in your Cygwin installation?
>>
>> I came across this effect because ssh-host-config did not recognize me as
>> administrator anymore. It's due to a check for a certain user group that
>> uses a command substitution with a pipe. Replacing this with an equivalent
>> command works. The original line used "id -G" and then a "grep -Eq" to check
>> whether a certain group is on that list.
>>
>> I am VERY curious now! I've rarely been puzzled that much by such a very
>> elementary shell expression (looking back at 34 years of Unix experience).
>
>
>
> Hi Jurgen.
>
> 90% chance it's what is called bloda in these parts.  It's in the FAQ on
> cygwin.com.  I'll go out on a limb and say you might have just
> installed/changed your AV/Firewall software.
>
> And if I want to be super-psychic, can I guess comodo?  Because I just
> changed to comodo a couple weeks ago and had the same subshell/command
> substitution/pipeline errors you're mentioning.
>
> If so, you need to exclude your cygwin folder from AV scanning. AND...  if
> the software does whitelisting or host intrusion protection (HIPS) or "run
> unknown executables in a container/sandbox" or something similar, you also
> need to trust all the executables, too.  Or switch to something else that
> doesn't trip cygwin's trigger.
>
> After doing that and a rebaseall, I haven't had a fork error in a week.  I
> can't wait to run setup and come up with an update process, though...
>
> Cheers!
>
>
>
> --
> Problem reports:       http://cygwin.com/problems.html
> FAQ:                   http://cygwin.com/faq/
> Documentation:         http://cygwin.com/docs.html
> Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
>

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Challenge: a VERY strange problem with command substitution in bash

[Jürgen Wagner]

[-- Attachment #1: Type: text/plain, Size: 4953 bytes --]

Hello,
  I apologize for not responding earlier - but a project kept me overly
busy, so extra cycles to reply here were not available.

The winner is: Richard Beels. Yes, you are right. Comodo was the source
of my problems.

What's funny is that when I migrated to Win10, I ran into this before
and had it fixed using some magic settings in Comodo. Magic because the
way the setting needed to be made to really work was against logic.

Now, just prior to my posting, the problem re-occurred, so I naturally
did not think of Comodo again causing this.
As it turns out, the culprit was a Windows Update that somehow undid the
settings in Comodo that I had carefully made before.
In the meantime, I had another run of Windows Update that influenced the
function of the Comodo firewall - resulting in rendering it partially
ineffective until reboot.

So, thanks for the quick feedback to all... yes, I managed to fix it again.

And for the record:
- Comodo Antivirus has not caused any problems.
- Comodo Auto-containment is fine, you may just want to exclude Cygwin
from the applications that may be contained. Not essential, though.
- Comodo HIPS I have on paranoid mode, anyway, just don't suppress any
pop-ups.
- Comodo VirusScope should only affect contained applications.
- Comodo Website Filterings is not really relevant, but I have it OFF,
anyway.
- Comodo Firewall has not caused any problems.
So, that's it? No. in the settings of Comodo, there is another section
"Advanced Protection" with the ominous title "Miscellanous".
- Heuristics: can cause issues with some scripts, I have it turned off.
- Detection of shellcode injections breaks some Cygwin mechanisms, turn
it off.
So, that's it? No. The shellcode injections option actually has a list
of exclusions.
If you simply turn the feature off, it won't help making Cygwin work.
You have to explicitly enable it, add all applications to the exclusion
list, then disable it.
Et voilá! Cygwin works.

Cheers,
--Jürgen


On 12.07.2017 01:22, Richard Beels via cygwin wrote:
> At 07/11/2017 at 15:12, Shakespearean monkeys danced on Jürgen
> Wagner's keyboard and said:
>> ...
>> Using backquotes instead of the command substitution with $(...) does
>> not change the results. I could swear this did work in an earlier
>> version of Cygwin on my Windows 7 machine.
>>
>> I tried this to see if the code in the parentheses is executed at all:
>>
>> $ value="$( date 2> foo | cat )"; echo "$? <$value>"
>>
>> The file "foo" was not created, i.e., it seems the commands don't
>> really get executed.
>>
>> $ value="$( date && pwd )"; echo "$? <$value>"
>> 0 <Tue Jul 11 20:49:09 CEST 2017
>> /home/juergen>
>>
>> $ value="$( date || pwd )"; echo "$? <$value>"
>> 0 <Tue Jul 11 20:32:27 CEST 2017>
>>
>> both work, so some control structures seem to be permissible... just
>> not a pipe.
>>
>> What is going on? Some misconfiguration? A Cygwin bug? Some
>> interaction with something weird in Windows 10? I am at loss to
>> understand what could be wrong... and am now most curious whether
>> anybody has an idea what is causing this. Does it work/not work in
>> the same way in your Cygwin installation?
>>
>> I came across this effect because ssh-host-config did not recognize
>> me as administrator anymore. It's due to a check for a certain user
>> group that uses a command substitution with a pipe. Replacing this
>> with an equivalent command works. The original line used "id -G" and
>> then a "grep -Eq" to check whether a certain group is on that list.
>>
>> I am VERY curious now! I've rarely been puzzled that much by such a
>> very elementary shell expression (looking back at 34 years of Unix
>> experience).
>
>
> Hi Jurgen.
>
> 90% chance it's what is called bloda in these parts.  It's in the FAQ
> on cygwin.com.  I'll go out on a limb and say you might have just
> installed/changed your AV/Firewall software.
>
> And if I want to be super-psychic, can I guess comodo?  Because I just
> changed to comodo a couple weeks ago and had the same subshell/command
> substitution/pipeline errors you're mentioning.
>
> If so, you need to exclude your cygwin folder from AV scanning.
> AND...  if the software does whitelisting or host intrusion protection
> (HIPS) or "run unknown executables in a container/sandbox" or
> something similar, you also need to trust all the executables, too. 
> Or switch to something else that doesn't trip cygwin's trigger.
>
> After doing that and a rebaseall, I haven't had a fork error in a
> week.  I can't wait to run setup and come up with an update process,
> though...
>
> Cheers!
>
>
> -- 
> Problem reports:       http://cygwin.com/problems.html
> FAQ:                   http://cygwin.com/faq/
> Documentation:         http://cygwin.com/docs.html
> Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
>
>



[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 3986 bytes --]

Re: Challenge: a VERY strange problem with command substitution in bash

[Brian Inglis]

On 2017-12-04 06:22, David Macek wrote:
> On 4. 12. 2017 14:01, Erik Haukjær Andersen wrote:
>> Hello
>>
>>
>> I have seen a similar problem on Windows 7, using Cygwin bash version
>> 4.4.12(3)-release (x86_64-unknown-cygwin).
>>
>> After repeating a simplified test on my own host and 6 other
>> colleagues, my conclusion is that the cause is found in BeyondTrust
>> SecureDesktop, which you also may have installed (I saw you wrote
>> something about TrustedInstaller).
>>
>>   I have raised an internal case with my IT service desk to resolve the
>> issue (and see what they can do).
> 
> My understanding is that you're just reporing your findings, not asking for
> support here.  Assuming that's the case, the only thing to do here is to add
> this to the BLODA.
> 
> I actually can't find any mention of a product nor feature called SecureDesktop,
> so unless there are better ideas, let's do only "BeyondTrust".

It's a marketing suggestion for referring to the PowerBroker product which
manages locking down desktop and server privileges.

-- 
Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Challenge: a VERY strange problem with command substitution in bash

[Andrey Repin]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=utf-8, Size: 523 bytes --]

Greetings, Erik Haukjær Andersen!

> (I saw you wrote something about TrustedInstaller).

TrustedInstalller is a Windows component.
%SystemRoot%\servicing\TrustedInstaller.exe


-- 
With best regards,
Andrey Repin
Monday, December 4, 2017 23:08:53

Sorry for my terrible english...\0ТÒÐÐ¥\a&ö&ÆVÒ\a&W\x06÷'G3¢\x02\x02\x02\x02\x02\x02\x06‡GG\x03¢òö7–wv–âæ6öÒ÷\a&ö&ÆV×2æ‡FÖÀФd\x15\x13¢\x02\x02\x02\x02\x02\x02\x02\x02\x02\x02\x02\x02\x02\x02\x02\x02\x02\x02\x06‡GG\x03¢òö7–wv–âæ6öÒöf\x17\x12ðФFö7VÖVçF\x17F–öã¢\x02\x02\x02\x02\x02\x02\x02\x02\x06‡GG\x03¢òö7–wv–âæ6öÒöFö72æ‡FÖÀÐ¥Vç7V'67&–&R\x06–æfó¢\x02\x02\x02\x02\x02\x06‡GG\x03¢òö7–wv–âæ6öÒöÖÂò7Vç7V'67&–&R×6–×\x06ÆPРÐ

Re: Challenge: a VERY strange problem with command substitution in bash

[David Macek]

[-- Attachment #1: Type: text/plain, Size: 895 bytes --]

On 4. 12. 2017 14:01, Erik Haukjær Andersen wrote:
> Hello
> 
> 
> I have seen a similar problem on Windows 7, using Cygwin bash version
> 4.4.12(3)-release (x86_64-unknown-cygwin).
> 
> After repeating a simplified test on my own host and 6 other
> colleagues, my conclusion is that the cause is found in BeyondTrust
> SecureDesktop, which you also may have installed (I saw you wrote
> something about TrustedInstaller).
> 
>   I have raised an internal case with my IT service desk to resolve the
> issue (and see what they can do).

My understanding is that you're just reporing your findings, not asking for support here.  Assuming that's the case, the only thing to do here is to add this to the BLODA.

I actually can't find any mention of a product nor feature called SecureDesktop, so unless there are better ideas, let's do only "BeyondTrust".

-- 
David Macek


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 3715 bytes --]

Re: Challenge: a VERY strange problem with command substitution in bash

[Erik Haukjær Andersen]

Hello


I have seen a similar problem on Windows 7, using Cygwin bash version
4.4.12(3)-release (x86_64-unknown-cygwin).

After repeating a simplified test on my own host and 6 other
colleagues, my conclusion is that the cause is found in BeyondTrust
SecureDesktop, which you also may have installed (I saw you wrote
something about TrustedInstaller).

 I have raised an internal case with my IT service desk to resolve the
issue (and see what they can do).

The observations are as follows, in bash, standing in a subdir with 3
files a,b and c:

(The results shown are on a host with SecureDesktop, but the only
difference is at the last command)
$ ls
a  b  c

$ echo $(ls)
a b c

$ echo $(ls|sort)
a b c

$ echo "$(ls)"
a
b
c

$ echo "$(ls|sort)"
$

The problem is seen in the last line, where no output is produced, and
the expected output is just like the previous command (the one without
the ‘sort’).

3 colleagues also with SecureDesktop sees the same as I do, and 3
other colleagues without SecureDesktop sees the expected result (the
last command produces output).


Best regards

Erik Haukjær Andersen
Sr Pncpl Software Developer

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple