Pipe behavior clarification?

Pipe behavior clarification?

[David Balažic]

Hi!

Is this a correct pipe behavior?

$ echo booo | tee >(md5sum --tag) >/dev/null
MD5 (-) = 9c8b79bdf79ef0ee73a77b8d36d27a2d

$ echo booo | tee >(md5sum --tag) | cat >/dev/null

It surprised me, that the output of md5sum is dealed the same way as
the std output of tee in the second case, but separately in the first
case.

Using cygwin 64 bit (up to date), with bash.

Regards,
David

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Pipe behavior clarification?

[Eliot Moss]

On 1/22/2017 3:19 PM, David Balažic wrote:
> Hi!
>
> Is this a correct pipe behavior?
>
> $ echo booo | tee >(md5sum --tag) >/dev/null
> MD5 (-) = 9c8b79bdf79ef0ee73a77b8d36d27a2d
>
> $ echo booo | tee >(md5sum --tag) | cat >/dev/null

Here's what I think happens, even if it is a bit counter-intuitive:

 >(...) creates a subprocess, whose input comes from some kind
of pipe or socket, and tee is presented with a filename it can
use to write to that socket.

The *output* of the >(...) subprocess is hooked up to what is
known to be the output of tee *at the time the subprocess is
created*.  This happens *before* any > redirections are done.

However, in the case of the | pipe, that plumbing is set up
*before* the >(...) construct is acted on.

Note that you could do >(md5sum --tag >whatever) if you want
to specifically control the output of md5sum.

I am sure someone more knowledgeable will correct me if I've
missed anything important here :-) ...

Regards - Eliot Moss

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Pipe behavior clarification?

[Eric Blake]

[-- Attachment #1.1: Type: text/plain, Size: 1693 bytes --]

On 01/22/2017 04:23 PM, Eliot Moss wrote:
> On 1/22/2017 3:19 PM, David Balažic wrote:
>> Hi!
>>
>> Is this a correct pipe behavior?
>>
>> $ echo booo | tee >(md5sum --tag) >/dev/null
>> MD5 (-) = 9c8b79bdf79ef0ee73a77b8d36d27a2d
>>
>> $ echo booo | tee >(md5sum --tag) | cat >/dev/null
> 
> Here's what I think happens, even if it is a bit counter-intuitive:
> 
>>(...) creates a subprocess, whose input comes from some kind
> of pipe or socket, and tee is presented with a filename it can
> use to write to that socket.
> 
> The *output* of the >(...) subprocess is hooked up to what is
> known to be the output of tee *at the time the subprocess is
> created*.  This happens *before* any > redirections are done.

Rather, all >() and > redirections are performed in left-to-right order.
 But you are correct that the second >/dev/null is overwriting the
stdout that was originally given by >(md5sum), and therefore tee is NOT
writing to the md5sum process.

> 
> However, in the case of the | pipe, that plumbing is set up
> *before* the >(...) construct is acted on.

Also correct. Mixing >() and | is usually not what you want, as you are
no longer writing to the pipeline.

> 
> Note that you could do >(md5sum --tag >whatever) if you want
> to specifically control the output of md5sum.
> 
> I am sure someone more knowledgeable will correct me if I've
> missed anything important here :-) ...

You got the gist of it.  Order matters, and specifying more than one
stdout (by any mix of >, >(), or |) is generally not what you want.

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: Pipe behavior clarification?

[Eliot Moss]

On 1/23/2017 9:32 AM, Eric Blake wrote:
> On 01/22/2017 04:23 PM, Eliot Moss wrote:
>> On 1/22/2017 3:19 PM, David Balažic wrote:
>>> Hi!
>>>
>>> Is this a correct pipe behavior?
>>>
>>> $ echo booo | tee >(md5sum --tag) >/dev/null
>>> MD5 (-) = 9c8b79bdf79ef0ee73a77b8d36d27a2d
>>>
>>> $ echo booo | tee >(md5sum --tag) | cat >/dev/null
>>
>> Here's what I think happens, even if it is a bit counter-intuitive:
>>
>>> (...) creates a subprocess, whose input comes from some kind
>> of pipe or socket, and tee is presented with a filename it can
>> use to write to that socket.
>>
>> The *output* of the >(...) subprocess is hooked up to what is
>> known to be the output of tee *at the time the subprocess is
>> created*.  This happens *before* any > redirections are done.
>
> Rather, all >() and > redirections are performed in left-to-right order.
>  But you are correct that the second >/dev/null is overwriting the
> stdout that was originally given by >(md5sum), and therefore tee is NOT
> writing to the md5sum process.
>
>>
>> However, in the case of the | pipe, that plumbing is set up
>> *before* the >(...) construct is acted on.
>
> Also correct. Mixing >() and | is usually not what you want, as you are
> no longer writing to the pipeline.
>
>>
>> Note that you could do >(md5sum --tag >whatever) if you want
>> to specifically control the output of md5sum.
>>
>> I am sure someone more knowledgeable will correct me if I've
>> missed anything important here :-) ...
>
> You got the gist of it.  Order matters, and specifying more than one
> stdout (by any mix of >, >(), or |) is generally not what you want.

Dear Eric (et al.) -- I *mostly* agree with this, with the exception
that >(...) is *not* an output redirection.  It will present tee with
the name of a file -- either a named pipe or a /dev/fdnnn file name.
By experimentation I discovered that this particular replacement
is not done in the same left-to-right pass as I/O redirections ...

Regards - EM

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Pipe behavior clarification?

[David Balažic]

On 23 January 2017 at 15:32, Eric Blake <eblake@redhat.com> wrote:
> On 01/22/2017 04:23 PM, Eliot Moss wrote:
>> On 1/22/2017 3:19 PM, David Balažic wrote:
>>> Hi!
>>>
>>> Is this a correct pipe behavior?
>>>
>>> $ echo booo | tee >(md5sum --tag) >/dev/null
>>> MD5 (-) = 9c8b79bdf79ef0ee73a77b8d36d27a2d
>>>
>>> $ echo booo | tee >(md5sum --tag) | cat >/dev/null
>>
>> Here's what I think happens, even if it is a bit counter-intuitive:
>>
>>>(...) creates a subprocess, whose input comes from some kind
>> of pipe or socket, and tee is presented with a filename it can
>> use to write to that socket.
>>
>> The *output* of the >(...) subprocess is hooked up to what is
>> known to be the output of tee *at the time the subprocess is
>> created*.  This happens *before* any > redirections are done.
>
> Rather, all >() and > redirections are performed in left-to-right order.
>  But you are correct that the second >/dev/null is overwriting the
> stdout that was originally given by >(md5sum), and therefore tee is NOT
> writing to the md5sum process.

The last part is incorrect. tee is always writing to the md5sum process.
It can be verified by redirecting the md5sum output to a file, like
>(md5sum > file1),
or redirecting the final output to a file instead of /dev/null. In
both cases the (correct)
md5 hash will be there.

Regards,
David

PS: I verified on SLES 11 that the behavior is the same as in cygwin,
so the thread is "technically closed".

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple