My pipe flushes late

My pipe flushes late

[Robert Schmidt]

I basically need to pass some output through d2u, then add a prefix to 
each line.  Most importantly, I need the line to flush through the pipe 
immediately.  However, d2u (or the pipe itself) caches/flushes late.
For example:

$ d2u | sed "s/^/prefix: /"
a
b^D
prefix: a
prefix: b
$

I need to see:

$ d2u | sed "s/^/prefix: /"
a
prefix: a
b^D
prefix: b
$

If I replace d2u with cat above, it flushes each line, but I need d2u.
If I replace d2u with sed s/\x0D//g, the output also flushes late.
I know I can probably write one sed script to replace the above pipe, 
but my prefix is actually not static (should be a time stamp).

How can I accomplish this?

My cygwin is up to date (Aug 20).

Cheers,
Robert

Re: My pipe flushes late

[Eric Blake]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Let's hope your plumbing doesn't overflow.  Although the hippos might like
the smell...

According to Robert Schmidt on 8/20/2008 5:54 AM:
> I basically need to pass some output through d2u, then add a prefix to
> each line.  Most importantly, I need the line to flush through the pipe
> immediately.  However, d2u (or the pipe itself) caches/flushes late.

What you want is to convert d2u's stdout from fully buffered (default, as
per POSIX, for all non-interactive file descriptors) to line-buffered or
unbuffered.  On Linux, this can be done with an LD_PRELOAD of a library
that calls setvbuf prior to letting main() run, but I'm not sure cygwin
quite has those hooks.  The coreutils list has even seen a proposal for
adding an app just for this purpose, although it is yet to materialize:
http://lists.gnu.org/archive/html/bug-coreutils/2006-03/msg00123.html

At this point, your best bet is to use something that presents d2u with a
pty as stdout (thus output would be line-buffered), rather than a pipe,
then consumes the pty output and throws it over the wall to the original
pipe to sed with flushing.  Have you tried expect - according to the link
above, expect provides a script unbuffer that does just what I described?

- --
Don't work too hard, make some time for fun as well!

Eric Blake             ebb9@byu.net
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Public key at home.comcast.net/~ericblake/eblake.gpg
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkisCNEACgkQ84KuGfSFAYBp5ACdEDjEB+ICCGsA9gkqiBInUkQ5
7R0AnjM5LDtEsQO1s7i3ConCUXkGNaUW
=NZf2
-----END PGP SIGNATURE-----

Re: My pipe flushes late

[Robert Schmidt]

Eric Blake wrote:
> At this point, your best bet is to use something that presents d2u with a
> pty as stdout (thus output would be line-buffered), rather than a pipe,
> then consumes the pty output and throws it over the wall to the original
> pipe to sed with flushing.  Have you tried expect - according to the link
> above, expect provides a script unbuffer that does just what I described?

Thanks for the informative answer - it confirmed my suspicions.  I 
wasn't able to get unbuffer working, but I may not need to - see my 
other followup.

Robert

Re: My pipe flushes late

[Brian Dessent]

Robert Schmidt wrote:

> How can I accomplish this?

Drop a hippo on it?

Seriously though, this is totally the wrong list to ask a serious
question.

Can't you just:

sed -e 's/\x0D//' -e 's/^/prefix: /'

The fact that there's only the one process eliminates the line buffering
in the pipe.  Well, presumably there is still a pipe in that you are
piping some unspecified output into this, but presumably also that
output flushes its buffer after each line so that's fine.

Brian

Re: My pipe flushes late

[Brian Dessent]

Robert Schmidt wrote:

> I know I can probably write one sed script to replace the above pipe,
> but my prefix is actually not static (should be a time stamp).

Oh, I missed that qualifier.  So, what you really mean is that sed is a
total red herring as you aren't actually using sed but something else
entirely in the real application?  Anyway, if you absolutely must have a
d2u that is line buffered instead of full buffered then you can use

perl -pe 'BEGIN { $| = 1; } s,\r$,,;' | whatever_the_actual_thing_is

Brian

Re: My pipe flushes late

[Robert Schmidt]

Brian Dessent wrote:
> Robert Schmidt wrote:
> Oh, I missed that qualifier.  So, what you really mean is that sed is a
> total red herring as you aren't actually using sed but something else
> entirely in the real application?  Anyway, if you absolutely must have a
> d2u that is line buffered instead of full buffered then you can use
> 
> perl -pe 'BEGIN { $| = 1; } s,\r$,,;' | whatever_the_actual_thing_is

Thanks!  In the meantime, I've discovered that the reason for d2u (the 
read builtin freezing on single 0x0D characters) seems to be gone in 
cygwin, so I can simplify my script.

For completeness, my real script is:

#!/bin/bash
# read stalls on 0x0d in input, d2u fixes that, but introduces buffering
d2u | while true
do
	read -r || exit
	echo `date +"%F %T"` "$REPLY"
done

... where d2u can probably be removed now.
It's used to timestamp output from various services, some of which 
output DOS line endings.  E.g. my-service.sh | prefix-time.sh.

If there was a way to execute a command (date, in my case) from within 
the sed replacement section, I'd be home free with a single sed process 
instead of a chunky bash process lurking about.


Robert

Re: My pipe flushes late

[Brian Dessent]

Robert Schmidt wrote:

> ... where d2u can probably be removed now.
> It's used to timestamp output from various services, some of which
> output DOS line endings.  E.g. my-service.sh | prefix-time.sh.
> 
> If there was a way to execute a command (date, in my case) from within
> the sed replacement section, I'd be home free with a single sed process
> instead of a chunky bash process lurking about.

my-service.sh | perl -MPOSIX -ne 'print strftime("%F %T ", localtime), $_'

Re: My pipe flushes late

[Christopher Faylor]

On Wed, Aug 20, 2008 at 03:14:35PM +0200, Robert Schmidt wrote:
> Thanks!  In the meantime, I've discovered that the reason for d2u (the read 
> builtin freezing on single 0x0D characters) seems to be gone in cygwin, so 
> I can simplify my script.

http://cygwin.com/lists.html#cygwin-talk

    cygwin-talk: a mindless chatter list for discussing things that are
    tangentially related to Cygwin.  This list is intended to be funny but
    not terribly helpful.  Do not send technical questions about Cygwin
    here.  Use the main Cygwin mailing list for serious Cygwin questions
    since responses to technical questions in this list are not guaranteed
    to be serious or definitive.  No profanity, no commercial posts, and
    minimal flaming, please.

Please use the main cygwin list for technical questions.

cgf