* My pipe flushes late
@ 2008-08-20 11:55 Robert Schmidt
2008-08-20 12:07 ` Eric Blake
` (2 more replies)
0 siblings, 3 replies; 8+ messages in thread
From: Robert Schmidt @ 2008-08-20 11:55 UTC (permalink / raw)
To: cygwin-talk
I basically need to pass some output through d2u, then add a prefix to
each line. Most importantly, I need the line to flush through the pipe
immediately. However, d2u (or the pipe itself) caches/flushes late.
For example:
$ d2u | sed "s/^/prefix: /"
a
b^D
prefix: a
prefix: b
$
I need to see:
$ d2u | sed "s/^/prefix: /"
a
prefix: a
b^D
prefix: b
$
If I replace d2u with cat above, it flushes each line, but I need d2u.
If I replace d2u with sed s/\x0D//g, the output also flushes late.
I know I can probably write one sed script to replace the above pipe,
but my prefix is actually not static (should be a time stamp).
How can I accomplish this?
My cygwin is up to date (Aug 20).
Cheers,
Robert
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: My pipe flushes late
2008-08-20 11:55 My pipe flushes late Robert Schmidt
@ 2008-08-20 12:07 ` Eric Blake
2008-08-20 13:20 ` Robert Schmidt
2008-08-20 12:16 ` Brian Dessent
2008-08-20 12:40 ` Brian Dessent
2 siblings, 1 reply; 8+ messages in thread
From: Eric Blake @ 2008-08-20 12:07 UTC (permalink / raw)
To: The Vulgar and Unprofessional Cygwin-Talk List
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Let's hope your plumbing doesn't overflow. Although the hippos might like
the smell...
According to Robert Schmidt on 8/20/2008 5:54 AM:
> I basically need to pass some output through d2u, then add a prefix to
> each line. Most importantly, I need the line to flush through the pipe
> immediately. However, d2u (or the pipe itself) caches/flushes late.
What you want is to convert d2u's stdout from fully buffered (default, as
per POSIX, for all non-interactive file descriptors) to line-buffered or
unbuffered. On Linux, this can be done with an LD_PRELOAD of a library
that calls setvbuf prior to letting main() run, but I'm not sure cygwin
quite has those hooks. The coreutils list has even seen a proposal for
adding an app just for this purpose, although it is yet to materialize:
http://lists.gnu.org/archive/html/bug-coreutils/2006-03/msg00123.html
At this point, your best bet is to use something that presents d2u with a
pty as stdout (thus output would be line-buffered), rather than a pipe,
then consumes the pty output and throws it over the wall to the original
pipe to sed with flushing. Have you tried expect - according to the link
above, expect provides a script unbuffer that does just what I described?
- --
Don't work too hard, make some time for fun as well!
Eric Blake ebb9@byu.net
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Public key at home.comcast.net/~ericblake/eblake.gpg
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iEYEARECAAYFAkisCNEACgkQ84KuGfSFAYBp5ACdEDjEB+ICCGsA9gkqiBInUkQ5
7R0AnjM5LDtEsQO1s7i3ConCUXkGNaUW
=NZf2
-----END PGP SIGNATURE-----
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: My pipe flushes late
2008-08-20 12:07 ` Eric Blake
@ 2008-08-20 13:20 ` Robert Schmidt
0 siblings, 0 replies; 8+ messages in thread
From: Robert Schmidt @ 2008-08-20 13:20 UTC (permalink / raw)
To: cygwin-talk
Eric Blake wrote:
> At this point, your best bet is to use something that presents d2u with a
> pty as stdout (thus output would be line-buffered), rather than a pipe,
> then consumes the pty output and throws it over the wall to the original
> pipe to sed with flushing. Have you tried expect - according to the link
> above, expect provides a script unbuffer that does just what I described?
Thanks for the informative answer - it confirmed my suspicions. I
wasn't able to get unbuffer working, but I may not need to - see my
other followup.
Robert
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: My pipe flushes late
2008-08-20 11:55 My pipe flushes late Robert Schmidt
2008-08-20 12:07 ` Eric Blake
@ 2008-08-20 12:16 ` Brian Dessent
2008-08-20 12:40 ` Brian Dessent
2 siblings, 0 replies; 8+ messages in thread
From: Brian Dessent @ 2008-08-20 12:16 UTC (permalink / raw)
To: The Vulgar and Unprofessional Cygwin-Talk List
Robert Schmidt wrote:
> How can I accomplish this?
Drop a hippo on it?
Seriously though, this is totally the wrong list to ask a serious
question.
Can't you just:
sed -e 's/\x0D//' -e 's/^/prefix: /'
The fact that there's only the one process eliminates the line buffering
in the pipe. Well, presumably there is still a pipe in that you are
piping some unspecified output into this, but presumably also that
output flushes its buffer after each line so that's fine.
Brian
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: My pipe flushes late
2008-08-20 11:55 My pipe flushes late Robert Schmidt
2008-08-20 12:07 ` Eric Blake
2008-08-20 12:16 ` Brian Dessent
@ 2008-08-20 12:40 ` Brian Dessent
2008-08-20 13:15 ` Robert Schmidt
2 siblings, 1 reply; 8+ messages in thread
From: Brian Dessent @ 2008-08-20 12:40 UTC (permalink / raw)
To: The Vulgar and Unprofessional Cygwin-Talk List
Robert Schmidt wrote:
> I know I can probably write one sed script to replace the above pipe,
> but my prefix is actually not static (should be a time stamp).
Oh, I missed that qualifier. So, what you really mean is that sed is a
total red herring as you aren't actually using sed but something else
entirely in the real application? Anyway, if you absolutely must have a
d2u that is line buffered instead of full buffered then you can use
perl -pe 'BEGIN { $| = 1; } s,\r$,,;' | whatever_the_actual_thing_is
Brian
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: My pipe flushes late
2008-08-20 12:40 ` Brian Dessent
@ 2008-08-20 13:15 ` Robert Schmidt
2008-08-20 13:31 ` Brian Dessent
2008-08-20 16:12 ` Christopher Faylor
0 siblings, 2 replies; 8+ messages in thread
From: Robert Schmidt @ 2008-08-20 13:15 UTC (permalink / raw)
To: cygwin-talk
Brian Dessent wrote:
> Robert Schmidt wrote:
> Oh, I missed that qualifier. So, what you really mean is that sed is a
> total red herring as you aren't actually using sed but something else
> entirely in the real application? Anyway, if you absolutely must have a
> d2u that is line buffered instead of full buffered then you can use
>
> perl -pe 'BEGIN { $| = 1; } s,\r$,,;' | whatever_the_actual_thing_is
Thanks! In the meantime, I've discovered that the reason for d2u (the
read builtin freezing on single 0x0D characters) seems to be gone in
cygwin, so I can simplify my script.
For completeness, my real script is:
#!/bin/bash
# read stalls on 0x0d in input, d2u fixes that, but introduces buffering
d2u | while true
do
read -r || exit
echo `date +"%F %T"` "$REPLY"
done
... where d2u can probably be removed now.
It's used to timestamp output from various services, some of which
output DOS line endings. E.g. my-service.sh | prefix-time.sh.
If there was a way to execute a command (date, in my case) from within
the sed replacement section, I'd be home free with a single sed process
instead of a chunky bash process lurking about.
Robert
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: My pipe flushes late
2008-08-20 13:15 ` Robert Schmidt
@ 2008-08-20 13:31 ` Brian Dessent
2008-08-20 16:12 ` Christopher Faylor
1 sibling, 0 replies; 8+ messages in thread
From: Brian Dessent @ 2008-08-20 13:31 UTC (permalink / raw)
To: The Vulgar and Unprofessional Cygwin-Talk List
Robert Schmidt wrote:
> ... where d2u can probably be removed now.
> It's used to timestamp output from various services, some of which
> output DOS line endings. E.g. my-service.sh | prefix-time.sh.
>
> If there was a way to execute a command (date, in my case) from within
> the sed replacement section, I'd be home free with a single sed process
> instead of a chunky bash process lurking about.
my-service.sh | perl -MPOSIX -ne 'print strftime("%F %T ", localtime), $_'
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: My pipe flushes late
2008-08-20 13:15 ` Robert Schmidt
2008-08-20 13:31 ` Brian Dessent
@ 2008-08-20 16:12 ` Christopher Faylor
1 sibling, 0 replies; 8+ messages in thread
From: Christopher Faylor @ 2008-08-20 16:12 UTC (permalink / raw)
To: The Vulgar and Unprofessional Cygwin-Talk List
On Wed, Aug 20, 2008 at 03:14:35PM +0200, Robert Schmidt wrote:
> Thanks! In the meantime, I've discovered that the reason for d2u (the read
> builtin freezing on single 0x0D characters) seems to be gone in cygwin, so
> I can simplify my script.
http://cygwin.com/lists.html#cygwin-talk
cygwin-talk: a mindless chatter list for discussing things that are
tangentially related to Cygwin. This list is intended to be funny but
not terribly helpful. Do not send technical questions about Cygwin
here. Use the main Cygwin mailing list for serious Cygwin questions
since responses to technical questions in this list are not guaranteed
to be serious or definitive. No profanity, no commercial posts, and
minimal flaming, please.
Please use the main cygwin list for technical questions.
cgf
^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2008-08-20 16:12 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2008-08-20 11:55 My pipe flushes late Robert Schmidt
2008-08-20 12:07 ` Eric Blake
2008-08-20 13:20 ` Robert Schmidt
2008-08-20 12:16 ` Brian Dessent
2008-08-20 12:40 ` Brian Dessent
2008-08-20 13:15 ` Robert Schmidt
2008-08-20 13:31 ` Brian Dessent
2008-08-20 16:12 ` Christopher Faylor
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).