Re: Tee and file redirections are very slow to write anything.

[Brian Inglis <Brian.Inglis@SystematicSw.ab.ca>]

On 2021-02-24 16:50, Brian Inglis wrote:
> On 2021-02-24 15:41, Duncan Roe wrote:
>> On Wed, Feb 24, 2021 at 04:58:24PM -0500, Eliot Moss wrote:
>>> On 2/24/2021 3:48 PM, ASSI wrote:
>>>> Hamish McIntyre-Bhatty via Cygwin writes:
>>>>> I found recently when trying to save output from a script for later
>>>>> inspection that "tee" and file redirections seem to have massive
>>>>> delays when run in Cygwin - usually nothing is written to file or
>>>>> stdout until after the command has finished - not very helpful.
>>>>
>>>> You will want to switch from fully buffered to line-buffered or even
>>>> unbuffered output.
> 
>>> And this does not have to do with Cygwin.  The same happens on Linux.
>>> The default is that terminal I/O is unbuffered while other stream are
>>> buffered.  Pipes come under "other streams".  One can make programmatic
>>> changes to get around this, but most programs won't override the
>>> default behavior on their own ...
> 
>> The (Linux) default is that terminal I/O is *line* buffered
>>
>> The man page for tee doesn't show an option to change buffering, while that for
>> grep does.
> 
> I believe the default for both Cygwin and Linux is 64KB pipe buffer, so if you 
> want to see smaller chunks as they are generated, you need to add some utility 
> that may allow you to change that e.g.
> 
>      $ tail -f access.log | stdbuf -oL cut -d ' ' -f1 | uniq
> 
> but read the disclaimers on the stdbuf and grep man pages, which is why it is 
> not done more, especially under Cygwin where Windows adds its own performance 
> penalties.
> Some utilities may use read(2/3p), write(2/3p), or mmap(3) if they can and don't 
> care about text or lines, for more efficient access to disk files, rather than 
> buffered stream I/O functions.

 From what I have been able to find, Cygwin <stdio.h> BUFSIZ is only 1K, 
compared to Linux 8K, and Cygwin internal 64K, and that is used in many places 
in coreutils like tee, which will slow everything down by a factor of at least 8 
plus increased overhead.

Suggest <stdio.h> BUFSIZ be bumped to at least Linux value of 8K, if not 64K.

-- 
Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada

This email may be disturbing to some readers as it contains
too much technical detail. Reader discretion is advised.
[Data in binary units and prefixes, physical quantities in SI.]