On 25/02/2021 07:50, Brian Inglis wrote:
> On 2021-02-24 16:50, Brian Inglis wrote:
>> On 2021-02-24 15:41, Duncan Roe wrote:
>>> On Wed, Feb 24, 2021 at 04:58:24PM -0500, Eliot Moss wrote:
>>>> On 2/24/2021 3:48 PM, ASSI wrote:
>>>>> Hamish McIntyre-Bhatty via Cygwin writes:
>>>>>> I found recently when trying to save output from a script for later
>>>>>> inspection that "tee" and file redirections seem to have massive
>>>>>> delays when run in Cygwin - usually nothing is written to file or
>>>>>> stdout until after the command has finished - not very helpful.
>>>>>
>>>>> You will want to switch from fully buffered to line-buffered or even
>>>>> unbuffered output.
>>
>>>> And this does not have to do with Cygwin.  The same happens on Linux.
>>>> The default is that terminal I/O is unbuffered while other stream are
>>>> buffered.  Pipes come under "other streams".  One can make 
>>>> programmatic
>>>> changes to get around this, but most programs won't override the
>>>> default behavior on their own ...
>>
>>> The (Linux) default is that terminal I/O is *line* buffered
>>>
>>> The man page for tee doesn't show an option to change buffering, 
>>> while that for
>>> grep does.
>>
>> I believe the default for both Cygwin and Linux is 64KB pipe buffer, 
>> so if you want to see smaller chunks as they are generated, you need 
>> to add some utility that may allow you to change that e.g.
>>
>>      $ tail -f access.log | stdbuf -oL cut -d ' ' -f1 | uniq
>>
>> but read the disclaimers on the stdbuf and grep man pages, which is 
>> why it is not done more, especially under Cygwin where Windows adds 
>> its own performance penalties.
>> Some utilities may use read(2/3p), write(2/3p), or mmap(3) if they 
>> can and don't care about text or lines, for more efficient access to 
>> disk files, rather than buffered stream I/O functions.
>
> From what I have been able to find, Cygwin <stdio.h> BUFSIZ is only 
> 1K, compared to Linux 8K, and Cygwin internal 64K, and that is used in 
> many places in coreutils like tee, which will slow everything down by 
> a factor of at least 8 plus increased overhead.
>
> Suggest <stdio.h> BUFSIZ be bumped to at least Linux value of 8K, if 
> not 64K.

So from the discussion above, I've not sure I fully understand why the 
behaviour is different on Cygwin to Linux for me, especially if Linux's 
buffer size is sometimes bigger.

Perhaps:

#1: It depends on the program being run and how it sets its buffers up?

#2: stdbuf -o L enables line buffering and should fix my problem?

I'm now thinking it may have been a Python script that was behaving this 
way - I might go and double check in case this isn't what I think it is.

Hamish