From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from smtp-out-no.shaw.ca (smtp-out-no.shaw.ca [64.59.134.9]) by sourceware.org (Postfix) with ESMTPS id AD9353857C61 for ; Thu, 25 Feb 2021 07:50:55 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.3.2 sourceware.org AD9353857C61 Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=SystematicSw.ab.ca Authentication-Results: sourceware.org; spf=none smtp.mailfrom=brian.inglis@systematicsw.ab.ca Received: from [192.168.1.104] ([68.147.0.90]) by shaw.ca with ESMTP id FBQDlHQOieHr9FBQElKJQD; Thu, 25 Feb 2021 00:50:54 -0700 X-Authority-Analysis: v=2.4 cv=Yq/K+6UX c=1 sm=1 tr=0 ts=603756de a=T+ovY1NZ+FAi/xYICV7Bgg==:117 a=T+ovY1NZ+FAi/xYICV7Bgg==:17 a=IkcTkHD0fZMA:10 a=sBsfJAgXvrhmUhQbWtwA:9 a=QEXdDO2ut3YA:10 Reply-To: cygwin@cygwin.com To: cygwin@cygwin.com References: <87eeh5b39n.fsf@Otto.invalid> <49e01de1-f7de-5d60-2996-3c9afb834396@cs.umass.edu> <20210224224149.GM2979@dimstar.local.net> <7bdae030-12c7-b2fa-9386-5e279382637d@SystematicSw.ab.ca> From: Brian Inglis Organization: Systematic Software Subject: Re: Tee and file redirections are very slow to write anything. Message-ID: Date: Thu, 25 Feb 2021 00:50:53 -0700 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.7.1 MIME-Version: 1.0 In-Reply-To: <7bdae030-12c7-b2fa-9386-5e279382637d@SystematicSw.ab.ca> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-CA Content-Transfer-Encoding: 8bit X-CMAE-Envelope: MS4xfBy8BTJkZutZC4lmu0HKhButy2KqmyUfAremrJ+L12AN6ogPcJgsBeNLFGSbtdKVtbM7iSutyFoc2c9q+HGtcKaQG7HsnVAT57adZf6dmGcVL0g+iTI5 NeitAfCC1GqJbnj/5p3C9HTrGNroNUkiTBIQB4yBrrIuTTnwdRZebfpgxi0dkkpi4pQU12Q2Szku3Y3Dq7gi0lGzYXWYNbIoU70= X-Spam-Status: No, score=1.9 required=5.0 tests=BAYES_00, BODY_8BITS, KAM_DMARC_STATUS, KAM_LAZY_DOMAIN_SECURITY, NICE_REPLY_A, RCVD_IN_BARRACUDACENTRAL, RCVD_IN_DNSWL_LOW, RCVD_IN_MSPIKE_H3, RCVD_IN_MSPIKE_WL, SPF_HELO_NONE, SPF_NONE, TXREP autolearn=no autolearn_force=no version=3.4.2 X-Spam-Level: * X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on server2.sourceware.org X-BeenThere: cygwin@cygwin.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: General Cygwin discussions and problem reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 25 Feb 2021 07:50:57 -0000 On 2021-02-24 16:50, Brian Inglis wrote: > On 2021-02-24 15:41, Duncan Roe wrote: >> On Wed, Feb 24, 2021 at 04:58:24PM -0500, Eliot Moss wrote: >>> On 2/24/2021 3:48 PM, ASSI wrote: >>>> Hamish McIntyre-Bhatty via Cygwin writes: >>>>> I found recently when trying to save output from a script for later >>>>> inspection that "tee" and file redirections seem to have massive >>>>> delays when run in Cygwin - usually nothing is written to file or >>>>> stdout until after the command has finished - not very helpful. >>>> >>>> You will want to switch from fully buffered to line-buffered or even >>>> unbuffered output. > >>> And this does not have to do with Cygwin.  The same happens on Linux. >>> The default is that terminal I/O is unbuffered while other stream are >>> buffered.  Pipes come under "other streams".  One can make programmatic >>> changes to get around this, but most programs won't override the >>> default behavior on their own ... > >> The (Linux) default is that terminal I/O is *line* buffered >> >> The man page for tee doesn't show an option to change buffering, while that for >> grep does. > > I believe the default for both Cygwin and Linux is 64KB pipe buffer, so if you > want to see smaller chunks as they are generated, you need to add some utility > that may allow you to change that e.g. > >     $ tail -f access.log | stdbuf -oL cut -d ' ' -f1 | uniq > > but read the disclaimers on the stdbuf and grep man pages, which is why it is > not done more, especially under Cygwin where Windows adds its own performance > penalties. > Some utilities may use read(2/3p), write(2/3p), or mmap(3) if they can and don't > care about text or lines, for more efficient access to disk files, rather than > buffered stream I/O functions. From what I have been able to find, Cygwin BUFSIZ is only 1K, compared to Linux 8K, and Cygwin internal 64K, and that is used in many places in coreutils like tee, which will slow everything down by a factor of at least 8 plus increased overhead. Suggest BUFSIZ be bumped to at least Linux value of 8K, if not 64K. -- Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada This email may be disturbing to some readers as it contains too much technical detail. Reader discretion is advised. [Data in binary units and prefixes, physical quantities in SI.]