From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mout.kundenserver.de (mout.kundenserver.de [212.227.17.13]) by sourceware.org (Postfix) with ESMTPS id 150593858D39 for ; Mon, 15 Nov 2021 16:08:04 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org 150593858D39 Authentication-Results: sourceware.org; dmarc=fail (p=none dis=none) header.from=cygwin.com Authentication-Results: sourceware.org; spf=fail smtp.mailfrom=cygwin.com Received: from calimero.vinschen.de ([24.134.7.25]) by mrelayeu.kundenserver.de (mreue108 [212.227.15.183]) with ESMTPSA (Nemesis) id 1MRmo8-1nEoY51dWM-00TG4t for ; Mon, 15 Nov 2021 17:08:03 +0100 Received: by calimero.vinschen.de (Postfix, from userid 500) id 53E0FA80D6A; Mon, 15 Nov 2021 17:08:02 +0100 (CET) Date: Mon, 15 Nov 2021 17:08:02 +0100 From: Corinna Vinschen To: cygwin-developers@cygwin.com Subject: Re: cygwin 3.3.x: another problem that may be related to pipes Message-ID: Reply-To: cygwin-developers@cygwin.com Mail-Followup-To: cygwin-developers@cygwin.com References: <115203324.649908.1636923059546.ref@mail.yahoo.com> <115203324.649908.1636923059546@mail.yahoo.com> <20211115171811.844dce9cce2b4d13262d64f2@nifty.ne.jp> <20211115235021.0f0f64b1b0e2a7bd6d16be80@nifty.ne.jp> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <20211115235021.0f0f64b1b0e2a7bd6d16be80@nifty.ne.jp> X-Provags-ID: V03:K1:bmmk4HuS4GTvQZU1LIrFSMpwJQ5AUjeUJhCDTjS1ObTUScoeQFh zFKFUbjmFYudP734Y2b304yBguQlC14DWFkMzIq6LibtcVoUsShwcopk+D1HwvhwErznqhd 5QXvciR3VmzfvU0+pOTwBRgBzYGPBA1s8MQ857A8+DLQBg8x5LoE4MyvrHWKfBNJjGx3WPR W2HNxmPfBIV2Xxy96Teiw== X-UI-Out-Filterresults: notjunk:1;V03:K0:SjgkH5V+yqo=:FJjRhfAhoT8Ml+/8S7tXJR cUcf7M9TZGPRCendKGY6XkUArqSbNadiLbTKd8/7YJC+d5Zc4le/UbY7SRXVTcJU3A6dyQ/tu FEH+lkTn9tTs12Juso0YDXpiciZCq1ygdkKc7+DJESFkvjUndvEw9eu0BWPSNhGUAHG+lHW8R Dx9n8mtrwEupnYYwsMedwncsCgJQGN/oJCtBF63P6SXBmpteEd/1NET9G7jO3X4+b1Wfydr1i ljM+MEknCedoYWjdse/qjWfyJABGJppb65+6oGEm/T6sg3+cfxL5sUK2WF4tE8E+gtiNaoaLE iq59N+fD6K9rM0aMRZGY9mzqKxO5d91Gs3F7qm6Lorp4dBJgKN6TGWy/Qd3WyTAuJ5HWx5vTr UR3tM8/iOpspQY1HFywextW/mO1XV5IaEQXntm+5jzk0ZEfIGpTOjNxtWUnmIbNDoz3iVC5gO Y2qWU1A/4bfMoRAWlJY/2sKjWj59wgaPRQmI7wzr99MLj+Vxk/FxiUbNEvzuTOVm/YoWRzURO Igv3NzyanAmisvfLR3hPy/ZhlG87Mhk9Fgbj9SlXlbW22fUCgeAHOy7cQ36BoCm/S9tHK1ton kXP+2cwWLfY49EMsu5VPdZgaVfl5ivChDsisnQb5VWNXNz/Dst6JLRskF1spa2lO0eyl7CJFw N5Ht76A+qkeBlrCnUUtcG1e4urkuvbda9PSp0WGMkcyaFTaID63gPIL5BiPBczPO6yvAaKBND GkAcGphibfsFd27d X-Spam-Status: No, score=-99.3 required=5.0 tests=BAYES_00, GOOD_FROM_CORINNA_CYGWIN, KAM_DMARC_NONE, KAM_DMARC_STATUS, RCVD_IN_MSPIKE_H2, SPF_HELO_NONE, SPF_NEUTRAL, TXREP autolearn=ham autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on server2.sourceware.org X-BeenThere: cygwin-developers@cygwin.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Cygwin core component developers mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Nov 2021 16:08:06 -0000 On Nov 15 23:50, Takashi Yano wrote: > On Mon, 15 Nov 2021 14:01:12 +0100 > Corinna Vinschen wrote: > > [Redirected to cygwin-developers] > > > > On Nov 15 17:18, Takashi Yano via Cygwin wrote: > > > On Sun, 14 Nov 2021 20:50:59 +0000 (UTC) > > > bf wrote: > > > > I've a shell script that processes several hundred xz-compressed text files with a pipeline of the form: > > > > > > > > find . -depth -type f  ... -print0 | \ > > > > xargs -0tI @ xzcat -- @ | \ > > > > iconv --byte-subst='�' --unicode-subst='�' --widechar-subst='�' -f MS-ANSI -t UTF-8 -- | \ > > > > grep ... \ > > > > sed ... | \ > > > > sort ... | \ > > > > sort ... > > > > > > > > . Since updating to cygwin 3.3.x (but not on cygwin 3.2.x) the script consistently fails when decompressing a portion of the files, resulting in lost data, and error messages of the form: > > > > > > > > xzcat: (stdout): Write error: Bad address > > > > > > Thanks for the report. > > > I could reproduce your problem and found the cause. > > > > > > raw_read()/raw_write() in fhandler_pipe.cc seems to need handling > > > of STATUS_PENDING in nonblocking mode. > > > > > > I also confirmed the following patch fixes the issue. However, I > > > am not very sure that this is the right thing. > > > > > > Corinna, Ken, what do you think? > > > > I'm puzzled. non-blocking pipes return STATUS_PENDING? What on earth > > is that supposed to mean on non-blocking (as opposed to asynchronous) > > pipes? The problem is that STATUS_PENDING theoretically requires > > to wait for... something, the pipe handle, perhaps, and then to check > > io.Status. > > > > Is it possible that nonblocking pipes need to be synchronous, i. e., > > setting the FILE_SYNCHRONOUS_IO_NONALERT would fix that? > > Yes. Setting FILE_SYNCHRONOUS_IO_NONALERT for the write pipe seems > to fix the issue. That was just a hunch but it doesn't *actually* surprise me... > Proberbly, this needs the modification for > raw_write() like the modification I have made for raw_read() in > topic/pipe branch. The idea was to do change to synchronous pipes only for 3.4. I have a bit of headaches to change the pipes to synchronous as bug fix on the 3.3 branch. Your first ideas might be better there... ...unless, of course, you and Ken are convinced that this is the best approach even as 3.3 bugfix. In that case, just say so and push the stuff to the 3.3 branch, too, as you see fit. Thanks, Corinna