public inbox for cygwin-developers@cygwin.com
 help / color / mirror / Atom feed
From: Corinna Vinschen <corinna-cygwin@cygwin.com>
To: cygwin-developers@cygwin.com
Subject: Re: cygwin 3.3.x: another problem that may be related to pipes
Date: Mon, 15 Nov 2021 17:08:02 +0100	[thread overview]
Message-ID: <YZKF4oY4cBMM8F08@calimero.vinschen.de> (raw)
In-Reply-To: <20211115235021.0f0f64b1b0e2a7bd6d16be80@nifty.ne.jp>

On Nov 15 23:50, Takashi Yano wrote:
> On Mon, 15 Nov 2021 14:01:12 +0100
> Corinna Vinschen wrote:
> > [Redirected to cygwin-developers]
> > 
> > On Nov 15 17:18, Takashi Yano via Cygwin wrote:
> > > On Sun, 14 Nov 2021 20:50:59 +0000 (UTC)
> > > bf wrote:
> > > > I've a shell script that processes several hundred xz-compressed text files with a pipeline of the form:
> > > > 
> > > > find . -depth -type f  ... -print0 | \
> > > > xargs -0tI @ xzcat -- @ | \
> > > > iconv --byte-subst='�' --unicode-subst='�' --widechar-subst='�' -f MS-ANSI -t UTF-8 -- | \
> > > > grep ... \
> > > > sed ... | \
> > > > sort ... | \
> > > > sort ...
> > > > 
> > > > . Since updating to cygwin 3.3.x (but not on cygwin 3.2.x) the script consistently fails when decompressing a portion of the files, resulting in lost data, and error messages of the form:
> > > > 
> > > > xzcat: (stdout): Write error: Bad address
> > > 
> > > Thanks for the report.
> > > I could reproduce your problem and found the cause.
> > > 
> > > raw_read()/raw_write() in fhandler_pipe.cc seems to need handling
> > > of STATUS_PENDING in nonblocking mode.
> > > 
> > > I also confirmed the following patch fixes the issue. However, I
> > > am not very sure that this is the right thing.
> > > 
> > > Corinna, Ken, what do you think?
> > 
> > I'm puzzled.  non-blocking pipes return STATUS_PENDING?  What on earth
> > is that supposed to mean on non-blocking (as opposed to asynchronous)
> > pipes?  The problem is that STATUS_PENDING theoretically requires
> > to wait for... something, the pipe handle, perhaps, and then to check
> > io.Status.
> > 
> > Is it possible that nonblocking pipes need to be synchronous, i. e.,
> > setting the FILE_SYNCHRONOUS_IO_NONALERT would fix that?
> 
> Yes. Setting FILE_SYNCHRONOUS_IO_NONALERT for the write pipe seems
> to fix the issue.

That was just a hunch but it doesn't *actually* surprise me...

> Proberbly, this needs the modification for
> raw_write() like the modification I have made for raw_read() in
> topic/pipe branch.

The idea was to do change to synchronous pipes only for 3.4.  I have a
bit of headaches to change the pipes to synchronous as bug fix on the
3.3 branch.  Your first ideas might be better there...

...unless, of course, you and Ken are convinced that this is the best
approach even as 3.3 bugfix.  In that case, just say so and push the
stuff to the 3.3 branch, too, as you see fit.


Thanks,
Corinna

  reply	other threads:[~2021-11-15 16:08 UTC|newest]

Thread overview: 20+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
     [not found] <115203324.649908.1636923059546.ref@mail.yahoo.com>
     [not found] ` <115203324.649908.1636923059546@mail.yahoo.com>
     [not found]   ` <20211115171811.844dce9cce2b4d13262d64f2@nifty.ne.jp>
2021-11-15 13:01     ` Corinna Vinschen
2021-11-15 13:57       ` Ken Brown
2021-11-15 14:36         ` Takashi Yano
2021-11-15 15:11           ` Takashi Yano
2021-11-15 15:27             ` Ken Brown
2021-11-15 15:36               ` Ken Brown
2021-11-15 16:45                 ` Takashi Yano
2021-11-15 16:25           ` Corinna Vinschen
2021-11-15 14:50       ` Takashi Yano
2021-11-15 16:08         ` Corinna Vinschen [this message]
2021-11-15 16:37           ` Ken Brown
2021-11-15 16:52             ` Takashi Yano
2021-11-15 18:47               ` Ken Brown
2021-11-15 23:35                 ` Takashi Yano
2021-11-15 23:49                   ` Ken Brown
2021-11-16  3:28                     ` Takashi Yano
2021-11-16  9:28                       ` Corinna Vinschen
2021-11-16  9:45                     ` Takashi Yano
2021-11-16 10:20                       ` Takashi Yano
2021-11-16 14:35                         ` Takashi Yano

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=YZKF4oY4cBMM8F08@calimero.vinschen.de \
    --to=corinna-cygwin@cygwin.com \
    --cc=cygwin-developers@cygwin.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).