From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 13170 invoked by alias); 1 Dec 2013 13:24:54 -0000 Mailing-List: contact cygwin-help@cygwin.com; run by ezmlm Precedence: bulk List-Id: List-Subscribe: List-Archive: List-Post: List-Help: , Sender: cygwin-owner@cygwin.com Mail-Followup-To: cygwin@cygwin.com Received: (qmail 13156 invoked by uid 89); 1 Dec 2013 13:24:51 -0000 Authentication-Results: sourceware.org; auth=none X-Virus-Found: No X-Spam-SWARE-Status: No, score=1.6 required=5.0 tests=BAYES_50,RDNS_NONE autolearn=no version=3.3.2 X-HELO: mailscanner01.zoner.fi Received: from Unknown (HELO mailscanner01.zoner.fi) (84.34.166.10) by sourceware.org (qpsmtpd/0.93/v0.84-503-g423c35a) with (AES256-SHA encrypted) ESMTPS; Sun, 01 Dec 2013 13:24:49 +0000 Date: Sun, 01 Dec 2013 13:24:00 -0000 From: Lasse Collin To: Denis Excoffier Cc: Tom Honermann , Cygwin Mailing List Subject: Re: Antivirus strikes back (probably) (Was: Intermittent failures retrieving process exit codes) Message-ID: <20131201152415.64cab6e3@tukaani.org> In-Reply-To: <9E1F6369-88DF-4EDB-A752-1340ACBA7564@Denis-Excoffier.org> References: <50C2498C.2000003@coverity.com> <50C276AC.9090301@mailme.ath.cx> <50D401EF.9040705@coverity.com> <52844B2E.5050902@coverity.com> <20131125215842.0ff53134@tukaani.org> <9E1F6369-88DF-4EDB-A752-1340ACBA7564@Denis-Excoffier.org> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Antivirus-Scanner: Clean mail though you should still use an Antivirus Received-SPF: none X-SW-Source: 2013-12/txt/msg00018.txt.bz2 On 2013-11-26 Denis Excoffier wrote: > On 2013-11-25 =C3=A0 21:58 +02:00, Lasse Collin wrote: > > If you haven't already tried, please compile both 5.1.2alpha and > > 5.1.3alpha from source while keeping everything else unchanged, and > > see if the bug really only occurs with 5.1.3alpha. > Already done. I did some strace-ing, and since i=E2=80=99m not so fluent = with > the result, i=E2=80=99ll send it there in a while (when i=E2=80=99m back = on cygwin) > if someone is interested. But the bug (contrary to what i said > before) also _sometimes_ occurs with 5.1.2alpha or 5.0.5 and this > makes me think now that: >=20 > a) my antivirus-anti-intrusion-whatever-software (that i can=E2=80=99t re= move > of course) creates some kind of "background noise" where a certain > percentage of such =E2=80=98tar xf =E2=80=94use-compress-program=E2=80=99= commands will > always fail >=20 > b) nevertheless, xz-5.1.3alpha (with its new file I/O code etc.) > triggers some untypical configuration inside the antivirus that > increases drastically the percentage, making the failure almost > certain for some files. >=20 > It is not extraordinary that i cannot observe the failure on XP since > i do not have this particular antivirus on XP. OK, so the new I/O code in xz probably isn't the problem even if it may affect how easily the actual problem gets triggered. [...] > When it fails (usually or pathologically), the last file of the > archive gets truncated (see above), and _this_ is strange from an > antivirus behaviour. After all, perhaps some flush() or similar is > missing inside 5.1.3alpha. xz uses write() which uses a file descriptor argument, so there is nothing to flush separately. xz just has to write() everything. When used with tar, xz writes to standard output (FILENO_STDOUT) which with tar is a pipe. When xz finishes, it closes its end (the writer end) of the pipe. With xz 5.1.3alpha, O_NONBLOCK flag is set for FILENO_STDIN and FILENO_STDOUT if the flag wasn't already set. If xz set the flag, it will unset it before closing the file descriptor. The setting and unsetting can be seen in the trace you sent and it seems to work correctly. I don't have a guess if these fcntl() calls might cause the difference between 5.1.3alpha and other versions, but it doesn't sound too important since the bug occurs in some form with all versions. =46rom the trace file it seems that the last write() from xz gets lost. xz first makes 173 writes of 8192 bytes and then one 6144-byte write, totalling 1,423,360 bytes. tar gets 1,417,216 from xz, that is, 6144 bytes too little. Since things go wrong with old xz versions that don't use non-blocking I/O, I would expect you to see similar issues with other compressors too. Maybe it would be worth testing with gzip and bzip2 in the same way you did with xz 5.0.5. --=20 Lasse Collin | IRC: Larhzu @ IRCnet & Freenode -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple