Cygwin on Wine: changing buffer content immediately after aync write() to a pipe corrupts written result

Cygwin on Wine: changing buffer content immediately after aync write() to a pipe corrupts written result

[Qian Hong]

[-- Attachment #1: Type: text/plain, Size: 2379 bytes --]

Dear list,

Recently we are investigating a Wine (Wine Staging) bug encountered by
Cygwin/MSYS2, which also suggest a potential bug in Cygwin.

When running xz on Cygwin/MSYS2 on Wine to decompress data and write
to a pipe, we found data corruption in the final result. This is
reported as Wine Staging bug 394:

https://bugs.wine-staging.com/show_bug.cgi?id=394

The command line to reproduce the bug is below:
$ cd /usr/share/gettext
$ xz -d -c < archive.dir.tar.xz | sha1sum # output result is random on Wine

Tested version:

Cygwin version
2.1.0, 2015/07/14 C:\cygwin\bin\cygwin1.dll

XZ version
$ xz --version
xz (XZ Utils) 5.2.1
liblzma 5.2.1

Wine Staging:
$ wine --version
wine-1.7.49-717-g3404140 (Staging)

In case anyone interesting, we also tested on msys2:
MSYS_NT-5.1  2.2.1(0.289/5/3) 2015-08-09 14:43 i686 Msys

***

With help from Lazhu (xz developer), Corrina and Sebastian, the
problem become a bit clearer. For archive, here are some testing
results we got:


1. This problem can't be reproduce on WinXP or Win7 according to my tests.

2. This problem doesn't happen with xz 5.0.8. Lazhu pointed out a
relate issue which doesn't happen in xz 5.0.8 but discovered by 5.2.1:
https://cygwin.com/ml/cygwin/2015-02/msg00575.html

That's very helpful information, finally we made sure the two problem
are different but get a clear idea what might be wrong.

3. Modified from Lazhu's previous test case, we wrote a new test case
to simulate the new problem, as attachment writer.c:
$ gcc writer.c -o writer
$ ./writer | sha1sum # the result is randomly on Wine but constant on Windows

The test case reproduce the problem on Wine, which generate randomly sha1sum.

If pipe is not used, then there is no data corruption:
$ ./write > good.txt && sha1sume good.txt # always good on Wine and Windows

However, the same test case doesn't reproduce anything wrong on WinXP
and Win7, sha1sum is always constant.

I tried modified buffer size, loop counts, etc inside the test case, I
also made sure EAGAIN is trigger on both Wine and Windows. I also
tried something like below to slow down reader speed:
$ ./write | pv -L512k | sha1sume

In any case, it's 100% reproduce on Wine, but never reproduce on Windows.

***

We are going to do some more test, will update this thread when we
have something interesting.

Thanks.


-- 
Regards,
Qian Hong

-
http://www.winehq.org

[-- Attachment #2: writer.c --]
[-- Type: text/x-csrc, Size: 1123 bytes --]

// gcc -std=gnu99 -Wall -Wextra writer.c -o writer

#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <unistd.h>
#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include <errno.h>

static void
fail(const char *str)
{
	fprintf(stderr, "W: %s\n", str);
	exit(EXIT_FAILURE);
}

int
main(void)
{
	int flags = fcntl(STDOUT_FILENO, F_GETFL);
	if (flags == -1)
		fail("Cannot get stdout file status flags");

	if (fcntl(STDOUT_FILENO, F_SETFL, flags | O_NONBLOCK) == -1)
		fail("Setting stdout to non-blocking mode failed");

	static unsigned char buf[64 << 10];
	int blocking = 0;
	int i = 0;

	while (i < 256)
	{
		int ret = write(STDOUT_FILENO, buf, sizeof(buf));
		if (ret == -1 && errno == EAGAIN)
		{
			blocking++;
			continue;
		}

		if (ret != sizeof(buf))
			fail("Unexpected number of bytes");

		memset(buf, ++i, sizeof(buf));
	}

	if (!blocking)
		fail("Was never blocking, read was too fast");

	if (fcntl(STDOUT_FILENO, F_SETFL, flags) == -1)
		fail("Restoring stdout file status flags failed");

	if (close(STDOUT_FILENO))
		fail("Error closing stdout");

	return EXIT_SUCCESS;
}

[-- Attachment #3: Type: text/plain, Size: 218 bytes --]

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Cygwin on Wine: changing buffer content immediately after aync write() to a pipe corrupts written result

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 610 bytes --]

Hi Qian,

On Aug 15 19:11, Qian Hong wrote:
> Dear list,
> 
> Recently we are investigating a Wine (Wine Staging) bug encountered by
> Cygwin/MSYS2, which also suggest a potential bug in Cygwin.
> [...]

thanks for the detailed report and your efforts to reproduce the issue.
As discussed on IRC, I'm going to use this for the release notes.  The
proposed Cygwin fix is in the 2.2.1-0.1 test release I'm uploading right
now.


Thanks,
Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: Cygwin on Wine: changing buffer content immediately after aync write() to a pipe corrupts written result

[Qian Hong]

Hi Corinna,

On Sat, Aug 15, 2015 at 7:27 PM, Corinna Vinschen
<corinna-cygwin@cygwin.com> wrote:
> thanks for the detailed report and your efforts to reproduce the issue.
> As discussed on IRC, I'm going to use this for the release notes.  The
> proposed Cygwin fix is in the 2.2.1-0.1 test release I'm uploading right
> now.

Thanks for the great help on irc. Also thanks for the test release.

I tested 32bit version of the test release, on both Wine and Windows7,
with both the xz command line and the test case writer.c attached in
original post. I can confirm both Cygwin Wine and Cygwin Windows works
now.

$ sha1sum  cygwin1-20150815.dll
929cd14b16bad6e00f4f51f61b9105756cfab8f6  cygwin1-20150815.dll
$ file cygwin1-20150815.dll
cygwin1-20150815.dll: PE32 executable (DLL) (console) Intel 80386
(stripped to external PDB), for MS Windows

Will update if there is anything interesting.

-- 
Regards,
Qian Hong

-
http://www.winehq.org

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple