Re: fstat st_size on open files on Parallels filesystem is wrong

[lennox@cs.columbia.edu]

[-- Attachment #1: message body text --]
[-- Type: text/plain, Size: 3952 bytes --]

On Tuesday, April 22 2014, "Corinna Vinschen" wrote to "cygwin at cygwin.com" saying:

> On Apr 21 14:46, lennox at cs.columbia.edu wrote:
> > On Monday, April 21 2014, "Andrey Repin" wrote to "lennox at
> > cs.columbia.edu, cygwin at cygwin.com" saying:
> > 
> > > Greetings, lennox at cs.columbia.edu!
> > > 
> > > > I’m running cygwin64 1.7.29 in a Windows 8.1 Pro virtual machine, running in
> > > > Parallels Desktop 9.0.24229 on Mac OS X 10.9.2.
> > > 
> > > > Parallels Desktop automatically mounts my Mac OS X home directory as a Z:
> > > > drive in Windows.  Cygwin mount reports this drive as being type "prlsf".
> > > 
> > > > Unfortunately, I've discovered that if I have an open file on this
> > > > filesystem which has been written to, the size returned by Cygwin fstat() on
> > > > the open file is wrong.  A stat() of the file after it's been closed is
> > > > correct.
> > > 
> > > > This has the consequence that emacs always thinks saved files have been
> > > > modified externally, since emacs looks at files' sizes (as well as their
> > > > modification times) to detect external changes.  This makes emacs
> > > > near-unusable.
> > > 
> > > > This problem does not occur for files in my Cygwin home directory, or other
> > > > locations mounted on my Windows C: drive.
> > > 
> > > > I've attached a simple unit test program that illustrates the problem.
> > > > I've also attached my cygcheck -s -v -r output.
> > > 
> > > > Any ideas?  Is this a Cygwin bug, a Parallels bug, or something else?
> > > > Glancing over the Cygwin code, I see that there are a few cases where fstat
> > > > has special cases for certain filesystem types.
> > > 
> > > You never flushing the buffer in your test code, or I'm reading it wrong?
> > 
> > This is using Posix APIs -- open() / write() -- not C APIs, fopen() /
> > fwrite(), so there shouldn't be a buffer?  Notice that the test behaves as I
> > expect for a file on NTFS.
> > 
> > Adding a call to fsync() prior to the fstat() call doesn't change anything.
> 
> This is actually a bad sign.  The problem you're describing occurs on
> NFS, too.  If you write to the file, a subsequent call to fetch stat
> attributes does not return the actual size of the file, but the size at
> the time the handle has been opened.
> 
> However, on NFS, a call to FlushFileBuffers helps to kick stat back into
> shape.  That's the Win32 function called from fsync as well.  What is
> Cygwin supposed to do if that doesn't work?

Okay, interesting further investigation.

The Parallels filesystem appears to work correctly when I repeat the test
case using Windows kernel32 APIs -- specifically, FileInformationByHandle --
so something's different between the kernel32 APIs and the ntdll APIs that
Cygwin uses.

Sample code for Win32 test attached.  Works identically with Cygwin, MinGW,
or Visual C++.

Just spitballing here, but -- I see that cygwin's file_get_fnoi function
(which is where fhandler_base::fstat_by_handle gets its size parameter)
tries NtQueryInformationFile(FileNetworkOpenInformation) before
NtQueryInformationFile(FileStandardInformation).  Is it possible that the
Parallels filesystem isn't filling out all fields of that API properly?  Is
there a straightforward way I could debug this?



$ ./stat-size-test.exe '/cygdrive/z/foo'
/cygdrive/z/foo: fstat: st_size=0
/cygdrive/z/foo: stat: st_size=12

$ ./win32-size-test.exe 'z:\foo'
z:\foo: FileInformationByHandle: nFileSize=12

$ mount
C:/cygwin64/bin on /usr/bin type ntfs (binary,auto)
C:/cygwin64/lib on /usr/lib type ntfs (binary,auto)
C:/cygwin64 on / type ntfs (binary,auto)
C: on /cygdrive/c type ntfs (binary,posix=0,user,noumount,auto)
S: on /cygdrive/s type ntfs (binary,posix=0,user,noumount,auto)
U: on /cygdrive/u type ntfs (binary,posix=0,user,noumount,auto)
Z: on /cygdrive/z type prlsf (binary,posix=0,user,noumount,auto)


[-- Attachment #2: win32-size-test.c --]
[-- Type: text/plain, Size: 1748 bytes --]

#include <windows.h>
#include <stdio.h>
#include <inttypes.h>

#if __LP64__
#define PRI_DWORD "u"
#else
#define PRI_DWORD "lu"
#endif

static const char* geterr(DWORD err)
{
	static char msg[1024];

	FormatMessage(FORMAT_MESSAGE_FROM_SYSTEM | FORMAT_MESSAGE_IGNORE_INSERTS |
		FORMAT_MESSAGE_MAX_WIDTH_MASK, NULL, err,
		MAKELANGID(LANG_NEUTRAL, SUBLANG_DEFAULT),
		(LPSTR)msg, sizeof(msg), NULL);

	return msg;
}

int main(int argc, char* argv[])
{
	int i;
	int status = 0;
	for (i = 1; i < argc; i++) {
		HANDLE file;
		DWORD err, bytesWritten;
		BY_HANDLE_FILE_INFORMATION fileInfo;

		file = CreateFileA(argv[i], GENERIC_WRITE, FILE_SHARE_WRITE,
			NULL, CREATE_ALWAYS, FILE_ATTRIBUTE_NORMAL, NULL);
		if (file == INVALID_HANDLE_VALUE) {
			err = GetLastError();
			fprintf(stderr, "%s: CreateFile: %s (err %" PRI_DWORD")\n", argv[i],
				geterr(err), err);
			status++;
			continue;
		}

		if (!WriteFile(file, "Hello world!\n", 12, &bytesWritten, NULL) ||
			bytesWritten != 12) {
			err = GetLastError();
			fprintf(stderr, "%s: WriteFile: %s (err %" PRI_DWORD ")\n", argv[i],
				geterr(err), err);
			CloseHandle(file);
			status++;
			continue;
		}

		if (!GetFileInformationByHandle(file, &fileInfo)) {
			err = GetLastError();
			fprintf(stderr, "%s: GetFileInformationByHandle: %s (err %" PRI_DWORD ")\n",
				argv[i], geterr(err), err);
			CloseHandle(file);
			status++;
			continue;
		}

		printf("%s: FileInformationByHandle: nFileSize=%" PRIu64 "\n",
			argv[i], (uint64_t)fileInfo.nFileSizeHigh << 32 | fileInfo.nFileSizeLow);

		if (!CloseHandle(file)) {
			err = GetLastError();
			fprintf(stderr, "%s: CloseHandle: %s (err %" PRI_DWORD ")\n", argv[i],
				geterr(err), err);
			status++;
			continue;
		}
	}
	return status;
}

[-- Attachment #3: .signature --]
[-- Type: text/plain, Size: 49 bytes --]



-- 
Jonathan Lennox
lennox at cs.columbia.edu


[-- Attachment #4: Type: text/plain, Size: 218 bytes --]

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple