Redirecting output from running proc doesn't modify the "last modified time" field for target file

Redirecting output from running proc doesn't modify the "last modified time" field for target file

[ajshower]

I've got a bash script running in Cygwin whose output has been redirected to
another file.  MS-DOS exhibits this same behavior if I run the bash script
as a batch job.

    bash.exe &> log.txt

Adding content to log.txt through redirection does not change log.txt's last
modified time while bash.exe is executing.  

Baseline:
    # ls -la --full-time log.txt
    -r-xr-x---+ 1 user Domain Users 66455 2011-11-30 16:16:45.246664800
-0500 log.txt

Some time later:
    # ls -la --full-time log.txt
    -r-xr-x---+ 1 user Domain Users 66838 2011-11-30 16:16:45.246664800
-0500 log.txt

Note that even though log.txt has gotten larger, the last modified time has
not changed.  The last modified time is updated only when bash.exe
terminates.

Ubuntu 11.04 handles this scenario, IMO, correctly.  Is there a way I can
get the last modified time to be updated when the content of the file
changes and while bash.exe is executing?

Cygwin:  CYGWIN_NT-6.1  
MS-DOS:  Microsoft Windows [Version 6.1.7601]

Thanks
-- 
View this message in context: http://old.nabble.com/Redirecting-output-from-running-proc-doesn%27t-modify-the-%22last-modified-time%22-field-for-target-file-tp32903475p32903475.html
Sent from the Cygwin list mailing list archive at Nabble.com.


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Redirecting output from running proc doesn't modify the "last modified time" field for target file

[Jon Clugston]

On Fri, Dec 2, 2011 at 11:51 AM, ajshower <ajshower@gmail.com> wrote:
>
> I've got a bash script running in Cygwin whose output has been redirected to
> another file.  MS-DOS exhibits this same behavior if I run the bash script
> as a batch job.
>
>    bash.exe &> log.txt
>
> Adding content to log.txt through redirection does not change log.txt's last
> modified time while bash.exe is executing.
>
> Baseline:
>    # ls -la --full-time log.txt
>    -r-xr-x---+ 1 user Domain Users 66455 2011-11-30 16:16:45.246664800
> -0500 log.txt
>
> Some time later:
>    # ls -la --full-time log.txt
>    -r-xr-x---+ 1 user Domain Users 66838 2011-11-30 16:16:45.246664800
> -0500 log.txt
>
> Note that even though log.txt has gotten larger, the last modified time has
> not changed.  The last modified time is updated only when bash.exe
> terminates.
>
> Ubuntu 11.04 handles this scenario, IMO, correctly.  Is there a way I can
> get the last modified time to be updated when the content of the file
> changes and while bash.exe is executing?
>
> Cygwin:  CYGWIN_NT-6.1
> MS-DOS:  Microsoft Windows [Version 6.1.7601]
>

I don't see how this could be a Cygwin issue.  If writing to a file
(while it is open) doesn't immediately update the last modified time
on Windows, there is no reasonable way for Cygwin to "fix" that.

IMHO, this is a very dubious "feature" to depend upon.  If you want to
know if the file has changed, why not check its length instead?

Jon

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Redirecting output from running proc doesn't modify the "last modified time" field for target file

[ajshower]

>IMHO, this is a very dubious "feature" to depend upon.  If you want to
>know if the file has changed, why not check its length instead?

I have switched to using the length instead but I was interested in what
people thought of the issue.

I'm having trouble reproducing this using just Cygwin so I'm not presenting
the problem accurately.  My process is to call  Runtime.getRuntime.exec()
within a java process which opens cygwin/bin/bash.exe and then runs the
redirection command from within the bash.  From the same java process that
called exec(), I'm trying to monitor the last modified time based on what's
going on in the previously called bash.exe.  That does sound pretty
convoluted.  I guess it's some kind of scope problem, but checking length()
is working, so it's all moot anyway.

I'll close this thread.

-Alex
-- 
View this message in context: http://old.nabble.com/Redirecting-output-from-running-proc-doesn%27t-modify-the-%22last-modified-time%22-field-for-target-file-tp32903475p32904332.html
Sent from the Cygwin list mailing list archive at Nabble.com.


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Redirecting output from running proc doesn't modify the "last modified time" field for target file

[Jon Clugston]

On Fri, Dec 2, 2011 at 1:42 PM, ajshower <ajshower@gmail.com> wrote:
>
>>IMHO, this is a very dubious "feature" to depend upon.  If you want to
>>know if the file has changed, why not check its length instead?
>
> I have switched to using the length instead but I was interested in what
> people thought of the issue.
>
> I'm having trouble reproducing this using just Cygwin so I'm not presenting
> the problem accurately.  My process is to call  Runtime.getRuntime.exec()
> within a java process which opens cygwin/bin/bash.exe and then runs the
> redirection command from within the bash.  From the same java process that
> called exec(), I'm trying to monitor the last modified time based on what's
> going on in the previously called bash.exe.  That does sound pretty
> convoluted.  I guess it's some kind of scope problem, but checking length()
> is working, so it's all moot anyway.
>

I reproduced your symptoms with this simple shell script:

while sleep 10 ; do
  echo
done >x.log &

While this loop is running, the timestamp on "x.log" doesn't change
(whereas on Linux it changes every 10 seconds).  It sure looks to me
that Windows just doesn't bother updating the file timestamp while it
is open.  I don't know if this update is required by POSIX - I would
doubt that it is.


Jon

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Redirecting output from running proc doesn't modify the "last modified time" field for target file

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 909 bytes --]

On 12/02/2011 11:50 AM, Jon Clugston wrote:
> While this loop is running, the timestamp on "x.log" doesn't change
> (whereas on Linux it changes every 10 seconds).  It sure looks to me
> that Windows just doesn't bother updating the file timestamp while it
> is open.  I don't know if this update is required by POSIX - I would
> doubt that it is.

POSIX requires that any write() to an open file mark it for update; the
update doesn't have to occur right away (so you can batch up several
writes, but only change the mtime metadata once at the end of the
batch), but it DOES require that stat() and several similar functions
flush all marked updates prior to exposing timestamps to the user.  So
yes, Windows is violating POSIX, and I have no idea whether cygwin can
work around it.

-- 
Eric Blake   eblake@redhat.com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 620 bytes --]

Re: Redirecting output from running proc doesn't modify the "last modified time" field for target file

[Corinna Vinschen]

On Dec  2 13:04, Eric Blake wrote:
> On 12/02/2011 11:50 AM, Jon Clugston wrote:
> > While this loop is running, the timestamp on "x.log" doesn't change
> > (whereas on Linux it changes every 10 seconds).  It sure looks to me
> > that Windows just doesn't bother updating the file timestamp while it
> > is open.  I don't know if this update is required by POSIX - I would
> > doubt that it is.
> 
> POSIX requires that any write() to an open file mark it for update; the
> update doesn't have to occur right away (so you can batch up several
> writes, but only change the mtime metadata once at the end of the
> batch), but it DOES require that stat() and several similar functions
> flush all marked updates prior to exposing timestamps to the user.  So
> yes, Windows is violating POSIX, and I have no idea whether cygwin can
> work around it.

You can change all file operations to use FILE_WRITE_THROUGH and
FILE_NO_INTERMEDIATE_BUFFERING.  Downside:  No caching.  All file
operations must be sector aligned.  Degraded system performance.
Broken when a process has only write permissions.

Alternatively, change write(2) so that every WriteFile call is
accompanied by a FlushFileBuffers call.  Downside: Extremly degraded
write performance.

Alternatively:  Lie.  That's how SUA does it.  It has a background
service running which (among other things) keeps track of write
operations of SUA applications.  If a SUA application calls write(2)
the write timestamp is kept up to date internally, while the metadata
on disk is still lagging in Windows style.  A SUA application calling
stat(2) gets a POSIX compatible timestamp.  Non-SUA apps continue to
show the "wrong" timestamp.  If non-SUA apps write to a file, SUA apps
also show the Windows timestamp.  Cygwin could do the same.  Downside:
We don't have a mandatory background service running.  Quite a hoop to
jump through to implement a usually non-critical POSIX requirement.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Project Co-Leader          cygwin AT cygwin DOT com
Red Hat

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Redirecting output from running proc doesn't modify the "last modified time" field for target file

[Christopher Faylor]

On Sat, Dec 03, 2011 at 09:36:56PM +0100, Corinna Vinschen wrote:
>On Dec  2 13:04, Eric Blake wrote:
>> On 12/02/2011 11:50 AM, Jon Clugston wrote:
>> > While this loop is running, the timestamp on "x.log" doesn't change
>> > (whereas on Linux it changes every 10 seconds).  It sure looks to me
>> > that Windows just doesn't bother updating the file timestamp while it
>> > is open.  I don't know if this update is required by POSIX - I would
>> > doubt that it is.
>> 
>> POSIX requires that any write() to an open file mark it for update; the
>> update doesn't have to occur right away (so you can batch up several
>> writes, but only change the mtime metadata once at the end of the
>> batch), but it DOES require that stat() and several similar functions
>> flush all marked updates prior to exposing timestamps to the user.  So
>> yes, Windows is violating POSIX, and I have no idea whether cygwin can
>> work around it.
>
>You can change all file operations to use FILE_WRITE_THROUGH and
>FILE_NO_INTERMEDIATE_BUFFERING.  Downside:  No caching.  All file
>operations must be sector aligned.  Degraded system performance.
>Broken when a process has only write permissions.
>
>Alternatively, change write(2) so that every WriteFile call is
>accompanied by a FlushFileBuffers call.  Downside: Extremly degraded
>write performance.
>
>Alternatively:  Lie.  That's how SUA does it.  It has a background
>service running which (among other things) keeps track of write
>operations of SUA applications.  If a SUA application calls write(2)
>the write timestamp is kept up to date internally, while the metadata
>on disk is still lagging in Windows style.  A SUA application calling
>stat(2) gets a POSIX compatible timestamp.  Non-SUA apps continue to
>show the "wrong" timestamp.  If non-SUA apps write to a file, SUA apps
>also show the Windows timestamp.  Cygwin could do the same.  Downside:
>We don't have a mandatory background service running.  Quite a hoop to
>jump through to implement a usually non-critical POSIX requirement.

Since I believe that this has come up before is anyone willing to
provide a FAQ entry that we can point to in the future?

cgf

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Redirecting output from running proc doesn't modify the "last modified time" field for target file

[Eliot Moss]

In my experience such metadata tends to be
updated when the file is closed, but not
necessarily before that time. This may
be different from Unix, but (as mentioned)
is probably not something cygwin can "fix".

Best wishes -- Eliot Moss

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple