Problem with daylight saving time, off by one hour

Problem with daylight saving time, off by one hour

[Frank Farance]

I've had Cygwin running on a Windows XP workstation for years using rsync
as my primary backup tool for my data files.  The workstation just crashed
with a hard drive failure, I've replaced the drive and so on.  Initially,
I was recovering most of the files with WinSCP (5.0.5) from the backup
server.

While this was going on, I was working on the script to recover the files
(essentially swapping the arguments to the rsync script I used for
backup).  As I was testing this on already-recovered files (via WinSCP) I
noticed that some needed to be transferred again.  To make a long story
short, rsync (via Cygwin) has a different sense of time than WinSCP or
Windows for files modified during summer time.

For example, the file 100_1164.JPG copied by WinSCP reports a timestamp of
2005-09-01 10:46, which is consistent with the EXIF data in the file, and
consistent with Windows XP's report of the time 10:46.

Meanwhile, rsync belives the file is different and looks to update it. 
Furthermore, ls reports the wrong time (via --full-time) as 11:46 -0400. 
Yes, ls has the right timezone offset (it was summer time in NYC on
2005-09-01), but the time itself is wrong.  Even when I precede the
command with TZ=UTC0, the UTC time reported by ls is wrong (says 15:46,
should be 14:46).

I've read previous posts on this problem in a variety of places and they
mostly suggest that one use "--modify-window=3601" or something like that
to make timestamps fuzzy.  In my opinion, that is the wrong solution
because the rsync client (i.e., the files being backed up) will record the
wrong time on the rsync server because the rsync client itself has the
wrong sense of time.  And, as I pointed out above, the ls command has the
same problem, too.

My sense is the implementation of the stat() system call is wrong because
even with the TZ set to UTC, it still reports the wrong time.

Now this all used to work nicely with Cygwin before.  I don't know exactly
the prior version of Cygwin, but I had downloaded it in 2006-11.

My present version of Cygwin, which was downloaded yesterday is: DLL
version: 1.7.11.  The current version of rsync is 3.0.9-1.

Is this fixed by a new release or a different configuration?  Please don't
respond with "windows is stupid" because (1) it's cygwin's job to figure
out these quirks with Windows, (2) other tools get this right (e.g.,
WinSCP), and (3) it used to work right on old versions of cygwin (I've
been using cywin rsync for at least a dozen years).  I have several
million files to back up (with integrity), so I need an rsync-like
solution.

Thank you, in advance, for your help and insight.

Frank Farance



--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Problem with daylight saving time, off by one hour

[Corinna Vinschen]

On Feb 28 10:00, Frank Farance wrote:
> I've had Cygwin running on a Windows XP workstation for years using rsync
> as my primary backup tool for my data files.  The workstation just crashed
> with a hard drive failure, I've replaced the drive and so on.  Initially,
> I was recovering most of the files with WinSCP (5.0.5) from the backup
> server.
> 
> While this was going on, I was working on the script to recover the files
> (essentially swapping the arguments to the rsync script I used for
> backup).  As I was testing this on already-recovered files (via WinSCP) I
> noticed that some needed to be transferred again.  To make a long story
> short, rsync (via Cygwin) has a different sense of time than WinSCP or
> Windows for files modified during summer time.
> 
> For example, the file 100_1164.JPG copied by WinSCP reports a timestamp of
> 2005-09-01 10:46, which is consistent with the EXIF data in the file, and
> consistent with Windows XP's report of the time 10:46.
> 
> Meanwhile, rsync belives the file is different and looks to update it. 
> Furthermore, ls reports the wrong time (via --full-time) as 11:46 -0400. 
> Yes, ls has the right timezone offset (it was summer time in NYC on
> 2005-09-01), but the time itself is wrong.  Even when I precede the
> command with TZ=UTC0, the UTC time reported by ls is wrong (says 15:46,
> should be 14:46).

I can't reproduce this issue.  I have files created during summer time
as well, on my Linux machine and my Cygwin box.  The output of ls -l
in Linux and Cygwin is identical for the files on the Samba share,
and the timestamps of Windows and Cygwin are identical as well.

Are the timezone settings on the remote machine and the local machine
identical?


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Project Co-Leader          cygwin AT cygwin DOT com
Red Hat

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Problem with daylight saving time, off by one hour

[Frank Farance]

Corinna Vinschen wrote:
> On Feb 28 10:00, Frank Farance wrote:
>
>> I've had Cygwin running on a Windows XP workstation for years using
>> rsync as my primary backup tool for my data files.  The workstation just
>> crashed with a hard drive failure, I've replaced the drive and so on.
>> Initially,
>> I was recovering most of the files with WinSCP (5.0.5) from the backup
>> server.
>>
>> While this was going on, I was working on the script to recover the
>> files (essentially swapping the arguments to the rsync script I used for
>>  backup).  As I was testing this on already-recovered files (via
>> WinSCP) I
>> noticed that some needed to be transferred again.  To make a long story
>> short, rsync (via Cygwin) has a different sense of time than WinSCP or
>> Windows for files modified during summer time.
>>
>>
>> For example, the file 100_1164.JPG copied by WinSCP reports a timestamp
>> of 2005-09-01 10:46, which is consistent with the EXIF data in the file,
>> and consistent with Windows XP's report of the time 10:46.
>>
>> Meanwhile, rsync belives the file is different and looks to update it.
>> Furthermore, ls reports the wrong time (via --full-time) as 11:46 -0400.
>>  Yes, ls has the right timezone offset (it was summer time in NYC on
>> 2005-09-01), but the time itself is wrong.  Even when I precede the
>> command with TZ=UTC0, the UTC time reported by ls is wrong (says 15:46,
>> should be 14:46).
>
> I can't reproduce this issue.  I have files created during summer time
> as well, on my Linux machine and my Cygwin box.  The output of ls -l in
> Linux and Cygwin is identical for the files on the Samba share,
> and the timestamps of Windows and Cygwin are identical as well.
>
> Are the timezone settings on the remote machine and the local machine
> identical?
>
>
> Corinna
>
>
> --
> Corinna Vinschen                  Please, send mails regarding Cygwin to
> Cygwin Project Co-Leader          cygwin AT cygwin DOT com
> Red Hat

Corinna-

Thank you for your help.  Yes, both are America/New_York.

My configuration is: (1) workstation and backup server on same LAN, (2)
workstation running Windows XP, (3) backup server running FC8 (2.6.25.9
kernel), (4) no Samba shares.

To repeat:

(1) copy summer files from backup server to workstation via WinSCP
(2) inspect timestamp on server
(3) inspect timestamp on workstation using right-click -> properties
(4) use "ls --full-time" to see local files and they will be different
than what Windows reports

which produces:

(5) now rsync the files and you'll see the workstation files are to be
updated
(6) once rsync completes, Windows (via properties) reports the wrong time

Although this is a problem in the context of rsync, the real problem is
that cygwin has a different sense of time than Windows for the summer
files.

-FF



--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Problem with daylight saving time, off by one hour

[Corinna Vinschen]

On Feb 28 11:19, Frank Farance wrote:
> Corinna Vinschen wrote:
> >> Furthermore, ls reports the wrong time (via --full-time) as 11:46 -0400.
> >>  Yes, ls has the right timezone offset (it was summer time in NYC on
> >> 2005-09-01), but the time itself is wrong.  Even when I precede the
> >> command with TZ=UTC0, the UTC time reported by ls is wrong (says 15:46,
> >> should be 14:46).
> >
> > I can't reproduce this issue.  I have files created during summer time
> > as well, on my Linux machine and my Cygwin box.  The output of ls -l in
> > Linux and Cygwin is identical for the files on the Samba share,
> > and the timestamps of Windows and Cygwin are identical as well.
> >
> > Are the timezone settings on the remote machine and the local machine
> > identical?
> 
> Corinna-
> 
> Thank you for your help.  Yes, both are America/New_York.

Is that what you have set in Cygwin's TZ?  Did you set it manually or is
that the timezone setting fetched from Windows by the call to the tzset
utility in /etc/profile.d/tzset.sh?

> My configuration is: (1) workstation and backup server on same LAN, (2)
> workstation running Windows XP, (3) backup server running FC8 (2.6.25.9
> kernel), (4) no Samba shares.
> 
> To repeat:
> 
> (1) copy summer files from backup server to workstation via WinSCP
> (2) inspect timestamp on server
> (3) inspect timestamp on workstation using right-click -> properties
> (4) use "ls --full-time" to see local files and they will be different
> than what Windows reports

Yeah, I got that.  As I said, I also have "summer files" on the local
machine, and the timestamp printed in Windows Explorer is the same as
what Cygwin's ls --full-time prints.  The timezone information in the
--full-time output is correct as well.

> Although this is a problem in the context of rsync, the real problem is
> that cygwin has a different sense of time than Windows for the summer
> files.

Not for me.  The only difference I see is that I'm living in another
timezone.  I changed my timezone to America/New_York as well, but the
time is still correct in ls.
Without a reproducible scenario (which does not involve non-system,
non-Cygwin tools like winSCP) it's pretty hard to track down why you see
the wrong time information.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Project Co-Leader          cygwin AT cygwin DOT com
Red Hat

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Problem with daylight saving time, off by one hour

[Frank Farance]

> On Feb 28 11:19, Frank Farance wrote:
>
>> Corinna Vinschen wrote:
>>
>>>> Furthermore, ls reports the wrong time (via --full-time) as 11:46
>>>> -0400.
>>>> Yes, ls has the right timezone offset (it was summer time in NYC on
>>>> 2005-09-01), but the time itself is wrong.  Even when I precede the
>>>> command with TZ=UTC0, the UTC time reported by ls is wrong (says
>>>> 15:46,
>>>> should be 14:46).
>>>
>>> I can't reproduce this issue.  I have files created during summer
>>> time as well, on my Linux machine and my Cygwin box.  The output of ls
>>> -l in
>>> Linux and Cygwin is identical for the files on the Samba share,
>>> and the timestamps of Windows and Cygwin are identical as well.
>>>
>>> Are the timezone settings on the remote machine and the local machine
>>>  identical?
>>
>> Corinna-
>>
>>
>> Thank you for your help.  Yes, both are America/New_York.
>>
>
> Is that what you have set in Cygwin's TZ?  Did you set it manually or is
> that the timezone setting fetched from Windows by the call to the tzset
> utility in /etc/profile.d/tzset.sh?

TZ is set to that when I start the bash command by double-clicking the
cygwin terminal icon.  I did not set the timezone, it was set like that
(correctly) after I installed the software.

>> My configuration is: (1) workstation and backup server on same LAN, (2)
>>  workstation running Windows XP, (3) backup server running FC8
>> (2.6.25.9
>> kernel), (4) no Samba shares.
>>
>> To repeat:
>>
>>
>> (1) copy summer files from backup server to workstation via WinSCP
>> (2) inspect timestamp on server
>> (3) inspect timestamp on workstation using right-click -> properties
>> (4) use "ls --full-time" to see local files and they will be different
>> than what Windows reports
>
> Yeah, I got that.  As I said, I also have "summer files" on the local
> machine, and the timestamp printed in Windows Explorer is the same as what
> Cygwin's ls --full-time prints.  The timezone information in the
> --full-time output is correct as well.
>
>
>> Although this is a problem in the context of rsync, the real problem is
>>  that cygwin has a different sense of time than Windows for the summer
>> files.
>
> Not for me.  The only difference I see is that I'm living in another
> timezone.  I changed my timezone to America/New_York as well, but the time
> is still correct in ls. Without a reproducible scenario (which does not
> involve non-system, non-Cygwin tools like winSCP) it's pretty hard to
> track down why you see the wrong time information.
>
>
> Corinna

Corinna, maybe my point wasn't clear above.  I mentioned WinSCP to reveal
how the files arrived.  However, it was via ***Windows*** tools where I
was able to see the timestamp: (1) a right-click->properties on the file
shows the correct timestamp, (2) I just tried "dir" from the command line
and got the right time, (3) but "ls -l --full-time" shows the wrong time
<-- all reproduced by system or cygwin tools.

FYI, I just tried a different folder that has winter files and all three
steps above report the same time.  This would dispell the idea that I had
the wrong time zone, right?

How would you like me to debug this further?  Thanks!

-FF



--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Problem with daylight saving time, off by one hour

[Corinna Vinschen]

On Feb 28 12:23, Frank Farance wrote:
> > Not for me.  The only difference I see is that I'm living in another
> > timezone.  I changed my timezone to America/New_York as well, but the time
> > is still correct in ls. Without a reproducible scenario (which does not
> > involve non-system, non-Cygwin tools like winSCP) it's pretty hard to
> > track down why you see the wrong time information.
> 
> Corinna, maybe my point wasn't clear above.  I mentioned WinSCP to reveal
> how the files arrived.  However, it was via ***Windows*** tools where I
> was able to see the timestamp: (1) a right-click->properties on the file

You made your point entirely clear.  But I really hope I made my point
as clear as well.  In theory it shouldn't matter how the files arrived
on the machine.  Still, there is the problem of reproducibility.  For
clearness:

- My Windows timezone is "(UTC+01:00) Amsterdam, Berlin, Bern, Rome,
  Stockholm, Vienna", which is +1 from UTC, +2 in DST.

- My C:\Windows folder has a modification timestamp of 2011-07-16 10:15
  in a standard Windows Explorer folder view.

- In CMD: dir C:\ prints
    [...]
    16.07.2011  09:15    <DIR>          Windows

  Note that it does NOT print 10:15!

- In a Cygwin shell:

  tcsh$ echo $TZ
  Europe/Berlin
  tcsh$ cd /cygdrive/c
  tcsh$ ls -ld --full-time Windows
  drwxrwx---+ 1 ???????? ???????? 0 2011-07-16 10:15:47.190400000 +0200 Windows

So Explorer and Cygwin agree on the time, only CMD is an hour off because
CMD apparently ignores the fact that the file's modification timestamp
is within DST, so it does not add an hour.

Another example involving a cp from a remote machine:

- Linux:

  tcsh$ echo $TZ
  Europe/Berlin
  tcsh$ ls -l --full-time x
  -rw-r--r--. 1 corinna vinschen 12849 2007-07-24 16:47:48.000000000 +0200 x

- Cygwin:

  tcsh$ cp -p //linux/share/x .
  tcsh$ ls -ld --full-time x
  -rw-r--r-- 1 corinna vinschen 12849 2007-07-24 16:47:48.000000000 +0200 x
 
- CMD:

  dir x
  24.07.2007  15:47            12.849 x

- Windows Explorer shows a timestamp of 2007-07-24 16:47.

Here, too, all affected parties agree on the timestamp, only CMD
doesn't.  A change to another Windows timezone only offsets the
timestamps, but doesn't change the general picture.

> shows the correct timestamp, (2) I just tried "dir" from the command line
> and got the right time, (3) but "ls -l --full-time" shows the wrong time
> <-- all reproduced by system or cygwin tools.

Which is kind of weird, because that's *exactly* inverse to what I observe
on my machine.

> FYI, I just tried a different folder that has winter files and all three
> steps above report the same time.  This would dispell the idea that I had
> the wrong time zone, right?

Well... probably.  What strikes me as weird is the exact opposite behaviour
from what I see.  If the file is a DST file, but Explorer and CMD don't
disagree by an hour, what does that mean?  That would mean that Explorer
doesn't treat the DST timestamp as DST timestamp for some reason.

> How would you like me to debug this further?  Thanks!

I don't know.  I can't see that anything's wrong with Cygwin here.  I
just searched the web and found other people having timestamp problems
with WinSCP without any Cygwin involvement:

  http://winscp.net/forum/viewtopic.php?t=8382

Maybe it *is* winSCP, after all?


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Project Co-Leader          cygwin AT cygwin DOT com
Red Hat

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Problem with daylight saving time, off by one hour

[Earnie Boyd]

On Tue, Feb 28, 2012 at 1:20 PM, Corinna Vinschen wrote:
>
> I don't know.  I can't see that anything's wrong with Cygwin here.  I
> just searched the web and found other people having timestamp problems
> with WinSCP without any Cygwin involvement:
>
>  http://winscp.net/forum/viewtopic.php?t=8382
>
> Maybe it *is* winSCP, after all?
>

Or maybe the winSCP settings?

http://winscp.net/eng/docs/ui_login_environment

-- 
Earnie
-- https://sites.google.com/site/earnieboyd

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Problem with daylight saving time, off by one hour

[Frank Farance]

Earnie Boyd wrote:
> On Tue, Feb 28, 2012 at 1:20 PM, Corinna Vinschen wrote:
>
>>
>> I don't know.  I can't see that anything's wrong with Cygwin here.  I
>>  just searched the web and found other people having timestamp problems
>>  with WinSCP without any Cygwin involvement:
>>
>>  http://winscp.net/forum/viewtopic.php?t=8382
>>
>>
>> Maybe it *is* winSCP, after all?
>>
>>
>
> Or maybe the winSCP settings?
>
>
> http://winscp.net/eng/docs/ui_login_environment
>
>
> --
> Earnie

Earnie-

Yes, I have looked into the WinSCP settings.  I've tried them all.  I am
using the setting:

        "Select Adjust remote timestamp to local conventions, if the
server behaves correctly and does not adjust file timestamp
(typically Unix-based servers and Windows 7 and newer)"

because (1) it is the one recommended for Unix servers, and (2) it is the
only one that works.

Regardless, once the file is copied, the time (as reported by Windows'
tools) is the correct time on the file.

So I don't believe this is a WinSCP problem, AND the problem is
demonstrated independent of WinSCP.

-FF



--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Problem with daylight saving time, off by one hour

[Earnie Boyd]

On Tue, Feb 28, 2012 at 3:24 PM, Frank Farance wrote:
>
> So I don't believe this is a WinSCP problem, AND the problem is
> demonstrated independent of WinSCP.
>

What is TZ set to in your Cygwin environment?  Try the string EST5EDT
to see if it helps.

-- 
Earnie
-- https://sites.google.com/site/earnieboyd

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Problem with daylight saving time, off by one hour

[Frank Farance]

Earnie Boyd wrote:
> On Tue, Feb 28, 2012 at 3:24 PM, Frank Farance wrote:
>
>>
>> So I don't believe this is a WinSCP problem, AND the problem is
>> demonstrated independent of WinSCP.
>>
>
> What is TZ set to in your Cygwin environment?  Try the string EST5EDT
> to see if it helps.
>
> --
> Earnie
> -- https://sites.google.com/site/earnieboyd

TZ setting is America/New_York on both machines.  Already tried setting
EST5EDT (and UTC0), all with same problem.

-FF



--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Problem with daylight saving time, off by one hour

[Frank Farance]

Corinna Vinschen wrote:
> On Feb 29 12:49, G.W. Haywood wrote:
>
>> Hi there,
>>
>>
>>> 175712 by: "Frank Farance"
>>> 175717 by: Corinna Vinschen
>>> 175719 by: "Frank Farance"
>>> 175720 by: Corinna Vinschen
>>> 175721 by: "Frank Farance"
>>> 175722 by: Corinna Vinschen
>>> 175725 by: Earnie Boyd
>>> 175728 by: "Frank Farance"
>>>
>>
>> What are the filesystems involved?  VFAT anywhere?
>>
>
> Apart from the CF cards for ny camera, not in my case, no.  NTFS-only on
> Windows and ext4-only on Linux.
>
>
>
> Corinna

NTFS on the windows system, ext3 on the Linux system.

-FF



--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Problem with daylight saving time, off by one hour

[Corinna Vinschen]

On Feb 29 12:49, G.W. Haywood wrote:
> Hi there,
> 
> >175712 by: "Frank Farance"
> >175717 by: Corinna Vinschen
> >175719 by: "Frank Farance"
> >175720 by: Corinna Vinschen
> >175721 by: "Frank Farance"
> >175722 by: Corinna Vinschen
> >175725 by: Earnie Boyd
> >175728 by: "Frank Farance"
> 
> What are the filesystems involved?  VFAT anywhere?

Apart from the CF cards for ny camera, not in my case, no.  NTFS-only on
Windows and ext4-only on Linux.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Project Co-Leader          cygwin AT cygwin DOT com
Red Hat

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Problem with daylight saving time, off by one hour

[G.W. Haywood]

Hi there,

> 175712 by: "Frank Farance"
> 175717 by: Corinna Vinschen
> 175719 by: "Frank Farance"
> 175720 by: Corinna Vinschen
> 175721 by: "Frank Farance"
> 175722 by: Corinna Vinschen
> 175725 by: Earnie Boyd
> 175728 by: "Frank Farance"

What are the filesystems involved?  VFAT anywhere?

--

73,
Ged.

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple