tzcode resync

tzcode resync

[Yaakov Selkowitz]

Our winsup/cygwin/localtime.cc is a mashup of a very old version of
upstream tzcode's private.h, tzfile.h, and localtime.c with Cygwin-
specific patches for getting the timezone from windows as a fallback,
as well as handling API compatibility with pre-tm_gmtoff binaries.

The upstream tzcode/tzdata have introduced database changes which
cannot be read by old code.  Currently, I am building tzdata in the old
("rearguard") format to keep compatibility, but at this point we're
*way* behind upstream and we really should update our code.  Given how
long it's been though, it's going to be a bit of work.

Is anyone willing to take on merging our changes into the latest
upstream code and creating a new localtime.cc?  As a bonus, if the file
layout were to be restructured to make this more easily maintainable
(e.g. by creating a winsup/cygwin/tzcode subdirectory, adding
$(srcdir)/tzcode to winsup/cygwin/Makefile.in:VPATH, and updating the
object file names), that would probably help going forward.

TIA,

--
Yaakov

Re: tzcode resync

[Corinna Vinschen]

On Apr 24 17:19, Yaakov Selkowitz wrote:
> Our winsup/cygwin/localtime.cc is a mashup of a very old version of
> upstream tzcode's private.h, tzfile.h, and localtime.c with Cygwin-
> specific patches for getting the timezone from windows as a fallback,
> as well as handling API compatibility with pre-tm_gmtoff binaries.
> 
> The upstream tzcode/tzdata have introduced database changes which
> cannot be read by old code.  Currently, I am building tzdata in the old
> ("rearguard") format to keep compatibility, but at this point we're
> *way* behind upstream and we really should update our code.  Given how
> long it's been though, it's going to be a bit of work.

When I updated localtime.cc back in 2012, I tried to keep it in a format
which allows further updates from the NetBSD version later on.

The last localtime.c update in NetBSD is from 2019.  That should be
sufficient, shouldn't it?

The upstream localtime.c diff from version 1.72 to current 1.122 is
3K lines of code, so it's still a lot of work, probably...


Thanks,
Corinna

-- 
Corinna Vinschen
Cygwin Maintainer

Re: tzcode resync

[Mark Geisert]

Hi folks,

On Mon, 27 Apr 2020, Corinna Vinschen wrote:
> On Apr 24 17:19, Yaakov Selkowitz wrote:
>> Our winsup/cygwin/localtime.cc is a mashup of a very old version of
>> upstream tzcode's private.h, tzfile.h, and localtime.c with Cygwin-
>> specific patches for getting the timezone from windows as a fallback,
>> as well as handling API compatibility with pre-tm_gmtoff binaries.
>>
>> The upstream tzcode/tzdata have introduced database changes which
>> cannot be read by old code.  Currently, I am building tzdata in the old
>> ("rearguard") format to keep compatibility, but at this point we're
>> *way* behind upstream and we really should update our code.  Given how
>> long it's been though, it's going to be a bit of work.
>
> When I updated localtime.cc back in 2012, I tried to keep it in a format
> which allows further updates from the NetBSD version later on.
>
> The last localtime.c update in NetBSD is from 2019.  That should be
> sufficient, shouldn't it?
>
> The upstream localtime.c diff from version 1.72 to current 1.122 is
> 3K lines of code, so it's still a lot of work, probably...

I can take this on.  I've looked at the current NetBSD code diffed against 
the version we use: there's a fair amount of prototype rejiggering and 
recoding of localized areas.  New code too, of course.  I've done this 
kind of thing before and this project seems doable to me.

I assume I can bring the current NetBSD code directly into Cygwin?  In a 
legal sense, I mean.  Code-wise I see what Yaakov's suggested and it seems 
like a great way to go.

The other question I have is, once I believe I've got it incorporated into 
Cygwin, what can I use to test that it's operating correctly?

..mark

Re: tzcode resync

[Corinna Vinschen]

Hi Mark,

On Apr 29 00:03, Mark Geisert wrote:
> Hi folks,
> 
> On Mon, 27 Apr 2020, Corinna Vinschen wrote:
> > On Apr 24 17:19, Yaakov Selkowitz wrote:
> > > Our winsup/cygwin/localtime.cc is a mashup of a very old version of
> > > upstream tzcode's private.h, tzfile.h, and localtime.c with Cygwin-
> > > specific patches for getting the timezone from windows as a fallback,
> > > as well as handling API compatibility with pre-tm_gmtoff binaries.
> > > 
> > > The upstream tzcode/tzdata have introduced database changes which
> > > cannot be read by old code.  Currently, I am building tzdata in the old
> > > ("rearguard") format to keep compatibility, but at this point we're
> > > *way* behind upstream and we really should update our code.  Given how
> > > long it's been though, it's going to be a bit of work.
> > 
> > When I updated localtime.cc back in 2012, I tried to keep it in a format
> > which allows further updates from the NetBSD version later on.
> > 
> > The last localtime.c update in NetBSD is from 2019.  That should be
> > sufficient, shouldn't it?
> > 
> > The upstream localtime.c diff from version 1.72 to current 1.122 is
> > 3K lines of code, so it's still a lot of work, probably...
> 
> I can take this on.  I've looked at the current NetBSD code diffed against
> the version we use: there's a fair amount of prototype rejiggering and
> recoding of localized areas.  New code too, of course.  I've done this kind
> of thing before and this project seems doable to me.

Sounds good to me.  I started with this yesterday but didn't make a lot
of progress, see below.  I'd be happy if you like to take over.

> I assume I can bring the current NetBSD code directly into Cygwin?  In a
> legal sense, I mean.

Yes, that's perfectly fine, given the BSD license.

> Code-wise I see what Yaakov's suggested and it seems
> like a great way to go.

Yesterday eve I was already looking into this and given how NetBSD
is still under CVS control, I decided trying to pull our localtime.cc
up from 1.72 to 1.122 version by version, just to be really careful.

I managed to get up to 1.83 and stumbled over a big problem in terms
of TZ data format.

Cygwin has a builtin binary representation of the file "posixrules" in
tz_posixrules.h.  Since 2000.  And unchanged since 2000!

This binary representation is used as posixrules data if the posixrules
file doesn't exist or is unreadable.  I have a vague feeling that this
data is hopelessly outdated and, probably, the format changed as well.

So I assume the data in tz_posixrules.h needs either an update, or
we drop the inline data entirely.

Dropping this inline data shouldn't be much of a problem, right?
Cygwin installations shouldn't need it and 3PPs can easily provide
a posixrules file, I guess...

Mark, Yaakov, what's your stance in terms of this posixrules data?


> The other question I have is, once I believe I've got it incorporated into
> Cygwin, what can I use to test that it's operating correctly?

Uhm... Yaakov?


Thanks,
Corinna

-- 
Corinna Vinschen
Cygwin Maintainer

Re: tzcode resync

[Mark Geisert]

Hi Corinna,

On Wed, 29 Apr 2020, Corinna Vinschen wrote:
> On Apr 29 00:03, Mark Geisert wrote:
>> On Mon, 27 Apr 2020, Corinna Vinschen wrote:
>>> On Apr 24 17:19, Yaakov Selkowitz wrote:
>>>> Our winsup/cygwin/localtime.cc is a mashup of a very old version of
>>>> upstream tzcode's private.h, tzfile.h, and localtime.c with Cygwin-
>>>> specific patches for getting the timezone from windows as a fallback,
>>>> as well as handling API compatibility with pre-tm_gmtoff binaries.
>>>>
>>>> The upstream tzcode/tzdata have introduced database changes which
>>>> cannot be read by old code.  Currently, I am building tzdata in the old
>>>> ("rearguard") format to keep compatibility, but at this point we're
>>>> *way* behind upstream and we really should update our code.  Given how
>>>> long it's been though, it's going to be a bit of work.
>>>
>>> When I updated localtime.cc back in 2012, I tried to keep it in a format
>>> which allows further updates from the NetBSD version later on.
>>>
>>> The last localtime.c update in NetBSD is from 2019.  That should be
>>> sufficient, shouldn't it?
>>>
>>> The upstream localtime.c diff from version 1.72 to current 1.122 is
>>> 3K lines of code, so it's still a lot of work, probably...
>>
>> I can take this on.  I've looked at the current NetBSD code diffed against
>> the version we use: there's a fair amount of prototype rejiggering and
>> recoding of localized areas.  New code too, of course.  I've done this kind
>> of thing before and this project seems doable to me.
>
> Sounds good to me.  I started with this yesterday but didn't make a lot
> of progress, see below.  I'd be happy if you like to take over.

Yes, please :-).

>> I assume I can bring the current NetBSD code directly into Cygwin?  In a
>> legal sense, I mean.
>
> Yes, that's perfectly fine, given the BSD license.
>
>> Code-wise I see what Yaakov's suggested and it seems
>> like a great way to go.
>
> Yesterday eve I was already looking into this and given how NetBSD
> is still under CVS control, I decided trying to pull our localtime.cc
> up from 1.72 to 1.122 version by version, just to be really careful.
>
> I managed to get up to 1.83 and stumbled over a big problem in terms
> of TZ data format.
>
> Cygwin has a builtin binary representation of the file "posixrules" in
> tz_posixrules.h.  Since 2000.  And unchanged since 2000!
>
> This binary representation is used as posixrules data if the posixrules
> file doesn't exist or is unreadable.  I have a vague feeling that this
> data is hopelessly outdated and, probably, the format changed as well.
>
> So I assume the data in tz_posixrules.h needs either an update, or
> we drop the inline data entirely.
>
> Dropping this inline data shouldn't be much of a problem, right?
> Cygwin installations shouldn't need it and 3PPs can easily provide
> a posixrules file, I guess...
>
> Mark, Yaakov, what's your stance in terms of this posixrules data?

I think this is an example of the very thing Yaakov mentioned in the 
quotes above.  Both this internal backup of posixrules and the usual 
posixrules file in /usr/share/zoneinfo are written in the obsolete Version 
0 format.  The current localtime code can only read this version.

When localtime.cc gets updated, it can read Version 2 (there was no 
Version 1 as far as I can tell).  Yaakov will generate V.2 formatted zone 
files at that point.

I can't see a reason to keep this internal backup copy, regardless of 
which Version it is.  But if we do need it, we can generate a V.2 version 
of it once localtime.cc is able to read it.

As I Understand It, for all the above...

..mark

Re: tzcode resync

[Corinna Vinschen]

On Apr 29 02:05, Mark Geisert wrote:
> On Wed, 29 Apr 2020, Corinna Vinschen wrote:
> > On Apr 29 00:03, Mark Geisert wrote:
> > > On Mon, 27 Apr 2020, Corinna Vinschen wrote:
> > > > On Apr 24 17:19, Yaakov Selkowitz wrote:
> > > > > Our winsup/cygwin/localtime.cc is a mashup of a very old version of
> > > > > upstream tzcode's private.h, tzfile.h, and localtime.c with Cygwin-
> > > > > specific patches for getting the timezone from windows as a fallback,
> > > > > as well as handling API compatibility with pre-tm_gmtoff binaries.
> > > > > 
> > > > > The upstream tzcode/tzdata have introduced database changes which
> > > > > cannot be read by old code.  Currently, I am building tzdata in the old
> > > > > ("rearguard") format to keep compatibility, but at this point we're
> > > > > *way* behind upstream and we really should update our code.  Given how
> > > > > long it's been though, it's going to be a bit of work.
> > > > [...]
> > > I can take this on.  I've looked at the current NetBSD code diffed against
> > > the version we use: there's a fair amount of prototype rejiggering and
> > > recoding of localized areas.  New code too, of course.  I've done this kind
> > > of thing before and this project seems doable to me.
> > 
> > Sounds good to me.  I started with this yesterday but didn't make a lot
> > of progress, see below.  I'd be happy if you like to take over.
> 
> Yes, please :-).
> [...]
> > [...]
> > Cygwin has a builtin binary representation of the file "posixrules" in
> > tz_posixrules.h.  Since 2000.  And unchanged since 2000!
> > 
> > This binary representation is used as posixrules data if the posixrules
> > file doesn't exist or is unreadable.  I have a vague feeling that this
> > data is hopelessly outdated and, probably, the format changed as well.
> > 
> > So I assume the data in tz_posixrules.h needs either an update, or
> > we drop the inline data entirely.
> > 
> > Dropping this inline data shouldn't be much of a problem, right?
> > Cygwin installations shouldn't need it and 3PPs can easily provide
> > a posixrules file, I guess...
> > 
> > Mark, Yaakov, what's your stance in terms of this posixrules data?
> 
> I think this is an example of the very thing Yaakov mentioned in the quotes
> above.  Both this internal backup of posixrules and the usual posixrules
> file in /usr/share/zoneinfo are written in the obsolete Version 0 format.
> The current localtime code can only read this version.
> 
> When localtime.cc gets updated, it can read Version 2 (there was no Version
> 1 as far as I can tell).  Yaakov will generate V.2 formatted zone files at
> that point.
> 
> I can't see a reason to keep this internal backup copy, regardless of which
> Version it is.  But if we do need it, we can generate a V.2 version of it
> once localtime.cc is able to read it.

Yeah, I guess that's the way to go.  The idea of all this was trying to
make sure a lone Cygwin with an executable would function outside of a
full installation.  That's also the reason we have builtin locale data.
How much sense does this make these days...?  I don't know.  It's just
a safety net for border scenarios.


Corinna

-- 
Corinna Vinschen
Cygwin Maintainer