New Cygwin family member

New Cygwin family member

[Corinna Vinschen]

Today, at about 15:50 CET, the first 64 bit Cygwin process printing
successfully "Hello World" to the Windows console saw the light of the
day.  The process is alive and well.  cyg64win1.dll, the proud mother,
is doing fine under the circumstances.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

Re: New Cygwin family member

[Teemu Nätkinniemi]

On 5.2.2013 18:11, Corinna Vinschen wrote:
> Today, at about 15:50 CET, the first 64 bit Cygwin process printing
> successfully "Hello World" to the Windows console saw the light of the
> day.  The process is alive and well.  cyg64win1.dll, the proud mother,
> is doing fine under the circumstances.

Congratulations Corinna!

Any idea when snapshots and toolchain are available for eager early 
adopters?

This reminds of the time when I got the first look of b19.

Teemu

Re: New Cygwin family member

[Corinna Vinschen]

On Feb  5 19:14, Teemu Nätkinniemi wrote:
> On 5.2.2013 18:11, Corinna Vinschen wrote:
> >Today, at about 15:50 CET, the first 64 bit Cygwin process printing
> >successfully "Hello World" to the Windows console saw the light of the
> >day.  The process is alive and well.  cyg64win1.dll, the proud mother,
> >is doing fine under the circumstances.
> 
> Congratulations Corinna!
> 
> Any idea when snapshots and toolchain are available for eager early
> adopters?

Not at all.  For a start I'm planning to provide the toolchain patches
and a binary cross toolchain x86_64-pc-linux -> x86_64-pc-cygwin this
month.

Right now I'm just building simple test applications and there's still
important stuff broken, probably even in the toolchain.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

Re: New Cygwin family member

[Chris Sutcliffe]

On 5 February 2013 12:57, Corinna Vinschen wrote:
> On Feb  5 19:14, Teemu Nätkinniemi wrote:
>> On 5.2.2013 18:11, Corinna Vinschen wrote:
>> >Today, at about 15:50 CET, the first 64 bit Cygwin process printing
>> >successfully "Hello World" to the Windows console saw the light of the
>> >day.  The process is alive and well.  cyg64win1.dll, the proud mother,
>> >is doing fine under the circumstances.
>>
>> Congratulations Corinna!
>>
>> Any idea when snapshots and toolchain are available for eager early
>> adopters?
>
> Not at all.  For a start I'm planning to provide the toolchain patches
> and a binary cross toolchain x86_64-pc-linux -> x86_64-pc-cygwin this
> month.

For us poor schmucks stuck in the 32-bit world, I'm hoping there will
eventually be a 32-to-64 bit native cross compiler (i.e. a compiler
that will run in 32-bit Cygwin that produces 64-bit Cygwin binaries)?
I'm guessing we're a ways out from that point yet, but when we get to
the point where Cygwin is officially offered in both architectures,
will the ask be of the various maintainers to produce binaries for
both (I personally don't mind as long as the tools exist to do it).

Cheers,

Chris

--
Chris Sutcliffe
http://emergedesktop.org
http://www.google.com/profiles/ir0nh34d

Re: New Cygwin family member

[Yaakov]

On Tue, 5 Feb 2013 21:04:11 -0500, Chris Sutcliffe wrote:
> On 5 February 2013 12:57, Corinna Vinschen wrote:
> > Not at all.  For a start I'm planning to provide the toolchain patches
> > and a binary cross toolchain x86_64-pc-linux -> x86_64-pc-cygwin this
> > month.
> 
> For us poor schmucks stuck in the 32-bit world, I'm hoping there will
> eventually be a 32-to-64 bit native cross compiler (i.e. a compiler
> that will run in 32-bit Cygwin that produces 64-bit Cygwin binaries)?

That will be possible as soon as the patches are released.

> I'm guessing we're a ways out from that point yet, but when we get to
> the point where Cygwin is officially offered in both architectures,
> will the ask be of the various maintainers to produce binaries for
> both (I personally don't mind as long as the tools exist to do it).

The problem is that not *everything* can be cross-compiled easily, if
at all.  Linux distributions generally use dedicated build
machines/farms (e.g. koji for Fedora) for this very reason.  This will
certainly need to be addressed when the time comes.


Yaakov

Re: New Cygwin family member

[Corinna Vinschen]

On Feb  5 22:14, Yaakov wrote:
> On Tue, 5 Feb 2013 21:04:11 -0500, Chris Sutcliffe wrote:
> > On 5 February 2013 12:57, Corinna Vinschen wrote:
> > > Not at all.  For a start I'm planning to provide the toolchain patches
> > > and a binary cross toolchain x86_64-pc-linux -> x86_64-pc-cygwin this
> > > month.
> > 
> > For us poor schmucks stuck in the 32-bit world, I'm hoping there will
> > eventually be a 32-to-64 bit native cross compiler (i.e. a compiler
> > that will run in 32-bit Cygwin that produces 64-bit Cygwin binaries)?
> 
> That will be possible as soon as the patches are released.

The patches will be relative to current binutils CVS / gcc SVN.  I'd
love to publish the patches, but just yesterday we encountered a big
problem in terms of the auto-import mechanism which I really hope to fix
before.

Without going into too much detail, the problem is that gcc references
external variables using pc-relative addressing with 32 bit displacement.
Only if a symbol is marked as dllimport, gcc will instead generate
indirect addressing opcodes.  So, if a source uses some external
variable from a DLL, but neglects to declare it as dllimport symbol,
the following will happen:

   extern char *optarg;
   char *o = optarg;

This code will be translated into something along these lines:

   movq optarg(%rip), %rax

The "optarg" here is a signed 32 bit displacement.  However, if the
offset between executable and the exporting DLL (in this case the cygwin
DLL) is bigger than 32 bit, which is *pretty* likely, given that we
want to place our DLLs where Windows will not collide with us.

The end result is either a SEGV, if the resulting address is outside of
readable virtual mem, or an access to the wrong data.

The easy solution is to stick to the current methods and play the
"serves them right" card if they didn't declare the variable as
dllimport.  However, from the POSIX POV the application didn't do
anything wrong.  And apart from the fact that a SEGV in this seemingly
innocent situation is pretty mind-boggling, it's also quite annoying to
have working 32 bit code but crashing 64 bit code generated from the
same source.

That's why I hope we can come up with a better solution.

If anybody is interested in the patches in the current state, I can
send them to the list anyway, though.

> > I'm guessing we're a ways out from that point yet, but when we get to
> > the point where Cygwin is officially offered in both architectures,
> > will the ask be of the various maintainers to produce binaries for
> > both (I personally don't mind as long as the tools exist to do it).
> 
> The problem is that not *everything* can be cross-compiled easily, if
> at all.  Linux distributions generally use dedicated build
> machines/farms (e.g. koji for Fedora) for this very reason.  This will
> certainly need to be addressed when the time comes.

Yeah.  This will still be a heck of a lot of work.  Given that "Hello
World" is barely a day old, you may understand that I didn't even dare
to call fork yet :}

Another really big problem is entirely unsolved yet:  32 and 64 bit
Cygwin processes execing each other, as well as working together.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

Re: New Cygwin family member

[Corinna Vinschen]

On Feb  6 12:20, Corinna Vinschen wrote:
> On Feb  5 22:14, Yaakov wrote:
> > On Tue, 5 Feb 2013 21:04:11 -0500, Chris Sutcliffe wrote:
> > > I'm guessing we're a ways out from that point yet, but when we get to
> > > the point where Cygwin is officially offered in both architectures,
> > > will the ask be of the various maintainers to produce binaries for
> > > both (I personally don't mind as long as the tools exist to do it).
> > 
> > The problem is that not *everything* can be cross-compiled easily, if
> > at all.  Linux distributions generally use dedicated build
> > machines/farms (e.g. koji for Fedora) for this very reason.  This will
> > certainly need to be addressed when the time comes.
> 
> Yeah.  This will still be a heck of a lot of work.  Given that "Hello
> World" is barely a day old, you may understand that I didn't even dare
> to call fork yet :}

Yay!  I have a running dash and fork works!  <at this point, imagine
the *plop* sound of opening a bottle of finest champagne>

> Another really big problem is entirely unsolved yet:  32 and 64 bit
> Cygwin processes execing each other, as well as working together.

Another problem is the memory layout (the "fork" problem).  Fortunately
we have a 43 bit address space now.  Here's is our current model, still
open for discussion:

- Keep clear of memory from 0x0 up to 0x0:7fffffff, and from
  0x700:00000000 up to 0x7ff:ffffffff since these areas are used
  by the OS for ... everything.

- Cygwin thread stacks will be located between 0x0:80000000 and
  0x0:ffffffff.  With a default stack size of 1 Meg, we have room for
  2048 threads.

- The Cygwin executables will be loaded to 0x1:00040000.

- The Cygwin DLL will be loaded to 0x1:80040000.  That leaves 2 Gigs of
  space for the executable.  The space from 0x1:8000000 up to
  0x1:80040000 will be used for Cygwin's shared memory areas.

- Other Cygwin distro DLLs are supposed to be rebased to the area
  between 0x2:00000000 up to 0x3:ffffffff.  This is an area of 8 Gigs
  for DLLs.  That should be enough for a while, I guess.

- The heap will be located at 0x4:00000000, and it will be 512Megs by
  default.

- Then ... a big void ...

- Eventually, mmap's will be allocated from 0x700:00000000 downwards.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

Re: New Cygwin family member

[Corinna Vinschen]

On Feb  8 14:55, Corinna Vinschen wrote:
> On Feb  6 12:20, Corinna Vinschen wrote:
> > On Feb  5 22:14, Yaakov wrote:
> > > On Tue, 5 Feb 2013 21:04:11 -0500, Chris Sutcliffe wrote:
> > > > I'm guessing we're a ways out from that point yet, but when we get to
> > > > the point where Cygwin is officially offered in both architectures,
> > > > will the ask be of the various maintainers to produce binaries for
> > > > both (I personally don't mind as long as the tools exist to do it).
> > > 
> > > The problem is that not *everything* can be cross-compiled easily, if
> > > at all.  Linux distributions generally use dedicated build
> > > machines/farms (e.g. koji for Fedora) for this very reason.  This will
> > > certainly need to be addressed when the time comes.
> > 
> > Yeah.  This will still be a heck of a lot of work.  Given that "Hello
> > World" is barely a day old, you may understand that I didn't even dare
> > to call fork yet :}
> 
> Yay!  I have a running dash and fork works!  <at this point, imagine
> the *plop* sound of opening a bottle of finest champagne>
> 
> > Another really big problem is entirely unsolved yet:  32 and 64 bit
> > Cygwin processes execing each other, as well as working together.
> 
> Another problem is the memory layout (the "fork" problem).  Fortunately
> we have a 43 bit address space now.  Here's is our current model, still
> open for discussion:
> 
> - Keep clear of memory from 0x0 up to 0x0:7fffffff, and from
>   0x700:00000000 up to 0x7ff:ffffffff since these areas are used
>   by the OS for ... everything.
> 
> - Cygwin thread stacks will be located between 0x0:80000000 and
>   0x0:ffffffff.  With a default stack size of 1 Meg, we have room for
>   2048 threads.
> 
> - The Cygwin executables will be loaded to 0x1:00040000.

Correction:  Not 0x1:0004.0000 but 0x1:0040.0000  Exactly the old
default start address for executables + 4 Gigs.

> - The Cygwin DLL will be loaded to 0x1:80040000.  That leaves 2 Gigs of
>   space for the executable.  The space from 0x1:8000000 up to
>   0x1:80040000 will be used for Cygwin's shared memory areas.
> 
> - Other Cygwin distro DLLs are supposed to be rebased to the area
>   between 0x2:00000000 up to 0x3:ffffffff.  This is an area of 8 Gigs
>   for DLLs.  That should be enough for a while, I guess.
> 
> - The heap will be located at 0x4:00000000, and it will be 512Megs by
>   default.
> 
> - Then ... a big void ...
> 
> - Eventually, mmap's will be allocated from 0x700:00000000 downwards.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

Re: New Cygwin family member

[Corinna Vinschen]

Hi guys,

On Feb  8 14:55, Corinna Vinschen wrote:
> Another problem is the memory layout (the "fork" problem).  Fortunately
> we have a 43 bit address space now.  Here's is our current model, still
> open for discussion:
> 
> - Keep clear of memory from 0x0 up to 0x0:7fffffff, and from
>   0x700:00000000 up to 0x7ff:ffffffff since these areas are used
>   by the OS for ... everything.
> 
> - Cygwin thread stacks will be located between 0x0:80000000 and
>   0x0:ffffffff.  With a default stack size of 1 Meg, we have room for
>   2048 threads.
> 
> - The Cygwin executables will be loaded to 0x1:00400000.
> 
> - The Cygwin DLL will be loaded to 0x1:80040000.  That leaves 2 Gigs of
>   space for the executable.  The space from 0x1:8000000 up to
>   0x1:80040000 will be used for Cygwin's shared memory areas.

A slight change to the  memory model occured to me, which should drop
the potential DLL collisions even more:

> - Other Cygwin distro DLLs are supposed to be rebased to the area
>   between 0x2:00000000 up to 0x3:ffffffff.  This is an area of 8 Gigs
>   for DLLs.  That should be enough for a while, I guess.

The memory slot from 0x2:00000000 up to 0x4:00000000 == 8 Gigs will be
used by rebase{all}...

...while ld's --auto-image-base will use the region from 0x4:00000000 up
to 0x6:00000000.

This means, rebased DLLs will never collide with auto-image-based DLLs,
and both types of DLLs have a full 8 Gigs space each, which should be
enough for the forseeable future.

> - The heap will be located at 0x4:00000000, and it will be 512Megs by
>   default.

The heap will then be located at 0x6:00000000, of course.

> - Then ... a big void ...
> 
> - Eventually, mmap's will be allocated from 0x700:00000000 downwards.

This means the VM space left for heap and mmaps is a mere 7152 Gigabyte,
about 7 Terabytes.  I hope that's ok, even for our friends on the
Fortran frontier.

Comments?


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

Re: New Cygwin family member

[Ryan Johnson]

On 12/02/2013 6:00 AM, Corinna Vinschen wrote:
> Hi guys,
>
> On Feb  8 14:55, Corinna Vinschen wrote:
>> Another problem is the memory layout (the "fork" problem).  Fortunately
>> we have a 43 bit address space now.  Here's is our current model, still
>> open for discussion:
>>
>> - Keep clear of memory from 0x0 up to 0x0:7fffffff, and from
>>    0x700:00000000 up to 0x7ff:ffffffff since these areas are used
>>    by the OS for ... everything.
>>
>> - Cygwin thread stacks will be located between 0x0:80000000 and
>>    0x0:ffffffff.  With a default stack size of 1 Meg, we have room for
>>    2048 threads.
>>
>> - The Cygwin executables will be loaded to 0x1:00400000.
>>
>> - The Cygwin DLL will be loaded to 0x1:80040000.  That leaves 2 Gigs of
>>    space for the executable.  The space from 0x1:8000000 up to
>>    0x1:80040000 will be used for Cygwin's shared memory areas.
> A slight change to the  memory model occured to me, which should drop
> the potential DLL collisions even more:
>
>> - Other Cygwin distro DLLs are supposed to be rebased to the area
>>    between 0x2:00000000 up to 0x3:ffffffff.  This is an area of 8 Gigs
>>    for DLLs.  That should be enough for a while, I guess.
> The memory slot from 0x2:00000000 up to 0x4:00000000 == 8 Gigs will be
> used by rebase{all}...
>
> ...while ld's --auto-image-base will use the region from 0x4:00000000 up
> to 0x6:00000000.
>
> This means, rebased DLLs will never collide with auto-image-based DLLs,
> and both types of DLLs have a full 8 Gigs space each, which should be
> enough for the forseeable future.
>
>> - The heap will be located at 0x4:00000000, and it will be 512Megs by
>>    default.
> The heap will then be located at 0x6:00000000, of course.
>
>> - Then ... a big void ...
>>
>> - Eventually, mmap's will be allocated from 0x700:00000000 downwards.
> This means the VM space left for heap and mmaps is a mere 7152 Gigabyte,
> about 7 Terabytes.  I hope that's ok, even for our friends on the
> Fortran frontier.
>
> Comments?
If I count correctly:
- Total address space available is 8TB
- Windows no-man's land regions occupy 2GB and 1TB, respectively
- Cygwin thread stacks occupy 2GB
- Executable image gets 2GB
- distro and auto-image dlls together occupy 16GB (including some tens 
of MB for the cygwin dll and shared memory areas)
- heap/mmap area gets from 0x6: to 0x700:, which I compute as 7144GB

In other words, the proposed layout occupies only 20GB, or less than 
0.3% of the address space. If somebody wants significantly more heap 
space, they'd have to brave the 1TB no-man's land and risk fork 
failures. Or, more likely, just wait for Intel and MS to bump the 
address space limit by a few powers of two (which should happen roughly 
when the "normal" amount of memory to have in a Windows box approaches 
8TB). At that point, the heap/mmap area can easily take over the 
additional space.

What's not to like?
Ryan

Re: New Cygwin family member

[Corinna Vinschen]

On Feb 12 08:37, Ryan Johnson wrote:
> On 12/02/2013 6:00 AM, Corinna Vinschen wrote:
> >Hi guys,
> >
> >On Feb  8 14:55, Corinna Vinschen wrote:
> >>Another problem is the memory layout (the "fork" problem).  Fortunately
> >>we have a 43 bit address space now.  Here's is our current model, still
> >>open for discussion:
> >>
> >>- Keep clear of memory from 0x0 up to 0x0:7fffffff, and from
> >>   0x700:00000000 up to 0x7ff:ffffffff since these areas are used
> >>   by the OS for ... everything.
> >>
> >>- Cygwin thread stacks will be located between 0x0:80000000 and
> >>   0x0:ffffffff.  With a default stack size of 1 Meg, we have room for
> >>   2048 threads.
> >>
> >>- The Cygwin executables will be loaded to 0x1:00400000.
> >>
> >>- The Cygwin DLL will be loaded to 0x1:80040000.  That leaves 2 Gigs of
> >>   space for the executable.  The space from 0x1:8000000 up to
> >>   0x1:80040000 will be used for Cygwin's shared memory areas.
> >A slight change to the  memory model occured to me, which should drop
> >the potential DLL collisions even more:
> >
> >>- Other Cygwin distro DLLs are supposed to be rebased to the area
> >>   between 0x2:00000000 up to 0x3:ffffffff.  This is an area of 8 Gigs
> >>   for DLLs.  That should be enough for a while, I guess.
> >The memory slot from 0x2:00000000 up to 0x4:00000000 == 8 Gigs will be
> >used by rebase{all}...
> >
> >...while ld's --auto-image-base will use the region from 0x4:00000000 up
> >to 0x6:00000000.
> >
> >This means, rebased DLLs will never collide with auto-image-based DLLs,
> >and both types of DLLs have a full 8 Gigs space each, which should be
> >enough for the forseeable future.
> >
> >>- The heap will be located at 0x4:00000000, and it will be 512Megs by
> >>   default.
> >The heap will then be located at 0x6:00000000, of course.
> >
> >>- Then ... a big void ...
> >>
> >>- Eventually, mmap's will be allocated from 0x700:00000000 downwards.
> >This means the VM space left for heap and mmaps is a mere 7152 Gigabyte,
> >about 7 Terabytes.  I hope that's ok, even for our friends on the
> >Fortran frontier.
> >
> >Comments?
> If I count correctly:
> - Total address space available is 8TB
> - Windows no-man's land regions occupy 2GB and 1TB, respectively
> - Cygwin thread stacks occupy 2GB
> - Executable image gets 2GB
> - distro and auto-image dlls together occupy 16GB (including some
> tens of MB for the cygwin dll and shared memory areas)

Not quite :)  You missed the fact that the Cygwin DLL is located
outside that region, at 0x1:80000000.  This includes Cygwin's
shared memory and allows a cygheap size of about 2 GiB.  Still,
that doesn't reduce the heap/mmap size at all, so there's still
7144 GB left.

> - heap/mmap area gets from 0x6: to 0x700:, which I compute as 7144GB
> 
> In other words, the proposed layout occupies only 20GB, or less than
> 0.3% of the address space. If somebody wants significantly more heap
> space, they'd have to brave the 1TB no-man's land and risk fork
> failures. Or, more likely, just wait for Intel and MS to bump the
> address space limit by a few powers of two (which should happen
> roughly when the "normal" amount of memory to have in a Windows box
> approaches 8TB). At that point, the heap/mmap area can easily take
> over the additional space.
> 
> What's not to like?
> Ryan

Yeah, that doesn't sound overly cramped, does it?


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

Re: New Cygwin family member

[Yaakov]

On Tue, 5 Feb 2013 18:57:28 +0100, Corinna Vinschen wrote:
> Not at all.  For a start I'm planning to provide the toolchain patches
> and a binary cross toolchain x86_64-pc-linux -> x86_64-pc-cygwin this
> month.

Congratulations, and nice work!  Please let me know if I can assist with
the new toolchains.


Yaakov

Re: New Cygwin family member

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 555 bytes --]

On 2/5/2013 8:11 AM, Corinna Vinschen wrote:
> Today, at about 15:50 CET, the first 64 bit Cygwin process printing
> successfully "Hello World" to the Windows console saw the light of the
> day.  The process is alive and well.  cyg64win1.dll, the proud mother,
> is doing fine under the circumstances.

Awesome! Is it too early to ask about performance? I'm interested in whether
Cygwin's running under WOW64 incurs much of a performance hit, and before now,
an apples-to-apples comparison with native process execution hasn't been possible.



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

Re: New Cygwin family member

[Corinna Vinschen]

On Feb  5 18:06, Daniel Colascione wrote:
> On 2/5/2013 8:11 AM, Corinna Vinschen wrote:
> > Today, at about 15:50 CET, the first 64 bit Cygwin process printing
> > successfully "Hello World" to the Windows console saw the light of the
> > day.  The process is alive and well.  cyg64win1.dll, the proud mother,
> > is doing fine under the circumstances.
> 
> Awesome! Is it too early to ask about performance?

Yes, very definitely!


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

Re: New Cygwin family member

[Larry Hall (Cygwin Developers)]

On 2/5/2013 11:11 AM, Corinna Vinschen wrote:
> Today, at about 15:50 CET, the first 64 bit Cygwin process printing
> successfully "Hello World" to the Windows console saw the light of the
> day.  The process is alive and well.  cyg64win1.dll, the proud mother,
> is doing fine under the circumstances.

And it must have all 64-digits!  Nice. :-)


-- 
Larry

Re: New Cygwin family member - And Big Thank You!

[Herbert Stocker]

Welcome from Munich, to this new citizen of the computing world.
Does it already have a name? Like "cyg64.exe".

Actually i really appreciate this work, Corinna.
Like i do appreciate the whole Cygwin project. A computer would not be 
complete
if it wouldn't have a command line interface with a full-featured 
command language
like bash provides.

Big Appreciation to all the programmers, testers and writers of the 
Cygwin project
from me!

Herbert



On 05.02.2013 17:11, Corinna Vinschen wrote:
> Today, at about 15:50 CET, the first 64 bit Cygwin process printing
> successfully "Hello World" to the Windows console saw the light of the
> day.  The process is alive and well.  cyg64win1.dll, the proud mother,
> is doing fine under the circumstances.
>
>
> Corinna
>