rebase-4.4.3-1 regression: Too many DLLs for available address space

rebase-4.4.3-1 regression: Too many DLLs for available address space

[Christian Franke]

After 4.4.3-1 upgrade, rebase always fails on 32- and 64-bit Cygwin:

$ rebase -s -T /var/cache/rebase/rebase_all
rebase: Too many DLLs for available address space: Cannot allocate memory

Using /bin/rebaseall does not help.

A downgrade to 4.4.2-1 fixes the problem.

The DLL address range on 32-bit Cygwin looks sane:

$ rebase -si
/usr/libexec/coreutils/libstdbuf.so   base 0x436b0000 size 0x0000b000
/usr/lib/zsh/5.3/zsh/zutil.dll        base 0x436c0000 size 0x0000e000
...
/usr/bin/cygEMF-1.dll                 base 0x6ff80000 size 0x00034000
/usr/bin/cygEGL-1.dll                 base 0x6ffc0000 size 0x00039000


Christian

PS: rebase should possibly warn/fail if a 32bit DLL would need to be 
rebased < 0x38000000.
Recent tests on Win10 1703 show that LoadLibrary() always loads such 
DLLs to other locations.


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: rebase-4.4.3-1 regression: Too many DLLs for available address space

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 887 bytes --]

On Jan 11 13:32, Christian Franke wrote:
> After 4.4.3-1 upgrade, rebase always fails on 32- and 64-bit Cygwin:
> 
> $ rebase -s -T /var/cache/rebase/rebase_all
> rebase: Too many DLLs for available address space: Cannot allocate memory

Uh, oh, a typo.  That should have been "if (end < 0)", not "if (end <= 0)".

Thanks for noticing.  I'll create an update soon.

> PS: rebase should possibly warn/fail if a 32bit DLL would need to be rebased
> < 0x38000000.
> Recent tests on Win10 1703 show that LoadLibrary() always loads such DLLs to
> other locations.

Warn or fail?  I tend to make this a fail.  There are just too many DLLs
on 32 bit and I guess this is the future anyway for most people.


Thanks,
Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: rebase-4.4.3-1 regression: Too many DLLs for available address space

[Achim Gratz]

Corinna Vinschen writes:
>> PS: rebase should possibly warn/fail if a 32bit DLL would need to be rebased
>> < 0x38000000.
>> Recent tests on Win10 1703 show that LoadLibrary() always loads such DLLs to
>> other locations.
>
> Warn or fail?  I tend to make this a fail.  There are just too many DLLs
> on 32 bit and I guess this is the future anyway for most people.

Please make it warn unless we can be certain that the system in question
won't work correctly with a DLL based to lower addresses.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Factory and User Sound Singles for Waldorf rackAttack:
http://Synth.Stromeko.net/Downloads.html#WaldorfSounds

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: rebase-4.4.3-1 regression: Too many DLLs for available address space

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 1306 bytes --]

On Jan 11 17:06, Corinna Vinschen wrote:
> On Jan 11 13:32, Christian Franke wrote:
> > After 4.4.3-1 upgrade, rebase always fails on 32- and 64-bit Cygwin:
> > 
> > $ rebase -s -T /var/cache/rebase/rebase_all
> > rebase: Too many DLLs for available address space: Cannot allocate memory
> 
> Uh, oh, a typo.  That should have been "if (end < 0)", not "if (end <= 0)".
> 
> Thanks for noticing.  I'll create an update soon.
> 
> > PS: rebase should possibly warn/fail if a 32bit DLL would need to be rebased
> > < 0x38000000.
> > Recent tests on Win10 1703 show that LoadLibrary() always loads such DLLs to
> > other locations.
> 
> Warn or fail?  I tend to make this a fail.  There are just too many DLLs
> on 32 bit and I guess this is the future anyway for most people.

Uhm... I think we have a problem here.  I really don't think I have the
most complete 32 bit installation, far from it, but I already have 1121
DLLs with addresses down to 0x37a00000.

If Windows willy-nilly rebases DLLs with start addresses < 0x38000000,
and if rebase starts to disable addresses below 0x38000000, we're in a
bit of a fix...


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: rebase-4.4.3-1 regression: Too many DLLs for available address space

[Denis Excoffier]

> On 2018-01-11 13:32, Christian Franke wrote:
> 
> After 4.4.3-1 upgrade, rebase always fails on 32- and 64-bit Cygwin:
> 
> $ rebase -s -T /var/cache/rebase/rebase_all
> rebase: Too many DLLs for available address space: Cannot allocate memory
> 
> Using /bin/rebaseall does not help.
> 
> A downgrade to 4.4.2-1 fixes the problem.
> 
Same for me. I also tried rebase-4.4.3 and it didn't work (message in setup.log and no /etc/re* created).
Therefore i also switched back to 4.4.2.

I would like to add that under my system (Windows 7 32bits), when i autorebase everything (full),
the 'rebase -si' function (rebase-4.4.2) produces the following result:

/usr/bin/cygvpx-3.dll                         base 0x002d0000 size 0x00201000
/usr/bin/cygvpx-1.dll                         base 0x004e0000 size 0x001ed000
/usr/bin/cygvorbisfile-3.dll                  base 0x006d0000 size 0x00010000
...
/usr/bin/cygGraphicsMagick-3.dll              base 0x6ecf0000 size 0x00321000
/usr/bin/cygGraphicsMagick++-12.dll           base 0x6f020000 size 0x00052000
...
/usr/bin/Scintilla.dll                        base 0x6fc70000 size 0x000c8000
/usr/bin/SciLexer.dll                         base 0x6fd40000 size 0x002bd000
/usr/share/regina-rexx/addons/rxtest2.dll     base 0xba330000 size 0x00009000
/usr/share/regina-rexx/addons/rxtest1.dll     base 0xba340000 size 0x00009000
/usr/share/regina-rexx/addons/regutil.dll     base 0xba350000 size 0x00018000
...
/usr/bin/cygvte-9.dll                         base 0xffee0000 size 0x00099000
/usr/bin/cygvte-2.91-0.dll                    base 0xfff80000 size 0x0005f000
/usr/bin/cygvpx-4.dll                         base 0xfffe0000 size 0x002e2000

The first DLL (listed in rebase_all) is /usr/bin/SciLexer.dll, the last is
/usr/share/regina-rexx/addons/rxtest2.dll
and /usr/bin/cygvpx-4.dll follows /usr/bin/cygvpx-3.dll (as you can imagine)
with 0xfffe0000 + (0x002e2000+0xe000) = 0x002d0000 (modulo 0x100000000) as you can check
and compare with the 1st line. Is this expected, this kind of arithmetic modulo 2^32?

In any case, i have no fork problems.

The full list contains 8006 lines, i have the complete Cygwin 32bit installation
except for:
- the debuginfo packages
- the gnome, kde, lua, lxde, mate, Ocaml, Scheme, video and xfce categories (but
  including all missing dependencies of course)
- the *.mex and *.oct files from the octave packages (they use nearly 0x30000000 bytes),
  i removed them by patching /usr/bin/rebaselst

Hope this helps,

Denis Excoffier.



--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: rebase-4.4.3-1 regression: Too many DLLs for available address space

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 3053 bytes --]

On Jan 11 22:52, Denis Excoffier wrote:
> > On 2018-01-11 13:32, Christian Franke wrote:
> > 
> > After 4.4.3-1 upgrade, rebase always fails on 32- and 64-bit Cygwin:
> > 
> > $ rebase -s -T /var/cache/rebase/rebase_all
> > rebase: Too many DLLs for available address space: Cannot allocate memory
> > 
> > Using /bin/rebaseall does not help.
> > 
> > A downgrade to 4.4.2-1 fixes the problem.
> > 
> Same for me. I also tried rebase-4.4.3 and it didn't work (message in setup.log and no /etc/re* created).
> Therefore i also switched back to 4.4.2.
> 
> I would like to add that under my system (Windows 7 32bits), when i autorebase everything (full),
> the 'rebase -si' function (rebase-4.4.2) produces the following result:
> 
> /usr/bin/cygvpx-3.dll                         base 0x002d0000 size 0x00201000
> /usr/bin/cygvpx-1.dll                         base 0x004e0000 size 0x001ed000
> /usr/bin/cygvorbisfile-3.dll                  base 0x006d0000 size 0x00010000
> ...
> [...]
> /usr/bin/cygvte-9.dll                         base 0xffee0000 size 0x00099000
> /usr/bin/cygvte-2.91-0.dll                    base 0xfff80000 size 0x0005f000
> /usr/bin/cygvpx-4.dll                         base 0xfffe0000 size 0x002e2000
> [...]
> Is this expected, this kind of arithmetic modulo 2^32?

Yes, because rebase was not designed originally with a lower address
boundary, so it just dilligently computes addresses and never checks if
a boundary is crossed.  At the time when this was designed we had
hundreds, rather than thousands of DLLs, and it didn't occur any of us
that the distro would grow to these dimensions.

I pushed a patch yesterday which let rebase bail out as soon as the
address gets < 0x1000000.  That's only barely above the usual address
executables are loaded to (0x400000).  The regions beneath that are
usually used for the main thread stack (0x20000 - 0x220000) and various
helper address mappings.

I releaase rebase 4.4.4 with the above patch today.

> In any case, i have no fork problems.

That may be as it is, but if you're really running a 32 bit Windows, in
contrast to under WOW64 on a 64 bit Windows, the above does not at all
work as desired and you're just lucky.  Any DLL in the upper region
beyond 0x80000000 will be rebased at runtime by the Windows loader.

> The full list contains 8006 lines, i have the complete Cygwin 32bit installation

The bottom line of this is, and it has been said before and I can't
stress this enough, we can't support this scenario at all, for the
simple fact that we have more DLLs than fit into the 32 bit address
space.  It's not much of a problem on 64 bit, but on 32 bit it's just
not feasible anymore.

Ultimately, You should (must) not install all of Cygwin on 32 bit, only
the set of stuff you need on top of the base category.  Or install 64
bit Cygwin.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: rebase-4.4.3-1 regression: Too many DLLs for available address space

[Thomas Waldmann]

On 01/11/2018 01:32 PM, Christian Franke wrote:
> After 4.4.3-1 upgrade, rebase always fails on 32- and 64-bit Cygwin:
> 
> $ rebase -s -T /var/cache/rebase/rebase_all
> rebase: Too many DLLs for available address space: Cannot allocate memory

Confirmed, I suddenly had the same issue.

> A downgrade to 4.4.2-1 fixes the problem.

Also confirmed, thanks for the hint - 4.4.2 works fine.


-- 

GPG ID: 9F88FB52FAF7B393
GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Future of 32-bit distro (was: rebase-4.4.3-1 regression: Too many DLLs for available address space)

[Yaakov Selkowitz]

[-- Attachment #1.1: Type: text/plain, Size: 859 bytes --]

On 2018-01-12 03:13, Corinna Vinschen wrote:
> On Jan 11 22:52, Denis Excoffier wrote:
>> The full list contains 8006 lines, i have the complete Cygwin 32bit installation
> 
> The bottom line of this is, and it has been said before and I can't
> stress this enough, we can't support this scenario at all, for the
> simple fact that we have more DLLs than fit into the 32 bit address
> space.  It's not much of a problem on 64 bit, but on 32 bit it's just
> not feasible anymore.
> 
> Ultimately, You should (must) not install all of Cygwin on 32 bit, only
> the set of stuff you need on top of the base category.  Or install 64
> bit Cygwin.

If it is not possible for the entire 32-bit distribution to function as
a whole, is it time to reconsider how much we provide for 32-bit?  And
when can we just drop 32-bit entirely?

-- 
Yaakov


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: Future of 32-bit distro

[Brian Inglis]

On 2018-01-12 11:11, Yaakov Selkowitz wrote:
> On 2018-01-12 03:13, Corinna Vinschen wrote:
>> On Jan 11 22:52, Denis Excoffier wrote:
>>> The full list contains 8006 lines, i have the complete Cygwin 32bit installation
>>
>> The bottom line of this is, and it has been said before and I can't
>> stress this enough, we can't support this scenario at all, for the
>> simple fact that we have more DLLs than fit into the 32 bit address
>> space.  It's not much of a problem on 64 bit, but on 32 bit it's just
>> not feasible anymore.
>>
>> Ultimately, You should (must) not install all of Cygwin on 32 bit, only
>> the set of stuff you need on top of the base category.  Or install 64
>> bit Cygwin.
> 
> If it is not possible for the entire 32-bit distribution to function as
> a whole, is it time to reconsider how much we provide for 32-bit?  And
> when can we just drop 32-bit entirely?

When the 32 bit toolchain is available under x86_64 for cross builds and
testing, and Windows X EoLs 32 bit? It could take a long while for US and other
governments and orgs to upgrade all hardware to support 64 bit desktops.

-- 
Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Future of 32-bit distro

[Keith Christian]

I run the 32 bit Cygwin on a 64 bit Windows 7 machine and install
everything, because it has been unnecessary to waste time picking and
choosing packages even though I don't use them all.  I've been doing
this for years and suspect many other Cygwin aficionados do the same.

I asked a related question In this link:
https://cygwin.com/ml/cygwin/2016-11/msg00014.html and was offered
this command line to remove KDE and Gnome:

                setup-x86.exe -c GNOME,KDE


My first question was how to remove both KDE and Gnome which are
already installed, and I used the suggested command to successfully
remove the existing KDE and Gnome categories:

When I updated Cygwin a few weeks later, I did not know how to omit
KDE and Gnome during the install/update on the setup-x86.exe command
line so KDE and Gnome were re-installed.

For me, a final solution is the set of switches to tell setup-x86.exe
to install everything EXCEPT KDE and Gnome.

With KDE and Gnome out of the picture, I'd think there would be plenty
of space for those people who find 32 bit Cygwin perfectly
satisfactory, despite the fact that (by now, I assume) 64 bit and 32
bit have feature equality.

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Future of 32-bit distro

[Steven Penny]

On Fri, 12 Jan 2018 12:02:18, Brian Inglis wrote:
> When the 32 bit toolchain is available under x86_64 for cross builds and
> testing

this doesnt make sense. at least with Mingw-w64, this is already the case:

- mingw64-x86_64-gcc-core: 64-bit tools for building 64-bit EXEs
- mingw64-i686-gcc-core: 64-bit tools for building 32-bit EXEs

> and Windows X EoLs 32 bit? It could take a long while for US and other
> governments and orgs to upgrade all hardware to support 64 bit desktops.

no. the end users dont decide this, cygwin maintainers do. govt or whoever may
not like it, but if the DLLs dont fit in the address space then that is the
start of a losing battle. a situation which will get worse, not better. yes
corinna has some creative workarounds, but its a stopgap.

i say we should deprecate it now with a warning, then after a year or so
officially drop 32-bit support. this doesnt stop people from using cygwin for
making 32-bit EXEs as i have shown - it only stops them from running cygwin
itself as 32-bit processes.


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Future of 32-bit distro (was: rebase-4.4.3-1 regression: Too many DLLs for available address space)

[Vince Rice]

> On Jan 12, 2018, at 12:11 PM, Yaakov Selkowitz wrote:
> 
> On 2018-01-12 03:13, Corinna Vinschen wrote:
>> On Jan 11 22:52, Denis Excoffier wrote:
>>> The full list contains 8006 lines, i have the complete Cygwin 32bit installation
>> 
>> The bottom line of this is, and it has been said before and I can't
>> stress this enough, we can't support this scenario at all, for the
>> simple fact that we have more DLLs than fit into the 32 bit address
>> space.  It's not much of a problem on 64 bit, but on 32 bit it's just
>> not feasible anymore.
>> 
>> Ultimately, You should (must) not install all of Cygwin on 32 bit, only
>> the set of stuff you need on top of the base category.  Or install 64
>> bit Cygwin.
> 
> If it is not possible for the entire 32-bit distribution to function as
> a whole, is it time to reconsider how much we provide for 32-bit?  And
> when can we just drop 32-bit entirely?

I have no dog in this hunt: I'm currently 64-bit (Windows and Cygwin). Having said that…

I think this is an extreme reaction. 32-bit Windows isn't going away anytime soon (as in not in the next five and probably more like ten years). There's nothing wrong with Cygwin-32. No one* needs to install all of Cygwin (And for those running 32-bit Cygwin on a 64-bit Windows, a giant rhetorical WHY?!?!). For many (most?) of us who use a relatively small part of Cygwin, there is no issue with the 32-bit address space. (I never had a problem when I ran 32-bit, which I did through last year (on a 32-bit Windows.)

For those that install everything out of laziness, well, that's not going to work anymore (and, from reports, hasn't worked very well for quite some time). But that's no reason to punish everyone else. There's also no reason to stop creating 32-bit packages. 32-bit Windows has limited address space. This isn't new, and it isn't news. If you run out of space, you run out of space. Switch to 64-bit or do without something. But it's not a reason to force everyone (on 32-bit windows) to do without everything.


*Within three decimal points.
--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Future of 32-bit distro (was: rebase-4.4.3-1 regression: Too many DLLs for available address space)

[Keith Christian]

See my post in the other "Re: Future of 32-bit distro" thread about
using setup-x86.exe to remove KDE and GNOME.

Keith

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Future of 32-bit distro

[Brian Inglis]

On 2018-01-12 13:23, Houder wrote:
> On Fri, 12 Jan 2018 12:33:46, Keith Christian wrote:
>> I run the 32 bit Cygwin on a 64 bit Windows 7 machine and install
>> everything, because it has been unnecessary to waste time picking and
>> choosing packages even though I don't use them all.  I've been doing
>> this for years and suspect many other Cygwin aficionados do the same.
> Every package? For years?
> Have there always been that many packages? (currently more than 8000).
> Out of the more than 8000 packages, I only install a couple of hundred.

More than 10k with ~650(/800) obsolete pointing to newer replacements.
I've got over 1300 on x64 with X, plus doc, devel, debuginfo variants.

-- 
Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Future of 32-bit distro

[Keith Christian]

Brian,

Yes, I install everything.  Cygwin has so many great utilities.

I'd like to change this to remove Gnome and KDE, and run updates from
the command line with setup-x86.exe, ignoring downloading or updating
Gnome and KDE.

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Future of 32-bit distro

[David Stacey]

On 12/01/18 18:11, Yaakov Selkowitz wrote:
> If it is not possible for the entire 32-bit distribution to function as
> a whole, is it time to reconsider how much we provide for 32-bit?  And
> when can we just drop 32-bit entirely?

I suspect there are a great number of people running 32-bit Windows, 
even if the underlying hardware is capable of running 64-bit. For that 
reason, I would argue against dropping 32-bit support at the moment - 
even if it increases my work as a package maintainer.

Would it be possible for 'setup_x86.exe' to give a warning if run on a 
64-bit OS? That might help.

Dave.


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple