[JAVA,libtool] Big libjava is biiiig.

[JAVA,libtool] Big libjava is biiiig.

[Dave Korn]

    Hi,

  As I'm sure everyone concerned is aware, libgcj is currently a bit of a
monolith.  Wighing in at 93M for a static archive, 73M for a shared library
(win32), it exports 82720 symbols.  Which is unfortunately 17184 more than the
system limit (64k) for a Windows DLL.

  The idea of breaking libjava up into smaller sublibraries has been mooted at
least a couple of times before (e.g. [*], [**]), but it's always raised issues
relating to backward compatibility.

  On windows we have no such back-compat issues to worry about; libjava has
not worked as a DLL in who-knows-how-long-if-ever.  I envisage that we could
very easily break it up into a bunch of separate (but presumably quite
inter-dependent) DLLs, and as a convenience we could provide a 'top-level'
libjava import library[***] that merged all the import libraries for the
individual DLLs.

  So I'm currently experimenting with a patch that adds a new option
"--enable-libgcj-sublibs" in libjava/configure.ac.  I may need to add a
dummy-link-and-relink stage to get the interdependencies working right, or I
might have to hack something in libtool, but the basic approach of adding a
bunch of extra libtool declarations based on $(filter)ing the full list of
dependencies from the complete libgcj_la_LIBADD definition seemed a reasonable
way to go:

+if BUILD_SUBLIBS
+libgcj_gnu_la_LIBADD = $(filter gnu/%.lo,$(libgcj_la_LIBADD)) -L$(here)/.libs
libgcj.la
+libgcj_java_la_LIBADD = $(filter java/%.lo,$(libgcj_la_LIBADD))
-L$(here)/.libs libgcj.la
+libgcj_javax_la_LIBADD = $(filter javax/%.lo,$(libgcj_la_LIBADD))
-L$(here)/.libs libgcj.la
+libgcj_misc_la_LIBADD = $(filter-out gnu/%.lo java/%.lo
javax/%.lo,$(libgcj_la_LIBADD)) -L$(here)/.libs libgcj.la
+endif

  Questions:

1)  Would this be a reasonable approach, specifically i) in adding a configure
option to cause sublibraries to be built, and ii) in using gmake's $(filter)
construct to crudely subdivide the libraries like this?

2)  Given that there's a bit of a logjam upstream, and not likely to be
another libtool release betwen now and the end of stage1, would it be
acceptable (in general) to hack on our in-tree libtool first and send patches
upstream second (thus still avoiding any potential future merge lossage)?


    cheers,
      DaveK
-- 
[*]   - http://gcc.gnu.org/ml/gcc/2005-04/threads.html#01450
[**]  - http://gcc.gnu.org/ml/java-patches/2005-q1/threads.html#00225
[***] - For those not familiar, when windows executables import symbols from
DLLs, they do so by statically linking against a so-named 'import library'
that contains .data section stubs that build the structures that constitute
the final exe's table of imports as understood by the runtime loader.

Re: [JAVA,libtool] Big libjava is biiiig.

[Dave Korn]

[ Boh!  I allowed my emailer to autocomplete the address and misdirected this
to the -patches list.  Apologies for the reposting to gcc@, but it'll break
the threading if I just send a forward to java@. ]


    Hi,

  As I'm sure everyone concerned is aware, libgcj is currently a bit of a
monolith.  Wighing in at 93M for a static archive, 73M for a shared library
(win32), it exports 82720 symbols.  Which is unfortunately 17184 more than the
system limit (64k) for a Windows DLL.

  The idea of breaking libjava up into smaller sublibraries has been mooted at
least a couple of times before (e.g. [*], [**]), but it's always raised issues
relating to backward compatibility.

  On windows we have no such back-compat issues to worry about; libjava has
not worked as a DLL in who-knows-how-long-if-ever.  I envisage that we could
very easily break it up into a bunch of separate (but presumably quite
inter-dependent) DLLs, and as a convenience we could provide a 'top-level'
libjava import library[***] that merged all the import libraries for the
individual DLLs.

  So I'm currently experimenting with a patch that adds a new option
"--enable-libgcj-sublibs" in libjava/configure.ac.  I may need to add a
dummy-link-and-relink stage to get the interdependencies working right, or I
might have to hack something in libtool, but the basic approach of adding a
bunch of extra libtool declarations based on $(filter)ing the full list of
dependencies from the complete libgcj_la_LIBADD definition seemed a reasonable
way to go:

+if BUILD_SUBLIBS
+libgcj_gnu_la_LIBADD = $(filter gnu/%.lo,$(libgcj_la_LIBADD)) -L$(here)/.libs
libgcj.la
+libgcj_java_la_LIBADD = $(filter java/%.lo,$(libgcj_la_LIBADD))
-L$(here)/.libs libgcj.la
+libgcj_javax_la_LIBADD = $(filter javax/%.lo,$(libgcj_la_LIBADD))
-L$(here)/.libs libgcj.la
+libgcj_misc_la_LIBADD = $(filter-out gnu/%.lo java/%.lo
javax/%.lo,$(libgcj_la_LIBADD)) -L$(here)/.libs libgcj.la
+endif

  Questions:

1)  Would this be a reasonable approach, specifically i) in adding a configure
option to cause sublibraries to be built, and ii) in using gmake's $(filter)
construct to crudely subdivide the libraries like this?

2)  Given that there's a bit of a logjam upstream, and not likely to be
another libtool release betwen now and the end of stage1, would it be
acceptable (in general) to hack on our in-tree libtool first and send patches
upstream second (thus still avoiding any potential future merge lossage)?


    cheers,
      DaveK
-- 
[*]   - http://gcc.gnu.org/ml/gcc/2005-04/threads.html#01450
[**]  - http://gcc.gnu.org/ml/java-patches/2005-q1/threads.html#00225
[***] - For those not familiar, when windows executables import symbols from
DLLs, they do so by statically linking against a so-named 'import library'
that contains .data section stubs that build the structures that constitute
the final exe's table of imports as understood by the runtime loader.

Re: [JAVA,libtool] Big libjava is biiiig.

[Ralf Wildenhues]

Hello Dave,

* Dave Korn wrote on Wed, May 06, 2009 at 06:09:05PM CEST:
> 
>   So I'm currently experimenting with a patch that adds a new option
> "--enable-libgcj-sublibs" in libjava/configure.ac.  I may need to add a
> dummy-link-and-relink stage to get the interdependencies working right, or I
> might have to hack something in libtool, but the basic approach of adding a
> bunch of extra libtool declarations based on $(filter)ing the full list of
> dependencies from the complete libgcj_la_LIBADD definition seemed a reasonable
> way to go:
> 
> +if BUILD_SUBLIBS
> +libgcj_gnu_la_LIBADD = $(filter gnu/%.lo,$(libgcj_la_LIBADD)) -L$(here)/.libs
> libgcj.la
> +libgcj_java_la_LIBADD = $(filter java/%.lo,$(libgcj_la_LIBADD))
> -L$(here)/.libs libgcj.la
> +libgcj_javax_la_LIBADD = $(filter javax/%.lo,$(libgcj_la_LIBADD))
> -L$(here)/.libs libgcj.la
> +libgcj_misc_la_LIBADD = $(filter-out gnu/%.lo java/%.lo
> javax/%.lo,$(libgcj_la_LIBADD)) -L$(here)/.libs libgcj.la
> +endif

I don't yet see why you would need any kind of libtool hacking.

I also don't see why the GCC tree *ever* uses -L$(something)/.libs in
directories where libtool is used.  Just specify in-tree library
dependencies as 'libgcj.la' and automake and libtool will do The Right
Thing.  You adding an explicit -L.libs can only lead to installed .la
files retaining paths to build tree locations.

> 1)  Would this be a reasonable approach, specifically i) in adding a configure
> option to cause sublibraries to be built, and ii) in using gmake's $(filter)
> construct to crudely subdivide the libraries like this?

You are aware of the fact that it is part of the ABI in which of the
linked DLLs a given symbol was found, and any shuffling of that later
will break that ABI?

You also have to ensure that the sub libraries are self-contained, or at
least their interdependencies form a directed non-cyclic graph (or you
will need very ugly hacks on w32).

> 2)  Given that there's a bit of a logjam upstream, and not likely to be
> another libtool release betwen now and the end of stage1,

What makes you think so?  There is a backlog of some patches, but that
doesn't mean
- important patches won't go in,
- GCC can't update to a newer unreleased Libtool version; it has done
  this several times, and it does not require the GCC developers to
  upgrade their build tools.

> would it be
> acceptable (in general) to hack on our in-tree libtool first and send patches
> upstream second (thus still avoiding any potential future merge lossage)?

I don't see a reason for that in your message.  Do you have other
reasons for suggesting this?

Cheers,
Ralf

Re: [JAVA,libtool] Big libjava is biiiig.

[David Daney]

Ralf Wildenhues wrote:
> Hello Dave,
> 
> * Dave Korn wrote on Wed, May 06, 2009 at 06:09:05PM CEST:
[...]
>> 1)  Would this be a reasonable approach, specifically i) in adding a configure
>> option to cause sublibraries to be built, and ii) in using gmake's $(filter)
>> construct to crudely subdivide the libraries like this?
> 
> You are aware of the fact that it is part of the ABI in which of the
> linked DLLs a given symbol was found, and any shuffling of that later
> will break that ABI?
> 
> You also have to ensure that the sub libraries are self-contained, or at
> least their interdependencies form a directed non-cyclic graph (or you
> will need very ugly hacks on w32).
> 

Unfortunately it may not be a simple task to find a suitably large set 
of packages that satisfy this 'directed non-cyclic graph' criterion.

I might suggest looking at grouping a bunch of various protocol handlers 
together that are all accessed via mechanisms like the URLConnection, 
and the various crypto implementations.


David Daney

Re: [JAVA,libtool] Big libjava is biiiig.

[Richard Guenther]

On Wed, May 6, 2009 at 6:26 PM, David Daney <ddaney@caviumnetworks.com> wrote:
> Ralf Wildenhues wrote:
>>
>> Hello Dave,
>>
>> * Dave Korn wrote on Wed, May 06, 2009 at 06:09:05PM CEST:
>
> [...]
>>>
>>> 1)  Would this be a reasonable approach, specifically i) in adding a
>>> configure
>>> option to cause sublibraries to be built, and ii) in using gmake's
>>> $(filter)
>>> construct to crudely subdivide the libraries like this?
>>
>> You are aware of the fact that it is part of the ABI in which of the
>> linked DLLs a given symbol was found, and any shuffling of that later
>> will break that ABI?
>>
>> You also have to ensure that the sub libraries are self-contained, or at
>> least their interdependencies form a directed non-cyclic graph (or you
>> will need very ugly hacks on w32).
>>
>
> Unfortunately it may not be a simple task to find a suitably large set of
> packages that satisfy this 'directed non-cyclic graph' criterion.
>
> I might suggest looking at grouping a bunch of various protocol handlers
> together that are all accessed via mechanisms like the URLConnection, and
> the various crypto implementations.

Is it not that maybe most of the exported symbols are not necessary and can
be made hidden?

Richard.

Re: [JAVA,libtool] Big libjava is biiiig.

[Andrew Haley]

Dave Korn wrote:

> 1)  Would this be a reasonable approach, specifically i) in adding a configure
> option to cause sublibraries to be built, and ii) in using gmake's $(filter)
> construct to crudely subdivide the libraries like this?

At program startup the first library would be loaded, it would load
the next, and so on.  There are a few parts of libgcj that are truly
independent, but I suspect that you'd always load almost all of it.
So, you'd have longer startup time for loading all those files.


With regard to GNU libc platforms:

You'd no longer be able to make so much use of fast calls between
functions in the same library; you'd have to go via the PLT.

Also, dl_iterate_phdr() is used a great deal (for finding exception
regions, garbage collection, etc.) and it linearly scans the libraries
that are loaded.  So, the more libraries you have loaded, the slower
it goes.

Now, I don't know how much of these characteristics are shared by
Windows, but I imagine some of them are.

So, I suspect your best bet would be to split libgcj into core and
non-core libraries and not slice much more thinly than that.  I can
advise you what is core and what isn't.

Andrew.

Re: [JAVA,libtool] Big libjava is biiiig.

[Andrew Haley]

Richard Guenther wrote:

> Is it not that maybe most of the exported symbols are not necessary and can
> be made hidden?

We already did that.  This is the number of symbols in the public API.

Andrew.

Re: [JAVA,libtool] Big libjava is biiiig.

[Dave Korn]

Andrew Haley wrote:
> Dave Korn wrote:
> 
>> 1)  Would this be a reasonable approach, specifically i) in adding a configure
>> option to cause sublibraries to be built, and ii) in using gmake's $(filter)
>> construct to crudely subdivide the libraries like this?
> 
> At program startup the first library would be loaded, it would load
> the next, and so on.  There are a few parts of libgcj that are truly
> independent, but I suspect that you'd always load almost all of it.
> So, you'd have longer startup time for loading all those files.

  Compared to a single DLL that is unusably malformed because it exceeds
system limits, that's still an improvement :)

> With regard to GNU libc platforms:
> 
> You'd no longer be able to make so much use of fast calls between
> functions in the same library; you'd have to go via the PLT.
> 
> Also, dl_iterate_phdr() is used a great deal (for finding exception
> regions, garbage collection, etc.) and it linearly scans the libraries
> that are loaded.  So, the more libraries you have loaded, the slower
> it goes.
> 
> Now, I don't know how much of these characteristics are shared by
> Windows, but I imagine some of them are.

  Yes, the inter-library calls would have to be dllimports and go through
stubs (one extra indirect branch).  I don't know how dl_iterate_phdr works on
win32 but I imagine that it's also linear in the number of libs.

> So, I suspect your best bet would be to split libgcj into core and
> non-core libraries and not slice much more thinly than that.  I can
> advise you what is core and what isn't.

  Please do, I'll happily try that approach since it might significantly
simplify my inter-dependences problems.

    cheers,
      DaveK

Re: [JAVA,libtool] Big libjava is biiiig.

[Dave Korn]

Ralf Wildenhues wrote:
> Hello Dave,

  Hiya Ralf,

> I don't yet see why you would need any kind of libtool hacking.

  Because of this:

> You also have to ensure that the sub libraries are self-contained, or at
> least their interdependencies form a directed non-cyclic graph (or you
> will need very ugly hacks on w32).

  I fully expect there to be cyclic interdependencies, which I could resolve
by building import libs first with dlltool.  I'm not sure yet whether to do
this in libtool, but it occurred to me as one possibility.  Another approach
would have been to do it in the Makefile, by first building the sub-DLLs all
linked against the monolithic libjava DLL, then rebuilding them all but this
time linking against their own import libs generated in the previous step; it
occurred to me that even this might need a little help from libtool.

> I also don't see why the GCC tree *ever* uses -L$(something)/.libs in
> directories where libtool is used.  Just specify in-tree library
> dependencies as 'libgcj.la' and automake and libtool will do The Right
> Thing.  You adding an explicit -L.libs can only lead to installed .la
> files retaining paths to build tree locations.

  Right, I'll try it; I have no idea why it's like that, I just cut and pasted
existing examples without fully understanding.  The comment on the bit I cut
and pasted reads:

## We don't explicitly link in the libraries we need; libgcj.la brings
## in all dependencies.  We need the -L so that gcj can find libgcj
## with `-lgcj', but it must come first, otherwise the -L flags
## brought in from libgcj.la would cause the install directories to be
## searched before the build-tree ones, and we'd get errors because of
## different libraries with the same SONAME from picky linkers such as
## Solaris'.  FIXME: should be _libs on some systems.
jv_convert_LDADD = -L$(here)/.libs libgcj.la

>> 1)  Would this be a reasonable approach, specifically i) in adding a configure
>> option to cause sublibraries to be built, and ii) in using gmake's $(filter)
>> construct to crudely subdivide the libraries like this?
> 
> You are aware of the fact that it is part of the ABI in which of the
> linked DLLs a given symbol was found, and any shuffling of that later
> will break that ABI?

  Yep.  I guess I didn't think that the top-level hierarchy might be
rearranged, or objects moved from one part of the sub-tree to another; is that
likely?

>> 2)  Given that there's a bit of a logjam upstream, and not likely to be
>> another libtool release betwen now and the end of stage1,
> 
> What makes you think so?  There is a backlog of some patches, but that
> doesn't mean
> - important patches won't go in,

  Oh, great.  I didn't want to /rely/ on being able to draw on your time when
you've already got a lot to deal with, but in combination with this ...

> - GCC can't update to a newer unreleased Libtool version; it has done
>   this several times, and it does not require the GCC developers to
>   upgrade their build tools.

  (which I didn't know this was standard practice) that makes it seem far more
possible to do things the right way round.

> I don't see a reason for that in your message.  Do you have other
> reasons for suggesting this?

  Nope, I was only planning ahead against the contingency of it either not
being practicable or convenient upstream, or there being subtle problems that
transpire with doing an entire libtool update/merge.

    cheers,
      DaveK

Re: [JAVA,libtool] Big libjava is biiiig.

[Dave Korn]

Bryce McKinlay wrote:

> If you're going to go to all this trouble, why not consider making libgcj
> compilable with the BC-ABI. 

  Well, in my case, because I have no idea what that involves or implies.  I
was hoping to find a solution that I can get in during this current stage1 and
it not be the only thing I spend all of stage1 doing.

  From your disclaimer, it sounds like this might be a slightly longer-term
project?

    cheers,
      DaveK

Re: [JAVA,libtool] Big libjava is biiiig.

[Andrew Haley]

Dave Korn wrote:
> Andrew Haley wrote:
>> Dave Korn wrote:
>>
>>> 1)  Would this be a reasonable approach, specifically i) in adding a configure
>>> option to cause sublibraries to be built, and ii) in using gmake's $(filter)
>>> construct to crudely subdivide the libraries like this?
>> At program startup the first library would be loaded, it would load
>> the next, and so on.  There are a few parts of libgcj that are truly
>> independent, but I suspect that you'd always load almost all of it.
>> So, you'd have longer startup time for loading all those files.
> 
>   Compared to a single DLL that is unusably malformed because it exceeds
> system limits, that's still an improvement :)
> 
>> With regard to GNU libc platforms:
>>
>> You'd no longer be able to make so much use of fast calls between
>> functions in the same library; you'd have to go via the PLT.
>>
>> Also, dl_iterate_phdr() is used a great deal (for finding exception
>> regions, garbage collection, etc.) and it linearly scans the libraries
>> that are loaded.  So, the more libraries you have loaded, the slower
>> it goes.
>>
>> Now, I don't know how much of these characteristics are shared by
>> Windows, but I imagine some of them are.
> 
>   Yes, the inter-library calls would have to be dllimports and go through
> stubs (one extra indirect branch).  I don't know how dl_iterate_phdr works on
> win32 but I imagine that it's also linear in the number of libs.
> 
>> So, I suspect your best bet would be to split libgcj into core and
>> non-core libraries and not slice much more thinly than that.  I can
>> advise you what is core and what isn't.
> 
>   Please do, I'll happily try that approach since it might significantly
> simplify my inter-dependences problems.

Here's a starter list of non-core packages:

gnu/CORBA
gnu/classpath/management
gnu/java/awt/dnd/peer/gtk
gnu/java/awt/dnd/peer/gtk
gnu/java/awt/peer/gtk
gnu/java/awt/peer/qt
gnu/java/awt/peer/x
gnu/java/beans
gnu/java/lang/managementheaders
gnu/java/math
gnu/java/util/prefs/gconf
gnu/java/util/prefs/gconf
gnu/javax/management
gnu/javax/rmi
gnu/javax/sound/midi
gnu/javax/sound/sampled/gstreamer) ignore
gnu/xml/aelfred2
gnu/xml/dom
gnu/xml/libxmlj
gnu/xml/pipeline
gnu/xml/stream
gnu/xml/transform
gnu/xml/util
gnu/xml/validation
gnu/xml/xpath
java/lang/management
javax/imageio
javax/rmi
javax/xml
org/omg/CORBA
org/omg/CORBA_2_3
org/omg/CosNaming
org/omg/Dynamic
org/omg/DynamicAny
org/omg/IOP
org/omg/Messaging
org/omg/PortableInterceptor
org/omg/PortableServer
org/omg/SendingContext
org/omg/stub
org/relaxng
org/w3c
org/xml

Andrew.

Re: [JAVA,libtool] Big libjava is biiiig.

[Ralf Wildenhues]

* Dave Korn wrote on Wed, May 06, 2009 at 07:08:17PM CEST:
> Ralf Wildenhues wrote:
> > I don't yet see why you would need any kind of libtool hacking.
> 
>   Because of this:
> 
> > You also have to ensure that the sub libraries are self-contained, or at
> > least their interdependencies form a directed non-cyclic graph (or you
> > will need very ugly hacks on w32).
> 
>   I fully expect there to be cyclic interdependencies, which I could resolve
> by building import libs first with dlltool.  I'm not sure yet whether to do
> this in libtool, but it occurred to me as one possibility.

If you could show the commands that would be needed without libtool,
then it would be easier for me to understand how libtool could help.
Of course, the more generally usable the approach, the better.

> Another approach
> would have been to do it in the Makefile, by first building the sub-DLLs all
> linked against the monolithic libjava DLL, then rebuilding them all but this
> time linking against their own import libs generated in the previous step; it
> occurred to me that even this might need a little help from libtool.

I agree that this sounds a bit awkward.

> > I also don't see why the GCC tree *ever* uses -L$(something)/.libs in
> > directories where libtool is used.  Just specify in-tree library
> > dependencies as 'libgcj.la' and automake and libtool will do The Right
> > Thing.  You adding an explicit -L.libs can only lead to installed .la
> > files retaining paths to build tree locations.
> 
>   Right, I'll try it; I have no idea why it's like that, I just cut and pasted
> existing examples without fully understanding.  The comment on the bit I cut
> and pasted reads:
> 
> ## We don't explicitly link in the libraries we need; libgcj.la brings
> ## in all dependencies.  We need the -L so that gcj can find libgcj
> ## with `-lgcj', but it must come first, otherwise the -L flags
> ## brought in from libgcj.la would cause the install directories to be
> ## searched before the build-tree ones, and we'd get errors because of
> ## different libraries with the same SONAME from picky linkers such as
> ## Solaris'.  FIXME: should be _libs on some systems.
> jv_convert_LDADD = -L$(here)/.libs libgcj.la

I understand that one might need -L$(here)/.libs with non-libtool
linking, where without it, -lgcj will not be found.  But it makes
no sense to me when libtool _is_ used for linking.  And here, libtool
must be the only user, if only for the simple reason that passing
'libgcj.la' (as opposed to '-lgcj') to the compiler driver (rather than
to libtool) will cause an error.

> >> 2)  Given that there's a bit of a logjam upstream, and not likely to be
> >> another libtool release betwen now and the end of stage1,
> > 
> > What makes you think so?  There is a backlog of some patches, but that
> > doesn't mean
> > - important patches won't go in,
> 
>   Oh, great.  I didn't want to /rely/ on being able to draw on your time when
> you've already got a lot to deal with, but in combination with this ...

If this turns out to be a lot of work to even stabilize, we can still
decide to let GCC do its own stuff for a while and reintegrate later.

> > I don't see a reason for that in your message.  Do you have other
> > reasons for suggesting this?
> 
>   Nope, I was only planning ahead against the contingency of it either not
> being practicable or convenient upstream, or there being subtle problems that
> transpire with doing an entire libtool update/merge.

Oh, I was intending for a complete autotool update for the GCC tree,
but given the delay on the license exception thing for Autoconf, I'm
not sure how well that will align with GCC being in Stage 1 ...

Cheers,
Ralf

Re: [JAVA,libtool] Big libjava is biiiig.

[Dave Korn]

Ralf Wildenhues wrote:
> * Dave Korn wrote on Wed, May 06, 2009 at 07:08:17PM CEST:
>> Ralf Wildenhues wrote:
>>> I don't yet see why you would need any kind of libtool hacking.
>>   Because of this:
>>
>>> You also have to ensure that the sub libraries are self-contained, or at
>>> least their interdependencies form a directed non-cyclic graph (or you
>>> will need very ugly hacks on w32).
>>   I fully expect there to be cyclic interdependencies, which I could resolve
>> by building import libs first with dlltool.  I'm not sure yet whether to do
>> this in libtool, but it occurred to me as one possibility.
> 
> If you could show the commands that would be needed without libtool,
> then it would be easier for me to understand how libtool could help.
> Of course, the more generally usable the approach, the better.

  Ok, so what I have is a big bunch of objects, that would normally be linked
together to make a single DLL, with all interdependencies resolved internally
and no undefined references:

g++ -shared a1.o a2.o a3.o b1.o b2.o b3.o c1.o c2.o c3.o \
    -o cygexample.dll -Wl,--out-implb libexample.dll.a

  What I instead want to do is partition the objects into (for example) three
separate sublibraries:

g++ -shared a1.o a2.o a3.o \
    -o cygexample-a.dll -Wl,--out-implb libexample-a.dll.a
g++ -shared b1.o b2.o b3.o \
    -o cygexample-b.dll -Wl,--out-implb libexample-b.dll.a
g++ -shared c1.o c2.o c3.o \
    -o cygexample-c.dll -Wl,--out-implb libexample-c.dll.a

  That won't work as-is because of the interdependencies; we can assume any of
the a/b/c objects may refer to any of the others.  So we need to do:

g++ -shared a1.o a2.o a3.o -lexample-b -lexample-c \
    -o cygexample-a.dll -Wl,--out-implb libexample-a.dll.a
g++ -shared b1.o b2.o b3.o -lexample-a -lexample-c \
    -o cygexample-b.dll -Wl,--out-implb libexample-b.dll.a
g++ -shared c1.o c2.o c3.o -lexample-a -lexample-b \
    -o cygexample-c.dll -Wl,--out-implb libexample-c.dll.a

... but as the import libs libexample-*.dll.a are generated as side-effects of
the link that builds the DLLs, they aren't available in time.  So we have to
use dlltool first to generate import libs ahead of final-link time (without
attmepting to build a complete DLL):

dlltool a1.o a2.o a3.o -D cygexample-a.dll -e libexample-a.dll.a
dlltool b1.o b2.o b3.o -D cygexample-b.dll -e libexample-b.dll.a
dlltool c1.o c2.o c3.o -D cygexample-c.dll -e libexample-c.dll.a
g++ -shared a1.o a2.o a3.o -lexample-b -lexample-c \
    -o cygexample-a.dll
g++ -shared b1.o b2.o b3.o -lexample-a -lexample-c \
    -o cygexample-b.dll
g++ -shared c1.o c2.o c3.o -lexample-a -lexample-b \
    -o cygexample-c.dll

>> Another approach
>> would have been to do it in the Makefile, by first building the sub-DLLs all
>> linked against the monolithic libjava DLL, then rebuilding them all but this
>> time linking against their own import libs generated in the previous step; it
>> occurred to me that even this might need a little help from libtool.
> 
> I agree that this sounds a bit awkward.

  True, but I'm not sure how well the libtool way-of-doing-things will adapt
to building in two stages.  It might be simplest just to invoke dlltool from
the libjava makefile to get the import libs built first, and feed them as
ready-made inputs to bog-standard libtool invocation, do you think?  I think I
might give that approach a try.

    cheers,
      DaveK

Re: [JAVA,libtool] Big libjava is biiiig.

[Andrew Haley]

Dave Korn wrote:
> Ralf Wildenhues wrote:
>> * Dave Korn wrote on Wed, May 06, 2009 at 07:08:17PM CEST:
>>> Ralf Wildenhues wrote:
>>>> I don't yet see why you would need any kind of libtool hacking.
>>>   Because of this:
>>>
>>>> You also have to ensure that the sub libraries are self-contained, or at
>>>> least their interdependencies form a directed non-cyclic graph (or you
>>>> will need very ugly hacks on w32).
>>>   I fully expect there to be cyclic interdependencies, which I could resolve
>>> by building import libs first with dlltool.  I'm not sure yet whether to do
>>> this in libtool, but it occurred to me as one possibility.
>> If you could show the commands that would be needed without libtool,
>> then it would be easier for me to understand how libtool could help.
>> Of course, the more generally usable the approach, the better.
> 
>   Ok, so what I have is a big bunch of objects, that would normally be linked
> together to make a single DLL, with all interdependencies resolved internally
> and no undefined references:
> 
> g++ -shared a1.o a2.o a3.o b1.o b2.o b3.o c1.o c2.o c3.o \
>     -o cygexample.dll -Wl,--out-implb libexample.dll.a
> 
>   What I instead want to do is partition the objects into (for example) three
> separate sublibraries:
> 
> g++ -shared a1.o a2.o a3.o \
>     -o cygexample-a.dll -Wl,--out-implb libexample-a.dll.a
> g++ -shared b1.o b2.o b3.o \
>     -o cygexample-b.dll -Wl,--out-implb libexample-b.dll.a
> g++ -shared c1.o c2.o c3.o \
>     -o cygexample-c.dll -Wl,--out-implb libexample-c.dll.a
> 
>   That won't work as-is because of the interdependencies; we can assume any of
> the a/b/c objects may refer to any of the others.  So we need to do:
> 
> g++ -shared a1.o a2.o a3.o -lexample-b -lexample-c \
>     -o cygexample-a.dll -Wl,--out-implb libexample-a.dll.a
> g++ -shared b1.o b2.o b3.o -lexample-a -lexample-c \
>     -o cygexample-b.dll -Wl,--out-implb libexample-b.dll.a
> g++ -shared c1.o c2.o c3.o -lexample-a -lexample-b \
>     -o cygexample-c.dll -Wl,--out-implb libexample-c.dll.a
> 
> ... but as the import libs libexample-*.dll.a are generated as side-effects of
> the link that builds the DLLs, they aren't available in time.  So we have to
> use dlltool first to generate import libs ahead of final-link time (without
> attmepting to build a complete DLL):
> 
> dlltool a1.o a2.o a3.o -D cygexample-a.dll -e libexample-a.dll.a
> dlltool b1.o b2.o b3.o -D cygexample-b.dll -e libexample-b.dll.a
> dlltool c1.o c2.o c3.o -D cygexample-c.dll -e libexample-c.dll.a
> g++ -shared a1.o a2.o a3.o -lexample-b -lexample-c \
>     -o cygexample-a.dll
> g++ -shared b1.o b2.o b3.o -lexample-a -lexample-c \
>     -o cygexample-b.dll
> g++ -shared c1.o c2.o c3.o -lexample-a -lexample-b \
>     -o cygexample-c.dll
> 
>>> Another approach
>>> would have been to do it in the Makefile, by first building the sub-DLLs all
>>> linked against the monolithic libjava DLL, then rebuilding them all but this
>>> time linking against their own import libs generated in the previous step; it
>>> occurred to me that even this might need a little help from libtool.
>> I agree that this sounds a bit awkward.
> 
>   True, but I'm not sure how well the libtool way-of-doing-things will adapt
> to building in two stages.  It might be simplest just to invoke dlltool from
> the libjava makefile to get the import libs built first, and feed them as
> ready-made inputs to bog-standard libtool invocation, do you think?  I think I
> might give that approach a try.

Did you try my list of things to lift out?  I don't think there will be any
interdependencies; the only problem might be that the reduction is not enough.

Andrew.

Re: [JAVA,libtool] Big libjava is biiiig.

[Dave Korn]

Andrew Haley wrote:

> Here's a starter list of non-core packages:
> 
> gnu/CORBA

  By the time I got done annotating all those packages with "non-core" in
makemake.tcl, it looked like the rule is "all packages of style 'bc' or
'bcheader'", is that correct?  I'd rather infer the list of non-core packages
from the existing annotations if it is correct to do so.

    cheers,
      DaveK

Re: [JAVA,libtool] Big libjava is biiiig.

[Andrew Haley]

Dave Korn wrote:
> Andrew Haley wrote:
> 
>> Here's a starter list of non-core packages:
>>
>> gnu/CORBA
> 
>   By the time I got done annotating all those packages with "non-core" in
> makemake.tcl, it looked like the rule is "all packages of style 'bc' or
> 'bcheader'", is that correct?  I'd rather infer the list of non-core packages
> from the existing annotations if it is correct to do so.

Makes sense.

Andrew.

Re: [JAVA,libtool] Big libjava is biiiig.

[Dave Korn]

Andrew Haley wrote:

> Did you try my list of things to lift out?  I don't think there will be any
> interdependencies; the only problem might be that the reduction is not enough.

  As my other post probably implied, I've just come back to this after a while
 pursuing other things, so I haven't tried it yet; if there are no
interdependencies, that makes my job almost trivial - thanks!

    cheers,
      DaveK

Re: [JAVA,libtool] Big libjava is biiiig.

[Dave Korn]

Andrew Haley wrote:

> Did you try my list of things to lift out?  I don't think there will be any
> interdependencies; the only problem might be that the reduction is not enough.

    Hi Andrew,

  I've had a quick hack at it now, and it's not doing what I'd hoped, so
possibly I've misunderstood what you intended, or am not grokking the
structure of java packages correctly.

  If I got the essence of your list right, you were indicating all the
packages that collectively amount to $(bc_objects) in the libjava makefile, yes?

  I tried adding libtool definitions to build a libjava-core which should if
I've got it right be exactly the same as libjava, but exclude the
$(bc_objects), and a libjava-noncore that only links in the $(bc_objects).  So
I've got these definitions in Makefile.am, immediately after all the
corresponding default libgcj_la_XXXX definitions:

-----------------------------<snip>-----------------------------
if BUILD_SUBLIBS
toolexeclib_LTLIBRARIES += libgcj-core.la libgcj-noncore.la
endif
-----------------------------<snip>
if BUILD_SUBLIBS
libgcj_core_la_SOURCES = $(libgcj_la_SOURCES)
libgcj_noncore_la_SOURCES =
endif
-----------------------------<snip>
if BUILD_SUBLIBS
libgcj_core_la_LDFLAGS = $(libgcj_la_LDFLAGS)
libgcj_noncore_la_LDFLAGS = $(libgcj_la_LDFLAGS)
endif
-----------------------------<snip>
if BUILD_SUBLIBS
libgcj_core_la_LIBADD = \
	classpath/native/fdlibm/libfdlibm.la \
	java/lang/Object.lo \
	java/lang/Class.lo \
	java/process-$(PLATFORM).lo \
	$(all_packages_source_files:.list=.lo) \
--> note $(bc_objects) missing here as compared to libgcj_la_LIBADD.
	$(propertyo_files) \
	$(LIBFFI) $(ZLIBS) $(GCLIBS)
libgcj_noncore_la_LIBADD = $(bc_objects)
endif
-----------------------------<snip>
if BUILD_SUBLIBS
libgcj_core_la_DEPENDENCIES = $(libgcj_la_DEPENDENCIES)
libgcj_noncore_la_DEPENDENCIES = $(libgcj_la_DEPENDENCIES) libgcj-core.la
endif
-----------------------------<snip>
if BUILD_SUBLIBS
libgcj_core_la_DEPENDENCIES += $(srcdir)/libgcj.ver
libgcj_noncore_la_DEPENDENCIES += $(srcdir)/libgcj.ver
endif
-----------------------------<snip>
if BUILD_SUBLIBS
libgcj_core_la_LINK = $(libgcj_la_LINK)
libgcj_noncore_la_LINK = $(libgcj_la_LINK)
endif
-----------------------------<snip>-----------------------------

... where BUILD_SUBLIBS is my new AM_CONDITIONAL flag enabled by
--enable-libgcj-sublibs.

  However, the new libjava-core doesn't link, and appears to contain many
dependencies on the $(bc_objects) files, for e.g. fundamental language features:

gnu/java/awt/.libs/java2d.o:/gnu/gcc/gcc/libjava/classpath/gnu/java/awt/java2d/PolyEdge.java:135:
undefined reference to `int gnu::java::math::Fixed::div(int, int, int)'
gnu/java/awt/.libs/java2d.o:/gnu/gcc/gcc/libjava/classpath/gnu/java/awt/java2d/ScanlineConverter.java:419:
undefined reference to `int gnu::java::math::Fixed::div(int, int, int)'

... and even entire complex packages:

java/.libs/util.o:/gnu/gcc/gcc/libjava/classpath/java/util/Properties.java:689:
undefined reference to `org::w3c::dom::bootstrap::DOMImplementationRegistry*
org::w3c::dom::bootstrap::DOMImplementationRegistry::newInstance()'
java/.libs/util.o:/gnu/gcc/gcc/libjava/classpath/java/util/Properties.java:690:
undefined reference to `org::w3c::dom::DOMImplementation*
org::w3c::dom::bootstrap::DOMImplementationRegistry::getDOMImplementation(java::lang::String*)'
java/.libs/util.o:/gnu/gcc/gcc/libjava/classpath/java/util/Properties.java:692:
undefined reference to `org::w3c::dom::DOMImplementation::class$'
java/.libs/util.o:/gnu/gcc/gcc/libjava/classpath/java/util/Properties.java:694:
undefined reference to `org::w3c::dom::DOMImplementation::class$'
java/.libs/util.o:/gnu/gcc/gcc/libjava/classpath/java/util/Properties.java:695:
undefined reference to `org::w3c::dom::Document::class$'
java/.libs/util.o:/gnu/gcc/gcc/libjava/classpath/java/util/Properties.java:698:
undefined reference to `org::w3c::dom::Document::class$'
java/.libs/util.o:/gnu/gcc/gcc/libjava/classpath/java/util/Properties.java:699:
undefined reference to `org::w3c::dom::Document::class$'
java/.libs/util.o:/gnu/gcc/gcc/libjava/classpath/java/util/Properties.java:699:
undefined reference to `org::w3c::dom::Element::class$'

  So, did I do the partitioning wrong here?  Or have I misunderstood what you
were referring to as "interdependencies"?

    cheers,
      DaveK

Re: [JAVA,libtool] Big libjava is biiiig.

[Dave Korn]

Dave Korn wrote:
> So we have to
> use dlltool first to generate import libs ahead of final-link time (without
> attmepting to build a complete DLL):
> 
> dlltool a1.o a2.o a3.o -D cygexample-a.dll -e libexample-a.dll.a
> dlltool b1.o b2.o b3.o -D cygexample-b.dll -e libexample-b.dll.a
> dlltool c1.o c2.o c3.o -D cygexample-c.dll -e libexample-c.dll.a

  Oops, typo.  Those should be -l, not -e options.

    cheers,
      DaveK

Re: [JAVA,libtool] Big libjava is biiiig.

[Andrew Haley]

Dave Korn wrote:
> Andrew Haley wrote:
> 
>> Did you try my list of things to lift out?  I don't think there will be any
>> interdependencies; the only problem might be that the reduction is not enough.
> 
>     Hi Andrew,
> 
>   I've had a quick hack at it now, and it's not doing what I'd hoped, so
> possibly I've misunderstood what you intended, or am not grokking the
> structure of java packages correctly.
> 
> 
>   So, did I do the partitioning wrong here?  Or have I misunderstood what you
> were referring to as "interdependencies"?

Oh, I'm _so_ sorry, you're absolutely right.  I misread (or, perhaps, miswrote)
the makemake.tcl file.  It turns out that even those files that are compiled BC
are compiled with -fno-indirect-classes, so that they can be (and indeed are)
called from code compiled with the C++ ABI.

How embarrassing.  :-(

Andrew.

Re: [JAVA,libtool] Big libjava is biiiig.

[Dave Korn]

Andrew Haley wrote:
> Dave Korn wrote:
>> Andrew Haley wrote:
>>
>>> Did you try my list of things to lift out?  I don't think there will be any
>>> interdependencies; the only problem might be that the reduction is not enough.
>>     Hi Andrew,
>>
>>   I've had a quick hack at it now, and it's not doing what I'd hoped, so
>> possibly I've misunderstood what you intended, or am not grokking the
>> structure of java packages correctly.
>>
>>
>>   So, did I do the partitioning wrong here?  Or have I misunderstood what you
>> were referring to as "interdependencies"?
> 
> Oh, I'm _so_ sorry, you're absolutely right.  I misread (or, perhaps, miswrote)
> the makemake.tcl file.  It turns out that even those files that are compiled BC
> are compiled with -fno-indirect-classes, so that they can be (and indeed are)
> called from code compiled with the C++ ABI.
> 
> How embarrassing.  :-(


  That's ok, thanks for trying to help.  I'll just have to go down the route
of preparing import libraries ahead-of-time.  (Actually I'm already doing this
as we speak, and it is /possibly/ showing up some binutils bugs, but that's
another story....)

  On the bright side, this means that if they're going to be interdependent
sublibraries anyway, I can parcel them out more evenly, and get a greater
reduction in the number of symbols each one individually exports.

    cheers,
      DaveK

Re: [JAVA,libtool] Big libjava is biiiig.

[Dave Korn]

[ Reviving a thread from 20090506, as the first step toward raising this issue
again on the lists, and just in case anyone was interested in the follow-on ... ]

David Daney wrote:
> Ralf Wildenhues wrote:
>> Hello Dave,
>>
>> * Dave Korn wrote on Wed, May 06, 2009 at 06:09:05PM CEST:
> [...]
>>> 1)  Would this be a reasonable approach, specifically i) in adding a
>>> configure
>>> option to cause sublibraries to be built, and ii) in using gmake's
>>> $(filter)
>>> construct to crudely subdivide the libraries like this?
>>
>> You are aware of the fact that it is part of the ABI in which of the
>> linked DLLs a given symbol was found, and any shuffling of that later
>> will break that ABI?
>>
>> You also have to ensure that the sub libraries are self-contained, or at
>> least their interdependencies form a directed non-cyclic graph (or you
>> will need very ugly hacks on w32).
>>
> 
> Unfortunately it may not be a simple task to find a suitably large set
> of packages that satisfy this 'directed non-cyclic graph' criterion.

  Not simple, but not so hard as to be impossible either; as it turns out, the
internal structure of libgcj looks a lot like a turnip, with a bunch of skinny
branchy foliage waving around on top, a few shallow roots spreading under the
ground, and a big ball of hair in the middle holding it all together, which
makes it actually quite easy to manually find a partition.

  There are a couple of regressions to solve first, but it appears that I've
more-or-less cracked it.  Full details are written up here:

        http://gcc.gnu.org/wiki/Internal_dependencies_of_libgcj

    cheers,
      DaveK

Re: [JAVA,libtool] Big libjava is biiiig.

[Andrew Haley]

Dave Korn wrote:

>   Not simple, but not so hard as to be impossible either; as it turns out, the
> internal structure of libgcj looks a lot like a turnip, with a bunch of skinny
> branchy foliage waving around on top, a few shallow roots spreading under the
> ground, and a big ball of hair in the middle holding it all together, which
> makes it actually quite easy to manually find a partition.
> 
>   There are a couple of regressions to solve first, but it appears that I've
> more-or-less cracked it.  Full details are written up here:
> 
>         http://gcc.gnu.org/wiki/Internal_dependencies_of_libgcj


Thank you for this.  One thing that you may like to consider is compiling
more packages with -findirect-dispatch -fno-indirect-classes.  This has a
smallish performance penalty, but packages compiled this way have no link-
time dependencies on other packages.  Instead, everything gets fixed up
at runtime.

Andrew.

Re: [JAVA,libtool] Big libjava is biiiig.

[Dave Korn]

Andrew Haley wrote:

> One thing that you may like to consider is compiling
> more packages with -findirect-dispatch -fno-indirect-classes.  This has a
> smallish performance penalty, but packages compiled this way have no link-
> time dependencies on other packages.  Instead, everything gets fixed up
> at runtime.

  That's useful to know, it sounds more or less like a "get out of jail free"
card in the event that we ever need to break any dependency cycles, which
might happen as libjava grows.

    cheers,
      DaveK

Re: [JAVA,libtool] Big libjava is biiiig.

[Tom Tromey]

>>>>> "Dave" == Dave Korn <dave.korn.cygwin@googlemail.com> writes:

Dave> There are a couple of regressions to solve first, but it appears
Dave> that I've more-or-less cracked it.  Full details are written up
Dave> here:

Dave>         http://gcc.gnu.org/wiki/Internal_dependencies_of_libgcj

One thing worth considering is that you may be able to shrink things
even more by splitting up some existing objects.

I didn't see AWT in the cluster 48 list, which seems weird.
I would expect it to be in the core due to AWTPermission.

I'm curious why sun.awt and swing ended up in there.

I would expect that with minor tweaks you could probably get AWT, the
peers, and Swing to drop out.  That was true for AWT, at least, last
time I looked at this (years ago) -- but I needed a special case to keep
AWTPermission in.

Tom

Re: [JAVA,libtool] Big libjava is biiiig.

[David Daney]

Tom Tromey wrote:
>>>>>> "Dave" == Dave Korn <dave.korn.cygwin@googlemail.com> writes:
> 
> Dave> There are a couple of regressions to solve first, but it appears
> Dave> that I've more-or-less cracked it.  Full details are written up
> Dave> here:
> 
> Dave>         http://gcc.gnu.org/wiki/Internal_dependencies_of_libgcj
> 
> One thing worth considering is that you may be able to shrink things
> even more by splitting up some existing objects.
> 
> I didn't see AWT in the cluster 48 list, which seems weird.
> I would expect it to be in the core due to AWTPermission.
> 
> I'm curious why sun.awt and swing ended up in there.
> 
> I would expect that with minor tweaks you could probably get AWT, the
> peers, and Swing to drop out.  That was true for AWT, at least, last
> time I looked at this (years ago) -- but I needed a special case to keep
> AWTPermission in.

I have patches that do a lot of the things Tom mentions.  They are only 
lightly tested, but they could be a starting point.

I will dig them out and post them this weekend.

David Daney.

Re: [JAVA,libtool] Big libjava is biiiig.

[Dave Korn]

David Daney wrote:
> Tom Tromey wrote:
>>>>>>> "Dave" == Dave Korn <dave.korn.cygwin@googlemail.com> writes:
>>
>> Dave> There are a couple of regressions to solve first, but it appears
>> Dave> that I've more-or-less cracked it.  Full details are written up
>> Dave> here:
>>
>> Dave>         http://gcc.gnu.org/wiki/Internal_dependencies_of_libgcj
>>
>> One thing worth considering is that you may be able to shrink things
>> even more by splitting up some existing objects.
>>
>> I didn't see AWT in the cluster 48 list, which seems weird.
>> I would expect it to be in the core due to AWTPermission.
>>
>> I'm curious why sun.awt and swing ended up in there.

  All the raw data is there to take a look at if you're determinedly curious.
 I don't know a lot about the theory behind the graphviz sccmap tool, and how
exactly it breaks the graph up.

>> I would expect that with minor tweaks you could probably get AWT, the
>> peers, and Swing to drop out.  That was true for AWT, at least, last
>> time I looked at this (years ago) -- but I needed a special case to keep
>> AWTPermission in.
> 
> I have patches that do a lot of the things Tom mentions.  They are only
> lightly tested, but they could be a starting point.
> 
> I will dig them out and post them this weekend.

  Thanks for the suggestions!  I'm going to finish the last tweaks and submit
my patch with the list as it currently stands, because of the scale of the
change and the looming-yet-still-a-little-vague end of stage1, but I think
tweaking the list of files subsequently is something we can do incrementally
once it's reviewed and checked in, no?

    cheers,
      DaveK