Re: ld-auto-import memory bug fixing

Re: ld-auto-import memory bug fixing

[Ralf Habacker]

Hi,

I like to analyse how ld grab memory in which functions. For that I have patched
malloc relating functions like xmalloc, bfd_malloc,... to print out the function
in which it is called, the params/result and accumulated memory usage.

At next I like to know the tree of functions which have this functions called
(like backtrace).
I know that gcc has support of basic block profiling. Could this used in anyway
to automated print out functions name on calling the relating functions ?

--Ralf

Re: ld-auto-import memory bug fixing

[DJ Delorie]

> I know that gcc has support of basic block profiling. Could this used in anyway
> to automated print out functions name on calling the relating functions ?

With newer gcc's, you want:

  -finstrument-functions  Instrument function entry/exit with profiling calls

Older gcc's should have -pg (call graph counting).

Either one should tell you which functions get called by which
functions and how many times for each combination.

You'll have to look the options up to see how to use them and which
extra programs to run (gprof, gcov, etc) to get reports.

RE: ld-auto-import memory bug fixing

[Ralf Habacker]

>
>
> > I know that gcc has support of basic block profiling. Could this
> used in anyway
> > to automated print out functions name on calling the relating functions ?
>
> With newer gcc's, you want:
>
>   -finstrument-functions  Instrument function entry/exit with profiling calls
>
> Older gcc's should have -pg (call graph counting).
>
> Either one should tell you which functions get called by which
> functions and how many times for each combination.
>
> You'll have to look the options up to see how to use them and which
> extra programs to run (gprof, gcov, etc) to get reports.
>
It seems to be to much overhead for this,because I need this in relating of used
memory,
so I'm trying to check manual which functions are involved.

... much debugging...

I have found that the currently mostly problem seems to be
pe_walk_relocs_of_symbol() like mentioned below.
To find this I have added some printfs before and
after any called function in ldemul.c and later in some subroutines.

pe_walk_relocs_of_symbol() seems to need about 4 MB for any auto imported
symbol,
so if there are 30 vars to import it needs 120 MB. :-(((

This depends on the number of used libs and there are many libs in the test case
( 
look at the link line below) 

BTW: Can anyone tell me, what pe_find_data_imports () and pe_walk_relocs_of_symbol() really do ? 

What I have seen, is that pe_walk_relocs_of_symbol() goes through every input bfd, 
that means through all loaded libs and object files and read the relating symbol table 
and relocation data and make something with it. The relocation data are freed, 
but the symbol table not. That is the problem. 

Perhaps it is possible to load the symbols table only if not already done and cache the pointer as 
read every loop again. (1) 

A note says 
	  /* Warning: the allocated symbols are remembered in BFD and reused
	     later, so don't free them! */
This makes me some headache, because I don't know where to look and if (1) is really a possible solution. 
Additional if the symbols are rembered in BFD, why should they allocated every time again ???

Any suggestions welcome. 

void
pe_walk_relocs_of_symbol (info, name, cb)
     struct bfd_link_info *info;
     CONST char *name;
     int (*cb) (arelent *);
{
  bfd *b;
  struct sec *s;
	
  for (b = info->input_bfds; b; b = b->link_next)
    {
      arelent **relocs;
      int relsize, nrelocs, i;

      for (s = b->sections; s; s = s->next)
	{
	  asymbol **symbols;
	  int nsyms, symsize;
	  int flags = bfd_get_section_flags (b, s);

	  /* Skip discarded linkonce sections */
	  if (flags & SEC_LINK_ONCE
	      && s->output_section == bfd_abs_section_ptr)
	    continue;

	  current_sec=s;

	  symsize = bfd_get_symtab_upper_bound (b);
	  symbols = (asymbol **) xmalloc (symsize);
	  nsyms = bfd_canonicalize_symtab (b, symbols);

	  relsize = bfd_get_reloc_upper_bound (b, s);
	  relocs = (arelent **) xmalloc ((size_t) relsize);
	  nrelocs = bfd_canonicalize_reloc (b, s, relocs, symbols);

	  for (i = 0; i < nrelocs; i++)
	    {
	      struct symbol_cache_entry *sym = *relocs[i]->sym_ptr_ptr;
	      if (!strcmp(name,sym->name)) cb(relocs[i]);
	    }
	  free (relocs);
	  /* Warning: the allocated symbols are remembered in BFD and reused
	     later, so don't free them! */
	  /* free (symbols); */
	}
    }
}

<snip>
ld logging 
stage: ldemul_open_dynamic_archive - END
stage: ldemul_open_dynamic_archive - START
stage: ldemul_open_dynamic_archive - END
stage: ldemul_open_dynamic_archive - START
stage: ldemul_open_dynamic_archive - END
stage: ldemul_open_dynamic_archive - START
        gld_i386pe_open_dynamic_archive: archive name=/usr/lib/gcc-lib/i686-pc-cygwin/2.95.3-5/libgcc.a
stage: ldemul_open_dynamic_archive - END
stage: ldemul_recognized_file - START
stage: ldemul_recognized_file - END
stage: ldemul_after_open - START
Warning: resolving _qt_x_time by linking to __imp__qt_x_time (auto-import)
stage: pe_walk_relocs_of_symbol - START
      
  pe_walk_relocs_of_symbol: malloc = 4090408
stage: pe_walk_relocs_of_symbol - END
Warning: resolving __12QApplication$focus_widget by linking to
__imp___12QApplication$focus_widget (auto-import)
stage: pe_walk_relocs_of_symbol - START
        pe_walk_relocs_of_symbol: malloc = 4090604
stage: pe_walk_relocs_of_symbol - END
Warning: resolving _qApp by linking to __imp__qApp (auto-import)
stage: pe_walk_relocs_of_symbol - START
        pe_walk_relocs_of_symbol: malloc = 4090920
stage: pe_walk_relocs_of_symbol - END
Warning: resolving _qt_wm_protocols by linking to __imp__qt_wm_protocols
(auto-import)
stage: pe_walk_relocs_of_symbol - START
        pe_walk_relocs_of_symbol: malloc = 4090996
stage: pe_walk_relocs_of_symbol - END
Warning: resolving _qt_wm_take_focus by linking to __imp__qt_wm_take_focus
(auto-import)
stage: pe_walk_relocs_of_symbol - START
        pe_walk_relocs_of_symbol: malloc = 4091032
stage: pe_walk_relocs_of_symbol - END
Warning: resolving _qt_wm_delete_window by linking to __imp__qt_wm_delete_window
(auto-import)
stage: pe_walk_relocs_of_symbol - START
        pe_walk_relocs_of_symbol: malloc = 4091068
stage: pe_walk_relocs_of_symbol - END



$ /bin/ld --shared -Bdynamic -e __cygwin_dll_entry@12 --dll-search-prefix=cyg -o
.libs/cygkdeui-3.dll -L/home/habacker/
tmp/kde/anoncvs.kde.org/kdelibs/dcop/KDE-ICE/.libs -L/opt/kde2/lib -L/home/habac
ker/tmp/kde/anoncvs.kde.org/kdelibs/dco
p/.libs -L/usr/X11R6/lib -L/usr/lib/qt2/lib -L/usr/local/lib -L/usr/lib -L/usr/l
ib/w32api -L/usr/lib/gcc-lib/i686-pc-cy
gwin/2.95.3-5 -L/usr/lib/gcc-lib/i686-pc-cygwin/2.95.3-5/../../../../i686-pc-cyg
win/lib --enable-auto-import --export-a
ll-symbols --whole-archive .libs/qxembed.o .libs/ksharedpixmap.o
.libs/kthemestyle.o .libs/kurllabel.o .libs/knuminput.
o .libs/ktoolbar.o .libs/klineedit.o .libs/klistview.o .libs/kprogress.o
.libs/kimageeffect.o .libs/kcolordlg.o .libs/k
select.o .libs/kdatepik.o .libs/kdatetbl.o .libs/kfontdialog.o
.libs/kpopupmenu.o .libs/ktabctl.o .libs/kstatusbar.o .l
ibs/kmainwindow.o .libs/ktmainwindow.o .libs/kmenubar.o .libs/kpixmapio.o
.libs/keditlistbox.o .libs/kseparator.o .libs
/krestrictedline.o .libs/kcolorbtn.o .libs/ksystemtray.o .libs/kbuttonbox.o
.libs/keditcl1.o .libs/keditcl2.o .libs/kle
d.o .libs/kdbtn.o .libs/kwizard.o .libs/kcontainer.o .libs/kkeydialog.o
.libs/kruler.o .libs/kcursor.o .libs/kaccelmenu
.o .libs/klineeditdlg.o .libs/kthemebase.o .libs/kcharselect.o
.libs/kcolordrag.o .libs/knumvalidator.o .libs/kdialog.o
 .libs/kdialogbase.o .libs/kjanuswidget.o .libs/kaboutdialog.o
.libs/kpixmapeffect.o .libs/kauthicon.o .libs/kmessagebo
x.o .libs/kdualcolorbtn.o .libs/ktoolbarradiogroup.o .libs/ktoolbarbutton.o
.libs/ktextbrowser.o .libs/kaction.o .libs/
khelpmenu.o .libs/kcmenumngr.o .libs/kpanelmenu.o .libs/kstdaction.o
.libs/kdrawutil.o .libs/kiconview.o .libs/klistbox
.o .libs/kbugreport.o .libs/kcombobox.o .libs/kpassdlg.o .libs/kxmlguiclient.o
.libs/kxmlgui.o .libs/kxmlguibuilder.o .
libs/kedittoolbar.o .libs/kdocktabctl.o .libs/kdockwidget.o
.libs/kdockwidget_private.o .libs/kanimwidget.o .libs/kroot
pixmap.o .libs/kaboutkde.o .libs/kaboutapplication.o .libs/kpanelapplet.o
.libs/kdcopactionproxy.o .libs/kcolorcombo.o
.libs/kpushbutton.o .libs/kpanelextension.o .libs/kcompletionbox.o
.libs/kalphapainter.o .libs/ksqueezedtextlabel.o .li
bs/kcommand.o .libs/kwindowlistmenu.o .libs/kfontcombo.o .libs/ktip.o
.libs/kdatewidget.o .libs/karrowbutton.o .libs/li
bkdeui_la_meta_unload.o --no-whole-archive -lcygwin
../kdecore/.libs/libkdecore.dll.a /home/habacker/tmp/kde/anoncvs.kd
e.org/kdelibs/dcop/.libs/libDCOP.dll.a
/home/habacker/tmp/kde/anoncvs.kde.org/kdelibs/dcop/KDE-ICE/.libs/libkICE.dll.a
-lqt -lpng -lz -ljpeg -lXext -lX11 -lipc -lSM -lICE --out-implib
.libs/libkdeui.dll.a -lstdc++ -lgcc -lcygwin -luser32
-lkernel32 -ladvapi32 -lshell32 -lgcc --enable-extra-pe-debug --enable-extra-pe-
debug

RE: ld-auto-import memory bug fixing

[Ralf Habacker]

I have looked into the relating stuff and found, that for any section of any
input bfd a symbol tab is allocated.
For this I have two questions:

1. I recognized, that for any input file read by bfd one symboltab is present,
not one for any section. Isn't it 	?
If this is true (and it seems so, because I have applied a patch and have
compiled about 20 dll's with it), I have
found a way to solve this problem.  (moving symboltable allocating in the first
for loop)

(This patch contains some debugging stuff, which needs other files patch. It is
for dicsussion only)

pe-dll.c
   for (b = info->input_bfds; b; b = b->link_next)
     {
-      arelent **relocs;
-      int relsize, nrelocs, i;
+         int nsyms, symsize;
+         asymbol **symbols;

+         symsize = bfd_get_symtab_upper_bound (b);
+         symbols = (asymbol **) xmalloc (symsize);
+         nsyms = bfd_canonicalize_symtab (b, symbols);
+
       for (s = b->sections; s; s = s->next)
        {
-         asymbol **symbols;
-         int nsyms, symsize;
+      int relsize, nrelocs, i;
+           arelent **relocs;
          int flags = bfd_get_section_flags (b, s);

          /* Skip discarded linkonce sections */
          if (flags & SEC_LINK_ONCE
-             && s->output_section == bfd_abs_section_ptr)
+                     && s->output_section == bfd_abs_section_ptr) {
+                               if (pe_dll_extra_pe_debug & 4)
+                                       printf("\t section->name=%s -->
ignored\n",s->name);
            continue;
-
+                       }
          current_sec=s;

-         symsize = bfd_get_symtab_upper_bound (b);
-         symbols = (asymbol **) xmalloc (symsize);
-         nsyms = bfd_canonicalize_symtab (b, symbols);
-
          relsize = bfd_get_reloc_upper_bound (b, s);
          relocs = (arelent **) xmalloc ((size_t) relsize);
          nrelocs = bfd_canonicalize_reloc (b, s, relocs, symbols);

2. Relating to this bug, I have found that allocating symbol tables is done
several times in ld. Using a symbol
   cache would be save much memory.

			pe_find_data_imports ()
	ei386pe.c:         symsize = bfd_get_symtab_upper_bound (b);

			gld_i386pe_after_open ()
	ei386pe.c:         symsize = bfd_get_symtab_upper_bound (is->the_bfd);

			pe_walk_relocs_of_symbol (info, name, cb)
	pe-dll.c:         symsize = bfd_get_symtab_upper_bound (b);

			process_def_file (abfd, info)
	pe-dll.c:         symsize = bfd_get_symtab_upper_bound (b);

			generate_reloc (abfd, info)
	pe-dll.c:         symsize = bfd_get_symtab_upper_bound (b);

	ldcref.c:      symsize = bfd_get_symtab_upper_bound (abfd);
	ldmain.c:         symsize = bfd_get_symtab_upper_bound (abfd);
	ldmisc.c:                   symsize = bfd_get_symtab_upper_bound (abfd);

What about creating a symbol cache list in ld using pointer to loaded symbols
tables or by extending bfd structure with a pointer to a "symbol cache entry",
which can be filled by client code and later reused by other code.
Extending bfd would minimize the overhead to handle this cache, so I think this
would be the best way.
I have to say one limitation to this. Such cached symbol tables should not be
manipulated. They should be used only read only. Is this real ?

Regards Ralf

RE: ld-auto-import memory bug fixing

[Ralf Habacker]

>
> I have looked into the relating stuff and found, that for any section of any
> input bfd a symbol tab is allocated.
> For this I have two questions:
>
> 1. I recognized, that for any input file read by bfd one symboltab is present,
> not one for any section. Isn't it 	?
> If this is true (and it seems so, because I have applied a patch and have
> compiled about 20 dll's with it), I have
> found a way to solve this problem.  (moving symboltable allocating in
> the first
> for loop)
>
> (This patch contains some debugging stuff, which needs other files
> patch. It is
> for dicsussion only)
>
> pe-dll.c
>    for (b = info->input_bfds; b; b = b->link_next)
>      {
> -      arelent **relocs;
> -      int relsize, nrelocs, i;
> +         int nsyms, symsize;
> +         asymbol **symbols;
>
> +         symsize = bfd_get_symtab_upper_bound (b);
> +         symbols = (asymbol **) xmalloc (symsize);
> +         nsyms = bfd_canonicalize_symtab (b, symbols);
> +
>        for (s = b->sections; s; s = s->next)
>         {
> -         asymbol **symbols;
> -         int nsyms, symsize;
> +      int relsize, nrelocs, i;
> +           arelent **relocs;
>           int flags = bfd_get_section_flags (b, s);
>
>           /* Skip discarded linkonce sections */
>           if (flags & SEC_LINK_ONCE
> -             && s->output_section == bfd_abs_section_ptr)
> +                     && s->output_section == bfd_abs_section_ptr) {
> +                               if (pe_dll_extra_pe_debug & 4)
> +                                       printf("\t section->name=%s -->
> ignored\n",s->name);
>             continue;
> -
> +                       }
>           current_sec=s;
>
> -         symsize = bfd_get_symtab_upper_bound (b);
> -         symbols = (asymbol **) xmalloc (symsize);
> -         nsyms = bfd_canonicalize_symtab (b, symbols);
> -
>           relsize = bfd_get_reloc_upper_bound (b, s);
>           relocs = (arelent **) xmalloc ((size_t) relsize);
>           nrelocs = bfd_canonicalize_reloc (b, s, relocs, symbols);
>
> 2. Relating to this bug, I have found that allocating symbol tables is done
> several times in ld. Using a symbol
>    cache would be save much memory.
>
> 			pe_find_data_imports ()
> 	ei386pe.c:         symsize = bfd_get_symtab_upper_bound (b);
>
> 			gld_i386pe_after_open ()
> 	ei386pe.c:         symsize = bfd_get_symtab_upper_bound (is->the_bfd);
>
> 			pe_walk_relocs_of_symbol (info, name, cb)
> 	pe-dll.c:         symsize = bfd_get_symtab_upper_bound (b);
>
> 			process_def_file (abfd, info)
> 	pe-dll.c:         symsize = bfd_get_symtab_upper_bound (b);
>
> 			generate_reloc (abfd, info)
> 	pe-dll.c:         symsize = bfd_get_symtab_upper_bound (b);
>
> 	ldcref.c:      symsize = bfd_get_symtab_upper_bound (abfd);
> 	ldmain.c:         symsize = bfd_get_symtab_upper_bound (abfd);
> 	ldmisc.c:                   symsize = bfd_get_symtab_upper_bound (abfd);
>
> What about creating a symbol cache list in ld using pointer to loaded symbols
> tables or by extending bfd structure with a pointer to a "symbol cache entry",
> which can be filled by client code and later reused by other code.
> Extending bfd would minimize the overhead to handle this cache, so I
> think this would be the best way.

Currently I have implemented a symbol cache table for holding bfd symbol tables
(referenced by pe-dll.c functions) in a simple vector . It works, but needs too
much lookup time because of the number of contained entries (compiling kdelibs
produces about 10.000 bfd's). Using this cache saves additional about 15 MB on
compiling kdeui with regards to the previous fix, which saves about 100 MB).

The alternatives are to implement a better lookup function like binary search or
use another strategy.

Te best thing, I think, is in using the bfd structure to hold a pointer to the
cached canonicalized bfd symbol table.
Ive found a bfd struct member called "usrdata" which was used by ldlang.c, which
seems to hold ld specific data relating to bfd's in a lang_input_statement_type
struct

My question currently is which the better is, adding members to this struct or
to add additional members to the bfd struct.


> I have to say one limitation to this. Such cached symbol tables should not be
> manipulated. They should be used only read only. Is this real ?
>
> Regards Ralf
>
>

Re: ld-auto-import memory bug fixing

[Nick Clifton]

Hi Ralf,

> 1. I recognized, that for any input file read by bfd one symboltab
> is present, not one for any section. Isn't it 	?

Err, I do not think that this is quite right.  For example dynamic
sections have their own symbol table, seperate from the normal symbol
table.

> The best thing, I think, is in using the bfd structure to hold a
> pointer to the cached canonicalized bfd symbol table. I've found a
> bfd struct member called "usrdata" which was used by ldlang.c, which
> seems to hold ld specific data relating to bfd's in a
> lang_input_statement_type struct
> 
> My question currently is which the better is, adding members to this
> struct or to add additional members to the bfd struct.

I would suggest keeping the number of changes to BFD small and so
using usrdata to point to your own structure.

> I have to say one limitation to this. Such cached symbol tables
> should not be manipulated. They should be used only read only. Is
> this real ? 

Unless you want to modify large parts of bfd then I would say yes.

Cheers
        Nick

RE: ld-auto-import memory bug fixing

[Ralf Habacker]

> Hi Ralf,
>
> > 1. I recognized, that for any input file read by bfd one symboltab
> > is present, not one for any section. Isn't it 	?
>
> Err, I do not think that this is quite right.  For example dynamic
> sections have their own symbol table, seperate from the normal symbol
> table.
How are dynamic sections accessed ?
>
> > The best thing, I think, is in using the bfd structure to hold a
> > pointer to the cached canonicalized bfd symbol table. I've found a
> > bfd struct member called "usrdata" which was used by ldlang.c, which
> > seems to hold ld specific data relating to bfd's in a
> > lang_input_statement_type struct
> >
> > My question currently is which the better is, adding members to this
> > struct or to add additional members to the bfd struct.
>
> I would suggest keeping the number of changes to BFD small and so
> using usrdata to point to your own structure.

In the meantime I have already found such a structure in ldlang.h

typedef struct lang_input_statement_struct {
<snip>
  /* Symbol table of the file.  */
  asymbol **asymbols;
  unsigned int symbol_count;
<snip>

This structure is already used by ldmain.c: warning_callback() and ldcref.c:
check_refs(),
so I have created a new function ld_get_symbol_table(bfd *abfd, int *nsyms) and
added this
to the below mentioned functions:

ldmain.c: warning_callback()

ldcref.c: check_refs()

ldmisc:c  vfinfo()

pe-dll.c:
  process_def_file ()

  pe_walk_relocs_of_symbol()

  generate_reloc()

ei386pe.c: pe_find_data_imports ()

The result was saving about 16 MB of RAM on compiling a c++ lib with about 10000
input bfd (kde2libs/kdeui)

>
> > I have to say one limitation to this. Such cached symbol tables
> > should not be manipulated. They should be used only read only. Is
> > this real ?
>
> Unless you want to modify large parts of bfd then I would say yes.
>
I have looked into the listed functions and with my current knowledge of ld it
seems to me, but I'm not 100% shure.

After applying this I have compiled some kde libs and apps (with the
auto-import-options) without errors,
but I don't know if there are be some side effects. Is a test suite for ld
available to verify this ?

> Cheers
>         Nick
>
>

Re: ld-auto-import memory bug fixing

[Nick Clifton]

Hi Ralf,

> > Err, I do not think that this is quite right.  For example dynamic 
> > sections have their own symbol table, seperate from the normal
> > symbol table.
> How are dynamic sections accessed ?

The same way as any other sections.  Dynamic sections are normal
sections.  It is just that they contain information that is used by
dynamic linkers/loaders to resolve symbols and relocations are load
time instead of link time.


> The result was saving about 16 MB of RAM on compiling a c++ lib with
> about 10000 input bfd (kde2libs/kdeui)

Sounds excellent.

> After applying this I have compiled some kde libs and apps (with the 
> auto-import-options) without errors,
> but I don't know if there are be some side effects. Is a test suite
> for ld available to verify this ?

Indeed there is.  In the ld sources there is a directory called
'testsuite' which contains tests which can be run using a dejagnu /
expect framework.  Tis testsuite can be run by the build target "make
check" in much the same way that gcc can be tested.

Cheers
        Nick

RE: ld-auto-import memory bug fixing

[Ralf Habacker]

Hi,
I have looked into the relating stuff and found, that for any section of any
input bfd a symbol tab is allocated.

For this I have two questions:

1. I recognized, that for any input file read by bfd one symboltab is present,
not one for any section. Isn't it 	?
If this is true (and it seems so, because I have applied a patch and have
compiled about 20 dll's with it), I have
found a way to solve this problem.  (moving symboltable allocating in the first
for loop)

(This patch contains some debugging stuff, which needs other files patch. It is
for dicsussion only)

pe-dll.c
   for (b = info->input_bfds; b; b = b->link_next)
     {
-      arelent **relocs;
-      int relsize, nrelocs, i;
+         int nsyms, symsize;
+         asymbol **symbols;

+         symsize = bfd_get_symtab_upper_bound (b);
+         symbols = (asymbol **) xmalloc (symsize);
+         nsyms = bfd_canonicalize_symtab (b, symbols);
+
       for (s = b->sections; s; s = s->next)
        {
-         asymbol **symbols;
-         int nsyms, symsize;
+      int relsize, nrelocs, i;
+           arelent **relocs;
          int flags = bfd_get_section_flags (b, s);

          /* Skip discarded linkonce sections */
          if (flags & SEC_LINK_ONCE
-             && s->output_section == bfd_abs_section_ptr)
+                     && s->output_section == bfd_abs_section_ptr) {
+                               if (pe_dll_extra_pe_debug & 4)
+                                       printf("\t section->name=%s -->
ignored\n",s->name);
            continue;
-
+                       }
          current_sec=s;

-         symsize = bfd_get_symtab_upper_bound (b);
-         symbols = (asymbol **) xmalloc (symsize);
-         nsyms = bfd_canonicalize_symtab (b, symbols);
-
          relsize = bfd_get_reloc_upper_bound (b, s);
          relocs = (arelent **) xmalloc ((size_t) relsize);
          nrelocs = bfd_canonicalize_reloc (b, s, relocs, symbols);

2. Relating to this bug, I have found that allocating symbol tables is done
several times in ld. Using a symbol
   cache would be save much memory.

			pe_find_data_imports ()
	ei386pe.c:         symsize = bfd_get_symtab_upper_bound (b);

			gld_i386pe_after_open ()
	ei386pe.c:         symsize = bfd_get_symtab_upper_bound (is->the_bfd);

			pe_walk_relocs_of_symbol (info, name, cb)
	pe-dll.c:         symsize = bfd_get_symtab_upper_bound (b);

			process_def_file (abfd, info)
	pe-dll.c:         symsize = bfd_get_symtab_upper_bound (b);

			generate_reloc (abfd, info)
	pe-dll.c:         symsize = bfd_get_symtab_upper_bound (b);

	ldcref.c:      symsize = bfd_get_symtab_upper_bound (abfd);
	ldmain.c:         symsize = bfd_get_symtab_upper_bound (abfd);
	ldmisc.c:                   symsize = bfd_get_symtab_upper_bound (abfd);

What about creating a symbol cache list in ld using pointer to loaded symbols
tables or by extending bfd structure with a pointer to a "symbol cache entry",
which can be filled by client code and later reused by other code.
Extending bfd would minimize the overhead to handle this cache, so I think this
would be the best way.
I have to say one limitation to this. Such cached symbol tables should not be
manipulated. They should be used only read only. Is this real ?

Re: ld-auto-import memory bug fixing

[Christopher Faylor]

On Fri, Sep 07, 2001 at 10:14:04PM +0200, Ralf Habacker wrote:
>Hi,
>
>is there any way to get the size of by malloc allocated byte of an application ?
>If have tried malloc_stats() and mallinfo(), which is described in
>/usr/include/malloc.h
>and implemented in dlmalloc.cc of the cygwin source, but they seems to be not
>defined
>in the cygwin1.dll import library.

dlmalloc is the debugging malloc library that cygwin uses when compiled
with malloc debugging (configure --enable-malloc-debugging).

The normal malloc comes from newlib.

>Another library mmalloc contains mmstats() and mmtrace() which looks equal
>malloc_stats() but using
>this prints nothing. :-(

I don't think that anything in binutils uses mmalloc.

cgf

Re: ld-auto-import memory bug fixing

[Ralf Habacker]

Hi,

is there any way to get the size of by malloc allocated byte of an application ?
If have tried malloc_stats() and mallinfo(), which is described in
/usr/include/malloc.h
and implemented in dlmalloc.cc of the cygwin source, but they seems to be not
defined
in the cygwin1.dll import library.

Another library mmalloc contains mmstats() and mmtrace() which looks equal
malloc_stats() but using
this prints nothing. :-(

Can I use the mmalloc lib with ld to archieve this or is there any other way for
this ?

Ralf

RE: ld-auto-import memory bug fixing

[Ralf Habacker]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 1382 bytes --]

> -----Ursprüngliche Nachricht-----
> Von: nickc@north-pole.nickc.cambridge.redhat.com
> [ mailto:nickc@north-pole.nickc.cambridge.redhat.com]Im Auftrag von Nick
> Clifton
> Gesendet am: Freitag, 7. September 2001 10:43
> An: Ralf Habacker
> Cc: Binutils
> Betreff: Re: ld-auto-import memory bug fixing
>
> Hi Ralf,
>
> > When linking a dll with g++, it calls collect which calls ld, but ld
> > is not in the ps list.
> >
> > Can anynone tell me how I can even though debug ld with gdb ?
>
> The best method would be to capture the command line that collect2
> uses to invoke ld and then reproduce that invocation inside gdb.  If
> you pass -debug to collect2 it should show you the commands that it is
> executing.
Thanks, I will try. With - debug you mean -Wl,--debug on g++ line ?

>
> Another thing you can try is to add print statements to ld.  I know
> that this is not as sophisticated as using gdb, but it is a very
> effective debugging tool.
>
Yes, I have done this too, but on linking big dll's/apps, the output seems to be
buffered, so I can't see in realtime, where ld currently is. I will try to use
fflush after printing.

Debugging state: I have tried to debug ld with a big lib, but unfortunally gdb
need much memory to load all the dll, that ld break very early in linking, so I
can't capture the origin problem.

Any ideas ?

Ralf

> Cheers
>         Nick
>
>

Re: ld-auto-import memory bug fixing

[Nick Clifton]

Hi Ralf,

> When linking a dll with g++, it calls collect which calls ld, but ld
> is not in the ps list.
> 
> Can anynone tell me how I can even though debug ld with gdb ?

The best method would be to capture the command line that collect2
uses to invoke ld and then reproduce that invocation inside gdb.  If
you pass -debug to collect2 it should show you the commands that it is
executing.

Another thing you can try is to add print statements to ld.  I know
that this is not as sophisticated as using gdb, but it is a very
effective debugging tool.

Cheers
        Nick

ld-auto-import memory bug fixing

[Ralf Habacker]

Hi,
currently I'm trying to fix the bug described in
http://sources.redhat.com/ml/binutils/2001-06/msg00742.html
With that I have a question:

When linking a dll with g++, it calls collect which calls ld, but ld is not in
the ps list.

Can anynone tell me how I can even though debug ld with gdb ?

Regards
Ralf

PS: For those, who doesn't know how the auto-import feature works, see below

/************************************************************************

 Auto-import feature by Paul Sokolovsky

 Quick facts:

 1. With this feature on, DLL clients can import variables from DLL
 without any concern from their side (for example, without any source
 code modifications).

 2. This is done completely in bounds of the PE specification (to be fair,
 there's a place where it pokes nose out of, but in practise it works).
 So, resulting module can be used with any other PE compiler/linker.

 3. Auto-import is fully compatible with standard import method and they
 can be mixed together.

 4. Overheads: space: 8 bytes per imported symbol, plus 20 for each
 reference to it; load time: negligible; virtual/physical memory: should be
 less than effect of DLL relocation, and I sincerely hope it doesn't affect
 DLL sharability (too much).

 Idea

 The obvious and only way to get rid of dllimport insanity is to make client
 access variable directly in the DLL, bypassing extra dereference. I.e.,
 whenever client contains someting like

 mov dll_var,%eax,

 address of dll_var in the command should be relocated to point into loaded
 DLL. The aim is to make OS loader do so, and than make ld help with that.
 Import section of PE made following way: there's a vector of structures
 each describing imports from particular DLL. Each such structure points
 to two other parellel vectors: one holding imported names, and one which
 will hold address of corresponding imported name. So, the solution is
 de-vectorize these structures, making import locations be sparse and
 pointing directly into code. Before continuing, it is worth a note that,
 while authors strives to make PE act ELF-like, there're some other people
 make ELF act PE-like: elfvector, ;-) .

 Implementation

 For each reference of data symbol to be imported from DLL (to set of which
 belong symbols with name <sym>, if __imp_<sym> is found in implib), the
 import fixup entry is generated. That entry is of type
 IMAGE_IMPORT_DESCRIPTOR and stored in .idata$3 subsection. Each
 fixup entry contains pointer to symbol's address within .text section
 (marked with __fuN_<sym> symbol, where N is integer), pointer to DLL name
 (so, DLL name is referenced by multiple entries), and pointer to symbol
 name thunk. Symbol name thunk is singleton vector (__nm_th_<symbol>)
 pointing to IMAGE_IMPORT_BY_NAME structure (__nm_<symbol>) directly
 containing imported name. Here comes that "om the edge" problem mentioned
 above: PE specification rambles that name vector (OriginalFirstThunk)
 should run in parallel with addresses vector (FirstThunk), i.e. that they
 should have same number of elements and terminated with zero. We violate
 this, since FirstThunk points directly into machine code. But in practise,
 OS loader implemented the sane way: it goes thru OriginalFirstThunk and
 puts addresses to FirstThunk, not something else. It once again should be
 noted that dll and symbol name structures are reused across fixup entries
 and should be there anyway to support standard import stuff, so sustained
 overhead is 20 bytes per reference. Other question is whether having several
 IMAGE_IMPORT_DESCRIPTORS for the same DLL is possible. Answer is yes, it is
 done even by native compiler/linker (libth32's functions are in fact reside
 in windows9x kernel32.dll, so if you use it, you have two
 IMAGE_IMPORT_DESCRIPTORS for kernel32.dll). Yet other question is whether
 referencing the same PE structures several times is valid. The answer is why
 not, prohibitting that (detecting violation) would require more work on
 behalf of loader than not doing it.