static executable bloat

static executable bloat

[Zack Weinberg]

H.J. Lu wrote:
> > This is what I'm trying to solve:
> > 
> > $ cat t1.c
> > int main(void) { return 0; }
> > $ cat t2.c
> > int main(void) { return 0; }
> > 
> > void abort(void) { while(1); }
> > void atexit(void *x) { return; }
> > void exit(int x) { _exit(x); }
> > 
> > void *__libc_stack_end;
> > $ gcc -static t1.c -o t1
> > $ gcc -static t2.c -o t2
> > $ ls -l t1 t2
> > -rwxrwxr-x   1 zack     zack       222102 Aug 18 09:12 t1
> > -rwxrwxr-x   1 zack     zack         6215 Aug 18 09:12 t2
> > 
> > I want t1 to not be 220k.  The problem appears to be that the
> > implementations of abort and atexit suck in all kinds of stuff
> > transitively - mainly via libio.  I thought, late last night, that it
> > had to do with pthreads being referenced despite those being weak
> > symbols, but I was wrong.
> > 
> 
> Add -Wl,-Map,map. You should see what/why they are included in "map".

Yes, that works beautifully, thank you.

The problem is nothing to do with weak symbols.  It's very simple:

- abort flushes all open FILEs, so it pulls in stdio
- atexit and exit refer to malloc, and the malloc checking code pulls
  in stdio
- __libc_stack_end is defined only in dl-support.c, which pulls in the
  rest of the static libdl, which uses stdio

and

- stdio is 100k by itself
- the wide char support in stdio references the gconv infrastructure and
  the static libdl, which (with everything that uses) is another 100k

I spent a lot of time today bashing at this, but there is no easy fix.
I guess this is just a "libio should be smaller" whine.  Note that an
awful lot of places in there know when a stream is wide; it seems to
me that this ought to be hidden in the jump table.  There's no reason
why not to redirect _IO_setb instead of having _IO_setb and _IO_wsetb,
for example.

zw

Re: static executable bloat

[Andreas Schwab]

Zack Weinberg <zack@bitmover.com> writes:

|> I spent a lot of time today bashing at this, but there is no easy fix.
|> I guess this is just a "libio should be smaller" whine.  Note that an
|> awful lot of places in there know when a stream is wide; it seems to
|> me that this ought to be hidden in the jump table.  There's no reason
|> why not to redirect _IO_setb instead of having _IO_setb and _IO_wsetb,
|> for example.

The wide stream stuff isn't actually that big.  I was only able to reduce
the size by about 8.3K by leaving out the wide stream support (see the
attached patch for how I did that).  The rest from libio is mostly the
printf stuff that is pulled in by malloc.

A much bigger problem is the gconv stuff, which is indirectly referenced
in strtol via btowc.  The biggest single object file that is included is
regex.o, which is referenced by gconv_db.c.  I don't see a way to reduce
this.

Andreas.

Re: static executable bloat

[Roland McGrath]

> A much bigger problem is the gconv stuff, which is indirectly referenced
> in strtol via btowc.  The biggest single object file that is included is
> regex.o, which is referenced by gconv_db.c.  I don't see a way to reduce
> this.

Originally all this stuff was avoided completely in a static program that
doesn't call setlocale.

Re: static executable bloat

[Ulrich Drepper]

Roland McGrath <roland@frob.com> writes:

> Originally all this stuff was avoided completely in a static program that
> doesn't call setlocale.

I never saw that your clever hacks using weak in the locale data parts
ever worked.  Therefore I remove the traces of it quite some time
back.

-- 
---------------.      drepper at gnu.org  ,-.   1325 Chesapeake Terrace
Ulrich Drepper  \    ,-------------------'   \  Sunnyvale, CA 94089 USA
Cygnus Solutions `--' drepper at cygnus.com   `------------------------

Re: static executable bloat

[Roland McGrath]

> Roland McGrath <roland@frob.com> writes:
> 
> > Originally all this stuff was avoided completely in a static program that
> > doesn't call setlocale.
> 
> I never saw that your clever hacks using weak in the locale data parts
> ever worked.  Therefore I remove the traces of it quite some time
> back.

They certainly did work (obviously only for static linking) when I stopped
maintaining libc.  It should not be very difficult to make it work, it's
not all that complicated.

Re: static executable bloat

[Ulrich Drepper]

Roland McGrath <roland@frob.com> writes:

> They certainly did work (obviously only for static linking) when I stopped
> maintaining libc.  It should not be very difficult to make it work, it's
> not all that complicated.

No, it's not easy.  I don't remember exactly anymore want are the
exact problems but the locale loading code always had problems which
doing only parts of the work.  You might remember, there was code to
prevent loading entire categories if they are not used.  This never
really worked but at that time nobody used the code for real and so
the problems were not discovered.

I personally don't care at all how big static binaries are.  Nobody
must use them except for very few exceptions and for those larger code
size is ok.

-- 
---------------.      drepper at gnu.org  ,-.   1325 Chesapeake Terrace
Ulrich Drepper  \    ,-------------------'   \  Sunnyvale, CA 94089 USA
Cygnus Solutions `--' drepper at cygnus.com   `------------------------

Re: static executable bloat

[Thorsten Kukuk]

On Thu, Aug 19, Ulrich Drepper wrote:

> Roland McGrath <roland@frob.com> writes:
> 
> > They certainly did work (obviously only for static linking) when I stopped
> > maintaining libc.  It should not be very difficult to make it work, it's
> > not all that complicated.
> 
> No, it's not easy.  I don't remember exactly anymore want are the
> exact problems but the locale loading code always had problems which
> doing only parts of the work.  You might remember, there was code to
> prevent loading entire categories if they are not used.  This never
> really worked but at that time nobody used the code for real and so
> the problems were not discovered.
> 
> I personally don't care at all how big static binaries are.  Nobody
> must use them except for very few exceptions and for those larger code
> size is ok.

Sorry, you are absolutly wrong. I know some cases where you need static
binaries and were the larger code size is not Ok. We have the problem
at the moment here, going back to libc5 or glibc 2.0 is no solution.
But it seems we have to maintain our own hacked libc5 if we don't find
a better solution.
Not all systems have plenty of disk space.

  Thorsten

-- 
Thorsten Kukuk       http://www.suse.de/~kukuk/       kukuk@suse.de
SuSE GmbH            Schanzaeckerstr. 10            90443 Nuernberg
Linux is like a Vorlon.  It is incredibly powerful, gives terse,
cryptic answers and has a lot of things going on in the background.

Re: static executable bloat

[Ulrich Drepper]

Thorsten Kukuk <kukuk@suse.de> writes:

> Sorry, you are absolutly wrong.

This is your opinion.  People are not installing from floppies
anymore, they have CDROMs.

-- 
---------------.      drepper at gnu.org  ,-.   1325 Chesapeake Terrace
Ulrich Drepper  \    ,-------------------'   \  Sunnyvale, CA 94089 USA
Cygnus Solutions `--' drepper at cygnus.com   `------------------------

Re: static executable bloat

[Thorsten Kukuk]

On Thu, Aug 19, Ulrich Drepper wrote:

> Thorsten Kukuk <kukuk@suse.de> writes:
> 
> > Sorry, you are absolutly wrong.
> 
> This is your opinion.  People are not installing from floppies
> anymore, they have CDROMs.

Sorry, but do you really life in this world ? Most of my computers
don't have a CDROM drive. My Router for example don't need one. From
my PCs with a CDROM only one could boot from it. Yes, new PCs have
very big disks, have a CDROM and could boot from this CDROM. But
most of the Hardare Linux is running on is old and couldn't boot from
the CDROM. And do you know what the bootsektor of a CDROM is ? 
A 1,44MB Floppy Image. Using a 2.88 MB Floppy Image isn't a solution
because most CDROM drives could not boot from it.

 Thorsten

-- 
Thorsten Kukuk       http://www.suse.de/~kukuk/       kukuk@suse.de
SuSE GmbH            Schanzaeckerstr. 10            90443 Nuernberg
Linux is like a Vorlon.  It is incredibly powerful, gives terse,
cryptic answers and has a lot of things going on in the background.

Re: static executable bloat

[Ulrich Drepper]

Thorsten Kukuk <kukuk@suse.de> writes:

> And do you know what the bootsektor of a CDROM is ? 
> A 1,44MB Floppy Image. Using a 2.88 MB Floppy Image isn't a solution
> because most CDROM drives could not boot from it.

You don't have to boot frmo the CDROM.  And if you need a router you
have a network and therefore can connect to another machine.

-- 
---------------.      drepper at gnu.org  ,-.   1325 Chesapeake Terrace
Ulrich Drepper  \    ,-------------------'   \  Sunnyvale, CA 94089 USA
Cygnus Solutions `--' drepper at cygnus.com   `------------------------

Re: static executable bloat

[scottb]

I have to agree with Thorsten.  The size of executables matters.
I have to squeeze a bootable image for thin clients into 4 MB.  It was 
far easier with the older a.out tools I used to use with libc4.  I
don't want to go back, but I do wish there was a finer granularity for 
the linker to strip code.  For instance:

#include <stdio.h>
int main(int argc, char **argv)
{
  printf ("Hello World\n");
}

compiles to 985,018 bytes.  This is almost all libc overhead.
Most of which is never going to be needed.

Scott

Re: static executable bloat

[Philip Blundell]

>#include <stdio.h>
>int main(int argc, char **argv)
>{
>  printf ("Hello World\n");
>}
>
>compiles to 985,018 bytes.  This is almost all libc overhead.
>Most of which is never going to be needed.

Yes, it is a bit sad.  Somewhere I have some half-baked patches that allow you 
to stub out the majority of the gconv and wchar code at configure time; I did 
this for an internal project where the resulting binaries had to fit into ROM.

The other thing I looked at was dropping the stdio implementation from newlib 
(which is far less featureful but also a whole lot smaller) into libc.  I 
think I got that to mostly work and again I can probably dig out some patches 
if anyone is interested.

p.

Re: static executable bloat

[H.J. Lu]

> 
> >#include <stdio.h>
> >int main(int argc, char **argv)
> >{
> >  printf ("Hello World\n");
> >}
> >
> >compiles to 985,018 bytes.  This is almost all libc overhead.
> >Most of which is never going to be needed.
> 
> Yes, it is a bit sad.  Somewhere I have some half-baked patches that allow you 
> to stub out the majority of the gconv and wchar code at configure time; I did 
> this for an internal project where the resulting binaries had to fit into ROM.
> 
> The other thing I looked at was dropping the stdio implementation from newlib 
> (which is far less featureful but also a whole lot smaller) into libc.  I 
> think I got that to mostly work and again I can probably dig out some patches 
> if anyone is interested.
> 

May I ask if it is ok to add some flags to get a smaller static
binary without gconv and/or wchar support? I am thinking to write
libNoGconv.a and libNoWchar.a. People can stub out gconv and/or wchar
with

# gcc -static .... -lNoGconv -lNoWchar

Will that work for people who want smaller static binaries or do you
want smaller static binaries with supports for everything?


-- 
H.J. Lu (hjl@gnu.org)

Re: static executable bloat

[Ulrich Drepper]

hjl@lucon.org (H.J. Lu) writes:

> May I ask if it is ok to add some flags to get a smaller static
> binary without gconv and/or wchar support? I am thinking to write
> libNoGconv.a and libNoWchar.a. People can stub out gconv and/or wchar
> with
> 
> # gcc -static .... -lNoGconv -lNoWchar

With this approach no libc change would be necessary so it is of
course ok with me.

-- 
---------------.      drepper at gnu.org  ,-.   1325 Chesapeake Terrace
Ulrich Drepper  \    ,-------------------'   \  Sunnyvale, CA 94089 USA
Cygnus Solutions `--' drepper at cygnus.com   `------------------------

Re: static executable bloat

[H.J. Lu]

> 
> hjl@lucon.org (H.J. Lu) writes:
> 
> > May I ask if it is ok to add some flags to get a smaller static
> > binary without gconv and/or wchar support? I am thinking to write
> > libNoGconv.a and libNoWchar.a. People can stub out gconv and/or wchar
> > with
> > 
> > # gcc -static .... -lNoGconv -lNoWchar
> 
> With this approach no libc change would be necessary so it is of
> course ok with me.
> 

I guessed you wouldn't object too much. But I am not sure if it will do
for those people who want smaller static binaries.

BTW, I want to change Makefiles in libc to build libNoXXX.a. But no
source code changes in libc. I will just add those stub functions
for libNoXXX.a.

Thanks.

-- 
H.J. Lu (hjl@gnu.org)

Re: static executable bloat

[Ulrich Drepper]

hjl@lucon.org (H.J. Lu) writes:

> BTW, I want to change Makefiles in libc to build libNoXXX.a. But no
> source code changes in libc. I will just add those stub functions
> for libNoXXX.a.

You can put this in an add-on.

-- 
---------------.      drepper at gnu.org  ,-.   1325 Chesapeake Terrace
Ulrich Drepper  \    ,-------------------'   \  Sunnyvale, CA 94089 USA
Cygnus Solutions `--' drepper at cygnus.com   `------------------------

Re: static executable bloat

[Philip Blundell]

>May I ask if it is ok to add some flags to get a smaller static
>binary without gconv and/or wchar support? I am thinking to write
>libNoGconv.a and libNoWchar.a. People can stub out gconv and/or wchar
>with
>
># gcc -static .... -lNoGconv -lNoWchar
>
>Will that work for people who want smaller static binaries or do you
>want smaller static binaries with supports for everything?

That would certainly have been OK for my situation.  I think that if you want 
smaller binaries you have to be prepared to sacrifice some functionality.

p.

Re: static executable bloat

[Cristian Gafton]

On 19 Aug 1999, Ulrich Drepper wrote:

> > And do you know what the bootsektor of a CDROM is ? 
> > A 1,44MB Floppy Image. Using a 2.88 MB Floppy Image isn't a solution
> > because most CDROM drives could not boot from it.
> 
> You don't have to boot frmo the CDROM.

So, one has Windows and wants to install Linux. Do you expect me to write
a Windows App to install Linux becvause glibc is big?!

>  And if you need a router you
> have a network and therefore can connect to another machine.

... booting from a floppy disk, right?

Cristian
--
----------------------------------------------------------------------
Cristian Gafton     --     gafton@redhat.com      --     Red Hat, Inc.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  "How could this be a problem in a country where we have Intel and 
   Microsoft?"  --Al Gore on Y2K

Re: static executable bloat

[Cristian Gafton]

On Fri, 20 Aug 1999, Thorsten Kukuk wrote:

> the CDROM. And do you know what the bootsektor of a CDROM is ? 
> A 1,44MB Floppy Image. Using a 2.88 MB Floppy Image isn't a solution
> because most CDROM drives could not boot from it.

And in general you'll need a BIOS that is less than 18 months old.

Cristian
--
----------------------------------------------------------------------
Cristian Gafton     --     gafton@redhat.com      --     Red Hat, Inc.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  "How could this be a problem in a country where we have Intel and 
   Microsoft?"  --Al Gore on Y2K

Re: static executable bloat

[Cristian Gafton]

On 19 Aug 1999, Ulrich Drepper wrote:

> > Sorry, you are absolutly wrong.
> 
> This is your opinion.  People are not installing from floppies
> anymore, they have CDROMs.

I happen to agree with thorsten on this one - as one that has been thorugh
this ordeal several times now: even on CD-ROM, you *have* to use a floppy
image to boot from it, and that floppy image has only that much space on
it. One can not just automagically boot from CD-Rom

So no, CD-ROM is not the answer if you are talking about a bootable
CD-Rom.

Cristian
--
----------------------------------------------------------------------
Cristian Gafton     --     gafton@redhat.com      --     Red Hat, Inc.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  "How could this be a problem in a country where we have Intel and 
   Microsoft?"  --Al Gore on Y2K

Re: static executable bloat

[Geoff Keating]

> From: Ulrich Drepper <drepper@cygnus.com>
> Date: 19 Aug 1999 11:02:02 -0700

> I personally don't care at all how big static binaries are.  Nobody
> must use them except for very few exceptions and for those larger code
> size is ok.

If people really want small static binaries, I'd suggest newlib;
that's one of the reasons it exists.

-- 
Geoffrey Keating <geoffk@cygnus.com>

Re: static executable bloat

[Ulrich Drepper]

Geoff Keating <geoffk@ozemail.com.au> writes:

> If people really want small static binaries, I'd suggest newlib;
> that's one of the reasons it exists.

I'm not sure.  Mostly those guys want to use it for programs like rpm
or equivalent.  But these programs require completel
internationalization (at least everybody wants to do it in the next
release).  So they want to have tiny little binaries with the full
functionality.

-- 
---------------.      drepper at gnu.org  ,-.   1325 Chesapeake Terrace
Ulrich Drepper  \    ,-------------------'   \  Sunnyvale, CA 94089 USA
Cygnus Solutions `--' drepper at cygnus.com   `------------------------

Re: static executable bloat

[Zack Weinberg]

Andreas Schwab wrote:
> 
> --=-=-=
> 
> Zack Weinberg <zack@bitmover.com> writes:
> 
> |> I spent a lot of time today bashing at this, but there is no easy fix.
> |> I guess this is just a "libio should be smaller" whine.  Note that an
> |> awful lot of places in there know when a stream is wide; it seems to
> |> me that this ought to be hidden in the jump table.  There's no reason
> |> why not to redirect _IO_setb instead of having _IO_setb and _IO_wsetb,
> |> for example.
> 
> The wide stream stuff isn't actually that big.  I was only able to reduce
> the size by about 8.3K by leaving out the wide stream support (see the
> attached patch for how I did that).  The rest from libio is mostly the
> printf stuff that is pulled in by malloc.
> 
> A much bigger problem is the gconv stuff, which is indirectly referenced
> in strtol via btowc.  The biggest single object file that is included is
> regex.o, which is referenced by gconv_db.c.  I don't see a way to reduce
> this.

It's fairly easy to prevent strtol from referencing btowc, and I think
the change is permissible by the standard.  You just use 
_NL_CURRENT (LC_NUMERIC, THOUSANDS_SEP) directly unless compiling
wcstol and friends.

But if you do that, you find that stdio does the same sort of thing
all over the place - mainly in the wide stream support, which is why I
was grousing about them, but printf too.  You might look at
wcsmbsload.c, that seems to be the trigger for the inclusion of gconv.

zw