Decreasing size of final executable

Decreasing size of final executable

[Jędrzej Dudkiewicz]

[-- Attachment #1: Type: text/plain, Size: 3582 bytes --]

This is basically the same as
https://stackoverflow.com/questions/76580329/hide-symbols-from-executable
I pasted it below in its entirety.

If this is primarily a binutils question I will paste it there, but I
*think* it is mostly a gcc related question.

------8< ------8< ------8< ------8< ------8< ------8< ------8< ------
<https://stackoverflow.com/posts/76580329/timeline>

I use gcc 9.2.0 to compile c++ program for arm. I use boost 1.71.0 (mainly
asio). My final executable works fine, but even after stripping readelf -W
prog --dyn-syms shows e.g. these entries:

   904: 000dc518     1 OBJECT  UNIQUE DEFAULT   25
_ZN5boost4asio6detail30execution_context_service_baseINS1_16resolver_serviceINS0_2ip3tcpEEEE2idE
   912: 000dc498     1 OBJECT  UNIQUE DEFAULT   25
_ZN5boost4asio6detail30execution_context_service_baseINS1_18signal_set_serviceEE2idE
   943: 000dc46c     4 OBJECT  UNIQUE DEFAULT   25
_ZN5boost4asio6detail17posix_global_implINS0_14system_contextEE11static_ptr_E
   951: 000dc4b0     1 OBJECT  UNIQUE DEFAULT   25
_ZN5boost4asio6detail12service_baseINS1_14strand_serviceEE2idE
   990: 000dc48c     4 OBJECT  UNIQUE DEFAULT   25
_ZGVN5boost4asio6detail10call_stackINS1_14strand_service11strand_implEhE4top_E
  1005: 0009a484    38 OBJECT  WEAK   DEFAULT   15
_ZTSN5boost4asio22service_already_existsE
  1006: 000a3808    87 OBJECT  WEAK   DEFAULT   15
_ZTSN5boost16exception_detail10clone_implINS0_19error_info_injectorINS_12bad_any_castEEEEE
  1024: 0009ad9c    65 OBJECT  WEAK   DEFAULT   15
_ZTSN5boost16exception_detail19error_info_injectorISt11logic_errorEE
  1038: 0009ee34   128 OBJECT  WEAK   DEFAULT   15
_ZTSN5boost4asio6detail30execution_context_service_baseINS1_22deadline_timer_serviceINS0_11time_traitsINS_10posix_time5ptimeEEEEEEE
  1045: 0009ade0    84 OBJECT  WEAK   DEFAULT   15
_ZTSN5boost16exception_detail10clone_implINS0_19error_info_injectorISt11logic_errorEEEE

As far as I can tell, these symbols are known to the compiler/linker, as
their Ndx is not UND (so undefined), which means their presence in the
symbol table is not necessary. Since the whole program has 690kB and as
strings shows these entries in the symbol table take more or less 140kB,
I'd like to remove them.

Currently I am compiling using the following compiler/linker options:
-fvisibility=hidden -fvisibility-inlines-hidden -flto -Wl,--exclude-libs,ALL
-Wl,-r,--discard-all -Wl,-flto

Without these options the binary is >1.3MB.

I have two questions:

   1. Why are these symbols WEAK? These symbols are not from an external
   library (static or dynamic), they come from templates (so from include
   files) and there is no need for them to be defined as WEAK as far as I can
   tell.
   2. How can I remove them from the symbols table?

Note: I neither want nor can define visibility by hand - modifying boost to
this end is tedious, also if we ever upgrade it I'd have to do it again. So
it is not an option.

Using objcopy with any option (-x, --localize-hidden, --strip-unneeded)
does nothing, as the symbol table is not modified whatsoever. I guess all
these functions/methods are used. I guess they must be marked differently
by linker (ld in my case), otherwise nothing can be done as they are
strictly required.

------8< ------8< ------8< ------8< ------8< ------8< ------8< ------
<https://stackoverflow.com/posts/76580329/timeline>

-- 
Jędrzej Dudkiewicz

I really hate this damn machine, I wish that they would sell it.
It never does just what I want, but only what I tell it.

Re: Decreasing size of final executable

[Xi Ruoyao]

On Fri, 2023-06-30 at 08:28 +0200, Jędrzej Dudkiewicz via Gcc-help wrote:
> This is basically the same as
> https://stackoverflow.com/questions/76580329/hide-symbols-from-executable
> I pasted it below in its entirety.
> 
> If this is primarily a binutils question I will paste it there, but I
> *think* it is mostly a gcc related question.
> 
> ------8< ------8< ------8< ------8< ------8< ------8< ------8< ------
> <https://stackoverflow.com/posts/76580329/timeline>
> 
> I use gcc 9.2.0 to compile c++ program for arm. I use boost 1.71.0 (mainly
> asio). My final executable works fine, but even after stripping readelf -W
> prog --dyn-syms shows e.g. these entries:
> 
>    904: 000dc518     1 OBJECT  UNIQUE DEFAULT   25
> _ZN5boost4asio6detail30execution_context_service_baseINS1_16resolver_serviceINS0_2ip3tcpEEEE2idE
>    912: 000dc498     1 OBJECT  UNIQUE DEFAULT   25
> _ZN5boost4asio6detail30execution_context_service_baseINS1_18signal_set_serviceEE2idE
>    943: 000dc46c     4 OBJECT  UNIQUE DEFAULT   25
> _ZN5boost4asio6detail17posix_global_implINS0_14system_contextEE11static_ptr_E
>    951: 000dc4b0     1 OBJECT  UNIQUE DEFAULT   25
> _ZN5boost4asio6detail12service_baseINS1_14strand_serviceEE2idE
>    990: 000dc48c     4 OBJECT  UNIQUE DEFAULT   25
> _ZGVN5boost4asio6detail10call_stackINS1_14strand_service11strand_implEhE4top_E
>   1005: 0009a484    38 OBJECT  WEAK   DEFAULT   15
> _ZTSN5boost4asio22service_already_existsE
>   1006: 000a3808    87 OBJECT  WEAK   DEFAULT   15
> _ZTSN5boost16exception_detail10clone_implINS0_19error_info_injectorINS_12bad_any_castEEEEE
>   1024: 0009ad9c    65 OBJECT  WEAK   DEFAULT   15
> _ZTSN5boost16exception_detail19error_info_injectorISt11logic_errorEE
>   1038: 0009ee34   128 OBJECT  WEAK   DEFAULT   15
> _ZTSN5boost4asio6detail30execution_context_service_baseINS1_22deadline_timer_serviceINS0_11time_traitsINS_10posix_time5ptimeEEEEEEE
>   1045: 0009ade0    84 OBJECT  WEAK   DEFAULT   15
> _ZTSN5boost16exception_detail10clone_implINS0_19error_info_injectorISt11logic_errorEEEE
> 
> As far as I can tell, these symbols are known to the compiler/linker, as
> their Ndx is not UND (so undefined), which means their presence in the
> symbol table is not necessary. Since the whole program has 690kB and as
> strings shows these entries in the symbol table take more or less 140kB,
> I'd like to remove them.
> 
> Currently I am compiling using the following compiler/linker options:
> -fvisibility=hidden -fvisibility-inlines-hidden -flto -Wl,--exclude-libs,ALL
> -Wl,-r,--discard-all -Wl,-flto

Hmm, isn't "-r" for partial linking?  How can you use it to link an
executable?

-- 
Xi Ruoyao <xry111@xry111.site>
School of Aerospace Science and Technology, Xidian University

Re: Decreasing size of final executable

[Jędrzej Dudkiewicz]

[-- Attachment #1: Type: text/plain, Size: 3355 bytes --]

On Fri, Jun 30, 2023 at 9:11 AM Xi Ruoyao <xry111@xry111.site> wrote:

> On Fri, 2023-06-30 at 08:28 +0200, Jędrzej Dudkiewicz via Gcc-help wrote:
> > This is basically the same as
> >
> https://stackoverflow.com/questions/76580329/hide-symbols-from-executable
> > I pasted it below in its entirety.
> >
> > If this is primarily a binutils question I will paste it there, but I
> > *think* it is mostly a gcc related question.
> >
> > ------8< ------8< ------8< ------8< ------8< ------8< ------8< ------
> > <https://stackoverflow.com/posts/76580329/timeline>
> >
> > I use gcc 9.2.0 to compile c++ program for arm. I use boost 1.71.0
> (mainly
> > asio). My final executable works fine, but even after stripping readelf
> -W
> > prog --dyn-syms shows e.g. these entries:
> >
> >    904: 000dc518     1 OBJECT  UNIQUE DEFAULT   25
> >
> _ZN5boost4asio6detail30execution_context_service_baseINS1_16resolver_serviceINS0_2ip3tcpEEEE2idE
> >    912: 000dc498     1 OBJECT  UNIQUE DEFAULT   25
> >
> _ZN5boost4asio6detail30execution_context_service_baseINS1_18signal_set_serviceEE2idE
> >    943: 000dc46c     4 OBJECT  UNIQUE DEFAULT   25
> >
> _ZN5boost4asio6detail17posix_global_implINS0_14system_contextEE11static_ptr_E
> >    951: 000dc4b0     1 OBJECT  UNIQUE DEFAULT   25
> > _ZN5boost4asio6detail12service_baseINS1_14strand_serviceEE2idE
> >    990: 000dc48c     4 OBJECT  UNIQUE DEFAULT   25
> >
> _ZGVN5boost4asio6detail10call_stackINS1_14strand_service11strand_implEhE4top_E
> >   1005: 0009a484    38 OBJECT  WEAK   DEFAULT   15
> > _ZTSN5boost4asio22service_already_existsE
> >   1006: 000a3808    87 OBJECT  WEAK   DEFAULT   15
> >
> _ZTSN5boost16exception_detail10clone_implINS0_19error_info_injectorINS_12bad_any_castEEEEE
> >   1024: 0009ad9c    65 OBJECT  WEAK   DEFAULT   15
> > _ZTSN5boost16exception_detail19error_info_injectorISt11logic_errorEE
> >   1038: 0009ee34   128 OBJECT  WEAK   DEFAULT   15
> >
> _ZTSN5boost4asio6detail30execution_context_service_baseINS1_22deadline_timer_serviceINS0_11time_traitsINS_10posix_time5ptimeEEEEEEE
> >   1045: 0009ade0    84 OBJECT  WEAK   DEFAULT   15
> >
> _ZTSN5boost16exception_detail10clone_implINS0_19error_info_injectorISt11logic_errorEEEE
> >
> > As far as I can tell, these symbols are known to the compiler/linker, as
> > their Ndx is not UND (so undefined), which means their presence in the
> > symbol table is not necessary. Since the whole program has 690kB and as
> > strings shows these entries in the symbol table take more or less 140kB,
> > I'd like to remove them.
> >
> > Currently I am compiling using the following compiler/linker options:
> > -fvisibility=hidden -fvisibility-inlines-hidden -flto
> -Wl,--exclude-libs,ALL
> > -Wl,-r,--discard-all -Wl,-flto
>
> Hmm, isn't "-r" for partial linking?  How can you use it to link an
> executable?
>

Well, final binary works so I suppose it simply makes no difference. Reason
for it being here is simple: I've been trying basically all combinations of
options that according to The Experts Of The Internet should somehow make
my binary smaller. Since this is combination that ended with smallest
binary.
-- 
Jędrzej Dudkiewicz

I really hate this damn machine, I wish that they would sell it.
It never does just what I want, but only what I tell it.

Re: Decreasing size of final executable

[Xi Ruoyao]

On Fri, 2023-06-30 at 09:29 +0200, Jędrzej Dudkiewicz wrote:
> > > Currently I am compiling using the following compiler/linker options:
> > > -fvisibility=hidden -fvisibility-inlines-hidden -flto -Wl,--exclude-libs,ALL
> > > -Wl,-r,--discard-all -Wl,-flto
> > 
> > Hmm, isn't "-r" for partial linking?  How can you use it to link an
> > executable?
> 
> Well, final binary works so I suppose it simply makes no difference.
> Reason for it being here is simple: I've been trying basically all
> combinations of options that according to The Experts Of The Internet
> should somehow make my binary smaller. Since this is combination that
> ended with smallest binary.

I've no idea why it even works in your case.  When I try them for a
Hello World program, the produced "a.out" file is an ELF relocatable and
I cannot run it at all.

-- 
Xi Ruoyao <xry111@xry111.site>
School of Aerospace Science and Technology, Xidian University

Re: Decreasing size of final executable

[Jędrzej Dudkiewicz]

[-- Attachment #1: Type: text/plain, Size: 1419 bytes --]

On Fri, Jun 30, 2023 at 9:34 AM Xi Ruoyao <xry111@xry111.site> wrote:

> On Fri, 2023-06-30 at 09:29 +0200, Jędrzej Dudkiewicz wrote:
> > > > Currently I am compiling using the following compiler/linker options:
> > > > -fvisibility=hidden -fvisibility-inlines-hidden -flto
> -Wl,--exclude-libs,ALL
> > > > -Wl,-r,--discard-all -Wl,-flto
> > >
> > > Hmm, isn't "-r" for partial linking?  How can you use it to link an
> > > executable?
> >
> > Well, final binary works so I suppose it simply makes no difference.
> > Reason for it being here is simple: I've been trying basically all
> > combinations of options that according to The Experts Of The Internet
> > should somehow make my binary smaller. Since this is combination that
> > ended with smallest binary.
>
> I've no idea why it even works in your case.  When I try them for a
> Hello World program, the produced "a.out" file is an ELF relocatable and
> I cannot run it at all.
>

You are right, thank you. I checked it twice and this option simply wasn't
passed to the linker. I removed both -r and -Ur from linker options and it
builds again - final binary is slightly slower, but main problem (symbols
from boost::asio) still persist - if this is a problem at all, of course.
-- 
Jędrzej Dudkiewicz

I really hate this damn machine, I wish that they would sell it.
It never does just what I want, but only what I tell it.

Re: Decreasing size of final executable

[Jędrzej Dudkiewicz]

[-- Attachment #1: Type: text/plain, Size: 1606 bytes --]

On Fri, Jun 30, 2023 at 10:02 AM Jędrzej Dudkiewicz <
jedrzej.dudkiewicz@gmail.com> wrote:

> On Fri, Jun 30, 2023 at 9:34 AM Xi Ruoyao <xry111@xry111.site> wrote:
>
>> On Fri, 2023-06-30 at 09:29 +0200, Jędrzej Dudkiewicz wrote:
>> > > > Currently I am compiling using the following compiler/linker
>> options:
>> > > > -fvisibility=hidden -fvisibility-inlines-hidden -flto
>> -Wl,--exclude-libs,ALL
>> > > > -Wl,-r,--discard-all -Wl,-flto
>> > >
>> > > Hmm, isn't "-r" for partial linking?  How can you use it to link an
>> > > executable?
>> >
>> > Well, final binary works so I suppose it simply makes no difference.
>> > Reason for it being here is simple: I've been trying basically all
>> > combinations of options that according to The Experts Of The Internet
>> > should somehow make my binary smaller. Since this is combination that
>> > ended with smallest binary.
>>
>> I've no idea why it even works in your case.  When I try them for a
>> Hello World program, the produced "a.out" file is an ELF relocatable and
>> I cannot run it at all.
>>
>
> You are right, thank you. I checked it twice and this option simply wasn't
> passed to the linker. I removed both -r and -Ur from linker options and it
> builds again - final binary is slightly slower, but main problem (symbols
> from boost::asio) still persist - if this is a problem at all, of course.
>

Not slower, smaller. Sorry for the confusion.

-- 
Jędrzej Dudkiewicz

I really hate this damn machine, I wish that they would sell it.
It never does just what I want, but only what I tell it.

Re: Decreasing size of final executable

[Andrew Haley]

On 6/30/23 07:28, Jędrzej Dudkiewicz via Gcc-help wrote:
> This is basically the same as
> https://stackoverflow.com/questions/76580329/hide-symbols-from-executable
> I pasted it below in its entirety.
> 
> If this is primarily a binutils question I will paste it there, but I
> *think*  it is mostly a gcc related question.

You don't say whether you are using -ffunction-sections.

-- 
Andrew Haley  (he/him)
Java Platform Lead Engineer
Red Hat UK Ltd. <https://www.redhat.com>
https://keybase.io/andrewhaley
EAC8 43EB D3EF DB98 CC77 2FAD A5CD 6035 332F A671

Re: Decreasing size of final executable

[Jędrzej Dudkiewicz]

On Sat, Jul 1, 2023 at 5:34 PM Andrew Haley via Gcc-help
<gcc-help@gcc.gnu.org> wrote:
>
> On 6/30/23 07:28, Jędrzej Dudkiewicz via Gcc-help wrote:
> > This is basically the same as
> > https://stackoverflow.com/questions/76580329/hide-symbols-from-executable
> > I pasted it below in its entirety.
> >
> > If this is primarily a binutils question I will paste it there, but I
> > *think*  it is mostly a gcc related question.
>
> You don't say whether you are using -ffunction-sections.

Andrew,

Thank you for responding. I have added `-ffunction-sections` and
`-fdata-sections` and also `-Wl,--gc-sections` to the linker. This did
nothing (size is identical as without these switches).

Let's try a different way:

`arm-linux-gnueabihf-nm -C prog` shows for example:

mqtt::client<mqtt::tcp_endpoint<boost::asio::ssl::stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp,
boost::asio::executor> >, boost::asio::io_context::strand>,
2u>::async_connect_impl(std::vector<boost::variant<mqtt::v5::property::payload_format_indicator,
mqtt::v5::property::message_expiry_interval,
mqtt::v5::property::content_type, mqtt::v5::property::response_topic,
mqtt::v5::property::correlation_data,
mqtt::v5::property::subscription_identifier,
mqtt::v5::property::session_expiry_interval,
mqtt::v5::property::assigned_client_identifier,
mqtt::v5::property::server_keep_alive,
mqtt::v5::property::authentication_method,
mqtt::v5::property::authentication_data,
mqtt::v5::property::request_problem_information,
mqtt::v5::property::will_delay_interval,
mqtt::v5::property::request_response_information,
mqtt::v5::property::response_information,
mqtt::v5::property::server_reference,
mqtt::v5::property::reason_string,
mqtt::v5::property::receive_maximum,
mqtt::v5::property::topic_alias_maximum,
mqtt::v5::property::topic_alias, mqtt::v5::property::maximum_qos,
mqtt::v5::property::retain_available,
mqtt::v5::property::user_property,
mqtt::v5::property::maximum_packet_size,
mqtt::v5::property::wildcard_subscription_available,
mqtt::v5::property::subscription_identifier_available,
mqtt::v5::property::shared_subscription_available>,
std::allocator<boost::variant<mqtt::v5::property::payload_format_indicator,
mqtt::v5::property::message_expiry_interval,
mqtt::v5::property::content_type, mqtt::v5::property::response_topic,
mqtt::v5::property::correlation_data,
mqtt::v5::property::subscription_identifier,
mqtt::v5::property::session_expiry_interval,
mqtt::v5::property::assigned_client_identifier,
mqtt::v5::property::server_keep_alive,
mqtt::v5::property::authentication_method,
mqtt::v5::property::authentication_data,
mqtt::v5::property::request_problem_information,
mqtt::v5::property::will_delay_interval,
mqtt::v5::property::request_response_information,
mqtt::v5::property::response_information,
mqtt::v5::property::server_reference,
mqtt::v5::property::reason_string,
mqtt::v5::property::receive_maximum,
mqtt::v5::property::topic_alias_maximum,
mqtt::v5::property::topic_alias, mqtt::v5::property::maximum_qos,
mqtt::v5::property::retain_available,
mqtt::v5::property::user_property,
mqtt::v5::property::maximum_packet_size,
mqtt::v5::property::wildcard_subscription_available,
mqtt::v5::property::subscription_identifier_available,
mqtt::v5::property::shared_subscription_available> > >, boost::any,
std::function<void
(boost::system::error_code)>)::{lambda(boost::system::error_code,
boost::asio::ip::basic_resolver_results<boost::asio::ip::tcp>)#1}::operator()(boost::system::error_code,
boost::asio::ip::basic_resolver_results<boost::asio::ip::tcp>)::{lambda(boost::system::error_code,
auto:1)#1}::~basic_resolver_results()

My questions is why this symbols is visible (via `nm` or `readelf`)
externally - my understanding is that since binaries are not expected
to be used as shared objects, it is enough to keep in symbols tables
only point of entry (so "main" or "_start" or whatever appropriate) OR
symbols that are to be imported from shared library. And probably some
other other stuff about which I don't know. But the main idea is that
since this symbol is not read from an external source (it is defined
in the binary itself), it shouldn't be visible. So why is it visible?

Also, the only file where "mqtt::client" is used is compiled as
follows (split for easier reading):

/usr/local/bin/arm-linux-gnueabihf-g++-9.2.0
    -DMQTT_USE_TLS
    -I/home/jd/repos/pac/zs-comm/inc
    -I/home/jd/repos/pac/zs-comm/svcbase/inc
    -I/home/jd/repos/pac/zs-comm/svcbase/config/inc
    -I/home/jd/repos/pac/zs-comm/svcbase/logger/inc
    -isystem
    /opt/ZS/deps/include
    -O3
    -DNDEBUG
    -Wall -Wextra -pedantic -Wno-psabi-Os
    -fvisibility=hidden
    -fvisibility-inlines-hidden
    -flto -ffunction-sections
    -fdata-sections
    -pthread
    -std=gnu++17
    -MD -MT
    CMakeFiles/paccomm.dir/src/endpoint_mqtt.cpp.o
    -MF CMakeFiles/paccomm.dir/src/endpoint_mqtt.cpp.o.d
    -o CMakeFiles/paccomm.dir/src/endpoint_mqtt.cpp.o
    -c /home/jd/repos/pac/zs-comm/src/endpoint_mqtt.cpp

Whole binary is linked like this:

/usr/local/bin/arm-linux-gnueabihf-g++-9.2.0
    -O3 -DNDEBUG
    -rdynamic
    -Wl,--exclude-libs,ALL
    -Wl,--discard-all
    -Wl,-flto
    -Wl,--gc-sections
    <object files>
    -o <program_name>
    -Wl,-rpath,<value_of_rpath>
    -lboost_system -lboost_filesystem -lssl -lcrypto
    svcbase/libimbase.a
    -pthread
    -lboost_system
    -lboost_filesystem
    -lssl
    -lcrypto
    -luuid
    svcbase/config/libineffcfg.a
    svcbase/logger/libzslog.a

I hope this is more than enough information.

Thanks,
-- 
Jędrzej Dudkiewicz

I really hate this damn machine, I wish that they would sell it.
It never does just what I want, but only what I tell it.

Re: Decreasing size of final executable

[Florian Weimer]

* Jędrzej Dudkiewicz via Gcc-help:

>     -rdynamic

This is what causes the exported symbols.  Perhaps you don't need it.

Thanks,
Florian