Diagnosing symbol conflicts in modified glibc build

Diagnosing symbol conflicts in modified glibc build

[George Hodgkins]

Hi all,
I'm a grad student working on modifying glibc for research purposes. Our
modifications interpose on or replace the public functions for some file
descriptor operations (open/openat/creat/close/socket/bind/connect) and
virtual memory control (mmap/sbrk/brk/munmap). Our code is contained in
three files: one in /misc that implements core system functionality, and
two in /sysdeps/unix/sysv/linux that implement the file and virtual memory
components respectively. To interpose on functions, we remove
their public-facing alias (pointing those to our own functions instead) and
call the internal versions (eg __mmap64). We also added a common header
included in all of our source files to /sysdeps/unix/sysv/linux/include/sys.

To include our files in the build, we added the file in /misc to the
routine list in /misc/Makefile, and our files in sysdeps to the
sysdep_routine list in the misc section of
/sysdeps/unix/sysv/linux/Makefile. We also added our header in both
locations. We completely deleted /misc/munmap.c and removed its Makefile
entry, and changed some filenames in /sysdeps/unix/syscalls.list to point
to our own code. Other than that, the only changes we have made to existing
glibc source are removing aliases for the functions we are replacing (no
changes to function bodies). We are working off of the glibc-2.32 tag,
building and installing as system libc inside a Ubuntu 18.04 Docker
container using GCC 8 (7 is the default for 18.04) with no non-default
configuration options.

With that extensive preamble, the actual problem we are facing is the
failure of this command:
#################
cc   -nostdlib -nostartfiles -r -o /glibc-wsp/build/elf/librtld.map.o
-Wl,--defsym=calloc=0 -Wl,--defsym=free=0 -Wl,--defsym=malloc=0
-Wl,--defsym=realloc=0 -Wl,--defsym=__stack_chk_fail=0
-Wl,--defsym=__stack_chk_fail_local=0 \
        '-Wl,-(' /glibc-wsp/build/elf/dl-allobjs.os
/glibc-wsp/build/libc_pic.a -lgcc '-Wl,-)'
-Wl,-Map,/glibc-wsp/build/elf/librtld.mapT
/glibc-wsp/build/libc_pic.a(getcwd.os): In function `__GI___getcwd':
/glibc-wsp/src/io/../sysdeps/unix/sysv/linux/getcwd.c:47: multiple
definition of `__getcwd'
/glibc-wsp/build/elf/dl-allobjs.os:/glibc-wsp/src/elf/../sysdeps/unix/sysv/linux/getcwd.c:47:
first defined here
/glibc-wsp/build/libc_pic.a(dl-error.os): In function
`__GI__dl_signal_exception':
/glibc-wsp/src/elf/dl-error-skeleton.c:91: multiple definition of
`_dl_signal_exception'
/glibc-wsp/build/elf/dl-allobjs.os:/glibc-wsp/src/elf/dl-error-skeleton.c:91:
first defined here
/glibc-wsp/build/libc_pic.a(dl-error.os): In function
`__GI__dl_signal_error':
/glibc-wsp/src/elf/dl-error-skeleton.c:109: multiple definition of
`_dl_signal_error'
/glibc-wsp/build/elf/dl-allobjs.os:/glibc-wsp/src/elf/dl-error-skeleton.c:109:
first defined here
/glibc-wsp/build/libc_pic.a(dl-error.os): In function
`__GI__dl_catch_exception':
/glibc-wsp/src/elf/dl-error-skeleton.c:175: multiple definition of
`_dl_catch_exception'
/glibc-wsp/build/elf/dl-allobjs.os:/glibc-wsp/src/elf/dl-error-skeleton.c:175:
first defined here
/glibc-wsp/build/libc_pic.a(dl-error.os): In function
`__GI__dl_catch_error':
/glibc-wsp/src/elf/dl-error-skeleton.c:225: multiple definition of
`_dl_catch_error'
/glibc-wsp/build/elf/dl-allobjs.os:/glibc-wsp/src/elf/dl-error-skeleton.c:225:
first defined here
/glibc-wsp/build/libc_pic.a(init-first.os):(.data+0x0): multiple definition
of `__libc_multiple_libcs'
/glibc-wsp/build/elf/dl-allobjs.os:(.bss+0xd8): first defined here
/glibc-wsp/build/libc_pic.a(libc_fatal.os): In function `__GI___libc_fatal':
/glibc-wsp/src/libio/../sysdeps/posix/libc_fatal.c:161: multiple definition
of `__GI___libc_fatal'
/glibc-wsp/build/elf/dl-allobjs.os:/glibc-wsp/src/elf/dl-minimal.c:259:
first defined here
/glibc-wsp/build/libc_pic.a(libc_fatal.os): In function `__GI___libc_fatal':
/glibc-wsp/src/libio/../sysdeps/posix/libc_fatal.c:161: multiple definition
of `__libc_fatal'
/glibc-wsp/build/elf/dl-allobjs.os:/glibc-wsp/src/elf/dl-minimal.c:259:
first defined here
collect2: error: ld returned 1 exit status
Makefile:524: recipe for target '/glibc-wsp/build/elf/librtld.map' failed
#################

It seems like we're accidentally adding some symbols in either libc_pic.a
or elf/dl-allobjs.os that shouldn't be there. However, I can't figure out
why this is. When I compare the symbols in each object to those from an
unmodified glibc build using objdump + diff, they seem to be essentially
the same -- all the conflicting symbols are present in both objects in the
unmodified version, from what I can see. We don't modify anything in the
elf or dlfcn modules, or do anything at all to interact with any of the
listed symbols as far as I can tell. What's going on here, and how can I
fix it?
Thanks,
George

P.S. More general guidance on our build scheme is also welcome if something
looks wrong to you. This is my first time modifying glibc and the build
system is still mostly a foreign country to me.

Re: Diagnosing symbol conflicts in modified glibc build

[Florian Weimer]

* George Hodgkins via Libc-help:

> With that extensive preamble, the actual problem we are facing is the
> failure of this command:
> #################
> cc   -nostdlib -nostartfiles -r -o /glibc-wsp/build/elf/librtld.map.o
> -Wl,--defsym=calloc=0 -Wl,--defsym=free=0 -Wl,--defsym=malloc=0
> -Wl,--defsym=realloc=0 -Wl,--defsym=__stack_chk_fail=0
> -Wl,--defsym=__stack_chk_fail_local=0 \
>         '-Wl,-(' /glibc-wsp/build/elf/dl-allobjs.os
> /glibc-wsp/build/libc_pic.a -lgcc '-Wl,-)'
> -Wl,-Map,/glibc-wsp/build/elf/librtld.mapT
> /glibc-wsp/build/libc_pic.a(getcwd.os): In function `__GI___getcwd':
> /glibc-wsp/src/io/../sysdeps/unix/sysv/linux/getcwd.c:47: multiple
> definition of `__getcwd'
> /glibc-wsp/build/elf/dl-allobjs.os:/glibc-wsp/src/elf/../sysdeps/unix/sysv/linux/getcwd.c:47:
> first defined here

This probably means that you are linking more into the dynamic loader
than you intend.  You may have to cover your own additions with
#if IS_IN (libc) or #if !IS_IN (rtld).

Thanks,
Florian

Re: Diagnosing symbol conflicts in modified glibc build

[George Hodgkins]

Unfortunately, that doesn't seem to resolve the issue. There are actually
parts of our code that need to be linked into the loader, but as an
experiment I tried wrapping all of our code in the macros you suggested.
The same errors occurred.
-George


On Wed, Jun 23, 2021 at 2:35 PM Florian Weimer <fweimer@redhat.com> wrote:

> * George Hodgkins via Libc-help:
>
> > With that extensive preamble, the actual problem we are facing is the
> > failure of this command:
> > #################
> > cc   -nostdlib -nostartfiles -r -o /glibc-wsp/build/elf/librtld.map.o
> > -Wl,--defsym=calloc=0 -Wl,--defsym=free=0 -Wl,--defsym=malloc=0
> > -Wl,--defsym=realloc=0 -Wl,--defsym=__stack_chk_fail=0
> > -Wl,--defsym=__stack_chk_fail_local=0 \
> >         '-Wl,-(' /glibc-wsp/build/elf/dl-allobjs.os
> > /glibc-wsp/build/libc_pic.a -lgcc '-Wl,-)'
> > -Wl,-Map,/glibc-wsp/build/elf/librtld.mapT
> > /glibc-wsp/build/libc_pic.a(getcwd.os): In function `__GI___getcwd':
> > /glibc-wsp/src/io/../sysdeps/unix/sysv/linux/getcwd.c:47: multiple
> > definition of `__getcwd'
> >
> /glibc-wsp/build/elf/dl-allobjs.os:/glibc-wsp/src/elf/../sysdeps/unix/sysv/linux/getcwd.c:47:
> > first defined here
>
> This probably means that you are linking more into the dynamic loader
> than you intend.  You may have to cover your own additions with
> #if IS_IN (libc) or #if !IS_IN (rtld).
>
> Thanks,
> Florian
>
>

Re: Diagnosing symbol conflicts in modified glibc build

[Florian Weimer]

* George Hodgkins via Libc-help:

> Unfortunately, that doesn't seem to resolve the issue. There are actually
> parts of our code that need to be linked into the loader, but as an
> experiment I tried wrapping all of our code in the macros you suggested.
> The same errors occurred.

Then you'll have to use the link editor's tracing capabilities (map
files, the -y option, etc.) to figure out what is pulling the multiple
definitions.

Thanks,
Florian