RE: undefined reference to symbol 'dlsym@@GLIBC_2.2.5' during qemu build

[<ckim@etri.re.kr>]

Hello Florian,
Thank you for the reply.
I examined if the -ldl option is really used. I added V=1 during the qemu
make to see the detail.

This is the compile command for axpu_if.c which calls dlsym function.

cc -iquote
/home/ckim/prj/abdsn/ab21sim/ab21tsim/QEMU/qemu-5.1.0/build/hw/misc -iquote
hw/misc -iquote
/home/ckim/prj/abdsn/ab21sim/ab21tsim/QEMU/qemu-5.1.0/tcg/i386 -isystem
/home/ckim/prj/abdsn/ab21sim/ab21tsim/QEMU/qemu-5.1.0/linux-headers -isystem
/home/ckim/prj/abdsn/ab21sim/ab21tsim/QEMU/qemu-5.1.0/build/linux-headers
-iquote . -iquote /home/ckim/prj/abdsn/ab21sim/ab21tsim/QEMU/qemu-5.1.0
-iquote /home/ckim/prj/abdsn/ab21sim/ab21tsim/QEMU/qemu-5.1.0/accel/tcg
-iquote /home/ckim/prj/abdsn/ab21sim/ab21tsim/QEMU/qemu-5.1.0/include
-iquote /home/ckim/prj/abdsn/ab21sim/ab21tsim/QEMU/qemu-5.1.0/disas/libvixl
-I/usr/include/pixman-1   -pthread -I/usr/include/glib-2.0
-I/usr/lib/x86_64-linux-gnu/glib-2.0/include -fPIE -DPIE -m64 -mcx16
-D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes
-Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes
-fno-strict-aliasing -fno-common -fwrapv -std=gnu99  -DBAREMETAL_USE_AXPUSIM
-ldl -Wold-style-declaration -Wold-style-definition -Wtype-limits
-Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wempty-body
-Wnested-externs -Wendif-labels -Wexpansion-to-defined
-Wno-missing-include-dirs -Wno-shift-negative-value -Wno-psabi
-fstack-protector-strong  -I/usr/include/p11-kit-1  -DSTRUCT_IOVEC_DEFINED
-I/usr/include/libpng16  -I/usr/include/spice-server -I/usr/include/spice-1
-I/home/ckim/prj/abdsn/ab21sim/ab21tsim/QEMU/qemu-5.1.0/capstone/include
-I/home/ckim/prj/abdsn/ab21sim/ab21tsim/QEMU/qemu-5.1.0/tests
-I/home/ckim/prj/abdsn/ab21sim/ab21tsim/QEMU/qemu-5.1.0/tests/qtest -MMD -MP
-MT hw/misc/axpu_if.o -MF hw/misc/axpu_if.d -g   -c -o hw/misc/axpu_if.o
/home/ckim/prj/abdsn/ab21sim/ab21tsim/QEMU/qemu-5.1.0/hw/misc/axpu_if.c

Here cc is used to make  axpu_if.o.

and This is the last link command to build the final executable
qemu-system-aarch64.

c++ -g  -Wl,--warn-common -Wl,-z,relro -Wl,-z,now -pie -m64
-DBAREMETAL_USE_AXPUSIM -ldl -fstack-protector-strong  -o
qemu-system-aarch64 exec.o exec-vary.o tcg/tcg.o tcg/tcg-op.o
tcg/tcg-op-vec.o tcg/tcg-op-gvec.o tcg/tcg-common.o tcg/optimize.o
fpu/softfloat.o disas.o gdbstub-xml.o gdbstub.o migration/ram.o
accel/qtest.o accel/stubs/hax-stub.o accel/stubs/hvf-stub.o
accel/stubs/whpx-stub.o accel/stubs/kvm-stub.o accel/stubs/xen-stub.o
accel/tcg/tcg-all.o accel/tcg/cputlb.o accel/tcg/tcg-runtime.o
accel/tcg/tcg-runtime-gvec.o accel/tcg/cpu-exec.o
accel/tcg/cpu-exec-common.o accel/tcg/translate-all.o accel/tcg/translator.o
dump/dump.o hw/9pfs/virtio-9p-device.o hw/block/virtio-blk.o
hw/block/vhost-user-blk.o hw/block/dataplane/virtio-blk.o
hw/char/virtio-serial-bus.o hw/core/machine-qmp-cmds.o hw/cpu/arm11mpcore.o
hw/cpu/realview_mpcore.o hw/cpu/a9mpcore.o hw/c.... (reduced) ...  verbs
-libumad -lslirp -lglib-2.0 -lgio-2.0 -lgobject-2.0 -lglib-2.0 -lgio-2.0
-lgobject-2.0 -lglib-2.0 -lepoxy -lgbm -lepoxy -lgbm -lepoxy -lgbm -lepoxy
-lgbm -lepoxy -lgbm -lpulse -lbrlapi -lcacard -lglib-2.0 -lcacard -lglib-2.0
-lusbredirparser -lusbredirparser -lgtk-3 -lgdk-3 -lpangocairo-1.0
-lpango-1.0 -lharfbuzz -latk-1.0 -lcairo-gobject -lcairo -lgdk_pixbuf-2.0
-lgio-2.0 -lgobject-2.0 -lglib-2.0 -lX11 -lvte-2.91 -lgtk-3 -lgdk-3
-lpangocairo-1.0 -lpango-1.0 -lharfbuzz -latk-1.0 -lcairo-gobject -lcairo
-lgdk_pixbuf-2.0 -lgio-2.0 -lgobject-2.0 -lglib-2.0 -lepoxy -lgbm -lgtk-3
-lgdk-3 -lpangocairo-1.0 -lpango-1.0 -lharfbuzz -latk-1.0 -lcairo-gobject
-lcairo -lgdk_pixbuf-2.0 -lgio-2.0 -lgobject-2.0 -lglib-2.0 -lX11 -lvte-2.91
-lgtk-3 -lgdk-3 -lpangocairo-1.0 -lpango-1.0 -lharfbuzz -latk-1.0
-lcairo-gobject -lcairo -lgdk_pixbuf-2.0 -lgio-2.0 -lgobject-2.0 -lglib-2.0
-lepoxy -lgbm -lgtk-3 -lgdk-3 -lpangocairo-1.0 -lpango-1.0 -lharfbuzz
-latk-1.0 -lcairo-gobject -lcairo -lgdk_pixbuf-2.0 -lgio-2.0 -lgobject-2.0
-lglib-2.0 -lX11 -lvte-2.91 -lgtk-3 -lgdk-3 -lpangocairo-1.0 -lpango-1.0
-lharfbuzz -latk-1.0 -lcairo-gobject -lcairo -lgdk_pixbuf-2.0 -lgio-2.0
-lgobject-2.0 -lglib-2.0 -lepoxy -lgbm -lgtk-3 -lgdk-3 -lpangocairo-1.0
-lpango-1.0 -lharfbuzz -latk-1.0 -lcairo-gobject -lcairo -lgdk_pixbuf-2.0
-lgio-2.0 -lgobject-2.0 -lglib-2.0 -lX11 -lvte-2.91 -lgtk-3 -lgdk-3
-lpangocairo-1.0 -lpango-1.0 -lharfbuzz -latk-1.0 -lcairo-gobject -lcairo
-lgdk_pixbuf-2.0 -lgio-2.0 -lgobject-2.0 -lglib-2.0 -lepoxy -lgbm
-Wl,-Bsymbolic-functions -lncursesw -ltinfo -L/usr/local/lib -lz -lxml2
-laio -liscsi -lcurl -lrbd -lrados -lssh -lpixman-1 -lutil -lnuma -lpng16
-lz -ljpeg -lsasl2 -lxenctrl -lxenstore -lxenguest -lxenforeignmemory
-lxengnttab -lxenevtchn -lxendevicemodel -lxentoolcore  -lrdmacm -libverbs
-libumad -lfdt -lspice-server -lpmem
-L/home/ckim/prj/abdsn/ab21sim/ab21tsim/QEMU/qemu-5.1.0/build/capstone
-lcapstone  -lm -lz -lgthread-2.0 -pthread -lglib-2.0 -lnettle -lgnutls
-lrt

I reduced the .o file list above. But I could see all the .o files are there
with -ldl option. One thing I didn't know was that the final link command
was using c++ not cc. The default library and include settings of my c++ are
like this : 

ckim@ckim-ubuntu:~$ echo | c++ -v -x c -E -
Using built-in specs.
COLLECT_GCC=c++
OFFLOAD_TARGET_NAMES=nvptx-none:hsa
OFFLOAD_TARGET_DEFAULT=1
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu
9.3.0-17ubuntu1~20.04' --with-bugurl=file:///usr/share/doc/gcc-9/README.Bugs
--enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++,gm2
--prefix=/usr --with-gcc-major-version-only --program-suffix=-9
--program-prefix=x86_64-linux-gnu- --enable-shared --enable-linker-build-id
--libexecdir=/usr/lib --without-included-gettext --enable-threads=posix
--libdir=/usr/lib --enable-nls --enable-clocale=gnu --enable-libstdcxx-debug
--enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new
--enable-gnu-unique-object --disable-vtable-verify --enable-plugin
--enable-default-pie --with-system-zlib --with-target-system-zlib=auto
--enable-objc-gc=auto --enable-multiarch --disable-werror
--with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32
--enable-multilib --with-tune=generic
--enable-offload-targets=nvptx-none=/build/gcc-9-HskZEa/gcc-9-9.3.0/debian/t
mp-nvptx/usr,hsa --without-cuda-driver --enable-checking=release
--build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 9.3.0 (Ubuntu 9.3.0-17ubuntu1~20.04) 
COLLECT_GCC_OPTIONS='-v' '-E' '-shared-libgcc' '-mtune=generic'
'-march=x86-64'
 /usr/lib/gcc/x86_64-linux-gnu/9/cc1 -E -quiet -v -imultiarch
x86_64-linux-gnu - -mtune=generic -march=x86-64 -fasynchronous-unwind-tables
-fstack-protector-strong -Wformat -Wformat-security -fstack-clash-protection
-fcf-protection
ignoring nonexistent directory "/usr/local/include/x86_64-linux-gnu"
ignoring nonexistent directory
"/usr/lib/gcc/x86_64-linux-gnu/9/include-fixed"
ignoring nonexistent directory
"/usr/lib/gcc/x86_64-linux-gnu/9/../../../../x86_64-linux-gnu/include"
#include "..." search starts here:
#include <...> search starts here:
 /usr/lib/gcc/x86_64-linux-gnu/9/include
 /usr/local/include
 /usr/include/x86_64-linux-gnu
 /usr/include
End of search list.
# 1 "<stdin>"
# 1 "<built-in>"
# 1 "<command-line>"
# 31 "<command-line>"
# 1 "/usr/include/stdc-predef.h" 1 3 4
# 32 "<command-line>" 2
# 1 "<stdin>"
COMPILER_PATH=/usr/lib/gcc/x86_64-linux-gnu/9/:/usr/lib/gcc/x86_64-linux-gnu
/9/:/usr/lib/gcc/x86_64-linux-gnu/:/usr/lib/gcc/x86_64-linux-gnu/9/:/usr/lib
/gcc/x86_64-linux-gnu/
LIBRARY_PATH=/usr/lib/gcc/x86_64-linux-gnu/9/:/usr/lib/gcc/x86_64-linux-gnu/
9/../../../x86_64-linux-gnu/:/usr/lib/gcc/x86_64-linux-gnu/9/../../../../lib
/:/lib/x86_64-linux-gnu/:/lib/../lib/:/usr/lib/x86_64-linux-gnu/:/usr/lib/..
/lib/:/usr/lib/gcc/x86_64-linux-gnu/9/../../../:/lib/:/usr/lib/
COLLECT_GCC_OPTIONS='-v' '-E' '-shared-libgcc' '-mtune=generic'
'-march=x86-64'

I can see the LIBRARY_PATH includes /usr/lib/x86_64-linux-gnu which contains
libdl.so file.

ckim@ckim-ubuntu:~$ F libdl.so /usr  (F: my find function)
/usr/lib32/libdl.so
/usr/alpha-linux-gnu/lib/libdl.so
/usr/libx32/libdl.so
/usr/lib/i386-linux-gnu/libdl.so
/usr/lib/x86_64-linux-gnu/libdl.so

Using "nm -D /usr/lib/x86_64-linux-gnu/libdl.so", I could see it contains
correctly dlsym (not __libc_dlsym). (I mistakenly grepped dlsym in libc.so
file yesterday and it gave me __libc_dlsym). So yes, it is not being linked
to the /usr/lib/x86_64-linux-gnu/libdl.so. (Actually it should be linked
during the run time and what I need is the include file with "extern
dlsym(...)" declaration. Right? 
The axpu_if.c file has #include <dlfcn.h> which is in
/usr/include/x86_64-linux-gnu directory which is in one of the default
include path of cc and c++. (I checked both)

So include path is correct and -ldl is used for final link command. The
dlsym is in libdl.so but I don't know why it's looking for a symbol
'dlsym@@GLIBC_2.2.5'. Can you give me any suggestions for check?
(In the source code I kept them as dlopen, dlsym now, and I guess if this
undefined reference error goes, the seg-fault will go too) 
Thank you!
Chan Kim

> -----Original Message-----
> From: Florian Weimer <fweimer@redhat.com>
> Sent: Tuesday, May 25, 2021 4:36 PM
> To: Chan Kim via Libc-help <libc-help@sourceware.org>
> Cc: ckim@etri.re.kr
> Subject: Re: undefined reference to symbol 'dlsym@@GLIBC_2.2.5' during
> qemu build
> 
> * Chan Kim via Libc-help:
> 
> >     LINK    aarch64-softmmu/qemu-system-aarch64
> >
> >     /usr/bin/ld: ../hw/misc/axpu_if.o: undefined reference to symbol
> > 'dlsym@@GLIBC_2.2.5'
> >
> >     /usr/bin/ld:
> > /usr/lib/gcc/x86_64-linux-gnu/9/../../../x86_64-linux-gnu/libdl.so:
> > error adding symbols: DSO missing from command line
> 
> This error message attempts to tell you to link with -ldl.
> 
> Thanks,
> Florian
>