RE: undefined reference to symbol 'dlsym@@GLIBC_2.2.5' during qemu build

[<ckim@etri.re.kr>]

Hi, Florian

Yes,as you said, the '-ldl' option was being ignored I guess.
I copied the last link command and in it, moved the -ldl option to the end.
When I separately run this modified last command, the final target
qemu-system-aarch64 is built without error message.
Now I have to figure out how to make '-Wl,as-needed' as not default or
automatically place the -ld option to the last in qemu-5.1.0 build system.
(And the debugger(ddd with gdb inside) prompt is a wait pointer. I can see
prints when the program runs))
Anyway thanks a lot for the help.

Thank you and Regards,
Chan Kim

> -----Original Message-----
> From: Libc-help <libc-help-bounces@sourceware.org> On Behalf Of Chan Kim
> via Libc-help
> Sent: Tuesday, May 25, 2021 9:17 PM
> To: 'Florian Weimer' <fweimer@redhat.com>; 'Chan Kim via Libc-help' <libc-
> help@sourceware.org>
> Subject: RE: undefined reference to symbol 'dlsym@@GLIBC_2.2.5' during
> qemu build
> 
> Hello Florian,
> Thank you for the reply.
> I examined if the -ldl option is really used. I added V=1 during the qemu
> make to see the detail.
> 
> This is the compile command for axpu_if.c which calls dlsym function.
> 
> cc -iquote
> /home/ckim/prj/abdsn/ab21sim/ab21tsim/QEMU/qemu-5.1.0/build/hw/misc -
> iquote hw/misc -iquote
> /home/ckim/prj/abdsn/ab21sim/ab21tsim/QEMU/qemu-5.1.0/tcg/i386 -isystem
> /home/ckim/prj/abdsn/ab21sim/ab21tsim/QEMU/qemu-5.1.0/linux-headers -
> isystem /home/ckim/prj/abdsn/ab21sim/ab21tsim/QEMU/qemu-5.1.0/build/linux-
> headers
> -iquote . -iquote /home/ckim/prj/abdsn/ab21sim/ab21tsim/QEMU/qemu-5.1.0
> -iquote /home/ckim/prj/abdsn/ab21sim/ab21tsim/QEMU/qemu-5.1.0/accel/tcg
> -iquote /home/ckim/prj/abdsn/ab21sim/ab21tsim/QEMU/qemu-5.1.0/include
> -iquote /home/ckim/prj/abdsn/ab21sim/ab21tsim/QEMU/qemu-
> 5.1.0/disas/libvixl
> -I/usr/include/pixman-1   -pthread -I/usr/include/glib-2.0
> -I/usr/lib/x86_64-linux-gnu/glib-2.0/include -fPIE -DPIE -m64 -mcx16 -
> D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-
> prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-
> prototypes -fno-strict-aliasing -fno-common -fwrapv -std=gnu99  -
> DBAREMETAL_USE_AXPUSIM -ldl -Wold-style-declaration -Wold-style-definition
> -Wtype-limits -Wformat-security -Wformat-y2k -Winit-self -Wignored-
> qualifiers -Wempty-body -Wnested-externs -Wendif-labels -Wexpansion-to-
> defined -Wno-missing-include-dirs -Wno-shift-negative-value -Wno-psabi -
> fstack-protector-strong  -I/usr/include/p11-kit-1  -DSTRUCT_IOVEC_DEFINED
> -I/usr/include/libpng16  -I/usr/include/spice-server -I/usr/include/spice-
> 1 -I/home/ckim/prj/abdsn/ab21sim/ab21tsim/QEMU/qemu-5.1.0/capstone/include
> -I/home/ckim/prj/abdsn/ab21sim/ab21tsim/QEMU/qemu-5.1.0/tests
> -I/home/ckim/prj/abdsn/ab21sim/ab21tsim/QEMU/qemu-5.1.0/tests/qtest -MMD -
> MP
> -MT hw/misc/axpu_if.o -MF hw/misc/axpu_if.d -g   -c -o hw/misc/axpu_if.o
> /home/ckim/prj/abdsn/ab21sim/ab21tsim/QEMU/qemu-5.1.0/hw/misc/axpu_if.c
> 
> Here cc is used to make  axpu_if.o.
> 
> and This is the last link command to build the final executable qemu-
> system-aarch64.
> 
> c++ -g  -Wl,--warn-common -Wl,-z,relro -Wl,-z,now -pie -m64
> -DBAREMETAL_USE_AXPUSIM -ldl -fstack-protector-strong  -o
> qemu-system-aarch64 exec.o exec-vary.o tcg/tcg.o tcg/tcg-op.o tcg/tcg-op-
> vec.o tcg/tcg-op-gvec.o tcg/tcg-common.o tcg/optimize.o fpu/softfloat.o
> disas.o gdbstub-xml.o gdbstub.o migration/ram.o accel/qtest.o
> accel/stubs/hax-stub.o accel/stubs/hvf-stub.o accel/stubs/whpx-stub.o
> accel/stubs/kvm-stub.o accel/stubs/xen-stub.o accel/tcg/tcg-all.o
> accel/tcg/cputlb.o accel/tcg/tcg-runtime.o accel/tcg/tcg-runtime-gvec.o
> accel/tcg/cpu-exec.o accel/tcg/cpu-exec-common.o accel/tcg/translate-all.o
> accel/tcg/translator.o dump/dump.o hw/9pfs/virtio-9p-device.o
> hw/block/virtio-blk.o hw/block/vhost-user-blk.o hw/block/dataplane/virtio-
> blk.o hw/char/virtio-serial-bus.o hw/core/machine-qmp-cmds.o
> hw/cpu/arm11mpcore.o hw/cpu/realview_mpcore.o hw/cpu/a9mpcore.o hw/c....
> (reduced) ...  verbs -libumad -lslirp -lglib-2.0 -lgio-2.0 -lgobject-2.0 -
> lglib-2.0 -lgio-2.0
> -lgobject-2.0 -lglib-2.0 -lepoxy -lgbm -lepoxy -lgbm -lepoxy -lgbm -lepoxy
> -lgbm -lepoxy -lgbm -lpulse -lbrlapi -lcacard -lglib-2.0 -lcacard -lglib-
> 2.0 -lusbredirparser -lusbredirparser -lgtk-3 -lgdk-3 -lpangocairo-1.0
> -lpango-1.0 -lharfbuzz -latk-1.0 -lcairo-gobject -lcairo -lgdk_pixbuf-2.0
> -lgio-2.0 -lgobject-2.0 -lglib-2.0 -lX11 -lvte-2.91 -lgtk-3 -lgdk-3
> -lpangocairo-1.0 -lpango-1.0 -lharfbuzz -latk-1.0 -lcairo-gobject -lcairo
> -lgdk_pixbuf-2.0 -lgio-2.0 -lgobject-2.0 -lglib-2.0 -lepoxy -lgbm -lgtk-3
> -lgdk-3 -lpangocairo-1.0 -lpango-1.0 -lharfbuzz -latk-1.0 -lcairo-gobject
> -lcairo -lgdk_pixbuf-2.0 -lgio-2.0 -lgobject-2.0 -lglib-2.0 -lX11 -lvte-
> 2.91
> -lgtk-3 -lgdk-3 -lpangocairo-1.0 -lpango-1.0 -lharfbuzz -latk-1.0 -lcairo-
> gobject -lcairo -lgdk_pixbuf-2.0 -lgio-2.0 -lgobject-2.0 -lglib-2.0 -
> lepoxy -lgbm -lgtk-3 -lgdk-3 -lpangocairo-1.0 -lpango-1.0 -lharfbuzz
> -latk-1.0 -lcairo-gobject -lcairo -lgdk_pixbuf-2.0 -lgio-2.0 -lgobject-2.0
> -lglib-2.0 -lX11 -lvte-2.91 -lgtk-3 -lgdk-3 -lpangocairo-1.0 -lpango-1.0 -
> lharfbuzz -latk-1.0 -lcairo-gobject -lcairo -lgdk_pixbuf-2.0 -lgio-2.0
> -lgobject-2.0 -lglib-2.0 -lepoxy -lgbm -lgtk-3 -lgdk-3 -lpangocairo-1.0
> -lpango-1.0 -lharfbuzz -latk-1.0 -lcairo-gobject -lcairo -lgdk_pixbuf-2.0
> -lgio-2.0 -lgobject-2.0 -lglib-2.0 -lX11 -lvte-2.91 -lgtk-3 -lgdk-3
> -lpangocairo-1.0 -lpango-1.0 -lharfbuzz -latk-1.0 -lcairo-gobject -lcairo
> -lgdk_pixbuf-2.0 -lgio-2.0 -lgobject-2.0 -lglib-2.0 -lepoxy -lgbm -Wl,-
> Bsymbolic-functions -lncursesw -ltinfo -L/usr/local/lib -lz -lxml2 -laio -
> liscsi -lcurl -lrbd -lrados -lssh -lpixman-1 -lutil -lnuma -lpng16 -lz -
> ljpeg -lsasl2 -lxenctrl -lxenstore -lxenguest -lxenforeignmemory -
> lxengnttab -lxenevtchn -lxendevicemodel -lxentoolcore  -lrdmacm -libverbs
> -libumad -lfdt -lspice-server -lpmem -
> L/home/ckim/prj/abdsn/ab21sim/ab21tsim/QEMU/qemu-5.1.0/build/capstone
> -lcapstone  -lm -lz -lgthread-2.0 -pthread -lglib-2.0 -lnettle -lgnutls -
> lrt
> 
> I reduced the .o file list above. But I could see all the .o files are
> there with -ldl option. One thing I didn't know was that the final link
> command was using c++ not cc. The default library and include settings of
> my c++ are like this :
> 
> ckim@ckim-ubuntu:~$ echo | c++ -v -x c -E - Using built-in specs.
> COLLECT_GCC=c++
> OFFLOAD_TARGET_NAMES=nvptx-none:hsa
> OFFLOAD_TARGET_DEFAULT=1
> Target: x86_64-linux-gnu
> Configured with: ../src/configure -v --with-pkgversion='Ubuntu 9.3.0-
> 17ubuntu1~20.04' --with-bugurl=file:///usr/share/doc/gcc-9/README.Bugs
> --enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++,gm2
> --prefix=/usr --with-gcc-major-version-only --program-suffix=-9
> --program-prefix=x86_64-linux-gnu- --enable-shared --enable-linker-build-
> id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix
> --libdir=/usr/lib --enable-nls --enable-clocale=gnu --enable-libstdcxx-
> debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --
> enable-gnu-unique-object --disable-vtable-verify --enable-plugin --enable-
> default-pie --with-system-zlib --with-target-system-zlib=auto --enable-
> objc-gc=auto --enable-multiarch --disable-werror
> --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --
> enable-multilib --with-tune=generic --enable-offload-targets=nvptx-
> none=/build/gcc-9-HskZEa/gcc-9-9.3.0/debian/t
> mp-nvptx/usr,hsa --without-cuda-driver --enable-checking=release --
> build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
> Thread model: posix gcc version 9.3.0 (Ubuntu 9.3.0-17ubuntu1~20.04)
> COLLECT_GCC_OPTIONS='-v' '-E' '-shared-libgcc' '-mtune=generic'
> '-march=x86-64'
>  /usr/lib/gcc/x86_64-linux-gnu/9/cc1 -E -quiet -v -imultiarch x86_64-
> linux-gnu - -mtune=generic -march=x86-64 -fasynchronous-unwind-tables -
> fstack-protector-strong -Wformat -Wformat-security -fstack-clash-
> protection -fcf-protection ignoring nonexistent directory
> "/usr/local/include/x86_64-linux-gnu"
> ignoring nonexistent directory
> "/usr/lib/gcc/x86_64-linux-gnu/9/include-fixed"
> ignoring nonexistent directory
> "/usr/lib/gcc/x86_64-linux-gnu/9/../../../../x86_64-linux-gnu/include"
> #include "..." search starts here:
> #include <...> search starts here:
>  /usr/lib/gcc/x86_64-linux-gnu/9/include
>  /usr/local/include
>  /usr/include/x86_64-linux-gnu
>  /usr/include
> End of search list.
> # 1 "<stdin>"
> # 1 "<built-in>"
> # 1 "<command-line>"
> # 31 "<command-line>"
> # 1 "/usr/include/stdc-predef.h" 1 3 4
> # 32 "<command-line>" 2
> # 1 "<stdin>"
> COMPILER_PATH=/usr/lib/gcc/x86_64-linux-gnu/9/:/usr/lib/gcc/x86_64-linux-
> gnu
> /9/:/usr/lib/gcc/x86_64-linux-gnu/:/usr/lib/gcc/x86_64-linux-
> gnu/9/:/usr/lib
> /gcc/x86_64-linux-gnu/
> LIBRARY_PATH=/usr/lib/gcc/x86_64-linux-gnu/9/:/usr/lib/gcc/x86_64-linux-
> gnu/
> 9/../../../x86_64-linux-gnu/:/usr/lib/gcc/x86_64-linux-
> gnu/9/../../../../lib
> /:/lib/x86_64-linux-gnu/:/lib/../lib/:/usr/lib/x86_64-linux-
> gnu/:/usr/lib/..
> /lib/:/usr/lib/gcc/x86_64-linux-gnu/9/../../../:/lib/:/usr/lib/
> COLLECT_GCC_OPTIONS='-v' '-E' '-shared-libgcc' '-mtune=generic'
> '-march=x86-64'
> 
> I can see the LIBRARY_PATH includes /usr/lib/x86_64-linux-gnu which
> contains libdl.so file.
> 
> ckim@ckim-ubuntu:~$ F libdl.so /usr  (F: my find function)
> /usr/lib32/libdl.so /usr/alpha-linux-gnu/lib/libdl.so /usr/libx32/libdl.so
> /usr/lib/i386-linux-gnu/libdl.so /usr/lib/x86_64-linux-gnu/libdl.so
> 
> Using "nm -D /usr/lib/x86_64-linux-gnu/libdl.so", I could see it contains
> correctly dlsym (not __libc_dlsym). (I mistakenly grepped dlsym in libc.so
> file yesterday and it gave me __libc_dlsym). So yes, it is not being
> linked to the /usr/lib/x86_64-linux-gnu/libdl.so. (Actually it should be
> linked during the run time and what I need is the include file with
> "extern dlsym(...)" declaration. Right?
> The axpu_if.c file has #include <dlfcn.h> which is in /usr/include/x86_64-
> linux-gnu directory which is in one of the default include path of cc and
> c++. (I checked both)
> 
> So include path is correct and -ldl is used for final link command. The
> dlsym is in libdl.so but I don't know why it's looking for a symbol
> 'dlsym@@GLIBC_2.2.5'. Can you give me any suggestions for check?
> (In the source code I kept them as dlopen, dlsym now, and I guess if this
> undefined reference error goes, the seg-fault will go too) Thank you!
> Chan Kim
> 
> > -----Original Message-----
> > From: Florian Weimer <fweimer@redhat.com>
> > Sent: Tuesday, May 25, 2021 4:36 PM
> > To: Chan Kim via Libc-help <libc-help@sourceware.org>
> > Cc: ckim@etri.re.kr
> > Subject: Re: undefined reference to symbol 'dlsym@@GLIBC_2.2.5' during
> > qemu build
> >
> > * Chan Kim via Libc-help:
> >
> > >     LINK    aarch64-softmmu/qemu-system-aarch64
> > >
> > >     /usr/bin/ld: ../hw/misc/axpu_if.o: undefined reference to symbol
> > > 'dlsym@@GLIBC_2.2.5'
> > >
> > >     /usr/bin/ld:
> > > /usr/lib/gcc/x86_64-linux-gnu/9/../../../x86_64-linux-gnu/libdl.so:
> > > error adding symbols: DSO missing from command line
> >
> > This error message attempts to tell you to link with -ldl.
> >
> > Thanks,
> > Florian
> >
> 
> 
> 
>