undefined reference to symbol 'dlsym@@GLIBC_2.2.5' during qemu build

undefined reference to symbol 'dlsym@@GLIBC_2.2.5' during qemu build

[ckim]

Hello, glibc users/developers,

 

I'm adding a code into qemu-5.1.0 and this code should use an external
shared library file (libaxpusim.so) that one of my colleagues made. (I'm on
ubuntu 20.04,  x86_64 machine).

I added -ldl to the build flags for qemu using the configure
--extra-cflags/--extra-cxxflags/--extra-ldflags options and can see they are
really set.  

But when I actually do "make", it gives me this error and I can't figure out
why.  

    (error message at the end)

    LINK    aarch64-softmmu/qemu-system-aarch64

    /usr/bin/ld: ../hw/misc/axpu_if.o: undefined reference to symbol
'dlsym@@GLIBC_2.2.5'

    /usr/bin/ld:
/usr/lib/gcc/x86_64-linux-gnu/9/../../../x86_64-linux-gnu/libdl.so: error
adding symbols: DSO missing from command line

    collect2: error: ld returned 1 exit status

    make[1]: *** [Makefile:219: qemu-system-aarch64] Error 1

    make: *** [Makefile:527: aarch64-softmmu/all] Error 2

 

Here is the "ldd --version" command output. My system's glibc version seems
to be 2.31 (ubuntu 20.04), but I don't know why it's looking for the symbol
dlsym from glibc_2.2.5.  

 

    ckim@ckim-ubuntu:~/xxx/qemu-5.1.0/build$ ldd --version

    ldd (Ubuntu GLIBC 2.31-0ubuntu9.2) 2.31

    Copyright (C) 2020 Free Software Foundation, Inc.

    This is free software; see the source for copying conditions.  There is
NO

    warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR
PURPOSE.

    Written by Roland McGrath and Ulrich Drepper.

 

Later, I found /usr/lib/x86_64-linux-gnu/libc.so is a text file with
contents

 

/* GNU ld script

   Use the shared library, but some functions are only in

   the static library, so try that secondarily.  */

OUTPUT_FORMAT(elf64-x86-64)

GROUP ( /lib/x86_64-linux-gnu/libc.so.6
/usr/lib/x86_64-linux-gnu/libc_nonshared.a  AS_NEEDED (
/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 ) )

 

So I used "nm -D /lib/x86_64-linux-gnu/libc.so.6 | grep dlsym" and found the
functions name is in the .so file is __glibc_dlsym, not dlsym. So I changed
the name dlsym to __libc_dlsym and likewise changed dlopen to __glibc_dlopen
and since dlerror was not defined there, I substituted it with a print. With
those changes it builds ok.

But whe the code runs, the __libc_dlopen opens the .so file, but the
__libc_dlsym causes segmentation fault.

 

Strange thing is, with the same external libaxpusim.so file, another
independent application was built ok and runs ok (using same code using
dlopen and dlsym, without changing dlopen, dlsym function name). So I guess
I should be able to build modified qemu without the names changes to dlopen
and dlsym.

 

Could anyone tell me in what situation I might be in now, and how I can
solve this segmentation fault error problem?

(the __libc_dloopen_model return value seems normal, a nonzero pointer).

If I follow glibc using gdb, the segmentation fault occurs in this code.
Glibc-2.31/elf/dl-libc.c)

 

106   args->loadbase = GLRO(dl_lookup_symbol_x) (args->name, args->map,
&args->ref,

107                                              args->map->l_local_scope,
NULL, 0,

108                                              DL_LOOKUP_RETURN_NEWEST,
NULL);

 

Any suggestion or hint will also be appreciated.

Thanks!

 

Chan Kim

Re: undefined reference to symbol 'dlsym@@GLIBC_2.2.5' during qemu build

[Florian Weimer]

* Chan Kim via Libc-help:

>     LINK    aarch64-softmmu/qemu-system-aarch64
>
>     /usr/bin/ld: ../hw/misc/axpu_if.o: undefined reference to symbol
> 'dlsym@@GLIBC_2.2.5'
>
>     /usr/bin/ld:
> /usr/lib/gcc/x86_64-linux-gnu/9/../../../x86_64-linux-gnu/libdl.so: error
> adding symbols: DSO missing from command line

This error message attempts to tell you to link with -ldl.

Thanks,
Florian

RE: undefined reference to symbol 'dlsym@@GLIBC_2.2.5' during qemu build

[ckim]

Hello Florian,
Thank you for the reply.
I examined if the -ldl option is really used. I added V=1 during the qemu
make to see the detail.

This is the compile command for axpu_if.c which calls dlsym function.

cc -iquote
/home/ckim/prj/abdsn/ab21sim/ab21tsim/QEMU/qemu-5.1.0/build/hw/misc -iquote
hw/misc -iquote
/home/ckim/prj/abdsn/ab21sim/ab21tsim/QEMU/qemu-5.1.0/tcg/i386 -isystem
/home/ckim/prj/abdsn/ab21sim/ab21tsim/QEMU/qemu-5.1.0/linux-headers -isystem
/home/ckim/prj/abdsn/ab21sim/ab21tsim/QEMU/qemu-5.1.0/build/linux-headers
-iquote . -iquote /home/ckim/prj/abdsn/ab21sim/ab21tsim/QEMU/qemu-5.1.0
-iquote /home/ckim/prj/abdsn/ab21sim/ab21tsim/QEMU/qemu-5.1.0/accel/tcg
-iquote /home/ckim/prj/abdsn/ab21sim/ab21tsim/QEMU/qemu-5.1.0/include
-iquote /home/ckim/prj/abdsn/ab21sim/ab21tsim/QEMU/qemu-5.1.0/disas/libvixl
-I/usr/include/pixman-1   -pthread -I/usr/include/glib-2.0
-I/usr/lib/x86_64-linux-gnu/glib-2.0/include -fPIE -DPIE -m64 -mcx16
-D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes
-Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes
-fno-strict-aliasing -fno-common -fwrapv -std=gnu99  -DBAREMETAL_USE_AXPUSIM
-ldl -Wold-style-declaration -Wold-style-definition -Wtype-limits
-Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wempty-body
-Wnested-externs -Wendif-labels -Wexpansion-to-defined
-Wno-missing-include-dirs -Wno-shift-negative-value -Wno-psabi
-fstack-protector-strong  -I/usr/include/p11-kit-1  -DSTRUCT_IOVEC_DEFINED
-I/usr/include/libpng16  -I/usr/include/spice-server -I/usr/include/spice-1
-I/home/ckim/prj/abdsn/ab21sim/ab21tsim/QEMU/qemu-5.1.0/capstone/include
-I/home/ckim/prj/abdsn/ab21sim/ab21tsim/QEMU/qemu-5.1.0/tests
-I/home/ckim/prj/abdsn/ab21sim/ab21tsim/QEMU/qemu-5.1.0/tests/qtest -MMD -MP
-MT hw/misc/axpu_if.o -MF hw/misc/axpu_if.d -g   -c -o hw/misc/axpu_if.o
/home/ckim/prj/abdsn/ab21sim/ab21tsim/QEMU/qemu-5.1.0/hw/misc/axpu_if.c

Here cc is used to make  axpu_if.o.

and This is the last link command to build the final executable
qemu-system-aarch64.

c++ -g  -Wl,--warn-common -Wl,-z,relro -Wl,-z,now -pie -m64
-DBAREMETAL_USE_AXPUSIM -ldl -fstack-protector-strong  -o
qemu-system-aarch64 exec.o exec-vary.o tcg/tcg.o tcg/tcg-op.o
tcg/tcg-op-vec.o tcg/tcg-op-gvec.o tcg/tcg-common.o tcg/optimize.o
fpu/softfloat.o disas.o gdbstub-xml.o gdbstub.o migration/ram.o
accel/qtest.o accel/stubs/hax-stub.o accel/stubs/hvf-stub.o
accel/stubs/whpx-stub.o accel/stubs/kvm-stub.o accel/stubs/xen-stub.o
accel/tcg/tcg-all.o accel/tcg/cputlb.o accel/tcg/tcg-runtime.o
accel/tcg/tcg-runtime-gvec.o accel/tcg/cpu-exec.o
accel/tcg/cpu-exec-common.o accel/tcg/translate-all.o accel/tcg/translator.o
dump/dump.o hw/9pfs/virtio-9p-device.o hw/block/virtio-blk.o
hw/block/vhost-user-blk.o hw/block/dataplane/virtio-blk.o
hw/char/virtio-serial-bus.o hw/core/machine-qmp-cmds.o hw/cpu/arm11mpcore.o
hw/cpu/realview_mpcore.o hw/cpu/a9mpcore.o hw/c.... (reduced) ...  verbs
-libumad -lslirp -lglib-2.0 -lgio-2.0 -lgobject-2.0 -lglib-2.0 -lgio-2.0
-lgobject-2.0 -lglib-2.0 -lepoxy -lgbm -lepoxy -lgbm -lepoxy -lgbm -lepoxy
-lgbm -lepoxy -lgbm -lpulse -lbrlapi -lcacard -lglib-2.0 -lcacard -lglib-2.0
-lusbredirparser -lusbredirparser -lgtk-3 -lgdk-3 -lpangocairo-1.0
-lpango-1.0 -lharfbuzz -latk-1.0 -lcairo-gobject -lcairo -lgdk_pixbuf-2.0
-lgio-2.0 -lgobject-2.0 -lglib-2.0 -lX11 -lvte-2.91 -lgtk-3 -lgdk-3
-lpangocairo-1.0 -lpango-1.0 -lharfbuzz -latk-1.0 -lcairo-gobject -lcairo
-lgdk_pixbuf-2.0 -lgio-2.0 -lgobject-2.0 -lglib-2.0 -lepoxy -lgbm -lgtk-3
-lgdk-3 -lpangocairo-1.0 -lpango-1.0 -lharfbuzz -latk-1.0 -lcairo-gobject
-lcairo -lgdk_pixbuf-2.0 -lgio-2.0 -lgobject-2.0 -lglib-2.0 -lX11 -lvte-2.91
-lgtk-3 -lgdk-3 -lpangocairo-1.0 -lpango-1.0 -lharfbuzz -latk-1.0
-lcairo-gobject -lcairo -lgdk_pixbuf-2.0 -lgio-2.0 -lgobject-2.0 -lglib-2.0
-lepoxy -lgbm -lgtk-3 -lgdk-3 -lpangocairo-1.0 -lpango-1.0 -lharfbuzz
-latk-1.0 -lcairo-gobject -lcairo -lgdk_pixbuf-2.0 -lgio-2.0 -lgobject-2.0
-lglib-2.0 -lX11 -lvte-2.91 -lgtk-3 -lgdk-3 -lpangocairo-1.0 -lpango-1.0
-lharfbuzz -latk-1.0 -lcairo-gobject -lcairo -lgdk_pixbuf-2.0 -lgio-2.0
-lgobject-2.0 -lglib-2.0 -lepoxy -lgbm -lgtk-3 -lgdk-3 -lpangocairo-1.0
-lpango-1.0 -lharfbuzz -latk-1.0 -lcairo-gobject -lcairo -lgdk_pixbuf-2.0
-lgio-2.0 -lgobject-2.0 -lglib-2.0 -lX11 -lvte-2.91 -lgtk-3 -lgdk-3
-lpangocairo-1.0 -lpango-1.0 -lharfbuzz -latk-1.0 -lcairo-gobject -lcairo
-lgdk_pixbuf-2.0 -lgio-2.0 -lgobject-2.0 -lglib-2.0 -lepoxy -lgbm
-Wl,-Bsymbolic-functions -lncursesw -ltinfo -L/usr/local/lib -lz -lxml2
-laio -liscsi -lcurl -lrbd -lrados -lssh -lpixman-1 -lutil -lnuma -lpng16
-lz -ljpeg -lsasl2 -lxenctrl -lxenstore -lxenguest -lxenforeignmemory
-lxengnttab -lxenevtchn -lxendevicemodel -lxentoolcore  -lrdmacm -libverbs
-libumad -lfdt -lspice-server -lpmem
-L/home/ckim/prj/abdsn/ab21sim/ab21tsim/QEMU/qemu-5.1.0/build/capstone
-lcapstone  -lm -lz -lgthread-2.0 -pthread -lglib-2.0 -lnettle -lgnutls
-lrt

I reduced the .o file list above. But I could see all the .o files are there
with -ldl option. One thing I didn't know was that the final link command
was using c++ not cc. The default library and include settings of my c++ are
like this : 

ckim@ckim-ubuntu:~$ echo | c++ -v -x c -E -
Using built-in specs.
COLLECT_GCC=c++
OFFLOAD_TARGET_NAMES=nvptx-none:hsa
OFFLOAD_TARGET_DEFAULT=1
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu
9.3.0-17ubuntu1~20.04' --with-bugurl=file:///usr/share/doc/gcc-9/README.Bugs
--enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++,gm2
--prefix=/usr --with-gcc-major-version-only --program-suffix=-9
--program-prefix=x86_64-linux-gnu- --enable-shared --enable-linker-build-id
--libexecdir=/usr/lib --without-included-gettext --enable-threads=posix
--libdir=/usr/lib --enable-nls --enable-clocale=gnu --enable-libstdcxx-debug
--enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new
--enable-gnu-unique-object --disable-vtable-verify --enable-plugin
--enable-default-pie --with-system-zlib --with-target-system-zlib=auto
--enable-objc-gc=auto --enable-multiarch --disable-werror
--with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32
--enable-multilib --with-tune=generic
--enable-offload-targets=nvptx-none=/build/gcc-9-HskZEa/gcc-9-9.3.0/debian/t
mp-nvptx/usr,hsa --without-cuda-driver --enable-checking=release
--build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 9.3.0 (Ubuntu 9.3.0-17ubuntu1~20.04) 
COLLECT_GCC_OPTIONS='-v' '-E' '-shared-libgcc' '-mtune=generic'
'-march=x86-64'
 /usr/lib/gcc/x86_64-linux-gnu/9/cc1 -E -quiet -v -imultiarch
x86_64-linux-gnu - -mtune=generic -march=x86-64 -fasynchronous-unwind-tables
-fstack-protector-strong -Wformat -Wformat-security -fstack-clash-protection
-fcf-protection
ignoring nonexistent directory "/usr/local/include/x86_64-linux-gnu"
ignoring nonexistent directory
"/usr/lib/gcc/x86_64-linux-gnu/9/include-fixed"
ignoring nonexistent directory
"/usr/lib/gcc/x86_64-linux-gnu/9/../../../../x86_64-linux-gnu/include"
#include "..." search starts here:
#include <...> search starts here:
 /usr/lib/gcc/x86_64-linux-gnu/9/include
 /usr/local/include
 /usr/include/x86_64-linux-gnu
 /usr/include
End of search list.
# 1 "<stdin>"
# 1 "<built-in>"
# 1 "<command-line>"
# 31 "<command-line>"
# 1 "/usr/include/stdc-predef.h" 1 3 4
# 32 "<command-line>" 2
# 1 "<stdin>"
COMPILER_PATH=/usr/lib/gcc/x86_64-linux-gnu/9/:/usr/lib/gcc/x86_64-linux-gnu
/9/:/usr/lib/gcc/x86_64-linux-gnu/:/usr/lib/gcc/x86_64-linux-gnu/9/:/usr/lib
/gcc/x86_64-linux-gnu/
LIBRARY_PATH=/usr/lib/gcc/x86_64-linux-gnu/9/:/usr/lib/gcc/x86_64-linux-gnu/
9/../../../x86_64-linux-gnu/:/usr/lib/gcc/x86_64-linux-gnu/9/../../../../lib
/:/lib/x86_64-linux-gnu/:/lib/../lib/:/usr/lib/x86_64-linux-gnu/:/usr/lib/..
/lib/:/usr/lib/gcc/x86_64-linux-gnu/9/../../../:/lib/:/usr/lib/
COLLECT_GCC_OPTIONS='-v' '-E' '-shared-libgcc' '-mtune=generic'
'-march=x86-64'

I can see the LIBRARY_PATH includes /usr/lib/x86_64-linux-gnu which contains
libdl.so file.

ckim@ckim-ubuntu:~$ F libdl.so /usr  (F: my find function)
/usr/lib32/libdl.so
/usr/alpha-linux-gnu/lib/libdl.so
/usr/libx32/libdl.so
/usr/lib/i386-linux-gnu/libdl.so
/usr/lib/x86_64-linux-gnu/libdl.so

Using "nm -D /usr/lib/x86_64-linux-gnu/libdl.so", I could see it contains
correctly dlsym (not __libc_dlsym). (I mistakenly grepped dlsym in libc.so
file yesterday and it gave me __libc_dlsym). So yes, it is not being linked
to the /usr/lib/x86_64-linux-gnu/libdl.so. (Actually it should be linked
during the run time and what I need is the include file with "extern
dlsym(...)" declaration. Right? 
The axpu_if.c file has #include <dlfcn.h> which is in
/usr/include/x86_64-linux-gnu directory which is in one of the default
include path of cc and c++. (I checked both)

So include path is correct and -ldl is used for final link command. The
dlsym is in libdl.so but I don't know why it's looking for a symbol
'dlsym@@GLIBC_2.2.5'. Can you give me any suggestions for check?
(In the source code I kept them as dlopen, dlsym now, and I guess if this
undefined reference error goes, the seg-fault will go too) 
Thank you!
Chan Kim

> -----Original Message-----
> From: Florian Weimer <fweimer@redhat.com>
> Sent: Tuesday, May 25, 2021 4:36 PM
> To: Chan Kim via Libc-help <libc-help@sourceware.org>
> Cc: ckim@etri.re.kr
> Subject: Re: undefined reference to symbol 'dlsym@@GLIBC_2.2.5' during
> qemu build
> 
> * Chan Kim via Libc-help:
> 
> >     LINK    aarch64-softmmu/qemu-system-aarch64
> >
> >     /usr/bin/ld: ../hw/misc/axpu_if.o: undefined reference to symbol
> > 'dlsym@@GLIBC_2.2.5'
> >
> >     /usr/bin/ld:
> > /usr/lib/gcc/x86_64-linux-gnu/9/../../../x86_64-linux-gnu/libdl.so:
> > error adding symbols: DSO missing from command line
> 
> This error message attempts to tell you to link with -ldl.
> 
> Thanks,
> Florian
> 

Re: undefined reference to symbol 'dlsym@@GLIBC_2.2.5' during qemu build

[Florian Weimer]

> Hello Florian,
> Thank you for the reply.
> I examined if the -ldl option is really used. I added V=1 during the qemu
> make to see the detail.

Order of command line arguments matters.  -ldl is ignored if it comes to
early if the toolchain defaults to -Wl,--as-needed.

Thanks,
Florian

RE: undefined reference to symbol 'dlsym@@GLIBC_2.2.5' during qemu build

[ckim]

Hi, Florian

Yes,as you said, the '-ldl' option was being ignored I guess.
I copied the last link command and in it, moved the -ldl option to the end.
When I separately run this modified last command, the final target
qemu-system-aarch64 is built without error message.
Now I have to figure out how to make '-Wl,as-needed' as not default or
automatically place the -ld option to the last in qemu-5.1.0 build system.
(And the debugger(ddd with gdb inside) prompt is a wait pointer. I can see
prints when the program runs))
Anyway thanks a lot for the help.

Thank you and Regards,
Chan Kim

> -----Original Message-----
> From: Libc-help <libc-help-bounces@sourceware.org> On Behalf Of Chan Kim
> via Libc-help
> Sent: Tuesday, May 25, 2021 9:17 PM
> To: 'Florian Weimer' <fweimer@redhat.com>; 'Chan Kim via Libc-help' <libc-
> help@sourceware.org>
> Subject: RE: undefined reference to symbol 'dlsym@@GLIBC_2.2.5' during
> qemu build
> 
> Hello Florian,
> Thank you for the reply.
> I examined if the -ldl option is really used. I added V=1 during the qemu
> make to see the detail.
> 
> This is the compile command for axpu_if.c which calls dlsym function.
> 
> cc -iquote
> /home/ckim/prj/abdsn/ab21sim/ab21tsim/QEMU/qemu-5.1.0/build/hw/misc -
> iquote hw/misc -iquote
> /home/ckim/prj/abdsn/ab21sim/ab21tsim/QEMU/qemu-5.1.0/tcg/i386 -isystem
> /home/ckim/prj/abdsn/ab21sim/ab21tsim/QEMU/qemu-5.1.0/linux-headers -
> isystem /home/ckim/prj/abdsn/ab21sim/ab21tsim/QEMU/qemu-5.1.0/build/linux-
> headers
> -iquote . -iquote /home/ckim/prj/abdsn/ab21sim/ab21tsim/QEMU/qemu-5.1.0
> -iquote /home/ckim/prj/abdsn/ab21sim/ab21tsim/QEMU/qemu-5.1.0/accel/tcg
> -iquote /home/ckim/prj/abdsn/ab21sim/ab21tsim/QEMU/qemu-5.1.0/include
> -iquote /home/ckim/prj/abdsn/ab21sim/ab21tsim/QEMU/qemu-
> 5.1.0/disas/libvixl
> -I/usr/include/pixman-1   -pthread -I/usr/include/glib-2.0
> -I/usr/lib/x86_64-linux-gnu/glib-2.0/include -fPIE -DPIE -m64 -mcx16 -
> D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-
> prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-
> prototypes -fno-strict-aliasing -fno-common -fwrapv -std=gnu99  -
> DBAREMETAL_USE_AXPUSIM -ldl -Wold-style-declaration -Wold-style-definition
> -Wtype-limits -Wformat-security -Wformat-y2k -Winit-self -Wignored-
> qualifiers -Wempty-body -Wnested-externs -Wendif-labels -Wexpansion-to-
> defined -Wno-missing-include-dirs -Wno-shift-negative-value -Wno-psabi -
> fstack-protector-strong  -I/usr/include/p11-kit-1  -DSTRUCT_IOVEC_DEFINED
> -I/usr/include/libpng16  -I/usr/include/spice-server -I/usr/include/spice-
> 1 -I/home/ckim/prj/abdsn/ab21sim/ab21tsim/QEMU/qemu-5.1.0/capstone/include
> -I/home/ckim/prj/abdsn/ab21sim/ab21tsim/QEMU/qemu-5.1.0/tests
> -I/home/ckim/prj/abdsn/ab21sim/ab21tsim/QEMU/qemu-5.1.0/tests/qtest -MMD -
> MP
> -MT hw/misc/axpu_if.o -MF hw/misc/axpu_if.d -g   -c -o hw/misc/axpu_if.o
> /home/ckim/prj/abdsn/ab21sim/ab21tsim/QEMU/qemu-5.1.0/hw/misc/axpu_if.c
> 
> Here cc is used to make  axpu_if.o.
> 
> and This is the last link command to build the final executable qemu-
> system-aarch64.
> 
> c++ -g  -Wl,--warn-common -Wl,-z,relro -Wl,-z,now -pie -m64
> -DBAREMETAL_USE_AXPUSIM -ldl -fstack-protector-strong  -o
> qemu-system-aarch64 exec.o exec-vary.o tcg/tcg.o tcg/tcg-op.o tcg/tcg-op-
> vec.o tcg/tcg-op-gvec.o tcg/tcg-common.o tcg/optimize.o fpu/softfloat.o
> disas.o gdbstub-xml.o gdbstub.o migration/ram.o accel/qtest.o
> accel/stubs/hax-stub.o accel/stubs/hvf-stub.o accel/stubs/whpx-stub.o
> accel/stubs/kvm-stub.o accel/stubs/xen-stub.o accel/tcg/tcg-all.o
> accel/tcg/cputlb.o accel/tcg/tcg-runtime.o accel/tcg/tcg-runtime-gvec.o
> accel/tcg/cpu-exec.o accel/tcg/cpu-exec-common.o accel/tcg/translate-all.o
> accel/tcg/translator.o dump/dump.o hw/9pfs/virtio-9p-device.o
> hw/block/virtio-blk.o hw/block/vhost-user-blk.o hw/block/dataplane/virtio-
> blk.o hw/char/virtio-serial-bus.o hw/core/machine-qmp-cmds.o
> hw/cpu/arm11mpcore.o hw/cpu/realview_mpcore.o hw/cpu/a9mpcore.o hw/c....
> (reduced) ...  verbs -libumad -lslirp -lglib-2.0 -lgio-2.0 -lgobject-2.0 -
> lglib-2.0 -lgio-2.0
> -lgobject-2.0 -lglib-2.0 -lepoxy -lgbm -lepoxy -lgbm -lepoxy -lgbm -lepoxy
> -lgbm -lepoxy -lgbm -lpulse -lbrlapi -lcacard -lglib-2.0 -lcacard -lglib-
> 2.0 -lusbredirparser -lusbredirparser -lgtk-3 -lgdk-3 -lpangocairo-1.0
> -lpango-1.0 -lharfbuzz -latk-1.0 -lcairo-gobject -lcairo -lgdk_pixbuf-2.0
> -lgio-2.0 -lgobject-2.0 -lglib-2.0 -lX11 -lvte-2.91 -lgtk-3 -lgdk-3
> -lpangocairo-1.0 -lpango-1.0 -lharfbuzz -latk-1.0 -lcairo-gobject -lcairo
> -lgdk_pixbuf-2.0 -lgio-2.0 -lgobject-2.0 -lglib-2.0 -lepoxy -lgbm -lgtk-3
> -lgdk-3 -lpangocairo-1.0 -lpango-1.0 -lharfbuzz -latk-1.0 -lcairo-gobject
> -lcairo -lgdk_pixbuf-2.0 -lgio-2.0 -lgobject-2.0 -lglib-2.0 -lX11 -lvte-
> 2.91
> -lgtk-3 -lgdk-3 -lpangocairo-1.0 -lpango-1.0 -lharfbuzz -latk-1.0 -lcairo-
> gobject -lcairo -lgdk_pixbuf-2.0 -lgio-2.0 -lgobject-2.0 -lglib-2.0 -
> lepoxy -lgbm -lgtk-3 -lgdk-3 -lpangocairo-1.0 -lpango-1.0 -lharfbuzz
> -latk-1.0 -lcairo-gobject -lcairo -lgdk_pixbuf-2.0 -lgio-2.0 -lgobject-2.0
> -lglib-2.0 -lX11 -lvte-2.91 -lgtk-3 -lgdk-3 -lpangocairo-1.0 -lpango-1.0 -
> lharfbuzz -latk-1.0 -lcairo-gobject -lcairo -lgdk_pixbuf-2.0 -lgio-2.0
> -lgobject-2.0 -lglib-2.0 -lepoxy -lgbm -lgtk-3 -lgdk-3 -lpangocairo-1.0
> -lpango-1.0 -lharfbuzz -latk-1.0 -lcairo-gobject -lcairo -lgdk_pixbuf-2.0
> -lgio-2.0 -lgobject-2.0 -lglib-2.0 -lX11 -lvte-2.91 -lgtk-3 -lgdk-3
> -lpangocairo-1.0 -lpango-1.0 -lharfbuzz -latk-1.0 -lcairo-gobject -lcairo
> -lgdk_pixbuf-2.0 -lgio-2.0 -lgobject-2.0 -lglib-2.0 -lepoxy -lgbm -Wl,-
> Bsymbolic-functions -lncursesw -ltinfo -L/usr/local/lib -lz -lxml2 -laio -
> liscsi -lcurl -lrbd -lrados -lssh -lpixman-1 -lutil -lnuma -lpng16 -lz -
> ljpeg -lsasl2 -lxenctrl -lxenstore -lxenguest -lxenforeignmemory -
> lxengnttab -lxenevtchn -lxendevicemodel -lxentoolcore  -lrdmacm -libverbs
> -libumad -lfdt -lspice-server -lpmem -
> L/home/ckim/prj/abdsn/ab21sim/ab21tsim/QEMU/qemu-5.1.0/build/capstone
> -lcapstone  -lm -lz -lgthread-2.0 -pthread -lglib-2.0 -lnettle -lgnutls -
> lrt
> 
> I reduced the .o file list above. But I could see all the .o files are
> there with -ldl option. One thing I didn't know was that the final link
> command was using c++ not cc. The default library and include settings of
> my c++ are like this :
> 
> ckim@ckim-ubuntu:~$ echo | c++ -v -x c -E - Using built-in specs.
> COLLECT_GCC=c++
> OFFLOAD_TARGET_NAMES=nvptx-none:hsa
> OFFLOAD_TARGET_DEFAULT=1
> Target: x86_64-linux-gnu
> Configured with: ../src/configure -v --with-pkgversion='Ubuntu 9.3.0-
> 17ubuntu1~20.04' --with-bugurl=file:///usr/share/doc/gcc-9/README.Bugs
> --enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++,gm2
> --prefix=/usr --with-gcc-major-version-only --program-suffix=-9
> --program-prefix=x86_64-linux-gnu- --enable-shared --enable-linker-build-
> id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix
> --libdir=/usr/lib --enable-nls --enable-clocale=gnu --enable-libstdcxx-
> debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --
> enable-gnu-unique-object --disable-vtable-verify --enable-plugin --enable-
> default-pie --with-system-zlib --with-target-system-zlib=auto --enable-
> objc-gc=auto --enable-multiarch --disable-werror
> --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --
> enable-multilib --with-tune=generic --enable-offload-targets=nvptx-
> none=/build/gcc-9-HskZEa/gcc-9-9.3.0/debian/t
> mp-nvptx/usr,hsa --without-cuda-driver --enable-checking=release --
> build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
> Thread model: posix gcc version 9.3.0 (Ubuntu 9.3.0-17ubuntu1~20.04)
> COLLECT_GCC_OPTIONS='-v' '-E' '-shared-libgcc' '-mtune=generic'
> '-march=x86-64'
>  /usr/lib/gcc/x86_64-linux-gnu/9/cc1 -E -quiet -v -imultiarch x86_64-
> linux-gnu - -mtune=generic -march=x86-64 -fasynchronous-unwind-tables -
> fstack-protector-strong -Wformat -Wformat-security -fstack-clash-
> protection -fcf-protection ignoring nonexistent directory
> "/usr/local/include/x86_64-linux-gnu"
> ignoring nonexistent directory
> "/usr/lib/gcc/x86_64-linux-gnu/9/include-fixed"
> ignoring nonexistent directory
> "/usr/lib/gcc/x86_64-linux-gnu/9/../../../../x86_64-linux-gnu/include"
> #include "..." search starts here:
> #include <...> search starts here:
>  /usr/lib/gcc/x86_64-linux-gnu/9/include
>  /usr/local/include
>  /usr/include/x86_64-linux-gnu
>  /usr/include
> End of search list.
> # 1 "<stdin>"
> # 1 "<built-in>"
> # 1 "<command-line>"
> # 31 "<command-line>"
> # 1 "/usr/include/stdc-predef.h" 1 3 4
> # 32 "<command-line>" 2
> # 1 "<stdin>"
> COMPILER_PATH=/usr/lib/gcc/x86_64-linux-gnu/9/:/usr/lib/gcc/x86_64-linux-
> gnu
> /9/:/usr/lib/gcc/x86_64-linux-gnu/:/usr/lib/gcc/x86_64-linux-
> gnu/9/:/usr/lib
> /gcc/x86_64-linux-gnu/
> LIBRARY_PATH=/usr/lib/gcc/x86_64-linux-gnu/9/:/usr/lib/gcc/x86_64-linux-
> gnu/
> 9/../../../x86_64-linux-gnu/:/usr/lib/gcc/x86_64-linux-
> gnu/9/../../../../lib
> /:/lib/x86_64-linux-gnu/:/lib/../lib/:/usr/lib/x86_64-linux-
> gnu/:/usr/lib/..
> /lib/:/usr/lib/gcc/x86_64-linux-gnu/9/../../../:/lib/:/usr/lib/
> COLLECT_GCC_OPTIONS='-v' '-E' '-shared-libgcc' '-mtune=generic'
> '-march=x86-64'
> 
> I can see the LIBRARY_PATH includes /usr/lib/x86_64-linux-gnu which
> contains libdl.so file.
> 
> ckim@ckim-ubuntu:~$ F libdl.so /usr  (F: my find function)
> /usr/lib32/libdl.so /usr/alpha-linux-gnu/lib/libdl.so /usr/libx32/libdl.so
> /usr/lib/i386-linux-gnu/libdl.so /usr/lib/x86_64-linux-gnu/libdl.so
> 
> Using "nm -D /usr/lib/x86_64-linux-gnu/libdl.so", I could see it contains
> correctly dlsym (not __libc_dlsym). (I mistakenly grepped dlsym in libc.so
> file yesterday and it gave me __libc_dlsym). So yes, it is not being
> linked to the /usr/lib/x86_64-linux-gnu/libdl.so. (Actually it should be
> linked during the run time and what I need is the include file with
> "extern dlsym(...)" declaration. Right?
> The axpu_if.c file has #include <dlfcn.h> which is in /usr/include/x86_64-
> linux-gnu directory which is in one of the default include path of cc and
> c++. (I checked both)
> 
> So include path is correct and -ldl is used for final link command. The
> dlsym is in libdl.so but I don't know why it's looking for a symbol
> 'dlsym@@GLIBC_2.2.5'. Can you give me any suggestions for check?
> (In the source code I kept them as dlopen, dlsym now, and I guess if this
> undefined reference error goes, the seg-fault will go too) Thank you!
> Chan Kim
> 
> > -----Original Message-----
> > From: Florian Weimer <fweimer@redhat.com>
> > Sent: Tuesday, May 25, 2021 4:36 PM
> > To: Chan Kim via Libc-help <libc-help@sourceware.org>
> > Cc: ckim@etri.re.kr
> > Subject: Re: undefined reference to symbol 'dlsym@@GLIBC_2.2.5' during
> > qemu build
> >
> > * Chan Kim via Libc-help:
> >
> > >     LINK    aarch64-softmmu/qemu-system-aarch64
> > >
> > >     /usr/bin/ld: ../hw/misc/axpu_if.o: undefined reference to symbol
> > > 'dlsym@@GLIBC_2.2.5'
> > >
> > >     /usr/bin/ld:
> > > /usr/lib/gcc/x86_64-linux-gnu/9/../../../x86_64-linux-gnu/libdl.so:
> > > error adding symbols: DSO missing from command line
> >
> > This error message attempts to tell you to link with -ldl.
> >
> > Thanks,
> > Florian
> >
> 
> 
> 
> 

Re: undefined reference to symbol 'dlsym@@GLIBC_2.2.5' during qemu build

[Jeffrey Walton]

On Tue, May 25, 2021 at 9:29 PM Chan Kim via Libc-help
<libc-help@sourceware.org> wrote:
>
> Hi, Florian
>
> Yes,as you said, the '-ldl' option was being ignored I guess.
> I copied the last link command and in it, moved the -ldl option to the end.
> When I separately run this modified last command, the final target
> qemu-system-aarch64 is built without error message.
> Now I have to figure out how to make '-Wl,as-needed' as not default or
> automatically place the -ld option to the last in qemu-5.1.0 build system.
> (And the debugger(ddd with gdb inside) prompt is a wait pointer. I can see
> prints when the program runs))
> Anyway thanks a lot for the help.

The make recipe is off a bit.

Ld is a single pass linker. -ldl needs to be near the end of the list
of object files and libraries, not the beginning. I.e., it should be
something like:

foo: <objs>
    $(CC) $(CFLAGS) <objs> $(LDFLAGS) $(LDLIBS) -ldl

Jeff

RE: undefined reference to symbol 'dlsym@@GLIBC_2.2.5' during qemu build

[ckim]

Hi, Jeffery, and all,

After reading your comment, I briefly looked into the qemu build script (configure script and the generated Makefiles)
And looks like it follows your flag placement rule.
So I figured out how to do it. Now the configure script is this (under build directory):

../configure --target-list=aarch64-softmmu --enable-debug --enable-gtk --extra-ldflags="-Wl,--no-as-needed -ldl"

This makes the link flag placed at the end. Now the undefined reference to symbol 'dlsym@@GLIBC_2.2.5' error is gone, and dlopen, dlsym works ok.
The debugger hang problem was in the ./ddd/init file. 
(I changed "set extended-prompt not set\n\"  to "set extended-prompt (gdb) \n\"  https://stackoverflow.com/questions/2914003/ddd-hangs-on-start)

Now I can do the normal debugging.
Thanks a lot!
Chan Kim

> The make recipe is off a bit.
> 
> Ld is a single pass linker. -ldl needs to be near the end of the list of
> object files and libraries, not the beginning. I.e., it should be
> something like:
> 
> foo: <objs>
>     $(CC) $(CFLAGS) <objs> $(LDFLAGS) $(LDLIBS) -ldl
> 
> Jeff