puzzling ld behavior, failing to find symbols from .so after adding more .o file

[Michael Tokarev <mjt@tls.msk.ru>]

Hi!

I've got a puzzling issue here which I need help with,
at least some pointers as of where to dig.

The context is - a debian system, with its current gcc & binutils
(unstable). I'm trying to update samba in debian to current version.
And the build fails during linking of one particular .so file (so far).
And fails in somewhat interesting way.  The whole thing is large with
a complex build system.

I took out this link command line and work with it only. From a long
list of .o files which was there, I started with just one .o file,
which links fine. And am adding the other .o files one by one.  And
at one point it starts failing.

Basically, it boils down to this.

  gcc -shared -o foo.so a.o -lbar -lbaz  -- works
  gcc -shared -o foo.so a.o b.o -lbar -lbaz -- fails

but it reports symbols like bar_init which are used by a.o as missing.

It is more.  When I replace -lbar (which is a system library,
and it uses .so file by default) with static libbar.a, it works
again even with b.o added to the mix -- "works" means it is now
able to find bar_init just fine, but it now start listing symbols
used by a.o from -lbaz too - which, again, can be fixed by replacing
-lbaz with libbaz.a. And so on.

it *looks* like after adding b.o into the mix, the linker stops
seeing symbols in shared libraries, only can use static libs.

b.o is a regular object file, I see nothing wrong with it. Yet it
makes whole thing to fail.

The complete command line is this (I preprocess this with grep -v # to run):

--- cut ---
gcc
-Wl,--as-needed
-Wl,--version-script=source3/smbconf.vscript
-shared
-Wl,-h,libsmbconf.so.0
lib/util/util_pw.c.99.o
librpc/gen_ndr/ndr_server_id.c.40.o
lib/async_req/async_sock.c.1.o
#source3/lib/cluster_support.c.45.o
source3/lib/dbwrap/dbwrap_ctdb.c.45.o
#source3/lib/messages_ctdb.c.45.o
#source3/lib/messages_ctdb_ref.c.45.o
#source3/lib/ctdbd_conn.c.45.o
#source3/lib/smbconf/smbconf_init.c.77.o
#source3/lib/smbconf/smbconf_reg.c.77.o
source3/lib/system.c.44.o
source3/lib/sendfile.c.44.o
source3/lib/recvfile.c.44.o
source3/lib/time.c.44.o
source3/lib/util_sid.c.44.o
source3/lib/util_specialsids.c.44.o
source3/lib/util_file.c.44.o
#source3/lib/util.c.44.o
#source3/lib/util_path.c.44.o
#source3/lib/util_matching.c.44.o
source3/lib/util_procid.c.44.o
#source3/lib/util_sock.c.44.o
source3/lib/util_tsock.c.44.o
source3/lib/util_transfer_file.c.44.o
source3/lib/util_macstreams.c.44.o
#source3/lib/util_tdb.c.116.o
source3/param/util.c.1.o
#source3/lib/messages.c.52.o
#source3/lib/util_cluster.c.52.o
#source3/lib/id_cache.c.52.o
#source3/lib/serverid.c.52.o
#source3/lib/server_id_watch.c.52.o
#source3/lib/server_id_db_util.c.52.o
source3/lib/addrchange.c.52.o
#lib/util/debug_s3.c.52.o
#source3/lib/dumpcore.c.52.o
#source3/lib/interface.c.52.o
#source3/lib/username.c.52.o
source3/lib/smbrun.c.52.o
#source3/lib/wins_srv.c.52.o
#source3/lib/substitute.c.52.o
source3/lib/substitute_generic.c.52.o
source3/lib/ms_fnmatch.c.52.o
#source3/lib/tallocmsg.c.52.o
#source3/lib/dmallocmsg.c.52.o
#source3/lib/gencache.c.52.o
source3/lib/util_event.c.52.o
#source3/lib/global_contexts.c.52.o
source3/lib/ldap_escape.c.52.o
source3/lib/system_smbd.c.52.o
source3/lib/audit.c.52.o
#source3/lib/idmap_cache.c.52.o
#source3/lib/namemap_cache.c.52.o
source3/lib/util_ea.c.52.o
#source3/lib/background.c.52.o
source3/lib/version.c.109.o
#source3/lib/dbwrap/dbwrap_open.c.48.o
#source3/lib/dbwrap/dbwrap_watch.c.48.o
#source3/lib/g_lock.c.48.o
lib/smbconf/smbconf.c.1.o
lib/smbconf/smbconf_txt.c.1.o
lib/smbconf/smbconf_util.c.1.o
#source3/registry/reg_backend_smbconf.c.36.o
#source3/registry/reg_init_smbconf.c.36.o
source3/registry/reg_util_token.c.36.o
#source3/registry/reg_api_util.c.36.o
#source3/registry/reg_api.c.35.o
source3/registry/reg_dispatcher.c.35.o
#source3/registry/reg_cachehook.c.35.o
source3/registry/reg_objects.c.35.o
source3/registry/reg_util_internal.c.35.o
source3/lib/util_nttoken.c.35.o
#source3/registry/reg_backend_db.c.35.o
#source3/registry/reg_parse_internal.c.35.o
#source3/registry/reg_parse.c.35.o
source3/lib/srprs.c.35.o
#source3/registry/reg_init_basic.c.35.o
#source3/param/loadparm_ctx.c.2.o
librpc/gen_ndr/ndr_messaging.c.154.o
source3/libsmb/errormap.c.126.o
source3/libsmb/smberr.c.126.o
source3/lib/errmap_unix.c.126.o
#source3/param/loadparm.c.76.o
#source3/lib/sharesec.c.76.o
#source3/lib/ldap_debug_handler.c.76.o
#source3/lib/util_names.c.76.o
-osource3/libsmbconf.so
-Lthird_party/heimdal_build
-Lsource4/lib/events
-Llib/tdb_wrap
-Llib
-Llibrpc
-Llib/replace
-Llib/socket
-Llibcli/util
-Llib/param
-Llib/messaging
-Llibcli/registry
-Llib/dbwrap
-Llib/util
-Lsource3
-Llibcli/security
-L/usr/local/lib
-L/usr/local/lib
-lsamba-security-samba4
-lmessages-util-samba4
-lserver-id-db-samba4
-lsamba-debug-samba4
-lsamba-util
-ldbwrap-samba4
-ltevent-util
-lsamba3-util-samba4
-lutil-tdb-samba4
-lutil-reg-samba4
-lsys-rw-samba4
-ltalloc-report-printf-samba4
-lCHARSET3-samba4
-lmessages-dgm-samba4
-lsamba-hostconfig
-lsamba-errors
-linterfaces-samba4
-lreplace-samba4
-lndr
-liov-buf-samba4
-lsamba-sockets-samba4
-lsmbd-shim-samba4
-lsocket-blocking-samba4
-ltdb-wrap-samba4
-ltime-basic-samba4
-lutil-setid-samba4
-lgenrand-samba4
-lmsghdr-samba4
-lserver-role-samba4
-lndr-nbt
-levents-samba4
-lroken-samba4
-lutil
-lresolv
-licudata
-licui18n
-licuuc
-lbsd
-lz
-ldl
-lgnutls
-ltdb
-lcups
-ltalloc
-lpthread
-llber
-lldap
-lcap
-ltevent
-ltalloc
-lsystemd
-Wl,-z,relro
-Wl,-z,now
-Wl,-z,relro,-z,now
-Wl,-no-undefined
-Wl,--export-dynamic
--- cut ---

There, the b.o which makes things fail is, for example,
source3/lib/dbwrap/dbwrap_ctdb.c.45.o (not only this one,
some other .o files trigger this same issue too).

Here's the output of the above command:

--- cut ---
/usr/bin/ld: lib/async_req/async_sock.c.1.o: in function `async_connect_connected':
./bin/default/../../lib/async_req/async_sock.c:194: undefined reference to `_talloc_get_type_abort'
/usr/bin/ld: ./bin/default/../../lib/async_req/async_sock.c:197: undefined reference to `_talloc_get_type_abort'
/usr/bin/ld: lib/async_req/async_sock.c.1.o: in function `writev_cleanup':
./bin/default/../../lib/async_req/async_sock.c:318: undefined reference to `_talloc_get_type_abort'
/usr/bin/ld: ./bin/default/../../lib/async_req/async_sock.c:320: undefined reference to `_talloc_free'
/usr/bin/ld: ./bin/default/../../lib/async_req/async_sock.c:321: undefined reference to `_talloc_free'
/usr/bin/ld: lib/async_req/async_sock.c.1.o: in function `read_packet_cleanup':
./bin/default/../../lib/async_req/async_sock.c:509: undefined reference to `_talloc_get_type_abort'
/usr/bin/ld: ./bin/default/../../lib/async_req/async_sock.c:511: undefined reference to `_talloc_free'
/usr/bin/ld: lib/async_req/async_sock.c.1.o: in function `wait_for_read_cleanup':
./bin/default/../../lib/async_req/async_sock.c:641: undefined reference to `_talloc_get_type_abort'
/usr/bin/ld: ./bin/default/../../lib/async_req/async_sock.c:643: undefined reference to `_talloc_free'
/usr/bin/ld: lib/async_req/async_sock.c.1.o: in function `async_connect_cleanup':
./bin/default/../../lib/async_req/async_sock.c:167: undefined reference to `_talloc_get_type_abort'
/usr/bin/ld: ./bin/default/../../lib/async_req/async_sock.c:169: undefined reference to `_talloc_free'
/usr/bin/ld: lib/async_req/async_sock.c.1.o: in function `writev_cancel':
./bin/default/../../lib/async_req/async_sock.c:326: undefined reference to `_talloc_get_type_abort'
/usr/bin/ld: ./bin/default/../../lib/async_req/async_sock.c:335: undefined reference to `_talloc_free'
/usr/bin/ld: ./bin/default/../../lib/async_req/async_sock.c:336: undefined reference to `_talloc_free'
/usr/bin/ld: lib/async_req/async_sock.c.1.o: in function `writev_trigger':
./bin/default/../../lib/async_req/async_sock.c:380: undefined reference to `_talloc_get_type_abort'
/usr/bin/ld: lib/async_req/async_sock.c.1.o: in function `wait_for_read_done':
./bin/default/../../lib/async_req/async_sock.c:651: undefined reference to `_talloc_get_type_abort'
/usr/bin/ld: ./bin/default/../../lib/async_req/async_sock.c:654: undefined reference to `_talloc_get_type_abort'
/usr/bin/ld: lib/async_req/async_sock.c.1.o: in function `writev_handler':
./bin/default/../../lib/async_req/async_sock.c:399: undefined reference to `_talloc_get_type_abort'
/usr/bin/ld: ./bin/default/../../lib/async_req/async_sock.c:402: undefined reference to `_talloc_get_type_abort'
/usr/bin/ld: lib/async_req/async_sock.c.1.o:./bin/default/../../lib/async_req/async_sock.c:518: more undefined references to `_talloc_get_type_abort' 
follow
/usr/bin/ld: lib/async_req/async_sock.c.1.o: in function `read_packet_handler':
./bin/default/../../lib/async_req/async_sock.c:522: undefined reference to `talloc_get_size'
/usr/bin/ld: lib/async_req/async_sock.c.1.o: in function `accept_handler':
./bin/default/../../lib/async_req/async_sock.c:732: undefined reference to `_talloc_get_type_abort'
/usr/bin/ld: ./bin/default/../../lib/async_req/async_sock.c:734: undefined reference to `_talloc_get_type_abort'
/usr/bin/ld: ./bin/default/../../lib/async_req/async_sock.c:737: undefined reference to `_talloc_free'
/usr/bin/ld: lib/async_req/async_sock.c.1.o: in function `writev_send':
./bin/default/../../lib/async_req/async_sock.c:275: undefined reference to `_talloc_memdup'
/usr/bin/ld: lib/async_req/async_sock.c.1.o: in function `writev_recv':
./bin/default/../../lib/async_req/async_sock.c:442: undefined reference to `_talloc_get_type_abort'
/usr/bin/ld: lib/async_req/async_sock.c.1.o: in function `read_packet_recv':
./bin/default/../../lib/async_req/async_sock.c:588: undefined reference to `_talloc_get_type_abort'
/usr/bin/ld: lib/async_req/async_sock.c.1.o: in function `accept_recv':
./bin/default/../../lib/async_req/async_sock.c:767: undefined reference to `_talloc_get_type_abort'
/usr/bin/ld: lib/async_req/async_sock.c.1.o: in function `read_packet_recv':
./bin/default/../../lib/async_req/async_sock.c:596: undefined reference to `talloc_get_size'
/usr/bin/ld: lib/util/util_pw.c.99.o: in function `tcopy_passwd':
./bin/default/../../lib/util/util_pw.c:50: undefined reference to `talloc_strdup'
/usr/bin/ld: ./bin/default/../../lib/util/util_pw.c:51: undefined reference to `talloc_strdup'
/usr/bin/ld: ./bin/default/../../lib/util/util_pw.c:54: undefined reference to `talloc_strdup'
/usr/bin/ld: ./bin/default/../../lib/util/util_pw.c:55: undefined reference to `talloc_strdup'
/usr/bin/ld: ./bin/default/../../lib/util/util_pw.c:56: undefined reference to `talloc_strdup'
/usr/bin/ld: source3/lib/dbwrap/dbwrap_ctdb.c.45.o: in function `db_ctdb_transaction_cancel':
./bin/default/../../source3/lib/dbwrap/dbwrap_ctdb.c:876: undefined reference to `_talloc_get_type_abort'
/usr/bin/ld: ./bin/default/../../source3/lib/dbwrap/dbwrap_ctdb.c:896: undefined reference to `_talloc_free'
/usr/bin/ld: source3/lib/dbwrap/dbwrap_ctdb.c.45.o: in function `db_ctdb_record_destructor':
./bin/default/../../source3/lib/dbwrap/dbwrap_ctdb.c:563: undefined reference to `_talloc_get_type_abort'
/usr/bin/ld: ./bin/default/../../source3/lib/dbwrap/dbwrap_ctdb.c:564: undefined reference to `_talloc_get_type_abort'
/usr/bin/ld: source3/lib/dbwrap/dbwrap_ctdb.c.45.o: in function `db_ctdb_id':
./bin/default/../../source3/lib/dbwrap/dbwrap_ctdb.c:1772: undefined reference to `_talloc_get_type_abort'
/usr/bin/ld: source3/lib/dbwrap/dbwrap_ctdb.c.45.o: in function `pull_newest_from_marshall_buffer_parser':
./bin/default/../../source3/lib/dbwrap/dbwrap_ctdb.c:463: undefined reference to `_talloc_memdup'
/usr/bin/ld: source3/lib/dbwrap/dbwrap_ctdb.c.45.o: in function `db_ctdb_transaction_start':
./bin/default/../../source3/lib/dbwrap/dbwrap_ctdb.c:343: undefined reference to `_talloc_get_type_abort'
/usr/bin/ld: ./bin/default/../../source3/lib/dbwrap/dbwrap_ctdb.c:359: undefined reference to `_talloc_zero'
/usr/bin/ld: ./bin/default/../../source3/lib/dbwrap/dbwrap_ctdb.c:367: undefined reference to `talloc_asprintf'
/usr/bin/ld: ./bin/default/../../source3/lib/dbwrap/dbwrap_ctdb.c:378: undefined reference to `g_lock_lock'
/usr/bin/ld: ./bin/default/../../source3/lib/dbwrap/dbwrap_ctdb.c:386: undefined reference to `_talloc_set_destructor'
/usr/bin/ld: ./bin/default/../../source3/lib/dbwrap/dbwrap_ctdb.c:382: undefined reference to `_talloc_free'
/usr/bin/ld: ./bin/default/../../source3/lib/dbwrap/dbwrap_ctdb.c:371: undefined reference to `_talloc_free'
/usr/bin/ld: source3/lib/dbwrap/dbwrap_ctdb.c.45.o: in function `db_ctdb_transaction_destructor':
./bin/default/../../source3/lib/dbwrap/dbwrap_ctdb.c:325: undefined reference to `g_lock_unlock'
/usr/bin/ld: source3/lib/dbwrap/dbwrap_ctdb.c.45.o: in function `db_ctdb_get_seqnum':
./bin/default/../../source3/lib/dbwrap/dbwrap_ctdb.c:1765: undefined reference to `_talloc_get_type_abort'
/usr/bin/ld: source3/lib/dbwrap/dbwrap_ctdb.c.45.o: in function `db_ctdbd_traverse':
./bin/default/../../source3/lib/dbwrap/dbwrap_ctdb.c:1569: undefined reference to `lp_ctdb_timeout'
/usr/bin/ld: ./bin/default/../../source3/lib/dbwrap/dbwrap_ctdb.c:1569: undefined reference to `lp_ctdbd_socket'
/usr/bin/ld: ./bin/default/../../source3/lib/dbwrap/dbwrap_ctdb.c:1569: undefined reference to `ctdbd_init_connection'
/usr/bin/ld: ./bin/default/../../source3/lib/dbwrap/dbwrap_ctdb.c:1578: undefined reference to `ctdbd_traverse'
/usr/bin/ld: ./bin/default/../../source3/lib/dbwrap/dbwrap_ctdb.c:1579: undefined reference to `_talloc_free'
/usr/bin/ld: source3/lib/dbwrap/dbwrap_ctdb.c.45.o: in function `db_ctdb_traverse_read':
./bin/default/../../source3/lib/dbwrap/dbwrap_ctdb.c:1732: undefined reference to `_talloc_get_type_abort'
/usr/bin/ld: ./bin/default/../../source3/lib/dbwrap/dbwrap_ctdb.c:1747: undefined reference to `tdb_traverse_read'
/usr/bin/ld: source3/lib/dbwrap/dbwrap_ctdb.c.45.o: in function `db_ctdb_traverse':
./bin/default/../../source3/lib/dbwrap/dbwrap_ctdb.c:1597: undefined reference to `_talloc_get_type_abort'
/usr/bin/ld: ./bin/default/../../source3/lib/dbwrap/dbwrap_ctdb.c:1612: undefined reference to `tdb_traverse'
/usr/bin/ld: ./bin/default/../../source3/lib/dbwrap/dbwrap_ctdb.c:1638: undefined reference to `tdb_exists'
/usr/bin/ld: ./bin/default/../../source3/lib/dbwrap/dbwrap_ctdb.c:1639: undefined reference to `tdb_null'
/usr/bin/ld: source3/lib/dbwrap/dbwrap_ctdb.c.45.o: relocation R_X86_64_PC32 against undefined hidden symbol `tdb_null' can not be used when making a 
shared object
/usr/bin/ld: final link failed: bad value
collect2: error: ld returned 1 exit status
--- cut ---

The symbols which are now reported as missing (in util_pw.o or async_sock.o) are
all talloc_* -- al the symbols from the system -ltalloc.  Once again, after
omitting dbwrap_ctdb.o, it all links fine, finding all talloc symbols without
an issue.

http://www.corpit.ru/mjt/tmp/dbwrap_ctdb.c.45.o is this .o file (x86_64).

(The tdb_null message is most likely due to it being unable to find this
symbol in -ltdb the same way as it can't find symbols in -ltalloc).

I'm quite lost here. The question is: why after ecountering dbwrap_ctdb.o,
ld stops finding symbols in shared libraries which it were finding before,
but still being able to find symbols in static libs?  How to debug this,
maybe to make a test case?

Is there a way to tell ld to emit some debugging info about its symbol
lookup/resolution happenings?

gcc (Debian 11.2.0-19) 11.2.0, from 4:11.2.0-2 debian package.
binutils 2.38, from 2.38-3 debian package.

Thank you!

/mjt