MIPSEL GLIBC sem_init() not shared

MIPSEL GLIBC sem_init() not shared

[Alessandro Carminati]

I'm currently developing an application that is meant to run on Linux
for MIPSEL platform. The application uses POSIX semaphores inside one
of its DSO, and the attention in my case is for the sem_init()
function used inside one DSO on which the app depends.

The application is currently using the GLIBC_2.31, but I observed the
same phenomenon also using the GLIBC_2_26.

Now I'm going to explain the anomaly and the current status of the
troubleshooting.

All happened when I noticed a process waiting on a semaphore wouldn't
wake when the semaphore incremented its value.

Just for the records, I'm aware that for multiprocessing, the
semaphore must sit in a memory area that is shared between the
processes that use the semaphore.

Now, investigating further using strace, I also noticed that the
semaphore I initialized using sem_init(&semaphore, 1, 0) resulted in
being private, despite I initialized it to be shared.

futex(0x774ce0bc, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1,
{tv_sec=1615204364, tv_nsec=515578961}, FUTEX_BITSET_MATCH_ANY) = -1
ETIMEDOUT (Connection timed out)

The next step I performed on this issue has been to debug using gdb
the semaphore initialization process. There I discovered that in this
particular combination, MIPSEL - LINUX - GLIBC, the sem_init() exists
in two versions:

$ mipsel-linux-objdump -T libpthread.so.0  | grep sem_init
00012e30 g    DF .text  00000040 (GLIBC_2.0)  sem_init
00012dd0 g    DF .text  00000060  GLIBC_2.2   sem_init

For some reason, the linker decided to link my code against the older
(not supporting shared semaphores) instead of the newer.

Looking at the GLIBC code, particularly where the sem_init() is
implemented (/nptl/sem_init.c), I realized that sem_init() is an alias
and that two symbols identify the functions' implementation
univocally.

My next move then has been to call directly the function I needed
__new_sem_init(), just to be sure any linker alchemy I'm not aware of
would interfere with my intentions.

Unfortunately, the symbol is not exported, and it can't be used.

Looking at my library liba.so which uses sem_wait(), sem_init(), and
sem_post() I noticed that these symbols are all exported by
libpthread.so.0.

$ mipsel-linux-nm -D libpthread.so.0 | grep sem_
00014a10 T sem_clockwait
00013784 T sem_close
00012e70 T sem_destroy
00012e70 T sem_destroy
00013b14 T sem_getvalue
00013b00 T sem_getvalue
00012e30 T sem_init
00012dd0 T sem_init
000131b8 T sem_open
00014ab0 T sem_post
00014b90 T sem_post
00014508 T sem_timedwait
00013f40 T sem_trywait
00013fac T sem_trywait
00013920 T sem_unlink
00013eb0 T sem_wait
00014020 T sem_wait

But looking at the liba.so dependencies, I've been surprised by the
fact that the library depends only on the main libc library
(libc.so.6) and has no dependency on libpthread.so.0 where the actual
sem_* symbols are implemented.

$ /lib/ld-2.31.so --list /usr/lib/liba.so
        linux-vdso.so.1 (0x7ffaa000)
        libc.so.6 => /lib/libc.so.6 (0x77ddc000)
        /lib/ld-2.31.so (0x77f8a000)

I guess this might be related to my problem, but I have no clue how to
link these two facts. How the liba.so could be compiled and linked
using symbols its dependencies do not provide?

$ mipsel-linux-objdump -T libc.so.6  | grep sem
000fab80 g    DF .text  00000070  GLIBC_2.0   semget
000fabf0 g    DF .text  000000b0  GLIBC_2.2   semctl
000faca0  w   DF .text  00000074  GLIBC_2.3.3 semtimedop
000359d0 g    DF .text  00000074  GLIBC_2.0   sigisemptyset
00149854 g    DF .text  000000b0 (GLIBC_2.0)  semctl
000fab60 g    DF .text  00000018  GLIBC_2.0   semop
$

How to force the GNU linker to link against sem_init@@GLIBC_2.2
instead of sem_init@GLIBC_2.0?

Re: MIPSEL GLIBC sem_init() not shared

[Florian Weimer]

* Alessandro Carminati via Libc-help:

> But looking at the liba.so dependencies, I've been surprised by the
> fact that the library depends only on the main libc library
> (libc.so.6) and has no dependency on libpthread.so.0 where the actual
> sem_* symbols are implemented.
>
> $ /lib/ld-2.31.so --list /usr/lib/liba.so
>         linux-vdso.so.1 (0x7ffaa000)
>         libc.so.6 => /lib/libc.so.6 (0x77ddc000)
>         /lib/ld-2.31.so (0x77f8a000)
>
> I guess this might be related to my problem, but I have no clue how to
> link these two facts. How the liba.so could be compiled and linked
> using symbols its dependencies do not provide?

This is called underlinking.  It produces invalid objects.

> How to force the GNU linker to link against sem_init@@GLIBC_2.2
> instead of sem_init@GLIBC_2.0?

Just link with -lpthread, it will add the symbol version and call the
right function.

We are working on changes to glibc that makes it much less likely that
such underlink can happen, so hopefully such difficult-to-debug problems
will become a non-issue at some point in the future.

Thanks,
Florian

Fwd: MIPSEL GLIBC sem_init() not shared

[Alessandro Carminati]

On Fri, Mar 12, 2021 at 11:49 AM Florian Weimer <fweimer@redhat.com> wrote:
>
> * Alessandro Carminati via Libc-help:
>
> > But looking at the liba.so dependencies, I've been surprised by the
> > fact that the library depends only on the main libc library
> > (libc.so.6) and has no dependency on libpthread.so.0 where the actual
> > sem_* symbols are implemented.
> >
> > $ /lib/ld-2.31.so --list /usr/lib/liba.so
> >         linux-vdso.so.1 (0x7ffaa000)
> >         libc.so.6 => /lib/libc.so.6 (0x77ddc000)
> >         /lib/ld-2.31.so (0x77f8a000)
> >
> > I guess this might be related to my problem, but I have no clue how to
> > link these two facts. How the liba.so could be compiled and linked
> > using symbols its dependencies do not provide?
>
> This is called underlinking.  It produces invalid objects.
>
> > How to force the GNU linker to link against sem_init@@GLIBC_2.2
> > instead of sem_init@GLIBC_2.0?
>
> Just link with -lpthread, it will add the symbol version and call the
> right function.
>
> We are working on changes to glibc that makes it much less likely that
> such underlink can happen, so hopefully such difficult-to-debug problems
> will become a non-issue at some point in the future.
>
> Thanks,
> Florian
>
Yes, it was just as you said. Underlinking. My fault. I knew it was
related, but I put the -lptread in the wrong place.
Thanks for the answer, it resolved the main issue.

If I can, I'd like to ask you a couple of questions about the same scenario.
This time was the port on MIPSEL, but before this port, I compiled
successfully (without -lpthread) on two other architectures: ARM32 and
x64.
Which is the mechanism why in other architecture, the libpthread was
included and in MIPS it didn't?

I understand that because the libpthread is a dependency of the
executable that also depends on liba.so, at runtime, the dynamic
linker has been able to resolve the sem_init call.
My question is, why it resolved the call picking the
sem_init@GLIBC_2.0 instead of sem_init@@GLIBC_2.2?

Re: Fwd: MIPSEL GLIBC sem_init() not shared

[Florian Weimer]

* Alessandro Carminati via Libc-help:

> This time was the port on MIPSEL, but before this port, I compiled
> successfully (without -lpthread) on two other architectures: ARM32 and
> x64.  Which is the mechanism why in other architecture, the libpthread
> was included and in MIPS it didn't?

32-bit Arm starts at GLIBC_2.4, x86-64 starts at GLIBC_2.2.5.  The old
versions aren't there, so an underlinked library cannot bind against
them.

> My question is, why it resolved the call picking the
> sem_init@GLIBC_2.0 instead of sem_init@@GLIBC_2.2?

glibc originally did not have symbol versioning, so for backwards
compatibility, unversioned binaries need to pick the oldest available
version.  In theory, we could provide better diagnostics.  But we're
working on moving all libpthread symbols in to libc.so.6, so that their
versions are always visible to the link editor (unless you link with
-nostdblib, but then you are on your own anyway).  That's why the
diagnostics do not seem to be particularly useful in the medium term.

Thanks,
Florian

Re: Fwd: MIPSEL GLIBC sem_init() not shared

[Alessandro Carminati]

On Fri, Mar 12, 2021 at 1:17 PM Florian Weimer <fweimer@redhat.com> wrote:
>
> * Alessandro Carminati via Libc-help:
>
> > This time was the port on MIPSEL, but before this port, I compiled
> > successfully (without -lpthread) on two other architectures: ARM32 and
> > x64.  Which is the mechanism why in other architecture, the libpthread
> > was included and in MIPS it didn't?
>
> 32-bit Arm starts at GLIBC_2.4, x86-64 starts at GLIBC_2.2.5.  The old
> versions aren't there, so an underlinked library cannot bind against
> them.
I noticed this fact, and I wondered if I could change anything in the
configuration of the glibc to remove the __old_sem_init.
Obviously, I now have a better solution, so I'll stick with that.
I also checked the DT_NEEDED in the liba.so in these two architectures.
They have libpthread even if I compiled them without -lpthread
I must assume this as a specificity of their version of the GCC.

>
> > My question is, why it resolved the call picking the
> > sem_init@GLIBC_2.0 instead of sem_init@@GLIBC_2.2?
>
> glibc originally did not have symbol versioning, so for backwards
> compatibility, unversioned binaries need to pick the oldest available
> version.  In theory, we could provide better diagnostics.  But we're
> working on moving all libpthread symbols in to libc.so.6, so that their
> versions are always visible to the link editor (unless you link with
> -nostdblib, but then you are on your own anyway).  That's why the
> diagnostics do not seem to be particularly useful in the medium term.
What you say makes sense. And explains the behavior I observed in my code.
But "unversioned binaries need to pick the oldest available version"
is in contrast with what you can read on the internet about the same
argument.
I must admit that before stepping into this issue, I ignored the
symbol versioning even exists.

By the way, here's what I found on the internet:
http://peeterjoot.com/2019/09/20/an-example-of-linux-glibc-symbol-versioning/
"The @@ one means that it applies to new code, whereas the
@MYSTUFF_1.1 is a load only function, and no new code can use that
symbol."
https://developers.redhat.com/blog/2019/08/01/how-the-gnu-c-library-handles-backward-compatibility/
"The @@ tells the dynamic linker that this version is the default
version."
https://web.archive.org/web/20100430151127/http://www.trevorpounds.com/blog/?33
"The double @@ can only be defined once for a given symbol since it
denotes the default version to use."

Al the sources I have found seems to agree on the fact that if a
symbol is unversioned, like in my case, the symbol the dynamic linker
should use is the one that carries the "@@"
In my scenario, the symbol which carries the double @@ is the
sem_init@@GLIBC_2_2.
If the dynamic linker would have picked it, as it should, if I
correctly interpret what I read, I wouldn't be here asking.
Could you add a comment on that?
Thank you
Alessandro

Re: Fwd: MIPSEL GLIBC sem_init() not shared

[Florian Weimer]

* Alessandro Carminati via Libc-help:

> Al the sources I have found seems to agree on the fact that if a
> symbol is unversioned, like in my case, the symbol the dynamic linker
> should use is the one that carries the "@@"

I think they talk about the link editor (static linker) chosing the
symbol version, and not about what happens at run time.

> In my scenario, the symbol which carries the double @@ is the
> sem_init@@GLIBC_2_2.
> If the dynamic linker would have picked it, as it should, if I
> correctly interpret what I read, I wouldn't be here asking.

Old binaries originally linked against glibc without symbol versioning
would also use sem_init@@GLIBC_2.2 as the result, but that would result
in breakage because the type sizes are different.

Thanks,
Florian

Re: Fwd: MIPSEL GLIBC sem_init() not shared

[Alessandro Carminati]

On Fri, Mar 12, 2021 at 5:04 PM Florian Weimer <fweimer@redhat.com> wrote:
>
> * Alessandro Carminati via Libc-help:
>
> > Al the sources I have found seems to agree on the fact that if a
> > symbol is unversioned, like in my case, the symbol the dynamic linker
> > should use is the one that carries the "@@"
>
> I think they talk about the link editor (static linker) chosing the
> symbol version, and not about what happens at run time.

Thank you for your explanation. I want you to know my appreciation for your
contribution.
Back to the topic; about your last statement, I'm quite confused.
For example, if the target of the double @@ feature is the static linker,
why they talk about it in a post where the topic is DSO?
Why this double @@ is mainteined inside the DSOs?
Reflecting on my scenario I see that the dynamic linker finds quite unusual
situation where the symbol is required and there's no dependency on the DSO
containing it, and no indication (.gnu.version_r) about how to handle it.
My assumption was that the double @@ would handle these situation, but it
is reasonable that this situation to have another handling.
For this reason my current understanding is that the dynamic linker would
use default the double @@ indicated function, where the .gnu.version_r is
present but not populated, and use the older symbol where
the .gnu.version_r is missing. This way it would handle correctly the
situation where old executables do not have the section because it didn't
exist yet.
I will check the dynamic linker source code to confute or confirm this
hypothesis.

Thanks
Alessandro

Re: Fwd: MIPSEL GLIBC sem_init() not shared

[Florian Weimer]

* Alessandro Carminati via Libc-help:

> Reflecting on my scenario I see that the dynamic linker finds quite unusual
> situation where the symbol is required and there's no dependency on the DSO
> containing it, and no indication (.gnu.version_r) about how to handle it.
> My assumption was that the double @@ would handle these situation, but it
> is reasonable that this situation to have another handling.
> For this reason my current understanding is that the dynamic linker would
> use default the double @@ indicated function, where the .gnu.version_r is
> present but not populated, and use the older symbol where
> the .gnu.version_r is missing. This way it would handle correctly the
> situation where old executables do not have the section because it didn't
> exist yet.

We could have done this for glibc itself perhaps, but it would break in
the case of another shared object adding symbol versions after a release
without symbols.

Thanks,
Florian