Bulding syscall wrapper: how glibc determines if assembly or macro wrapper will generated?

Bulding syscall wrapper: how glibc determines if assembly or macro wrapper will generated?

[Dmitrowski, Lukasz]

Hello,

I create a documentation describing how glibc builds syscall wrappers. I am currently limited to Linux x86_64 architecture only and base on https://sourceware.org/glibc/wiki/SyscallWrappers + source code analysis.

I want to to know how glibc build system decides what type of syscall wrapper will be generated, assembly wrapper or macro wrapper. I have troubles trying to figure out how the glibc build system works on this level.

1. make-syscall.sh produces sysd-syscalls, that contains build formulas for assembly wrappers.
How sysd-syscalls is used by the build system after it is generated?

2. I found out that if sysd-syscalls contains a build formula for assembly wrapper then this formula is always used when building glibc, but in some cases sysd-syscalls also shows that a *.c wrapper exists, even that it contains assembly wrapper formula. This is because make-syscall.sh searches multiple syscalls.list files and multiple related sysdirs for *.c wrappers, so sometimes *.c wrapper exists in such relation and sometimes not. Both are included into sysd-syscalls.

Example:
#### DIRECTORY = sysdeps/unix/sysv/linux/x86_64
#### SYSDIRS = sysdeps/unix/sysv/linux/x86_64/64
...
#### CALL=socket NUMBER=41 ARGS=i:iii SOURCE=-
ifeq (,$(filter socket,$(unix-syscalls)))
unix-syscalls += socket
$(foreach p,$(sysd-rules-targets),$(foreach o,$(object-suffixes),$(objpfx)$(patsubst %,$p,socket)$o)): \
                $(..)sysdeps/unix/make-syscalls.sh
        $(make-target-directory)
        (echo '#define SYSCALL_NAME socket'; \
         echo '#define SYSCALL_NARGS 3'; \
         echo '#define SYSCALL_SYMBOL __socket'; \
         echo '#define SYSCALL_CANCELLABLE 0'; \
         echo '#define SYSCALL_NOERRNO 0'; \
         echo '#define SYSCALL_ERRVAL 0'; \
         echo '#include <syscall-template.S>'; \
         echo 'weak_alias (__socket, socket)'; \
         echo 'hidden_weak (socket)'; \
        ) | $(compile-syscall) $(foreach p,$(patsubst %socket,%,$(basename $(@F))),$($(p)CPPFLAGS))
endif

...

#### DIRECTORY = sysdeps/unix
#### SYSDIRS = sysdeps/unix/sysv/linux/x86_64/64 sysdeps/unix/sysv/linux/x86_64 sysdeps/unix/sysv/linux/x86 sysdeps/x86/nptl sysdeps/unix/sysv/linux/wordsize-64 sysdeps/x86_64/nptl sysdeps/unix/sysv/linux sysdeps/nptl sysdeps/pthread sysdeps/gnu sysdeps/unix/inet sysdeps/unix/sysv sysdeps/unix/x86_64
...
#### CALL=socket NUMBER=41 ARGS=i:iii SOURCE=sysdeps/unix/sysv/linux/socket.c

--
Where is the exact point in the build system that decides if assembly wrapper (from template-syscall.S) or macro wrapper (from *.c file) will be generated?

3. If syscall wrapper is generated from *.c file, where can I find how the build formula is prepared by the build system?

4. How glibc determines a set of directories where make-syscall.sh looks for syscalls.list and *.c syscall wrappers? ($thisdir and $sysdirs variables in make-syscalls.sh)

Will be greateful for help or reference to some docs. Thank you in advance!

Re: Bulding syscall wrapper: how glibc determines if assembly or macro wrapper will generated?

[Adhemerval Zanella]

On 19/07/2021 06:26, Dmitrowski, Lukasz wrote:
> Hello,
> 
> I create a documentation describing how glibc builds syscall wrappers. I am currently limited to Linux x86_64 architecture only and base on https://sourceware.org/glibc/wiki/SyscallWrappers + source code analysis.
> 
> I want to to know how glibc build system decides what type of syscall wrapper will be generated, assembly wrapper or macro wrapper. I have troubles trying to figure out how the glibc build system works on this level.
> 
> 1. make-syscall.sh produces sysd-syscalls, that contains build formulas for assembly wrappers.
> How sysd-syscalls is used by the build system after it is generated?

By including the generated file:

sysdeps/unix/Makefile

 20 # Sysdep dirs unix/... can contain a file syscalls.list,
 21 # which specifies objects to be compiled as simple Unix system calls.
 22 
 23 -include $(common-objpfx)sysd-syscalls

The '-' on the include is a directive for gnumake [1] to make it not fail
if the file does not exists (since it will be regenerated).

> 
> 2. I found out that if sysd-syscalls contains a build formula for assembly wrapper then this formula is always used when building glibc, but in some cases sysd-syscalls also shows that a *.c wrapper exists, even that it contains assembly wrapper formula. This is because make-syscall.sh searches multiple syscalls.list files and multiple related sysdirs for *.c wrappers, so sometimes *.c wrapper exists in such relation and sometimes not. Both are included into sysd-syscalls.

The only C file that is used on the auto-generation scheme is the
stub-syscalls.c one:

sysdeps/unix/Makefile

 35 sysdep_routines += stub-syscalls
 36 $(objpfx)stub-syscalls.c: $(common-objpfx)sysd-syscalls \
 37                           $(..)sysdeps/unix/Makefile


It contains a set of symbols that are not implemented neither by glibc nor
by kernel but should exist due ABI compatibility.  For instance, on x86_64
it is bdflush@GLIBC_2.0, but it may vary depending of the ABI.

The file is just an size optimization, since multiple symbols will be
aliases to only one implementation, _no_syscall, that returns -1/ENOSYS.


> 
> Example:
> #### DIRECTORY = sysdeps/unix/sysv/linux/x86_64
> #### SYSDIRS = sysdeps/unix/sysv/linux/x86_64/64
> ...
> #### CALL=socket NUMBER=41 ARGS=i:iii SOURCE=-
> ifeq (,$(filter socket,$(unix-syscalls)))
> unix-syscalls += socket
> $(foreach p,$(sysd-rules-targets),$(foreach o,$(object-suffixes),$(objpfx)$(patsubst %,$p,socket)$o)): \
>                 $(..)sysdeps/unix/make-syscalls.sh
>         $(make-target-directory)
>         (echo '#define SYSCALL_NAME socket'; \
>          echo '#define SYSCALL_NARGS 3'; \
>          echo '#define SYSCALL_SYMBOL __socket'; \
>          echo '#define SYSCALL_CANCELLABLE 0'; \
>          echo '#define SYSCALL_NOERRNO 0'; \
>          echo '#define SYSCALL_ERRVAL 0'; \
>          echo '#include <syscall-template.S>'; \
>          echo 'weak_alias (__socket, socket)'; \
>          echo 'hidden_weak (socket)'; \
>         ) | $(compile-syscall) $(foreach p,$(patsubst %socket,%,$(basename $(@F))),$($(p)CPPFLAGS))
> endif
> 
> ...
> 
> #### DIRECTORY = sysdeps/unix
> #### SYSDIRS = sysdeps/unix/sysv/linux/x86_64/64 sysdeps/unix/sysv/linux/x86_64 sysdeps/unix/sysv/linux/x86 sysdeps/x86/nptl sysdeps/unix/sysv/linux/wordsize-64 sysdeps/x86_64/nptl sysdeps/unix/sysv/linux sysdeps/nptl sysdeps/pthread sysdeps/gnu sysdeps/unix/inet sysdeps/unix/sysv sysdeps/unix/x86_64
> ...
> #### CALL=socket NUMBER=41 ARGS=i:iii SOURCE=sysdeps/unix/sysv/linux/socket.c
> 
> --
> Where is the exact point in the build system that decides if assembly wrapper (from template-syscall.S) or macro wrapper (from *.c file) will be generated?

If I recall correctly, this is due the operator precedence on the 
o-iterator-doit on Makefules:

 393 define o-iterator-doit
 394 $(objpfx)%$o: %.S $(before-compile); $$(compile-command.S)
 395 endef
 396 object-suffixes-left := $(all-object-suffixes)
 397 include $(o-iterator)
 398 
 399 define o-iterator-doit
 400 $(objpfx)%$o: %.c $(before-compile); $$(compile-command.c)
 401 endef
 402 object-suffixes-left := $(all-object-suffixes)
 403 include $(o-iterator)
 404 
 405 define o-iterator-doit
 406 $(objpfx)%$o: %.cc $(before-compile); $$(compile-command.cc)
 407 endef
 408 object-suffixes-left := $(all-object-suffixes)
 409 include $(o-iterator)

The idea is an .S file will have precedence over the .c and .cc file on
sysdeps folder usage.

> 
> 3. If syscall wrapper is generated from *.c file, where can I find how the build formula is prepared by the build system?

Currently there is no .c file besides stub-syscalls.c.  All other
implementation are done through the assembly auto-generation, done on
the fly by the build.

> 
> 4. How glibc determines a set of directories where make-syscall.sh looks for syscalls.list and *.c syscall wrappers? ($thisdir and $sysdirs variables in make-syscalls.sh)

This is obtained from the sysdep_dirs dirs, created by configure:

sysdeps/unix/Makefile

 92 ifndef avoid-generated
 93 $(common-objpfx)sysd-syscalls: $(..)sysdeps/unix/make-syscalls.sh \
 94                                $(wildcard $(+sysdep_dirs:%=%/syscalls.list)) \
 95                                $(wildcard $(+sysdep_dirs:%=%/arch-syscall.h)) \
 96                                $(common-objpfx)libc-modules.stmp
 97         for dir in $(+sysdep_dirs); do \
 98           test -f $$dir/syscalls.list && \
 99           { sysdirs='$(sysdirs)' \
100             asm_CPP='$(COMPILE.S) -E -x assembler-with-cpp' \
101             $(SHELL) $(dir $<)$(notdir $<) $$dir || exit 1; }; \
102           test $$dir = $(..)sysdeps/unix && break; \
103         done > $@T
104         mv -f $@T $@
105 endif

Makeconfig

  58 # Complete path to sysdep dirs.
  59 # `configure' writes a definition of `config-sysdirs' in `config.make'.
  60 sysdirs := $(foreach D,$(config-sysdirs),$(firstword $(filter /%,$D) $(..)$D))
  61
  62 +sysdep_dirs = $(sysdirs)
  63 ifdef objdir
  64 +sysdep_dirs := $(objdir) $(+sysdep_dirs)
  65 endif

For instance on x86_64:

$ grep config-sysdirs config.make 
config-sysdirs =  sysdeps/unix/sysv/linux/x86_64/64 sysdeps/unix/sysv/linux/x86_64 sysdeps/unix/sysv/linux/x86 sysdeps/x86/nptl sysdeps/unix/sysv/linux/wordsize-64 sysdeps/x86_64/nptl sysdeps/unix/sysv/linux sysdeps/np
tl sysdeps/pthread sysdeps/gnu sysdeps/unix/inet sysdeps/unix/sysv sysdeps/unix/x86_64 sysdeps/unix sysdeps/posix sysdeps/x86_64/64 sysdeps/x86_64/fpu/multiarch sysdeps/x86_64/fpu sysdeps/x86/fpu sysdeps/x86_64/multiar
ch sysdeps/x86_64 sysdeps/x86 sysdeps/ieee754/float128 sysdeps/ieee754/ldbl-96 sysdeps/ieee754/dbl-64 sysdeps/ieee754/flt-32 sysdeps/wordsize-64 sysdeps/ieee754 sysdeps/generic


> 
> Will be greateful for help or reference to some docs. Thank you in advance!

Unfortunately we don't have all of this proper documented.  As a side note,
I also have a project to get rid of the assembly routines and have all 
syscalls wrapper being done by C implementations instead. 

> 

[1] https://www.gnu.org/software/make/manual/html_node/Include.html