Principles for syscall wrappers, again

Principles for syscall wrappers, again

[Joseph Myers]

We've had various past discussions of when to add wrappers for Linux 
kernel syscalls that have failed to reach a conclusion.  The result is 
a de facto status of "syscall wrappers present for almost all syscalls 
added up to Linux 3.2 / glibc 2.15 but for nothing added since then", 
which certainly doesn't make sense.

Existing bugs asking for such wrappers include: 6399 (gettid) 9712 (futex) 
17252 (getrandom) 17662 (renameat2) 18271 (bpf).

Sample messages from past discussions include: 
<https://sourceware.org/ml/libc-alpha/2012-05/msg01733.html> 
<https://sourceware.org/ml/libc-alpha/2012-06/msg00048.html> 
<https://sourceware.org/ml/libc-alpha/2013-01/msg01007.html> 
<https://sourceware.org/ml/libc-alpha/2013-02/msg00030.html> 
<https://sourceware.org/ml/libc-alpha/2014-10/msg00535.html> 
<https://sourceware.org/ml/libc-alpha/2014-10/msg00541.html> 
<https://sourceware.org/ml/libc-alpha/2014-10/msg00591.html> 
<https://sourceware.org/ml/libc-alpha/2015-01/msg00315.html>.

There were objections to what I suggested in 
<https://sourceware.org/ml/libc-alpha/2014-10/msg00535.html> regarding 
defaulting to adding wrappers unless there's a reason (syscall being 
obsolete, can't meaningfully be used behind glibc's back, or a glibc 
function provides all the useful functionality without being a direct 
wrapper) not to do so.  So here is a narrower proposal:

Direct wrappers to non-architecture-specific Linux syscalls are 
appropriate if none of the previously mentioned issues apply, *and* the 
functionality provided by the syscall is naturally useful in 
non-Linux-specific applications (although those applications might use 
different interfaces to access it on other OSes), *and* it is expected 
that more than one application or library might wish to use that 
functionality directly, rather than a single library providing 
higher-level wrappers that everything else will then use, or a single 
package containing all the uses of the syscall.

That is, wrappers are appropriate when the functionality could plausibly 
form part of the OS-independent GNU API.  However, there would be no 
expectation that such a wrapper is added to the OS-independent GNU API 
(i.e., that an ENOSYS version is added for other OSes) at the time it's 
added to glibc, nor is there an expectation that an attempt is made to 
develop an OS-independent abstraction.  If someone decides such an 
interface is useful to implement on another OS, at that point we can make 
it part of the OS-independent GNU API (with an ENOSYS version for all 
other OSes added at that point).

For example: under these principles it's fine that we have direct wrappers 
for the Linux epoll interfaces.  If the kFreeBSD port gets added it would 
be fine for that to provide direct access to kqueue.  There would be no 
expectation that either interface needs to become part of the 
OS-independent GNU API and be emulated in terms of the other on the other 
OS (if that were even possible).  If a new interface for the same use 
cases were added to POSIX in future, it would become part of the 
OS-independent GNU API and get implemented in terms of whatever underlying 
interfaces are available on each OS.

Under these principles, at least the following existing syscall wrappers 
would probably not be added now (or at least these principles would not by 
themselves provide reason to add them): bdflush create_module 
delete_module get_kernel_syms init_module query_module uselib.  However, I 
think most wrappers added up to Linux 3.2 / glibc 2.15 are appropriate 
under these principles (albeit generally lacking documentation and 
testcases).

* bdflush create_module get_kernel_syms query_module uselib: obsolete.  
Despite this, glibc provides them as ABIs even on architectures that have 
never had those syscalls (in which case they just give ENOSYS errors).  
They should become compat symbols (requiring a new syscalls.list feature 
to create a compat symbol with a SHLIB_COMPAT conditional so that new 
architectures don't get the symbol at all).

* delete_module init_module: inherently Linux-specific and only expected 
to be used by a single package.  A few other existing wrappers might be 
similarly suspect (e.g. klogctl nfsservctl).

Under these principles, I think the first four above-mentioned requested 
syscalls (futex gettid getrandom renameat2) would be appropriate for 
wrappers, but I don't know about bpf.  Of other post-3.2 syscalls, I think 
the following would be appropriate for wrappers: sched_setattr 
sched_getattr memfd_create; finit_module wouldn't be and I don't know 
about kcmp and seccomp.  execveat would be appropriate for a wrapper *and* 
making part of the OS-independent GNU API immediately.

All of that would be conditional on someone with a copyright assignment 
implementing such patches that conform to the usual standards for patches 
adding new APIs (header, documentation, tests at least that uses of the 
APIs compile and link, symbol versions, updating all ABI baselines).  
Patches may or may not already have been posted in some cases.  
Documentation for some interfaces might defer to external Linux 
documentation, but at least a statement to that effect would need to be in 
the glibc manual alongside the function prototype.  The documentation 
would also need to state the return value / errno setting, since errno 
setting is a userspace thing not something done by the kernel, and it 
would be necessary in each case to consider whether the syscall should be 
a cancellation point.

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: Principles for syscall wrappers, again

[Rich Felker]

On Mon, May 18, 2015 at 10:15:59PM +0000, Joseph Myers wrote:
> We've had various past discussions of when to add wrappers for Linux 
> kernel syscalls that have failed to reach a conclusion.  The result is 
> a de facto status of "syscall wrappers present for almost all syscalls 
> added up to Linux 3.2 / glibc 2.15 but for nothing added since then", 
> which certainly doesn't make sense.
> 
> Existing bugs asking for such wrappers include: 6399 (gettid) 9712 (futex) 
> 17252 (getrandom) 17662 (renameat2) 18271 (bpf).
> 
> Sample messages from past discussions include: 
> <https://sourceware.org/ml/libc-alpha/2012-05/msg01733.html> 
> <https://sourceware.org/ml/libc-alpha/2012-06/msg00048.html> 
> <https://sourceware.org/ml/libc-alpha/2013-01/msg01007.html> 
> <https://sourceware.org/ml/libc-alpha/2013-02/msg00030.html> 
> <https://sourceware.org/ml/libc-alpha/2014-10/msg00535.html> 
> <https://sourceware.org/ml/libc-alpha/2014-10/msg00541.html> 
> <https://sourceware.org/ml/libc-alpha/2014-10/msg00591.html> 
> <https://sourceware.org/ml/libc-alpha/2015-01/msg00315.html>.
> 
> There were objections to what I suggested in 
> <https://sourceware.org/ml/libc-alpha/2014-10/msg00535.html> regarding 
> defaulting to adding wrappers unless there's a reason (syscall being 
> obsolete, can't meaningfully be used behind glibc's back, or a glibc 
> function provides all the useful functionality without being a direct 
> wrapper) not to do so.  So here is a narrower proposal:
> 
> Direct wrappers to non-architecture-specific Linux syscalls are 
> appropriate if none of the previously mentioned issues apply, *and* the 
> functionality provided by the syscall is naturally useful in 
> non-Linux-specific applications (although those applications might use 
> different interfaces to access it on other OSes), *and* it is expected 
> that more than one application or library might wish to use that 
> functionality directly, rather than a single library providing 
> higher-level wrappers that everything else will then use, or a single 
> package containing all the uses of the syscall.
> 
> That is, wrappers are appropriate when the functionality could plausibly 
> form part of the OS-independent GNU API.  However, there would be no 
> expectation that such a wrapper is added to the OS-independent GNU API 
> (i.e., that an ENOSYS version is added for other OSes) at the time it's 
> added to glibc, nor is there an expectation that an attempt is made to 
> develop an OS-independent abstraction.  If someone decides such an 
> interface is useful to implement on another OS, at that point we can make 
> it part of the OS-independent GNU API (with an ENOSYS version for all 
> other OSes added at that point).
> 
> For example: under these principles it's fine that we have direct wrappers 
> for the Linux epoll interfaces.  If the kFreeBSD port gets added it would 
> be fine for that to provide direct access to kqueue.  There would be no 
> expectation that either interface needs to become part of the 
> OS-independent GNU API and be emulated in terms of the other on the other 
> OS (if that were even possible).  If a new interface for the same use 
> cases were added to POSIX in future, it would become part of the 
> OS-independent GNU API and get implemented in terms of whatever underlying 
> interfaces are available on each OS.
> 
> Under these principles, at least the following existing syscall wrappers 
> would probably not be added now (or at least these principles would not by 
> themselves provide reason to add them): bdflush create_module 
> delete_module get_kernel_syms init_module query_module uselib.  However, I 
> think most wrappers added up to Linux 3.2 / glibc 2.15 are appropriate 
> under these principles (albeit generally lacking documentation and 
> testcases).
> 
> * bdflush create_module get_kernel_syms query_module uselib: obsolete.  
> Despite this, glibc provides them as ABIs even on architectures that have 
> never had those syscalls (in which case they just give ENOSYS errors).  
> They should become compat symbols (requiring a new syscalls.list feature 
> to create a compat symbol with a SHLIB_COMPAT conditional so that new 
> architectures don't get the symbol at all).
> 
> * delete_module init_module: inherently Linux-specific and only expected 
> to be used by a single package.  A few other existing wrappers might be 
> similarly suspect (e.g. klogctl nfsservctl).
> 
> Under these principles, I think the first four above-mentioned requested 
> syscalls (futex gettid getrandom renameat2) would be appropriate for 
> wrappers, but I don't know about bpf.  Of other post-3.2 syscalls, I think 
> the following would be appropriate for wrappers: sched_setattr 
> sched_getattr memfd_create; finit_module wouldn't be and I don't know 
> about kcmp and seccomp.  execveat would be appropriate for a wrapper *and* 
> making part of the OS-independent GNU API immediately.

The big practical questions left are where to expose the interfaces,
and with what signatures.

I would like to see futex in its own header, sys/futex.h, since it has
a number of macros. I also tend to think the function itself should be
variadic since a few of the argument slots have different types
depending on the command in use.

For gettid, I would not mind having it in unistd.h under _ALL_SOURCE.
I don't think it warrants its own header. sched.h might also be
appropriate. Even sys/futex.h might be ok as a stretch. Its return
type should probably be pid_t or int (not sure which). For use of tids
in the futex API, a tid is necessarily an int with the sign bit and
highest two value bits clear, so int might be appropriate to match
futex.

For renameat2 I think it's obvious that it belongs in the same place
as renameat, stdio.h.

Not sure about getrandom. There's also an open question of whether
getrandom should be a pure syscall wrapper or have emulation for
kernels too old to provide it.

Rich

Re: Principles for syscall wrappers, again

[Joseph Myers]

On Mon, 18 May 2015, Rich Felker wrote:

> The big practical questions left are where to expose the interfaces,
> and with what signatures.

All these questions can be resolved in the discussions of the individual 
interfaces if and when anyone works on patches to add wrappers for them.  
I'm simply trying to get an answer for "when should such wrappers be added 
at all?" other than "only for interfaces already in Linux 3.2 / glibc 
2.15", along with "what does a complete patch adding such a wrapper look 
like?", so there is a common basis for reviewing all such patches.

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: Principles for syscall wrappers, again

[Rich Felker]

On Tue, May 19, 2015 at 01:17:49AM +0000, Joseph Myers wrote:
> On Mon, 18 May 2015, Rich Felker wrote:
> 
> > The big practical questions left are where to expose the interfaces,
> > and with what signatures.
> 
> All these questions can be resolved in the discussions of the individual 
> interfaces if and when anyone works on patches to add wrappers for them.  
> I'm simply trying to get an answer for "when should such wrappers be added 
> at all?" other than "only for interfaces already in Linux 3.2 / glibc 
> 2.15", along with "what does a complete patch adding such a wrapper look 
> like?", so there is a common basis for reviewing all such patches.

Indeed. I didn't mean to derail your (very good) question, just to
offer some thoughts to try to get the process started.

Rich

Re: Principles for syscall wrappers, again

[Joseph Myers]

Does anyone have any comments on these proposed principles 
<https://sourceware.org/ml/libc-alpha/2015-05/msg00351.html>?  Especially 
those who objected to my previous suggestion of a more general default to 
adding such wrappers.

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: Principles for syscall wrappers, again

[Roland McGrath]

I will try to follow up today.

Re: Principles for syscall wrappers, again

[Adhemerval Zanella]

On 22-05-2015 11:30, Joseph Myers wrote:
> Does anyone have any comments on these proposed principles 
> <https://sourceware.org/ml/libc-alpha/2015-05/msg00351.html>?  Especially 
> those who objected to my previous suggestion of a more general default to 
> adding such wrappers.
> 

I think your points are reasonable and I see no impose arguments
about not use it the guideline for future wrappers.  How should
we document this principles for future references: a documentation
file in source code or just wiki entry in sourceware.org?

Also, for arch-specific syscalls, should we apply to same principles
or may the maintainer be able to discuss more specific points (for
instance, the usage is limited to a set of programs/libraries)?

Re: Principles for syscall wrappers, again

[Joseph Myers]

On Fri, 22 May 2015, Adhemerval Zanella wrote:

> I think your points are reasonable and I see no impose arguments
> about not use it the guideline for future wrappers.  How should
> we document this principles for future references: a documentation
> file in source code or just wiki entry in sourceware.org?

My assumption is that if we get consensus then it would be documented on 
the wiki.

> Also, for arch-specific syscalls, should we apply to same principles
> or may the maintainer be able to discuss more specific points (for
> instance, the usage is limited to a set of programs/libraries)?

I excluded arch-specific syscalls from my proposed principles since (a) 
Mike's objection in 
<https://sourceware.org/ml/libc-alpha/2014-10/msg00591.html> said 
"especially for arch-specific syscalls" and (b) I don't think 
arch-specific syscalls are the main problem with glibc's syscall API 
having been frozen since Linux 3.2 / glibc 2.15.

My proposed principles are primarily principles for what to add rather 
than not what to add.  If you wish to add something not covered by those 
principles, you can always seek consensus that the given syscall should be 
added despite not meeting the criteria I suggest (or indeed propose 
supplementary criteria, such that wrappers for new syscalls should be 
added if they meet either set of criteria).

What I hope with my principles is that for *most* cases, the discussion 
doesn't need to be about whether to add a binding at all, but rather about 
such things as what header the declaration goes in and all the ordinary 
things checked in any patch review for a new API.  That is, it's a 
shortcut to consensus for "should we add this API at all?", to avoid the 
same issues being rehashed every time, much like other shortcuts to 
consensus we have (e.g. that certain patches can be committed without 
prior review).

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: Principles for syscall wrappers, again

[Roland McGrath]

I think I've been the most strident skeptic about adding Linuxisms to
libc, so if we manage to get me to stop complaining then we are
probably in a pretty good place.

To me there is only one category where things are clear.  That's when
there is consensus that a new function is worthwhile to have as part
of the OS-independent GNU API.  In that case we aren't talking about
adding a wrapper for a Linux syscall.  We're talking about adding a
new GNU API, and it might happen to be that the sysdeps implementation
of that for Linux is nothing more than a syscall wrapper.

My top concern is adding cruft to the core libc ABIs.  That means
specifically symbols in the shared objects for libc, libpthread,
librt, libdl, libm, and libutil.

I propose that we rule out adding any symbols to the core libc ABIs
that are not entering the OS-independent GNU API.

My second concern is adding cruft to the core libc APIs.  That means
what you can see in header files that are installed in all OS
configurations, and what you can link to without special options
beyond -lc, -lpthread (or -pthread), -lrt, -ldl, -lm, or -lutil.  I'm
more concerned about the header files than the link-time availability.

I propose that we rule out adding any declarations to core libc API
header files that are not entering the OS-independent GNU API.

If we follow those rules, then I'm not very concerned about the other
details.  I'll elaborate.

We could provide OS-specific ABIs in an OS-specific shared library,
e.g. libinux-syscalls.so.N.  This library's SONAME could change
without any changes to core libc ABIs.  (We would change the SONAME
when we want to completely drop some obsolete syscalls.)  If this
library contains nothing but syscall wrappers or equivalently trivial
code (importantly, stateless code that doesn't need any data objects
of permanent extent), then it won't be a practical problem to have
multiple versions of the library loaded in the same process at the
same time--so all the usual issues that make changing SONAMEs very
hard don't really apply.

My preference would be that we not put such OS-specific ABIs into the
common link-time API either.  That is, programs would be required to
link explicitly with -linux-syscalls.  But I will not stand in the way
of a contrary consensus, which presumably would be implemented by
adding 'AS_NEEDED ( libinux-syscalls.so.N )' into the libc.so linker
script.

In this context, the principles Joseph described seem fine enough to
me.  They might even be more conservative than we need to be.


Thanks,
Roland

Re: Principles for syscall wrappers, again

[Andreas Schwab]

Roland McGrath <roland@hack.frob.com> writes:

> We could provide OS-specific ABIs in an OS-specific shared library,
> e.g. libinux-syscalls.so.N.

Wouldn't it be small enough to make it static-only?  I think the
OS-specific syscalls are unlikely be used in many places.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Principles for syscall wrappers, again

[Carlos O'Donell]

On 05/23/2015 04:20 AM, Andreas Schwab wrote:
> Roland McGrath <roland@hack.frob.com> writes:
> 
>> We could provide OS-specific ABIs in an OS-specific shared library,
>> e.g. libinux-syscalls.so.N.
> 
> Wouldn't it be small enough to make it static-only?  I think the
> OS-specific syscalls are unlikely be used in many places.

What about the security implications of this? Say if we find a bug
in the argument handling on the glibc side? Or cancellation on the
glibc side?

For the record I like Roland's propose of a libinux-syscalls.so.N,
but would want it to be AS_NEEDED, since to be useful it must be
transparent to the average GNU/Linux user.

Cheers,
Carlos.

Re: Principles for syscall wrappers, again

[Torvald Riegel]

On Mon, 2015-05-18 at 20:09 -0400, Rich Felker wrote:
> I would like to see futex in its own header, sys/futex.h, since it has
> a number of macros. I also tend to think the function itself should be
> variadic since a few of the argument slots have different types
> depending on the command in use.

Roland has argued that we should be adding GNU API extensions, not just
Linuxisms.  I think futex is a good example of that: I'd prefer us to
expose functionality that is trimmed down to what's currently widely
used (e.g., futex_wake(), futex_wait(), ... vs. a single variadic-arg
futex()).  That makes it easier for things like the Native Client to
support it, and we can expose a refined interface to users (the futex
syscalls has seen quite a few changes over time and not all of the
initial functionality proved to be really useful).

Thus, it seems to me we should explicitly discuss the GNU API we want to
expose for each syscall's functionality that we want to offer, and not
just expose syscalls interfaces as-is by default.

Re: Principles for syscall wrappers, again

[Rich Felker]

On Tue, May 26, 2015 at 10:55:25AM +0200, Torvald Riegel wrote:
> On Mon, 2015-05-18 at 20:09 -0400, Rich Felker wrote:
> > I would like to see futex in its own header, sys/futex.h, since it has
> > a number of macros. I also tend to think the function itself should be
> > variadic since a few of the argument slots have different types
> > depending on the command in use.
> 
> Roland has argued that we should be adding GNU API extensions, not just
> Linuxisms.  I think futex is a good example of that: I'd prefer us to
> expose functionality that is trimmed down to what's currently widely
> used (e.g., futex_wake(), futex_wait(), ... vs. a single variadic-arg
> futex()).  That makes it easier for things like the Native Client to
> support it, and we can expose a refined interface to users (the futex
> syscalls has seen quite a few changes over time and not all of the
> initial functionality proved to be really useful).
> 
> Thus, it seems to me we should explicitly discuss the GNU API we want to
> expose for each syscall's functionality that we want to offer, and not
> just expose syscalls interfaces as-is by default.

I think this is gratuitous NIH'ing and a disservice to applications.
Code which wants to use the futex API is already doing so via
syscall() with the existing API, and is most easily updated to use
futex(). Your proposed API is not any more general or easier to
provide on NaCl or elsewhere; the existing API can easily be provided
because it's equivalent.

Rich

Re: Principles for syscall wrappers, again

[Andreas Schwab]

Rich Felker <dalias@libc.org> writes:

> I think this is gratuitous NIH'ing and a disservice to applications.
> Code which wants to use the futex API is already doing so via
> syscall() with the existing API, and is most easily updated to use
> futex().

futex is much like socketcall: a multiplexer between more or less
loosely related functions.  See nptl/DESIGN*.txt for how the API really
should look like.

Andreas.

-- 
Andreas Schwab, SUSE Labs, schwab@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

Re: Principles for syscall wrappers, again

[Rich Felker]

On Tue, May 26, 2015 at 05:48:41PM +0200, Andreas Schwab wrote:
> Rich Felker <dalias@libc.org> writes:
> 
> > I think this is gratuitous NIH'ing and a disservice to applications.
> > Code which wants to use the futex API is already doing so via
> > syscall() with the existing API, and is most easily updated to use
> > futex().
> 
> futex is much like socketcall: a multiplexer between more or less
> loosely related functions.  See nptl/DESIGN*.txt for how the API really
> should look like.

That may be how it's used internally in glibc, but not elsewhere. And
that API lacks important things like the private/shared distinction
and features beyond basic wait/wake..

Rich

Re: Principles for syscall wrappers, again

[Torvald Riegel]

On Tue, 2015-05-26 at 11:10 -0400, Rich Felker wrote:
> On Tue, May 26, 2015 at 10:55:25AM +0200, Torvald Riegel wrote:
> > On Mon, 2015-05-18 at 20:09 -0400, Rich Felker wrote:
> > > I would like to see futex in its own header, sys/futex.h, since it has
> > > a number of macros. I also tend to think the function itself should be
> > > variadic since a few of the argument slots have different types
> > > depending on the command in use.
> > 
> > Roland has argued that we should be adding GNU API extensions, not just
> > Linuxisms.  I think futex is a good example of that: I'd prefer us to
> > expose functionality that is trimmed down to what's currently widely
> > used (e.g., futex_wake(), futex_wait(), ... vs. a single variadic-arg
> > futex()).  That makes it easier for things like the Native Client to
> > support it, and we can expose a refined interface to users (the futex
> > syscalls has seen quite a few changes over time and not all of the
> > initial functionality proved to be really useful).
> > 
> > Thus, it seems to me we should explicitly discuss the GNU API we want to
> > expose for each syscall's functionality that we want to offer, and not
> > just expose syscalls interfaces as-is by default.
> 
> I think this is gratuitous NIH'ing and a disservice to applications.

I can't see how offering a subset of the functionality provided by the
syscall is NIH'ing.  We wouldn't offer additional stuff for obvious
reasons.

Do you really want to provide FUTEX_FD or FUTEX_REQUEUE?

> Code which wants to use the futex API is already doing so via
> syscall() with the existing API, and is most easily updated to use
> futex().

Nothing is breaking that, or is it?

Anyway, as Joseph requested, the specifics of the futex case need to be
discussed elsewhere.  I just wanted to point out that we should make a
conscious decision about the (GNU) API that we offer and not just copy
stuff without further consideration.

Re: Principles for syscall wrappers, again

[Rich Felker]

On Tue, May 26, 2015 at 06:57:11PM +0200, Torvald Riegel wrote:
> On Tue, 2015-05-26 at 11:10 -0400, Rich Felker wrote:
> > On Tue, May 26, 2015 at 10:55:25AM +0200, Torvald Riegel wrote:
> > > On Mon, 2015-05-18 at 20:09 -0400, Rich Felker wrote:
> > > > I would like to see futex in its own header, sys/futex.h, since it has
> > > > a number of macros. I also tend to think the function itself should be
> > > > variadic since a few of the argument slots have different types
> > > > depending on the command in use.
> > > 
> > > Roland has argued that we should be adding GNU API extensions, not just
> > > Linuxisms.  I think futex is a good example of that: I'd prefer us to
> > > expose functionality that is trimmed down to what's currently widely
> > > used (e.g., futex_wake(), futex_wait(), ... vs. a single variadic-arg
> > > futex()).  That makes it easier for things like the Native Client to
> > > support it, and we can expose a refined interface to users (the futex
> > > syscalls has seen quite a few changes over time and not all of the
> > > initial functionality proved to be really useful).
> > > 
> > > Thus, it seems to me we should explicitly discuss the GNU API we want to
> > > expose for each syscall's functionality that we want to offer, and not
> > > just expose syscalls interfaces as-is by default.
> > 
> > I think this is gratuitous NIH'ing and a disservice to applications.
> 
> I can't see how offering a subset of the functionality provided by the
> syscall is NIH'ing.  We wouldn't offer additional stuff for obvious
> reasons.
> 
> Do you really want to provide FUTEX_FD or FUTEX_REQUEUE?

Linux does not provide FUTEX_FD, so no. I don't see why you wouldn't
want to support FUTEX_REQUEUE. The remarks about it having inherent
race conditions are wrong; they only apply to glibc's use of it. It's
perfectly valid to requeue a futex waiter to a lock as long as the
thread doing the requeue currently holds said lock, and in fact my
cond var implementation uses this (the lock being requeued to is an
internal lock, not the mutex).

> > Code which wants to use the futex API is already doing so via
> > syscall() with the existing API, and is most easily updated to use
> > futex().
> 
> Nothing is breaking that, or is it?

No, but the NIH'd API would preclude a trivial mechanical change to
use the syscall wrappers.

> Anyway, as Joseph requested, the specifics of the futex case need to be
> discussed elsewhere.  I just wanted to point out that we should make a
> conscious decision about the (GNU) API that we offer and not just copy
> stuff without further consideration.

Plenty of stuff has been copied without further consideration even
recently -- the worst example is sendmmsg/recvmmsg which use the wrong
types. What I would like to see more than making minor but annoying
API differences in glibc vs Linux is some serious additional scrutiny
over the kernel folks when they add new interfaces without thinking
about problems they're going to make in userspace, so this stops
happening.

Rich

Re: Principles for syscall wrappers, again

[Andreas Schwab]

Rich Felker <dalias@libc.org> writes:

> That may be how it's used internally in glibc, but not elsewhere. 

Of course, since the API doesn't exist yet.

> And that API lacks important things like the private/shared
> distinction and features beyond basic wait/wake..

Use your imagination.

Andreas.

-- 
Andreas Schwab, SUSE Labs, schwab@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

Re: Principles for syscall wrappers, again

[Rich Felker]

On Wed, May 27, 2015 at 10:12:49AM +0200, Andreas Schwab wrote:
> Rich Felker <dalias@libc.org> writes:
> 
> > That may be how it's used internally in glibc, but not elsewhere. 
> 
> Of course, since the API doesn't exist yet.
> 
> > And that API lacks important things like the private/shared
> > distinction and features beyond basic wait/wake..
> 
> Use your imagination.

Sorry, I guess I wasn't clear; my point was just that inventing a new
API has lots of requirements that might not be apparent at first, and
that could easily be overlooked, whereas using the existing API is
guaranteed not to be a regression.

Rich

Re: Principles for syscall wrappers, again

[Andreas Schwab]

Rich Felker <dalias@libc.org> writes:

> Sorry, I guess I wasn't clear; my point was just that inventing a new
> API has lots of requirements that might not be apparent at first, and
> that could easily be overlooked, whereas using the existing API is
> guaranteed not to be a regression.

Stopping development is guaranteed to not causing regressions.

Andreas.

-- 
Andreas Schwab, SUSE Labs, schwab@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

Re: Principles for syscall wrappers, again

[Joseph Myers]

On Fri, 22 May 2015, Roland McGrath wrote:

> I think I've been the most strident skeptic about adding Linuxisms to
> libc, so if we manage to get me to stop complaining then we are
> probably in a pretty good place.
> 
> To me there is only one category where things are clear.  That's when
> there is consensus that a new function is worthwhile to have as part
> of the OS-independent GNU API.  In that case we aren't talking about
> adding a wrapper for a Linux syscall.  We're talking about adding a
> new GNU API, and it might happen to be that the sysdeps implementation
> of that for Linux is nothing more than a syscall wrapper.

I think it's appropriate for the additions of APIs taken from Linux 
syscalls to be roughly consistent with historical practice for when APIs 
go in glibc.

To me, that suggests that if we don't add such APIs as Linux-specific 
APIs, we should instead treat Linux like BSD and SysV were historically 
treated as sources of APIs, as described in "Berkeley Unix, SVID, POSIX, 
Standards and Portability" in intro.texi.  So if a Linux syscall API seems 
useful for non-Linux-specific applications, the starting point would be 
that it is appropriate for the GNU API - and while we'd need to consider 
issues such as the header for the declaration, the userspace types to use, 
whether it is a cancellation point, and whether it sets errno or uses some 
other error return convention, we should not generally get into 
second-guessing the API design and reworking the whole design of an API 
which has already passed the review process for Linux kernel/userspace 
APIs on linux-api.  (That review process is a lot more thorough than it 
used to be; there have been past cases before where glibc's version of an 
API added a flags argument where the original syscall lacked one, and I 
think it less likely now that a new syscall would be added without such 
extensibility.)

It would then be possible for such APIs to move from a syscalls.list entry 
to a more complicated fallback providing compatibility for older kernels, 
if desired in a particular case, and possible for other OS ports of glibc 
to add implementations if they desire (otherwise the ENOSYS version would 
be used - just as several functions from other OSes exist in glibc with 
the default ENOSYS implementations used under the Linux kernel).

Thus, I'd suggest that if we don't want new Linux-specific APIs in libc, 
we should set standards that would consider all of futex gettid getrandom 
renameat2 sched_setattr sched_getattr memfd_create execveat appropriate 
for the OS-independent GNU API (and I don't know about bpf kcmp seccomp).

> My second concern is adding cruft to the core libc APIs.  That means
> what you can see in header files that are installed in all OS
> configurations, and what you can link to without special options
> beyond -lc, -lpthread (or -pthread), -lrt, -ldl, -lm, or -lutil.  I'm
> more concerned about the header files than the link-time availability.
> 
> I propose that we rule out adding any declarations to core libc API
> header files that are not entering the OS-independent GNU API.

It's already the case that Linux-specific APIs are declared in sys/ 
headers that are only installed from sysdeps/unix/sysv/linux/Makefile (or 
in a few cases, in Linux-specific versions of bits/ headers included from 
more generic headers).

> We could provide OS-specific ABIs in an OS-specific shared library,
> e.g. libinux-syscalls.so.N.  This library's SONAME could change
> without any changes to core libc ABIs.  (We would change the SONAME
> when we want to completely drop some obsolete syscalls.)  If this
> library contains nothing but syscall wrappers or equivalently trivial
> code (importantly, stateless code that doesn't need any data objects
> of permanent extent), then it won't be a practical problem to have
> multiple versions of the library loaded in the same process at the
> same time--so all the usual issues that make changing SONAMEs very
> hard don't really apply.

I'd be wary of assuming that (for example) a syscall wrapper with 
cancellation from an old glibc version will work with libc.so from a new 
glibc version that handles cancellation differently, so if you changed 
SONAME that would suggest building all the versions with different SONAMEs 
as part of the same glibc build (which seems like an undue complication).  
So changing the SONAME of such a library seems dangerous to me.

I'd think of such a library as only being for syscalls that fail my 
suggested criteria for being too Linux-specific (or maybe for being 
architecture-specific) - not for anything that could plausibly be useful 
on another OS, or where a fallback implementation for older kernels might 
make sense.

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: Principles for syscall wrappers, again

[Joseph Myers]

On Tue, 26 May 2015, Rich Felker wrote:

> > Thus, it seems to me we should explicitly discuss the GNU API we want to
> > expose for each syscall's functionality that we want to offer, and not
> > just expose syscalls interfaces as-is by default.
> 
> I think this is gratuitous NIH'ing and a disservice to applications.
> Code which wants to use the futex API is already doing so via
> syscall() with the existing API, and is most easily updated to use
> futex(). Your proposed API is not any more general or easier to
> provide on NaCl or elsewhere; the existing API can easily be provided
> because it's equivalent.

Agreed.  As I said in 
<https://sourceware.org/ml/libc-alpha/2015-05/msg00764.html>, I think the 
API details to discuss are things such as the userspace types and the 
header with the declaration, not any more substantial rearrangement of the 
API - treating Linux as an API source like BSD and SysV, except for API 
details outside the scope of what the kernel defines.

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: Principles for syscall wrappers, again

[Torvald Riegel]

On Thu, 2015-05-28 at 16:51 +0000, Joseph Myers wrote:
> On Tue, 26 May 2015, Rich Felker wrote:
> 
> > > Thus, it seems to me we should explicitly discuss the GNU API we want to
> > > expose for each syscall's functionality that we want to offer, and not
> > > just expose syscalls interfaces as-is by default.
> > 
> > I think this is gratuitous NIH'ing and a disservice to applications.
> > Code which wants to use the futex API is already doing so via
> > syscall() with the existing API, and is most easily updated to use
> > futex(). Your proposed API is not any more general or easier to
> > provide on NaCl or elsewhere; the existing API can easily be provided
> > because it's equivalent.
> 
> Agreed.  As I said in 
> <https://sourceware.org/ml/libc-alpha/2015-05/msg00764.html>, I think the 
> API details to discuss are things such as the userspace types and the 
> header with the declaration, not any more substantial rearrangement of the 
> API - treating Linux as an API source like BSD and SysV, except for API 
> details outside the scope of what the kernel defines.

Joseph, the fact that you comment on futex specifically tells me you're
okay with discussing futex in detail in this thread (which I tried to
avoid so far).

I suppose both you and Rich are thinking about this as the futex() API
you want to expose:

int futex(int *uaddr, int op, int val, const struct timespec *timeout,
                 int *uaddr2, int val3);


syscall() is multiplexing.  And so is futex().  Thus, by your logic,
there's no need for offering futex() because syscall() is doing it
already, just with *one part* of the multiplexing being variable.

There is no API-design reason for the futex syscall to do multiplexing
itself.  Look at the manpages -- the documentation clearly speaks about
logically separate operations that are exposed through this single
syscall.  We do not need this multiplexing at all, or do we need to
minimize the number of functions for some significant reason?  Why not
just have a file() function then that has as first param an enum
signaling whether one actually wants to read or write from a file?

Thus, the specific futex() signature is an artifact of having to
multiplex.  We don't have to multiplex, so this isn't a reason to keep
it.

Keeping the multiplexing is bad for users.  Can you tell me off-hand
what goes in "uaddr2", "val", or "val3" for all the ops?  Is it easy to
remember based on the function signature?
Can you remember in which cases "timeout" is actually "val2" and not a
pointer but cast to uint32_t?  So are we going to expect users to cast
uint32_t's to a pointer to call one of the operations and consider that
a useful API design?  It's a nice way to potentially trigger compiler
warnings though.


In contrast, consider an API such as this (just for exposure, we should
discuss the details such as whether we want to break split out the
flags, or split out waiting with timeouts):

int futex_wake(int* futex_word, int flags, int max_wakeups);
int futex_wait(int* futex_word, int flags, int expected_value,
  const struct timespec *timeout);
int futex_cmp_requeue(int* futex_word, int flags, int expected_value,
  int max_wakeups, int max_requeue, int* requeue_futex_word);
...

This makes it easy for users to use the right parameters at the right
positions at call sites.  Things like code completion will just show the
function signatures, or you look it up, and you're fine.  Parameters can
be used in a type-safe way too, so no odd complaints by your compiler.

And this is *not* an API redesign or anything like that.  If you think
this is NIH, go look at the draft of the futex man-pages:
http://git.kernel.org/cgit/docs/man-pages/man-pages.git/tree/?h=draft_futex
We are just exposing the logical API documented there.

Terms such as "futex word" come from the kernel's documentation.  We do
pick a position for the flags parameter, for example, but this isn't
covered by the kernel interface because it wants to multiplex.  But the
rest of the parameters are all just taken from the kernel interface
unmodified.  What's the point of working on clear futex documentation if
we then unnecessarily obfuscate it through exposing a multiplexing
function?

Thus, not exposing a clean form of the logical API documented by the
kernel and instead exposing a multiplexed form for no good reason would
be the disservice to users.

Rich, whether a transition from syscall() to futex() would be easier is
irrelevant because doing that doesn't buy you anything significant in
terms of easy of use, type-safe code, or similar.  We need to think
about what is best for users that want to use futexes, period -- not
what is best for users that already do so using syscall().

If we expose futex() and thus think that this is the interface people
should be using, I suppose you all are fine with using futex() as
glibc-internal interface too, right?  If not, surely neither will be the
case for external users.  We should not expose interfaces that force
users to use their own wrappers to make sense of it, especially not if
we ourselves are already aware of the need for such wrappers.


Thus, I do think exposing futex() with the multiplexing in place would
be a *poor* API to expose to users, and a clear mistake.  I do not know
whether other syscall interfaces show similar issues, but the futex case
shows to me that clearly we need to consider and discuss what APIs we
actually expose.

Re: Principles for syscall wrappers, again

[Joseph Myers]

On Fri, 29 May 2015, Torvald Riegel wrote:

> syscall() is multiplexing.  And so is futex().  Thus, by your logic,
> there's no need for offering futex() because syscall() is doing it
> already, just with *one part* of the multiplexing being variable.

The syscall function doesn't deal with cancellation.

My proposed principles are non-exclusive.  I consider the futex call to be 
useful in the same way as open or ptrace or ioctl - a function providing 
access to a range of related operations, some system-independent and some 
with system-specific aspects.  This does not exclude the possibility of 
other APIs being useful for particular cases of the futex call, or indeed 
of applying my proposed principles recursively to cases of particular 
multiplexing system calls.

A futex function is useful to provide access to new futex operations 
without needing libc changes for them, much like open and ptrace provide 
access to new operations.  Sometimes we might need to change the futex 
implementation in libc to adjust cancellation handling for new operations, 
but that's no different from the adjustment of open regarding O_TMPFILE 
(bug 17523).  Providing access to new operations - at the most specific 
level at which this can be done, rather than forcing everything through 
syscall - is useful even if occasionally they do need libc changes, as 
with O_TMPFILE.

I think it would have been harmful to users to insist that all unknown 
ioctl uses go through the syscall function unless we've individually 
reviewed and second-guessed the interface in every case - it would just 
gratuitously complicate their code.  And I think the same applies to 
futex.  The fact that, with hindsight, we might not have designed an API 
the way it was in fact designed does not mean we should embed that 
viewpoint in the choice of APIs provided to users.  This is a C library 
and in C we should provide a range of interfaces at different levels, some 
easier to use and some closer to the underlying OS interfaces.

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: Principles for syscall wrappers, again

[Torvald Riegel]

On Fri, 2015-05-29 at 11:19 +0000, Joseph Myers wrote:
> On Fri, 29 May 2015, Torvald Riegel wrote:
> 
> > syscall() is multiplexing.  And so is futex().  Thus, by your logic,
> > there's no need for offering futex() because syscall() is doing it
> > already, just with *one part* of the multiplexing being variable.
> 
> The syscall function doesn't deal with cancellation.

Agreed.  But not all futex operations are cancellation points either.

And what if we had syscall() and a variant of that that supports
cancellation?  Then the cancellation argument you make is taken care of,
or am I missing something?

> My proposed principles are non-exclusive.  I consider the futex call to be 
> useful in the same way as open or ptrace or ioctl - a function providing 
> access to a range of related operations, some system-independent and some 
> with system-specific aspects.  This does not exclude the possibility of 
> other APIs being useful for particular cases of the futex call, or indeed 
> of applying my proposed principles recursively to cases of particular 
> multiplexing system calls.
> 
> A futex function is useful to provide access to new futex operations 
> without needing libc changes for them, much like open and ptrace provide 
> access to new operations.

I would not be opposed to *additionally* offering a futex() operation
that can be used as a fallback should we ever get new futex operations.
However, the existence of an exposed futex() fallback should never be a
reason to not provide a proper futex interface.

> I think it would have been harmful to users to insist that all unknown 
> ioctl uses go through the syscall function unless we've individually 
> reviewed and second-guessed the interface in every case - it would just 
> gratuitously complicate their code.  And I think the same applies to 
> futex.

That may be a proper analogy if we'd be looking at adding a futex API
while the kernel-level set of futex operations still evolved.  But right
now, we're looking at a pretty stable set of operations, and we have
documentation from the kernel that actually describes the intended
logical API (which is then transformed into the multiplexing scheme
used).

Thus, if people start with the futex docs to understand how the futex
operations work -- and they should do that --, then what complicates a
futex interface would be the multiplexing interface itself (e.g., the
wild parameter reuse), not the alternative interface I outlined.

Also, it's not like it should take us long to decide on a proper futex
interface, nor would that matter much given that we haven't had any
futex interface for a long time.  Or are you thinking about other kinds
of complications?

And I still don't really see the harm for users.  Can you be more
specific?  Are you concerned about availability, or timely availability,
or deviation to the lowest-level OS interface, or usability, or ...?

> The fact that, with hindsight, we might not have designed an API 
> the way it was in fact designed does not mean we should embed that 
> viewpoint in the choice of APIs provided to users.

If we have the hindsight, why should we ignore it?  It's not even just
our hindsight in this case, we're following the kernel docs mostly.

Re: Principles for syscall wrappers, again

[Joseph Myers]

On Fri, 29 May 2015, Torvald Riegel wrote:

> On Fri, 2015-05-29 at 11:19 +0000, Joseph Myers wrote:
> > On Fri, 29 May 2015, Torvald Riegel wrote:
> > 
> > > syscall() is multiplexing.  And so is futex().  Thus, by your logic,
> > > there's no need for offering futex() because syscall() is doing it
> > > already, just with *one part* of the multiplexing being variable.
> > 
> > The syscall function doesn't deal with cancellation.
> 
> Agreed.  But not all futex operations are cancellation points either.
> 
> And what if we had syscall() and a variant of that that supports
> cancellation?  Then the cancellation argument you make is taken care of,
> or am I missing something?

I think we should have such a variant - but again, that's non-exclusive of 
also providing futex bindings.

> I would not be opposed to *additionally* offering a futex() operation
> that can be used as a fallback should we ever get new futex operations.
> However, the existence of an exposed futex() fallback should never be a
> reason to not provide a proper futex interface.

I don't claim it is.  I don't object to someone proposing a patch adding 
multiple functions with different futex interfaces - I simply don't 
consider my principles to have anything to say on such a patch.

> > I think it would have been harmful to users to insist that all unknown 
> > ioctl uses go through the syscall function unless we've individually 
> > reviewed and second-guessed the interface in every case - it would just 
> > gratuitously complicate their code.  And I think the same applies to 
> > futex.
> 
> That may be a proper analogy if we'd be looking at adding a futex API
> while the kernel-level set of futex operations still evolved.  But right
> now, we're looking at a pretty stable set of operations, and we have
> documentation from the kernel that actually describes the intended
> logical API (which is then transformed into the multiplexing scheme
> used).
> 
> Thus, if people start with the futex docs to understand how the futex
> operations work -- and they should do that --, then what complicates a
> futex interface would be the multiplexing interface itself (e.g., the
> wild parameter reuse), not the alternative interface I outlined.

I'd expect people to look at "man futex" on their distribution, and to 
expect a C function with the interface described there.  I looked at the 
branch you referenced 
<http://git.kernel.org/cgit/docs/man-pages/man-pages.git/tree/man2/futex.2?h=draft_futex> 
and I still don't see descriptions of the different C interfaces you 
describe.

Now, if the kernel stops providing the futex syscall in the asm-generic 
API for new architectures, instead providing separate syscalls, I'd 
consider that a strong indication that the interface *is* obsolete (and so 
excluded by my principles).  Or even if it states somewhere that new 
operations will use new syscalls and no new operations will be added to 
the existing syscall.

> Also, it's not like it should take us long to decide on a proper futex
> interface, nor would that matter much given that we haven't had any
> futex interface for a long time.  Or are you thinking about other kinds
> of complications?
> 
> And I still don't really see the harm for users.  Can you be more
> specific?  Are you concerned about availability, or timely availability,
> or deviation to the lowest-level OS interface, or usability, or ...?

All of these.  I think glibc will be more useful in practice if we are 
more willing to delegate certain areas of low-level API design and 
implement APIs designed in the Linux kernel community like we do with 
POSIX APIs.  I think this discussion has illustrated my point: discussions 
here of the detailed design of such APIs get into ratholes, consuming 
limited resources (look at the size of the patch review backlog in 
patchwork), and so we should have principles that as far as possible 
delegate the design to external communities and respect their choices, 
reducing the set of choices needing review here.  Only in the cases where 
there isn't a suitable external API do we then need to do the detailed 
design ourselves.

> > The fact that, with hindsight, we might not have designed an API 
> > the way it was in fact designed does not mean we should embed that 
> > viewpoint in the choice of APIs provided to users.
> 
> If we have the hindsight, why should we ignore it?  It's not even just
> our hindsight in this case, we're following the kernel docs mostly.

Because whatever the issues with the kernel API, at least it's an API 
people know and can see in their futex manpages.

If you wish to define a more precise rule for determining the kernel's 
intended API - for determining the set of userspace C functions intended 
to correspond to a C syscall - that's fine, as long as it's generally 
clear how to apply that rule.  What I don't want is detailed redesign of 
APIs in the absence of strong evidence of kernel intent.

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: Principles for syscall wrappers, again

[Rich Felker]

On Fri, May 29, 2015 at 10:37:33AM +0200, Torvald Riegel wrote:
> On Thu, 2015-05-28 at 16:51 +0000, Joseph Myers wrote:
> > On Tue, 26 May 2015, Rich Felker wrote:
> > 
> > > > Thus, it seems to me we should explicitly discuss the GNU API we want to
> > > > expose for each syscall's functionality that we want to offer, and not
> > > > just expose syscalls interfaces as-is by default.
> > > 
> > > I think this is gratuitous NIH'ing and a disservice to applications.
> > > Code which wants to use the futex API is already doing so via
> > > syscall() with the existing API, and is most easily updated to use
> > > futex(). Your proposed API is not any more general or easier to
> > > provide on NaCl or elsewhere; the existing API can easily be provided
> > > because it's equivalent.
> > 
> > Agreed.  As I said in 
> > <https://sourceware.org/ml/libc-alpha/2015-05/msg00764.html>, I think the 
> > API details to discuss are things such as the userspace types and the 
> > header with the declaration, not any more substantial rearrangement of the 
> > API - treating Linux as an API source like BSD and SysV, except for API 
> > details outside the scope of what the kernel defines.
> 
> Joseph, the fact that you comment on futex specifically tells me you're
> okay with discussing futex in detail in this thread (which I tried to
> avoid so far).
> 
> I suppose both you and Rich are thinking about this as the futex() API
> you want to expose:
> 
> int futex(int *uaddr, int op, int val, const struct timespec *timeout,
>                  int *uaddr2, int val3);

I would make it variadic with all but the first 3 optional; the types
for the rest are not consistent across commands.

> syscall() is multiplexing.  And so is futex().  Thus, by your logic,
> there's no need for offering futex() because syscall() is doing it
> already, just with *one part* of the multiplexing being variable.

No, the need to add futex is because it should be a cancellation
point, and syscall is not a cancellation point. If you want to write
custom sync primitives that interact with cancellation, futex must be
a cancellation point.

I'm not going to reply to the rest line-by-line, but I don't see how
futex() command multiplexing is any worse than fcntl(). Yes I agree
that, if this were being done fresh from day 1, a non-multiplexed API
is cleaner. That's not what my argument is about. What I believe here
is that:

1. As long as it doesn't have fundamental flaws(*), adopting an API
   with existing usage is better than trying to invent a new one to do
   the same thing.

2. There's a nontrivial risk of omitting access to some features in
   the proposed new non-multiplexed API, and then there's no way to
   get access to a cancellation-point futex command that has those
   features.

(*) By "fundamental flaws", I mean things that make the API unusable
for some purposes and that can't be worked around by wrapping, like
lack of thread-safety. If the existing API can be used to implement
your new API as wrappers, then the existing API does not have the kind
of "fundamental flaws" I'm talking about.

Rich

Re: Principles for syscall wrappers, again

[Rich Felker]

On Fri, May 29, 2015 at 01:58:17PM +0200, Torvald Riegel wrote:
> On Fri, 2015-05-29 at 11:19 +0000, Joseph Myers wrote:
> > On Fri, 29 May 2015, Torvald Riegel wrote:
> > 
> > > syscall() is multiplexing.  And so is futex().  Thus, by your logic,
> > > there's no need for offering futex() because syscall() is doing it
> > > already, just with *one part* of the multiplexing being variable.
> > 
> > The syscall function doesn't deal with cancellation.
> 
> Agreed.  But not all futex operations are cancellation points either.
> 
> And what if we had syscall() and a variant of that that supports
> cancellation?  Then the cancellation argument you make is taken care of,
> or am I missing something?

I think that would solve the technical problem, yes.

> > My proposed principles are non-exclusive.  I consider the futex call to be 
> > useful in the same way as open or ptrace or ioctl - a function providing 
> > access to a range of related operations, some system-independent and some 
> > with system-specific aspects.  This does not exclude the possibility of 
> > other APIs being useful for particular cases of the futex call, or indeed 
> > of applying my proposed principles recursively to cases of particular 
> > multiplexing system calls.
> > 
> > A futex function is useful to provide access to new futex operations 
> > without needing libc changes for them, much like open and ptrace provide 
> > access to new operations.
> 
> I would not be opposed to *additionally* offering a futex() operation
> that can be used as a fallback should we ever get new futex operations.
> However, the existence of an exposed futex() fallback should never be a
> reason to not provide a proper futex interface.

I would fully support having both a full-featured futex() multiplexing
function and individual clean-API wrappers for common operations. I
would even be fine with only the latter being part of the
"cross-platform GNU API" and the former being Linux-only, though of
course I would prefer both be treated as first-class APIs.

> > I think it would have been harmful to users to insist that all unknown 
> > ioctl uses go through the syscall function unless we've individually 
> > reviewed and second-guessed the interface in every case - it would just 
> > gratuitously complicate their code.  And I think the same applies to 
> > futex.
> 
> That may be a proper analogy if we'd be looking at adding a futex API
> while the kernel-level set of futex operations still evolved.  But right
> now, we're looking at a pretty stable set of operations, and we have
> documentation from the kernel that actually describes the intended
> logical API (which is then transformed into the multiplexing scheme
> used).

Look at it this way: if the kernel adds a new futex command, how long
will it take to get glibc support for it with multiplexer vs
individual functions? And to get that support deployed into mainstream
distros? :-)

Rich

Re: Principles for syscall wrappers, again

[Torvald Riegel]

On Fri, 2015-05-29 at 13:21 +0000, Joseph Myers wrote:
> On Fri, 29 May 2015, Torvald Riegel wrote:
> 
> > On Fri, 2015-05-29 at 11:19 +0000, Joseph Myers wrote:
> > > I think it would have been harmful to users to insist that all unknown 
> > > ioctl uses go through the syscall function unless we've individually 
> > > reviewed and second-guessed the interface in every case - it would just 
> > > gratuitously complicate their code.  And I think the same applies to 
> > > futex.
> > 
> > That may be a proper analogy if we'd be looking at adding a futex API
> > while the kernel-level set of futex operations still evolved.  But right
> > now, we're looking at a pretty stable set of operations, and we have
> > documentation from the kernel that actually describes the intended
> > logical API (which is then transformed into the multiplexing scheme
> > used).
> > 
> > Thus, if people start with the futex docs to understand how the futex
> > operations work -- and they should do that --, then what complicates a
> > futex interface would be the multiplexing interface itself (e.g., the
> > wild parameter reuse), not the alternative interface I outlined.
> 
> I'd expect people to look at "man futex" on their distribution, and to 
> expect a C function with the interface described there.  I looked at the 
> branch you referenced 
> <http://git.kernel.org/cgit/docs/man-pages/man-pages.git/tree/man2/futex.2?h=draft_futex> 
> and I still don't see descriptions of the different C interfaces you 
> describe.

Well, the C interface they show is the multiplexing one -- that's why I
spoke about the "logical", intended API.  Look into the actual
descriptions of the individual operations.  Those do read like separate
functions to, and that would also what would make sense.  For example
futex_wait:
   This operation tests that the value at the futex word pointed to
   by  the address uaddr still contains the expected value val, and
   if so, then sleeps awaiting FUTEX_WAKE on the futex  word. [...]

Don't you think it is easier to remember how to use it if we have this?:
  int futex_wait(int* futex_word, int flags, int expected_value,
    const struct timespec *timeout);

Note that the rest of the manpage uses "futex word" consistently to name
the int variable that holds the userspace state for the futex, for
example.

> > Also, it's not like it should take us long to decide on a proper futex
> > interface, nor would that matter much given that we haven't had any
> > futex interface for a long time.  Or are you thinking about other kinds
> > of complications?
> > 
> > And I still don't really see the harm for users.  Can you be more
> > specific?  Are you concerned about availability, or timely availability,
> > or deviation to the lowest-level OS interface, or usability, or ...?
> 
> All of these.

Really?  Usability includes ease-of-use to me, and having to put an int
into a timespec* parameter that has a completely unintuitive name for
that use really does conflict with ease-of-use to me.  Why do you think
that's not the case?

> I think glibc will be more useful in practice if we are 
> more willing to delegate certain areas of low-level API design and 
> implement APIs designed in the Linux kernel community like we do with 
> POSIX APIs.

I doubt the POSIX API and the Linux syscall APIs have the same target
users.  If we had a futex API that would be at a level similar to POSIX,
that may be fine.  And I also agree that we might want to expose the
lowest-level API we can do (ie, futex() if we ignore syscall()) -- but
that doesn't mean that the lowest-level API is what would actually be
useful to users.

futexes are most useful for people that want to implement their own
blocking synchronization, and for that to be useful in the GNU system
(which you seem to be concerned about), we need to have an API that
makes sense in this context and is well-integrated with how we do
atomics on the GNU system.  That's why I have proposed patches for the
futex kernel docs that clarify what is atomic wrt. what etc.

> I think this discussion has illustrated my point: discussions 
> here of the detailed design of such APIs get into ratholes

Please note that we're not even discussing here how a proper futex API
should look, so we don't even know if we'd disagree about it.  Thus, I
disagree with your claim that we're in *that* rathole.

> consuming 
> limited resources (look at the size of the patch review backlog in 
> patchwork), and so we should have principles that as far as possible 
> delegate the design to external communities and respect their choices, 
> reducing the set of choices needing review here.  Only in the cases where 
> there isn't a suitable external API do we then need to do the detailed 
> design ourselves.

I'm confused.  You said you don't see adding futex() as a reason to not
add a proper futex interface, yet here you seem to say that the latter
would be bad for us.  Can you clarify?

> > > The fact that, with hindsight, we might not have designed an API 
> > > the way it was in fact designed does not mean we should embed that 
> > > viewpoint in the choice of APIs provided to users.
> > 
> > If we have the hindsight, why should we ignore it?  It's not even just
> > our hindsight in this case, we're following the kernel docs mostly.
> 
> Because whatever the issues with the kernel API, at least it's an API 
> people know and can see in their futex manpages.

But that manpage doesn't discuss cancellation, or does it?  So it's not
complete anyway from our POV.

I have worked on that futex manpage draft; based on that experience, I
bet that the manpage would be easier to consume and the interface easier
to use if we wouldn't multiplex in the interface.

Also, the futex API is, I'd argue, a collaborative kernel/glibc API.
For quite a while, the most useful docs on it where a combination of
Ulrich's paper and the futex syscall manpage.  Some of the error
conditions (e.g., spurious wake-ups) are a direct result of how glibc is
using futexes (and how similar synchronization code in other libraries
would do).

> If you wish to define a more precise rule for determining the kernel's 
> intended API - for determining the set of userspace C functions intended 
> to correspond to a C syscall - that's fine, as long as it's generally 
> clear how to apply that rule.  What I don't want is detailed redesign of 
> APIs in the absence of strong evidence of kernel intent.

As I said, I'm speaking about futexes specifically here, and claim that
a different API than the multiplexing one is significantly easier and
less error-prone to use.  I'm also claiming that a different API would
be easier to follow even with just the current Linux manpage.

In the general case, I just want us to (1) consider whether, for a
particular syscall, we are in a similar situation as for futexes; (2)
discuss the situation and see whether there is consensus on offering an
API that is more useful to the majority of glibc users or GNU system
developers; and (3) try to provide such an API in the same release in
which we might provide a lowest-level API as well.

(I'm adding (3) because I don't want us to provide futex() suport in
release N, followed by users creating their own wrappers for it if they
want cleaner interfaces -- and then we provide a proper interface in
release N+1 and everyone has to change again or live with their own
variants.)

Re: Principles for syscall wrappers, again

[Torvald Riegel]

On Fri, 2015-05-29 at 10:07 -0400, Rich Felker wrote:
> On Fri, May 29, 2015 at 01:58:17PM +0200, Torvald Riegel wrote:
> > On Fri, 2015-05-29 at 11:19 +0000, Joseph Myers wrote:
> > > On Fri, 29 May 2015, Torvald Riegel wrote:
> > > 
> > > > syscall() is multiplexing.  And so is futex().  Thus, by your logic,
> > > > there's no need for offering futex() because syscall() is doing it
> > > > already, just with *one part* of the multiplexing being variable.
> > > 
> > > The syscall function doesn't deal with cancellation.
> > 
> > Agreed.  But not all futex operations are cancellation points either.
> > 
> > And what if we had syscall() and a variant of that that supports
> > cancellation?  Then the cancellation argument you make is taken care of,
> > or am I missing something?
> 
> I think that would solve the technical problem, yes.
> 
> > > My proposed principles are non-exclusive.  I consider the futex call to be 
> > > useful in the same way as open or ptrace or ioctl - a function providing 
> > > access to a range of related operations, some system-independent and some 
> > > with system-specific aspects.  This does not exclude the possibility of 
> > > other APIs being useful for particular cases of the futex call, or indeed 
> > > of applying my proposed principles recursively to cases of particular 
> > > multiplexing system calls.
> > > 
> > > A futex function is useful to provide access to new futex operations 
> > > without needing libc changes for them, much like open and ptrace provide 
> > > access to new operations.
> > 
> > I would not be opposed to *additionally* offering a futex() operation
> > that can be used as a fallback should we ever get new futex operations.
> > However, the existence of an exposed futex() fallback should never be a
> > reason to not provide a proper futex interface.
> 
> I would fully support having both a full-featured futex() multiplexing
> function and individual clean-API wrappers for common operations. I
> would even be fine with only the latter being part of the
> "cross-platform GNU API" and the former being Linux-only, though of
> course I would prefer both be treated as first-class APIs.

Good.

Would you want futex() that is cancellable for some ops specifically, or
would you be fine with a cancellable syscall?  Would you prefer having
futex()?

> > > I think it would have been harmful to users to insist that all unknown 
> > > ioctl uses go through the syscall function unless we've individually 
> > > reviewed and second-guessed the interface in every case - it would just 
> > > gratuitously complicate their code.  And I think the same applies to 
> > > futex.
> > 
> > That may be a proper analogy if we'd be looking at adding a futex API
> > while the kernel-level set of futex operations still evolved.  But right
> > now, we're looking at a pretty stable set of operations, and we have
> > documentation from the kernel that actually describes the intended
> > logical API (which is then transformed into the multiplexing scheme
> > used).
> 
> Look at it this way: if the kernel adds a new futex command, how long
> will it take to get glibc support for it with multiplexer vs
> individual functions? And to get that support deployed into mainstream
> distros? :-)

That depends on the distro.  If the kernel is as old as glibc, there's
no problem... :)

Re: Principles for syscall wrappers, again

[Rich Felker]

On Fri, May 29, 2015 at 04:15:11PM +0200, Torvald Riegel wrote:
> On Fri, 2015-05-29 at 10:07 -0400, Rich Felker wrote:
> > On Fri, May 29, 2015 at 01:58:17PM +0200, Torvald Riegel wrote:
> > > On Fri, 2015-05-29 at 11:19 +0000, Joseph Myers wrote:
> > > > On Fri, 29 May 2015, Torvald Riegel wrote:
> > > > 
> > > > > syscall() is multiplexing.  And so is futex().  Thus, by your logic,
> > > > > there's no need for offering futex() because syscall() is doing it
> > > > > already, just with *one part* of the multiplexing being variable.
> > > > 
> > > > The syscall function doesn't deal with cancellation.
> > > 
> > > Agreed.  But not all futex operations are cancellation points either.
> > > 
> > > And what if we had syscall() and a variant of that that supports
> > > cancellation?  Then the cancellation argument you make is taken care of,
> > > or am I missing something?
> > 
> > I think that would solve the technical problem, yes.
> > 
> > > > My proposed principles are non-exclusive.  I consider the futex call to be 
> > > > useful in the same way as open or ptrace or ioctl - a function providing 
> > > > access to a range of related operations, some system-independent and some 
> > > > with system-specific aspects.  This does not exclude the possibility of 
> > > > other APIs being useful for particular cases of the futex call, or indeed 
> > > > of applying my proposed principles recursively to cases of particular 
> > > > multiplexing system calls.
> > > > 
> > > > A futex function is useful to provide access to new futex operations 
> > > > without needing libc changes for them, much like open and ptrace provide 
> > > > access to new operations.
> > > 
> > > I would not be opposed to *additionally* offering a futex() operation
> > > that can be used as a fallback should we ever get new futex operations.
> > > However, the existence of an exposed futex() fallback should never be a
> > > reason to not provide a proper futex interface.
> > 
> > I would fully support having both a full-featured futex() multiplexing
> > function and individual clean-API wrappers for common operations. I
> > would even be fine with only the latter being part of the
> > "cross-platform GNU API" and the former being Linux-only, though of
> > course I would prefer both be treated as first-class APIs.
> 
> Good.
> 
> Would you want futex() that is cancellable for some ops specifically, or
> would you be fine with a cancellable syscall?  Would you prefer having
> futex()?

I strongly prefer having futex(). Using syscall() is conceptually a
bad idea except in arch-specific code. This is even more the case
nowadays with x32's wackiness where the published syscall() API (using
'long' for the variadic arguments) does not even work. But it's always
been the case that various syscalls have arch-specific variations that
you need to be aware of to use them safely -- things like different
argument orders, etc. And if you're ever in a situation where the libc
types differ from the kernel types, using syscall() is just wrong
there too. This would apply to futex, for example, if glibc eventually
has an option like _FILE_OFFSET_BITS but for time_t -- when time_t is
set to 64-bit, futex() would need to use the 64-bit timespec
structure. 

Even if not for that last issue (the only one that applies to futex),
though, users should not have to be aware of which syscalls are "safe"
to make with syscall() and which are not, and we shouldn't be
encouraging use of syscall() for non-arch-specific purposes.

Also, if you want to emulate futex() on a system that doesn't have the
same native API, that's not going to work for code calling
syscall(SYS_futex, ...).

Rich

Re: Principles for syscall wrappers, again

[Joseph Myers]

On Fri, 29 May 2015, Torvald Riegel wrote:

> > > And I still don't really see the harm for users.  Can you be more
> > > specific?  Are you concerned about availability, or timely availability,
> > > or deviation to the lowest-level OS interface, or usability, or ...?
> > 
> > All of these.
> 
> Really?  Usability includes ease-of-use to me, and having to put an int
> into a timespec* parameter that has a completely unintuitive name for
> that use really does conflict with ease-of-use to me.  Why do you think
> that's not the case?

Usability includes being able to follow the existing documented interface 
rather than "the glibc people decided that was NIH, so you need to apply 
these transformations and hope your use case was one they thought 
relevant".

> > I think glibc will be more useful in practice if we are 
> > more willing to delegate certain areas of low-level API design and 
> > implement APIs designed in the Linux kernel community like we do with 
> > POSIX APIs.
> 
> I doubt the POSIX API and the Linux syscall APIs have the same target
> users.  If we had a futex API that would be at a level similar to POSIX,

They don't (POSIX provides the pthreads interfaces for synchronization), 
but libc caters to a range of different target users.

> I'm confused.  You said you don't see adding futex() as a reason to not
> add a proper futex interface, yet here you seem to say that the latter
> would be bad for us.  Can you clarify?

What would be bad is letting discussion of a multiple-function interface 
obstruct consideration and acceptance of a patch adding the basic 
interface.

> In the general case, I just want us to (1) consider whether, for a
> particular syscall, we are in a similar situation as for futexes; (2)

Any such consideration should only be in the basis of compelling Linux 
kernel consensus that the existing API is deficient and a particular 
alternative API is how it would be designed now, not on the basis of 
substituting our own opinion for the review that already took place on 
linux-api when the syscall was added (I think the principles for new 
syscalls that postdate the present review process on linux-api are more 
important than the application to older syscalls).

> discuss the situation and see whether there is consensus on offering an
> API that is more useful to the majority of glibc users or GNU system
> developers; and (3) try to provide such an API in the same release in
> which we might provide a lowest-level API as well.
> 
> (I'm adding (3) because I don't want us to provide futex() suport in
> release N, followed by users creating their own wrappers for it if they
> want cleaner interfaces -- and then we provide a proper interface in
> release N+1 and everyone has to change again or live with their own
> variants.)

But actually we're closer to the situation of having provided the 
low-level syscall() interface to futex in release N with any better 
interface being derailed and delayed to N+20.

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: Principles for syscall wrappers, again

[Torvald Riegel]

On Fri, 2015-05-29 at 15:34 +0000, Joseph Myers wrote:
> On Fri, 29 May 2015, Torvald Riegel wrote:
> 
> > > > And I still don't really see the harm for users.  Can you be more
> > > > specific?  Are you concerned about availability, or timely availability,
> > > > or deviation to the lowest-level OS interface, or usability, or ...?
> > > 
> > > All of these.
> > 
> > Really?  Usability includes ease-of-use to me, and having to put an int
> > into a timespec* parameter that has a completely unintuitive name for
> > that use really does conflict with ease-of-use to me.  Why do you think
> > that's not the case?
> 
> Usability includes being able to follow the existing documented interface 
> rather than "the glibc people decided that was NIH, so you need to apply 
> these transformations and hope your use case was one they thought 
> relevant".

You can certainly disagree that an improvement I claim is not actually
an improvement, but characterizing it as NIH is inappropriate IMO.  NIH
tries to sum up a person's or group's mindset or objectives, so I'd like
to ask you to stop trying to speculate about my intentions.  If you
disagree on the claimed improvement, do that instead.

> > > I think glibc will be more useful in practice if we are 
> > > more willing to delegate certain areas of low-level API design and 
> > > implement APIs designed in the Linux kernel community like we do with 
> > > POSIX APIs.
> > 
> > I doubt the POSIX API and the Linux syscall APIs have the same target
> > users.  If we had a futex API that would be at a level similar to POSIX,
> 
> They don't (POSIX provides the pthreads interfaces for synchronization), 
> but libc caters to a range of different target users.

I was arguing that your analogy doesn't hold.

> > I'm confused.  You said you don't see adding futex() as a reason to not
> > add a proper futex interface, yet here you seem to say that the latter
> > would be bad for us.  Can you clarify?
> 
> What would be bad is letting discussion of a multiple-function interface 
> obstruct consideration and acceptance of a patch adding the basic 
> interface.

Sure, but we haven't actually found out whether there would be any
obstruction.  Rich says he wouldn't mind having both, others haven't
commented.  What's your thought on having both?  Are you opposed to
having both, or where do you think the obstruction would come from?

> > In the general case, I just want us to (1) consider whether, for a
> > particular syscall, we are in a similar situation as for futexes; (2)
> 
> Any such consideration should only be in the basis of compelling Linux 
> kernel consensus that the existing API is deficient and a particular 
> alternative API is how it would be designed now, not on the basis of 
> substituting our own opinion for the review that already took place on 
> linux-api when the syscall was added (I think the principles for new 
> syscalls that postdate the present review process on linux-api are more 
> important than the application to older syscalls).

What they exposed has different constraints than what we expose, do you
agree?  For example, number of syscalls may matter to them, number of
functions we expose doesn't really matter much.  Or differences between
syscalls and function calls.

Thus, what can be right for a kernel interface can differ from a
function-based interface, right?  Remember that we're not even
discussing a difference in functionality here, just basics like no
unnecessary multiplexing, no type-unsafe reuse of parameters, and
actually using parameter names that are meaningful.

From the kernel's perspective, there's no reason to change the futex API
now because they won't change an interface anyway unless it's obviously
dysfunctional.  But that's simply not the question nor the constraint
set that we face.  We have to provide something useful to our users, not
blindly copy design decisions made for a different situations (ie, for a
kernel interface, not for a glibc / GNU / ... interface).  Note that I'm
not saying the kernel's choice is bad, it's just in a different context.

I mean, in glibc we already do use a different interface than futex(),
and always have, right?  And IIRC, futexes have been developed at least
with a lot of input from glibc.  Isn't that enough indication that just
offering futex() wouldn't be what people actually need?

Also, in what you wrote above you express your opinion how the rules
should be -- could you also offer reasoning why this should be the case?

> > discuss the situation and see whether there is consensus on offering an
> > API that is more useful to the majority of glibc users or GNU system
> > developers; and (3) try to provide such an API in the same release in
> > which we might provide a lowest-level API as well.
> > 
> > (I'm adding (3) because I don't want us to provide futex() suport in
> > release N, followed by users creating their own wrappers for it if they
> > want cleaner interfaces -- and then we provide a proper interface in
> > release N+1 and everyone has to change again or live with their own
> > variants.)
> 
> But actually we're closer to the situation of having provided the 
> low-level syscall() interface to futex in release N with any better 
> interface being derailed and delayed to N+20.

How long we had syscall is irrelevant here, strictly speaking.  You seem
to worry about some improvement over syscall taking more time -- but the
reason for this could only mean that we two disagree whether we should
have both futex() and proper or just futex().  Based on that, it's not
clear who is actually preventing consensus here.

Re: Principles for syscall wrappers, again

[Joseph Myers]

On Fri, 29 May 2015, Torvald Riegel wrote:

> > > I'm confused.  You said you don't see adding futex() as a reason to not
> > > add a proper futex interface, yet here you seem to say that the latter
> > > would be bad for us.  Can you clarify?
> > 
> > What would be bad is letting discussion of a multiple-function interface 
> > obstruct consideration and acceptance of a patch adding the basic 
> > interface.
> 
> Sure, but we haven't actually found out whether there would be any
> obstruction.  Rich says he wouldn't mind having both, others haven't
> commented.  What's your thought on having both?  Are you opposed to
> having both, or where do you think the obstruction would come from?

I do not object to having both.  I do object to a new API being any sort 
of prerequisite for adding the known and understood API.

> > Any such consideration should only be in the basis of compelling Linux 
> > kernel consensus that the existing API is deficient and a particular 
> > alternative API is how it would be designed now, not on the basis of 
> > substituting our own opinion for the review that already took place on 
> > linux-api when the syscall was added (I think the principles for new 
> > syscalls that postdate the present review process on linux-api are more 
> > important than the application to older syscalls).
> 
> What they exposed has different constraints than what we expose, do you
> agree?  For example, number of syscalls may matter to them, number of
> functions we expose doesn't really matter much.  Or differences between
> syscalls and function calls.

I don't see any real difference in significance between number of syscalls 
and number of functions.  A syscall is essentially a function call that 
happens to cross a privilege boundary and all new syscalls come with an 
intended C API (modulo details of exactly what the libc-level typedefs 
are, etc.) - the design considerations are essentially the same.

> I mean, in glibc we already do use a different interface than futex(),
> and always have, right?  And IIRC, futexes have been developed at least
> with a lot of input from glibc.  Isn't that enough indication that just
> offering futex() wouldn't be what people actually need?

The point of providing the futex interface is to provide for any use 
someone comes up with, not just those we envisage or that are useful 
directly within glibc.

> Also, in what you wrote above you express your opinion how the rules
> should be -- could you also offer reasoning why this should be the case?

I think that having the kernel interfaces conveniently available to users, 
so they can use whatever combinations of userspace and kernel interfaces 
seem appropriate without needing to be concerned with the details of which 
parts of the functionality are implemented in userspace or with 
architecture-specific syscall argument handling, is better for glibc than 
only providing higher-level interfaces, or than providing documentation 
that presents one particular viewpoint on the deficiencies of an external 
standard and declares a feature unsupported (something common in older GNU 
documentation), and that the kernel API review process is (now) 
sufficiently good that the benefits of compatibility with the kernel and 
its documentation outweight any disadvantages of imperfections in the 
interfaces.

I apply this principle both ways - I think the kernel should do more to 
support POSIX requirements (e.g. set*id for multi-threaded processes) 
rather than declaring those requirements, or the parts that are 
problematic to implement in userspace, to be uninteresting.

I think that if the manpage for a syscall needs to document the absence of 
a corresponding glibc function, that's an embarrassment to glibc - that 
we're not keeping up with external changes (Linux is at least as 
appropriate an API source now as BSD and SysV were originally) - in the 
absence of a clear reason making the syscall inappropriate for glibc.

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: Principles for syscall wrappers, again

[Szabolcs Nagy]

On 29/05/15 19:14, Torvald Riegel wrote:
> On Fri, 2015-05-29 at 15:34 +0000, Joseph Myers wrote:
>> What would be bad is letting discussion of a multiple-function interface 
>> obstruct consideration and acceptance of a patch adding the basic 
>> interface.
> 
> Sure, but we haven't actually found out whether there would be any
> obstruction.  Rich says he wouldn't mind having both, others haven't
> commented.  What's your thought on having both?  Are you opposed to
> having both, or where do you think the obstruction would come from?

i think it would be nice if ppl could stop calling syscall directly
(eg. it can be problematic on ilp32 syscall abi).

i assume providing the kernel futex api (as a variadic function)
is the easiest way to get there.

eg. syscall users in gcc on linux:

- libstdc++ uses SYS_futex.
- libgomp uses SYS_futex and SYS_gettid (and it has inline asm to call
futex syscall directly on various archs).
- libcilkrts uses SYS_gettid.
- libitm uses SYS_futex.

> I mean, in glibc we already do use a different interface than futex(),
> and always have, right?  And IIRC, futexes have been developed at least
> with a lot of input from glibc.  Isn't that enough indication that just
> offering futex() wouldn't be what people actually need?

some projects already use syscall(SYS_futex,...).

some of them indeed wrap it as futex_wake/futex_wait but not all.

debian codesearch finds these futex users:

android-androresolvd
aqsis (includes tbb)
capnproto
chromium-browser
ghdl
groonga
haproxy
libxshmfence
linux-tools
mariadb-10.0 (includes groonga)
mumble
openimageio (includes tbb)
percona-xtrabackup
phantomjs (includes qt base)
qt4-x11 (includes qt base)
qtbase-opensource-src
rr
sbcl
simgrid
stress-ng
tbb

Re: Principles for syscall wrappers, again

[Rich Felker]

On Mon, Jun 01, 2015 at 02:42:56PM +0100, Szabolcs Nagy wrote:
> On 29/05/15 19:14, Torvald Riegel wrote:
> > On Fri, 2015-05-29 at 15:34 +0000, Joseph Myers wrote:
> >> What would be bad is letting discussion of a multiple-function interface 
> >> obstruct consideration and acceptance of a patch adding the basic 
> >> interface.
> > 
> > Sure, but we haven't actually found out whether there would be any
> > obstruction.  Rich says he wouldn't mind having both, others haven't
> > commented.  What's your thought on having both?  Are you opposed to
> > having both, or where do you think the obstruction would come from?
> 
> i think it would be nice if ppl could stop calling syscall directly
> (eg. it can be problematic on ilp32 syscall abi).
> 
> i assume providing the kernel futex api (as a variadic function)
> is the easiest way to get there.
> 
> eg. syscall users in gcc on linux:
> 
> - libstdc++ uses SYS_futex.
> - libgomp uses SYS_futex and SYS_gettid (and it has inline asm to call
> futex syscall directly on various archs).
> - libcilkrts uses SYS_gettid.
> - libitm uses SYS_futex.
> 
> > I mean, in glibc we already do use a different interface than futex(),
> > and always have, right?  And IIRC, futexes have been developed at least
> > with a lot of input from glibc.  Isn't that enough indication that just
> > offering futex() wouldn't be what people actually need?
> 
> some projects already use syscall(SYS_futex,...).
> 
> some of them indeed wrap it as futex_wake/futex_wait but not all.

Another risk of using the names futex_wake/futex_wait is that they may
clash with different wrappers by the same name that an application has
used, making refitting the code to use sys/futex.h versions more work
and more error-prone.

Still I'd rather have both approaches in sys/futex.h than neither...

> debian codesearch finds these futex users:
> 
> android-androresolvd
> aqsis (includes tbb)
> capnproto
> chromium-browser
> ghdl
> groonga
> haproxy
> libxshmfence
> linux-tools
> mariadb-10.0 (includes groonga)
> mumble
> openimageio (includes tbb)
> percona-xtrabackup
> phantomjs (includes qt base)
> qt4-x11 (includes qt base)
> qtbase-opensource-src
> rr
> sbcl
> simgrid
> stress-ng
> tbb

Thanks for doing the research here.

Rich

Re: Principles for syscall wrappers, again

[Andreas Schwab]

Szabolcs Nagy <szabolcs.nagy@arm.com> writes:

> i assume providing the kernel futex api (as a variadic function)

Variadic functions are problematic.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Principles for syscall wrappers, again

[Rich Felker]

On Tue, Jun 02, 2015 at 12:32:20AM +0200, Andreas Schwab wrote:
> Szabolcs Nagy <szabolcs.nagy@arm.com> writes:
> 
> > i assume providing the kernel futex api (as a variadic function)
> 
> Variadic functions are problematic.

I don't much like them either, but I also don't like casts to pass
val2 to FUTEX_*REQUEUE* and FUTEX_WAKE_OP. I also don't like that the
caller would have to unconditionally pass 6 arguments even for
commands (especially futex_wake) that only need 3 or 4. On lots of
targets 4 is the cutoff for arguments passed in registers, so always
passing 6 arguments will seriously pessimize code generation (stack
frame setup, register allocation, spillage, etc.) in the caller,
whereas futex is expected only to be called in an unlikely code path.

In any case this is a fairly minor issue and I don't want disagreement
over whether to make futex() variadic or not to derail it.

Rich

Re: Principles for syscall wrappers, again

[Andreas Schwab]

Congratulations, you just argued for a proper futex API.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Principles for syscall wrappers, again

[Rich Felker]

On Tue, Jun 02, 2015 at 01:28:45AM +0200, Andreas Schwab wrote:
> Congratulations, you just argued for a proper futex API.

I don't see anything wrong with that. This isn't a competition to see
who's right and wrong. In fact I think this is a good reason to have
the simple futex_wait and futex_wake functions that cover a large
portion of real-world usage cases, but not a reason to omit the
general function that follows the existing API and provides all
functionality.

Rich

Re: Principles for syscall wrappers, again

[Torvald Riegel]

On Mon, 2015-06-01 at 14:42 +0100, Szabolcs Nagy wrote:
> On 29/05/15 19:14, Torvald Riegel wrote:
> > On Fri, 2015-05-29 at 15:34 +0000, Joseph Myers wrote:
> >> What would be bad is letting discussion of a multiple-function interface 
> >> obstruct consideration and acceptance of a patch adding the basic 
> >> interface.
> > 
> > Sure, but we haven't actually found out whether there would be any
> > obstruction.  Rich says he wouldn't mind having both, others haven't
> > commented.  What's your thought on having both?  Are you opposed to
> > having both, or where do you think the obstruction would come from?
> 
> i think it would be nice if ppl could stop calling syscall directly
> (eg. it can be problematic on ilp32 syscall abi).
> 
> i assume providing the kernel futex api (as a variadic function)
> is the easiest way to get there.
> 
> eg. syscall users in gcc on linux:
> 
> - libstdc++ uses SYS_futex.

We use syscall() here, but code would use a proper futex API if
available.  If the latter would offer a ready-to-use timedwait with
absolute timeouts, code that duplicates what we do in glibc could be
removed.

> - libgomp uses SYS_futex and SYS_gettid (and it has inline asm to call
> futex syscall directly on various archs).

I believe we'd also use wrappers here.

> - libitm uses SYS_futex.

Same as for libstdc++.  Has it's own futex_wait and futex_wake wrappers
already.

> > I mean, in glibc we already do use a different interface than futex(),
> > and always have, right?  And IIRC, futexes have been developed at least
> > with a lot of input from glibc.  Isn't that enough indication that just
> > offering futex() wouldn't be what people actually need?
> 
> some projects already use syscall(SYS_futex,...).
> 
> some of them indeed wrap it as futex_wake/futex_wait but not all.

Good idea using the Debian code search.  Here's what I found for the
first few hits returned in my search ("futex filetype:c", ignoring those
packages that don't seem to actually use futexes, but just include the
syscall numbers, for example):

> mariadb-10.0 (includes groonga)
- groonga seems to have use futex_wake but futex_wait is just a sleep?

openmprtl:
-seems to build its own locks based on syscall()

qemu: creates it's own futex_wait etc. wrappers

> simgrid
- creates it's own futex_wait etc. wrappers based on syscall()

> tbb
- creates it's own futex_wait etc. wrappers

wine:
- creates it's own futex_wait and _wake wrappers

I haven't looked further, but based on that sample it seems that quite a
few projects see a need to build and use their own custom wrappers.
That indicates to me that futex() isn't quite what they were looking
for.

Re: Principles for syscall wrappers, again

[Torvald Riegel]

On Fri, 2015-05-29 at 20:16 +0000, Joseph Myers wrote:
> On Fri, 29 May 2015, Torvald Riegel wrote:
> 
> > > > I'm confused.  You said you don't see adding futex() as a reason to not
> > > > add a proper futex interface, yet here you seem to say that the latter
> > > > would be bad for us.  Can you clarify?
> > > 
> > > What would be bad is letting discussion of a multiple-function interface 
> > > obstruct consideration and acceptance of a patch adding the basic 
> > > interface.
> > 
> > Sure, but we haven't actually found out whether there would be any
> > obstruction.  Rich says he wouldn't mind having both, others haven't
> > commented.  What's your thought on having both?  Are you opposed to
> > having both, or where do you think the obstruction would come from?
> 
> I do not object to having both.  I do object to a new API being any sort 
> of prerequisite for adding the known and understood API.

I can understand that you don't want to stall an existing API for a
significant time to discuss a new API.  But can we at least agree that
we should spend some thought on whether a new improved API provides a
better service to our users before we just rush out just the existing
API?  I'm not asking for bike-shedding but more of sanity check.

> > > Any such consideration should only be in the basis of compelling Linux 
> > > kernel consensus that the existing API is deficient and a particular 
> > > alternative API is how it would be designed now, not on the basis of 
> > > substituting our own opinion for the review that already took place on 
> > > linux-api when the syscall was added (I think the principles for new 
> > > syscalls that postdate the present review process on linux-api are more 
> > > important than the application to older syscalls).
> > 
> > What they exposed has different constraints than what we expose, do you
> > agree?  For example, number of syscalls may matter to them, number of
> > functions we expose doesn't really matter much.  Or differences between
> > syscalls and function calls.
> 
> I don't see any real difference in significance between number of syscalls 
> and number of functions.  A syscall is essentially a function call that 
> happens to cross a privilege boundary and all new syscalls come with an 
> intended C API (modulo details of exactly what the libc-level typedefs 
> are, etc.) - the design considerations are essentially the same.

There's still more manual setup required for a syscall than for a normal
C function where almost all of the machinery is handled by the
toolchain.

Also, why would anyone multiplex several logically different operations,
with type-unsafe parameter multiplexing and all that, into a single C
function -- instead of just using several C functions?  I can't think of
any reason but wanting to limit the number of functions/syscalls.

> > I mean, in glibc we already do use a different interface than futex(),
> > and always have, right?  And IIRC, futexes have been developed at least
> > with a lot of input from glibc.  Isn't that enough indication that just
> > offering futex() wouldn't be what people actually need?
> 
> The point of providing the futex interface is to provide for any use 
> someone comes up with, not just those we envisage or that are useful 
> directly within glibc.

I said "*just* offering futex()", and I meant only futex().  We can
offer both futex() and the proper interface and still support the use
case you mentioned above.

> > Also, in what you wrote above you express your opinion how the rules
> > should be -- could you also offer reasoning why this should be the case?
> 
> I think that having the kernel interfaces conveniently available to users, 
> so they can use whatever combinations of userspace and kernel interfaces 
> seem appropriate without needing to be concerned with the details of which 
> parts of the functionality are implemented in userspace or with 
> architecture-specific syscall argument handling, is better for glibc than 
> only providing higher-level interfaces, or than providing documentation 
> that presents one particular viewpoint on the deficiencies of an external 
> standard and declares a feature unsupported (something common in older GNU 
> documentation), and that the kernel API review process is (now) 
> sufficiently good that the benefits of compatibility with the kernel and 
> its documentation outweight any disadvantages of imperfections in the 
> interfaces.

OK.  I notice you said "only providing higher-level" interfaces; so, do
I understand you correctly that you're primary concern here is to
provide futex(), and that you don't necessarily object to a proper
interface, nor think that offering futex() renders offering a proper
interface obsolete?

> I apply this principle both ways - I think the kernel should do more to 
> support POSIX requirements (e.g. set*id for multi-threaded processes) 
> rather than declaring those requirements, or the parts that are 
> problematic to implement in userspace, to be uninteresting.
> 
> I think that if the manpage for a syscall needs to document the absence of 
> a corresponding glibc function, that's an embarrassment to glibc - that 
> we're not keeping up with external changes (Linux is at least as 
> appropriate an API source now as BSD and SysV were originally) - in the 
> absence of a clear reason making the syscall inappropriate for glibc.

OK.

Re: Principles for syscall wrappers, again

[Joseph Myers]

On Thu, 4 Jun 2015, Torvald Riegel wrote:

> Also, why would anyone multiplex several logically different operations,
> with type-unsafe parameter multiplexing and all that, into a single C
> function -- instead of just using several C functions?  I can't think of
> any reason but wanting to limit the number of functions/syscalls.

Well, I think ioctl is better as a libc interface than providing large 
numbers of extremely specialized functions - limiting the size of the libc 
interface, while still providing access to specialized operations not 
known at the time the interface is added to libc, can make sense.  And 
then in a few cases we provide higher-level interfaces to ioctls (e.g. 
termios).

> OK.  I notice you said "only providing higher-level" interfaces; so, do
> I understand you correctly that you're primary concern here is to
> provide futex(), and that you don't necessarily object to a proper
> interface, nor think that offering futex() renders offering a proper
> interface obsolete?

Yes.

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: Principles for syscall wrappers, again

[Florian Weimer]

On 05/26/2015 10:55 AM, Torvald Riegel wrote:
> On Mon, 2015-05-18 at 20:09 -0400, Rich Felker wrote:
>> I would like to see futex in its own header, sys/futex.h, since it has
>> a number of macros. I also tend to think the function itself should be
>> variadic since a few of the argument slots have different types
>> depending on the command in use.
> 
> Roland has argued that we should be adding GNU API extensions, not just
> Linuxisms.  I think futex is a good example of that: I'd prefer us to
> expose functionality that is trimmed down to what's currently widely
> used (e.g., futex_wake(), futex_wait(), ... vs. a single variadic-arg
> futex()).  That makes it easier for things like the Native Client to
> support it, and we can expose a refined interface to users (the futex
> syscalls has seen quite a few changes over time and not all of the
> initial functionality proved to be really useful).

These non-variadic interfaces can be implemented in C.  A direct wrapper
for futex would have to be written in assembler and pass everything
directly to the kernel.  But perhaps the system call wrapper generator
could take of it.

A C implementation which tries to reconstruct the argument list would
have to know all the operations/flags, similar separate to separate
functions for each operation.  See bug 17523 for an example why this
could turn out to be relevant.

-- 
Florian Weimer / Red Hat Product Security