Re: [PATCH] NUMA spinlock [BZ #23962]

[Torvald Riegel <triegel@redhat.com>]

On Sat, 2019-01-05 at 07:34 -0500, Carlos O'Donell wrote:
> On 1/3/19 2:58 PM, H.J. Lu wrote:
> > +libpthread {
> > +  GLIBC_2.29 {
> > +    numa_spinlock_alloc;
> > +    numa_spinlock_free;
> > +    numa_spinlock_init;
> > +    numa_spinlock_apply;
> > +  }
> > +}
> 
> Why are we adding these non-standard interfaces to glibc?

I also think that there shouldn't be a new API added for these.  There
would be the option of adding a new pthread_mutex_t type, and thus even get
some more space for the implementation, but I wouldn't like that either.

What I think we should be doing instead:

(1) Finally add proper spinning and back-off throughout our synchronization
abstractions (spinlocks, mutexes, etc.; see my other comment on this
thread).  This should improve performance significantly, either through
less contention during spinning, or through doing more spinning and thus
less trips through futexes and the kernel.

(2) Change the synchronization abstractions (especially mutexes) so that
they can efficiently run different code paths when they are of non-process-
shared type, and then start with doing things like MCS in the non-process-
shared mutexes.

For both of these, add proper benchmarks so that tuning decisions can be
checked automatically and tested for regressions.  IOW, we want tests for
whether the tuning decisions are still the correct ones (in the future, or
on future HW).

The main reason why I think that's a better approach is that the biggest
return on the investments by the glibc community is improving performance
for as many users as possible, even if it may not be ideal performance for
those unchanged programs.  Adding a special API+implementation instead
might enable somewhat larger performance improvements, but it requires
programs to change and programmers to be aware of it, so will likely remain
a niche case in practice.

> It could be implemented as a distinct library, allowed to evolve quickly
> in response to customer need, and eventually integrated into glibc if the
> API proves stable. A similar model has been setup by Boost and C++ just to
> draw some parallels.

I think first of all, we should try hard to get as much as performance out
of the interfaces, POSIX semantics constraints, and ABI constraints we have
today.  Then maybe change the ABI if it really unlocks further benefits.

If that's not sufficient to get decent performance, then I think the next
venue to look at is C++.  That is, if implementing these ideas to the full
extent is not possible in glibc, go to libstdc++ instead and see what's
possible there.  C++'s synchronization constructs have saner semantics than
POSIX's, and ABI breaks in the future are more likely for C++ than for
glibc.
If the current C++ synchronization constructs have semantics that inhibit
performance, then ISO C++ Study Group 1 will likely want to hear about it. 
And they are a much better group to discuss this than the glibc community
is, simply because they are focused on just parallelism and concurrency.