Re: [PATCH] Unify pthread_spin_[try]lock implementations.

[Chris Metcalf <cmetcalf@tilera.com>]

On 7/11/2012 7:22 AM, Roland McGrath wrote:
>> OK, but this will be a change on its own in a follow up patch.  I want to
>> keep this patch just a mechanical change.
> I don't want to put anything in the generic code that isn't clearly correct
> or isn't clearly justified or doesn't have thorough comments explaining any
> nonobvious issues.  So get it right first before the generic code goes in.
> Each machine dropping its old code in favor of the new generic code can
> wait as long as each machine maintainer wants.

And this kind of thing definitely varies by architecture.  For example, on
TILE-Gx the atomic operation is almost as fast as a regular store or load,
since there is no bus locking involved: the atomic instruction is
dispatched over the memory interconnect to the home cache for the address
in question, and returns a result.  However, you definitely don't want to
have dozens and dozens of cores all hammering compare-exchanges into the
memory network, so some kind of backoff turns out to be critical.

In addition, the "test and test-and-set" idiom [1] that is being proposed
here is actually an anti-pattern for Tilera (and likely for any other
cache-coherent mesh-based memory interconnect).  The problem is that each
"test" instruction (i.e. a load) pulls a shared copy of the cache line into
the tester's local cache, and then when some other core does the
test-and-set operation, the first thing done by the home cache is to
invalidate all the shared copies, which is slow and also loads the memory
network unnecessarily.  So plain test-and-set in a loop, with some kind of
backoff (we use bounded exponential backoff) ends up being the best thing
to do.

This isn't to say that the generic version might not be plausible for many
existing architectures, just that it certainly isn't going to be right for
all of them.

[1] http://en.wikipedia.org/wiki/Test_and_Test-and-set

-- 
Chris Metcalf, Tilera Corp.
http://www.tilera.com