thread-ready ABIs

thread-ready ABIs

[Ulrich Drepper]

The time is near when we (well, I) well start a drastic move toward
generally using thread registers.  Even in non-threaded code.

This means that unless all architectures get thread registers (or
equivalent things like Alpha's special code) we'll have a two class
society of platforms where all code written for the platforms without
thread register can be run on the other systems, but not vice versa.

From what I see today we have thread registers only on Alpha, x86,
IA-64, SH, and x86_64.  SPARC shouldn't be too much of a problem.  Sun
is using %g6 or %g7 (forgot which one) and since they define the ABI
no big complications are expected.

Now, what is about the rest?  I assume cris isn't much of a problem
since it's a purely embedded machine.


Arm: don't know whether this should fall in the same category.
Philip?


m68k: Well, maybe it's time to retire these machines.  But on the
other hand, there are those useless address registers.  Andreas, Jes?


PPC (32-bit) is known to be a problem.  I've seen several proposals as
to what register to use but haven't seen a final decision.  Problems
with the different PPC implementations are probably hindering this.
Geoff, could you please make a decision?  I hope the PPC64 ABI already
allocated a thread register.


S390: I have no idea.  Martin, please comment and make a decision.


MIPS: Who feels responsible?  Andreas, HJ?


PA: no idea.  HP has no 32-bit ELF so.  But they have 64-bit ELF and
it definitely has a thread register.



Please consider this a high priority task now.  I've been warning
about this for a long time.  Jakub is working on some code and once
this is ready for me to use I'll make lots of changes to ld.so and the
locale handling and from that point on we have the two classes of
architectures.

Oh, this now also concerns Hurd.  So, Roland, how far is using LDTs on
Hurd/x86?

-- 
---------------.                          ,-.   1325 Chesapeake Terrace
Ulrich Drepper  \    ,-------------------'   \  Sunnyvale, CA 94089 USA
Red Hat          `--' drepper at redhat.com   `------------------------

Re: thread-ready ABIs

[Ulrich Drepper]

Of course I should have looked closer: SPARC and S390 already have
thread register support.

-- 
---------------.                          ,-.   1325 Chesapeake Terrace
Ulrich Drepper  \    ,-------------------'   \  Sunnyvale, CA 94089 USA
Red Hat          `--' drepper at redhat.com   `------------------------

Re: thread-ready ABIs

[Andreas Schwab]

Ulrich Drepper <drepper@redhat.com> writes:

|> m68k: Well, maybe it's time to retire these machines.  But on the
|> other hand, there are those useless address registers.  Andreas, Jes?

None of the address registers are useless, they are all needed.  Reserving
one of them as a thread register means an ABI change, since currently all
(address or data) registers are either scratch or callee-saved.

Andreas.

-- 
Andreas Schwab, SuSE Labs, schwab@suse.de
SuSE GmbH, Deutschherrnstr. 15-19, D-90429 Nürnberg
Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: thread-ready ABIs

[Ulrich Drepper]

Andreas Schwab <schwab@suse.de> writes:

> None of the address registers are useless, they are all needed.  Reserving
> one of them as a thread register means an ABI change, since currently all
> (address or data) registers are either scratch or callee-saved.

And this means?  Make a decision.  Will you break the ABI or never
have a thread register?

-- 
---------------.                          ,-.   1325 Chesapeake Terrace
Ulrich Drepper  \    ,-------------------'   \  Sunnyvale, CA 94089 USA
Red Hat          `--' drepper at redhat.com   `------------------------

Re: thread-ready ABIs

[Ulrich Drepper]

Geoff Keating <geoffk@geoffk.org> writes:

> It will be %r2 for ELF.  No register is available for AIX and I will
> leave that unless someone can make a proposal.

AIX is no issue.  We don't determine the ABI, it's IBM's problem.
glibc will work as before on AIX.

Is %r2 currently unused?  If this a break in the ABI?

-- 
---------------.                          ,-.   1325 Chesapeake Terrace
Ulrich Drepper  \    ,-------------------'   \  Sunnyvale, CA 94089 USA
Red Hat          `--' drepper at redhat.com   `------------------------

Re: thread-ready ABIs

[Andreas Schwab]

Ulrich Drepper <drepper@redhat.com> writes:

|> Andreas Schwab <schwab@suse.de> writes:
|> 
|> > None of the address registers are useless, they are all needed.  Reserving
|> > one of them as a thread register means an ABI change, since currently all
|> > (address or data) registers are either scratch or callee-saved.
|> 
|> And this means?  Make a decision.  Will you break the ABI or never
|> have a thread register?

I have raised the issue on the linux-m68k mailinglist.  Let's see what
they say about this.

Andreas.

-- 
Andreas Schwab, SuSE Labs, schwab@suse.de
SuSE GmbH, Deutschherrnstr. 15-19, D-90429 Nürnberg
Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: thread-ready ABIs

[H . J . Lu]

On Thu, Jan 17, 2002 at 04:07:23PM -0800, Ulrich Drepper wrote:
> The time is near when we (well, I) well start a drastic move toward
> generally using thread registers.  Even in non-threaded code.
> 
> This means that unless all architectures get thread registers (or
> equivalent things like Alpha's special code) we'll have a two class
> society of platforms where all code written for the platforms without
> thread register can be run on the other systems, but not vice versa.
> 
> >From what I see today we have thread registers only on Alpha, x86,
> IA-64, SH, and x86_64.  SPARC shouldn't be too much of a problem.  Sun
> is using %g6 or %g7 (forgot which one) and since they define the ABI
> no big complications are expected.
> 
> Now, what is about the rest?  I assume cris isn't much of a problem
> since it's a purely embedded machine.
> 
> 
> Arm: don't know whether this should fall in the same category.
> Philip?
> 
> 
> m68k: Well, maybe it's time to retire these machines.  But on the
> other hand, there are those useless address registers.  Andreas, Jes?
> 
> 
> PPC (32-bit) is known to be a problem.  I've seen several proposals as
> to what register to use but haven't seen a final decision.  Problems
> with the different PPC implementations are probably hindering this.
> Geoff, could you please make a decision?  I hope the PPC64 ABI already
> allocated a thread register.
> 
> 
> S390: I have no idea.  Martin, please comment and make a decision.
> 
> 
> MIPS: Who feels responsible?  Andreas, HJ?
> 

I don't see there are any registers we can use without breaking ABI.
On the other hand, can we change the mips kernel to save k0 or k1 for
user space?

> 
> PA: no idea.  HP has no 32-bit ELF so.  But they have 64-bit ELF and
> it definitely has a thread register.
> 
> 
> 
> Please consider this a high priority task now.  I've been warning
> about this for a long time.  Jakub is working on some code and once
> this is ready for me to use I'll make lots of changes to ld.so and the
> locale handling and from that point on we have the two classes of
> architectures.
> 
> Oh, this now also concerns Hurd.  So, Roland, how far is using LDTs on
> Hurd/x86?
> 


H.J.

Re: thread-ready ABIs

[Ulrich Drepper]

"H . J . Lu" <hjl@lucon.org> writes:

> I don't see there are any registers we can use without breaking ABI.
> On the other hand, can we change the mips kernel to save k0 or k1 for
> user space?

Are these registers which are readable by normal users but writable
only in ring 0?  If yes, this is definitely worthwhile (similar to how
x86 works).  The only problem will be the MIPS variants which don't
have this register.  I bet there are some.

-- 
---------------.                          ,-.   1325 Chesapeake Terrace
Ulrich Drepper  \    ,-------------------'   \  Sunnyvale, CA 94089 USA
Red Hat          `--' drepper at redhat.com   `------------------------

Re: thread-ready ABIs

[H . J . Lu]

On Fri, Jan 18, 2002 at 10:31:17AM -0800, Ulrich Drepper wrote:
> "H . J . Lu" <hjl@lucon.org> writes:
> 
> > I don't see there are any registers we can use without breaking ABI.
> > On the other hand, can we change the mips kernel to save k0 or k1 for
> > user space?
> 
> Are these registers which are readable by normal users but writable
> only in ring 0?  If yes, this is definitely worthwhile (similar to how

I can write to k0/k1. But the value is not perserved by kernel.

> x86 works).  The only problem will be the MIPS variants which don't
> have this register.  I bet there are some.

I don't think so. k0/k1 is reserved for OS. I don't know if OS can
restore it for use space or not.


H.J.

Re: thread-ready ABIs

[Ulrich Drepper]

"H . J . Lu" <hjl@lucon.org> writes:

> I can write to k0/k1. But the value is not perserved by kernel.

Strange.  This means the registers cannot have been used so far and if
the kernel can be changed it is free.

> I don't think so. k0/k1 is reserved for OS. I don't know if OS can
> restore it for use space or not.

There are so many different MIPS implementations that I wouldn't bet
on it.  One would have to look at the minimum architecture definition.
Also, what do the new MIPS32 cores do?

-- 
---------------.                          ,-.   1325 Chesapeake Terrace
Ulrich Drepper  \    ,-------------------'   \  Sunnyvale, CA 94089 USA
Red Hat          `--' drepper at redhat.com   `------------------------

Re: thread-ready ABIs

[Ulrich Drepper]

Geoff Keating <geoffk@geoffk.org> writes:

> It is currently reserved for the system.  I believe some programs use
> it, but not many.

Let's start by modifying the current linuxthreads code to use this
register then.  This is not how it necessarily has to work in future
but it'll help shaking out problems.  Just define the few needed
macros in pt-machine.h.

-- 
---------------.                          ,-.   1325 Chesapeake Terrace
Ulrich Drepper  \    ,-------------------'   \  Sunnyvale, CA 94089 USA
Red Hat          `--' drepper at redhat.com   `------------------------

Re: thread-ready ABIs

[Geoff Keating]

> From: Ulrich Drepper <drepper@redhat.com>
> Date: 18 Jan 2002 08:09:52 -0800

> Geoff Keating <geoffk@geoffk.org> writes:
> 
> > It will be %r2 for ELF.  No register is available for AIX and I will
> > leave that unless someone can make a proposal.
> 
> AIX is no issue.  We don't determine the ABI, it's IBM's problem.
> glibc will work as before on AIX.
> 
> Is %r2 currently unused?  If this a break in the ABI?

It is currently reserved for the system.  I believe some programs use
it, but not many.

-- 
- Geoffrey Keating <geoffk@geoffk.org> <geoffk@redhat.com>

Re: thread-ready ABIs

[Jakub Jelinek]

On Fri, Jan 18, 2002 at 12:16:33PM -0800, Geoff Keating wrote:
> 
> > From: Ulrich Drepper <drepper@redhat.com>
> > Date: 18 Jan 2002 08:09:52 -0800
> 
> > Geoff Keating <geoffk@geoffk.org> writes:
> > 
> > > It will be %r2 for ELF.  No register is available for AIX and I will
> > > leave that unless someone can make a proposal.
> > 
> > AIX is no issue.  We don't determine the ABI, it's IBM's problem.
> > glibc will work as before on AIX.
> > 
> > Is %r2 currently unused?  If this a break in the ABI?
> 
> It is currently reserved for the system.  I believe some programs use
> it, but not many.

But if it is reseerved by the ABI, then it is certainly their fault
(remember fixing two or three packages which used %g6 on sparc).

	Jakub

Re: thread-ready ABIs

[Geoff Keating]

> From: Ulrich Drepper <drepper@redhat.com>
> Date: 17 Jan 2002 16:07:23 -0800

> PPC (32-bit) is known to be a problem.  I've seen several proposals as
> to what register to use but haven't seen a final decision.  Problems
> with the different PPC implementations are probably hindering this.
> Geoff, could you please make a decision?  I hope the PPC64 ABI already
> allocated a thread register.

It will be %r2 for ELF.  No register is available for AIX and I will
leave that unless someone can make a proposal.

-- 
- Geoffrey Keating <geoffk@geoffk.org> <geoffk@redhat.com>

Re: thread-ready ABIs

[H . J . Lu]

On Sat, Jan 19, 2002 at 01:35:38AM +0100, Kevin D. Kissell wrote:
> 
> It would, in principle, be possible to save/restore k0
> or k1 (but not both) if no other clever solution can be found.  
> There are other VM OSes that manage to do so for MIPS, 
> for other outside-the-old-ABI reasons.  It does, of course,
> add some instructions and some memory traffic to the 
> low-level exception handling , and we would have to look 
> at whether we would want to make such a feature standard 
> or specific to a "thread-ready" kernel build.

I like the read-only k0 idea. We just need to make a system call to
tell kernel what value to put in k0 before returning to the user space.
It shouldn't be too hard to implement. I will try it next week.


H.J.

Re: thread-ready ABIs

[H . J . Lu]

On Sat, Jan 19, 2002 at 12:27:52PM +0000, Dominic Sweetman wrote:
> 
> H . J . Lu (hjl@lucon.org) writes:
> 
> > > It would, in principle, be possible to save/restore k0
> > > or k1 (but not both) if no other clever solution can be found.  
> > > There are other VM OSes that manage to do so for MIPS, 
> > > for other outside-the-old-ABI reasons.  It does, of course,
> > > add some instructions and some memory traffic to the 
> > > low-level exception handling , and we would have to look 
> > > at whether we would want to make such a feature standard 
> > > or specific to a "thread-ready" kernel build.
> > 
> > I like the read-only k0 idea. We just need to make a system call to
> > tell kernel what value to put in k0 before returning to the user space.
> > It shouldn't be too hard to implement. I will try it next week.
> 
> You could, I guess, wire a TLB entry to map the thread register into
> the highest virtual memory region of the machine (the top of 'kseg2'),
> which is accessible in a single instruction as a negative offset from
> $0.  The kernel can write it through kseg0 or 64-bit equivalent, if
> you're a bit careful about cache aliases.

But it has to be a per thread value.

> 
> Reading something out of the cache is pretty cheap: would that be
> close enough to a 'register' to do the job?  There's no change to
> critical routines, that way.
> 

This is a patch against 2.4.16. Will this restore k1 to a known per
thread value?


H.J.
--- include/asm-mips/stackframe.h.thread	Wed Dec 12 12:34:53 2001
+++ include/asm-mips/stackframe.h	Sat Jan 19 11:36:38 2002
@@ -191,6 +191,7 @@ __asm__ (                               
 		lw	$2,  PT_R2(sp)
 
 #define RESTORE_SP_AND_RET                               \
+		lw	$27, PT_R27(sp);                 \
 		.set	push;				 \
 		.set	noreorder;			 \
 		lw	k0, PT_EPC(sp);                  \
@@ -229,6 +230,7 @@ __asm__ (                               
 		lw	$2,  PT_R2(sp)
 
 #define RESTORE_SP_AND_RET                               \
+		lw	$27, PT_R27(sp);                 \
 		lw	sp,  PT_R29(sp);                 \
 		.set	mips3;				 \
 		eret;					 \
@@ -237,6 +239,7 @@ __asm__ (                               
 #endif
 
 #define RESTORE_SP                                       \
+		lw	$27, PT_R27(sp);                 \
 		lw	sp,  PT_R29(sp);                 \
 
 #define RESTORE_ALL                                      \

Re: thread-ready ABIs

[Roland McGrath]

> The time is near when we (well, I) well start a drastic move toward
> generally using thread registers.  Even in non-threaded code.

Can you elaborate on this a little?  Do you just mean removing the
conditionalization in __libc_tsd_{get,set} and always calling the
functions?  A platform without thread registers could always just have
implementations of __libc_internal_tsd_* in -lpthread and -lc.

> Oh, this now also concerns Hurd.  So, Roland, how far is using LDTs on
> Hurd/x86?

I think it might be broken now, but we cna fix it.  We certainly intend to
have the feature (and/or something else equivalent for this purpose) and I
don't have a problem committing to relying on that in the libc-2.3 timeframe.

(The LDT approach is kind of gratuitous overhead for just this use of
emulating a thread register.  I might instead implement a simpler plan of a
constant %gs value pointing to a single location per CPU that the kernel
updates with a thread-specific word at context-switch time.)

Re: thread-ready ABIs

[Ulrich Drepper]

Roland McGrath <roland@frob.com> writes:

> Can you elaborate on this a little?  Do you just mean removing the
> conditionalization in __libc_tsd_{get,set} and always calling the
> functions?  A platform without thread registers could always just have
> implementations of __libc_internal_tsd_* in -lpthread and -lc.

There will of course always be support for platforms without thread
registers.  The existing code won't go away.  But for the platforms
with such support these functions indeed should go away.  As already
mentioned, I don't have a document describing this for all
architectures.  But you can look at the IA-64 ABI specification which
should give you a pretty good idea.

-- 
---------------.                          ,-.   1325 Chesapeake Terrace
Ulrich Drepper  \    ,-------------------'   \  Sunnyvale, CA 94089 USA
Red Hat          `--' drepper at redhat.com   `------------------------

Re: thread-ready ABIs

[Ulrich Drepper]

Ralf Baechle <ralf@oss.sgi.com> writes:

> Changing the kernel for the small number of threaded applications that
> exists and taking a performance impact for the kernel itself and anything
> that's using threads is an exquisite example for a bad tradeoff.

Well, it seems you haven't read what I wrote.  It's not about a small
number of threaded applications anymore.  The thread register will be
part of the ABI and all applications, threaded or not, will use it.

-- 
---------------.                          ,-.   1325 Chesapeake Terrace
Ulrich Drepper  \    ,-------------------'   \  Sunnyvale, CA 94089 USA
Red Hat          `--' drepper at redhat.com   `------------------------

Re: thread-ready ABIs

[Richard Henderson]

On Fri, Jan 18, 2002 at 10:19:08AM -0800, H . J . Lu wrote:
> On the other hand, can we change the mips kernel to save k0 or k1 for
> user space?

I doubt it.  Traditionally these are clobbered by the TLB fill trap.


r~

Re: thread-ready ABIs

[Martin Schwidefsky]

> S390: I have no idea.  Martin, please comment and make a decision.

For 31 bit S/390 we use the access register 0 as the thread register
and for 64 bit S/390 we use the access registers 0 and 1.
In short, everthing is already in place for S/390.

blue skies,
   Martin

Linux/390 Design & Development, IBM Deutschland Entwicklung GmbH
Schönaicherstr. 220, D-71032 Böblingen, Telefon: 49 - (0)7031 - 16-2247
E-Mail: schwidefsky@de.ibm.com

Re: thread-ready ABIs

[Ulrich Drepper]

"Maciej W. Rozycki" <macro@ds2.pg.gda.pl> writes:

> But what about that Alpha's special code?  It could possibly be
> reused given the large Alpha's similarity to MIPS.

No.  Alpha has certain builtin code which looks similar to calls or
software interrupts but are executed in the CPU.  This allows access
to some memory in the CPU which is almost as fast as a normal register
access.  MIPS doesn't have such hardware.  If you cannot find a
register you're doomed.

-- 
---------------.                          ,-.   1325 Chesapeake Terrace
Ulrich Drepper  \    ,-------------------'   \  Sunnyvale, CA 94089 USA
Red Hat          `--' drepper at redhat.com   `------------------------

Re: thread-ready ABIs

[Ulrich Drepper]

Andi Kleen <ak@suse.de> writes:

> I was just talking about the overhead for unthreaded apps (paying
> for things you don't use) 

That's the same.  There won't be special versions of libraries etc
which can be used in multi-threaded apps only.  Instead the code will
always use the same code and therefore the thread register handling
must always be enabled.

-- 
---------------.                          ,-.   1325 Chesapeake Terrace
Ulrich Drepper  \    ,-------------------'   \  Sunnyvale, CA 94089 USA
Red Hat          `--' drepper at redhat.com   `------------------------

Re: thread-ready ABIs

[Ulrich Drepper]

Andi Kleen <ak@suse.de> writes:

> That would be certainly not worth it. I hope at least that it is an
> big improvement for glibc, offseting the cost it adds to the context
> switch. 

This changes reaches far beyond glibc, it's an ABI change which can be
used by the compiler to generate much better code for threaded apps.
Look at the IA-64 ABI.  I'll have at some time similar documentation
for the other architectures but not now.

-- 
---------------.                          ,-.   1325 Chesapeake Terrace
Ulrich Drepper  \    ,-------------------'   \  Sunnyvale, CA 94089 USA
Red Hat          `--' drepper at redhat.com   `------------------------

Re: thread-ready ABIs

[Jakub Jelinek]

On Fri, Jan 18, 2002 at 08:07:15AM -0800, Ulrich Drepper wrote:
> Andi Kleen <ak@suse.de> writes:
> 
> > On x86-64 we have the situation that context switches get somewhat more 
> > expensive as soon as thread registers are used. It would be nice if 
> > use of thread registers in non threaded programs could be avoided.
> 
> Not possible.

But what can be certainly done is not use %gs as thread register on x86-64,
but instead %r13 or whatever you choose in the ABI.

	Jakub

Re: thread-ready ABIs

[Ulrich Drepper]

Andi Kleen <ak@suse.de> writes:

> On x86-64 we have the situation that context switches get somewhat more 
> expensive as soon as thread registers are used. It would be nice if 
> use of thread registers in non threaded programs could be avoided.

Not possible.

-- 
---------------.                          ,-.   1325 Chesapeake Terrace
Ulrich Drepper  \    ,-------------------'   \  Sunnyvale, CA 94089 USA
Red Hat          `--' drepper at redhat.com   `------------------------

Re: thread-ready ABIs

[Phil Blundell]

>Now, what is about the rest?  I assume cris isn't much of a problem
>since it's a purely embedded machine.
>
>Arm: don't know whether this should fall in the same category.
>Philip?

I am planning an ABI change for Linux in the libc 2.3 timeframe in any
case, so this would be a good opportunity to introduce a thread
register.  I'll bring it up with the ARM EABI group and see if we can
get consensus on which register to use.

p.