RFC: Shadow Stack support in glibc

RFC: Shadow Stack support in glibc

[H.J. Lu]

To support Shadow Stack in Intel Control-flow Enforcement Technology (CET)
instructions:

https://software.intel.com/sites/default/files/managed/4d/2a/control-flow-enforcement-technology-preview.pdf

Most of glibc functions are compatible with Shadow Stack, except for

1. setjmp/longjmp need to be extended to support Shadow Stack.
2. getcontext/setcontext may be extended to support Shadow Stack.
3. makecontext/swapcontext are hard to support Shadow Stack.

To enable Shadow Stack in glibc, there are 2 approaches:

1. Only support Shadow Stack in getcontext/setcontext and disallow
makecontext/swapcontext when Shadow Stack is used.
2. Disallow getcontext/setcontext/makecontext/swapcontext when
when Shadow Stack is used.

If we decide to disallow getcontext/setcontext/makecontext/swapcontext
when Shadow Stack is used, should we add a new set of functions
compatible with Shadow Stack to manipulate user context?


-- 
H.J.

Re: RFC: Shadow Stack support in glibc

[Paul Eggert]

On 06/05/2017 02:36 PM, H.J. Lu wrote:
> To enable Shadow Stack in glibc, there are 2 approaches:
>
> 1. Only support Shadow Stack in getcontext/setcontext and disallow
> makecontext/swapcontext when Shadow Stack is used.
> 2. Disallow getcontext/setcontext/makecontext/swapcontext when
> when Shadow Stack is used.

The latter sounds simpler. Do you know of applications that would 
benefit from the former?

By "disallow" do you mean that getcontext and makecontext return -1 and 
set errno to ENOTSUP, or were you thinking of something else?

> If we decide to disallow getcontext/setcontext/makecontext/swapcontext
> when Shadow Stack is used, should we add a new set of functions
> compatible with Shadow Stack to manipulate user context?

What would this new set of functions look like, and how would one modify 
an application to use them? If getcontext/etc. are rarely used it might 
not be worth designing replacements.

Re: RFC: Shadow Stack support in glibc

[H.J. Lu]

On Mon, Jun 5, 2017 at 3:41 PM, Paul Eggert <eggert@cs.ucla.edu> wrote:
> On 06/05/2017 02:36 PM, H.J. Lu wrote:
>>
>> To enable Shadow Stack in glibc, there are 2 approaches:
>>
>> 1. Only support Shadow Stack in getcontext/setcontext and disallow
>> makecontext/swapcontext when Shadow Stack is used.
>> 2. Disallow getcontext/setcontext/makecontext/swapcontext when
>> when Shadow Stack is used.
>
>
> The latter sounds simpler. Do you know of applications that would benefit
> from the former?

I don't know how useful they are without makecontext/swapcontex.

> By "disallow" do you mean that getcontext and makecontext return -1 and set
> errno to ENOTSUP, or were you thinking of something else?

A single copy of libc has getcontext/setcontext/makecontext/swapcontext.
They work fine as long as Shadow Stack isn't used.  We can add

#if SHADOW STACK is enabled
# error getcontext/setcontext/makecontext/swapcontext can't be used
with Shadow Stack.
#endif

to <ucontext.h>.

>> If we decide to disallow getcontext/setcontext/makecontext/swapcontext
>> when Shadow Stack is used, should we add a new set of functions
>> compatible with Shadow Stack to manipulate user context?
>
>
> What would this new set of functions look like, and how would one modify an

The main issue is that an explicit "freecontext" function call is needed
to free shadow stack when context is no longer needed.   We need to
investigate how to modify each program to add freecontext call.

> application to use them? If getcontext/etc. are rarely used it might not be
> worth designing replacements.
>

makecontext, getcontext and setcontext are used in libgo.

-- 
H.J.

Re: RFC: Shadow Stack support in glibc

[Florian Weimer]

On 06/05/2017 11:36 PM, H.J. Lu wrote:
> Most of glibc functions are compatible with Shadow Stack, except for
> 
> 1. setjmp/longjmp need to be extended to support Shadow Stack.
> 2. getcontext/setcontext may be extended to support Shadow Stack.
> 3. makecontext/swapcontext are hard to support Shadow Stack.

What about these?

clone
sigaltstack
pthread_attr_setstack
pthread_attr_setstackaddr
pthread_attr_setstacksize

Thanks,
Florian

Re: RFC: Shadow Stack support in glibc

[Paul Eggert]

H.J. Lu wrote:

> #if SHADOW STACK is enabled

It's a compile-time option? (Somehow I expected runtime.)

Does the compiler need to generate special code for it? Can some modules be 
compiled with it, and some without? How does that work in the presence of setjmp?

> makecontext, getcontext and setcontext are used in libgo.

The libgo developers might be a good source for advice here.

Come to think of it, surely these functions can all be implemented via a 
syscall, which could check its arguments at least as well as the hardware checks 
ordinary call and return. (Just thinking out loud, admittedly.)

Re: RFC: Shadow Stack support in glibc

[Yu-cheng Yu]

On Mon, 2017-06-05 at 23:18 -0700, Florian Weimer wrote:
> On 06/05/2017 11:36 PM, H.J. Lu wrote:
> > Most of glibc functions are compatible with Shadow Stack, except for
> >
> > 1. setjmp/longjmp need to be extended to support Shadow Stack.
> > 2. getcontext/setcontext may be extended to support Shadow Stack.
> > 3. makecontext/swapcontext are hard to support Shadow Stack.
> 
> What about these?
> 
> clone
> sigaltstack
> pthread_attr_setstack
> pthread_attr_setstackaddr
> pthread_attr_setstacksize
> 

Please comment on the following.

Thanks,
Yu-cheng


clone:

The child's shadow stack can be copied on access, similar to
copy-on-write of a regular memory page.  A shadow stack PTE has to be
dirty and read-only.  When a task is cloned, the kernel makes shadow
stack PTEs clean until they are accessed again.  At that time, a copy is
made.

sigaltstack:

When the kernel gets the sigaltstack syscall, it allocates a
shadow_sigaltstack and records the pointer in the task header (similar
to the existing sigaltstack pointer).  If there is another sigaltstack
syscall, the kernel frees the previous shadow_sigaltstack; else, the
shadow_sigaltstack is freed upon task exit.

pthread_attr_xxx:

Since shadow stack stores only return pointers, it is not affected by
the address/size of the program stack.

Re: RFC: Shadow Stack support in glibc

[Szabolcs Nagy]

On 08/06/17 00:00, Yu-cheng Yu wrote:
> On Mon, 2017-06-05 at 23:18 -0700, Florian Weimer wrote:
>> On 06/05/2017 11:36 PM, H.J. Lu wrote:
>>> Most of glibc functions are compatible with Shadow Stack, except for
>>>
>>> 1. setjmp/longjmp need to be extended to support Shadow Stack.
>>> 2. getcontext/setcontext may be extended to support Shadow Stack.
>>> 3. makecontext/swapcontext are hard to support Shadow Stack.
>>
>> What about these?
>>
>> clone
>> sigaltstack
>> pthread_attr_setstack
>> pthread_attr_setstackaddr
>> pthread_attr_setstacksize
>>
> 
> Please comment on the following.
> 
> Thanks,
> Yu-cheng
> 
> 
> clone:
> 
> The child's shadow stack can be copied on access, similar to
> copy-on-write of a regular memory page.  A shadow stack PTE has to be
> dirty and read-only.  When a task is cloned, the kernel makes shadow
> stack PTEs clean until they are accessed again.  At that time, a copy is
> made.
> 
> sigaltstack:
> 
> When the kernel gets the sigaltstack syscall, it allocates a
> shadow_sigaltstack and records the pointer in the task header (similar
> to the existing sigaltstack pointer).  If there is another sigaltstack
> syscall, the kernel frees the previous shadow_sigaltstack; else, the
> shadow_sigaltstack is freed upon task exit.
> 
> pthread_attr_xxx:
> 
> Since shadow stack stores only return pointers, it is not affected by
> the address/size of the program stack.

why is it not affected by the size of the program stack?
how is the size of the shadow stack determined?

thread and signal stacks may be user allocated with special
mmap flags, moving return addresses to a different place
is observable and introduces new failure modes (shadow stack
allocation failure, shadow stack overflow), it is not clear
how an unwinder may access the shadow stack nor how the
ucontext/jmpbuf structs may store the new ssp register without
breaking abi.

Re: RFC: Shadow Stack support in glibc

[Florian Weimer]

On 06/08/2017 01:00 AM, Yu-cheng Yu wrote:
> clone:
> 
> The child's shadow stack can be copied on access, similar to
> copy-on-write of a regular memory page.  A shadow stack PTE has to be
> dirty and read-only.  When a task is cloned, the kernel makes shadow
> stack PTEs clean until they are accessed again.  At that time, a copy is
> made.

There are clone modes which do not duplicate the address space, and the
caller supplies a stack allocation.  Surely this new stack needs a
shadow stack stored somewhere, too?

> sigaltstack:
> 
> When the kernel gets the sigaltstack syscall, it allocates a
> shadow_sigaltstack and records the pointer in the task header (similar
> to the existing sigaltstack pointer).  If there is another sigaltstack
> syscall, the kernel frees the previous shadow_sigaltstack; else, the
> shadow_sigaltstack is freed upon task exit.

Interesting.  Is the allocated shadow stack visible from user space?
How it is sized?

> pthread_attr_xxx:
> 
> Since shadow stack stores only return pointers, it is not affected by
> the address/size of the program stack.

See Szabolcs' question.

Thanks,
Florian

Re: RFC: Shadow Stack support in glibc

[Yu-cheng Yu]

On Thu, 2017-06-08 at 11:46 +0200, Florian Weimer wrote:
> On 06/08/2017 01:00 AM, Yu-cheng Yu wrote:
> > clone:
> > 
> > The child's shadow stack can be copied on access, similar to
> > copy-on-write of a regular memory page.  A shadow stack PTE has to be
> > dirty and read-only.  When a task is cloned, the kernel makes shadow
> > stack PTEs clean until they are accessed again.  At that time, a copy is
> > made.
> 
> There are clone modes which do not duplicate the address space, and the
> caller supplies a stack allocation.  Surely this new stack needs a
> shadow stack stored somewhere, too?
> 

The shadow stack is always copied on access.

> > sigaltstack:
> > 
> > When the kernel gets the sigaltstack syscall, it allocates a
> > shadow_sigaltstack and records the pointer in the task header (similar
> > to the existing sigaltstack pointer).  If there is another sigaltstack
> > syscall, the kernel frees the previous shadow_sigaltstack; else, the
> > shadow_sigaltstack is freed upon task exit.
> 
> Interesting.  Is the allocated shadow stack visible from user space?
> How it is sized?
> 

When a task is created from exec(), the shadow stack is allocated to the
same size as the program stack.  The reasoning is the following.
(1) Physical memory is not committed until used,
(2) Shadow stack will not overflow before the program stack does,
(3) This design relieves any shadow stack management work from
applications.

For shadow_sigaltstack, it is still allocated to the same size as the
program stack for similar reasons.

Shadow stack is readable by the application.

> > pthread_attr_xxx:
> > 
> > Since shadow stack stores only return pointers, it is not affected by
> > the address/size of the program stack.
> 
> See Szabolcs' question.
> 

I will reply to his email.

Thanks,
Yu-cheng

Re: RFC: Shadow Stack support in glibc

[Yu-cheng Yu]

On Thu, 2017-06-08 at 09:46 +0100, Szabolcs Nagy wrote:
> >On 08/06/17 00:00, Yu-cheng Yu wrote:
> > 
> > pthread_attr_xxx:
> > 
> > Since shadow stack stores only return pointers, it is not affected by
> > the address/size of the program stack.
> 
> why is it not affected by the size of the program stack?
> how is the size of the shadow stack determined?
> 

Please see my reply to Florian on the reasoning of sizing the shadow
stack.  In summary, shadow stack is allocated to the same size as the
program stack.

> thread and signal stacks may be user allocated with special
> map flags, moving return addresses to a different place
> is observable and introduces new failure modes (shadow stack
> allocation failure, shadow stack overflow),
>

The shadow stack keeps only secure copies of return addresses (in
addition to ones already on the program stack) and cannot be directly
written by the application.  There is no requirement to correlate the
two stacks' location.  Allocation failure is treated the same as other
memory allocation failure.

> it is not clear
> how an unwinder may access the shadow stack nor how the
> ucontext/jmpbuf structs may store the new ssp register without
> breaking abi.
>

Shadow stack unwinding is done with the INCSSP instruction. The
ucontext/jmpbuf structs are likely to change and being discussed in the
thread started by H.J.

Thanks,
Yu-cheng

Re: RFC: Shadow Stack support in glibc

[Szabolcs Nagy]

On 08/06/17 20:46, Yu-cheng Yu wrote:
> 
> On Thu, 2017-06-08 at 09:46 +0100, Szabolcs Nagy wrote:
>>> On 08/06/17 00:00, Yu-cheng Yu wrote:
>>>
>>> pthread_attr_xxx:
>>>
>>> Since shadow stack stores only return pointers, it is not affected by
>>> the address/size of the program stack.
>>
>> why is it not affected by the size of the program stack?
>> how is the size of the shadow stack determined?
>>
> 
> Please see my reply to Florian on the reasoning of sizing the shadow
> stack.  In summary, shadow stack is allocated to the same size as the
> program stack.

as far as i understand the main thread stack can grow
(up to the rlimit or until the stack hits an already
mapped page) only 128K is committed when the process
starts (on linux).

so with a large rlimit it is in principle possible to
overflow the shadow stack.

>> thread and signal stacks may be user allocated with special
>> map flags, moving return addresses to a different place
>> is observable and introduces new failure modes (shadow stack
>> allocation failure, shadow stack overflow),
>>
> 
> The shadow stack keeps only secure copies of return addresses (in
> addition to ones already on the program stack) and cannot be directly
> written by the application.  There is no requirement to correlate the
> two stacks' location.  Allocation failure is treated the same as other
> memory allocation failure.
> 

ok, i don't think this is a major issue, but

allocation failure cannot be treated the same way as before
because sigaltstack only fails with ENOMEM if the user
supplied stack is small.

as for thread creation: the allocation of the thread stack
and the allocation of the shadow stack can happen at different
calls so the failure happens elsewhere than expected.

>> it is not clear
>> how an unwinder may access the shadow stack nor how the
>> ucontext/jmpbuf structs may store the new ssp register without
>> breaking abi.
>>
> 
> Shadow stack unwinding is done with the INCSSP instruction. The
> ucontext/jmpbuf structs are likely to change and being discussed in the
> thread started by H.J.
> 

makes sense

fixing setjmp/longjmp is non-trivial since jmpbuf size is abi,
it may be possible to do without saving ssp into jmpbuf though.

> Thanks,
> Yu-cheng
> 
> 

Re: RFC: Shadow Stack support in glibc

[H.J. Lu]

On Fri, Jun 9, 2017 at 2:05 AM, Szabolcs Nagy <szabolcs.nagy@arm.com> wrote:
>
> fixing setjmp/longjmp is non-trivial since jmpbuf size is abi,
> it may be possible to do without saving ssp into jmpbuf though.
>

Shadow stack is enabled at run-time only if all sources are compiled
with shadow stack enabled compiler against shadow stack enabled
glibc.  setjmp/longjmp are implemented with IFUNC, which uses
the old setjmp/longjmp if shadow stack is off and uses the new
setjmp/longjmp if shadow stack is on.

-- 
H.J.

Re: RFC: Shadow Stack support in glibc

[Szabolcs Nagy]

On 09/06/17 12:13, H.J. Lu wrote:
> On Fri, Jun 9, 2017 at 2:05 AM, Szabolcs Nagy <szabolcs.nagy@arm.com> wrote:
>>
>> fixing setjmp/longjmp is non-trivial since jmpbuf size is abi,
>> it may be possible to do without saving ssp into jmpbuf though.
>>
> 
> Shadow stack is enabled at run-time only if all sources are compiled
> with shadow stack enabled compiler against shadow stack enabled
> glibc.  setjmp/longjmp are implemented with IFUNC, which uses
> the old setjmp/longjmp if shadow stack is off and uses the new
> setjmp/longjmp if shadow stack is on.
> 

that is not enough.

jmpbuf may be used in user code independently of setjmp/longjmp.

Re: RFC: Shadow Stack support in glibc

[H.J. Lu]

On Fri, Jun 9, 2017 at 4:15 AM, Szabolcs Nagy <szabolcs.nagy@arm.com> wrote:
> On 09/06/17 12:13, H.J. Lu wrote:
>> On Fri, Jun 9, 2017 at 2:05 AM, Szabolcs Nagy <szabolcs.nagy@arm.com> wrote:
>>>
>>> fixing setjmp/longjmp is non-trivial since jmpbuf size is abi,
>>> it may be possible to do without saving ssp into jmpbuf though.
>>>
>>
>> Shadow stack is enabled at run-time only if all sources are compiled
>> with shadow stack enabled compiler against shadow stack enabled
>> glibc.  setjmp/longjmp are implemented with IFUNC, which uses
>> the old setjmp/longjmp if shadow stack is off and uses the new
>> setjmp/longjmp if shadow stack is on.
>>
>
> that is not enough.
>
> jmpbuf may be used in user code independently of setjmp/longjmp.
>

If that piece of code isn't shadow stack enabled, shadow stack won't
be enabled for the whole program and old setjmp/long will be used.

-- 
H.J.

Re: RFC: Shadow Stack support in glibc

[Szabolcs Nagy]

On 09/06/17 12:15, Szabolcs Nagy wrote:
> On 09/06/17 12:13, H.J. Lu wrote:
>> On Fri, Jun 9, 2017 at 2:05 AM, Szabolcs Nagy <szabolcs.nagy@arm.com> wrote:
>>>
>>> fixing setjmp/longjmp is non-trivial since jmpbuf size is abi,
>>> it may be possible to do without saving ssp into jmpbuf though.
>>>
>>
>> Shadow stack is enabled at run-time only if all sources are compiled
>> with shadow stack enabled compiler against shadow stack enabled
>> glibc.  setjmp/longjmp are implemented with IFUNC, which uses
>> the old setjmp/longjmp if shadow stack is off and uses the new
>> setjmp/longjmp if shadow stack is on.
>>
> 
> that is not enough.
> 
> jmpbuf may be used in user code independently of setjmp/longjmp.
> 
> 

oh wait, are you saying shadow stack is a new abi
and all libraries on the system need to be compiled
for that abi?

Re: RFC: Shadow Stack support in glibc

[H.J. Lu]

On Fri, Jun 9, 2017 at 4:21 AM, Szabolcs Nagy <szabolcs.nagy@arm.com> wrote:
> On 09/06/17 12:15, Szabolcs Nagy wrote:
>> On 09/06/17 12:13, H.J. Lu wrote:
>>> On Fri, Jun 9, 2017 at 2:05 AM, Szabolcs Nagy <szabolcs.nagy@arm.com> wrote:
>>>>
>>>> fixing setjmp/longjmp is non-trivial since jmpbuf size is abi,
>>>> it may be possible to do without saving ssp into jmpbuf though.
>>>>
>>>
>>> Shadow stack is enabled at run-time only if all sources are compiled
>>> with shadow stack enabled compiler against shadow stack enabled
>>> glibc.  setjmp/longjmp are implemented with IFUNC, which uses
>>> the old setjmp/longjmp if shadow stack is off and uses the new
>>> setjmp/longjmp if shadow stack is on.
>>>
>>
>> that is not enough.
>>
>> jmpbuf may be used in user code independently of setjmp/longjmp.
>>
>>
>
> oh wait, are you saying shadow stack is a new abi
> and all libraries on the system need to be compiled
> for that abi?
>

Yes,  99% of codes just need to recompile.  If context functions
are used or jmpbuf is used independently of setjmp/longjmp,
enabling is needed.

-- 
H.J.

Re: RFC: Shadow Stack support in glibc

[Florian Weimer]

On 06/09/2017 01:28 PM, H.J. Lu wrote:
> Yes,  99% of codes just need to recompile.  If context functions
> are used or jmpbuf is used independently of setjmp/longjmp,
> enabling is needed.

Would you please clarify if the size of the type jmp_buf changes in the
new compilation mode?

Thanks,
Florian

Re: RFC: Shadow Stack support in glibc

[H.J. Lu]

On Fri, Jun 9, 2017 at 4:32 AM, Florian Weimer <fweimer@redhat.com> wrote:
> On 06/09/2017 01:28 PM, H.J. Lu wrote:
>> Yes,  99% of codes just need to recompile.  If context functions
>> are used or jmpbuf is used independently of setjmp/longjmp,
>> enabling is needed.
>
> Would you please clarify if the size of the type jmp_buf changes in the
> new compilation mode?
>

jmp_buf size will be increased unconditionally in glibc whose <setjmp.h>
has

#if shadow stack is enabled
asm ("Turn on shadow stack in program property"):
#endif

The old setjmp/longjuimp only use the old jmp_buf size and the new
setjmp/longjuimp use the new jmp_buf size.  The new setjmp/longjmp
are used only if all input object files have shadow stack enabled in
program property.


-- 
H.J.

Re: RFC: Shadow Stack support in glibc

[Szabolcs Nagy]

On 09/06/17 12:40, H.J. Lu wrote:
> On Fri, Jun 9, 2017 at 4:32 AM, Florian Weimer <fweimer@redhat.com> wrote:
>> On 06/09/2017 01:28 PM, H.J. Lu wrote:
>>> Yes,  99% of codes just need to recompile.  If context functions
>>> are used or jmpbuf is used independently of setjmp/longjmp,
>>> enabling is needed.
>>
>> Would you please clarify if the size of the type jmp_buf changes in the
>> new compilation mode?
>>
> 
> jmp_buf size will be increased unconditionally in glibc whose <setjmp.h>
> has
> 
> #if shadow stack is enabled
> asm ("Turn on shadow stack in program property"):
> #endif
> 
> The old setjmp/longjuimp only use the old jmp_buf size and the new
> setjmp/longjuimp use the new jmp_buf size.  The new setjmp/longjmp
> are used only if all input object files have shadow stack enabled in
> program property.
> 

what happens if a process with shadow stack enabled
dlopens a lib with shadow stack disabled?

Re: RFC: Shadow Stack support in glibc

[H.J. Lu]

On Fri, Jun 9, 2017 at 4:44 AM, Szabolcs Nagy <szabolcs.nagy@arm.com> wrote:
> On 09/06/17 12:40, H.J. Lu wrote:
>> On Fri, Jun 9, 2017 at 4:32 AM, Florian Weimer <fweimer@redhat.com> wrote:
>>> On 06/09/2017 01:28 PM, H.J. Lu wrote:
>>>> Yes,  99% of codes just need to recompile.  If context functions
>>>> are used or jmpbuf is used independently of setjmp/longjmp,
>>>> enabling is needed.
>>>
>>> Would you please clarify if the size of the type jmp_buf changes in the
>>> new compilation mode?
>>>
>>
>> jmp_buf size will be increased unconditionally in glibc whose <setjmp.h>
>> has
>>
>> #if shadow stack is enabled
>> asm ("Turn on shadow stack in program property"):
>> #endif
>>
>> The old setjmp/longjuimp only use the old jmp_buf size and the new
>> setjmp/longjuimp use the new jmp_buf size.  The new setjmp/longjmp
>> are used only if all input object files have shadow stack enabled in
>> program property.
>>
>
> what happens if a process with shadow stack enabled
> dlopens a lib with shadow stack disabled?
>

My understanding is shadow stack can be disabled at any time.
If we can't do that, dlopen will fail.

-- 
H.J.

Re: RFC: Shadow Stack support in glibc

[Szabolcs Nagy]

On 09/06/17 12:45, H.J. Lu wrote:
> On Fri, Jun 9, 2017 at 4:44 AM, Szabolcs Nagy <szabolcs.nagy@arm.com> wrote:
>> On 09/06/17 12:40, H.J. Lu wrote:
>>> On Fri, Jun 9, 2017 at 4:32 AM, Florian Weimer <fweimer@redhat.com> wrote:
>>>> On 06/09/2017 01:28 PM, H.J. Lu wrote:
>>>>> Yes,  99% of codes just need to recompile.  If context functions
>>>>> are used or jmpbuf is used independently of setjmp/longjmp,
>>>>> enabling is needed.
>>>>
>>>> Would you please clarify if the size of the type jmp_buf changes in the
>>>> new compilation mode?
>>>>
>>>
>>> jmp_buf size will be increased unconditionally in glibc whose <setjmp.h>
>>> has
>>>

note that there is a large unused space in jmp_buf
on linux because of the hurd sigset (__saved_mask member),
if that space can be used in target specific ways then
the abi complications may be possible to avoid.

>>> #if shadow stack is enabled
>>> asm ("Turn on shadow stack in program property"):
>>> #endif
>>>
>>> The old setjmp/longjuimp only use the old jmp_buf size and the new
>>> setjmp/longjuimp use the new jmp_buf size.  The new setjmp/longjmp
>>> are used only if all input object files have shadow stack enabled in
>>> program property.
>>>
>>
>> what happens if a process with shadow stack enabled
>> dlopens a lib with shadow stack disabled?
>>
> 
> My understanding is shadow stack can be disabled at any time.
> If we can't do that, dlopen will fail.
> 

Re: RFC: Shadow Stack support in glibc

[H.J. Lu]

On Fri, Jun 9, 2017 at 5:57 AM, Szabolcs Nagy <szabolcs.nagy@arm.com> wrote:
> On 09/06/17 12:45, H.J. Lu wrote:
>> On Fri, Jun 9, 2017 at 4:44 AM, Szabolcs Nagy <szabolcs.nagy@arm.com> wrote:
>>> On 09/06/17 12:40, H.J. Lu wrote:
>>>> On Fri, Jun 9, 2017 at 4:32 AM, Florian Weimer <fweimer@redhat.com> wrote:
>>>>> On 06/09/2017 01:28 PM, H.J. Lu wrote:
>>>>>> Yes,  99% of codes just need to recompile.  If context functions
>>>>>> are used or jmpbuf is used independently of setjmp/longjmp,
>>>>>> enabling is needed.
>>>>>
>>>>> Would you please clarify if the size of the type jmp_buf changes in the
>>>>> new compilation mode?
>>>>>
>>>>
>>>> jmp_buf size will be increased unconditionally in glibc whose <setjmp.h>
>>>> has
>>>>
>
> note that there is a large unused space in jmp_buf
> on linux because of the hurd sigset (__saved_mask member),
> if that space can be used in target specific ways then
> the abi complications may be possible to avoid.

On x86, we have

# if __WORDSIZE == 64
typedef long int __jmp_buf[8];
# elif defined  __x86_64__
__extension__ typedef long long int __jmp_buf[8];
# else
typedef int __jmp_buf[6];
# endif

We need to grow it for shadow stack.

>>>> #if shadow stack is enabled
>>>> asm ("Turn on shadow stack in program property"):
>>>> #endif
>>>>
>>>> The old setjmp/longjuimp only use the old jmp_buf size and the new
>>>> setjmp/longjuimp use the new jmp_buf size.  The new setjmp/longjmp
>>>> are used only if all input object files have shadow stack enabled in
>>>> program property.
>>>>
>>>
>>> what happens if a process with shadow stack enabled
>>> dlopens a lib with shadow stack disabled?
>>>
>>
>> My understanding is shadow stack can be disabled at any time.
>> If we can't do that, dlopen will fail.
>>
>



-- 
H.J.

Re: RFC: Shadow Stack support in glibc

[Szabolcs Nagy]

On 09/06/17 14:06, H.J. Lu wrote:
> On Fri, Jun 9, 2017 at 5:57 AM, Szabolcs Nagy <szabolcs.nagy@arm.com> wrote:
>> On 09/06/17 12:45, H.J. Lu wrote:
>>> On Fri, Jun 9, 2017 at 4:44 AM, Szabolcs Nagy <szabolcs.nagy@arm.com> wrote:
>>>> On 09/06/17 12:40, H.J. Lu wrote:
>>>>> On Fri, Jun 9, 2017 at 4:32 AM, Florian Weimer <fweimer@redhat.com> wrote:
>>>>>> On 06/09/2017 01:28 PM, H.J. Lu wrote:
>>>>>>> Yes,  99% of codes just need to recompile.  If context functions
>>>>>>> are used or jmpbuf is used independently of setjmp/longjmp,
>>>>>>> enabling is needed.
>>>>>>
>>>>>> Would you please clarify if the size of the type jmp_buf changes in the
>>>>>> new compilation mode?
>>>>>>
>>>>>
>>>>> jmp_buf size will be increased unconditionally in glibc whose <setjmp.h>
>>>>> has
>>>>>
>>
>> note that there is a large unused space in jmp_buf
>> on linux because of the hurd sigset (__saved_mask member),
>> if that space can be used in target specific ways then
>> the abi complications may be possible to avoid.
> 
> On x86, we have
> 
> # if __WORDSIZE == 64
> typedef long int __jmp_buf[8];
> # elif defined  __x86_64__
> __extension__ typedef long long int __jmp_buf[8];
> # else
> typedef int __jmp_buf[6];
> # endif
> 
> We need to grow it for shadow stack.
> 

that's just the first part of the jmp_buf struct,
what matters is the size of jmp_buf which is

struct __jmp_buf_tag
  {
    /* NOTE: The machine-dependent definitions of `__sigsetjmp'
       assume that a `jmp_buf' begins with a `__jmp_buf' and that
       `__mask_was_saved' follows it.  Do not move these members
       or add others before it.  */
    __jmp_buf __jmpbuf;		/* Calling environment.  */
    int __mask_was_saved;	/* Saved the signal mask?  */
    __sigset_t __saved_mask;	/* Saved signal mask.  */
  };

typedef struct __jmp_buf_tag jmp_buf[1];

and only the first few bytes of __saved_mask is used on linux,
there is lot of free space there, it's just not setup for
target specific tinkering currently.


>>>>> #if shadow stack is enabled
>>>>> asm ("Turn on shadow stack in program property"):
>>>>> #endif
>>>>>
>>>>> The old setjmp/longjuimp only use the old jmp_buf size and the new
>>>>> setjmp/longjuimp use the new jmp_buf size.  The new setjmp/longjmp
>>>>> are used only if all input object files have shadow stack enabled in
>>>>> program property.
>>>>>
>>>>
>>>> what happens if a process with shadow stack enabled
>>>> dlopens a lib with shadow stack disabled?
>>>>
>>>
>>> My understanding is shadow stack can be disabled at any time.
>>> If we can't do that, dlopen will fail.
>>>
>>
> 
> 
> 

Re: RFC: Shadow Stack support in glibc

[Florian Weimer]

Szabolcs Nagy <szabolcs.nagy@arm.com> writes:

> that's just the first part of the jmp_buf struct,
> what matters is the size of jmp_buf which is
>
> struct __jmp_buf_tag
>   {
>     /* NOTE: The machine-dependent definitions of `__sigsetjmp'
>        assume that a `jmp_buf' begins with a `__jmp_buf' and that
>        `__mask_was_saved' follows it.  Do not move these members
>        or add others before it.  */
>     __jmp_buf __jmpbuf;		/* Calling environment.  */
>     int __mask_was_saved;	/* Saved the signal mask?  */
>     __sigset_t __saved_mask;	/* Saved signal mask.  */
>   };
>
> typedef struct __jmp_buf_tag jmp_buf[1];
>
> and only the first few bytes of __saved_mask is used on linux,
> there is lot of free space there, it's just not setup for
> target specific tinkering currently.

Yes, please investigate if you can make this work by stealing some
bits.  setjmp/longjmp isn't performance-critical, and if we can avoid
the ABI bump, I'd appreciate that very much.

Thanks,
Florian

Re: RFC: Shadow Stack support in glibc

[Joseph Myers]

On Fri, 9 Jun 2017, H.J. Lu wrote:

> On Fri, Jun 9, 2017 at 4:32 AM, Florian Weimer <fweimer@redhat.com> wrote:
> > On 06/09/2017 01:28 PM, H.J. Lu wrote:
> >> Yes,  99% of codes just need to recompile.  If context functions
> >> are used or jmpbuf is used independently of setjmp/longjmp,
> >> enabling is needed.
> >
> > Would you please clarify if the size of the type jmp_buf changes in the
> > new compilation mode?
> >
> 
> jmp_buf size will be increased unconditionally in glibc whose <setjmp.h>
> has

Previous experience (the s390 change in 2.19, reverted in 2.20) showed 
that changing jmp_buf size, even with symbol versioning, is very 
problematic.  jmp_buf gets embedded in types involved in the ABIs of 
third-party libraries such as libpng.

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: RFC: Shadow Stack support in glibc

[H.J. Lu]

On Fri, Jun 9, 2017 at 6:41 AM, Szabolcs Nagy <szabolcs.nagy@arm.com> wrote:
> On 09/06/17 14:06, H.J. Lu wrote:
>> On Fri, Jun 9, 2017 at 5:57 AM, Szabolcs Nagy <szabolcs.nagy@arm.com> wrote:
>>> On 09/06/17 12:45, H.J. Lu wrote:
>>>> On Fri, Jun 9, 2017 at 4:44 AM, Szabolcs Nagy <szabolcs.nagy@arm.com> wrote:
>>>>> On 09/06/17 12:40, H.J. Lu wrote:
>>>>>> On Fri, Jun 9, 2017 at 4:32 AM, Florian Weimer <fweimer@redhat.com> wrote:
>>>>>>> On 06/09/2017 01:28 PM, H.J. Lu wrote:
>>>>>>>> Yes,  99% of codes just need to recompile.  If context functions
>>>>>>>> are used or jmpbuf is used independently of setjmp/longjmp,
>>>>>>>> enabling is needed.
>>>>>>>
>>>>>>> Would you please clarify if the size of the type jmp_buf changes in the
>>>>>>> new compilation mode?
>>>>>>>
>>>>>>
>>>>>> jmp_buf size will be increased unconditionally in glibc whose <setjmp.h>
>>>>>> has
>>>>>>
>>>
>>> note that there is a large unused space in jmp_buf
>>> on linux because of the hurd sigset (__saved_mask member),
>>> if that space can be used in target specific ways then
>>> the abi complications may be possible to avoid.
>>
>> On x86, we have
>>
>> # if __WORDSIZE == 64
>> typedef long int __jmp_buf[8];
>> # elif defined  __x86_64__
>> __extension__ typedef long long int __jmp_buf[8];
>> # else
>> typedef int __jmp_buf[6];
>> # endif
>>
>> We need to grow it for shadow stack.
>>
>
> that's just the first part of the jmp_buf struct,
> what matters is the size of jmp_buf which is
>
> struct __jmp_buf_tag
>   {
>     /* NOTE: The machine-dependent definitions of `__sigsetjmp'
>        assume that a `jmp_buf' begins with a `__jmp_buf' and that
>        `__mask_was_saved' follows it.  Do not move these members
>        or add others before it.  */
>     __jmp_buf __jmpbuf;         /* Calling environment.  */
>     int __mask_was_saved;       /* Saved the signal mask?  */
>     __sigset_t __saved_mask;    /* Saved signal mask.  */
>   };
>
> typedef struct __jmp_buf_tag jmp_buf[1];
>
> and only the first few bytes of __saved_mask is used on linux,
> there is lot of free space there, it's just not setup for
> target specific tinkering currently.
>
>

We can use the unused space for SSP.

-- 
H.J.

Re: RFC: Shadow Stack support in glibc

[Florian Weimer]

"H.J. Lu" <hjl.tools@gmail.com> writes:

> We can use the unused space for SSP.

Great. :)

Please note that SSP already stands for “stack smashing protector”, so
it's probably best not reuse the same abbreviation for another hardening
feature.

Thanks,
Florian

Re: RFC: Shadow Stack support in glibc

[H.J. Lu]

[-- Attachment #1: Type: text/plain, Size: 428 bytes --]

On Fri, Jun 9, 2017 at 8:03 AM, Florian Weimer <fweimer@redhat.com> wrote:
> "H.J. Lu" <hjl.tools@gmail.com> writes:
>
>> We can use the unused space for SSP.
>
> Great. :)

Something like the patch here?

> Please note that SSP already stands for “stack smashing protector”, so
> it's probably best not reuse the same abbreviation for another hardening
> feature.

SSR (Shadow Stack Register)?


-- 
H.J.

[-- Attachment #2: 0001-x86-Add-a-field-to-jmp_buf-to-save-shadow-stack-poin.patch --]
[-- Type: text/x-patch, Size: 4692 bytes --]

From 96f7b89f1850a0daa771b812eead76076525eadd Mon Sep 17 00:00:00 2001
From: "H.J. Lu" <hjl.tools@gmail.com>
Date: Fri, 9 Jun 2017 08:34:44 -0700
Subject: [PATCH] x86: Add a field to jmp_buf to save shadow stack pointer

Since the __saved_mask field in jmp_buf is unused for x86, replace
__saved_mask with a union to save save shadow stack pointer while
keeping the size of jmp_buf unchanged.

	* sysdeps/x86/setjmp.h: New file.
---
 sysdeps/x86/setjmp.h | 112 +++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 112 insertions(+)
 create mode 100644 sysdeps/x86/setjmp.h

diff --git a/sysdeps/x86/setjmp.h b/sysdeps/x86/setjmp.h
new file mode 100644
index 0000000..8d32583
--- /dev/null
+++ b/sysdeps/x86/setjmp.h
@@ -0,0 +1,112 @@
+/* Copyright (C) 1991-2017 Free Software Foundation, Inc.
+   This file is part of the GNU C Library.
+
+   The GNU C Library is free software; you can redistribute it and/or
+   modify it under the terms of the GNU Lesser General Public
+   License as published by the Free Software Foundation; either
+   version 2.1 of the License, or (at your option) any later version.
+
+   The GNU C Library is distributed in the hope that it will be useful,
+   but WITHOUT ANY WARRANTY; without even the implied warranty of
+   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+   Lesser General Public License for more details.
+
+   You should have received a copy of the GNU Lesser General Public
+   License along with the GNU C Library; if not, see
+   <http://www.gnu.org/licenses/>.  */
+
+/*
+ *	ISO C99 Standard: 7.13 Nonlocal jumps	<setjmp.h>
+ */
+
+#ifndef	_SETJMP_H
+#define	_SETJMP_H	1
+
+#include <features.h>
+
+__BEGIN_DECLS
+
+#include <stdint.h>			/* Get uintptr_t.    */
+#include <bits/setjmp.h>		/* Get `__jmp_buf'.  */
+#include <bits/types/__sigset_t.h>
+
+/* Calling environment, plus possibly a saved signal mask.  */
+struct __jmp_buf_tag
+  {
+    /* NOTE: The machine-dependent definitions of `__sigsetjmp'
+       assume that a `jmp_buf' begins with a `__jmp_buf' and that
+       `__mask_was_saved' follows it.  Do not move these members
+       or add others before it.  */
+    __jmp_buf __jmpbuf;		/* Calling environment.  */
+    int __mask_was_saved;	/* Saved the signal mask?  */
+    union
+      {
+	/* Saved shadow stack pointer.  */
+	uintptr_t __saved_shadow_stack_pointer;
+	/* Saved signal mask.  */
+	__sigset_t __saved_mask;
+      } u;
+  };
+
+
+typedef struct __jmp_buf_tag jmp_buf[1];
+
+/* Store the calling environment in ENV, also saving the signal mask.
+   Return 0.  */
+extern int setjmp (jmp_buf __env) __THROWNL;
+
+/* Store the calling environment in ENV, also saving the
+   signal mask if SAVEMASK is nonzero.  Return 0.
+   This is the internal name for `sigsetjmp'.  */
+extern int __sigsetjmp (struct __jmp_buf_tag __env[1], int __savemask) __THROWNL;
+
+/* Store the calling environment in ENV, not saving the signal mask.
+   Return 0.  */
+extern int _setjmp (struct __jmp_buf_tag __env[1]) __THROWNL;
+
+/* Do not save the signal mask.  This is equivalent to the `_setjmp'
+   BSD function.  */
+#define setjmp(env)	_setjmp (env)
+
+
+/* Jump to the environment saved in ENV, making the
+   `setjmp' call there return VAL, or 1 if VAL is 0.  */
+extern void longjmp (struct __jmp_buf_tag __env[1], int __val)
+     __THROWNL __attribute__ ((__noreturn__));
+
+#if defined __USE_MISC || defined __USE_XOPEN
+/* Same.  Usually `_longjmp' is used with `_setjmp', which does not save
+   the signal mask.  But it is how ENV was saved that determines whether
+   `longjmp' restores the mask; `_longjmp' is just an alias.  */
+extern void _longjmp (struct __jmp_buf_tag __env[1], int __val)
+     __THROWNL __attribute__ ((__noreturn__));
+#endif
+
+
+#ifdef	__USE_POSIX
+/* Use the same type for `jmp_buf' and `sigjmp_buf'.
+   The `__mask_was_saved' flag determines whether
+   or not `longjmp' will restore the signal mask.  */
+typedef struct __jmp_buf_tag sigjmp_buf[1];
+
+/* Store the calling environment in ENV, also saving the
+   signal mask if SAVEMASK is nonzero.  Return 0.  */
+# define sigsetjmp(env, savemask)	__sigsetjmp (env, savemask)
+
+/* Jump to the environment saved in ENV, making the
+   sigsetjmp call there return VAL, or 1 if VAL is 0.
+   Restore the signal mask if that sigsetjmp call saved it.
+   This is just an alias `longjmp'.  */
+extern void siglongjmp (sigjmp_buf __env, int __val)
+     __THROWNL __attribute__ ((__noreturn__));
+#endif /* Use POSIX.  */
+
+
+/* Define helper functions to catch unsafe code.  */
+#if __USE_FORTIFY_LEVEL > 0
+# include <bits/setjmp2.h>
+#endif
+
+__END_DECLS
+
+#endif /* setjmp.h  */
-- 
2.9.4

Re: RFC: Shadow Stack support in glibc

[Yu-cheng Yu]

On Fri, 2017-06-09 at 10:05 +0100, Szabolcs Nagy wrote:
> On 08/06/17 20:46, Yu-cheng Yu wrote:
> > 
> > On Thu, 2017-06-08 at 09:46 +0100, Szabolcs Nagy wrote:
> >>> On 08/06/17 00:00, Yu-cheng Yu wrote:
> >>>
> >>> pthread_attr_xxx:
> >>>
> >>> Since shadow stack stores only return pointers, it is not affected by
> >>> the address/size of the program stack.
> >>
> >> why is it not affected by the size of the program stack?
> >> how is the size of the shadow stack determined?
> >>
> > 
> > Please see my reply to Florian on the reasoning of sizing the shadow
> > stack.  In summary, shadow stack is allocated to the same size as the
> > program stack.
> 
> as far as i understand the main thread stack can grow
> (up to the rlimit or until the stack hits an already
> mapped page) only 128K is committed when the process
> starts (on linux).
> 
> so with a large rlimit it is in principle possible to
> overflow the shadow stack.
> 

That is a good point.  I will look into it.

Thanks,
Yu-cheng

Re: RFC: Shadow Stack support in glibc

[Joseph Myers]

On Fri, 9 Jun 2017, H.J. Lu wrote:

> On Fri, Jun 9, 2017 at 8:03 AM, Florian Weimer <fweimer@redhat.com> wrote:
> > "H.J. Lu" <hjl.tools@gmail.com> writes:
> >
> >> We can use the unused space for SSP.
> >
> > Great. :)
> 
> Something like the patch here?

Both "u" and the contents of <stdint.h> are outside the <setjmp.h> 
namespace.  And I'd strongly discourage having sysdeps versions of any 
headers other than bits/ and sys/ ones, since they tend to get out of sync 
with the generic versions (and in this case, the include/setjmp.h wouldn't 
properly include your sysdeps version).

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: RFC: Shadow Stack support in glibc

[Szabolcs Nagy]

On 09/06/17 16:39, H.J. Lu wrote:
> Since the __saved_mask field in jmp_buf is unused for x86, replace
> __saved_mask with a union to save save shadow stack pointer while
> keeping the size of jmp_buf unchanged.
> 
> 	* sysdeps/x86/setjmp.h: New file.

sigsetjmp/siglongjmp has to save/restore the signal mask
but the signal mask size on linux is at most 16bytes (?)
and glibc uses 128 byte sigset_t.

so instead of the union below, i'd expect a solution where
if !HURD then the first 16bytes of __saved_mask and the rest
can be accessed separately and the tail bytes are usable
for target specific data.

> +/* Calling environment, plus possibly a saved signal mask.  */
> +struct __jmp_buf_tag
> +  {
> +    /* NOTE: The machine-dependent definitions of `__sigsetjmp'
> +       assume that a `jmp_buf' begins with a `__jmp_buf' and that
> +       `__mask_was_saved' follows it.  Do not move these members
> +       or add others before it.  */
> +    __jmp_buf __jmpbuf;		/* Calling environment.  */
> +    int __mask_was_saved;	/* Saved the signal mask?  */
> +    union
> +      {
> +	/* Saved shadow stack pointer.  */
> +	uintptr_t __saved_shadow_stack_pointer;
> +	/* Saved signal mask.  */
> +	__sigset_t __saved_mask;
> +      } u;
> +  };

Re: RFC: Shadow Stack support in glibc

[H.J. Lu]

[-- Attachment #1: Type: text/plain, Size: 1566 bytes --]

On Fri, Jun 9, 2017 at 8:58 AM, Szabolcs Nagy <szabolcs.nagy@arm.com> wrote:
> On 09/06/17 16:39, H.J. Lu wrote:
>> Since the __saved_mask field in jmp_buf is unused for x86, replace
>> __saved_mask with a union to save save shadow stack pointer while
>> keeping the size of jmp_buf unchanged.
>>
>>       * sysdeps/x86/setjmp.h: New file.
>
> sigsetjmp/siglongjmp has to save/restore the signal mask
> but the signal mask size on linux is at most 16bytes (?)
> and glibc uses 128 byte sigset_t.
>
> so instead of the union below, i'd expect a solution where
> if !HURD then the first 16bytes of __saved_mask and the rest
> can be accessed separately and the tail bytes are usable
> for target specific data.
>

How about this?  I allocated32 bytes for signal mask:

/* The biggest signal number + 1  */
#define _JUMP_BUF_SIGSET_NSIG 257
/* Number of longs to hold all signals.  */
#define _JUMP_BUF_SIGSET_NWORDS \
  ((_JUMP_BUF_SIGSET_NSIG - 1 + 7) / (8 * sizeof (unsigned long int)))

typedef struct
  {
    unsigned long int __val[_JUMP_BUF_SIGSET_NWORDS];
  } __jmp_buf_sigset_t;

typedef union
  {
    __sigset_t __saved_mask_compat;
    struct
      {
__jmp_buf_sigset_t __saved_mask;
void *__padding[12];
      } __saved;
  } __jmpbuf_target_t;

/* Saved signal mask.  */
#define __saved_mask __target.__saved.__saved_mask

and used

#if _JUMP_BUF_SIGSET_NSIG < _NSIG
# error _JUMP_BUF_SIGSET_NSIG <  _NSIG
#endif

 _Static_assert (sizeof (env[0].__target) == sizeof (__sigset_t),
 "__jmpbuf_target_t == __sigset_t");

to catch any future issues.


-- 
H.J.

[-- Attachment #2: 0001-Add-bits-setjmp3.h.patch --]
[-- Type: text/x-patch, Size: 5466 bytes --]

From 10fdcdb92c73bcbf170d764ee86196e8999d3357 Mon Sep 17 00:00:00 2001
From: "H.J. Lu" <hjl.tools@gmail.com>
Date: Sat, 10 Jun 2017 17:23:06 -0700
Subject: [PATCH] Add bits/setjmp3.h

---
 include/bits/setjmp3.h |  1 +
 setjmp/Makefile        |  2 +-
 setjmp/bits/setjmp3.h  | 47 +++++++++++++++++++++++++++++++++++++++++++++++
 setjmp/longjmp.c       | 13 ++++++++++---
 setjmp/setjmp.h        |  3 ++-
 setjmp/sigjmp.c        |  9 ++++++++-
 6 files changed, 69 insertions(+), 6 deletions(-)
 create mode 100644 include/bits/setjmp3.h
 create mode 100644 setjmp/bits/setjmp3.h

diff --git a/include/bits/setjmp3.h b/include/bits/setjmp3.h
new file mode 100644
index 0000000..ea343af
--- /dev/null
+++ b/include/bits/setjmp3.h
@@ -0,0 +1 @@
+#include <setjmp/bits/setjmp3.h>
diff --git a/setjmp/Makefile b/setjmp/Makefile
index ca80b8e..b52685f 100644
--- a/setjmp/Makefile
+++ b/setjmp/Makefile
@@ -22,7 +22,7 @@ subdir	:= setjmp
 
 include ../Makeconfig
 
-headers	:= setjmp.h bits/setjmp.h bits/setjmp2.h
+headers	:= setjmp.h bits/setjmp.h bits/setjmp2.h bits/setjmp3.h
 
 routines	:= setjmp sigjmp bsd-setjmp bsd-_setjmp \
 		   longjmp __longjmp jmp-unwind
diff --git a/setjmp/bits/setjmp3.h b/setjmp/bits/setjmp3.h
new file mode 100644
index 0000000..a21b037
--- /dev/null
+++ b/setjmp/bits/setjmp3.h
@@ -0,0 +1,47 @@
+/* __jmpbuf_target_t defition.
+   Copyright (C) 2017 Free Software Foundation, Inc.
+   This file is part of the GNU C Library.
+
+   The GNU C Library is free software; you can redistribute it and/or
+   modify it under the terms of the GNU Lesser General Public
+   License as published by the Free Software Foundation; either
+   version 2.1 of the License, or (at your option) any later version.
+
+   The GNU C Library is distributed in the hope that it will be useful,
+   but WITHOUT ANY WARRANTY; without even the implied warranty of
+   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+   Lesser General Public License for more details.
+
+   You should have received a copy of the GNU Lesser General Public
+   License along with the GNU C Library; if not, see
+   <http://www.gnu.org/licenses/>.  */
+
+#ifndef _SETJMP_H
+# error "Never include <bits/setjmp3.h> directly; use <setjmp.h> instead."
+#endif
+
+#include <bits/types/__sigset_t.h>
+
+/* The biggest signal number + 1  */
+#define _JUMP_BUF_SIGSET_NSIG	257
+/* Number of longs to hold all signals.  */
+#define _JUMP_BUF_SIGSET_NWORDS \
+  ((_JUMP_BUF_SIGSET_NSIG - 1 + 7) / (8 * sizeof (unsigned long int)))
+
+typedef struct
+  {
+    unsigned long int __val[_JUMP_BUF_SIGSET_NWORDS];
+  } __jmp_buf_sigset_t;
+
+typedef union
+  {
+    __sigset_t __saved_mask_compat;
+    struct
+      {
+	__jmp_buf_sigset_t __saved_mask;
+	void *__padding[12];
+      } __saved;
+  } __jmpbuf_target_t;
+
+/* Saved signal mask.  */
+#define __saved_mask __target.__saved.__saved_mask
diff --git a/setjmp/longjmp.c b/setjmp/longjmp.c
index 2453c2c..8418894 100644
--- a/setjmp/longjmp.c
+++ b/setjmp/longjmp.c
@@ -19,6 +19,9 @@
 #include <setjmp.h>
 #include <signal.h>
 
+#if _JUMP_BUF_SIGSET_NSIG < _NSIG
+# error _JUMP_BUF_SIGSET_NSIG <  _NSIG
+#endif
 
 /* Set the signal mask to the one specified in ENV, and jump
    to the position specified in ENV, causing the setjmp
@@ -30,9 +33,13 @@ __libc_siglongjmp (sigjmp_buf env, int val)
   _longjmp_unwind (env, val);
 
   if (env[0].__mask_was_saved)
-    /* Restore the saved signal mask.  */
-    (void) __sigprocmask (SIG_SETMASK, &env[0].__saved_mask,
-			  (sigset_t *) NULL);
+    {
+      _Static_assert (sizeof (env[0].__target) == sizeof (__sigset_t),
+		      "__jmpbuf_target_t == __sigset_t");
+      __sigset_t *saved_mask = (__sigset_t *) &env[0].__saved_mask;
+      /* Restore the saved signal mask.  */
+      (void) __sigprocmask (SIG_SETMASK, saved_mask, (sigset_t *) NULL);
+    }
 
   /* Call the machine-dependent function to restore machine state.  */
   __longjmp (env[0].__jmpbuf, val ?: 1);
diff --git a/setjmp/setjmp.h b/setjmp/setjmp.h
index 86fb2ed..4763ca1 100644
--- a/setjmp/setjmp.h
+++ b/setjmp/setjmp.h
@@ -27,6 +27,7 @@
 __BEGIN_DECLS
 
 #include <bits/setjmp.h>		/* Get `__jmp_buf'.  */
+#include <bits/setjmp3.h>
 #include <bits/types/__sigset_t.h>
 
 /* Calling environment, plus possibly a saved signal mask.  */
@@ -38,7 +39,7 @@ struct __jmp_buf_tag
        or add others before it.  */
     __jmp_buf __jmpbuf;		/* Calling environment.  */
     int __mask_was_saved;	/* Saved the signal mask?  */
-    __sigset_t __saved_mask;	/* Saved signal mask.  */
+    __jmpbuf_target_t __target;	/* Target specific data.  */
   };
 
 
diff --git a/setjmp/sigjmp.c b/setjmp/sigjmp.c
index 30839ae..91c6f56 100644
--- a/setjmp/sigjmp.c
+++ b/setjmp/sigjmp.c
@@ -19,6 +19,10 @@
 #include <setjmp.h>
 #include <signal.h>
 
+#if _JUMP_BUF_SIGSET_NSIG < _NSIG
+# error _JUMP_BUF_SIGSET_NSIG <  _NSIG
+#endif
+
 /* This function is called by the `sigsetjmp' macro
    before doing a `__setjmp' on ENV[0].__jmpbuf.
    Always return zero.  */
@@ -26,9 +30,12 @@
 int
 __sigjmp_save (sigjmp_buf env, int savemask)
 {
+  _Static_assert (sizeof (env[0].__target) == sizeof (__sigset_t),
+		  "__jmpbuf_target_t == __sigset_t");
+  __sigset_t *saved_mask = (__sigset_t *) &env[0].__saved_mask;
   env[0].__mask_was_saved = (savemask &&
 			     __sigprocmask (SIG_BLOCK, (sigset_t *) NULL,
-					    &env[0].__saved_mask) == 0);
+					    saved_mask) == 0);
 
   return 0;
 }
-- 
2.9.4