Re: [PATCH v3] Introduce strub: machine-independent stack scrubbing

[Alexandre Oliva <oliva@adacore.com>]

Hello, Qing,

On Jun 27, 2023, Qing Zhao <qing.zhao@oracle.com> wrote:

> I am wondering why stack scrubbing, proposed in this patch series, cannot do the stack scrubbing in the routine’s epilogue similar as
> register scrubbing?

There were multiple considerations that led to this design decision:

- Stack scrubbing in epilogues would be highly target-dependent

An epilogue expected to scrub the stack of its containing function would
not usually be able to call memset; there might not even be registers
available to do the cleaning, let alone to do it efficiently.  Since
epilogues are output after register allocation, the epilogue code
generator would have to allocate registers itself to do the job,
avoiding call-saved registers (that would have to be restored before
scrubbing the stack holding them), those holding return values, and
taking care of any machine- or ABI-specific conventions that apply to
epilogues.

- Exception Handling

Raising or propagating an exception requires a function's stack frame to
be active.  It wouldn't be possible for e.g. a cleanup handler to clean
up the stack frame holding it and then propagating the exception: either
the scrubbing would have to leave much of the stack frame alone for
propagation to work, or it would scrub too much and propagation would
fail.

So we had to devise a way for stack frames to be scrubbed and protect
the sensitive data in them even if an exception is raised or propagated
out of the sensitive frame.

- Variable frame size

Though many functions have static frame sizes, there are cases in which
a function dynamically allocates and releases stack space, and that
extra space should be scrubbed as well.  So the improvements out of a
known frame size are not a given, and we may need a watermark to handle
the general case.

Now consider that this watermark needs to survive past the point in
which the epilogue restores call-saved registers, so that the save area
can be scrubbed.  Call-clobbered registers might not be available, or
need scrubbing themselves.

A caller-owned watermark relieves the callee from these contradictory
requirements, enables the register pointing to the watermark to be
reused by the callee as soon as it's no longer needed; aggregation of
scrubbing, passing on the watermark when tail-calling another scrubbed
subprogram; caller and callee to be compiled separately, circumstances
in which the caller (in the strub("at-calls") mode) wouldn't know how
much stack space used by the callee is to be scrubbed.

- Watermark as in/out argument

Thus, watermarks, and caller-based scrubbing were required, so we might
as well use the same strategy for non-exceptional exit paths to make it
portable.

We've explored various possibilities of watermark passing to reduce the
impact on the ABI:

-- a single global variable wouldn't do in multi-threaded programs;
we need per-thread stack information.  TLS is not available on every
target, it's emulated with high overhead on some, and even when it
doesn't use part of the thread's stack for static thread-local
storage, each caller of a scrubbing function would have to preserve
that variable somehow (presumably in its own stack frame) before
reusing it to communicate with its callee.

-- a thread-local pointer to a heap-allocated parallel stack of
stack-scrubbing ranges might avoid holding the watermarks in the
stack, or passing pointers to them as arguments, leaving the entire
scrub range management in the library.  that would make the __strub_*
library components heavy enough that inlining them would not be
viable.  Furthermore, making such low-level APIs heap allocators
normally makes for problems of async-signal safety, and prevents heap
implementations from relying on such low-level APIs.

-- using the static chain machinery to convey to scrubbed callees
access to the callee's watermark seems viable, if onerous, but the
chained records live in the stack anyway, and there are targets that
do not support static chains.

-- an out parameter might do for "amount of stack used", but
making it an in/out watermark enabled aggregation and tail-calling;
early set-and-forget on fixed-size stack frames; and assured
initialization, even in case of an early asynchronous exception.

- Internal scrubbing

Though we have implemented strub("internal") through wrappers that call
the actual function and then scrub its stack space, we have envisioned
an alternate implementation that, through machine-specific support,
performs actual internal scrubbing, arranging the stack frame in such a
way that epilogues and EH cleanups can scrub most, if not all of the
stack frame (analogous to how the wrapper only scrubs the wrapped frame,
not its own), and taking advantage of constant frame sizes where
possible.  At least with variable frame sizes, the amount of stack space
to be scrubbed in the epilogue (or in an EH cleanup) will have to be
held in a local variable or somesuch, and at least for nonleaf
functions, that surely will end up in the stack one way or another.

> 2.  I have concerns on the runtime performance overhead, do you have any data on this for your current implementation?

Though one could conceivable build entire applications with the testing
option -fstrub=all, and that works AFAICT, the expected use case is
marking sensitive functions or variables for strubbing, and there aren't
benchmarks for this use case.


> 3. You mentioned that there are several “modes” for this feature,
> could you please provide more details on the modes and their
> description?

There's strict vs relaxed, and there's internal vs at-calls.  The
documentation for these modes included in the patch in quite extensive.
Rather than duplicating it here in other words, I suppose it would be a
better "test" for the documentation to have others go through it, try to
make sense of it, and point out passages that are unclear or hard to
understand.  WDYT?

Thanks,

-- 
Alexandre Oliva, happy hacker                https://FSFLA.org/blogs/lxo/
   Free Software Activist                       GNU Toolchain Engineer
Disinformation flourishes because many people care deeply about injustice
but very few check the facts.  Ask me about <https://stallmansupport.org>