question regarding two security features

question regarding two security features

[Tom Henderson]

Hello, I'm writing with a question concerning whether two glibc security 
features could be optionally disabled for a specific use case.  The ns-3 
network simulator has a framework called Direct Code Execution [1, 2] 
that has been around for about ten years.  DCE allows us to load and 
simulate multiple instances of C or C++ executables and libraries that 
have been compiled as position independent code.  We link these with the 
system glibc.  Starting after glibc-2.25 or so, some vtable verification 
checks were introduced, and later, some restrictions on opening PIE 
objects were added, which blocked our usage of the system glibc for 
subsequent Linux releases (newer than Ubuntu 16.04).  We have been 
holding back on upgrades since then because of this issue.

Going forward, we can work around this by building a custom glibc (see 
sample patch at [3]) and using that within the DCE framework, but this 
approach is more heavyweight for users, so we are wondering if glibc 
would entertain the possibility of defining a link option to disable 
these features?

- Tom

[1] https://hal.inria.fr/hal-00880870/document
[2] https://github.com/direct-code-execution
[3] 
https://github.com/ParthPratim/ns-3-dce/commit/790b70674061eb11e94d7cf49d90b0373016178f

Re: question regarding two security features

[Adhemerval Zanella]

On 07/07/2021 22:58, Tom Henderson wrote:
> Hello, I'm writing with a question concerning whether two glibc security features could be optionally disabled for a specific use case.  The ns-3 network simulator has a framework called Direct Code Execution [1, 2] that has been around for about ten years.  DCE allows us to load and simulate multiple instances of C or C++ executables and libraries that have been compiled as position independent code.  We link these with the system glibc.  Starting after glibc-2.25 or so, some vtable verification checks were introduced, and later, some restrictions on opening PIE objects were added, which blocked our usage of the system glibc for subsequent Linux releases (newer than Ubuntu 16.04).  We have been holding back on upgrades since then because of this issue.
> 
> Going forward, we can work around this by building a custom glibc (see sample patch at [3]) and using that within the DCE framework, but this approach is more heavyweight for users, so we are wondering if glibc would entertain the possibility of defining a link option to disable these features?

Every configuration adds maintenance burden that we need take in consideration,
there are some cases where some options has bit-rotten and needed fixing 
(--disable-shared for instance, BZ#20845).  And as fair as I am aware new options
regarding security are opt-in, opt-out security issue are risky and usually
used in very fringe cases.  Another options is we might provide a way to disable
it through tunables, but even then it adds more surface where an attacker might
find a easier way to just disable it.

I have only skimmed through the paper, but I couldn't identify why exactly these
two security features are preventing you to use you framework. If you could describe
in more details what exactly is failing for newer glibcs we might come with ideas
of how to make it work.

> 
> - Tom
> 
> [1] https://hal.inria.fr/hal-00880870/document
> [2] https://github.com/direct-code-execution
> [3] https://github.com/ParthPratim/ns-3-dce/commit/790b70674061eb11e94d7cf49d90b0373016178f

Re: question regarding two security features

[Tom Henderson]

On 7/12/21 5:12 AM, Adhemerval Zanella wrote:
>
> On 07/07/2021 22:58, Tom Henderson wrote:
>> Hello, I'm writing with a question concerning whether two glibc security features could be optionally disabled for a specific use case.  The ns-3 network simulator has a framework called Direct Code Execution [1, 2] that has been around for about ten years.  DCE allows us to load and simulate multiple instances of C or C++ executables and libraries that have been compiled as position independent code.  We link these with the system glibc.  Starting after glibc-2.25 or so, some vtable verification checks were introduced, and later, some restrictions on opening PIE objects were added, which blocked our usage of the system glibc for subsequent Linux releases (newer than Ubuntu 16.04).  We have been holding back on upgrades since then because of this issue.
>>
>> Going forward, we can work around this by building a custom glibc (see sample patch at [3]) and using that within the DCE framework, but this approach is more heavyweight for users, so we are wondering if glibc would entertain the possibility of defining a link option to disable these features?
> Every configuration adds maintenance burden that we need take in consideration,
> there are some cases where some options has bit-rotten and needed fixing
> (--disable-shared for instance, BZ#20845).  And as fair as I am aware new options
> regarding security are opt-in, opt-out security issue are risky and usually
> used in very fringe cases.  Another options is we might provide a way to disable
> it through tunables, but even then it adds more surface where an attacker might
> find a easier way to just disable it.
>
> I have only skimmed through the paper, but I couldn't identify why exactly these
> two security features are preventing you to use you framework. If you could describe
> in more details what exactly is failing for newer glibcs we might come with ideas
> of how to make it work.

Thanks for replying; that paper doesn't get into the implementation 
details.  The only other online documentation that I know of is the DCE 
documentation here:

https://ns-3-dce.readthedocs.io/en/latest/how-it-works.html

but the better details are in a thesis that I can't find online right 
now.  Briefly, the issues are as follows:

DCE allows multiple instances of the same application to run in the 
simulator (itself a C++ process).  A good example application is the 
quagga routing suite (https://www.quagga.net) that provides routing 
protocol daemons.  DCE ensures that each instance does not share global 
and static variables, and overrides many system calls to redirect them 
to simulation-specific versions (e.g., so that 'gettimeofday' returns 
simulation time, not wall clock time).

The main technique used is to compile the application with -fPIC and 
link with -pie, and then to use either the system dlmopen() variant that 
supports the lmid namespaces argument, or a custom ELF dynamic loader 
[1].  Our ability to use the default dlmopen() seems to have been 
blocked by this patch:

https://patchwork.ozlabs.org/project/glibc/patch/20190312130235.8E82C89CE49C@oldenburg2.str.redhat.com/

The second issue is that DCE performs a vtable override [2] so that 
descriptors such as stdio are isolated and redirected. However, this has 
been blocked recently, and DCE returns the runtime error 'Fatal error: 
glibc detected an invalid stdio handle'.  Of course, there is a good 
reason to do this for avoiding FILE structure exploitation, but I 
thought I would ask whether anyone had an idea on how to bypass it for 
our use case without undoing the security hole it is designed to plug.

- Tom

[1] https://gitlab.com/nsnam/elf-loader

[2] 
https://github.com/direct-code-execution/ns-3-dce/blob/master/model/dce-stdio.cc#L184

[3] https://github.com/direct-code-execution/ns-3-dce/issues/57

Re: question regarding two security features

[Florian Weimer]

* Tom Henderson:

> The second issue is that DCE performs a vtable override [2] so that
> descriptors such as stdio are isolated and redirected. However, this
> has been blocked recently, and DCE returns the runtime error 'Fatal
> error: glibc detected an invalid stdio handle'.  Of course, there is a
> good reason to do this for avoiding FILE structure exploitation, but I 
> thought I would ask whether anyone had an idea on how to bypass it for
> our use case without undoing the security hole it is designed to plug.

Could you use fopencookie instead of a vtable override?

Thanks,
Florian

Re: question regarding two security features

[Tom Henderson]

On 7/13/21 1:23 PM, Florian Weimer wrote:
> * Tom Henderson:
> 
>> The second issue is that DCE performs a vtable override [2] so that
>> descriptors such as stdio are isolated and redirected. However, this
>> has been blocked recently, and DCE returns the runtime error 'Fatal
>> error: glibc detected an invalid stdio handle'.  Of course, there is a
>> good reason to do this for avoiding FILE structure exploitation, but I
>> thought I would ask whether anyone had an idea on how to bypass it for
>> our use case without undoing the security hole it is designed to plug.
> 
> Could you use fopencookie instead of a vtable override?
> 

Someone suggested this approach last year in our tracker but I have not 
seen any code around it; I will try to follow up based on your suggestion.

Thanks,
Tom

Re: question regarding two security features

[Parth Pratim Chatterjee]

Hello Florian,

Thanks for the suggestion on our problem with DCE.

I attempted to implement a solution based on fopencookie, and I'm facing a
few issues with it. Please find the commit link below [1].

In DCE, we basically open FILE streams to real files on the system, but try
to map all the stdio functionalities through our own handlers by
overwriting the vtable, also keeping track of the file descriptor number.
With fopencookie, we are being returned back a completely new FILE *
pointer, with custom callbacks to our own read/write/seek/close callbacks,
but in the original vtable hijacking implementation, we were also
overwriting the stat function callback, which we aren't able to do when
using fopencookie. Also the previously hijacked callback my_stat, was being
invoked multiple times during simulation as a sub call from fseek (based on
a gdb backtrace), making me wonder if it's important for us for a
consistent execution.

In addition to this, functionalities like freopen, which requires us to
return back the same pointer with a new instance of the FILE stream with
our custom vtable callbacks, is causing trouble, as fopencookie returns
back a new FILE pointer with a new address, causing our assertion tests
(new_fp == old_fp) to fail. Other functionalities like ftell after a fseek
and fcloseall, are failing at assert checks as well.

Also, after the ns-3 simulation ends, all fopencookie FILE streams are
being flushed by the libc, which forces write to be called, but we need an
ns-3 Simulation context to perform a write operation on a given FILE*
stream, not having access to any, leads to a SIGSEGV. This particularly
could be avoided if we tried to close a FILE stream in our
dce_fclose_unconditional(..) [2], which is responsible for closing an
already open FILE stream without an ns-3 simulation context. But, the
previous implementation goes as follows. There's a FILE stream X, with
vtable callbacks mapped to our handlers(which requires an ns-3 context),
but when it has to be closed post simulation, it is passed to
dce_fclose_unconditional, which overwrites the vtable back again with
handlers that do not require a context. It also considers FILE streams
which were previously not created with our custom callbacks just like the
FILE stream X was, making it consistent. But, with fopencookie,
implementing this seems to be a problem as fopencookie returns back a
completely new FILE stream if we try to map our new handlers.

These are pretty much the issues I've been facing right now trying to
implement your idea.
Please let me know if you have any insights to this, or if there are any
possible solutions to these problems.

[1] :
https://github.com/ParthPratim/ns-3-dce/commit/65217db7bec67679d748696a144ac5622e08c063
[2] :
https://github.com/direct-code-execution/ns-3-dce/blob/master/model/dce-stdio.cc#L340

Thank You,
Parth Pratim Chatterjee


On Fri, 23 Jul 2021 at 20:10, Tom Henderson <tomh@tomh.org> wrote:

>
> Email to reply to
>
> -------- Forwarded Message --------
> Subject: Re: question regarding two security features
> Date: Tue, 13 Jul 2021 22:23:53 +0200
> From: Florian Weimer <fweimer@redhat.com>
> To: Tom Henderson <tomh@tomh.org>
> CC: Adhemerval Zanella <adhemerval.zanella@linaro.org>,
> libc-alpha@sourceware.org
>
> * Tom Henderson:
>
> > The second issue is that DCE performs a vtable override [2] so that
> > descriptors such as stdio are isolated and redirected. However, this
> > has been blocked recently, and DCE returns the runtime error 'Fatal
> > error: glibc detected an invalid stdio handle'.  Of course, there is a
> > good reason to do this for avoiding FILE structure exploitation, but I
> > thought I would ask whether anyone had an idea on how to bypass it for
> > our use case without undoing the security hole it is designed to plug.
>
> Could you use fopencookie instead of a vtable override?
>
> Thanks,
> Florian
>
>

Re: question regarding two security features

[Florian Weimer]

* Parth Pratim Chatterjee:

> Hello Florian,
>
> Thanks for the suggestion on our problem with DCE. 
>
> I attempted to implement a solution based on fopencookie, and I'm
> facing a few issues with it. Please find the commit link below [1].
>
> In DCE, we basically open FILE streams to real files on the system,
> but try to map all the stdio functionalities through our own handlers
> by overwriting the vtable, also keeping track of the file descriptor
> number. With fopencookie, we are being returned back a completely new
> FILE * pointer, with custom callbacks to our own read/write/seek/close
> callbacks, but in the original vtable hijacking implementation, we
> were also overwriting the stat function callback, which we aren't able
> to do when using fopencookie. Also the previously hijacked callback
> my_stat, was being invoked multiple times during simulation as a sub
> call from fseek (based on a gdb backtrace), making me wonder if it's
> important for us for a consistent execution.
>
> In addition to this, functionalities like freopen, which requires us
> to return back the same pointer with a new instance of the FILE stream
> with our custom vtable callbacks, is causing trouble, as fopencookie
> returns back a new FILE pointer with a new address, causing our
> assertion tests (new_fp == old_fp) to fail. Other functionalities like
> ftell after a fseek and fcloseall, are failing at assert checks as
> well.

Could you wrap fopen, detect that it is your own file handle, and swap
the implementation details underneath?

In general, this hooking is really not supportable.  You should consider
using your own patched libc build for this, it will probably be easier
to maintain in the long return.

But I do wonder if saving a few context switching is really worth the
effort.

Thanks,
Florian

Re: question regarding two security features

[Tom Henderson]

On 9/17/21 3:33 AM, Florian Weimer via Libc-alpha wrote:
> * Parth Pratim Chatterjee:
> 
>> Hello Florian,
>>
>> Thanks for the suggestion on our problem with DCE.
>>
>> I attempted to implement a solution based on fopencookie, and I'm
>> facing a few issues with it. Please find the commit link below [1].
>>
>> In DCE, we basically open FILE streams to real files on the system,
>> but try to map all the stdio functionalities through our own handlers
>> by overwriting the vtable, also keeping track of the file descriptor
>> number. With fopencookie, we are being returned back a completely new
>> FILE * pointer, with custom callbacks to our own read/write/seek/close
>> callbacks, but in the original vtable hijacking implementation, we
>> were also overwriting the stat function callback, which we aren't able
>> to do when using fopencookie. Also the previously hijacked callback
>> my_stat, was being invoked multiple times during simulation as a sub
>> call from fseek (based on a gdb backtrace), making me wonder if it's
>> important for us for a consistent execution.
>>
>> In addition to this, functionalities like freopen, which requires us
>> to return back the same pointer with a new instance of the FILE stream
>> with our custom vtable callbacks, is causing trouble, as fopencookie
>> returns back a new FILE pointer with a new address, causing our
>> assertion tests (new_fp == old_fp) to fail. Other functionalities like
>> ftell after a fseek and fcloseall, are failing at assert checks as
>> well.
> 
> Could you wrap fopen, detect that it is your own file handle, and swap
> the implementation details underneath?
> 
> In general, this hooking is really not supportable.  You should consider
> using your own patched libc build for this, it will probably be easier
> to maintain in the long return.
> 
> But I do wonder if saving a few context switching is really worth the
> effort.
> 

Florian, thank you for taking the time to answer.  I did not understand 
your last sentence above; could you please elaborate?

Thanks,
Tom