[Bug dynamic-link/30134] New: DT_AUDIT is ignored for dlopen()ed solib

[Bug dynamic-link/30134] New: DT_AUDIT is ignored for dlopen()ed solib

[stsp at users dot sourceforge.net]

https://sourceware.org/bugzilla/show_bug.cgi?id=30134

            Bug ID: 30134
           Summary: DT_AUDIT is ignored for dlopen()ed solib
           Product: glibc
           Version: 2.38
            Status: UNCONFIRMED
          Severity: normal
          Priority: P2
         Component: dynamic-link
          Assignee: unassigned at sourceware dot org
          Reporter: stsp at users dot sourceforge.net
  Target Milestone: ---

Created attachment 14691
  --> https://sourceware.org/bugzilla/attachment.cgi?id=14691&action=edit
test case

The attached test-case links a solib
with --audit and then dlopen()s it.
If you do `ldd glreflib1.so` you'll see
the auditor printouts, so DT_AUDIT is set.
But dlopen() seems to ignore DT_AUDIT.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug dynamic-link/30134] DT_AUDIT is ignored for dlopen()ed solib

[fweimer at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=30134

Florian Weimer <fweimer at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |fweimer at redhat dot com

--- Comment #1 from Florian Weimer <fweimer at redhat dot com> ---
(In reply to Stas Sergeev from comment #0)
> Created attachment 14691 [details]
> test case
> 
> The attached test-case links a solib
> with --audit and then dlopen()s it.
> If you do `ldd glreflib1.so` you'll see
> the auditor printouts, so DT_AUDIT is set.
> But dlopen() seems to ignore DT_AUDIT.

This is currently the expected behavior. The link editor (ld) is expected to
copy audit entries to the main program as DT_DEPAUDIT entries, so that the
auditors are loaded at run time. But that only works for explicit linking, it
fails with dlopen.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug dynamic-link/30134] DT_AUDIT is ignored for dlopen()ed solib

[stsp at users dot sourceforge.net]

https://sourceware.org/bugzilla/show_bug.cgi?id=30134

--- Comment #2 from Stas Sergeev <stsp at users dot sourceforge.net> ---
(In reply to Florian Weimer from comment #1)
> This is currently the expected behavior.

But is it a desired/final behavior?

> The link editor (ld) is expected to
> copy audit entries to the main program as DT_DEPAUDIT entries, so that the
> auditors are loaded at run time. But that only works for explicit linking,
> it fails with dlopen.

It is possible to implement DT_AUDIT for
dlopen(), isn't it?
Many programs use plugins today, which is
dlopen() instead of a direct linking.
If dlopen() doesn't handle DT_AUDIT then
you can't provide a plugin that uses DT_AUDIT.
Is there any reason to not implement it
for dlopen?

It will likely make DT_DEPAUDIT unneeded
as a bonus. :)

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug dynamic-link/30134] DT_AUDIT is ignored for dlopen()ed solib

[stsp at users dot sourceforge.net]

https://sourceware.org/bugzilla/show_bug.cgi?id=30134

--- Comment #3 from Stas Sergeev <stsp at users dot sourceforge.net> ---
(In reply to Stas Sergeev from comment #2)
> It will likely make DT_DEPAUDIT unneeded
> as a bonus. :)

Or actually not: dlopen() should just
process both DT_AUDIT and DT_DEPAUDIT
entries because the dep libs could also
have DT_AUDIT. DT_DEPAUDIT will allow
dlopen() to load everything in advance.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug dynamic-link/30134] DT_AUDIT is ignored for dlopen()ed solib

[fweimer at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=30134

--- Comment #4 from Florian Weimer <fweimer at redhat dot com> ---
(In reply to Stas Sergeev from comment #2)
> (In reply to Florian Weimer from comment #1)
> > The link editor (ld) is expected to
> > copy audit entries to the main program as DT_DEPAUDIT entries, so that the
> > auditors are loaded at run time. But that only works for explicit linking,
> > it fails with dlopen.
> 
> It is possible to implement DT_AUDIT for
> dlopen(), isn't it?

Maybe in theory, but it's not how the current implementation is structured. It
depends on an auditing flag in RELRO memory for hardening, so that the auditing
(function) pointers cannot be overwritten if the process did not start with
auditing enabled.

We would also have to figure out all the small corner cases and work out what
to do for them. For example, auditing is currently a process-global operation,
and the auditing modules are never unloaded after being created. Adding dlopen
support might change that: auditing could perhaps be restricted to the local
search scope, and dlclose might unload the auditor as well. Or not, it's hard
to tell without some exploration.

We also have hard limits on the number of auditors. The explicit one (namespace
count) is perhaps somewhat easy to overcome, but there is also the static TLS
space consumption by new auditing namespaces.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug dynamic-link/30134] DT_AUDIT is ignored for dlopen()ed solib

[stsp at users dot sourceforge.net]

https://sourceware.org/bugzilla/show_bug.cgi?id=30134

--- Comment #5 from Stas Sergeev <stsp at users dot sourceforge.net> ---
(In reply to Florian Weimer from comment #4)
> Maybe in theory, but it's not how the current implementation is structured.
> It depends on an auditing flag in RELRO memory for hardening, so that the
> auditing (function) pointers cannot be overwritten if the process did not
> start with auditing enabled.

Yes, I already "noticed" that.
That's why my dlload_audit_module()
patch is still not posted. :(
But it will be posted sooner or later.
Yes, I realize it may be rejected
because it moves the audit list to
rw space, but what should I do if I
need that patch...
Do you have any suggestion in that
area while I am still polishing it?


> We would also have to figure out all the small corner cases and work out
> what to do for them. For example, auditing is currently a process-global
> operation, and the auditing modules are never unloaded after being created.
> Adding dlopen support might change that: auditing could perhaps be
> restricted to the local search scope, and dlclose might unload the auditor
> as well. Or not, it's hard to tell without some exploration.

Yes, closing the module that had
DT_AUDIT looks like a good place
to unload the audit module as well,
unless some other solib also had
the same DT_AUDIT. So perhaps that
would need a refcount.


> We also have hard limits on the number of auditors. The explicit one
> (namespace count) is perhaps somewhat easy to overcome, but there is also
> the static TLS space consumption by new auditing namespaces.

I am yet to understand why every
audit module needs a new NS. Wouldn't
it be possible to use one separate
NS for all audit modules?

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug dynamic-link/30134] DT_AUDIT is ignored for dlopen()ed solib

[fweimer at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=30134

--- Comment #6 from Florian Weimer <fweimer at redhat dot com> ---
(In reply to Stas Sergeev from comment #5)
> (In reply to Florian Weimer from comment #4)
> > We also have hard limits on the number of auditors. The explicit one
> > (namespace count) is perhaps somewhat easy to overcome, but there is also
> > the static TLS space consumption by new auditing namespaces.
> 
> I am yet to understand why every
> audit module needs a new NS. Wouldn't
> it be possible to use one separate
> NS for all audit modules?

I don't know. It does seem wasteful. I haven't seen many production examples of
audit modules. Multiple auditors loaded at startup may be rare (even rarer than
auditing itself), so it's possible that we could switch to a single separate
namespace without impacting backwards compatibility.

If the auditor cannot intercept libc.so.6 calls because it is already loaded,
then it can even use the main libc.so.6, and no fully separate namespace is
needed at all. There are many open questions and things to explore in this
area.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug dynamic-link/30134] DT_AUDIT is ignored for dlopen()ed solib

[stsp at users dot sourceforge.net]

https://sourceware.org/bugzilla/show_bug.cgi?id=30134

--- Comment #7 from Stas Sergeev <stsp at users dot sourceforge.net> ---
(In reply to Florian Weimer from comment #6)
> If the auditor cannot intercept libc.so.6 calls because it is already
> loaded, then it can even use the main libc.so.6, and no fully separate
> namespace is needed at all.

Certainly I don't know the implications,
but in all projects where I needed to
intercept the glibc calls I was using
LD_PRELOAD (or the more advanced techniques
like dlmopen() the libc into another
namespace and intercept it there), but
not LD_AUDIT. I've yet to understand is
there something special in those complex
pltenter/pltexit call-backs that you can't
do by some other means, and are there
many/any users of them?
But having audit modules in the main
namespace would help me a lot.

Also wrt security hardening you mentioned,
I wonder if such a measure (making audit
list R/O) was applied exactly because of
those pltenter/pltexit call-backs, that
looks like the good target for code injection?
If you had some subclass of "simple"
auditors that only assist the dlopen()
functionality, then would you still mark
the list of such "simple" auditors read-only?
Maybe we need the subclass of "simple"
or "safe" auditors that would be allowed
to load at run-time?
Is there any write-up about the objectives
and the desired functionality of that
auditing frame-work, or was it evolving
chaotically and eventually became a
potential security risk?

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug dynamic-link/30134] DT_AUDIT is ignored for dlopen()ed solib

[janderson at rice dot edu]

https://sourceware.org/bugzilla/show_bug.cgi?id=30134

Jonathon Anderson <janderson at rice dot edu> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |janderson at rice dot edu

--- Comment #8 from Jonathon Anderson <janderson at rice dot edu> ---
(In reply to Stas Sergeev from comment #7)
> (In reply to Florian Weimer from comment #6)
> > If the auditor cannot intercept libc.so.6 calls because it is already
> > loaded, then it can even use the main libc.so.6, and no fully separate
> > namespace is needed at all.
> 
> Certainly I don't know the implications,
> but in all projects where I needed to
> intercept the glibc calls I was using
> LD_PRELOAD (or the more advanced techniques
> like dlmopen() the libc into another
> namespace and intercept it there), but
> not LD_AUDIT. I've yet to understand is
> there something special in those complex
> pltenter/pltexit call-backs that you can't
> do by some other means, and are there
> many/any users of them?

pltenter/pltexit allows you to wrap symbols without knowing the signature of
the wrapped function. AFAIK there isn't a way to write a function that does
that in standard or GNU C (...without writing straight assembly based on the
call ABI, not fun).

That said, I don't personally know any users besides sotruss, one of the Glibc
dev tools. So do with that what you will. :D

> But having audit modules in the main
> namespace would help me a lot.

My group relies on LD_AUDIT to reliably "escape" the main namespace and acquire
symbols that won't be overridden by any other in-process tools operating at the
same time. Merging those namespaces will cause us some significant grief.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug dynamic-link/30134] DT_AUDIT is ignored for dlopen()ed solib

[stsp at users dot sourceforge.net]

https://sourceware.org/bugzilla/show_bug.cgi?id=30134

--- Comment #9 from Stas Sergeev <stsp at users dot sourceforge.net> ---
(In reply to Jonathon Anderson from comment #8)
> My group relies on LD_AUDIT to reliably "escape" the main namespace and
> acquire symbols that won't be overridden by any other in-process tools
> operating at the same time. Merging those namespaces will cause us some
> significant grief.

Even if merging in a still separate,
single auditing namespace?

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug dynamic-link/30134] DT_AUDIT is ignored for dlopen()ed solib

[janderson at rice dot edu]

https://sourceware.org/bugzilla/show_bug.cgi?id=30134

--- Comment #10 from Jonathon Anderson <janderson at rice dot edu> ---
(In reply to Stas Sergeev from comment #9)
> (In reply to Jonathon Anderson from comment #8)
> > My group relies on LD_AUDIT to reliably "escape" the main namespace and
> > acquire symbols that won't be overridden by any other in-process tools
> > operating at the same time. Merging those namespaces will cause us some
> > significant grief.
> 
> Even if merging in a still separate,
> single auditing namespace?

Unfortunately yes. We don't expect our software to be compiled with the same
libstdc++.so as the application or any other auditor, so sharing a namespace
won't end well for us.

I thought about a fake lmid for dlmopen to create "auditor-read-only"
namespaces, that auditors can see but aren't allowed to modify (our own can be
included). That would also provide the "escape" we need, but I presume there
are use cases under the "security" umbrella where hiding from the auditor(s)
(while not being an auditor yourself) would be highly undesirable.

One namespace per auditor is very convenient for us, please don't change it.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug dynamic-link/30134] DT_AUDIT is ignored for dlopen()ed solib

[stsp at users dot sourceforge.net]

https://sourceware.org/bugzilla/show_bug.cgi?id=30134

--- Comment #11 from Stas Sergeev <stsp at users dot sourceforge.net> ---
(In reply to Jonathon Anderson from comment #10)
> One namespace per auditor is very convenient for us, please don't change it.

Thanks for info.
But I think the "audit_hook from solib"
trick you described in another ticket, can
be used here too (with DT_AUDIT on dlopened
solib) because DT_AUDIT should be loaded
before solib itself. Which makes separate
namespaces not a big problem here.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug dynamic-link/30134] DT_AUDIT is ignored for dlopen()ed solib

[janderson at rice dot edu]

https://sourceware.org/bugzilla/show_bug.cgi?id=30134

--- Comment #12 from Jonathon Anderson <janderson at rice dot edu> ---
(In reply to Stas Sergeev from comment #11)
> (In reply to Jonathon Anderson from comment #10)
> > One namespace per auditor is very convenient for us, please don't change it.
> 
> Thanks for info.
> But I think the "audit_hook from solib"
> trick you described in another ticket, can
> be used here too (with DT_AUDIT on dlopened
> solib) because DT_AUDIT should be loaded
> before solib itself.

Continuing thoughts from the other ticket
(https://sourceware.org/bugzilla/show_bug.cgi?id=30127#c27).

Correct me if I'm wrong, but isn't the solib mapped before DT_AUDIT is read? So
the proposed la_premap wouldn't apply to the solib with the DT_AUDIT, only to
what it loads and whatever gets loaded later? Which doesn't sound at all like
what you want.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug dynamic-link/30134] DT_AUDIT is ignored for dlopen()ed solib

[stsp at users dot sourceforge.net]

https://sourceware.org/bugzilla/show_bug.cgi?id=30134

--- Comment #13 from Stas Sergeev <stsp at users dot sourceforge.net> ---
(In reply to Jonathon Anderson from comment #12)
> Correct me if I'm wrong, but isn't the solib mapped before DT_AUDIT is read?

See elf/rtld.c:dl_main.
At around line 1844 you see
  /* If we have auditing DSOs to load, do it now.  */
Then LD_PRELOADs are loaded, then DT_NEEDED.
Then you have la_activity and la_objopen.
So I think auditor is loaded first and that's
intentional.

> So the proposed la_premap wouldn't apply to the solib with the DT_AUDIT,
> only to what it loads and whatever gets loaded later? Which doesn't sound at
> all like what you want.

Well, both parts are not true. :)
It will be applied to the lib being loaded,
AND I want to customize another lib loaded
even later rather than this one.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug dynamic-link/30134] DT_AUDIT is ignored for dlopen()ed solib

[janderson at rice dot edu]

https://sourceware.org/bugzilla/show_bug.cgi?id=30134

--- Comment #14 from Jonathon Anderson <janderson at rice dot edu> ---
(In reply to Stas Sergeev from comment #13)
> (In reply to Jonathon Anderson from comment #12)
> > Correct me if I'm wrong, but isn't the solib mapped before DT_AUDIT is read?
> 
> See elf/rtld.c:dl_main.
> At around line 1844 you see
>   /* If we have auditing DSOs to load, do it now.  */
> Then LD_PRELOADs are loaded, then DT_NEEDED.
> Then you have la_activity and la_objopen.
> So I think auditor is loaded first and that's
> intentional.

Alright, made me look. :)

In elf/rtld.c:dl_main line 1821 calls load_audit_modules. That function loads
the auditors and then around line 1069 you see:
   /* Notify audit modules of the initially loaded modules (the main
      program and the dynamic linker itself).  */
   if (GLRO(dl_naudit) > 0)
     {
       _dl_audit_objopen (main_map, LM_ID_BASE);
       _dl_audit_objopen (&GL(dl_rtld_map), LM_ID_BASE);
     }
The first of these is the la_objopen callback for the applcation (main_map).
Note that (a) this is before the la_activity call on line 1842 and (b) the
main_map is already filled by this point.

Where does it get filled? Either when the executable is mapped when ld.so is
invoked as a command on line 1595:
           _dl_map_object (NULL, rtld_progname, lt_executable, 0,
                           __RTLD_OPENEXEC, LM_ID_BASE);
           rtld_timer_stop (&load_time, start);
         }

       /* Now the map for the main executable is available.  */
       main_map = GL(dl_ns)[LM_ID_BASE]._ns_loaded;
or in the "normal" case it gets stitched together from the auxiliary vector
(i.e. the kernel mapped the executable) on line 1650:
       /* Create a link_map for the executable itself.
          This will be what dlopen on "" returns.  */
       main_map = _dl_new_object ((char *) "", "", lt_executable, NULL,
                                  __RTLD_OPENEXEC, LM_ID_BASE);
       assert (main_map != NULL);
       main_map->l_phdr = phdr;
       main_map->l_phnum = phnum;
       main_map->l_entry = *user_entry;

In other words, the main executable gets mapped (long) before the auditors get
loaded. That doesn't seem like it can change soon, if at all.

I suppose solibs could be different, but AFAICT you would have to refactor
_dl_map_object_from_fd to *also* read the dynamic section and recursively load
any auditors *before* its call to _dl_map_segments. Not impossible but a much
more involved patch, and I haven't even considered how it would interact with
your dlmem() patch.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug dynamic-link/30134] DT_AUDIT is ignored for dlopen()ed solib

[stsp at users dot sourceforge.net]

https://sourceware.org/bugzilla/show_bug.cgi?id=30134

--- Comment #15 from Stas Sergeev <stsp at users dot sourceforge.net> ---
(In reply to Jonathon Anderson from comment #14)
> In other words, the main executable gets mapped (long) before the auditors
> get loaded.

Yes, but:
- We can see that la_objopen is explicitly
  called for main executable and ld.so, as if
  they were loaded later.
- As I pointed, solibs are processed later yet.

So no matter what, the auditor gets the calls
for every object, including the one that links
it in, and all its deps.

> I suppose solibs could be different, but AFAICT you would have to refactor
> _dl_map_object_from_fd to *also* read the dynamic section and recursively
> load any auditors *before* its call to _dl_map_segments.

Or just do similar to dl_main() and call the
new auditors at the end? Of course objsearch
will be skipped, but its the same for dl_main()
and no one complained. It would be really weird
if the auditor wants to alter the object path
of the object to which it linked to.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug dynamic-link/30134] DT_AUDIT is ignored for dlopen()ed solib

[janderson at rice dot edu]

https://sourceware.org/bugzilla/show_bug.cgi?id=30134

--- Comment #16 from Jonathon Anderson <janderson at rice dot edu> ---
(In reply to Stas Sergeev from comment #15)
> > I suppose solibs could be different, but AFAICT you would have to refactor
> > _dl_map_object_from_fd to *also* read the dynamic section and recursively
> > load any auditors *before* its call to _dl_map_segments.
> 
> Or just do similar to dl_main() and call the
> new auditors at the end? Of course objsearch
> will be skipped, but its the same for dl_main()
> and no one complained. It would be really weird
> if the auditor wants to alter the object path
> of the object to which it linked to.
Won't it skip la_premap too? Because DT_AUDIT auditors would only be loaded
after the object has been mapped (and the dynamic section is accessible).
la_premap has already fired before the auditor is even loaded, so the auditor
entirely unable to customize the mapping for the object itself. Based on
comment #13 isn't that a critical problem for your use case?

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug dynamic-link/30134] DT_AUDIT is ignored for dlopen()ed solib

[stsp at users dot sourceforge.net]

https://sourceware.org/bugzilla/show_bug.cgi?id=30134

--- Comment #17 from Stas Sergeev <stsp at users dot sourceforge.net> ---
Ah, I see your point.
So basically you mean, while DT_AUDIT
can well be supported for dlopen() right
now, it may be slightly inconsistent with
la_premap, as it would just skip it.

Well, for my use-case this is definitely
not a problem because my use-case is only
applying auditing to dlmem(), not dlopen().
dlmem() is done later by the dlopen()'ed
plugin. So for my case its perfectly fine.

Whether it is a generic consistency problem
or not, is hard to tell, as currently there
is no la_premap so we can't take a look at
how dl_main() would have supported it, be
it there. :) But I do share your concern,
especially since there was once that
LD_PREFER_MAP_32BIT_EXE thing that likely
didn't skip any objects. So if we are
modelling that, then la_premap is something
to think about for the implementer of
DT_AUDIT support for dlopen().

But please note one more thing.
I started dlload_audit_module() exactly
because I am not a big believer in DT_AUDIT
for dlopen(). DT_AUDIT for dlopen() is just
an "escape" scenario. So maybe if we get
dlload_audit_module() in, we can just close
this?

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug dynamic-link/30134] DT_AUDIT is ignored for dlopen()ed solib

[stsp at users dot sourceforge.net]

https://sourceware.org/bugzilla/show_bug.cgi?id=30134

--- Comment #18 from Stas Sergeev <stsp at users dot sourceforge.net> ---
(In reply to Jonathon Anderson from comment #14)
> I suppose solibs could be different, but AFAICT you would have to refactor
> _dl_map_object_from_fd to *also* read the dynamic section and recursively
> load any auditors *before* its call to _dl_map_segments.

Note that ld already does this and
adds DT_DEPAUDIT for any recursive
DT_AUDIT on a DT_NEEDED traversal.
So bringing up a list of DT_AUDIT+
DT_DEPAUDIT is probably not a rocket
science. But may still be too much
for the "escape" strategy, as this
RFE is. :)

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug dynamic-link/30134] DT_AUDIT is ignored for dlopen()ed solib

[stsp at users dot sourceforge.net]

https://sourceware.org/bugzilla/show_bug.cgi?id=30134

--- Comment #19 from Stas Sergeev <stsp at users dot sourceforge.net> ---
A bit more verbosely:
ld adds all sub-dependency DT_AUDITs
as DT_DEPAUDIT when linking an executable
or solib. So your concern about a recursive
search is not valid: the list is already
linearized.

-- 
You are receiving this mail because:
You are on the CC list for the bug.