[Bug dynamic-link/30127] New: [rfe]: enable ld audit at run-time

[Bug dynamic-link/30127] New: [rfe]: enable ld audit at run-time

[stsp at users dot sourceforge.net]

https://sourceware.org/bugzilla/show_bug.cgi?id=30127

            Bug ID: 30127
           Summary: [rfe]: enable ld audit at run-time
           Product: glibc
           Version: 2.38
            Status: UNCONFIRMED
          Severity: normal
          Priority: P2
         Component: dynamic-link
          Assignee: unassigned at sourceware dot org
          Reporter: stsp at users dot sourceforge.net
  Target Milestone: ---

Currently ld audit can only be enabled before starting
the process. One needs to set LD_AUDIT var or use --audit
switch of ld.so. Audit modules are unavailable for look-up
with dlopen(RTLD_NOLOAD) as they go to another link map, so
I've found no way of communicating with audit modules from
the program.

Would it be possible to allow loading the audit modules at
run-time? Perhaps just adding an RTLD_AUDIT flag would be
enough for that task. Then the user will be able to bind
an already dlopened solibs to audit, or load a new one and
get its handle.

LD_AUDIT allows a lot of customization of rtld behavior, but
without the run-time loading of audit modules that functionality
is not easy to use, it seems.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug dynamic-link/30127] [rfe]: enable ld audit at run-time

[fweimer at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=30127

Florian Weimer <fweimer at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |fweimer at redhat dot com

--- Comment #1 from Florian Weimer <fweimer at redhat dot com> ---
It's possible to activate audit modules via DT_AUDIT (settable via “ld
--audit”). This may help for at least some applications.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug dynamic-link/30127] [rfe]: enable ld audit at run-time

[stsp at users dot sourceforge.net]

https://sourceware.org/bugzilla/show_bug.cgi?id=30127

--- Comment #2 from Stas Sergeev <stsp at users dot sourceforge.net> ---
Thanks!
DT_AUDIT is good indeed.
But I still can't get a dl handle
of an audit object, so I still don't
know how to communicate with them.
Any trick?

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug dynamic-link/30127] [rfe]: enable ld audit at run-time

[stsp at users dot sourceforge.net]

https://sourceware.org/bugzilla/show_bug.cgi?id=30127

--- Comment #3 from Stas Sergeev <stsp at users dot sourceforge.net> ---
So if its not possible to get a dl handle
for DT_AUDIT object, I would propose the
following extension:

- void *dlaudit_load_module(const char *path, int flags)
  Loads the audit module at run-time and returns the handle.
  Flags are reserved for future needs.
  Such modules do not audit the normal dlopen() or dlmopen()
  calls. They only audit the dlopen_audit() call below.
  In the future the flag can be added to allow these modules
  to audit any calls, not only dlopen_audit() one.

- void *dlopen_audit(const char *path, int flags, void *cookie)
  Same as dlopen(), but allows the loading of a solib to be
  audited also by the modules loaded via dlaudit_load_module(),
  together with auditing modules loaded by other means.
  The pointer to "cookie" arg is passed to la_xxx() auditing
  funcs as a cookie (of course they can overwrite it, as before).

Will something like this be viable?
Or if its possible to get the dlhandle for
DT_AUDIT object, then I can use it too, except
the cookie passing would be a bit cumbersome.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug dynamic-link/30127] [rfe]: enable ld audit at run-time

[fweimer at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=30127

--- Comment #4 from Florian Weimer <fweimer at redhat dot com> ---
(In reply to Stas Sergeev from comment #2)
> Thanks!
> DT_AUDIT is good indeed.
> But I still can't get a dl handle
> of an audit object, so I still don't
> know how to communicate with them.
> Any trick?

It's possible to enumerate other namespaces using the extended variant of
_r_debug. But maybe we should add a global key-value store to make
communication among modules easier. We should perhaps make getenv/setenv
process-global, too, but it's not a good fit for this purpose because the
environment variables are inherited by most subprocesses.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug dynamic-link/30127] [rfe]: enable ld audit at run-time

[stsp at users dot sourceforge.net]

https://sourceware.org/bugzilla/show_bug.cgi?id=30127

--- Comment #5 from Stas Sergeev <stsp at users dot sourceforge.net> ---
(In reply to Florian Weimer from comment #4)
> It's possible to enumerate other namespaces using the extended variant of
> _r_debug.

Zero luck with that:

(gdb) p *(struct r_debug_extended *)&_r_debug
$4 = {base = {r_version = 2, r_map = 0x7ffff7ffe2e0, r_brk = 140737353913088, 
    r_state = RT_ADD, r_ldbase = 140737353904128}, r_next = 0x0}

r_next is NULL, as you can see.
And DT_AUDIT is not empty, I see all printfs
from auditors.
So audit namespace is not covered by _r_debug?
Or am I doing something wrong?

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug dynamic-link/30127] [rfe]: enable ld audit at run-time

[fweimer at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=30127

--- Comment #6 from Florian Weimer <fweimer at redhat dot com> ---
It works for me:

(gdb) print ((struct r_debug_extended)_r_debug).r_next.base.r_map.l_name
$5 = 0x7f13d58926f0 "elf/sotruss-lib.so"
(gdb) print ((struct r_debug_extended *)&_r_debug).r_next.base.r_map.l_name
$6 = 0x7f13d58926f0 "elf/sotruss-lib.so"

This is with glibc-2.36-9.fc37.x86_64. We don't have any downstream fixes for
audit support.

There is a problem with copy relocations against _r_debug. If you try to access
_r_debug programmatically, try to build with -fPIC.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug dynamic-link/30127] [rfe]: enable ld audit at run-time

[stsp at users dot sourceforge.net]

https://sourceware.org/bugzilla/show_bug.cgi?id=30127

--- Comment #7 from Stas Sergeev <stsp at users dot sourceforge.net> ---
(In reply to Florian Weimer from comment #6)
> There is a problem with copy relocations against _r_debug. If you try to
> access _r_debug programmatically, try to build with -fPIC.

Oh wow, I was using -fpie, but indeed adding also -fpic to executable
fixes _r_debug! Very confusing...

Still no luck though.
To get the handle, I convert the struct link_map ptr to 

struct link_map_extended {
    /* These first few members are part of the protocol with the debugger.
       This is the same format used in SVR4.  */

    ElfW(Addr) l_addr;          /* Difference between the address in the ELF
                                   file and the addresses in memory.  */
    char *l_name;               /* Absolute file name object was found in.  */
    ElfW(Dyn) *l_ld;            /* Dynamic section of the shared object.  */
    struct link_map *l_next, *l_prev; /* Chain of loaded objects.  */

    /* All following members are internal to the dynamic linker.
       They may change without notice.  */

    /* This is an element which is only ever different from a pointer to
       the very same copy of this type for ld.so when it is used in more
       than one namespace.  */
    struct link_map *l_real;

    /* Number of the namespace this link map belongs to.  */
    Lmid_t l_ns;
};

I can see that the ns member seems correct:
it is 0 for first namespace and 1 for audit namespace.
But this fails:

void *h = dlmopen(((struct link_map_extended *)l1)->l_ns, l1->l_name, RTLD_LAZY
| RTLD_LOCAL | RTLD_NOLOAD);

... with this dlerror():
./tst-auditmod18.so: invalid target namespace in dlmopen(): Invalid argument


So I still can't get a handle. :(
Plus I use undocumented struct link_map_extended, which
I probably shouldn't do.
So what is to try?

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug dynamic-link/30127] [rfe]: enable ld audit at run-time

[stsp at users dot sourceforge.net]

https://sourceware.org/bugzilla/show_bug.cgi?id=30127

--- Comment #8 from Stas Sergeev <stsp at users dot sourceforge.net> ---
  /* Never allow loading a DSO in a namespace which is empty.  Such
     direct placements is only causing problems.  Also don't allow
     loading into a namespace used for auditing.  */
  else if (__glibc_unlikely (nsid != LM_ID_BASE && nsid != __LM_ID_CALLER)
           && (__glibc_unlikely (nsid < 0 || nsid >= GL(dl_nns))
               /* This prevents the [NSID] index expressions from being
                  evaluated, so the compiler won't think that we are
                  accessing an invalid index here in the !SHARED case where
                  DL_NNS is 1 and so any NSID != 0 is invalid.  */
               || DL_NNS == 1
               || GL(dl_ns)[nsid]._ns_nloaded == 0
               || GL(dl_ns)[nsid]._ns_loaded->l_auditing))
    _dl_signal_error (EINVAL, file, NULL,
                      N_("invalid target namespace in dlmopen()"));


So getting a handle for objects in an audit
namespace is explicitly disallowed? :(
How is so?

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug dynamic-link/30127] [rfe]: enable ld audit at run-time

[stsp at users dot sourceforge.net]

https://sourceware.org/bugzilla/show_bug.cgi?id=30127

--- Comment #9 from Stas Sergeev <stsp at users dot sourceforge.net> ---
Can you make an exception for RTLD_NOLOAD flag
and allow getting a handle?

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug dynamic-link/30127] [rfe]: enable ld audit at run-time

[fweimer at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=30127

--- Comment #10 from Florian Weimer <fweimer at redhat dot com> ---
I think for experimentation, you can use the link map as a dlopen handle
directly, and pass it to dlsym, for example.

We need to gather consensus that this is the way to do it, and document it.
Presently, it's an undocumented implementation detail.

We also need to use a dedicated link map allocator which only reuses, but not
frees, the underlying storiage, so that it becomes safe to traverse the
_r_debug lists, despite concurrent dlopen/dlcose operation.

There should also be a way to obtain a stable handle from a link map pointer.
Using dlopen with RTLD_NOLOAD seems about right for that. Maybe we should
maintain a hash table with the l_name pointers (not the strings), so that we
can precisely identify the link map from its l_name member, even if the name is
not unique or concurrently overwritten.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug dynamic-link/30127] [rfe]: enable ld audit at run-time

[stsp at users dot sourceforge.net]

https://sourceware.org/bugzilla/show_bug.cgi?id=30127

--- Comment #11 from Stas Sergeev <stsp at users dot sourceforge.net> ---
(In reply to Florian Weimer from comment #10)
> I think for experimentation, you can use the link map as a dlopen handle
> directly, and pass it to dlsym, for example.

That finally works, thank you.
So with that and the patches already posted to ML,
I can at least start prototyping some code.


> We need to gather consensus that this is the way to do it, and document it.
> Presently, it's an undocumented implementation detail.
> 
> We also need to use a dedicated link map allocator which only reuses, but
> not frees, the underlying storiage, so that it becomes safe to traverse the
> _r_debug lists, despite concurrent dlopen/dlcose operation.

But I hope the traversal of _r_debug list
will not became the official solution, just
as well as using linkmap pointers directly.
So maybe you don't need such an allocator,
but of course I am not aware of a use-case
you are referring to.

> There should also be a way to obtain a stable handle from a link map
> pointer. Using dlopen with RTLD_NOLOAD seems about right for that.

So do you want this patch to get that working
right here right now:

--- a/elf/dl-open.c
+++ b/elf/dl-open.c
@@ -872,7 +872,8 @@ no more namespaces available for dlmopen()"));
                  DL_NNS is 1 and so any NSID != 0 is invalid.  */
               || DL_NNS == 1
               || GL(dl_ns)[nsid]._ns_nloaded == 0
-              || GL(dl_ns)[nsid]._ns_loaded->l_auditing))
+              || (GL(dl_ns)[nsid]._ns_loaded->l_auditing &&
+                                  !(mode & RTLD_NOLOAD))))
     _dl_signal_error (EINVAL, file, NULL,
                      N_("invalid target namespace in dlmopen()"));

I tested that patch and it works.
Test-suit shows no regressions.

Also what do you think about the aforementioned
void *dlaudit_load_module(const char *path, int flags);
void *dlopen_audit(const char *path, int flags, void *cookie);
extensions?

They are orthogonal to these namespace problems,
and in my case would allow to get rid of all those
_r_debug, fpic and lmpointer hacks, and get a good
use of the "cookie" audit functionality.
I can even code them up myself if there is an
interest in such functionality.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug dynamic-link/30127] [rfe]: enable ld audit at run-time

[fweimer at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=30127

--- Comment #12 from Florian Weimer <fweimer at redhat dot com> ---
(In reply to Stas Sergeev from comment #11)
> (In reply to Florian Weimer from comment #10)
> > I think for experimentation, you can use the link map as a dlopen handle
> > directly, and pass it to dlsym, for example.
> 
> That finally works, thank you.
> So with that and the patches already posted to ML,
> I can at least start prototyping some code.
> 
> 
> > We need to gather consensus that this is the way to do it, and document it.
> > Presently, it's an undocumented implementation detail.
> > 
> > We also need to use a dedicated link map allocator which only reuses, but
> > not frees, the underlying storiage, so that it becomes safe to traverse the
> > _r_debug lists, despite concurrent dlopen/dlcose operation.
> 
> But I hope the traversal of _r_debug list
> will not became the official solution, just
> as well as using linkmap pointers directly.

Why?  The handles are void *, so there isn't any type-checking today, and it's
hard to tell how many applications assume this undocumented equivalence.

Maybe we shouldn't commit to the other direction of reuse (casting a dlopen
handle to struct link_map *), that's a separate decision.

> So maybe you don't need such an allocator,
> but of course I am not aware of a use-case
> you are referring to.

There's no lock that synchronizes the traversal of _r_debug with concurrent
dlopen/dlclose. Introducing that is difficult. Such locks are easily misused,
and there is no precedent for exposing glibc locks in this way.

> > There should also be a way to obtain a stable handle from a link map
> > pointer. Using dlopen with RTLD_NOLOAD seems about right for that.
> 
> So do you want this patch to get that working
> right here right now:
> 
> --- a/elf/dl-open.c
> +++ b/elf/dl-open.c
> @@ -872,7 +872,8 @@ no more namespaces available for dlmopen()"));
>                   DL_NNS is 1 and so any NSID != 0 is invalid.  */
>                || DL_NNS == 1
>                || GL(dl_ns)[nsid]._ns_nloaded == 0
> -              || GL(dl_ns)[nsid]._ns_loaded->l_auditing))
> +              || (GL(dl_ns)[nsid]._ns_loaded->l_auditing &&
> +                                  !(mode & RTLD_NOLOAD))))
>      _dl_signal_error (EINVAL, file, NULL,
>                       N_("invalid target namespace in dlmopen()"));
>  
> I tested that patch and it works.
> Test-suit shows no regressions.

I think we need to do a pass first with pointer-comparison because the names
are not unique per namespace.

> Also what do you think about the aforementioned
> void *dlaudit_load_module(const char *path, int flags);
> void *dlopen_audit(const char *path, int flags, void *cookie);
> extensions?

They seem to have poor encapsulation. If more than one module uses them, the
auditor registered in that way might see unexpected auditing events, with
unrecognizable cookies.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug dynamic-link/30127] [rfe]: enable ld audit at run-time

[schwab@linux-m68k.org]

https://sourceware.org/bugzilla/show_bug.cgi?id=30127

--- Comment #13 from Andreas Schwab <schwab@linux-m68k.org> ---
My understanding is that _r_debug is a debugger-only interface which is only
supposed to be inspected while the world is stopped.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug dynamic-link/30127] [rfe]: enable ld audit at run-time

[stsp at users dot sourceforge.net]

https://sourceware.org/bugzilla/show_bug.cgi?id=30127

--- Comment #14 from Stas Sergeev <stsp at users dot sourceforge.net> ---
(In reply to Florian Weimer from comment #12)
> > But I hope the traversal of _r_debug list
> > will not became the official solution, just
> > as well as using linkmap pointers directly.
> 
> Why?  The handles are void *, so there isn't any type-checking today, and

For example getting a handle with dlopen()
can increment a refcount, making sure that
handle is not freed unexpectedly.

> > So maybe you don't need such an allocator,
> > but of course I am not aware of a use-case
> > you are referring to.
> 
> There's no lock that synchronizes the traversal of _r_debug with concurrent

Which is why I said I hope _r_debug is not
the final solution for me. :) I understand it has
multiple problems, like the one you mentioned
and more.

> dlopen/dlclose. Introducing that is difficult. Such locks are easily
> misused, and there is no precedent for exposing glibc locks in this way.

If you have a proper API for traversing, it
can use such a lock, or it can use more
advanced techniques to present the user with
the consistent list of objects that never
changes during traversal, even if someone
else does dlopen/dlclose in parallel.
Of course if its just an _r_debug and a manual
traversal, then there are much fewer options
to consider.

> I think we need to do a pass first with pointer-comparison because the names
> are not unique per namespace.

OK, as you wish, will not post a patch then,
although to me it looks like a bug-fix, not
even an extension. But it seems, after latest
"git pull" I have 32 test failures even w/o
any patches of mine, so its simpler for me
to not bother. :)
And I always had 1 test failure before the
latest "git pull" and never seen a full pass
yet (w/o any patches of mine).

> > Also what do you think about the aforementioned
> > void *dlaudit_load_module(const char *path, int flags);
> > void *dlopen_audit(const char *path, int flags, void *cookie);
> > extensions?
> 
> They seem to have poor encapsulation. If more than one module uses them, the
> auditor registered in that way might see unexpected auditing events, with
> unrecognizable cookies.

Good point!
Lets do 2 small amends then.
- void *dlaudit_load_module(const char *path, int flags);
  As above, no change.
- void *dlopen_audit(const char *path, int flags, void *audit_handle, void
*cookie2);

dlopen_audit() is changed: now the auditor handle is explicitly
passed, so this solib will only be audited by that auditor plus
all "legacy" ones (that are loaded by other means).
Also "cookie2" is not passed as a cookie to la_xxx() call-backs,
but instead put somewhere into the link_map struct. The designated
auditor will easily access that cookie2 and "legacy" auditors do
not care.
This seems to remove the problems you mentioned, doesn't it?

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug dynamic-link/30127] [rfe]: enable ld audit at run-time

[stsp at users dot sourceforge.net]

https://sourceware.org/bugzilla/show_bug.cgi?id=30127

--- Comment #15 from Stas Sergeev <stsp at users dot sourceforge.net> ---
(In reply to Andreas Schwab from comment #13)
> My understanding is that _r_debug is a debugger-only interface which is only
> supposed to be inspected while the world is stopped.

I think so too, so I am trying to propose
something else, to not use _r_debug...
Its good that its there so that I can
experiment and prototype, but as a final
solution... :(

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug dynamic-link/30127] [rfe]: enable ld audit at run-time

[fweimer at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=30127

--- Comment #16 from Florian Weimer <fweimer at redhat dot com> ---
(In reply to Andreas Schwab from comment #13)
> My understanding is that _r_debug is a debugger-only interface which is only
> supposed to be inspected while the world is stopped.

I'm not so sure about that. We declare the variable in <link.h>. Dyninst uses
it within the same process, for example.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug dynamic-link/30127] [rfe]: enable ld audit at run-time

[stsp at users dot sourceforge.net]

https://sourceware.org/bugzilla/show_bug.cgi?id=30127

--- Comment #17 from Stas Sergeev <stsp at users dot sourceforge.net> ---
I posted the patch set here:
https://sourceware.org/pipermail/libc-alpha/2023-February/145765.html

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug dynamic-link/30127] [rfe]: enable ld audit at run-time

[janderson at rice dot edu]

https://sourceware.org/bugzilla/show_bug.cgi?id=30127

Jonathon Anderson <janderson at rice dot edu> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |janderson at rice dot edu

--- Comment #18 from Jonathon Anderson <janderson at rice dot edu> ---
Hopping in as a fellow LD_AUDIT user.

If you control the application, there is already a viable way to communicate
between an auditor and the application with dlmopen and dlsym. The auditor
initiates by calling a function exposed by the application (compiled with
`-rdynamic`), e.g.:

void la_preinit(uintptr_t* cookie) {
  void* h = dlmopen(LM_ID_BASE, NULL, RTLD_LAZY | RTLD_NOLOAD);
  void (*hook)() = dlsym(h, "_auditor_hook");
  hook(...);
}

I'm a bit confused how the proposed interface would provide a better (or
alternate) solution to the problem. Echoing a few of Florian's concerns, if you
load an auditor after the application has been running for a time, you run a
large risk of being passed an unrecognizable cookie (since the la_objopen
callbacks are fired before the auditor is loaded). IMHO an auditor that can't
reliably implement la_symbind or la_objclose is not a very useful auditor. :)

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug dynamic-link/30127] [rfe]: enable ld audit at run-time

[stsp at users dot sourceforge.net]

https://sourceware.org/bugzilla/show_bug.cgi?id=30127

--- Comment #19 from Stas Sergeev <stsp at users dot sourceforge.net> ---
(In reply to Jonathon Anderson from comment #18)
> void la_preinit(uintptr_t* cookie) {
>   void* h = dlmopen(LM_ID_BASE, NULL, RTLD_LAZY | RTLD_NOLOAD);
>   void (*hook)() = dlsym(h, "_auditor_hook");
>   hook(...);
> }

Well, yes, auditor_hook is a viable
solution perhaps in some cases. Though
in my case the auditor is brought in with
a solib plugin, not with the main application.
And currently DT_AUDIT is not supported for
solibs, see #30134.
And even if #30134 is fixed, will you be
able to resolve the auditor hook with
LM_ID_BASE if it is in an RTLD_LOCAL solib
rather than in a main executable?
I think not, so the scope of your solution
is quite limited.

> I'm a bit confused how the proposed interface would provide a better (or
> alternate) solution to the problem.

By doing
void *auditor_handle = dlload_audit_module("my_module", 0);
void (*auditor_cb)(void) = dlsym(auditor_handle, "auditor_control_interface");
Just as simple as that. :)

> Echoing a few of Florian's concerns, if
> you load an auditor after the application has been running for a time, you
> run a large risk of being passed an unrecognizable cookie (since the
> la_objopen callbacks are fired before the auditor is loaded).

Good point, so I'll adjust the patch to
not call the auditor for the lib for which
he missed the initial cookie... which doesn't
look simple. :)
Oh, will think...
Thanks!

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug dynamic-link/30127] [rfe]: enable ld audit at run-time

[stsp at users dot sourceforge.net]

https://sourceware.org/bugzilla/show_bug.cgi?id=30127

--- Comment #20 from Stas Sergeev <stsp at users dot sourceforge.net> ---
(In reply to Jonathon Anderson from comment #18)
> run a large risk of being passed an unrecognizable cookie (since the
> la_objopen callbacks are fired before the auditor is loaded). IMHO an
> auditor that can't reliably implement la_symbind or la_objclose is not a
> very useful auditor. :)

I improved the test to catch all
unrecognized cookies, and found that
this bug is pre-existing: la_objclose()
is called for ld-linux*.so upon dlclose()
of every other solib, but la_objopen()
is not called for it - this is even without
my patch.
Will fix by preventing the audit calls
for ld.so.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug dynamic-link/30127] [rfe]: enable ld audit at run-time

[janderson at rice dot edu]

https://sourceware.org/bugzilla/show_bug.cgi?id=30127

--- Comment #21 from Jonathon Anderson <janderson at rice dot edu> ---
(In reply to Stas Sergeev from comment #19)
> (In reply to Jonathon Anderson from comment #18)
> > void la_preinit(uintptr_t* cookie) {
> >   void* h = dlmopen(LM_ID_BASE, NULL, RTLD_LAZY | RTLD_NOLOAD);
> >   void (*hook)() = dlsym(h, "_auditor_hook");
> >   hook(...);
> > }
> 
> Well, yes, auditor_hook is a viable
> solution perhaps in some cases. Though
> in my case the auditor is brought in with
> a solib plugin, not with the main application.
> And currently DT_AUDIT is not supported for
> solibs, see #30134.
> And even if #30134 is fixed, will you be
> able to resolve the auditor hook with
> LM_ID_BASE if it is in an RTLD_LOCAL solib
> rather than in a main executable?
> I think not, so the scope of your solution
> is quite limited.

Assuming you can get an auditor into the process (LD_AUDIT env var), if you can
identify the solib from it's loaded path, you can (recursively) load the solib
during the la_objopen callback and access the auditor hook that way:

unsigned int la_objopen(struct link_map* m, Lmid_t lmid, uintptr_t* cookie) {
  if(m->l_name /* ... is the target solib */) {
    void* h = dlmopen(lmid, m->l_name, RTLD_LAZY | RTLD_NOLOAD);
    void (*hook)() = dlsym(h, "_auditor_hook");
    hook();
  }
  return 0;
}

Like most of LD_AUDIT, this isn't a well-tested code path, so be careful and
expect some bugs. For example, don't try this during the later
la_activity(LA_ACT_CONSISTENT) callback, you'll get a glorious crash. :)

> 
> > I'm a bit confused how the proposed interface would provide a better (or
> > alternate) solution to the problem.
> 
> By doing
> void *auditor_handle = dlload_audit_module("my_module", 0);
> void (*auditor_cb)(void) = dlsym(auditor_handle,
> "auditor_control_interface");
> Just as simple as that. :)
> 
> > Echoing a few of Florian's concerns, if
> > you load an auditor after the application has been running for a time, you
> > run a large risk of being passed an unrecognizable cookie (since the
> > la_objopen callbacks are fired before the auditor is loaded).
> 
> Good point, so I'll adjust the patch to
> not call the auditor for the lib for which
> he missed the initial cookie... which doesn't
> look simple. :)
> Oh, will think...
> Thanks!

This still doesn't solve the issue. Expanding a bit: say you load
dlload_audit_module("my_auditor"), then dlopen("libmylib.so") and libmylib.so
binds malloc from libc.so. my_auditor gets an la_objopen call for libmylib.so
but not libc.so. What do you do about the la_symbind callback?:
  - If you send an la_symbind to my_auditor, the defcook will be an
unrecognized cookie.
  - If you skip the call, my_auditor won't be able to intercept malloc (and all
other libc functions). 
  - If you NULL the defcook, my_auditor won't be able to wrap malloc etc. since
it can't (reliably) get the "original" function it would bind to. And it's an
API break, so you need to bump LAV_CURRENT.

There are use cases that don't need la_symbind, e.g. an auditor that records
used libraries or denies access to libraries outside of a "whitelist." But I
also think most (if not all) of those use cases would operate far better as a
global auditor (LD_AUDIT) rather than one dynamically loaded late in the
application. Especially for any security-related purpose.

What is your proposed auditor supposed to do?

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug dynamic-link/30127] [rfe]: enable ld audit at run-time

[stsp at users dot sourceforge.net]

https://sourceware.org/bugzilla/show_bug.cgi?id=30127

--- Comment #22 from Stas Sergeev <stsp at users dot sourceforge.net> ---
(In reply to Jonathon Anderson from comment #21)
> Assuming you can get an auditor into the process (LD_AUDIT env var), if you
> can identify the solib from it's loaded path, you can (recursively) load the
> solib during the la_objopen callback and access the auditor hook that way:

Yes, in case the auditor is global,
you can find a trick to access the
hook from solib.
But that still means you can't make
an auditor a part of your plug-in.
Though with that technique and DT_AUDIT
implemented for dlopen(), you can get
things working perhaps. But DT_AUDIT
for dlopen() is probably a very distant
future, if at all.


> This still doesn't solve the issue. Expanding a bit: say you load
> dlload_audit_module("my_auditor"), then dlopen("libmylib.so") and
> libmylib.so binds malloc from libc.so. my_auditor gets an la_objopen call
> for libmylib.so but not libc.so. What do you do about the la_symbind
> callback?:
>   - If you send an la_symbind to my_auditor, the defcook will be an
> unrecognized cookie.
>   - If you skip the call, my_auditor won't be able to intercept malloc (and
> all other libc functions). 
>   - If you NULL the defcook, my_auditor won't be able to wrap malloc etc.
> since it can't (reliably) get the "original" function it would bind to. And
> it's an API break, so you need to bump LAV_CURRENT.

In that case I think it would make
sense to disable symbind and plt call-backs
for such dynamic auditors?

> What is your proposed auditor supposed to do?

It supposed to use these audit extensions
https://sourceware.org/pipermail/libc-alpha/2023-February/145638.html
and apply other solib loading customizations.
But its not going to do anything after the
load process, i.e. I don't need symbind/pltenter.
I guess it would be fair to let the app-wide
auditors to work with all libs, and have the
dynamic auditors without any access to the
"outer" libs?

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug dynamic-link/30127] [rfe]: enable ld audit at run-time

[stsp at users dot sourceforge.net]

https://sourceware.org/bugzilla/show_bug.cgi?id=30127

--- Comment #23 from Stas Sergeev <stsp at users dot sourceforge.net> ---
I don't understand why in ML archives
I only see the cover-letter of my patch
series:
https://sourceware.org/pipermail/libc-alpha/2023-February/145872.html
https://sourceware.org/pipermail/libc-alpha/2023-February/145827.html
The patches themselves are not there.
Why can that happen?
I posted a new patch that addresses the
comments above, but I can't even provide
the link to it.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug dynamic-link/30127] [rfe]: enable ld audit at run-time

[stsp at users dot sourceforge.net]

https://sourceware.org/bugzilla/show_bug.cgi?id=30127

--- Comment #24 from Stas Sergeev <stsp at users dot sourceforge.net> ---
So is this a problem with ML or an
intentional blocking? I don't understand
what's going on. If its a block then
perhaps you forgot to tell me about it?

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug dynamic-link/30127] [rfe]: enable ld audit at run-time

[carlos at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=30127

Carlos O'Donell <carlos at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |carlos at redhat dot com

--- Comment #25 from Carlos O'Donell <carlos at redhat dot com> ---
(In reply to Stas Sergeev from comment #24)
> So is this a problem with ML or an
> intentional blocking? I don't understand
> what's going on. If its a block then
> perhaps you forgot to tell me about it?

There is no intentional blocking that I am aware of, and I have the full patch
series in *my* mailbox.

You are correct and I can confirm that they do not appear in the pipermail
archives.

They *do* appear in the public-inbox archive:
https://inbox.sourceware.org/libc-alpha/20230224165400.3417104-1-stsp2@yandex.ru/T/#t

So this looks like a pipermail issue. Please submit an infrastructure bug using
this bugzilla instance, or email overseers@sourceware.org with the details.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug dynamic-link/30127] [rfe]: enable ld audit at run-time

[stsp at users dot sourceforge.net]

https://sourceware.org/bugzilla/show_bug.cgi?id=30127

--- Comment #26 from Stas Sergeev <stsp at users dot sourceforge.net> ---
Thanks, filled #30172.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug dynamic-link/30127] [rfe]: enable ld audit at run-time

[janderson at rice dot edu]

https://sourceware.org/bugzilla/show_bug.cgi?id=30127

--- Comment #27 from Jonathon Anderson <janderson at rice dot edu> ---
(In reply to Stas Sergeev from comment #22)
> > This still doesn't solve the issue. Expanding a bit: say you load
> > dlload_audit_module("my_auditor"), then dlopen("libmylib.so") and
> > libmylib.so binds malloc from libc.so. my_auditor gets an la_objopen call
> > for libmylib.so but not libc.so. What do you do about the la_symbind
> > callback?:
> >   - If you send an la_symbind to my_auditor, the defcook will be an
> > unrecognized cookie.
> >   - If you skip the call, my_auditor won't be able to intercept malloc (and
> > all other libc functions). 
> >   - If you NULL the defcook, my_auditor won't be able to wrap malloc etc.
> > since it can't (reliably) get the "original" function it would bind to. And
> > it's an API break, so you need to bump LAV_CURRENT.
> 
> In that case I think it would make
> sense to disable symbind and plt call-backs
> for such dynamic auditors?

There *will* be auditors that will not function properly under these
restrictions. At the very least there needs to be some kind of flag presented
to the auditor at la_version-time to let it know that it has been loaded in
this (very) degraded state, so the auditor has a chance to refuse to load.

> 
> > What is your proposed auditor supposed to do?
> 
> It supposed to use these audit extensions
> https://sourceware.org/pipermail/libc-alpha/2023-February/145638.html
> and apply other solib loading customizations.
> But its not going to do anything after the
> load process, i.e. I don't need symbind/pltenter.
> I guess it would be fair to let the app-wide
> auditors to work with all libs, and have the
> dynamic auditors without any access to the
> "outer" libs?

So, IIUC you have an solib plugin (i.e. you don't control the application)
where you want/need to customize (a) the location the solib is loaded (or where
what the solib loads is loaded?) with la_premap and (b) the paths that your
plugin loads with la_objsearch.

Solibs are "position-independent" and should work no matter where they are
loaded (unless you're doing *really* dangerous and non-portable tricks with
function pointers). So (a) can only be a performance optimization, maybe
improving I-cache locality? IMHO it makes more sense to apply such an
optimization across the entire application. Or is there a reason why this
should only apply to your plugin?

(b) gives me bad vibes, it seems very easy to run into a situation where
dlopen->dlclose->dlopen will unexpectedly give you different binaries between
the two dlopen calls. la_objsearch also doesn't affect binaries that are
already loaded, e.g. if an earlier plugin loads a library you wanted to replace
you will have lost the chance to do so (and you won't even get a callback
warning that this is happening). IMHO it seems delicate unless applied
consistently across the entire application. Is there a reason you can't
manually adjust the paths your solib loads? Are the wanted paths dynamically
generated (like e.g. Spindle[1])?

[1]: https://computing.llnl.gov/projects/spindle

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug dynamic-link/30127] [rfe]: enable ld audit at run-time

[stsp at users dot sourceforge.net]

https://sourceware.org/bugzilla/show_bug.cgi?id=30127

--- Comment #28 from Stas Sergeev <stsp at users dot sourceforge.net> ---
(In reply to Jonathon Anderson from comment #27)
> There *will* be auditors that will not function properly under these
> restrictions.

Where would they come from and how could
they became loaded by this API?


> At the very least there needs to be some kind of flag
> presented to the auditor at la_version-time to let it know that it has been
> loaded in this (very) degraded state, so the auditor has a chance to refuse
> to load.

Hmm, not really sure there are flags in
la_version(). But I can instead disallow
loading of such auditor and return an error.


> So, IIUC you have an solib plugin (i.e. you don't control the application)
> where you want/need to customize (a) the location the solib is loaded (or
> where what the solib loads is loaded?) with la_premap and (b) the paths that
> your plugin loads with la_objsearch.

a - yes
b - no

> Solibs are "position-independent" and should work no matter where they are
> loaded (unless you're doing *really* dangerous and non-portable tricks with
> function pointers).

I need that to make the 64bit solib to
cooperate with 32bit VM. The memory of
that VM is mapped to 64bit space. I need
to map the lib to that window, but it should
be relocated as if it is in the first 4Gb.
For that I need 2 extensions instead of just 1.
They are in that patch-set. But I am going
to split it into multiple patches for easier
understanding by reviewers. As currently no
one understands it, but this is simply because
the patch is too big.

> (b) gives me bad vibes,

Same here, I am not planning b.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug dynamic-link/30127] [rfe]: enable ld audit at run-time

[stsp at users dot sourceforge.net]

https://sourceware.org/bugzilla/show_bug.cgi?id=30127

--- Comment #29 from Stas Sergeev <stsp at users dot sourceforge.net> ---
Also please see LD_PREFER_MAP_32BIT_EXE.
It was removed in ea5814467a02 and that's
what I am trying to resurrect. But in a
much, much more powerful way. I wouldn't
be using LD_PREFER_MAP_32BIT_EXE, but its
for the same thing.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug dynamic-link/30127] [rfe]: enable ld audit at run-time

[stsp at users dot sourceforge.net]

https://sourceware.org/bugzilla/show_bug.cgi?id=30127

--- Comment #30 from Stas Sergeev <stsp at users dot sourceforge.net> ---
Hi Jonathon, so if you are satisfied with the current
solution (return EINVAL for disallowed audit call-backs),
then maybe you can give me an ACK? :)
I am not aware of any problems or a pending review
comments for the last (v6) version of the patch.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug dynamic-link/30127] [rfe]: enable ld audit at run-time

[janderson at rice dot edu]

https://sourceware.org/bugzilla/show_bug.cgi?id=30127

--- Comment #31 from Jonathon Anderson <janderson at rice dot edu> ---
(In reply to Stas Sergeev from comment #30)
> Hi Jonathon, so if you are satisfied with the current
> solution (return EINVAL for disallowed audit call-backs),
> then maybe you can give me an ACK? :)
> I am not aware of any problems or a pending review
> comments for the last (v6) version of the patch.

Busy week for me, expect more slow responses. :)

I can't say the current solutions are appealing. You've made it very difficult
for a dynamically-loaded auditor to accomplish any auditor-like task, outside
of a hypothetical callback (la_premap) that hasn't been mainstreamed yet. On
top of that, I'm still a bit leery of letting the application observe the
auditors at all (outside of _r_debug).

Before your patches, there are parts of your use case that I am not convinced
do not have alternative solutions. One doesn't hack Glibc before exploring all
other alternatives, so I assume you've tried these but I'd like the context to
know why they don't work:

You mentioned in another ticket
(https://sourceware.org/bugzilla/show_bug.cgi?id=30007#c2) that the whole point
of this is to access data objects in the solib. Do you write this solib? If you
do, can you replace these static objects with pointers that are mmap'd into the
right location (e.g. initialized in an init() function or init constructors)?
If you don't, these are data structures that (a) part of the public API for the
solib and (b) are certainly identical (alignment padding included) between the
64-bit and 32-bit ABI?

Could you override mmap itself for the files you want mapped in the right
locations? If LD_PRELOAD won't fly, newer kernels have syscall user dispatch
[1] for doing this at the syscall level.

You mentioned DT_AUDIT wouldn't work, so I presume the executable is out of
your control, e.g. you're extending a VM with an solib plugin? If so, do you
not have the access to set LD_AUDIT in the environment before launching the
executable? You can always make your auditor "no-op" except when needed.

[1]: https://docs.kernel.org/admin-guide/syscall-user-dispatch.html

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug dynamic-link/30127] [rfe]: enable ld audit at run-time

[stsp at users dot sourceforge.net]

https://sourceware.org/bugzilla/show_bug.cgi?id=30127

--- Comment #32 from Stas Sergeev <stsp at users dot sourceforge.net> ---
(In reply to Jonathon Anderson from comment #31)
> I can't say the current solutions are appealing. You've made it very
> difficult for a dynamically-loaded auditor to accomplish any auditor-like

Posted v7 which just calls la_objopen()
for pre-existing libs, similar to dl_main().

> You mentioned in another ticket
> (https://sourceware.org/bugzilla/show_bug.cgi?id=30007#c2) that the whole
> point of this is to access data objects in the solib. Do you write this
> solib?

I write a shim that should load (via dlmem()) other
libs, built from some 30 years old sources (that neither
me nor anyone else writes today), and allows them to work
on a current system. I.e. this is a compatibility layer.
Compatibility layer, unfortunately, can't require changing
the legacy code. What you propose, will require mostly a
full rewrites, eg I have no idea how would the one identify
all pointers to all vars and change all such vars to a heap
objects...

> Could you override mmap itself for the files you want mapped in the right
> locations?

But it won't be relocated to that address.

> You mentioned DT_AUDIT wouldn't work, so I presume the executable is out of
> your control, e.g. you're extending a VM with an solib plugin? If so, do you
> not have the access to set LD_AUDIT in the environment before launching the
> executable? You can always make your auditor "no-op" except when needed.

This is possible but a quite clumsy solution IMO.
And of course it would still need all other patches
(dlmem etc) except this one. So I'd rather concentrate
on making dlload_audit_module() suitable for your needs.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug dynamic-link/30127] [rfe]: enable ld audit at run-time

[janderson at rice dot edu]

https://sourceware.org/bugzilla/show_bug.cgi?id=30127

--- Comment #33 from Jonathon Anderson <janderson at rice dot edu> ---
(In reply to Stas Sergeev from comment #32)
> (In reply to Jonathon Anderson from comment #31)
> > I can't say the current solutions are appealing. You've made it very
> > difficult for a dynamically-loaded auditor to accomplish any auditor-like
> 
> Posted v7 which just calls la_objopen()
> for pre-existing libs, similar to dl_main().

So, my group currently assumes la_objopen() happens before the init
constructors of the referenced objects. It turns out to be an important
callback for registering callbacks in runtime libraries before transferring
control back to the application. Is there a way to indicate to the auditor that
it's "missed the boat" so auditors that need this timing can recover
gracefully?

> 
> > You mentioned in another ticket
> > (https://sourceware.org/bugzilla/show_bug.cgi?id=30007#c2) that the whole
> > point of this is to access data objects in the solib. Do you write this
> > solib?
> 
> I write a shim that should load (via dlmem()) other
> libs, built from some 30 years old sources (that neither
> me nor anyone else writes today), and allows them to work
> on a current system. I.e. this is a compatibility layer.
> Compatibility layer, unfortunately, can't require changing
> the legacy code. What you propose, will require mostly a
> full rewrites, eg I have no idea how would the one identify
> all pointers to all vars and change all such vars to a heap
> objects...

Why would you have to lift the variables in the legacy code? Those are already
"inside" the VM, right?

What it sounds like you're trying to do is pass a pointer to the data segment
of a 64-bit library "outside" the VM to the 32-bit legacy code running "inside"
the VM. Which it will then dereference to access or alter some variable shared
"across the VM gap." Does this match your use case?

If you control this 64-bit library, you should be able to lift the shared
variables to pointers. Just allocate them "inside" the VM's more restrictive
address space and reference from "outside." In this case Glibc doesn't even get
involved.

The only reason you would require dlmem() or la_premap is if you didn't control
the 64-bit library who's data segment needs to be shared "across the gap." Is
this the case?

> 
> > Could you override mmap itself for the files you want mapped in the right
> > locations?
> 
> But it won't be relocated to that address.

The result of the first call to mmap() for an solib decides the base address
for the solib, which is the address all relocations are relative to. So it
should relocate to the right address if your mmap wrapper is sane. See
_dl_map_segments (esp. lines 85-95 and 106) and _dl_map_segment.

There's a performance penalty to syscall wrapping (and it's arch-specific and
ugly), but AFAICT it should work just fine for this case.

> 
> > You mentioned DT_AUDIT wouldn't work, so I presume the executable is out of
> > your control, e.g. you're extending a VM with an solib plugin? If so, do you
> > not have the access to set LD_AUDIT in the environment before launching the
> > executable? You can always make your auditor "no-op" except when needed.
> 
> This is possible but a quite clumsy solution IMO.
> And of course it would still need all other patches
> (dlmem etc) except this one. So I'd rather concentrate
> on making dlload_audit_module() suitable for your needs.

This solution matches well with the (few) use cases auditors have been tested
in so far. If you want something that *works* (reliably) for this rather
specific use case, a little clumsiness is the least of your worries IMHO.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug dynamic-link/30127] [rfe]: enable ld audit at run-time

[stsp at users dot sourceforge.net]

https://sourceware.org/bugzilla/show_bug.cgi?id=30127

--- Comment #34 from Stas Sergeev <stsp at users dot sourceforge.net> ---
(In reply to Jonathon Anderson from comment #33)
> So, my group currently assumes la_objopen() happens before the init
> constructors of the referenced objects. It turns out to be an important
> callback for registering callbacks in runtime libraries before transferring
> control back to the application. Is there a way to indicate to the auditor
> that it's "missed the boat" so auditors that need this timing can recover
> gracefully?

Sure!
The sale point of that patch is to allow
more convenient ways of communicating with
an auditor. So of course there is such a way!
In v9. :)


> Why would you have to lift the variables in the legacy code? Those are
> already "inside" the VM, right?

No: VM is a part of compatibility layer.
It allows those solibs built from ancient
non-unix code, to work. And shim is gluing
these 2 components together.
So if solib calls something, shim forwards
the call to VM, but the call args must be
all accessible to VM. So the libs are mapped
to the 4Gb window where VM can see them, and
that 4Gb window is not in the lower 4Gb, but
rather somewhere in a 64bit space.


> What it sounds like you're trying to do is pass a pointer to the data
> segment of a 64-bit library "outside" the VM to the 32-bit legacy code
> running "inside" the VM. Which it will then dereference to access or alter
> some variable shared "across the VM gap." Does this match your use case?

Indeed.


> If you control this 64-bit library, you should be able to lift the shared
> variables to pointers. Just allocate them "inside" the VM's more restrictive
> address space and reference from "outside." In this case Glibc doesn't even
> get involved.

I still don't see how can this be done
w/o rewriting the libs from scratches -
the ones I do not control, I only have
the control over the shim lib.
Now suppose I do have a control over the
actual solib (actually not, but if), and
then what? It can do thousands of calls,
any of them can have pointers to its own
local or global data, variables, whatever.
I do not control these libs (the compatibility
layer is the final product, libs are on
user's side), but even if I do, I would
have to walk through all its intercepted
calls and convert all referenced objects
to a heap objects - heap objects can indeed
be allocated on the VMs side, and in fact,
the heap objects _will_ be allocated on a
VMs side. But I can't estimate the work of
converting all addressable objects to heap
objects, even if I could control all those
solibs.


> The only reason you would require dlmem() or la_premap is if you didn't
> control the 64-bit library who's data segment needs to be shared "across the
> gap." Is this the case?

Yes, it is the case.
I only control shim lib.
But I don't believe even the "otherwise" would
help much. Compatibility layer that requires
you to rewrite everything, is not of a big use.


> The result of the first call to mmap() for an solib decides the base address
> for the solib, which is the address all relocations are relative to.

Its indeed the nice trick, but the problem
is that I need a "disparity" between the
relocated and mapped address.
Relocated address is in MAP_32BIT, but mapped
address = relocated_address+VM_window_start.
Then VM sees the solib in its relocated address,
which is the goal.


> There's a performance penalty to syscall wrapping (and it's arch-specific
> and ugly), but AFAICT it should work just fine for this case.

I don't think the proposed scheme can be
replicated by any tricks and hacks. And
honestly I don't know why it should.
There was that LD_PREFER_MAP_32BIT_EXE but
it was dumb, so removed. Why not to replace
it with something small and smart? Esp given
that there is a demand in a functionality of
specifying the relocation address, it can be
found on stackoverflow.
There were multiple requests for dlmem(), too,
both here and on steckoverflow. There are even
some sketches on github from various people.
And finally you can also have that feature of
"mmapped_addr!=relocated_addr" - all by the
price of 1. :)
Why even try to replicate that with tricks
and hacks?
But I think its impossible w/o my patches,
anyway.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug dynamic-link/30127] [rfe]: enable ld audit at run-time

[stsp at users dot sourceforge.net]

https://sourceware.org/bugzilla/show_bug.cgi?id=30127

--- Comment #35 from Stas Sergeev <stsp at users dot sourceforge.net> ---
(In reply to Jonathon Anderson from comment #33)
> So, my group currently assumes la_objopen() happens before the init
> constructors of the referenced objects. It turns out to be an important
> callback for registering callbacks in runtime libraries before transferring
> control back to the application. Is there a way to indicate to the auditor
> that it's "missed the boat" so auditors that need this timing can recover
> gracefully?

So while I added what you asked for,
let me wonder why "missed the boat"?
It seems gcc allows to specify the
constructor priority. So why not to
create the "first" constructor for
loading audit modules, and actually
start trying my patch out in your
use-case? :) I think it accumulated
everything to aid your scenario.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug dynamic-link/30127] [rfe]: enable ld audit at run-time

[janderson at rice dot edu]

https://sourceware.org/bugzilla/show_bug.cgi?id=30127

--- Comment #36 from Jonathon Anderson <janderson at rice dot edu> ---
(In reply to Stas Sergeev from comment #34)
> (In reply to Jonathon Anderson from comment #33)
> > So, my group currently assumes la_objopen() happens before the init
> > constructors of the referenced objects. It turns out to be an important
> > callback for registering callbacks in runtime libraries before transferring
> > control back to the application. Is there a way to indicate to the auditor
> > that it's "missed the boat" so auditors that need this timing can recover
> > gracefully?
> 
> Sure!
> The sale point of that patch is to allow
> more convenient ways of communicating with
> an auditor. So of course there is such a way!
> In v9. :)

I must not be seeing this feature in the patch. Can you guide me a bit? I see
la_dynload which indicates when an auditor is dynamically loaded, but AFAICT
there isn't a way to tell which la_objopen callbacks are "real" (new objects)
or "fake" (objects already loaded).

I probably should have been clearer in my request, sorry.

(In reply to Stas Sergeev from comment #35)
> So while I added what you asked for,
> let me wonder why "missed the boat"?
> It seems gcc allows to specify the
> constructor priority. So why not to
> create the "first" constructor for
> loading audit modules,

How does that help? Constructor priority means nothing after the solib or
executable is linked. GCC/ld uses the priorities to sort the list of _init
functions within a single object. ld.so decides on the order objects run their
constructors based on dependencies between objects and nothing else.

(In reply to Stas Sergeev from comment #34)
> > Why would you have to lift the variables in the legacy code? Those are
> > already "inside" the VM, right?
> 
> No: VM is a part of compatibility layer.
> It allows those solibs built from ancient
> non-unix code, to work. And shim is gluing
> these 2 components together.
> So if solib calls something, shim forwards
> the call to VM,

So, let me just make sure I'm on the same page here: you have non-Unix (or even
pre-Unix?) code that's been compiled into 64-bit solibs. And you have
compatibility libraries to implement the non-Unix bits, but they're 32-bit only
and so need to run in a VM. So to get everything working, you need those two
kinds of libraries calling each other despite the complete ABI and ISA
difference, in a single process.

I assume there's a Very Good Reason (TM) the ancient code can't be compiled
32-bit (and run in the VM) nor the compatibility libraries compiled 64-bit (and
run without a VM). Otherwise you wouldn't need any of this.

I'm not confident this description is right based on your comments later... but
dang, this is quite a unique problem.

> but the call args must be
> all accessible to VM. So the libs are mapped
> to the 4Gb window where VM can see them, and
> that 4Gb window is not in the lower 4Gb, but
> rather somewhere in a 64bit space.

And the call args can contain pointers, thus the need to ensure the pointers
are in range. You can override malloc and new to (mostly) ensure heap
allocations are in the right region. The dlmem/la_premap patches(es) ensure
pointers to data segments will work.

What do you do about pointers to stack variables?

> > If you control this 64-bit library, you should be able to lift the shared
> > variables to pointers. Just allocate them "inside" the VM's more restrictive
> > address space and reference from "outside." In this case Glibc doesn't even
> > get involved.
> 
> I still don't see how can this be done
> w/o rewriting the libs from scratches -
> the ones I do not control, I only have
> the control over the shim lib.

I'm beginning to understand this. If so, the question is indeed moot.

> 
> 
> > The result of the first call to mmap() for an solib decides the base address
> > for the solib, which is the address all relocations are relative to.
> 
> Its indeed the nice trick, but the problem
> is that I need a "disparity" between the
> relocated and mapped address.
> Relocated address is in MAP_32BIT, but mapped
> address = relocated_address+VM_window_start.
> Then VM sees the solib in its relocated address,
> which is the goal.

That... makes absolutely no sense. In that arrangement the loaded solib will be
useless outside the VM, the relocations are wrong for the 64-bit address space.
So why does the 64-bit ld.so need to load it? Can't you load it inside the VM
using the 32-bit ld.so, since that's the only place it could possibly work?

This seems to contradict your prior explanation too. Doing this will make a
64-bit solib useless, the relocations are wrong in 64-bit address space and the
32-bit VM can't run the code (wrong ISA). If you don't fully control the 64-bit
solib you're doing this torture to, there's no way you can expect this to work
at all.

> 
> 
> > There's a performance penalty to syscall wrapping (and it's arch-specific
> > and ugly), but AFAICT it should work just fine for this case.
> 
> I don't think the proposed scheme can be
> replicated by any tricks and hacks. And
> honestly I don't know why it should.
> There was that LD_PREFER_MAP_32BIT_EXE but
> it was dumb, so removed. Why not to replace
> it with something small and smart?

IMHO this is well outside the kinds of problems where a "small and smart"
solution is a tractable goal. After all, we're talking about multiple
architectures in a single process image. :P

> Esp given
> that there is a demand in a functionality of
> specifying the relocation address, it can be
> found on stackoverflow.
> There were multiple requests for dlmem(), too,
> both here and on steckoverflow. There are even
> some sketches on github from various people.

Mind providing a couple good sources for what you're referring to here?

I didn't get any relevant hits for "dlmem" on these sites or on Google.

> And finally you can also have that feature of
> "mmapped_addr!=relocated_addr" - all by the
> price of 1. :)

That's the part that makes no sense to me. Why would ld.so EVER load an solib
in an arrangement it won't be able to use? It can't even call the solib's init
constructors!

Which of course makes me question the intended stability of your "small and
smart" patches. There are plenty of ways to crash an application, let's not add
more, shall we? ;)

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug dynamic-link/30127] [rfe]: enable ld audit at run-time

[stsp at users dot sourceforge.net]

https://sourceware.org/bugzilla/show_bug.cgi?id=30127

--- Comment #37 from Stas Sergeev <stsp at users dot sourceforge.net> ---
(In reply to Jonathon Anderson from comment #36)
> (In reply to Stas Sergeev from comment #34)
> > (In reply to Jonathon Anderson from comment #33)
> > > So, my group currently assumes la_objopen() happens before the init
> > > constructors of the referenced objects. It turns out to be an important
> > > callback for registering callbacks in runtime libraries before transferring
> > > control back to the application. Is there a way to indicate to the auditor
> > > that it's "missed the boat" so auditors that need this timing can recover
> > > gracefully?
> > 
> > Sure!
> > The sale point of that patch is to allow
> > more convenient ways of communicating with
> > an auditor. So of course there is such a way!
> > In v9. :)
> I must not be seeing this feature in the patch. Can you guide me a bit? I
> see la_dynload which indicates when an auditor is dynamically loaded, but

Well, v9 only addresses this:
"Is there a way to indicate to the auditor that it's \"missed the boat\""
by allowing you to tell anything you want to
an auditor.

> AFAICT there isn't a way to tell which la_objopen callbacks are "real" (new
> objects) or "fake" (objects already loaded).

There is no such thing simply because the
"late" objopen call is already an established
practice in dl_main(). Which means its a bit
out of the scope of this patch, but it doesn't
mean I can't add such a thing. For example I
can add la_objopen_late(). But let's see if it
is actually needed. For example can't you use
the fact that la_objsearch() and/or
la_activity(LA_ACT_ADD) wasn't called before
la_objopen()?


> How does that help? Constructor priority means nothing after the solib or
> executable is linked. GCC/ld uses the priorities to sort the list of _init
> functions within a single object. ld.so decides on the order objects run
> their constructors based on dependencies between objects and nothing else.

Yes, so you need to add the "primary" ctor
to the main object (not to any solib of it)
and from that load the audit module, no?


> So, let me just make sure I'm on the same page here: you have non-Unix (or
> even pre-Unix?) code that's been compiled into 64-bit solibs. And you have
> compatibility libraries to implement the non-Unix bits, but they're 32-bit
> only and so need to run in a VM. So to get everything working, you need
> those two kinds of libraries calling each other despite the complete ABI and
> ISA difference, in a single process.

Exactly.


> I assume there's a Very Good Reason (TM) the ancient code can't be compiled
> 32-bit (and run in the VM) nor the compatibility libraries compiled 64-bit
> (and run without a VM). Otherwise you wouldn't need any of this.

No, this is not the case. :)
I mean, the reason may be not Very Good (TM)
for your ears, but its just that it would need
the pretty much outdated tool-chains that no
one has these days. Also the ability to work
on host in 64bit mode provides performance and
other benefits. VM is much, much less involved
in that scenario, than when you run everything
inside VM. In theory, this will also allow to
extend the old code with some functionality
available on host, but that's not something to
anticipate at that early stage of a development.


> I'm not confident this description is right based on your comments later...

But actually your description is very relevant!


> And the call args can contain pointers, thus the need to ensure the pointers
> are in range. You can override malloc and new to (mostly) ensure heap
> allocations are in the right region. The dlmem/la_premap patches(es) ensure
> pointers to data segments will work.

Indeed!


> What do you do about pointers to stack variables?

I can set up the initial stack from my
shim lib, and if an app wants to switch
stacks (which is very unlikely), it will
have to use the heap space, which is under
my control.


> That... makes absolutely no sense. In that arrangement the loaded solib will
> be useless outside the VM, the relocations are wrong for the 64-bit address
> space.

la_premap_dlmem() is a very powerful extension. :)
It allows you to map the lib anywhere you
want, any amount of times in a row. You can
map it to VM window _in addition_ to also
mapping it to its relocation point. That's
what even the test-case of mine demonstrates
by doing 2 mappings of the same lib. And let
me assert its very small for such a powerful
functionality, because all it does is to pass
the backing-store fd to dlmem. With that fd
you can do whatever you want. So no lots of
new and convoluted APIs, whereas the solved
task is quite big. For the price of just 1
small dlmem() call!


> So why does the 64-bit ld.so need to load it? Can't you load it
> inside the VM using the 32-bit ld.so, since that's the only place it could
> possibly work?

It should work in both "worlds" because it has
the regular DT_NEEDED deps linked with its
own custom libc. So the host relocations should
work properly, and only the shim lib translates
the calls into VM.


> > I don't think the proposed scheme can be
> > replicated by any tricks and hacks. And
> > honestly I don't know why it should.
> > There was that LD_PREFER_MAP_32BIT_EXE but
> > it was dumb, so removed. Why not to replace
> > it with something small and smart?
> 
> IMHO this is well outside the kinds of problems where a "small and smart"
> solution is a tractable goal. After all, we're talking about multiple
> architectures in a single process image. :P

I think LD_PREFER_MAP_32BIT_EXE was also
about something like this. So my demand is
definitely not the first one.
But let's look at it from another side:
all I do is specify the reloc address and
pass a backing-store fd. No matter what
arch problems does this solve, its simple
by itself and only adds 1 API func that people
requested many times. I am sure next time
you'll deal with dozens of APIs to solve only
some small part of the same problem. :) My
impl is exceptionally minimalistic, I am quite
confident in that.


> Mind providing a couple good sources for what you're referring to here?

Will do in an hour or so.
Give me some time to collect the links for you.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug dynamic-link/30127] [rfe]: enable ld audit at run-time

[stsp at users dot sourceforge.net]

https://sourceware.org/bugzilla/show_bug.cgi?id=30127

--- Comment #38 from Stas Sergeev <stsp at users dot sourceforge.net> ---
So dlmem was requested:
https://sourceware.org/bugzilla/show_bug.cgi?id=11767
https://stackoverflow.com/questions/5053664/dlopen-from-memory
https://stackoverflow.com/questions/29637464/need-an-api-that-allows-me-to-load-a-shared-object-from-a-memory-address

And even implemented:
https://github.com/m1m1x/memdlopen

Mapping to specific address was requested:
https://stackoverflow.com/questions/62064806/is-there-a-way-to-specify-the-base-address-of-a-shared-library-using-dlopen

You can provide your mmap trick as an answer
to that stackoverflow question. Lets see if
that can solve his problem.

Not sure if dlopen to user buffer was ever
requested. Its probably my own request then,
because no one thought of layering that on
top of dlmem(). And any other impl would be
very problematic.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug dynamic-link/30127] [rfe]: enable ld audit at run-time

[janderson at rice dot edu]

https://sourceware.org/bugzilla/show_bug.cgi?id=30127

--- Comment #39 from Jonathon Anderson <janderson at rice dot edu> ---
(In reply to Stas Sergeev from comment #37)
> Well, v9 only addresses this:
> "Is there a way to indicate to the auditor that it's \"missed the boat\""
> by allowing you to tell anything you want to
> an auditor.

There's definitely some confusion here. Let me explain my use case a bit.

My team develops a tool that (among other things) monitors CUDA activity from
within the application's address space (to unwind the call stack, to associate
the activity with a calling context). We register a handful of callbacks with
CUDA (libcuda.so), we get called anytime the application invokes a CUDA
activity *after* we register. To avoid changing the application's behavior as
much as possible, we don't load libcuda.so into the address space ourselves,
instead we react to when the application loads it (the specific version that
the app was compiled with, may be different than ours) and register the
callbacks then.

To capture all possible activities, we have a very narrow window in which to
register these callbacks:
  1. We must register callbacks before control returns to the application,
since it might cause some activity to occur. The latest LD_AUDIT callback
before application control is the la_activity(ACT_CONSISTENT) following the
la_objopen(libcuda.so).
  2. We must register callbacks after the init constructors for libcuda.so (and
dependencies) have run, the registration functions rely on internal state that
isn't fully initialized until the constructors. However,
  3. We must register callbacks before the init constructors for *any other
library* runs. There are applications that indirectly depend on libcuda.so
through some libgpuframework.so, which performs some CUDA activities in its
init constructors. We want to capture the activities invoked by
libgpuframework.so too.

Our current solution (and AFAIK the only solution) fitting these 3 rules relies
on the fact that la_objopen and la_activity(ACT_CONSISTENT) are called *before*
the init constructors for the newly loaded objects are run. This fact holds
true even for the main application itself. Specifically, we record when
la_objopen(libcuda.so) occurs, and during the following
la_activity(ACT_CONSISTENT) recursively call dlopen(libcuda.so). This call runs
the init constructors for libcuda.so (and dependencies) and *nothing else*, so
we can register the callbacks at that moment before control is returned to
other libraries or the application.

dlload_audit_module introduces an auditor much later in the process, long after
the init constructors for the loaded objects have completed. The la_objopen
calls you propose to "introduce" the already-loaded objects to the auditor are
thus "late," in other words the auditor has completely "missed the boat" for
this very narrow timing window. My team's auditor will not function correctly
if the la_objopen for libcuda.so is "late," but it can be made to work with
"late" notifications for most other libraries. With the approach you propose, I
need some kind of indication that any *particular* la_objopen is "late" for
this graceful failure handling.

I think that should answer the later questions, but I'll try to be explicit.

> 
> > AFAICT there isn't a way to tell which la_objopen callbacks are "real" (new
> > objects) or "fake" (objects already loaded).
> 
> There is no such thing simply because the
> "late" objopen call is already an established
> practice in dl_main().

It's not. The la_objopen calls in dl_main() still happens before the init
constructors, so these calls are not late. ld.so currently contains no late
la_objopen calls.

> Which means its a bit
> out of the scope of this patch, but it doesn't
> mean I can't add such a thing. For example I
> can add la_objopen_late(). But let's see if it
> is actually needed. For example can't you use
> the fact that la_objsearch() and/or
> la_activity(LA_ACT_ADD) wasn't called before
> la_objopen()?

la_objsearch isn't always called before la_objopen. vDSO is an example of this,
IIRC there also used to be an internal networking (DNS?) library that skipped
la_objsearch too. The same should be true of your proposed dlmem() patch.

The la_activity(ACT_ADD) trick has some promise, but there are a couple of
issues. One is that dl_main doesn't call la_activity(ACT_ADD) before the
la_objopens for the main application and ld.so. IMHO it should, but it doesn't.
That will confuse this approach.

The bigger issue is that the interface definition is very unclear on *when*
la_activity is called around la_obj* and the scope in which it applies. Glibc
maintains the internal link_map state under a single lock, but one could
imagine a more flexible implementation where multiple threads could load and
unload objects in parallel in different namespaces. If an auditor receives an
la_activity(ACT_ADD) callback, should it expect only la_objopen until the next
la_activity? For just the referenced namespace or for all namespaces? At the
moment there isn't a consistent meaning to the order of the callbacks [1], so
attaching extra meaning to the order is... well, meaningless.

[1]:
https://docs.google.com/document/d/1dVaDBdzySecxQqD6hLLzDrEF18M1UtjDna9gL5BWWI0/edit#heading=h.2usdoz39dhyg

> 
> 
> > How does that help? Constructor priority means nothing after the solib or
> > executable is linked. GCC/ld uses the priorities to sort the list of _init
> > functions within a single object. ld.so decides on the order objects run
> > their constructors based on dependencies between objects and nothing else.
> 
> Yes, so you need to add the "primary" ctor
> to the main object (not to any solib of it)
> and from that load the audit module, no?

IMHO if you go that far, LD_AUDIT (the environment var) works just as well. And
doesn't require recompiling the application with non-standard C. :D

Anyway, my main concern is graceful handling for late notifications. Getting
the auditor to load very early is a separate but less troublesome issue.

---

I'll try to reply to the rest tomorrow, schedule permitting. Some of my
questions have been answered, but at first glance I still have a number of
points of confusion and contention.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug dynamic-link/30127] [rfe]: enable ld audit at run-time

[stsp at users dot sourceforge.net]

https://sourceware.org/bugzilla/show_bug.cgi?id=30127

--- Comment #40 from Stas Sergeev <stsp at users dot sourceforge.net> ---
(In reply to Jonathon Anderson from comment #39) 
> I think that should answer the later questions, but I'll try to be explicit.

So it seems you are saying that even the very
first ctor is too late because you want to audit
the loading of DT_NEEDED deps of the main executable
itself? Or have I missed your point?
If that's the case then am I right that you can't
even change the app you audit, so dynamic loading
of an audit modules is completely out of the scope
for you?
And do you mean that you need that only for libcuda,
while for other libs you can use dynamic loading,
which, presumably, means calling dlload_audit_module()
from some call-back of a libcuda auditor? Because
I think you aren't changing the app you audit.
From that picture I think you have 2 audit modules,
one should be loaded with LD_AUDIT no matter what,
and another one can be loaded late but it needs to
know which lib was "already there" and which not.
Is that picture remotely correct? I guess no, as
it is based on too many assumptions from me. :)


> It's not. The la_objopen calls in dl_main() still happens before the init
> constructors, so these calls are not late. ld.so currently contains no late
> la_objopen calls.

Yes, now I see what kind of "late" you mean:
late for DT_NEEDED deps of the main executable.
Is that correct?


> The bigger issue is that the interface definition is very unclear on *when*
> la_activity is called around la_obj* and the scope in which it applies.
> Glibc maintains the internal link_map state under a single lock, but one
> could imagine a more flexible implementation where multiple threads could
> load and unload objects in parallel in different namespaces. If an auditor
> receives an la_activity(ACT_ADD) callback, should it expect only la_objopen
> until the next la_activity?

Ah, that's right... I thought la_activity()
is called with all link-maps, but now I see
it is only called for NS head, so you can't
rely on such an ordering by using cookies. :(

Note however that struct link_map has
"l_init_called". Unfortunately there are 2
struct link_map's, one of which doesn't have
it. Is it a viable solution for you to peek
into an "extended" link_map structure?

Also I still can't discount la_objsearch().
Even if it isn't called for vdso, is it a big
problem for you to assume that "vdso ctors
already executed"? Does vdso have any ctors
at all?


> IMHO if you go that far, LD_AUDIT (the environment var) works just as well.
> And doesn't require recompiling the application with non-standard C. :D

You forgot that the requirement of calling
an auditor before any ctors, is not a global
one. :) Its your use-case, mine doesn't have
such a requirement. Which is why I think my
use-case deserves proper "tools", whereas for
your use-case LD_AUDIT is the proper tool.


> Anyway, my main concern is graceful handling for late notifications. Getting
> the auditor to load very early is a separate but less troublesome issue.

If you can load it early enough then why would
you care about telling it about "too late"?

And also I probably haven't yet spelled out
the v9 change. Since you now can have la_dynload(),
can you assume that after la_dynload() any objopen
is "late" until LA_ACT_CONSISTENT is called?
I think LA_ACT_CONSISTENT will be called even
at the end of dlload_audit_module() itself.
So la_dynload() and LA_ACT_CONSISTENT do look
like a good markers?

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug dynamic-link/30127] [rfe]: enable ld audit at run-time

[stsp at users dot sourceforge.net]

https://sourceware.org/bugzilla/show_bug.cgi?id=30127

--- Comment #41 from Stas Sergeev <stsp at users dot sourceforge.net> ---
Ping.
So any of the suggested methods are
good for your use-case?

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug dynamic-link/30127] [rfe]: enable ld audit at run-time

[janderson at rice dot edu]

https://sourceware.org/bugzilla/show_bug.cgi?id=30127

--- Comment #42 from Jonathon Anderson <janderson at rice dot edu> ---
Sorry for the delay, this fell lower on my list than intended.

(In reply to Stas Sergeev from comment #40)
> (In reply to Jonathon Anderson from comment #39) 
> > I think that should answer the later questions, but I'll try to be explicit.
> 
> So it seems you are saying that even the very
> first ctor is too late because you want to audit
> the loading of DT_NEEDED deps of the main executable
> itself? Or have I missed your point?
More or less right, IF libcuda.so is one of those deps. If libcuda.so isn't
involved or is loaded after the auditor is, I can recover and still operate as
intended.

> If that's the case then am I right that you can't
> even change the app you audit, so dynamic loading
> of an audit modules is completely out of the scope
> for you?
My team has managed to avoid changing the app so far, it's much more convenient
for us to keep it that way.

We have no reason to dynamically load our auditor. However, my team does
implement a real, live auditor that is used in the wild. I bring it up because
I would think that a feature that allows for dynamic loading of auditors should
support all auditors, as much as physically possible. That includes my team's
auditor.

After all, all solibs are perfectly functional regardless of whether you load
it via LD_PRELOAD or dlopen or DT_NEEDED. Shouldn't auditors follow the same
principle?

> And do you mean that you need that only for libcuda,
> while for other libs you can use dynamic loading,
> which, presumably, means calling dlload_audit_module()
> from some call-back of a libcuda auditor?
No. What I mean is that libcuda.so is one of the only libraries I need to
register callbacks for. For the vast majority of libraries I don't need to hit
the narrow timing window and can work fine with "late" la_objopen calls. It's
the la_objopen for libcuda.so that I cannot handle being "late," if that
happens I would need to error out to prevent crashes later down the line.

> Because
> I think you aren't changing the app you audit.
> From that picture I think you have 2 audit modules,
> one should be loaded with LD_AUDIT no matter what,
> and another one can be loaded late but it needs to
> know which lib was "already there" and which not.
A second auditor would be overkill, once you have one loaded with LD_AUDIT you
can "see" everything going on already and don't need a second auditor. I think
that logic would apply just as well to the vast majority of other auditor use
cases.

> Is that picture remotely correct? I guess no, as
> it is based on too many assumptions from me. :)
It's in the right direction. The only reason I bring up my use case is as an
example auditor that would be broken by this dynamic loading feature in its
current state.

> 
> 
> > It's not. The la_objopen calls in dl_main() still happens before the init
> > constructors, so these calls are not late. ld.so currently contains no late
> > la_objopen calls.
> 
> Yes, now I see what kind of "late" you mean:
> late for DT_NEEDED deps of the main executable.
> Is that correct?
And any other solibs loaded before the dlload_audit_module call, but yeah.

> 
> 
> > The bigger issue is that the interface definition is very unclear on *when*
> > la_activity is called around la_obj* and the scope in which it applies.
> > Glibc maintains the internal link_map state under a single lock, but one
> > could imagine a more flexible implementation where multiple threads could
> > load and unload objects in parallel in different namespaces. If an auditor
> > receives an la_activity(ACT_ADD) callback, should it expect only la_objopen
> > until the next la_activity?
> 
> Ah, that's right... I thought la_activity()
> is called with all link-maps, but now I see
> it is only called for NS head, so you can't
> rely on such an ordering by using cookies. :(
> 
> Note however that struct link_map has
> "l_init_called". Unfortunately there are 2
> struct link_map's, one of which doesn't have
> it. Is it a viable solution for you to peek
> into an "extended" link_map structure?
Not really, we support too many different versions of Glibc to make that
viable. :P

> 
> Also I still can't discount la_objsearch().
> Even if it isn't called for vdso, is it a big
> problem for you to assume that "vdso ctors
> already executed"? Does vdso have any ctors
> at all?
The finite set of "expected" non-la_objsearch objects (vDSO, main executable,
ld.so) are easy enough to deal with. The bigger problem is the much larger set
of unexpected non-la_objsearch objects, such as the old (DNS? libresolv?)
library or the objects loaded by the proposed dlmem().

The parallelism issue is present here too. There's no association between an
la_objsearch and the resulting la_objopen, so if the auditor receives the
callback sequence [la_objsearch, la_objsearch, la_objopen, la_objopen], should
it expect the second la_objopen to be "late" or not? (la_objsearch can also be
called multiple times for the same path as it gets transformed, so counting
isn't a solution.)

> 
> 
> > IMHO if you go that far, LD_AUDIT (the environment var) works just as well.
> > And doesn't require recompiling the application with non-standard C. :D
> 
> You forgot that the requirement of calling
> an auditor before any ctors, is not a global
> one. :) Its your use-case, mine doesn't have
> such a requirement. Which is why I think my
> use-case deserves proper "tools", whereas for
> your use-case LD_AUDIT is the proper tool.
Indeed. My primary goal is to ensure the proposed "tools" for your use case
don't unwittingly destroy my use case in the process. :)

> 
> 
> > Anyway, my main concern is graceful handling for late notifications. Getting
> > the auditor to load very early is a separate but less troublesome issue.
> 
> If you can load it early enough then why would
> you care about telling it about "too late"?
> 
> And also I probably haven't yet spelled out
> the v9 change. Since you now can have la_dynload(),
> can you assume that after la_dynload() any objopen
> is "late" until LA_ACT_CONSISTENT is called?
> I think LA_ACT_CONSISTENT will be called even
> at the end of dlload_audit_module() itself.
> So la_dynload() and LA_ACT_CONSISTENT do look
> like a good markers?
If the interface is defined such that no other la_objopen calls from threads
running in parallel can trigger before that first la_activity(CONSISTENT), then
I could consider that a reasonable enough solution.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug dynamic-link/30127] [rfe]: enable ld audit at run-time

[janderson at rice dot edu]

https://sourceware.org/bugzilla/show_bug.cgi?id=30127

--- Comment #43 from Jonathon Anderson <janderson at rice dot edu> ---
Going back to your use case:

(In reply to Stas Sergeev from comment #37)
> > I assume there's a Very Good Reason (TM) the ancient code can't be compiled
> > 32-bit (and run in the VM) nor the compatibility libraries compiled 64-bit
> > (and run without a VM). Otherwise you wouldn't need any of this.
> 
> No, this is not the case. :)
> I mean, the reason may be not Very Good (TM)
> for your ears, but its just that it would need
> the pretty much outdated tool-chains that no
> one has these days. 
I can consider that a good enough reason not to compile the ancient code as
32-bit. But the compatibility libraries are surely much newer, can't they be
compiled 64-bit?

> Also the ability to work
> on host in 64bit mode provides performance and
> other benefits. 
It does. Can you compile the compatibility libraries as 64-bit instead? :D

> VM is much, much less involved
> in that scenario, than when you run everything
> inside VM. In theory, this will also allow to
> extend the old code with some functionality
> available on host, but that's not something to
> anticipate at that early stage of a development.
> 
> 
> > What do you do about pointers to stack variables?
> 
> I can set up the initial stack from my
> shim lib, and if an app wants to switch
> stacks (which is very unlikely), it will
> have to use the heap space, which is under
> my control.
How? The initial stack is set up by the kernel before the executable is even
mapped. Do you switch stacks in your shim library, after the VM memory region
is decided upon?

While on the topic (and mostly out of curiosity), what do you do when the
64-bit code allocates more than 4GB of memory? It obviously won't all fit in
the VM's address space, so a pointer could be passed through the shim lib that
would end up out of bounds.

> 
> 
> > That... makes absolutely no sense. In that arrangement the loaded solib will
> > be useless outside the VM, the relocations are wrong for the 64-bit address
> > space.
> 
> la_premap_dlmem() is a very powerful extension. :)
> It allows you to map the lib anywhere you
> want, any amount of times in a row. You can
> map it to VM window _in addition_ to also
> mapping it to its relocation point. That's
> what even the test-case of mine demonstrates
> by doing 2 mappings of the same lib. 
...If all you wanted was to mmap the solib to another address, you can already
do that using mmap and /proc/self/map_files/. Maybe dl_iterate_phdr. No
la_premap or dlmem() needed.

The only reason any extension to Glibc is needed is if you wanted
(reloc_base_address != mmap_address). Which, to reiterate, is completely
nonsensical.

> 
> 
> > So why does the 64-bit ld.so need to load it? Can't you load it
> > inside the VM using the 32-bit ld.so, since that's the only place it could
> > possibly work?
> 
> It should work in both "worlds" because it has
> the regular DT_NEEDED deps linked with its
> own custom libc. So the host relocations should
> work properly, and only the shim lib translates
> the calls into VM.
I really don't understand. The ISA is different between the "worlds," the
relocation base address will always be wrong in one "world" and make the solib
useless there. Heck, even the ELFCLASS is different.

I see no way for an solib to function in both "worlds." AFAICT the best you can
do is "manually" (i.e. not via data symbols) access the data segment from the
"other world," functions simply won't work.

> 
> 
> > > I don't think the proposed scheme can be
> > > replicated by any tricks and hacks. And
> > > honestly I don't know why it should.
> > > There was that LD_PREFER_MAP_32BIT_EXE but
> > > it was dumb, so removed. Why not to replace
> > > it with something small and smart?
> > 
> > IMHO this is well outside the kinds of problems where a "small and smart"
> > solution is a tractable goal. After all, we're talking about multiple
> > architectures in a single process image. :P
> 
> I think LD_PREFER_MAP_32BIT_EXE was also
> about something like this. So my demand is
> definitely not the first one.
> But let's look at it from another side:
> all I do is specify the reloc address and
> pass a backing-store fd. 
You can specify the backing-store fd with classic dlopen and /proc/self/fd/.
That's a very simple solution, and requires no changes to Glibc.

You can choose the reloc base address (== mmap address) by overriding mmap.
That's not a simple solution, but it doesn't require a new interface or Glibc
patches. As noted above, I'm not yet confident your use case would work even if
you could specify the reloc base address.

I don't see any reason to support the (reloc base address != mmap address)
case. It's utter nonsense.

(In reply to Stas Sergeev from comment #38)
> So dlmem was requested:
> https://sourceware.org/bugzilla/show_bug.cgi?id=11767
> https://stackoverflow.com/questions/5053664/dlopen-from-memory
> https://stackoverflow.com/questions/29637464/need-an-api-that-allows-me-to-
> load-a-shared-object-from-a-memory-address
> 
> And even implemented:
> https://github.com/m1m1x/memdlopen
AFAICT these discussions are all solved by memfd_create. Almost all of the
complaints revolve around the memory vs. disk performance difference,
memfd_create is a very easy way to get a memory-backed file on modern Linux
boxes.

> 
> Mapping to specific address was requested:
> https://stackoverflow.com/questions/62064806/is-there-a-way-to-specify-the-
> base-address-of-a-shared-library-using-dlopen
I'm as confused as the answer's author on what the OP's actual use case is
here. Given the OP's lack of response to the request for context, I wouldn't
consider this worth much attention.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug dynamic-link/30127] [rfe]: enable ld audit at run-time

[stsp at users dot sourceforge.net]

https://sourceware.org/bugzilla/show_bug.cgi?id=30127

--- Comment #44 from Stas Sergeev <stsp at users dot sourceforge.net> ---
(In reply to Jonathon Anderson from comment #42)
> After all, all solibs are perfectly functional regardless of whether you
> load it via LD_PRELOAD or dlopen or DT_NEEDED. Shouldn't auditors follow the
> same principle?

Normally yes, but when you bring up
things like "but it should run before
any ctors", then these must be evaluated
case-by-case.


> > Is that picture remotely correct? I guess no, as
> > it is based on too many assumptions from me. :)
> It's in the right direction. The only reason I bring up my use case is as an
> example auditor that would be broken by this dynamic loading feature in its
> current state.

This is a very harsh statement.
Nothing will be broken. At worst, you won't
be able to use that dynamic loading for your
case. Given that you have only 1 auditor, no
intention to change the app and a requirement
for an auditor to run before any ctors, I think
this "worst case" is already unavoidable. :)


> The parallelism issue is present here too. There's no association between an
> la_objsearch and the resulting la_objopen,

I think you can associate by "name".
The name in objsearch and abjopen should
match I think. But I've heard your other
arguments against objsearch, yes.


> Indeed. My primary goal is to ensure the proposed "tools" for your use case
> don't unwittingly destroy my use case in the process. :)

I don't even see why this can be the case.
If it is, why aren't we discussing _that_,
instead of everything else? :)


> If the interface is defined such that no other la_objopen calls from threads
> running in parallel can trigger before that first la_activity(CONSISTENT),
> then I could consider that a reasonable enough solution.

This happens within a single dlmopen() call.
I think its up to glibc to make sure no one
touches the link-map before ACT_CONSISTENT is
reported. And I think you can also use pthread_self()
to check for yourself from which threads things
are called. So when you see la_dynload(), you
remember pthread_self(). Then you know that:
1. Any la_objopen() call with that pthread_self()
   is "late" unless ACT_CONSISTENT was seen with
   that pthread_self().
2. Any la_objopen() call with different pthread_self()
   is not "late".

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug dynamic-link/30127] [rfe]: enable ld audit at run-time

[stsp at users dot sourceforge.net]

https://sourceware.org/bugzilla/show_bug.cgi?id=30127

--- Comment #45 from Stas Sergeev <stsp at users dot sourceforge.net> ---
(In reply to Jonathon Anderson from comment #43)
> I can consider that a good enough reason not to compile the ancient code as
> 32-bit. But the compatibility libraries are surely much newer, can't they be
> compiled 64-bit?

Indeed, that's the case.
I wouldn't go into such a long and deep
hoops (where glibc is by far not the only
obstacle) to just compile things for 32bits. :)


> > I can set up the initial stack from my
> > shim lib, and if an app wants to switch
> > stacks (which is very unlikely), it will
> > have to use the heap space, which is under
> > my control.
> How? The initial stack is set up by the kernel before the executable is even
> mapped.

The initial stack of what?
Of an executable - indeed.
But not of the lib I run through dlmem().
I call its entry manually, after setting
up the things (including stack).


> Do you switch stacks in your shim library, after the VM memory
> region is decided upon?

Indeed. These things are done through
makecontext().


> While on the topic (and mostly out of curiosity), what do you do when the
> 64-bit code allocates more than 4GB of memory?

This 64bit code is compiled from ancient
sources, that shouldn't do that. So if it
does, it will just get the malloc() failure.
I run it with the custom libc - that allows
me to control everything. No direct host
calls are allowed. I am using RTLD_DEEPBIND
and a new NS. Well, "using" is a too loud
statement, as before things settled to glibc,
I have only a rough tests and prototypes.
But I tried similar concepts in the past,
so I am pretty sure the scheme is functional.


> > by doing 2 mappings of the same lib. 
> ...If all you wanted was to mmap the solib to another address, you can
> already do that using mmap and /proc/self/map_files/. Maybe dl_iterate_phdr.

That can only work for loadable sections,
I believe. .bss cannot be shared that way,
and likely much more.


> The only reason any extension to Glibc is needed is if you wanted
> (reloc_base_address != mmap_address). Which, to reiterate, is completely
> nonsensical.

But as I already mentioned, I do 2 identical
mappings. One at reloc address, one at mmap
address. So even though reloc_address!=mmap_address,
things do work, because because the one at
mmap_address is only used for data accesses,
and the one at reloc_address is used for
everything (like code calls).


> I really don't understand. The ISA is different between the "worlds," the
> relocation base address will always be wrong in one "world" and make the
> solib useless there.

Not quite useless: data access by 32bit
code is done there. Everything else is done
at another address. So every mapping does
only what it can.


> I see no way for an solib to function in both "worlds." AFAICT the best you
> can do is "manually" (i.e. not via data symbols) access the data segment
> from the "other world," functions simply won't work.

Indeed!
Functions will work when shim calls them
(via the right mapping).
"not via data symbols" is not a problem,
because when the call is in the VMs "world",
it has the ready to use pointers inside the
call arguments, and nothing else. Only these
pointers should work there. These pointers
lead to a reloc address, but because the VM
window is itself shifted, they need to go
through the alias at mmap_address to reach
their destination at reloc_address.
mmap_address=reloc_address+VM_window_start


> You can specify the backing-store fd with classic dlopen and /proc/self/fd/.
> That's a very simple solution, and requires no changes to Glibc.

Not sure I understand that part.
dlopen() uses source fd to map it to reloc_address,
but the data sections are just anonymously
mapped.
So I am lost, what does /proc/self/fd do with
all that?


> Glibc patches. As noted above, I'm not yet confident your use case would
> work even if you could specify the reloc base address.
> 
> I don't see any reason to support the (reloc base address != mmap address)
> case. It's utter nonsense.

If alone - yes.
But when you have multiple mappings, one of
which is still at reloc_address, then things
magically start to work again.
Probably you still haven't realized the whole
power of la_premap_dlmem(). Please note that
la_premap_dlmem() is not the same as la_premap().
These are different extensions, but when they
play together, that can do Real Things (TM).


> AFAICT these discussions are all solved by memfd_create. Almost all of the
> complaints revolve around the memory vs. disk performance difference,

I am getting a bit nervous already when people
mention memfd_create(). :) In what way is it
any better than shm_open(), that I used in my
la_premap_dlmem() example?
Yes, I could also use memfd_create() with
la_premap_dlmem(), but I prefer shm_open().
Why people think that memfd_create() is the
thing, is unclear to me. :) But it fits my
design very well, as does shm_open().


> memfd_create is a very easy way to get a memory-backed file on modern Linux
> boxes.

... and provide it to la_premap_dlmem().
But I'll use shm_open! :)


> > Mapping to specific address was requested:
> > https://stackoverflow.com/questions/62064806/is-there-a-way-to-specify-the-
> > base-address-of-a-shared-library-using-dlopen
> I'm as confused as the answer's author on what the OP's actual use case is
> here. Given the OP's lack of response to the request for context, I wouldn't
> consider this worth much attention.

Well, I am not saying these examples
deserve much of an attention. But I used
them to bring up an API that at least is
requested multiple times, and layer the
rest on top. The warning here is the next
time you'll deal with much more alien and
complicated API that won't solve the entire
task even. So why not to use mine, its was
requested multiple times, its written, it
will be proven (within this thread) to solve
the stated task, and very unlikely some hack
can be proposed as an alternative.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug dynamic-link/30127] [rfe]: enable ld audit at run-time

[stsp at users dot sourceforge.net]

https://sourceware.org/bugzilla/show_bug.cgi?id=30127

--- Comment #46 from Stas Sergeev <stsp at users dot sourceforge.net> ---
So let me summarize that memfd_create()
(shm_open() actually) is not a replacement,
but rather is an essential part of the
scheme. Using it together with la_premap_dlmem()
and la_premap() you can get the desired
picture. Desired picture is 2 identical
mappings of the same lib, one at relolc_addr,
one at mmap_addr=reloc_addr+VM_window_start.

There is basically nothing else!
That scheme is very simple to describe,
but not that simple to grok from that
description, as no one have tried that
layout yet.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug dynamic-link/30127] [rfe]: enable ld audit at run-time

[stsp at users dot sourceforge.net]

https://sourceware.org/bugzilla/show_bug.cgi?id=30127

--- Comment #47 from Stas Sergeev <stsp at users dot sourceforge.net> ---
I also need to correct myself referring
to 2 different entities as a "shim lib":
1. actual shim lib that sits in a same
   NS with dlmem()ed lib and forwards
   the calls to VM.
2. control library that does dlload_audit_module(),
   dlmem(), makecontext() and all the other
   housekeeping - that one sits in a main namespace
   and should not be referred to as a "shim lib".

Not sure what level of details you want,
but I'll better clarify that before the
confusion arises.

Another note may be a suggestion to not
yet look into the dlmem()&co impl itself.
There are good demos and test-cases in the
patch, worth taking a look into, as they
are there to answer most of the questions.
But regarding the dlmem()&co impl itself -
I'll rewrite it when this dlload_audit_module()
patch settles, as I definitely know more
about glibc than back when dlmem() was written.
So no need to look into it or criticise
it just yet. :) If you do look, please
restrict to a test-cases.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug dynamic-link/30127] [rfe]: enable ld audit at run-time

[stsp at users dot sourceforge.net]

https://sourceware.org/bugzilla/show_bug.cgi?id=30127

--- Comment #48 from Stas Sergeev <stsp at users dot sourceforge.net> ---
(In reply to Jonathon Anderson from comment #33)
> The result of the first call to mmap() for an solib decides the base address

While a bit outdated topic, I don't
think "the first call to mmap()" is a
good or reliable work-around. It may
change with an impl, or because of the
threads. In general, I wouldn't advise
that to the stackoverflow guy, as such
solution is quite limited.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug dynamic-link/30127] [rfe]: enable ld audit at run-time

[janderson at rice dot edu]

https://sourceware.org/bugzilla/show_bug.cgi?id=30127

--- Comment #49 from Jonathon Anderson <janderson at rice dot edu> ---
(In reply to Stas Sergeev from comment #46)
> So let me summarize that memfd_create()
> (shm_open() actually) is not a replacement,
> but rather is an essential part of the
> scheme. Using it together with la_premap_dlmem()
> and la_premap() you can get the desired
> picture. Desired picture is 2 identical
> mappings of the same lib, one at relolc_addr,
> one at mmap_addr=reloc_addr+VM_window_start.
> 
> There is basically nothing else!
> That scheme is very simple to describe,
> but not that simple to grok from that
> description, as no one have tried that
> layout yet.
FWIW I understand this succinct description of your intended use case much
better than your prior descriptions, thanks. If this is the core feature you
need, then I see where most of my questions about the VM go off-topic. Sorry
for the noise. (I honestly thought you were trying to achieve something far
more drastic...)

So in short, dynamic auditor loading is merely a "detail" in your full
approach, since you don't have access to the executable or pre-launch
environment and your patch to implement this core feature uses LD_AUDIT to host
the object mmap callback.

I'll redirect my thoughts that are really about object mapping over to
https://sourceware.org/bugzilla/show_bug.cgi?id=30007. I think it makes sense
to resolve that RFE first, since that is in fact the core feature you need for
your entire approach to function. After (or maybe in parallel with) that we can
continue to discuss dynamic auditor loading over here.

(In reply to Stas Sergeev from comment #47)
> But regarding the dlmem()&co impl itself -
> I'll rewrite it when this dlload_audit_module()
> patch settles, as I definitely know more
> about glibc than back when dlmem() was written.
> So no need to look into it or criticise
> it just yet. :) If you do look, please
> restrict to a test-cases.
Certainly, I'll stick to the API and test cases. :D

-- 
You are receiving this mail because:
You are on the CC list for the bug.