[Bug runtime/14490] New: Enable stapdyn in multiarch environments

[Bug runtime/14490] New: Enable stapdyn in multiarch environments

[jistone at redhat dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=14490

             Bug #: 14490
           Summary: Enable stapdyn in multiarch environments
           Product: systemtap
           Version: unspecified
            Status: NEW
          Severity: normal
          Priority: P2
         Component: runtime
        AssignedTo: systemtap@sourceware.org
        ReportedBy: jistone@redhat.com
            Blocks: 14178
    Classification: Unclassified


Right now, stapdyn and Dyninst itself are not well equipped to deal with
multiarch environments.  For one, Dyninst uses a single DYNINSTAPI_RT_LIB
environment variable, so it's not possible to set that differently for separate
target processes.  Then stap is currently only generating a native shared
library, but it might need to compile with flags like -m32 to be loaded in the
target.  And further, if PR14489 does have stapdyn dlopen the library, that
requires stapdyn to match architectures too.

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug dyninst/14490] Enable stapdyn in multiarch environments

[jistone at redhat dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=14490

Josh Stone <jistone at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
          Component|runtime                     |dyninst

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug dyninst/14490] Enable stapdyn in multiarch environments

[fche at redhat dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=14490

Frank Ch. Eigler <fche at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |fche at redhat dot com

--- Comment #1 from Frank Ch. Eigler <fche at redhat dot com> 2013-02-13 03:07:02 UTC ---
The cu-decl.exp test case demonstrates this case, running on x86-64, creating
an -m32 binary.

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug dyninst/14490] Enable stapdyn in multiarch environments

[dsmith at redhat dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=14490

David Smith <dsmith at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |dsmith at redhat dot com

--- Comment #2 from David Smith <dsmith at redhat dot com> ---
I wonder if we couldn't brute force this. On multiarch systems, compile a
'stapdyn' and a 'stapdyn32' executable and then run the correct one based on
the target executable. 'stap' itself would have to look at the target
executable to know which way to compile the shared library. (An even more brute
force approach would be to always compile 64-bit and 32-bit shared libraries,
and have a naming convention so we'll know which is which.)

Using a cached shared library becomes more tricky, since the 'bitness' of the
target executable could change (which might argue for the "always compile for
both architectures" approach).

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug dyninst/14490] Enable stapdyn in multiarch environments

[jistone at redhat dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=14490

--- Comment #3 from Josh Stone <jistone at redhat dot com> ---
If we create a stapdyn32, that means we also have to have dyninst in 32bit. 
This stapdyn32 also would *only* be capable of probing 32-bit targets.  So at
this level, I don't see much value in multiarch -- people who know they're
targeting 32-bit apps could use an entirely 32-bit stap to begin with.

The real value of multiarch IMO comes with mixed targets.  Perhaps a 64-bit
initial application will fork/exec a 32-bit worker process, etc.  All of this
monitored with a single stap session.

Of course, my opinion now doesn't mesh with my comment #0 about matching archs
in stapdyn.  We do now dlopen the stap module in stapdyn, and we obviously
can't do that for both -m64 and -m32.  To make those interoperable, we'll need
the same module compiled in both modes, able to talk to the *same* shared
memory.

So everything in shm would have to be constructed in an arch-neutral fashion. 
Structs have to be careful about padding, offptr_t has to use a fixed size for
its offset, etc.  Some parts are mostly just a shared reserve of memory, like
the context allocations, so that may not have to completely line up, though a
few bits within really are shared.


(Dare I throw mixed-endians in there?  No, no, that would be crazy...)

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug dyninst/14490] Enable stapdyn in multiarch environments

[dsmith at redhat dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=14490

--- Comment #4 from David Smith <dsmith at redhat dot com> ---
(In reply to Josh Stone from comment #3)
> If we create a stapdyn32, that means we also have to have dyninst in 32bit. 
> This stapdyn32 also would *only* be capable of probing 32-bit targets.  So
> at this level, I don't see much value in multiarch -- people who know
> they're targeting 32-bit apps could use an entirely 32-bit stap to begin
> with.

In case I didn't specify out my brute force idea very well, here you go. A user
would still run 'stap --dyninst -c target_exe script.stp'. Then, based on
whether target_exe was a 64 or 32 bit app, stap would run either stapdyn or
stapdyn32. If we want to push that down a level, stap could just run 'stapdyn'
which would be a simple program that looks at target_exe, and then exec either
'stapdyn_real' or 'stapdyn_real32'.

I guess this brute force approach is more 'biarch' (i.e. able to handle more
than 1 arch, but only one at at time) then 'multiarch' (i.e. able to handle
multiple arch simultaneously).

From a user's point of view, I think there is some value here. Perhaps not an
incredible amount, but some.

On to your next point...

> The real value of multiarch IMO comes with mixed targets.  Perhaps a 64-bit
> initial application will fork/exec a 32-bit worker process, etc.  All of
> this monitored with a single stap session.
> 
> Of course, my opinion now doesn't mesh with my comment #0 about matching
> archs in stapdyn.  We do now dlopen the stap module in stapdyn, and we
> obviously can't do that for both -m64 and -m32.  To make those
> interoperable, we'll need the same module compiled in both modes, able to
> talk to the *same* shared memory.
> 
> So everything in shm would have to be constructed in an arch-neutral
> fashion.  Structs have to be careful about padding, offptr_t has to use a
> fixed size for its offset, etc.  Some parts are mostly just a shared reserve
> of memory, like the context allocations, so that may not have to completely
> line up, though a few bits within really are shared.

OK, so here's my understanding of what you're thinking here.

- the shared memory segment created by the shared library needs to be bit size
agnostic, using arch neutral types and padding
- stap always compiles 64-bit and 32-bit shared libraries
- stap runs stapdyn, which is always 64-bit, and it always loads the 64-bit
shared library
- We load the appropriate shared library in the target based on the target's
'bitness'
- the target shared library (either 32-bit or 64-bit) and stapdyn's 64-bit
shared library communicate through the shared memory segment

I think from the stap side of things that looks reasonable and doable. Of
course we've still got the dyninst problem you mentioned in comment #0 (the
single DYNINSTAPI_RT_LIB variable).

Note #1. We've got a problem with cached shared libraries (and stapdyn
execution in general I guess). In theory you should be able to say 'stapdyn
foo.so -c target_exe', but we need both the 32-bit and 64-bit of foo.so if
target_exe is a a 32-bit executable (we need the 64-bit for stapdyn and the
32-bit for the target). Perhaps instead of the shared library name, stapdyn
will need to take a directory name (that is assumed to contain the 64-bit and
32-bit shared libraries). I guess our other option here would be that stapdyn
takes 2 shared library arguments.

Note #2: How do we ensure that the 64-bit and 32-bit shared libraries were
built from the same source (and therefore expect the shared memory to be laid
out the same way)? Perhaps we'll need to checksum the source and put that value
in the shared memory segment and both libraries need to agree on the value.

Note #3. Down the road we may have an RPM problem, since I'm not sure that a
64-bit stap RPM can require both the 32-bit and 64-bit versions of dyninst.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug dyninst/14490] Enable stapdyn in multiarch environments

[jistone at redhat dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=14490

--- Comment #5 from Josh Stone <jistone at redhat dot com> ---
(In reply to David Smith from comment #4)
> OK, so here's my understanding of what you're thinking here.
> 
> - the shared memory segment created by the shared library needs to be bit
> size agnostic, using arch neutral types and padding
> - stap always compiles 64-bit and 32-bit shared libraries
> - stap runs stapdyn, which is always 64-bit, and it always loads the 64-bit
> shared library
> - We load the appropriate shared library in the target based on the target's
> 'bitness'
> - the target shared library (either 32-bit or 64-bit) and stapdyn's 64-bit
> shared library communicate through the shared memory segment

Yes.

> I think from the stap side of things that looks reasonable and doable. Of
> course we've still got the dyninst problem you mentioned in comment #0 (the
> single DYNINSTAPI_RT_LIB variable).

Yes, that's something to address in dyninst itself.

> Note #1. We've got a problem with cached shared libraries (and stapdyn
> execution in general I guess). In theory you should be able to say 'stapdyn
> foo.so -c target_exe', but we need both the 32-bit and 64-bit of foo.so if
> target_exe is a a 32-bit executable (we need the 64-bit for stapdyn and the
> 32-bit for the target). Perhaps instead of the shared library name, stapdyn
> will need to take a directory name (that is assumed to contain the 64-bit
> and 32-bit shared libraries). I guess our other option here would be that
> stapdyn takes 2 shared library arguments.

I'd say just put m64/m32 in the cached names, give the native (m64) version as
normal to stapdyn, and add an optional --compat filename.so for the m32
version.

> Note #2: How do we ensure that the 64-bit and 32-bit shared libraries were
> built from the same source (and therefore expect the shared memory to be
> laid out the same way)? Perhaps we'll need to checksum the source and put
> that value in the shared memory segment and both libraries need to agree on
> the value.

Sure, just a checksum or uuid of some sort is fine - we don't need to be *too*
clever here.  We want to prevent accidents, but don't need to bother with
someone who deliberately tries to abuse this.  It's more like the buildid
checks we do for targets, just a basic sanity check.

> Note #3. Down the road we may have an RPM problem, since I'm not sure that a
> 64-bit stap RPM can require both the 32-bit and 64-bit versions of dyninst.

I'm not sure whether we may, but I don't think we should require it.  People
who are doing multiarch probing can be advised to install dyninst.i686 (or
ppc32, etc).  For the approach which does it all via shm compatibility, only
libdyninstAPI_RT.so is actually needed in 32-bit.  That's not packaged on its
own, but it could be.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug dyninst/14490] Enable stapdyn in multiarch environments

[dsmith at redhat dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=14490

--- Comment #6 from David Smith <dsmith at redhat dot com> ---
(In reply to Josh Stone from comment #5)
> (In reply to David Smith from comment #4)
> > OK, so here's my understanding of what you're thinking here.
> > 
> > - the shared memory segment created by the shared library needs to be bit
> > size agnostic, using arch neutral types and padding

I'm wondering if 32-bit and 64-bit processes can share a pthread mutexes. In a
64-bit exe, sizeof(pthread_mutex_t) is 40 bytes. In a 32-bit exe,
sizeof(pthread_mutex_t) is 24 bytes.

This can be clearly seen in /usr/include/bits/pthreadtypes.h (and confirmed by
a test program):

====
...
#ifdef __x86_64__
# if __WORDSIZE == 64
#  define __SIZEOF_PTHREAD_MUTEX_T 40
...
#else
# define __SIZEOF_PTHREAD_MUTEX_T 24
...
#endif
...
====

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug dyninst/14490] Enable stapdyn in multiarch environments

[jistone at redhat dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=14490

--- Comment #7 from Josh Stone <jistone at redhat dot com> ---
(In reply to David Smith from comment #6)
> I'm wondering if 32-bit and 64-bit processes can share a pthread mutexes. In
> a 64-bit exe, sizeof(pthread_mutex_t) is 40 bytes. In a 32-bit exe,
> sizeof(pthread_mutex_t) is 24 bytes.

That's a very good point.  Gross.  I suppose we could create our own locking
primitives, since futex deals in simple ints, but this is definitely a
complication I hadn't thought about.

-- 
You are receiving this mail because:
You are the assignee for the bug.