libffi & fork

libffi & fork

[Gaash Hazan]

Hi experts,

I am new to libffi, so forgive me for my ignorance.

I have a python crash involving ctypes and functions callbacks (read
closures) and fork. The crash happens when process is forked, some
callbacks are modified and a function callback is made. I suspect that
the problem could be related to libffi closures and fork.

dlmmap() at libffi/closures.c allocates memory block that has to be
both read-write and read-execute. To achieve that, libffi creates a
temp file and performs two mmap-s to the that file, the first
read-write and the 2nd read-execute.

7fe27d64f000-7fe27d650000 rw-s 00000000 08:02 1840563
  /tmp/ffiHf53x5 (deleted)
7fe27d650000-7fe27d651000 r-xs 00000000 08:02 1840563
  /tmp/ffiHf53x5 (deleted)

The thing is that those maps are marked as shared. Hence when the
process is forked the allocated(mapped) memory block is *shared*
between the processes (and not copied-on-write). So, when father
process changes something in that memory block, the child process will
be affected as well, creating unplanned shared-memory block between
the two processes. I suspect that this is not the intended behavior
and that it leads to the crash.

snips from /proc/XXX/maps: father PID=7975, child PID=7976
----------------------------------------------------------------------------------------
$cat /proc/7975/maps  | grep /tmp/
7fc537d49000-7fc537d4a000 rw-s 00000000 08:02 1840370
  /tmp/ffiqyLKFR (deleted)
7fc537d4a000-7fc537d4b000 r-xs 00000000 08:02 1840370
  /tmp/ffiqyLKFR (deleted)

$cat /proc/7976/maps  | grep /tmp/
7fc537d49000-7fc537d4a000 rw-s 00000000 08:02 1840370
  /tmp/ffiqyLKFR (deleted)
7fc537d4a000-7fc537d4b000 r-xs 00000000 08:02 1840370
  /tmp/ffiqyLKFR (deleted)

test .py:
-----------
import os,time
import ctypes
os.fork()
print str(os.getpid())
time.sleep( 1000)

system info: centos 6.0, pyhton 2.6.5, selinux disabled, x86_64. Also
tested with python 2.7.3

Thanks,
Gaash

Re: libffi & fork

[Andrew Haley]

On 04/24/2012 08:40 PM, Gaash Hazan wrote:

> I am new to libffi, so forgive me for my ignorance.
> 
> I have a python crash involving ctypes and functions callbacks (read
> closures) and fork. The crash happens when process is forked, some
> callbacks are modified and a function callback is made. I suspect that
> the problem could be related to libffi closures and fork.

There is, as far as I can see, only one way to remedy this: allow
python to map anonymous memory with both write and execute permission.
Then it should work fine.

Andrew.

Re: libffi & fork

[Gaash Hazan]

> There is, as far as I can see, only one way to remedy this: allow
> python to map anonymous memory with both write and execute permission.
> Then it should work fine.
>

I believe memory allocation is a service provided by the OS and not by
the application or interpreter. Python does not provide memory
allocation service to libffi. In this case libffi creates
read-write-exec memory block using mmap to a tmp file. I guess the
problem is common to libff users and it is not unique to python.

libffi uses mmap with MAP_SHARED at libffi:closures.c:dlmap(). What
was the reason for using MAP_SHARED  in the first place?

I think MAP_PRIVATE would create the desired behavior of copy-on-write
when forked. Would that be a proper fix?

Gaash

Re: libffi & fork

[Andrew Haley]

On 04/25/2012 06:32 PM, Gaash Hazan wrote:
>> There is, as far as I can see, only one way to remedy this: allow
>> python to map anonymous memory with both write and execute permission.
>> Then it should work fine.
>
> I believe memory allocation is a service provided by the OS and not by
> the application or interpreter.

The usual problem is a mechanism such as SELinux that is forbidding
Python from mmap()ing PROT_WRITE|PROT_EXEC .  If you label the
executable unconfined_execmem_exec_t it may work because then libffi
won't bother with all this multiple maps though a shared file
descriptor business.

> Python does not provide memory allocation service to libffi. In this
> case libffi creates read-write-exec memory block using mmap to a tmp
> file. I guess the problem is common to libff users and it is not
> unique to python.

It's special to Python because Python is the only common libffi client
that forks and then tries to use the closures.

> libffi uses mmap with MAP_SHARED at libffi:closures.c:dlmap(). What
> was the reason for using MAP_SHARED  in the first place?
>
> I think MAP_PRIVATE would create the desired behavior of copy-on-write
> when forked. Would that be a proper fix?

No, memory regions that are backed by a real file are always shared
between processes.  There is a single file descriptor that
references the block of memory that holds the closures.  The only
way you'll get it to work is to create a new file.

Andrew.

Re: libffi & fork

[Gaash Hazan]

Hi Andrew,

Thanks for the detailed answer.

I think I found the root cause of the problem. The libffi version that
I use (3.0.5) contains a bug in closures.c:selinux_enabled_check()
that incorrectly detected non-selinux systsem as selinux system.

Because system is detected (incorrectly) as selinux dlmap() does not
attempts to do mmap() for private anonymous block with write-exec
protection. It reverts to the trick with two mmap()s on a tmp file.
But trick does not with fork.

If the system is detected (correctly) as non-selinux, then dlmp()
would use anonymous write-exec mmap() and it would  not use the trick.
Hence the fork problem would not occurred.

The selinux detected problem was fixed in
https://github.com/atgreen/libffi/commit/eaf444eabc4c78703c0f98ac0197b1619c1b1bef#src/closures.c
(closures.c line 149)

Unfortunately Redhat 6.2 and 6.3-bets uses libffi 3.0.5 that has this problem.

Regards,
Gaash


On Thu, Apr 26, 2012 at 1:00 PM, Andrew Haley <aph@redhat.com> wrote:
>
> On 04/25/2012 06:32 PM, Gaash Hazan wrote:
> >> There is, as far as I can see, only one way to remedy this: allow
> >> python to map anonymous memory with both write and execute permission.
> >> Then it should work fine.
> >
> > I believe memory allocation is a service provided by the OS and not by
> > the application or interpreter.
>
> The usual problem is a mechanism such as SELinux that is forbidding
> Python from mmap()ing PROT_WRITE|PROT_EXEC .  If you label the
> executable unconfined_execmem_exec_t it may work because then libffi
> won't bother with all this multiple maps though a shared file
> descriptor business.
>
> > Python does not provide memory allocation service to libffi. In this
> > case libffi creates read-write-exec memory block using mmap to a tmp
> > file. I guess the problem is common to libff users and it is not
> > unique to python.
>
> It's special to Python because Python is the only common libffi client
> that forks and then tries to use the closures.
>
> > libffi uses mmap with MAP_SHARED at libffi:closures.c:dlmap(). What
> > was the reason for using MAP_SHARED  in the first place?
> >
> > I think MAP_PRIVATE would create the desired behavior of copy-on-write
> > when forked. Would that be a proper fix?
>
> No, memory regions that are backed by a real file are always shared
> between processes.  There is a single file descriptor that
> references the block of memory that holds the closures.  The only
> way you'll get it to work is to create a new file.
>
> Andrew.

Re: libffi & fork

[Anthony Green]

On Sun, Apr 29, 2012 at 11:34 AM, Gaash Hazan <gaash@qwilt.com> wrote:
> The selinux detected problem was fixed in
> https://github.com/atgreen/libffi/commit/eaf444eabc4c78703c0f98ac0197b1619c1b1bef#src/closures.c
> (closures.c line 149)
>
> Unfortunately Redhat 6.2 and 6.3-bets uses libffi 3.0.5 that has this problem.

Thanks Gaash.

Andrew - it would be nice if we could upgrade RHEL's libffi to a
non-ABI breaking 3.0.10.  If, however, it makes more sense to
cherry-pick patches since 3.0.5, I can see some nice ones from mjw and
jakub back in 2010, and there are likely more.  If you think we should
open an RFE I can propose some specific patches for consideration.
There may only be 4 or 5 and they are all pretty obvious.

AG

Re: libffi & fork

[Andrew Haley]

On 04/29/2012 06:58 PM, Anthony Green wrote:
> On Sun, Apr 29, 2012 at 11:34 AM, Gaash Hazan <gaash@qwilt.com> wrote:
>> The selinux detected problem was fixed in
>> https://github.com/atgreen/libffi/commit/eaf444eabc4c78703c0f98ac0197b1619c1b1bef#src/closures.c
>> (closures.c line 149)
>>
>> Unfortunately Redhat 6.2 and 6.3-bets uses libffi 3.0.5 that has this problem.
> 
> Thanks Gaash.
> 
> Andrew - it would be nice if we could upgrade RHEL's libffi to a
> non-ABI breaking 3.0.10.  If, however, it makes more sense to
> cherry-pick patches since 3.0.5, I can see some nice ones from mjw and
> jakub back in 2010, and there are likely more.  If you think we should
> open an RFE I can propose some specific patches for consideration.
> There may only be 4 or 5 and they are all pretty obvious.

I think we should do that.

Andrew.

Re: libffi & fork

[Anthony Green]

On Mon, Apr 30, 2012 at 4:32 AM, Andrew Haley <aph@redhat.com> wrote:
>> Andrew - it would be nice if we could upgrade RHEL's libffi to a
>> non-ABI breaking 3.0.10.  If, however, it makes more sense to
>> cherry-pick patches since 3.0.5, I can see some nice ones from mjw and
>> jakub back in 2010, and there are likely more.  If you think we should
>> open an RFE I can propose some specific patches for consideration.
>> There may only be 4 or 5 and they are all pretty obvious.
>
> I think we should do that.

Hmm.. check out this bug:
https://bugzilla.redhat.com/show_bug.cgi?id=772657

If our selinux state detection code really is this fragile, maybe we
should link against libselinux and use is_selinux_enabled() as
suggested in comment 4.

AG

Re: libffi & fork

[Andrew Haley]

On 04/30/2012 08:16 PM, Anthony Green wrote:
> On Mon, Apr 30, 2012 at 4:32 AM, Andrew Haley <aph@redhat.com> wrote:
>>> Andrew - it would be nice if we could upgrade RHEL's libffi to a
>>> non-ABI breaking 3.0.10.  If, however, it makes more sense to
>>> cherry-pick patches since 3.0.5, I can see some nice ones from mjw and
>>> jakub back in 2010, and there are likely more.  If you think we should
>>> open an RFE I can propose some specific patches for consideration.
>>> There may only be 4 or 5 and they are all pretty obvious.
>>
>> I think we should do that.
> 
> Hmm.. check out this bug:
> https://bugzilla.redhat.com/show_bug.cgi?id=772657
> 
> If our selinux state detection code really is this fragile, maybe we
> should link against libselinux and use is_selinux_enabled() as
> suggested in comment 4.

Would it not make far more sense simply to try to map the region
and fall back if that fails?

Andrew.

Re: libffi & fork

[Anthony Green]

On Tue, May 1, 2012 at 3:56 AM, Andrew Haley <aph@redhat.com> wrote:
>> Hmm.. check out this bug:
>> https://bugzilla.redhat.com/show_bug.cgi?id=772657
>>
>> If our selinux state detection code really is this fragile, maybe we
>> should link against libselinux and use is_selinux_enabled() as
>> suggested in comment 4.
>
> Would it not make far more sense simply to try to map the region
> and fall back if that fails?

Wouldn't that generate spurious SELinux warnings/errors?  I'd rather
not have that as expected behaviour.

AG

Re: libffi & fork

[Andrew Haley]

On 05/02/2012 09:23 PM, Anthony Green wrote:
> On Tue, May 1, 2012 at 3:56 AM, Andrew Haley <aph@redhat.com> wrote:
>>> Hmm.. check out this bug:
>>> https://bugzilla.redhat.com/show_bug.cgi?id=772657
>>>
>>> If our selinux state detection code really is this fragile, maybe we
>>> should link against libselinux and use is_selinux_enabled() as
>>> suggested in comment 4.
>>
>> Would it not make far more sense simply to try to map the region
>> and fall back if that fails?
> 
> Wouldn't that generate spurious SELinux warnings/errors?  I'd rather
> not have that as expected behaviour.

I suppose it would.  We could ask Dan Walsh what we should do.
Detecting the presence of SELinux is incorrect because:

1.  SELinux might be configured to allow write/exec regions.

2.  SELinux might be configured to allow write/exec regions for
    Python.

3.  Some non-SELinux mechanism might disallow write/exec regions.

One more alternative would be to probe but only at install time.
That's still wrong because someone could change SELinux settings.

Andrew.

Re: libffi & fork

[Anthony Green]

On Fri, May 4, 2012 at 4:59 AM, Andrew Haley <aph@redhat.com> wrote:
> I suppose it would.  We could ask Dan Walsh what we should do.

This is exactly what I've done in that bugzilla ticket.

Thanks,

AG