Re: /usr/bin/m4: internal error detected

Re: /usr/bin/m4: internal error detected

[John Paul Adrian Glaubitz]

Hi Daniel!

On 12/01/2017 06:08 AM, Daniel Kahn Gillmor wrote:
> ------------
> Copying file po/Makefile.in.in
> Copying file po/Makevars.template
> qemu: Unsupported syscall: -1
> m4: ../sysdeps/unix/sysv/linux/spawni.c:366: __spawnix: Assertion `ec >= 0' failed.
> /usr/bin/m4: internal error detected; please report this bug to <bug-m4@gnu.org>: Aborted
> -----------

This isn't a bug in m4 or anything architecture-specific, it's a regression
that was introduced by an upstream change in glibc [1] and mainly affects
qemu-user which we are using for m68k and sh4 [2].

While the change in glibc is most certainly correct (I don't have enough
background knowledge to comment on that), it broke qemu-user for everyone
and so far there is no possible fix in sight.

I am CC'ing this to libc-alpha in the hope that someone from glibc upstream
might give us a tip on how to resolve the issue. Not being able to use qemu-user
anymore is quite a deal breaker because lots of people use qemu-user for
debugging issues on foreign architectures which is now no longer possible.

Thanks,
Adrian

> [1] https://sourceware.org/git/?p=glibc.git;a=commit;h=4b4d4056bb154603f36c6f8845757c1012758158
> [2] https://bugs.launchpad.net/qemu/+bug/1673976

-- 
  .''`.  John Paul Adrian Glaubitz
: :' :  Debian Developer - glaubitz@debian.org
`. `'   Freie Universitaet Berlin - glaubitz@physik.fu-berlin.de
   `-    GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913

Re: /usr/bin/m4: internal error detected

[Florian Weimer]

On 12/01/2017 10:51 AM, John Paul Adrian Glaubitz wrote:
> Hi Daniel!
> 
> On 12/01/2017 06:08 AM, Daniel Kahn Gillmor wrote:
>> ------------
>> Copying file po/Makefile.in.in
>> Copying file po/Makevars.template
>> qemu: Unsupported syscall: -1
>> m4: ../sysdeps/unix/sysv/linux/spawni.c:366: __spawnix: Assertion `ec 
>> >= 0' failed.
>> /usr/bin/m4: internal error detected; please report this bug to 
>> <bug-m4@gnu.org>: Aborted
>> -----------
> 
> This isn't a bug in m4 or anything architecture-specific, it's a regression
> that was introduced by an upstream change in glibc [1] and mainly affects
> qemu-user which we are using for m68k and sh4 [2].
> 
> While the change in glibc is most certainly correct (I don't have enough
> background knowledge to comment on that), it broke qemu-user for everyone
> and so far there is no possible fix in sight.
> 
> I am CC'ing this to libc-alpha in the hope that someone from glibc upstream
> might give us a tip on how to resolve the issue. Not being able to use 
> qemu-user
> anymore is quite a deal breaker because lots of people use qemu-user for
> debugging issues on foreign architectures which is now no longer possible.

Is this <https://sourceware.org/bugzilla/show_bug.cgi?id=22273>?  If 
yes, it should be fixed in master and on the 2.25 and 2.26 branches. 
2.24 and earlier should not be affected.

Thanks,
Florian

Re: /usr/bin/m4: internal error detected

[Adhemerval Zanella]

On 01/12/2017 08:40, Florian Weimer wrote:
> On 12/01/2017 10:51 AM, John Paul Adrian Glaubitz wrote:
>> Hi Daniel!
>>
>> On 12/01/2017 06:08 AM, Daniel Kahn Gillmor wrote:
>>> ------------
>>> Copying file po/Makefile.in.in
>>> Copying file po/Makevars.template
>>> qemu: Unsupported syscall: -1
>>> m4: ../sysdeps/unix/sysv/linux/spawni.c:366: __spawnix: Assertion `ec >= 0' failed.
>>> /usr/bin/m4: internal error detected; please report this bug to <bug-m4@gnu.org>: Aborted
>>> -----------
>>
>> This isn't a bug in m4 or anything architecture-specific, it's a regression
>> that was introduced by an upstream change in glibc [1] and mainly affects
>> qemu-user which we are using for m68k and sh4 [2].
>>
>> While the change in glibc is most certainly correct (I don't have enough
>> background knowledge to comment on that), it broke qemu-user for everyone
>> and so far there is no possible fix in sight.
>>
>> I am CC'ing this to libc-alpha in the hope that someone from glibc upstream
>> might give us a tip on how to resolve the issue. Not being able to use qemu-user
>> anymore is quite a deal breaker because lots of people use qemu-user for
>> debugging issues on foreign architectures which is now no longer possible.
> 
> Is this <https://sourceware.org/bugzilla/show_bug.cgi?id=22273>?  If yes, it should be fixed in master and on the 2.25 and 2.26 branches. 2.24 and earlier should not be affected.

As I wrote in the qemu-devel mailist [1], current GLIBC won't trigger any 
assert anymore, however I am not sure if posix_spawn semantic will work
for all the expected scenarios in qemu user-mode.

Most likely any failure (sched_set{param,scheduler}, setsid, setpgid,
seteuid, any file action or execve itself) won't be advertise to main
process, since err is set 0 as default and the auxiliary process will
write to a expected shared memory to signalling an issue.  This is 
mostly likely to happen if either CLONE_VFORK (since err will
be access concurrently) or/and if CLONE_VM are not respected 

As I described in maillist, I am not very compelled to change internal
posix_spawn to use a pipe2 on Linux mainly because 

  1. it uses a slight less resources than the generic POSIX one 
     (check e83be730910c) and it works on Linux kernel as expected.

  2. I personally do not see it a good precedence because we can't
     really foretell what kind of kernel ABI or semantic the non default 
     emulation runtime will support or not, so we can't really plan to 
     set what will break or not. 

I still think relying on what a Linux kernel actually provides us is the 
right approach and I would expected qemu would need to better support 
CLONE_VM and CLONE_VFORK (WSL has improved its implementation for the 
flags it seems [2]) in user mode. 

[1] https://lists.gnu.org/archive/html/qemu-devel/2017-11/msg04729.html
[2] https://github.com/Microsoft/WSL/issues/1878

Re: /usr/bin/m4: internal error detected

[John Paul Adrian Glaubitz]

Hi Florian!

On 12/01/2017 11:40 AM, Florian Weimer wrote:
> Is this <https://sourceware.org/bugzilla/show_bug.cgi?id=22273>?  If yes, it
> should be fixed in master and on the 2.25 and 2.26 branches. 2.24 and earlier
> should not be affected.

You're right, this is indeed fixed by your patch.

I tried your patch first a few days ago but I my test setup was flawed so I
was still testing against the unpatched version of Debian's glibc package.

I now managed to test the patched version and this one does indeed work

Unpatched:

Generating locales (this might take a while)...
   en_US.UTF-8...localedef: ../sysdeps/unix/sysv/linux/spawni.c:366: __spawnix: Assertion `ec >= 0' failed.
qemu: uncaught target signal 6 (Aborted) - core dumped
Aborted
  done
Generation complete.
*** update-locale: Error: invalid locale settings:  LANG=en_US.UTF-8
(sid-m68k-sbuild)root@nofan:/tmp#

Patched:

Generating locales (this might take a while)...
   en_US.UTF-8... done
Generation complete.
(sid-m68k-sbuild)root@nofan:/tmp#

The fix is already in the packaging source of Debian's glibc [1] after
I reported the bug. But the updated package has not been uploaded
to the FTP servers yet. I'll ask Debian's glibc maintainers to push
it.

Adrian

> [1] https://anonscm.debian.org/cgit/pkg-glibc/glibc.git/commit/?id=0a94d5f3ce5785b07372a810f011c62679be910e

-- 
  .''`.  John Paul Adrian Glaubitz
: :' :  Debian Developer - glaubitz@debian.org
`. `'   Freie Universitaet Berlin - glaubitz@physik.fu-berlin.de
   `-    GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913

Re: /usr/bin/m4: internal error detected

[Andreas Schwab]

On Dez 01 2017, John Paul Adrian Glaubitz <glaubitz@physik.fu-berlin.de> wrote:

> This isn't a bug in m4 or anything architecture-specific, it's a regression
> that was introduced by an upstream change in glibc [1] and mainly affects
> qemu-user which we are using for m68k and sh4 [2].

It's a bug in qemu-linux-user, which ignores CLONE_VFORK, turning vfork
into fork.  This breaks the expected semantics of vfork (VM sharing and
blocking the child until exec).

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: /usr/bin/m4: internal error detected

[John Paul Adrian Glaubitz]

On 12/01/2017 01:18 PM, Andreas Schwab wrote:
>> This isn't a bug in m4 or anything architecture-specific, it's a regression
>> that was introduced by an upstream change in glibc [1] and mainly affects
>> qemu-user which we are using for m68k and sh4 [2].
> 
> It's a bug in qemu-linux-user, which ignores CLONE_VFORK, turning vfork
> into fork.  This breaks the expected semantics of vfork (VM sharing and
> blocking the child until exec).

Yes, I wasn't really arguing that it's a bug in QEMU as Adhemerval had already
explained. The problem was just that apparently resolving the issue in
QEMU isn't trivial as Peter Maydell mentioned in [1].

That's why I was looking for a workaround. However, I have re-tested Florian's
patch and it works for me now, it didn't when I tested it for the first time.

So, we have a workaround for the time being until the bug is resolved in QEMU.

Adrian

> [1] https://bugs.launchpad.net/qemu/+bug/1673976

-- 
  .''`.  John Paul Adrian Glaubitz
: :' :  Debian Developer - glaubitz@debian.org
`. `'   Freie Universitaet Berlin - glaubitz@physik.fu-berlin.de
   `-    GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913

Re: /usr/bin/m4: internal error detected

[Florian Weimer]

On 12/01/2017 12:58 PM, John Paul Adrian Glaubitz wrote:
> The fix is already in the packaging source of Debian's glibc [1] after
> I reported the bug. But the updated package has not been uploaded
> to the FTP servers yet. I'll ask Debian's glibc maintainers to push
> it.

Okay.  Based on the other responses, all these changes do is paper over 
bugs in the execution environment (wrong semantics of CLONE_VFORK), so 
patching glibc really isn't the proper fix.

Thanks,
Florian