nested popen()'s lead to fgets() failure on 64-bit only

nested popen()'s lead to fgets() failure on 64-bit only

[David Levine]

Hi,

The program below works just fine on 32-bit Cygwin:
  buf1 = /usr/bin/file.exe
  buf2 = /usr/bin/file.exe: application/x-dosexec

But on 64-bit Cygwin (2.831), the second fgets() fails:
  buf1 = /usr/bin/file.exe
fgets2: No error

It only seems to happen with nested popen()'s.  Is it due to a
problem in the Cygwin lib?

Thanks,
David


#include <stdio.h>

int
main() {
  FILE *f1, *f2;

  if ((f1 = popen("ls /usr/bin/file.exe", "r"))) {
    char buf1[BUFSIZ];

    while (fgets(buf1, sizeof buf1, f1)) {
      printf("  buf1 = %s", buf1);

      if ((f2 = popen("file --mime-type /usr/bin/file.exe", "r"))) {
        char buf2[BUFSIZ];

        /* This fgets call fails on 64-bit Cygwin, with "No error". */
        if (fgets(buf2, sizeof buf2, f2)) {
          printf("  buf2 = %s", buf2);
        } else {
          perror("fgets2");
        }

        pclose(f2);
      } else {
        perror("popen2");
      }
    }

    pclose(f1);
  } else {
    perror("popen1");
  }

  return 0;
}

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: nested popen()'s lead to fgets() failure on 64-bit only

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 1494 bytes --]

On Jan 12 21:15, David Levine wrote:
> Hi,
> 
> The program below works just fine on 32-bit Cygwin:
>   buf1 = /usr/bin/file.exe
>   buf2 = /usr/bin/file.exe: application/x-dosexec
> 
> But on 64-bit Cygwin (2.831), the second fgets() fails:

That's the version number of the setup-x86_64.exe tool, not the version
number of Cygwin.  Try `uname -r'.

>   buf1 = /usr/bin/file.exe
> fgets2: No error
> 
> It only seems to happen with nested popen()'s.  Is it due to a
> problem in the Cygwin lib?

Works fine for me on Cygwin 1.7.27:

  $ uname -rm
  1.7.27(0.271/5/3) x86_64
  $ file --version
  file-5.13
  magic file from /usr/share/misc/magic
  $ ./popen-test
    buf1 = /usr/bin/file.exe
    buf2 = /usr/bin/file.exe: application/x-dosexec
  $

The fact that fgets returns NULL doesn't mean an error occured.  fgets
also returns NULL at EOF, meaning no output in this case, which you
should test, too:

  if (fgets(buf2, sizeof buf2, f2))
    printf("  buf2 = %s", buf2);
  else if (feof (f2))
    fputs ("fgets2 EOF\n", stderr);
  else
    perror("fgets2");

Maybe you're calling the wrong file command?  What's your $PATH set to?
What if you change your popen call to run /usr/bin/file.exe with full
path?

  if ((f2 = popen("/usr/bin/file --mime-type /usr/bin/file.exe", "r")))


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: nested popen()'s lead to fgets() failure on 64-bit only

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 890 bytes --]

On Jan 13 22:41, David Levine wrote:
> Corinna wrote:
> 
> > Your PID values are pretty suspicious.  Even on a busy machine they
> > should seldomly have more than 4 digits, let alone 6.  The PIDs are only
> > getting high if process and other handles are hold and not closed by
> > another process.  This looks a lot like a BLODA problem.  Did you
> > check against http://cygwin.com/faq/faq.html#faq.using.bloda?
> 
> That's it.  The machine needs to be rebooted every once in
> a while because it runs out of memory.
> 
> I checked that FAQ but don't see the culprit.

Hmm.  Virus?  NSA keylogger?

> Should popen() report the failure?

It can't.  It doesn't see this failure and it's totally unexpected.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: nested popen()'s lead to fgets() failure on 64-bit only

[David Levine]

Corinna wrote:

> Your PID values are pretty suspicious.  Even on a busy machine they
> should seldomly have more than 4 digits, let alone 6.  The PIDs are only
> getting high if process and other handles are hold and not closed by
> another process.  This looks a lot like a BLODA problem.  Did you
> check against http://cygwin.com/faq/faq.html#faq.using.bloda?

That's it.  The machine needs to be rebooted every once in
a while because it runs out of memory.

I checked that FAQ but don't see the culprit.

Should popen() report the failure?

Thank you for your help.

David

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: nested popen()'s lead to fgets() failure on 64-bit only

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 1136 bytes --]

On Jan 13 11:38, David Levine wrote:
> Corinna wrote:
> 
> > Not really.  I have no idea why this fails for you.  Does an strace
> > show anything suspicious?
> 
> Yes, there's an exception, it looks like in WFMO.  It's
> happens when closing handles after forking sh.exe for the
> second popen().  strace excerpt is below.
> 
> The exception didn't happen after the first popen().  And I
> tried running with a minimal environment (env -i PATH=/usr/bin),
> which didn't help.
> 
> David
> 
> 
>     1       1 [main] sh (426948) **********************************************
                          ^^^^^^^^
Your PID values are pretty suspicious.  Even on a busy machine they
should seldomly have more than 4 digits, let alone 6.  The PIDs are only
getting high if process and other handles are hold and not closed by
another process.  This looks a lot like a BLODA problem.  Did you
check against http://cygwin.com/faq/faq.html#faq.using.bloda?


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: nested popen()'s lead to fgets() failure on 64-bit only

[David Levine]

Corinna wrote:

> Not really.  I have no idea why this fails for you.  Does an strace
> show anything suspicious?

Yes, there's an exception, it looks like in WFMO.  It's
happens when closing handles after forking sh.exe for the
second popen().  strace excerpt is below.

The exception didn't happen after the first popen().  And I
tried running with a minimal environment (env -i PATH=/usr/bin),
which didn't help.

David


    1       1 [main] sh (426948) **********************************************
   39      40 [main] sh (426948) Program name: C:\cygwin64\bin\sh.exe (windows pid 426948)
   16      56 [main] sh (426948) OS version:   Windows NT-6.1
   15      71 [main] sh (426948) **********************************************
   59     130 [main] sh (426948) sigprocmask: 0 = sigprocmask (0, 0x1802BCC08, 0x0)
   64     194 [main] sh 426948 child_copy: cygheap - hp 0x114 low 0x1802DB410, high 0x1802E2CD8, res 1
   20     214 [main] sh 426948 child_copy: done
   51     265 [main] sh 426948 open_shared: name shared.5, n 5, shared 0x180030000 (wanted 0x180030000), h 0x6C, *m 6
   24     289 [main] sh 426948 user_heap_info::init: heap base 0x600000000, heap top 0x600000000, heap size 0x20000000 (536870912)
   22     311 [main] sh 426948 open_shared: name (null), n 1, shared 0x180020000 (wanted 0x180020000), h 0x60, *m 6
   13     324 [main] sh 426948 user_info::create: opening user shared for '' at 0x180020000
   16     340 [main] sh 426948 user_info::create: user shared version AB1FCCE8
   35     375 [main] sh (426948) open_shared: name cygpid.426948, n 426948, shared 0x180010000 (wanted 0x180010000), h 0x74, *m 6
   21     396 [main] sh 426948 pinfo::thisproc: myself dwProcessId 426948
   19     415 [main] sh 426948 fhandler_pty_slave::fixup_after_fork: /dev/pty0 inherited, usecount 4
   19     434 [main] sh 426948 dtable::fixup_close: closing fd 1 since it is an archetype
   18     452 [main] sh 426948 fhandler_base::close_with_arch: line 1140:  /dev/pty0<0x1802DF208> usecount + -1 = 3
   16     468 [main] sh 426948 fhandler_base::close_with_arch: not closing archetype
   18     486 [main] sh 426948 fhandler_pty_slave::fixup_after_fork: /dev/pty0 inherited, usecount 3
   20     506 [main] sh 426948 fhandler_base::fixup_after_exec: here for '/dev/fd/pipe:[3]'
   16     522 [main] sh 426948 close: close(3)
--- Process 426948, exception c0000005 at 00000001800F21EF
13416 1272862 [main] nested_popen 426736 child_info::sync: pid 426948, WFMO returned 1, exit_code 0xC0000005, res 0
    0 1272862 [waitproc] nested_popen 426736 pinfo::maybe_set_exit_code_from_windows: pid 426948, exit value - old 0x0, windows 0xC0000005, cygwin 0x800000B
   53 1272915 [waitproc] nested_popen 426736 sig_send: sendsig 0x80, pid 426736, signal 20, its_me 1
   22 1272937 [waitproc] nested_popen 426736 sig_send: Not waiting for sigcomplete.  its_me 1 signal 20
   22 1272959 [waitproc] nested_popen 426736 sig_send: returning 0x0 from sending signal 20
   13 1272972 [main] nested_popen 426736 fcntl64: fcntl(1, 2, ...)
   19 1272991 [waitproc] nested_popen 426736 proc_waiter: exiting wait thread for pid 426948
   20 1273011 [sig] nested_popen 426736 sigpacket::process: signal 20 processing
   35 1273046 [sig] nested_popen 426736 init_cygheap::find_tls: sig 20
  -31 1273015 [main] nested_popen 426736 fcntl64: 0 = fcntl(1, 2, 0x0)
   65 1273080 [main] nested_popen 426736 close: close(5)
  -14 1273066 [sig] nested_popen 426736 sigpacket::process: using tls 0x22CE00
   70 1273136 [sig] nested_popen 426736 sigpacket::process: signal 20 default is currently ignore
    1 1273137 [main] nested_popen 426736 fhandler_base::close: closing 'pipe:[5]' handle 0xE0
   17 1273154 [sig] nested_popen 426736 sigpacket::process: returning 1
   32 1273186 [sig] nested_popen 426736 proc_subproc: args: 5, 0
   19 1273205 [sig] nested_popen 426736 proc_subproc: looking for processes to reap, nprocs 2
    2 1273207 [main] nested_popen 426736 close: 0 = close(5)
   16 1273223 [sig] nested_popen 426736 proc_subproc: finished processing terminated/stopped child

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: nested popen()'s lead to fgets() failure on 64-bit only

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 1885 bytes --]

On Jan 13 09:35, David Levine wrote:
> Corinna wrote:
> 
> > That's the version number of the setup-x86_64.exe tool, not the version
> > number of Cygwin.  Try `uname -r'.
> 
> 1.7.27(0.271/5/3)
> 
> > Maybe you're calling the wrong file command?  What's your $PATH set to?
> > What if you change your popen call to run /usr/bin/file.exe with full
> > path?
> > 
> >   if ((f2 = popen("/usr/bin/file --mime-type /usr/bin/file.exe", "r")))
> 
> My PATH picks /usr/bin/file, but just to be sure I hard coded it,
> and also used it for the first popen(), and also set PATH to /usr/bin:
> 
> $ gcc -o nested_popen nested_popen.c -Wall -Wextra && \
>   PATH=/usr/bin ./nested_popen
>   buf1 = /usr/bin/file.exe: application/x-dosexec
> popen: No error
> fgets2 EOF

Looks like this here:

  $ ./popen-test
    buf1 = /usr/bin/file.exe: application/x-dosexec
  popen: No error
    buf2 = /usr/bin/file.exe: application/x-dosexec

> It shouldn't be at EOF in this case.  If I change the
> second popen() to open anything, e.g.,
> popen("garbage,really", "r"), it still reports success.

Looks like this here:

  $ ./popen-test
    buf1 = /usr/bin/file.exe: application/x-dosexec
  popen: No error
  /bin/sh: garbage.really: command not found
  fgets2 EOF

Same as on Linux:

  $ ./popen-test
    buf1 = /usr/bin/file: application/x-executable
  popen: Success
    buf2 = /usr/bin/file: application/x-executable

  $ ./popen-test
    buf1 = /usr/bin/file: application/x-executable
  popen: Success
  sh: garbage.really: command not found
  fgets2 EOF

> We're getting closer?

Not really.  I have no idea why this fails for you.  Does an strace
show anything suspicious?


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: nested popen()'s lead to fgets() failure on 64-bit only

[David Levine]

Corinna wrote:

> That's the version number of the setup-x86_64.exe tool, not the version
> number of Cygwin.  Try `uname -r'.

1.7.27(0.271/5/3)

> Maybe you're calling the wrong file command?  What's your $PATH set to?
> What if you change your popen call to run /usr/bin/file.exe with full
> path?
> 
>   if ((f2 = popen("/usr/bin/file --mime-type /usr/bin/file.exe", "r")))

My PATH picks /usr/bin/file, but just to be sure I hard coded it,
and also used it for the first popen(), and also set PATH to /usr/bin:

$ gcc -o nested_popen nested_popen.c -Wall -Wextra && \
  PATH=/usr/bin ./nested_popen
  buf1 = /usr/bin/file.exe: application/x-dosexec
popen: No error
fgets2 EOF

It shouldn't be at EOF in this case.  If I change the
second popen() to open anything, e.g.,
popen("garbage,really", "r"), it still reports success.
We're getting closer?

David


#include <stdio.h>

int
main() {
  FILE *f1, *f2;

  if ((f1 = popen("file --mime-type /usr/bin/file.exe", "r"))) {
    char buf1[BUFSIZ];

    while (fgets(buf1, sizeof buf1, f1)) {
      printf("  buf1 = %s", buf1);

      if ((f2 = popen("/usr/bin/file.exe --mime-type /usr/bin/file.exe",
                      "r"))) {
        char buf2[BUFSIZ];

        perror("popen");

        /* This fgets call fails on 64-bit Cygwin 1,7.27(0.271/5/3),
           with "No error". */
        if (fgets(buf2, sizeof buf2, f2)) {
          printf("  buf2 = %s", buf2);
        } else if (feof (f2)) {
          fputs ("fgets2 EOF\n", stderr);
        } else {
          perror("fgets2");
        }

        pclose(f2);
      } else {
        perror("popen2");
      }
    }

    pclose(f1);
  } else {
    perror("popen1");
  }

  return 0;
}

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple