the redirected stdin/out/err for new console is wrong when the gdb's stdin/out/err is already redirected

the redirected stdin/out/err for new console is wrong when the gdb's stdin/out/err is already redirected

[Roy Qu]

I'm using gdb 9.2 of  mingw.org. (not mingw-w64)
With "set new-console on", gdb will create a new console and run the
debugged program in it.
Commands like "run  < somefile" will redirect STDIN to the input of
'somefile', and the STDOUT should leave to the created console.

but when the gdb is running embedded in some IDE such as Dev-CPP and its
STDIN/STDOUT/STDERR is already redirected into the host IDE,
Console created by commands like "run < somefile" will inherit gdb's
STDOUT/STDERR value and redirect all the output to host IDE too.
(The running result should show in the debugged console, not in the IDE's
debug log window)

So I create a patch to fix it.

--- gdb-9.2-new/gdb/windows-nat.c 2020-07-11 20:13:30.000000000 +0800
+++ gdb-9.2/gdb/windows-nat.c 2020-09-28 20:21:06.515524700 +0800
@@ -2736,20 +2736,20 @@
  si.hStdInput = (HANDLE) _get_osfhandle (fd_inp);
       else if (tty != INVALID_HANDLE_VALUE)
  si.hStdInput = tty;
-      else
- si.hStdInput = GetStdHandle (STD_INPUT_HANDLE);
+      else
+ si.hStdInput = (HANDLE)0;
       if (fd_out >= 0)
  si.hStdOutput = (HANDLE) _get_osfhandle (fd_out);
       else if (tty != INVALID_HANDLE_VALUE)
  si.hStdOutput = tty;
       else
- si.hStdOutput = GetStdHandle (STD_OUTPUT_HANDLE);
+ si.hStdOutput = NULL;
       if (fd_err >= 0)
  si.hStdError = (HANDLE) _get_osfhandle (fd_err);
       else if (tty != INVALID_HANDLE_VALUE)
  si.hStdError = tty;
       else
- si.hStdError = GetStdHandle (STD_ERROR_HANDLE);
+ si.hStdError = NULL;
       si.dwFlags |= STARTF_USESTDHANDLES;
     }

Re: the redirected stdin/out/err for new console is wrong when the gdb's stdin/out/err is already redirected

[Eli Zaretskii]

> Date: Mon, 28 Sep 2020 20:21:38 +0800
> From: Roy Qu via Gdb-patches <gdb-patches@sourceware.org>
> 
> I'm using gdb 9.2 of  mingw.org. (not mingw-w64)
> With "set new-console on", gdb will create a new console and run the
> debugged program in it.
> Commands like "run  < somefile" will redirect STDIN to the input of
> 'somefile', and the STDOUT should leave to the created console.
> 
> but when the gdb is running embedded in some IDE such as Dev-CPP and its
> STDIN/STDOUT/STDERR is already redirected into the host IDE,
> Console created by commands like "run < somefile" will inherit gdb's
> STDOUT/STDERR value and redirect all the output to host IDE too.
> (The running result should show in the debugged console, not in the IDE's
> debug log window)
> 
> So I create a patch to fix it.

Thanks.

I admit that I don't understand how the patch fixes the problem, and
what other effects it could have.  Could elaborate on each part of the
changes?  (Also, (HANDLE)0 is the same as NULL, isn't it?)

Re: the redirected stdin/out/err for new console is wrong when the gdb's stdin/out/err is already redirected

[Eli Zaretskii]

[Please use Reply All to reply, so that the list is CC'ed.]

> From: Roy Qu <royqh1979@gmail.com>
> Date: Tue, 29 Sep 2020 07:48:18 +0800
> 
> Yes, (HANLDE)0 is the same as NULL. 
> 
> These code will get running when we want to run/start the inferior  and at least one of inferior's
> STDIN/STDOUT/STDERR is to be redirected.  
>    si.hStdInput / si.hStdOutput/ si.hStdError are the handle for STDIN/STDOUT/STDERR you want the child
> process ( console window) to use.    
> 
> Let's the following old code for the STDOUT:
> 
>       if (fd_out >= 0)
> si.hStdOutput = (HANDLE) _get_osfhandle (fd_out);
>       else if (tty != INVALID_HANDLE_VALUE)
> si.hStdOutput = tty;
>       else
> si.hStdOutput =  GetStdHandle (STD_OUTPUT_HANDLE)  ;
> 
> If the inferior process's STDOUT is to be redirected,  fd_out will be a valid file_descriptor (>0) to the
> redirected source, and  _get_osfhandle will
> get its corresponding windows handle.
> If the STDOUT is not to be redirected (else if)   , tty is from get_inferior_io_terminal(), which should the
> inferior's current  tty's handle; (I'm not sure
> if here is correct, because STDIN/STDOUT/STDERR should be 3 HANDLEs， but here we use the same
> one）
> If the STDOUT is not to be redirected and we don't have a valid tty already (else ), then we should let the
> inferior use new console STDOUT ( output to the screen).
> 
> But here the following old code is wrong:
> si.hStdOutput =  GetStdHandle (STD_OUTPUT_HANDLE)  ; 
> It tries to fetch the main gdb's currrent STDOUT handle and let the inferior to use it. If the main gdb's
> STDOUT is not redirected, that's ok.
> If the main gdb's STDOUT is already redirected, this will get the inferior's STDOUT redirected too, that's
> wrong.
> 
> the new code:
>   si.hStdOutput = NULL;
> will let inferior's STDOUT use the console's default output handle (output to its console's screen). That's
> what we want.
> 
> Logic of the code for STDIN/STDERR is the same.That's ALL

Thanks.

My hesitation is that I don't see the significance of NULL as the
standard handle documented in the MS docs of CreateProcess.  What am I
missing there?

Another potential issue is whether, when GDB's standard output was
redirected, we might want stdout of the inferior to be redirected to
the same file/device.  In your case, that is definitely not so, but I
wonder if there are other use cases where the desired behavior could
be different?  Running the test suite, perhaps?

Re: the redirected stdin/out/err for new console is wrong when the gdb's stdin/out/err is already redirected

[Eli Zaretskii]

[Please use Reply All to reply, so that the list is CC'ed.]

> From: Roy Qu <royqh1979@gmail.com>
> Date: Fri, 2 Oct 2020 21:58:13 +0800
> 
> https://stackoverflow.com/questions/30494945/createprocess-with-new-console-window-but-override-some-std-i-o-handles
> 
> Point 1.You are right, NULL doesn't work under windows 7.
> So we need a better solution here.

OK.

> Point 2. When GDB's standard output is not redirected (started in console) , the inferior's ouput will go to it's
> own console, not the gdb's console.

Really?  I thought this happens only if you say

  (gdb) set new-console 1

If I'm wrong, then where is the code which causes that other console
to be created for the inferior?

> So if gdb's STDOUT is redirected,
> I think the inferior's output should go to it's own console ,too.

I'm asking whether we always want that.  I hope others who have
experience with running GDB on MS-Windows will chime in and comment on
this part.

Thanks.

Re: the redirected stdin/out/err for new console is wrong when the gdb's stdin/out/err is already redirected

[Eli Zaretskii]

[Please use Reply All to reply, so that the list is CC'ed.]

> From: Roy Qu <royqh1979@gmail.com>
> Date: Sat, 3 Oct 2020 00:22:46 +0800
> 
> Eli Zaretskii <eliz@gnu.org> 于2020年10月2日周五 下午10:49写道：
> 
>  [Please use Reply All to reply, so that the list is CC'ed.]

Please do, you continue replying only to me, which requires me to
quote the entire message sent by you to the list, and also prevents
others from replying in a timely manner.

> 
>  > From: Roy Qu <royqh1979@gmail.com>
>  > Date: Fri, 2 Oct 2020 21:58:13 +0800
>  > 
>  >
>  https://stackoverflow.com/questions/30494945/createprocess-with-new-console-window-but-override-some-std-i-o-handles
>  
>  > 
>  > Point 1.You are right, NULL doesn't work under windows 7.
>  > So we need a better solution here.
> 
>  OK.
> 
>  > Point 2. When GDB's standard output is not redirected (started in console) , the inferior's ouput will
>  go to it's
>  > own console, not the gdb's console.
> 
>  Really?  I thought this happens only if you say
> 
>    (gdb) set new-console 1
> 
>  If I'm wrong, then where is the code which causes that other console
>  to be created for the inferior?
> 
> Yes. Can we add a check here to see if there should be a new console ?

Probably, but I hope someone else will explain how to do that.

And if we do have such a test, how will that affect the issue you
would like to solve?  Does the problem only happen when the inferior
has a separate console?

Re: the redirected stdin/out/err for new console is wrong when the gdb's stdin/out/err is already redirected

[Roy Qu]

Eli Zaretskii <eliz@gnu.org> 于2020年10月3日周六 上午3:50写道：

> [Please use Reply All to reply, so that the list is CC'ed.]
>
> > From: Roy Qu <royqh1979@gmail.com>
> > Date: Sat, 3 Oct 2020 00:22:46 +0800
> >
> > Eli Zaretskii <eliz@gnu.org> 于2020年10月2日周五 下午10:49写道：
> >
> >  [Please use Reply All to reply, so that the list is CC'ed.]
>
> Please do, you continue replying only to me, which requires me to
> quote the entire message sent by you to the list, and also prevents
> others from replying in a timely manner.
>
> Sorry.

> >
> >  > From: Roy Qu <royqh1979@gmail.com>
> >  > Date: Fri, 2 Oct 2020 21:58:13 +0800
> >  >
> >  >
> >
> https://stackoverflow.com/questions/30494945/createprocess-with-new-console-window-but-override-some-std-i-o-handles
> >
> >  >
> >  > Point 1.You are right, NULL doesn't work under windows 7.
> >  > So we need a better solution here.
> >
> >  OK.
> >
> >  > Point 2. When GDB's standard output is not redirected (started in
> console) , the inferior's ouput will
> >  go to it's
> >  > own console, not the gdb's console.
> >
> >  Really?  I thought this happens only if you say
> >
> >    (gdb) set new-console 1
> >
> >  If I'm wrong, then where is the code which causes that other console
> >  to be created for the inferior?
> >
> > Yes. Can we add a check here to see if there should be a new console ?
>
> Probably, but I hope someone else will explain how to do that.
>
> And if we do have such a test, how will that affect the issue you
> would like to solve?  Does the problem only happen when the inferior
> has a separate console?
>

I think only console programs use STDIN/STDOUT for I/O. So only console
programs have this problem.
And if the inferior don't have its own console (it redirect its
STDIN/STDOUT to gdb's main console) ,  there's no such problem either.