Screen clearing in CMD without "Legacy Console Mode"

Screen clearing in CMD without "Legacy Console Mode"

[Kevin Locke]

Hi All,

I'm investigating an issue in Git for Windows[^1], which also affects
Cygwin.  The issue is that, when using CMD (i.e. Command Prompt) on
Windows 10 1703 or above with "Legacy Console Mode"[^2] disabled, if
TERM=cygwin is set in the environment, the console is not cleared when
vi exits.  To demonstrate, with Cygwin 3.2.0, in CMD with "Legacy
Console Mode" disabled:

cd C:\cygwin64
set TERM=cygwin
bin\vi etc\bash.bashrc
:q

After exiting vi, the console window has not been cleared and content
from etc\bash.bashrc remains visible, making further use of the console
difficult until cleared.

The difference appears to result from enabling
SetConsoleMode(ENABLE_VIRTUAL_TERMINAL_PROCESSING) and
SetConsoleMode(ENABLE_VIRTUAL_TERMINAL_INPUT)[^3] to behave like
xterm-256color when supported[^4] (they are not supported in "Legacy
Console Mode").

I'm not too familiar with TTY/PTY handling, much less Cygwin on top of
CMD.  It's not clear to me if the alternate screen buffer behaves
differently in CMD than xterm, whether Cygwin has any responsibility, or
if the issue is in how CMD handles ANSI escape sequences.  Johannes
Schindelin suggested it may make sense not to enable virtual terminal
processing when TERM=cygwin is set.[^5]  Thoughts?

Any advice or assistance for how to address this issue would be
appreciated.

Thanks,
Kevin

[^1]: https://github.com/git-for-windows/git/issues/3177
[^2]: https://docs.microsoft.com/windows/console/legacymode
[^3]: https://cygwin.com/git/?p=newlib-cygwin.git;a=blob;f=winsup/cygwin/fhandler_console.cc;h=ebe89fc9f8fd9fcb8102ee9884b8b91f04e88fb1;hb=5eb232ede72e394e280d1a23131267c7002d2c52#l464
[^4]: https://cygwin.com/git/?p=newlib-cygwin.git;a=blob;f=winsup/cygwin/fhandler_console.cc;h=ebe89fc9f8fd9fcb8102ee9884b8b91f04e88fb1;hb=5eb232ede72e394e280d1a23131267c7002d2c52#l1327
[^5]: https://github.com/git-for-windows/git/issues/3177#issuecomment-828494565

Re: Screen clearing in CMD without "Legacy Console Mode"

[Takashi Yano]

On Fri, 30 Apr 2021 08:25:12 -0600
Kevin Locke wrote:
> Hi All,
> 
> I'm investigating an issue in Git for Windows[^1], which also affects
> Cygwin.  The issue is that, when using CMD (i.e. Command Prompt) on
> Windows 10 1703 or above with "Legacy Console Mode"[^2] disabled, if
> TERM=cygwin is set in the environment, the console is not cleared when
> vi exits.  To demonstrate, with Cygwin 3.2.0, in CMD with "Legacy
> Console Mode" disabled:
> 
> cd C:\cygwin64
> set TERM=cygwin
> bin\vi etc\bash.bashrc
> :q
> 
> After exiting vi, the console window has not been cleared and content
> from etc\bash.bashrc remains visible, making further use of the console
> difficult until cleared.
> 
> The difference appears to result from enabling
> SetConsoleMode(ENABLE_VIRTUAL_TERMINAL_PROCESSING) and
> SetConsoleMode(ENABLE_VIRTUAL_TERMINAL_INPUT)[^3] to behave like
> xterm-256color when supported[^4] (they are not supported in "Legacy
> Console Mode").
> 
> I'm not too familiar with TTY/PTY handling, much less Cygwin on top of
> CMD.  It's not clear to me if the alternate screen buffer behaves
> differently in CMD than xterm, whether Cygwin has any responsibility, or
> if the issue is in how CMD handles ANSI escape sequences.  Johannes
> Schindelin suggested it may make sense not to enable virtual terminal
> processing when TERM=cygwin is set.[^5]  Thoughts?
> 
> Any advice or assistance for how to address this issue would be
> appreciated.
> 
> Thanks,
> Kevin
> 
> [^1]: https://github.com/git-for-windows/git/issues/3177
> [^2]: https://docs.microsoft.com/windows/console/legacymode
> [^3]: https://cygwin.com/git/?p=newlib-cygwin.git;a=blob;f=winsup/cygwin/fhandler_console.cc;h=ebe89fc9f8fd9fcb8102ee9884b8b91f04e88fb1;hb=5eb232ede72e394e280d1a23131267c7002d2c52#l464
> [^4]: https://cygwin.com/git/?p=newlib-cygwin.git;a=blob;f=winsup/cygwin/fhandler_console.cc;h=ebe89fc9f8fd9fcb8102ee9884b8b91f04e88fb1;hb=5eb232ede72e394e280d1a23131267c7002d2c52#l1327
> [^5]: https://github.com/git-for-windows/git/issues/3177#issuecomment-828494565

Why on earth do you want to set TERM=cygwin?
If you don't set TERM=cygwin, TERM is automatically set to
xterm-256color, in which the issue does not occur.

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

Re: Screen clearing in CMD without "Legacy Console Mode"

[Kevin Locke]

On Fri, 2021-04-30 at 23:53 +0900, Takashi Yano wrote:
> On Fri, 30 Apr 2021 08:25:12 -0600 Kevin Locke wrote:
>> I'm investigating an issue in Git for Windows[^1], which also affects
>> Cygwin.  The issue is that, when using CMD (i.e. Command Prompt) on
>> Windows 10 1703 or above with "Legacy Console Mode"[^2] disabled, if
>> TERM=cygwin is set in the environment, the console is not cleared when
>> vi exits.  To demonstrate, with Cygwin 3.2.0, in CMD with "Legacy
>> Console Mode" disabled:
>> 
>> cd C:\cygwin64
>> set TERM=cygwin
>> bin\vi etc\bash.bashrc
>> :q
>> 
>> After exiting vi, the console window has not been cleared and content
>> from etc\bash.bashrc remains visible, making further use of the console
>> difficult until cleared.
> 
> Why on earth do you want to set TERM=cygwin?
> If you don't set TERM=cygwin, TERM is automatically set to
> xterm-256color, in which the issue does not occur.

Unfortunately, I am not clear on that myself.[^6]  According to Johannes
Schindelin[^7]:

> We specifically set TERM so that Cygwin (or more correctly, the MSYS2 runtime) uses ANSI sequences...

Hopefully he (or another of the Git for Windows contributors) can add
more specifics.

Thanks,
Kevin

[^6]: https://github.com/git-for-windows/git/issues/3177#issuecomment-828507812
[^7]: https://github.com/git-for-windows/git/issues/3177#issuecomment-826834976

Re: Screen clearing in CMD without "Legacy Console Mode"

[Brian Inglis]

On 2021-04-30 08:53, Takashi Yano via Cygwin wrote:
> On Fri, 30 Apr 2021 08:25:12 -0600
> Kevin Locke wrote:
>> Hi All,
>>
>> I'm investigating an issue in Git for Windows[^1], which also affects
>> Cygwin.  The issue is that, when using CMD (i.e. Command Prompt) on
>> Windows 10 1703 or above with "Legacy Console Mode"[^2] disabled, if
>> TERM=cygwin is set in the environment, the console is not cleared when
>> vi exits.  To demonstrate, with Cygwin 3.2.0, in CMD with "Legacy
>> Console Mode" disabled:
>>
>> cd C:\cygwin64
>> set TERM=cygwin
>> bin\vi etc\bash.bashrc
>> :q
>>
>> After exiting vi, the console window has not been cleared and content
>> from etc\bash.bashrc remains visible, making further use of the console
>> difficult until cleared.
>>
>> The difference appears to result from enabling
>> SetConsoleMode(ENABLE_VIRTUAL_TERMINAL_PROCESSING) and
>> SetConsoleMode(ENABLE_VIRTUAL_TERMINAL_INPUT)[^3] to behave like
>> xterm-256color when supported[^4] (they are not supported in "Legacy
>> Console Mode").
>>
>> I'm not too familiar with TTY/PTY handling, much less Cygwin on top of
>> CMD.  It's not clear to me if the alternate screen buffer behaves
>> differently in CMD than xterm, whether Cygwin has any responsibility, or
>> if the issue is in how CMD handles ANSI escape sequences.  Johannes
>> Schindelin suggested it may make sense not to enable virtual terminal
>> processing when TERM=cygwin is set.[^5]  Thoughts?
>>
>> Any advice or assistance for how to address this issue would be
>> appreciated.
>>
>> Thanks,
>> Kevin
>>
>> [^1]: https://github.com/git-for-windows/git/issues/3177
>> [^2]: https://docs.microsoft.com/windows/console/legacymode
>> [^3]: https://cygwin.com/git/?p=newlib-cygwin.git;a=blob;f=winsup/cygwin/fhandler_console.cc;h=ebe89fc9f8fd9fcb8102ee9884b8b91f04e88fb1;hb=5eb232ede72e394e280d1a23131267c7002d2c52#l464
>> [^4]: https://cygwin.com/git/?p=newlib-cygwin.git;a=blob;f=winsup/cygwin/fhandler_console.cc;h=ebe89fc9f8fd9fcb8102ee9884b8b91f04e88fb1;hb=5eb232ede72e394e280d1a23131267c7002d2c52#l1327
>> [^5]: https://github.com/git-for-windows/git/issues/3177#issuecomment-828494565
> 
> Why on earth do you want to set TERM=cygwin?
> If you don't set TERM=cygwin, TERM is automatically set to
> xterm-256color, in which the issue does not occur.

Was this a change made to Cygwin and from which version?

-- 
Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada

This email may be disturbing to some readers as it contains
too much technical detail. Reader discretion is advised.
[Data in binary units and prefixes, physical quantities in SI.]

Re: Screen clearing in CMD without "Legacy Console Mode"

[Takashi Yano]

On Fri, 30 Apr 2021 10:40:33 -0600
Brian Inglis wrote:
> On 2021-04-30 08:53, Takashi Yano via Cygwin wrote:
> > On Fri, 30 Apr 2021 08:25:12 -0600
> > Kevin Locke wrote:
> >> Hi All,
> >>
> >> I'm investigating an issue in Git for Windows[^1], which also affects
> >> Cygwin.  The issue is that, when using CMD (i.e. Command Prompt) on
> >> Windows 10 1703 or above with "Legacy Console Mode"[^2] disabled, if
> >> TERM=cygwin is set in the environment, the console is not cleared when
> >> vi exits.  To demonstrate, with Cygwin 3.2.0, in CMD with "Legacy
> >> Console Mode" disabled:
> >>
> >> cd C:\cygwin64
> >> set TERM=cygwin
> >> bin\vi etc\bash.bashrc
> >> :q
> >>
> >> After exiting vi, the console window has not been cleared and content
> >> from etc\bash.bashrc remains visible, making further use of the console
> >> difficult until cleared.
> >>
> >> The difference appears to result from enabling
> >> SetConsoleMode(ENABLE_VIRTUAL_TERMINAL_PROCESSING) and
> >> SetConsoleMode(ENABLE_VIRTUAL_TERMINAL_INPUT)[^3] to behave like
> >> xterm-256color when supported[^4] (they are not supported in "Legacy
> >> Console Mode").
> >>
> >> I'm not too familiar with TTY/PTY handling, much less Cygwin on top of
> >> CMD.  It's not clear to me if the alternate screen buffer behaves
> >> differently in CMD than xterm, whether Cygwin has any responsibility, or
> >> if the issue is in how CMD handles ANSI escape sequences.  Johannes
> >> Schindelin suggested it may make sense not to enable virtual terminal
> >> processing when TERM=cygwin is set.[^5]  Thoughts?
> >>
> >> Any advice or assistance for how to address this issue would be
> >> appreciated.
> >>
> >> Thanks,
> >> Kevin
> >>
> >> [^1]: https://github.com/git-for-windows/git/issues/3177
> >> [^2]: https://docs.microsoft.com/windows/console/legacymode
> >> [^3]: https://cygwin.com/git/?p=newlib-cygwin.git;a=blob;f=winsup/cygwin/fhandler_console.cc;h=ebe89fc9f8fd9fcb8102ee9884b8b91f04e88fb1;hb=5eb232ede72e394e280d1a23131267c7002d2c52#l464
> >> [^4]: https://cygwin.com/git/?p=newlib-cygwin.git;a=blob;f=winsup/cygwin/fhandler_console.cc;h=ebe89fc9f8fd9fcb8102ee9884b8b91f04e88fb1;hb=5eb232ede72e394e280d1a23131267c7002d2c52#l1327
> >> [^5]: https://github.com/git-for-windows/git/issues/3177#issuecomment-828494565
> > 
> > Why on earth do you want to set TERM=cygwin?
> > If you don't set TERM=cygwin, TERM is automatically set to
> > xterm-256color, in which the issue does not occur.
> 
> Was this a change made to Cygwin and from which version?

IIRC, from cygwin 3.1.0.

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

Re: Screen clearing in CMD without "Legacy Console Mode"

[Jack Adrian Zappa]

On Fri, Apr 30, 2021 at 1:31 PM Takashi Yano via Cygwin <cygwin@cygwin.com>
wrote:

> Why on earth do you want to set TERM=cygwin?
> If you don't set TERM=cygwin, TERM is automatically set to
> xterm-256color, in which the issue does not occur.
>

Might be because for the longest time, if you didn't set it to cygwin, it
would mess up the line editing in the shell if your line would exceed a
certain length (I think 80 chars)

Re: Screen clearing in CMD without "Legacy Console Mode"

[Johannes Schindelin]

Hi,

On Fri, 30 Apr 2021, Kevin Locke wrote:

> On Fri, 2021-04-30 at 23:53 +0900, Takashi Yano wrote:
> > On Fri, 30 Apr 2021 08:25:12 -0600 Kevin Locke wrote:
> >> I'm investigating an issue in Git for Windows[^1], which also affects
> >> Cygwin.  The issue is that, when using CMD (i.e. Command Prompt) on
> >> Windows 10 1703 or above with "Legacy Console Mode"[^2] disabled, if
> >> TERM=cygwin is set in the environment, the console is not cleared when
> >> vi exits.  To demonstrate, with Cygwin 3.2.0, in CMD with "Legacy
> >> Console Mode" disabled:
> >>
> >> cd C:\cygwin64
> >> set TERM=cygwin
> >> bin\vi etc\bash.bashrc
> >> :q
> >>
> >> After exiting vi, the console window has not been cleared and content
> >> from etc\bash.bashrc remains visible, making further use of the console
> >> difficult until cleared.
> >
> > Why on earth do you want to set TERM=cygwin?
> > If you don't set TERM=cygwin, TERM is automatically set to
> > xterm-256color, in which the issue does not occur.

TERM=cygwin would be correct right until the time when you toggle
`ENABLE_VIRTUAL_TERMINAL_PROCESSING` under our feet.

That is, when the Cygwin process is called, this flag is not set, so
`TERM=xterm-256color` would be incorrect. And then when the Cygwin process
returns, the flag is set, and the calling process should somehow guess
that `TERM` should be set differently.

Oh, and you probably expect the caller to then figure out whether the flag
was _actually_ toggled, just in case we're running on a Windows 10 version
that is too old to understand that flag.

If you expect CMD to be no longer used after the Cygwin process returns,
that might be a valid world view. But that is probably not a very tenable
world view.

If Cygwin is unable to handle `TERM=cygwin` correctly when
`ENABLE_VIRTUAL_TERMINAL_PROCESSING` is toggled, then Cygwin should
definitely, absolutely, with 100% certainty _not_ toggle that flag when
`TERM` is already set to `cygwin`!

Ciao,
Johannes

> Unfortunately, I am not clear on that myself.[^6]  According to Johannes
> Schindelin[^7]:
>
> > We specifically set TERM so that Cygwin (or more correctly, the MSYS2 runtime) uses ANSI sequences...
>
> Hopefully he (or another of the Git for Windows contributors) can add
> more specifics.
>
> Thanks,
> Kevin
>
> [^6]: https://github.com/git-for-windows/git/issues/3177#issuecomment-828507812
> [^7]: https://github.com/git-for-windows/git/issues/3177#issuecomment-826834976
>

Re: Screen clearing in CMD without "Legacy Console Mode"

[Takashi Yano]

On Wed, 5 May 2021 15:07:04 +0200 (CEST)
Johannes Schindelin wrote:
> Hi,
> 
> On Fri, 30 Apr 2021, Kevin Locke wrote:
> 
> > On Fri, 2021-04-30 at 23:53 +0900, Takashi Yano wrote:
> > > On Fri, 30 Apr 2021 08:25:12 -0600 Kevin Locke wrote:
> > >> I'm investigating an issue in Git for Windows[^1], which also affects
> > >> Cygwin.  The issue is that, when using CMD (i.e. Command Prompt) on
> > >> Windows 10 1703 or above with "Legacy Console Mode"[^2] disabled, if
> > >> TERM=cygwin is set in the environment, the console is not cleared when
> > >> vi exits.  To demonstrate, with Cygwin 3.2.0, in CMD with "Legacy
> > >> Console Mode" disabled:
> > >>
> > >> cd C:\cygwin64
> > >> set TERM=cygwin
> > >> bin\vi etc\bash.bashrc
> > >> :q
> > >>
> > >> After exiting vi, the console window has not been cleared and content
> > >> from etc\bash.bashrc remains visible, making further use of the console
> > >> difficult until cleared.
> > >
> > > Why on earth do you want to set TERM=cygwin?
> > > If you don't set TERM=cygwin, TERM is automatically set to
> > > xterm-256color, in which the issue does not occur.
> 
> TERM=cygwin would be correct right until the time when you toggle
> `ENABLE_VIRTUAL_TERMINAL_PROCESSING` under our feet.
> 
> That is, when the Cygwin process is called, this flag is not set, so
> `TERM=xterm-256color` would be incorrect. And then when the Cygwin process
> returns, the flag is set, and the calling process should somehow guess
> that `TERM` should be set differently.
> 
> Oh, and you probably expect the caller to then figure out whether the flag
> was _actually_ toggled, just in case we're running on a Windows 10 version
> that is too old to understand that flag.
> 
> If you expect CMD to be no longer used after the Cygwin process returns,
> that might be a valid world view. But that is probably not a very tenable
> world view.

If you don't set TERM, cygwin sets right value automatically.
That is, in the case in which ENABLE_VIRTUAL_TERMINAL_PROCESSING
is not supported, TERM will be set to 'cygwin', otherwise, it
will be set to 'xterm-256color'.

Why do you want to set TERM? What is wrong with not setting TERM
manually? If you set TERM incorrectly, it is very natural result
that apps does not work correctly.

> If Cygwin is unable to handle `TERM=cygwin` correctly when
> `ENABLE_VIRTUAL_TERMINAL_PROCESSING` is toggled, then Cygwin should
> definitely, absolutely, with 100% certainty _not_ toggle that flag when
> `TERM` is already set to `cygwin`!

I don't think so AT ALL. If someone sets TERM to the value other 
than 'cygwin' and 'xterm-256color', what do you expect cygwin to
behave? For example, if TERM is set to 'linux', how should cygwin
behave, do you think?

P.S.
mintty overwrites the TERM at startup even if user sets TERM manually.
Shouldn't cygwin console also overwrite TERM to the correct value?

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

Re: Screen clearing in CMD without "Legacy Console Mode"

[Takashi Yano]

On Thu, 6 May 2021 18:12:38 +0900
Takashi Yano via Cygwin> wrote:
> On Wed, 5 May 2021 15:07:04 +0200 (CEST)
> Johannes Schindelin wrote:
> > Hi,
> > 
> > On Fri, 30 Apr 2021, Kevin Locke wrote:
> > 
> > > On Fri, 2021-04-30 at 23:53 +0900, Takashi Yano wrote:
> > > > On Fri, 30 Apr 2021 08:25:12 -0600 Kevin Locke wrote:
> > > >> I'm investigating an issue in Git for Windows[^1], which also affects
> > > >> Cygwin.  The issue is that, when using CMD (i.e. Command Prompt) on
> > > >> Windows 10 1703 or above with "Legacy Console Mode"[^2] disabled, if
> > > >> TERM=cygwin is set in the environment, the console is not cleared when
> > > >> vi exits.  To demonstrate, with Cygwin 3.2.0, in CMD with "Legacy
> > > >> Console Mode" disabled:
> > > >>
> > > >> cd C:\cygwin64
> > > >> set TERM=cygwin
> > > >> bin\vi etc\bash.bashrc
> > > >> :q
> > > >>
> > > >> After exiting vi, the console window has not been cleared and content
> > > >> from etc\bash.bashrc remains visible, making further use of the console
> > > >> difficult until cleared.
> > > >
> > > > Why on earth do you want to set TERM=cygwin?
> > > > If you don't set TERM=cygwin, TERM is automatically set to
> > > > xterm-256color, in which the issue does not occur.
> > 
> > TERM=cygwin would be correct right until the time when you toggle
> > `ENABLE_VIRTUAL_TERMINAL_PROCESSING` under our feet.
> > 
> > That is, when the Cygwin process is called, this flag is not set, so
> > `TERM=xterm-256color` would be incorrect. And then when the Cygwin process
> > returns, the flag is set, and the calling process should somehow guess
> > that `TERM` should be set differently.
> > 
> > Oh, and you probably expect the caller to then figure out whether the flag
> > was _actually_ toggled, just in case we're running on a Windows 10 version
> > that is too old to understand that flag.
> > 
> > If you expect CMD to be no longer used after the Cygwin process returns,
> > that might be a valid world view. But that is probably not a very tenable
> > world view.
> 
> If you don't set TERM, cygwin sets right value automatically.
> That is, in the case in which ENABLE_VIRTUAL_TERMINAL_PROCESSING
> is not supported, TERM will be set to 'cygwin', otherwise, it
> will be set to 'xterm-256color'.
> 
> Why do you want to set TERM? What is wrong with not setting TERM
> manually? If you set TERM incorrectly, it is very natural result
> that apps does not work correctly.
> 
> > If Cygwin is unable to handle `TERM=cygwin` correctly when
> > `ENABLE_VIRTUAL_TERMINAL_PROCESSING` is toggled, then Cygwin should
> > definitely, absolutely, with 100% certainty _not_ toggle that flag when
> > `TERM` is already set to `cygwin`!
> 
> I don't think so AT ALL. If someone sets TERM to the value other 
> than 'cygwin' and 'xterm-256color', what do you expect cygwin to
> behave? For example, if TERM is set to 'linux', how should cygwin
> behave, do you think?
> 
> P.S.
> mintty overwrites the TERM at startup even if user sets TERM manually.
> Shouldn't cygwin console also overwrite TERM to the correct value?

WSL also overwrites TERM value to 'xterm-256color'.

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>