"kubectl exec" in Cygwin gets "Upgrade request required", but not in cmd shell

"kubectl exec" in Cygwin gets "Upgrade request required", but not in cmd shell

[David Karr]

I've been using kubectl in Cygwin on Windows 10 for quite a while, to
communicate to our in-house k8s clusters. I often use "kubectl exec" to
open a shell in a container or directly execute a shell command.  This has
worked perfectly fine for a long time.

A couple of days ago, I discovered that all of these attempts were failing
with "Upgrade request required".  I hadn't upgraded kubectl or Cygwin in
quite a while. I doubt our clusters had a k8s upgrade, but it's entirely
possible.

A colleague of mine has a very similar desktop configuration (Windows 10,
Cygwin), and he's not seeing this symptom.

I noticed that when I ran "kubectl exec" with max verbosity, it shows the
resulting "curl" command that it runs. I tried that resulting command, and
it results in the same response. I then tried updating my Cygwin tools and
retesting, no change.

I then took the entire resulting "kubectl exec" command line and ran it in
a "cmd" shell.  No problem at all.  No error.

I know I haven't provided much useful information yet. I wanted to get an
initial response before I started providing those diagnostics. Is there a
clear issue here that I'm not aware of?

Re: "kubectl exec" in Cygwin gets "Upgrade request required", but not in cmd shell

[Marco Atzeri]

On 13.06.2020 20:53, David Karr via Cygwin wrote:
> I've been using kubectl in Cygwin on Windows 10 for quite a while, to
> communicate to our in-house k8s clusters. I often use "kubectl exec" to
> open a shell in a container or directly execute a shell command.  This has
> worked perfectly fine for a long time.
> 
> A couple of days ago, I discovered that all of these attempts were failing
> with "Upgrade request required".  I hadn't upgraded kubectl or Cygwin in
> quite a while. I doubt our clusters had a k8s upgrade, but it's entirely
> possible.
> 
> A colleague of mine has a very similar desktop configuration (Windows 10,
> Cygwin), and he's not seeing this symptom.
> 
> I noticed that when I ran "kubectl exec" with max verbosity, it shows the
> resulting "curl" command that it runs. I tried that resulting command, and
> it results in the same response. I then tried updating my Cygwin tools and
> retesting, no change.
> 
> I then took the entire resulting "kubectl exec" command line and ran it in
> a "cmd" shell.  No problem at all.  No error.
> 
> I know I haven't provided much useful information yet. I wanted to get an
> initial response before I started providing those diagnostics. Is there a
> clear issue here that I'm not aware of?
> --

from where is kubectl coming from ?

In cygwin I found only a kubectl.py in the ansible package

Re: "kubectl exec" in Cygwin gets "Upgrade request required", but not in cmd shell

[David Karr]

On Sat, Jun 13, 2020 at 10:31 PM Marco Atzeri via Cygwin wrote:

> On 13.06.2020 20:53, David Karr via Cygwin wrote:
> > I've been using kubectl in Cygwin on Windows 10 for quite a while, to
> > communicate to our in-house k8s clusters. I often use "kubectl exec" to
> > open a shell in a container or directly execute a shell command.  This
> has
> > worked perfectly fine for a long time.
> >
> > A couple of days ago, I discovered that all of these attempts were
> failing
> > with "Upgrade request required".  I hadn't upgraded kubectl or Cygwin in
> > quite a while. I doubt our clusters had a k8s upgrade, but it's entirely
> > possible.
> >
> > A colleague of mine has a very similar desktop configuration (Windows 10,
> > Cygwin), and he's not seeing this symptom.
> >
> > I noticed that when I ran "kubectl exec" with max verbosity, it shows the
> > resulting "curl" command that it runs. I tried that resulting command,
> and
> > it results in the same response. I then tried updating my Cygwin tools
> and
> > retesting, no change.
> >
> > I then took the entire resulting "kubectl exec" command line and ran it
> in
> > a "cmd" shell.  No problem at all.  No error.
> >
> > I know I haven't provided much useful information yet. I wanted to get an
> > initial response before I started providing those diagnostics. Is there a
> > clear issue here that I'm not aware of?
> > --
>
> from where is kubectl coming from ?
>
> In cygwin I found only a kubectl.py in the ansible package
>

It's from here:
https://kubernetes.io/docs/tasks/tools/install-kubectl/#install-kubectl-on-windows
.

Re: "kubectl exec" in Cygwin gets "Upgrade request required", but not in cmd shell

[Marco Atzeri]

On 14.06.2020 08:12, David Karr wrote:
> 
> 
> On Sat, Jun 13, 2020 at 10:31 PM Marco Atzeri via Cygwin wrote:
> 
>     On 13.06.2020 20:53, David Karr via Cygwin wrote:
>      > I've been using kubectl in Cygwin on Windows 10 for quite a while, to
>      > communicate to our in-house k8s clusters. I often use "kubectl
>     exec" to
>      > open a shell in a container or directly execute a shell command. 
>     This has
>      > worked perfectly fine for a long time.
>      >
>      > A couple of days ago, I discovered that all of these attempts
>     were failing
>      > with "Upgrade request required".  I hadn't upgraded kubectl or
>     Cygwin in
>      > quite a while. I doubt our clusters had a k8s upgrade, but it's
>     entirely
>      > possible.
>      >
>      > A colleague of mine has a very similar desktop configuration
>     (Windows 10,
>      > Cygwin), and he's not seeing this symptom.
>      >
>      > I noticed that when I ran "kubectl exec" with max verbosity, it
>     shows the
>      > resulting "curl" command that it runs. I tried that resulting
>     command, and
>      > it results in the same response. I then tried updating my Cygwin
>     tools and
>      > retesting, no change.
>      >
>      > I then took the entire resulting "kubectl exec" command line and
>     ran it in
>      > a "cmd" shell.  No problem at all.  No error.
>      >
>      > I know I haven't provided much useful information yet. I wanted
>     to get an
>      > initial response before I started providing those diagnostics. Is
>     there a
>      > clear issue here that I'm not aware of?
>      > --
> 
>     from where is kubectl coming from ?
> 
>     In cygwin I found only a kubectl.py in the ansible package
> 
> 
> It's from here: 
> https://kubernetes.io/docs/tasks/tools/install-kubectl/#install-kubectl-on-windows 
> .

so it is NOT a cygwin program.

If the warning is coming about curl, it is likely
that using from cygwin you are using the cygwin curl
and from CMD the windows one


$ which -a curl
/usr/bin/curl
/cygdrive/c/WINDOWS/system32/curl


$ /cygdrive/c/WINDOWS/system32/curl -V
curl 7.55.1 (Windows) libcurl/7.55.1 WinSSL
Release-Date: 2017-11-14, security patched: 2019-11-05
Protocols: dict file ftp ftps http https imap imaps pop3 pop3s smtp 
smtps telnet
  tftp
Features: AsynchDNS IPv6 Largefile SSPI Kerberos SPNEGO NTLM SSL

$ /usr/bin/curl -V
curl 7.66.0 (x86_64-pc-cygwin) libcurl/7.66.0 OpenSSL/1.1.1f zlib/1.2.11 
brotli/1.0.7 libidn2/2.2.0 libpsl/0.21.0 (+libidn2/2.0.4) 
libssh/0.8.7/openssl/zlib nghttp2/1.37.0
Release-Date: 2019-09-11
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps 
pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: AsynchDNS brotli Debug GSS-API HTTP2 HTTPS-proxy IDN IPv6 
Kerberos Largefile libz Metalink NTLM NTLM_WB PSL SPNEGO SSL TLS-SRP 
TrackMemory UnixSockets


the support Forum https://discuss.kubernetes.io/
is probably the most indicate place for guidance

Regards
Marco

Re: "kubectl exec" in Cygwin gets "Upgrade request required", but not in cmd shell

[David Karr]

On Sun, Jun 14, 2020 at 2:25 AM Marco Atzeri wrote:

> On 14.06.2020 08:12, David Karr wrote:
> >
> >
> > On Sat, Jun 13, 2020 at 10:31 PM Marco Atzeri via Cygwin wrote:
> >
> >     On 13.06.2020 20:53, David Karr via Cygwin wrote:
> >      > I've been using kubectl in Cygwin on Windows 10 for quite a
> while, to
> >      > communicate to our in-house k8s clusters. I often use "kubectl
> >     exec" to
> >      > open a shell in a container or directly execute a shell command.
> >     This has
> >      > worked perfectly fine for a long time.
> >      >
> >      > A couple of days ago, I discovered that all of these attempts
> >     were failing
> >      > with "Upgrade request required".  I hadn't upgraded kubectl or
> >     Cygwin in
> >      > quite a while. I doubt our clusters had a k8s upgrade, but it's
> >     entirely
> >      > possible.
> >      >
> >      > A colleague of mine has a very similar desktop configuration
> >     (Windows 10,
> >      > Cygwin), and he's not seeing this symptom.
> >      >
> >      > I noticed that when I ran "kubectl exec" with max verbosity, it
> >     shows the
> >      > resulting "curl" command that it runs. I tried that resulting
> >     command, and
> >      > it results in the same response. I then tried updating my Cygwin
> >     tools and
> >      > retesting, no change.
> >      >
> >      > I then took the entire resulting "kubectl exec" command line and
> >     ran it in
> >      > a "cmd" shell.  No problem at all.  No error.
> >      >
> >      > I know I haven't provided much useful information yet. I wanted
> >     to get an
> >      > initial response before I started providing those diagnostics. Is
> >     there a
> >      > clear issue here that I'm not aware of?
> >      > --
> >
> >     from where is kubectl coming from ?
> >
> >     In cygwin I found only a kubectl.py in the ansible package
> >
> >
> > It's from here:
> >
> https://kubernetes.io/docs/tasks/tools/install-kubectl/#install-kubectl-on-windows
> > .
>
> so it is NOT a cygwin program.
>
> If the warning is coming about curl, it is likely
> that using from cygwin you are using the cygwin curl
> and from CMD the windows one
>
>
> $ which -a curl
> /usr/bin/curl
> /cygdrive/c/WINDOWS/system32/curl
>
>
> $ /cygdrive/c/WINDOWS/system32/curl -V
> curl 7.55.1 (Windows) libcurl/7.55.1 WinSSL
> Release-Date: 2017-11-14, security patched: 2019-11-05
> Protocols: dict file ftp ftps http https imap imaps pop3 pop3s smtp
> smtps telnet
>   tftp
> Features: AsynchDNS IPv6 Largefile SSPI Kerberos SPNEGO NTLM SSL
>
> $ /usr/bin/curl -V
> curl 7.66.0 (x86_64-pc-cygwin) libcurl/7.66.0 OpenSSL/1.1.1f zlib/1.2.11
> brotli/1.0.7 libidn2/2.2.0 libpsl/0.21.0 (+libidn2/2.0.4)
> libssh/0.8.7/openssl/zlib nghttp2/1.37.0
> Release-Date: 2019-09-11
> Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps
> pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp
> Features: AsynchDNS brotli Debug GSS-API HTTP2 HTTPS-proxy IDN IPv6
> Kerberos Largefile libz Metalink NTLM NTLM_WB PSL SPNEGO SSL TLS-SRP
> TrackMemory UnixSockets
>
>
> the support Forum https://discuss.kubernetes.io/
> is probably the most indicate place for guidance
>
> Regards
> Marco
>

I thought it was obvious that it was not working because it was calling the
Cygwin curl. I wouldn't have posted here if that wasn't obvious to me.

And since I'm well aware of the k8s community, I already posted questions
about this in the appropriate place, before I posted here.

What I was hoping to get here was some indication or thoughts on why a
process using Windows curl doesn't have a problem, but does have a problem
when using Cygwin Curl. This isn't likely something that Cygwin curl is
doing "wrong", it's just that it's doing something different.

If it matters, the following is an elided version of the resulting curl
command:

    curl -k -v -XPOST  -H "User-Agent: kubectl.exe/v1.18.0 (windows/amd64)
kubernetes/9e99141" -H "Authorization: Bearer ..." -H
"X-Stream-Protocol-Version: v4.channel.k8s.io" -H
"X-Stream-Protocol-Version: v3.channel.k8s.io" -H
"X-Stream-Protocol-Version: v2.channel.k8s.io" -H
"X-Stream-Protocol-Version: channel.k8s.io" 'https://
.../api/v1/namespaces/.../pods/.../exec?command=%2Fbin%2Fls&container=...&stderr=true&stdin=true&stdout=true'

I can't tell from the logging what request body it sent. It's possible it
didn't send any.

Re: "kubectl exec" in Cygwin gets "Upgrade request required", but not in cmd shell

[Marco Atzeri]

On 14.06.2020 17:38, David Karr wrote:
> 
> On Sun, Jun 14, 2020 at 2:25 AM Marco Atzeri wrote:
> 
>     On 14.06.2020 08:12, David Karr wrote:
>      >
>      >
>      > On Sat, Jun 13, 2020 at 10:31 PM Marco Atzeri via Cygwin wrote:
>      >
>      >     On 13.06.2020 20:53, David Karr via Cygwin wrote:
>      >      > I've been using kubectl in Cygwin on Windows 10 for quite
>     a while, to
>      >      > communicate to our in-house k8s clusters. I often use "kubectl
>      >     exec" to
>      >      > open a shell in a container or directly execute a shell
>     command.
>      >     This has
>      >      > worked perfectly fine for a long time.
>      >      >
>      >      > A couple of days ago, I discovered that all of these attempts
>      >     were failing
>      >      > with "Upgrade request required".  I hadn't upgraded kubectl or
>      >     Cygwin in
>      >      > quite a while. I doubt our clusters had a k8s upgrade, but
>     it's
>      >     entirely
>      >      > possible.
>      >      >
>      >      > A colleague of mine has a very similar desktop configuration
>      >     (Windows 10,
>      >      > Cygwin), and he's not seeing this symptom.
>      >      >
>      >      > I noticed that when I ran "kubectl exec" with max
>     verbosity, it
>      >     shows the
>      >      > resulting "curl" command that it runs. I tried that resulting
>      >     command, and
>      >      > it results in the same response. I then tried updating my
>     Cygwin
>      >     tools and
>      >      > retesting, no change.
>      >      >
>      >      > I then took the entire resulting "kubectl exec" command
>     line and
>      >     ran it in
>      >      > a "cmd" shell.  No problem at all.  No error.
>      >      >
>      >      > I know I haven't provided much useful information yet. I
>     wanted
>      >     to get an
>      >      > initial response before I started providing those
>     diagnostics. Is
>      >     there a
>      >      > clear issue here that I'm not aware of?
>      >      > --
>      >
>      >     from where is kubectl coming from ?
>      >
>      >     In cygwin I found only a kubectl.py in the ansible package
>      >
>      >
>      > It's from here:
>      >
>     https://kubernetes.io/docs/tasks/tools/install-kubectl/#install-kubectl-on-windows
> 
>      > .
> 
>     so it is NOT a cygwin program.
> 
>     If the warning is coming about curl, it is likely
>     that using from cygwin you are using the cygwin curl
>     and from CMD the windows one
> 
> 
>     $ which -a curl
>     /usr/bin/curl
>     /cygdrive/c/WINDOWS/system32/curl

>     the support Forum https://discuss.kubernetes.io/
>     is probably the most indicate place for guidance
> 
>     Regards
>     Marco
> 
> 
> I thought it was obvious that it was not working because it was calling 
> the Cygwin curl. I wouldn't have posted here if that wasn't obvious to me.

Obvious not so much to me, evidently ;-)

> And since I'm well aware of the k8s community, I already posted 
> questions about this in the appropriate place, before I posted here.
> 
> What I was hoping to get here was some indication or thoughts on why a 
> process using Windows curl doesn't have a problem, but does have a 
> problem when using Cygwin Curl. This isn't likely something that Cygwin 
> curl is doing "wrong", it's just that it's doing something different.

the Cygwin curl was changed last time on 18 Sep 2019.
So it is not something directly depending from the package, maybe from
the dependencing libraries.

Comparing the help and version outputs the cygwin one is coming from a 
more recent version and has more features than the windows one ;
the only major difference I see is that the windows version
produces output with CRLF termination.

It is possible that kubectl is doing a version check and it misleading
reports a different version as older one.

> If it matters, the following is an elided version of the resulting curl 
> command:
> 
>      curl -k -v -XPOST  -H "User-Agent: kubectl.exe/v1.18.0 
> (windows/amd64) kubernetes/9e99141" -H "Authorization: Bearer ..." -H 
> "X-Stream-Protocol-Version: v4.channel.k8s.io 
> <http://v4.channel.k8s.io>" -H "X-Stream-Protocol-Version: 
> v3.channel.k8s.io <http://v3.channel.k8s.io>" -H 
> "X-Stream-Protocol-Version: v2.channel.k8s.io 
> <http://v2.channel.k8s.io>" -H "X-Stream-Protocol-Version: 
> channel.k8s.io <http://channel.k8s.io>" 
> 'https://.../api/v1/namespaces/.../pods/.../exec?command=%2Fbin%2Fls&container=...&stderr=true&stdin=true&stdout=true'
> 
> I can't tell from the logging what request body it sent. It's possible 
> it didn't send any.

all the options used are basic and present in both versions.
May be CRLF vs LF is present also on POST method

Regards
Marco

Re: "kubectl exec" in Cygwin gets "Upgrade request required", but not in cmd shell

[Brian Inglis]

On 2020-06-14 09:38, David Karr via Cygwin wrote:
> On Sun, Jun 14, 2020 at 2:25 AM Marco Atzeri wrote:
>> On 14.06.2020 08:12, David Karr wrote:
>>> On Sat, Jun 13, 2020 at 10:31 PM Marco Atzeri via Cygwin wrote:
>>>>     On 13.06.2020 20:53, David Karr via Cygwin wrote:
>>>>> I've been using kubectl in Cygwin on Windows 10 for quite a while,
>>>>> to communicate to our in-house k8s clusters. I often use "kubectl 
>>>>> exec" to open a shell in a container or directly execute a shell
>>>>> command.
>>>>> This has worked perfectly fine for a long time.
>>>>> A couple of days ago, I discovered that all of these attempts were
>>>>> failing with "Upgrade request required".  I hadn't upgraded kubectl
>>>>> or Cygwin in quite a while. I doubt our clusters had a k8s upgrade,
>>>>> but it's entirely possible.
>>>>> A colleague of mine has a very similar desktop configuration
>>>>> (Windows 10, Cygwin), and he's not seeing this symptom.
>>>>> I noticed that when I ran "kubectl exec" with max verbosity, it shows
>>>>> the resulting "curl" command that it runs. I tried that resulting 
>>>>> command, and it results in the same response. I then tried updating
>>>>> my Cygwin tools and retesting, no change.>>>>> I then took the entire resulting "kubectl exec" command line and ran
>>>>> it in a "cmd" shell.  No problem at all.  No error.
>>>>> I know I haven't provided much useful information yet. I wanted to
>>>>> get an initial response before I started providing those diagnostics.
>>>>> Is there a clear issue here that I'm not aware of?

>>> from where is kubectl coming from ?
>>> In cygwin I found only a kubectl.py in the ansible package

>>>> It's from here:
>>>> https://kubernetes.io/docs/tasks/tools/install-kubectl/#install-kubectl-on-windows

>> so it is NOT a cygwin program.
>> If the warning is coming about curl, it is likely
>> that using from cygwin you are using the cygwin curl
>> and from CMD the windows one
>> $ which -a curl
>> /usr/bin/curl
>> /cygdrive/c/WINDOWS/system32/curl
>> $ /cygdrive/c/WINDOWS/system32/curl -V
>> curl 7.55.1 (Windows) libcurl/7.55.1 WinSSL
>> Release-Date: 2017-11-14, security patched: 2019-11-05
>> Protocols: dict file ftp ftps http https imap imaps pop3 pop3s smtp smtps
>> telnet tftp
>> Features: AsynchDNS IPv6 Largefile SSPI Kerberos SPNEGO NTLM SSL
>>
>> $ /usr/bin/curl -V
>> curl 7.66.0 (x86_64-pc-cygwin) libcurl/7.66.0 OpenSSL/1.1.1f zlib/1.2.11
>> brotli/1.0.7 libidn2/2.2.0 libpsl/0.21.0 (+libidn2/2.0.4)
>> libssh/0.8.7/openssl/zlib nghttp2/1.37.0
>> Release-Date: 2019-09-11
>> Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps
>> pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp
>> Features: AsynchDNS brotli Debug GSS-API HTTP2 HTTPS-proxy IDN IPv6
>> Kerberos Largefile libz Metalink NTLM NTLM_WB PSL SPNEGO SSL TLS-SRP
>> TrackMemory UnixSockets
>> the support Forum https://discuss.kubernetes.io/
>> is probably the most indicate place for guidance

> I thought it was obvious that it was not working because it was calling the
> Cygwin curl. I wouldn't have posted here if that wasn't obvious to me.
> And since I'm well aware of the k8s community, I already posted questions
> about this in the appropriate place, before I posted here.
> What I was hoping to get here was some indication or thoughts on why a
> process using Windows curl doesn't have a problem, but does have a problem
> when using Cygwin Curl. This isn't likely something that Cygwin curl is
> doing "wrong", it's just that it's doing something different.
> If it matters, the following is an elided version of the resulting curl
> command:
>     curl -k -v -XPOST  -H "User-Agent: kubectl.exe/v1.18.0 (windows/amd64)
> kubernetes/9e99141" -H "Authorization: Bearer ..." -H
> "X-Stream-Protocol-Version: v4.channel.k8s.io" -H
> "X-Stream-Protocol-Version: v3.channel.k8s.io" -H
> "X-Stream-Protocol-Version: v2.channel.k8s.io" -H
> "X-Stream-Protocol-Version: channel.k8s.io" 'https://
> .../api/v1/namespaces/.../pods/.../exec?command=%2Fbin%2Fls&container=...&stderr=true&stdin=true&stdout=true'
> I can't tell from the logging what request body it sent. It's possible it
> didn't send any.

We can't tell, not knowing who you are, and from what you are posting, what may
or may not be obvious to you, where you are seeing "Upgrade request required",
or where that may be coming from.
That you need diagnostic help indicates that, what may appear obvious to you,
may not be the case, as it is often our assumptions which lead us astray, and we
get daily proof that we are imperfect, which is why most of seek to talk over
and explain our issues to inanimate objects or colleagues e.g. talk to the
rubber duck, teddy bear, plush hippo, etc. (My shelf holds two ducks and a
pelican awarded by former projects.)

For help with Cygwin, we need to see *whole* commands and all the output between
the shell prompts, preferably with context, in this case including PATH under
Cygwin and MS Windows and/or program paths typed or invoked.

You may retain privacy and security by substituting variables e.g. env vars $VAR
to sanitize sensitive information: to avoid problems I use a script to do so
that logs are never written containing sensitive content.

-- 
Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada

This email may be disturbing to some readers as it contains
too much technical detail. Reader discretion is advised.
[Data in IEC units and prefixes, physical quantities in SI.]

Re: "kubectl exec" in Cygwin gets "Upgrade request required", but not in cmd shell

[David Karr]

On Sun, Jun 14, 2020 at 10:20 AM Brian Inglis <
Brian.Inglis@systematicsw.ab.ca> wrote:

> On 2020-06-14 09:38, David Karr via Cygwin wrote:
> > On Sun, Jun 14, 2020 at 2:25 AM Marco Atzeri wrote:
> >> On 14.06.2020 08:12, David Karr wrote:
> >>> On Sat, Jun 13, 2020 at 10:31 PM Marco Atzeri via Cygwin wrote:
> >>>>     On 13.06.2020 20:53, David Karr via Cygwin wrote:
> >>>>> I've been using kubectl in Cygwin on Windows 10 for quite a while,
> >>>>> to communicate to our in-house k8s clusters. I often use "kubectl
> >>>>> exec" to open a shell in a container or directly execute a shell
> >>>>> command.
> >>>>> This has worked perfectly fine for a long time.
> >>>>> A couple of days ago, I discovered that all of these attempts were
> >>>>> failing with "Upgrade request required".  I hadn't upgraded kubectl
> >>>>> or Cygwin in quite a while. I doubt our clusters had a k8s upgrade,
> >>>>> but it's entirely possible.
> >>>>> A colleague of mine has a very similar desktop configuration
> >>>>> (Windows 10, Cygwin), and he's not seeing this symptom.
> >>>>> I noticed that when I ran "kubectl exec" with max verbosity, it shows
> >>>>> the resulting "curl" command that it runs. I tried that resulting
> >>>>> command, and it results in the same response. I then tried updating
> >>>>> my Cygwin tools and retesting, no change.>>>>> I then took the
> entire resulting "kubectl exec" command line and ran
> >>>>> it in a "cmd" shell.  No problem at all.  No error.
> >>>>> I know I haven't provided much useful information yet. I wanted to
> >>>>> get an initial response before I started providing those diagnostics.
> >>>>> Is there a clear issue here that I'm not aware of?
>
> >>> from where is kubectl coming from ?
> >>> In cygwin I found only a kubectl.py in the ansible package
>
> >>>> It's from here:
> >>>>
> https://kubernetes.io/docs/tasks/tools/install-kubectl/#install-kubectl-on-windows
>
> >> so it is NOT a cygwin program.
> >> If the warning is coming about curl, it is likely
> >> that using from cygwin you are using the cygwin curl
> >> and from CMD the windows one
> >> $ which -a curl
> >> /usr/bin/curl
> >> /cygdrive/c/WINDOWS/system32/curl
> >> $ /cygdrive/c/WINDOWS/system32/curl -V
> >> curl 7.55.1 (Windows) libcurl/7.55.1 WinSSL
> >> Release-Date: 2017-11-14, security patched: 2019-11-05
> >> Protocols: dict file ftp ftps http https imap imaps pop3 pop3s smtp
> smtps
> >> telnet tftp
> >> Features: AsynchDNS IPv6 Largefile SSPI Kerberos SPNEGO NTLM SSL
> >>
> >> $ /usr/bin/curl -V
> >> curl 7.66.0 (x86_64-pc-cygwin) libcurl/7.66.0 OpenSSL/1.1.1f zlib/1.2.11
> >> brotli/1.0.7 libidn2/2.2.0 libpsl/0.21.0 (+libidn2/2.0.4)
> >> libssh/0.8.7/openssl/zlib nghttp2/1.37.0
> >> Release-Date: 2019-09-11
> >> Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps
> >> pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp
> >> Features: AsynchDNS brotli Debug GSS-API HTTP2 HTTPS-proxy IDN IPv6
> >> Kerberos Largefile libz Metalink NTLM NTLM_WB PSL SPNEGO SSL TLS-SRP
> >> TrackMemory UnixSockets
> >> the support Forum https://discuss.kubernetes.io/
> >> is probably the most indicate place for guidance
>
> > I thought it was obvious that it was not working because it was calling
> the
> > Cygwin curl. I wouldn't have posted here if that wasn't obvious to me.
> > And since I'm well aware of the k8s community, I already posted questions
> > about this in the appropriate place, before I posted here.
> > What I was hoping to get here was some indication or thoughts on why a
> > process using Windows curl doesn't have a problem, but does have a
> problem
> > when using Cygwin Curl. This isn't likely something that Cygwin curl is
> > doing "wrong", it's just that it's doing something different.
> > If it matters, the following is an elided version of the resulting curl
> > command:
> >     curl -k -v -XPOST  -H "User-Agent: kubectl.exe/v1.18.0
> (windows/amd64)
> > kubernetes/9e99141" -H "Authorization: Bearer ..." -H
> > "X-Stream-Protocol-Version: v4.channel.k8s.io" -H
> > "X-Stream-Protocol-Version: v3.channel.k8s.io" -H
> > "X-Stream-Protocol-Version: v2.channel.k8s.io" -H
> > "X-Stream-Protocol-Version: channel.k8s.io" 'https://
> >
> .../api/v1/namespaces/.../pods/.../exec?command=%2Fbin%2Fls&container=...&stderr=true&stdin=true&stdout=true'
> > I can't tell from the logging what request body it sent. It's possible it
> > didn't send any.
>
> We can't tell, not knowing who you are, and from what you are posting,
> what may
> or may not be obvious to you, where you are seeing "Upgrade request
> required",
> or where that may be coming from.
> That you need diagnostic help indicates that, what may appear obvious to
> you,
> may not be the case, as it is often our assumptions which lead us astray,
> and we
> get daily proof that we are imperfect, which is why most of seek to talk
> over
> and explain our issues to inanimate objects or colleagues e.g. talk to the
> rubber duck, teddy bear, plush hippo, etc. (My shelf holds two ducks and a
> pelican awarded by former projects.)
>
> For help with Cygwin, we need to see *whole* commands and all the output
> between
> the shell prompts, preferably with context, in this case including PATH
> under
> Cygwin and MS Windows and/or program paths typed or invoked.
>
> You may retain privacy and security by substituting variables e.g. env
> vars $VAR
> to sanitize sensitive information: to avoid problems I use a script to do
> so
> that logs are never written containing sensitive content.
>

Acknowledged. The "Upgrade request required" is coming from the kubernetes
server.  At this point, I think I'm going to need to understand exactly
what that message means (other threads talking about this didn't make that
clear). Someone in that community did respond to my question, not with an
answer, but with a query for more information. I'll see what I can find out.

Re: "kubectl exec" in Cygwin gets "Upgrade request required", but not in cmd shell

[Brian Inglis]

On 2020-06-14 12:16, David Karr via Cygwin wrote:
> On Sun, Jun 14, 2020 at 10:20 AM Brian Inglis <
> Brian.Inglis@systematicsw.ab.ca> wrote:
> 
>> On 2020-06-14 09:38, David Karr via Cygwin wrote:
>>> On Sun, Jun 14, 2020 at 2:25 AM Marco Atzeri wrote:
>>>> On 14.06.2020 08:12, David Karr wrote:
>>>>> On Sat, Jun 13, 2020 at 10:31 PM Marco Atzeri via Cygwin wrote:
>>>>>>     On 13.06.2020 20:53, David Karr via Cygwin wrote:
>>>>>>> I've been using kubectl in Cygwin on Windows 10 for quite a while,
>>>>>>> to communicate to our in-house k8s clusters. I often use "kubectl
>>>>>>> exec" to open a shell in a container or directly execute a shell
>>>>>>> command.
>>>>>>> This has worked perfectly fine for a long time.
>>>>>>> A couple of days ago, I discovered that all of these attempts were
>>>>>>> failing with "Upgrade request required".  I hadn't upgraded kubectl
>>>>>>> or Cygwin in quite a while. I doubt our clusters had a k8s upgrade,
>>>>>>> but it's entirely possible.
>>>>>>> A colleague of mine has a very similar desktop configuration
>>>>>>> (Windows 10, Cygwin), and he's not seeing this symptom.
>>>>>>> I noticed that when I ran "kubectl exec" with max verbosity, it shows
>>>>>>> the resulting "curl" command that it runs. I tried that resulting
>>>>>>> command, and it results in the same response. I then tried updating
>>>>>>> my Cygwin tools and retesting, no change.>>>>> I then took the
>> entire resulting "kubectl exec" command line and ran
>>>>>>> it in a "cmd" shell.  No problem at all.  No error.
>>>>>>> I know I haven't provided much useful information yet. I wanted to
>>>>>>> get an initial response before I started providing those diagnostics.
>>>>>>> Is there a clear issue here that I'm not aware of?
>>
>>>>> from where is kubectl coming from ?
>>>>> In cygwin I found only a kubectl.py in the ansible package
>>
>>>>>> It's from here:
>>>>>>
>> https://kubernetes.io/docs/tasks/tools/install-kubectl/#install-kubectl-on-windows
>>
>>>> so it is NOT a cygwin program.
>>>> If the warning is coming about curl, it is likely
>>>> that using from cygwin you are using the cygwin curl
>>>> and from CMD the windows one
>>>> $ which -a curl
>>>> /usr/bin/curl
>>>> /cygdrive/c/WINDOWS/system32/curl
>>>> $ /cygdrive/c/WINDOWS/system32/curl -V
>>>> curl 7.55.1 (Windows) libcurl/7.55.1 WinSSL
>>>> Release-Date: 2017-11-14, security patched: 2019-11-05
>>>> Protocols: dict file ftp ftps http https imap imaps pop3 pop3s smtp
>> smtps
>>>> telnet tftp
>>>> Features: AsynchDNS IPv6 Largefile SSPI Kerberos SPNEGO NTLM SSL
>>>>
>>>> $ /usr/bin/curl -V
>>>> curl 7.66.0 (x86_64-pc-cygwin) libcurl/7.66.0 OpenSSL/1.1.1f zlib/1.2.11
>>>> brotli/1.0.7 libidn2/2.2.0 libpsl/0.21.0 (+libidn2/2.0.4)
>>>> libssh/0.8.7/openssl/zlib nghttp2/1.37.0
>>>> Release-Date: 2019-09-11
>>>> Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps
>>>> pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp
>>>> Features: AsynchDNS brotli Debug GSS-API HTTP2 HTTPS-proxy IDN IPv6
>>>> Kerberos Largefile libz Metalink NTLM NTLM_WB PSL SPNEGO SSL TLS-SRP
>>>> TrackMemory UnixSockets
>>>> the support Forum https://discuss.kubernetes.io/
>>>> is probably the most indicate place for guidance
>>
>>> I thought it was obvious that it was not working because it was calling
>> the
>>> Cygwin curl. I wouldn't have posted here if that wasn't obvious to me.
>>> And since I'm well aware of the k8s community, I already posted questions
>>> about this in the appropriate place, before I posted here.
>>> What I was hoping to get here was some indication or thoughts on why a
>>> process using Windows curl doesn't have a problem, but does have a
>> problem
>>> when using Cygwin Curl. This isn't likely something that Cygwin curl is
>>> doing "wrong", it's just that it's doing something different.
>>> If it matters, the following is an elided version of the resulting curl
>>> command:
>>>     curl -k -v -XPOST  -H "User-Agent: kubectl.exe/v1.18.0
>> (windows/amd64)
>>> kubernetes/9e99141" -H "Authorization: Bearer ..." -H
>>> "X-Stream-Protocol-Version: v4.channel.k8s.io" -H
>>> "X-Stream-Protocol-Version: v3.channel.k8s.io" -H
>>> "X-Stream-Protocol-Version: v2.channel.k8s.io" -H
>>> "X-Stream-Protocol-Version: channel.k8s.io" 'https://
>>>
>> .../api/v1/namespaces/.../pods/.../exec?command=%2Fbin%2Fls&container=...&stderr=true&stdin=true&stdout=true'
>>> I can't tell from the logging what request body it sent. It's possible it
>>> didn't send any.
>>
>> We can't tell, not knowing who you are, and from what you are posting,
>> what may
>> or may not be obvious to you, where you are seeing "Upgrade request
>> required",
>> or where that may be coming from.
>> That you need diagnostic help indicates that, what may appear obvious to
>> you,
>> may not be the case, as it is often our assumptions which lead us astray,
>> and we
>> get daily proof that we are imperfect, which is why most of seek to talk
>> over
>> and explain our issues to inanimate objects or colleagues e.g. talk to the
>> rubber duck, teddy bear, plush hippo, etc. (My shelf holds two ducks and a
>> pelican awarded by former projects.)
>>
>> For help with Cygwin, we need to see *whole* commands and all the output
>> between
>> the shell prompts, preferably with context, in this case including PATH
>> under
>> Cygwin and MS Windows and/or program paths typed or invoked.
>>
>> You may retain privacy and security by substituting variables e.g. env
>> vars $VAR
>> to sanitize sensitive information: to avoid problems I use a script to do
>> so
>> that logs are never written containing sensitive content.
>>
> 
> Acknowledged. The "Upgrade request required" is coming from the kubernetes
> server.  At this point, I think I'm going to need to understand exactly
> what that message means (other threads talking about this didn't make that
> clear). Someone in that community did respond to my question, not with an
> answer, but with a query for more information. I'll see what I can find out.

A short Google search turned up that the message is apparently an HTTP 400 Bad
Request response with Upgrade request message, meaning you need to use the
websocket API to talk to a kubernetes pod:

https://stackoverflow.com/questions/49250370/kubernetes-pod-exec-api-upgrade-request-required
https://tools.ietf.org/html/rfc6455
https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/101

-- 
Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada

This email may be disturbing to some readers as it contains
too much technical detail. Reader discretion is advised.
[Data in IEC units and prefixes, physical quantities in SI.]

Re: "kubectl exec" in Cygwin gets "Upgrade request required", but not in cmd shell

[David Karr]

On Sun, Jun 14, 2020 at 12:09 PM Brian Inglis <
Brian.Inglis@systematicsw.ab.ca> wrote:

> On 2020-06-14 12:16, David Karr via Cygwin wrote:
> > On Sun, Jun 14, 2020 at 10:20 AM Brian Inglis <
> > Brian.Inglis@systematicsw.ab.ca> wrote:
> >
> >> On 2020-06-14 09:38, David Karr via Cygwin wrote:
> >>> On Sun, Jun 14, 2020 at 2:25 AM Marco Atzeri wrote:
> >>>> On 14.06.2020 08:12, David Karr wrote:
> >>>>> On Sat, Jun 13, 2020 at 10:31 PM Marco Atzeri via Cygwin wrote:
> >>>>>>     On 13.06.2020 20:53, David Karr via Cygwin wrote:
> >>>>>>> I've been using kubectl in Cygwin on Windows 10 for quite a while,
> >>>>>>> to communicate to our in-house k8s clusters. I often use "kubectl
> >>>>>>> exec" to open a shell in a container or directly execute a shell
> >>>>>>> command.
> >>>>>>> This has worked perfectly fine for a long time.
> >>>>>>> A couple of days ago, I discovered that all of these attempts were
> >>>>>>> failing with "Upgrade request required".  I hadn't upgraded kubectl
> >>>>>>> or Cygwin in quite a while. I doubt our clusters had a k8s upgrade,
> >>>>>>> but it's entirely possible.
> >>>>>>> A colleague of mine has a very similar desktop configuration
> >>>>>>> (Windows 10, Cygwin), and he's not seeing this symptom.
> >>>>>>> I noticed that when I ran "kubectl exec" with max verbosity, it
> shows
> >>>>>>> the resulting "curl" command that it runs. I tried that resulting
> >>>>>>> command, and it results in the same response. I then tried updating
> >>>>>>> my Cygwin tools and retesting, no change.>>>>> I then took the
> >> entire resulting "kubectl exec" command line and ran
> >>>>>>> it in a "cmd" shell.  No problem at all.  No error.
> >>>>>>> I know I haven't provided much useful information yet. I wanted to
> >>>>>>> get an initial response before I started providing those
> diagnostics.
> >>>>>>> Is there a clear issue here that I'm not aware of?
> >>
> >>>>> from where is kubectl coming from ?
> >>>>> In cygwin I found only a kubectl.py in the ansible package
> >>
> >>>>>> It's from here:
> >>>>>>
> >>
> https://kubernetes.io/docs/tasks/tools/install-kubectl/#install-kubectl-on-windows
> >>
> >>>> so it is NOT a cygwin program.
> >>>> If the warning is coming about curl, it is likely
> >>>> that using from cygwin you are using the cygwin curl
> >>>> and from CMD the windows one
> >>>> $ which -a curl
> >>>> /usr/bin/curl
> >>>> /cygdrive/c/WINDOWS/system32/curl
> >>>> $ /cygdrive/c/WINDOWS/system32/curl -V
> >>>> curl 7.55.1 (Windows) libcurl/7.55.1 WinSSL
> >>>> Release-Date: 2017-11-14, security patched: 2019-11-05
> >>>> Protocols: dict file ftp ftps http https imap imaps pop3 pop3s smtp
> >> smtps
> >>>> telnet tftp
> >>>> Features: AsynchDNS IPv6 Largefile SSPI Kerberos SPNEGO NTLM SSL
> >>>>
> >>>> $ /usr/bin/curl -V
> >>>> curl 7.66.0 (x86_64-pc-cygwin) libcurl/7.66.0 OpenSSL/1.1.1f
> zlib/1.2.11
> >>>> brotli/1.0.7 libidn2/2.2.0 libpsl/0.21.0 (+libidn2/2.0.4)
> >>>> libssh/0.8.7/openssl/zlib nghttp2/1.37.0
> >>>> Release-Date: 2019-09-11
> >>>> Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps
> >>>> pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp
> >>>> Features: AsynchDNS brotli Debug GSS-API HTTP2 HTTPS-proxy IDN IPv6
> >>>> Kerberos Largefile libz Metalink NTLM NTLM_WB PSL SPNEGO SSL TLS-SRP
> >>>> TrackMemory UnixSockets
> >>>> the support Forum https://discuss.kubernetes.io/
> >>>> is probably the most indicate place for guidance
> >>
> >>> I thought it was obvious that it was not working because it was calling
> >> the
> >>> Cygwin curl. I wouldn't have posted here if that wasn't obvious to me.
> >>> And since I'm well aware of the k8s community, I already posted
> questions
> >>> about this in the appropriate place, before I posted here.
> >>> What I was hoping to get here was some indication or thoughts on why a
> >>> process using Windows curl doesn't have a problem, but does have a
> >> problem
> >>> when using Cygwin Curl. This isn't likely something that Cygwin curl is
> >>> doing "wrong", it's just that it's doing something different.
> >>> If it matters, the following is an elided version of the resulting curl
> >>> command:
> >>>     curl -k -v -XPOST  -H "User-Agent: kubectl.exe/v1.18.0
> >> (windows/amd64)
> >>> kubernetes/9e99141" -H "Authorization: Bearer ..." -H
> >>> "X-Stream-Protocol-Version: v4.channel.k8s.io" -H
> >>> "X-Stream-Protocol-Version: v3.channel.k8s.io" -H
> >>> "X-Stream-Protocol-Version: v2.channel.k8s.io" -H
> >>> "X-Stream-Protocol-Version: channel.k8s.io" 'https://
> >>>
> >>
> .../api/v1/namespaces/.../pods/.../exec?command=%2Fbin%2Fls&container=...&stderr=true&stdin=true&stdout=true'
> >>> I can't tell from the logging what request body it sent. It's possible
> it
> >>> didn't send any.
> >>
> >> We can't tell, not knowing who you are, and from what you are posting,
> >> what may
> >> or may not be obvious to you, where you are seeing "Upgrade request
> >> required",
> >> or where that may be coming from.
> >> That you need diagnostic help indicates that, what may appear obvious to
> >> you,
> >> may not be the case, as it is often our assumptions which lead us
> astray,
> >> and we
> >> get daily proof that we are imperfect, which is why most of seek to talk
> >> over
> >> and explain our issues to inanimate objects or colleagues e.g. talk to
> the
> >> rubber duck, teddy bear, plush hippo, etc. (My shelf holds two ducks
> and a
> >> pelican awarded by former projects.)
> >>
> >> For help with Cygwin, we need to see *whole* commands and all the output
> >> between
> >> the shell prompts, preferably with context, in this case including PATH
> >> under
> >> Cygwin and MS Windows and/or program paths typed or invoked.
> >>
> >> You may retain privacy and security by substituting variables e.g. env
> >> vars $VAR
> >> to sanitize sensitive information: to avoid problems I use a script to
> do
> >> so
> >> that logs are never written containing sensitive content.
> >>
> >
> > Acknowledged. The "Upgrade request required" is coming from the
> kubernetes
> > server.  At this point, I think I'm going to need to understand exactly
> > what that message means (other threads talking about this didn't make
> that
> > clear). Someone in that community did respond to my question, not with an
> > answer, but with a query for more information. I'll see what I can find
> out.
>
> A short Google search turned up that the message is apparently an HTTP 400
> Bad
> Request response with Upgrade request message, meaning you need to use the
> websocket API to talk to a kubernetes pod:
>
>
> https://stackoverflow.com/questions/49250370/kubernetes-pod-exec-api-upgrade-request-required
> https://tools.ietf.org/html/rfc6455
> https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/101


Yes, I saw that thread in my research on this, but note that that
conclusion about using WebSocket is only from the first sentence of the
answer, whereas by the end of the answer, including the comment, it's clear
that this should be mitigated by just using "kubectl exec", which is what
I'm doing. In any case, it appears that even kubectl uses curl under the
covers. I would assume that kubectl attempts to mitigate the issues around
not using WebSocket for this, but it seems like it's not working at least
in my case.

Another thought I had is whether I can coerce kubectl to use the Windows
curl by setting the PATH just before calling kubectl.  This didn't help. Is
there possibly something happening under the covers that would still make
this execute the Cygwin curl?


>
> --
> Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada
>
> This email may be disturbing to some readers as it contains
> too much technical detail. Reader discretion is advised.
> [Data in IEC units and prefixes, physical quantities in SI.]
> --
> Problem reports:      https://cygwin.com/problems.html
> FAQ:                  https://cygwin.com/faq/
> Documentation:        https://cygwin.com/docs.html
> Unsubscribe info:     https://cygwin.com/ml/#unsubscribe-simple
>

Re: "kubectl exec" in Cygwin gets "Upgrade request required", but not in cmd shell

[David Karr]

On Sun, Jun 14, 2020 at 12:32 PM David Karr wrote:

>
> On Sun, Jun 14, 2020 at 12:09 PM Brian Inglis <
> Brian.Inglis@systematicsw.ab.ca> wrote:
>
>> On 2020-06-14 12:16, David Karr via Cygwin wrote:
>> > On Sun, Jun 14, 2020 at 10:20 AM Brian Inglis <
>> > Brian.Inglis@systematicsw.ab.ca> wrote:
>> >
>> >> On 2020-06-14 09:38, David Karr via Cygwin wrote:
>> >>> On Sun, Jun 14, 2020 at 2:25 AM Marco Atzeri wrote:
>> >>>> On 14.06.2020 08:12, David Karr wrote:
>> >>>>> On Sat, Jun 13, 2020 at 10:31 PM Marco Atzeri via Cygwin wrote:
>> >>>>>>     On 13.06.2020 20:53, David Karr via Cygwin wrote:
>> >>>>>>> I've been using kubectl in Cygwin on Windows 10 for quite a while,
>> >>>>>>> to communicate to our in-house k8s clusters. I often use "kubectl
>> >>>>>>> exec" to open a shell in a container or directly execute a shell
>> >>>>>>> command.
>> >>>>>>> This has worked perfectly fine for a long time.
>> >>>>>>> A couple of days ago, I discovered that all of these attempts were
>> >>>>>>> failing with "Upgrade request required".  I hadn't upgraded
>> kubectl
>> >>>>>>> or Cygwin in quite a while. I doubt our clusters had a k8s
>> upgrade,
>> >>>>>>> but it's entirely possible.
>> >>>>>>> A colleague of mine has a very similar desktop configuration
>> >>>>>>> (Windows 10, Cygwin), and he's not seeing this symptom.
>> >>>>>>> I noticed that when I ran "kubectl exec" with max verbosity, it
>> shows
>> >>>>>>> the resulting "curl" command that it runs. I tried that resulting
>> >>>>>>> command, and it results in the same response. I then tried
>> updating
>> >>>>>>> my Cygwin tools and retesting, no change.>>>>> I then took the
>> >> entire resulting "kubectl exec" command line and ran
>> >>>>>>> it in a "cmd" shell.  No problem at all.  No error.
>> >>>>>>> I know I haven't provided much useful information yet. I wanted to
>> >>>>>>> get an initial response before I started providing those
>> diagnostics.
>> >>>>>>> Is there a clear issue here that I'm not aware of?
>> >>
>> >>>>> from where is kubectl coming from ?
>> >>>>> In cygwin I found only a kubectl.py in the ansible package
>> >>
>> >>>>>> It's from here:
>> >>>>>>
>> >>
>> https://kubernetes.io/docs/tasks/tools/install-kubectl/#install-kubectl-on-windows
>> >>
>> >>>> so it is NOT a cygwin program.
>> >>>> If the warning is coming about curl, it is likely
>> >>>> that using from cygwin you are using the cygwin curl
>> >>>> and from CMD the windows one
>> >>>> $ which -a curl
>> >>>> /usr/bin/curl
>> >>>> /cygdrive/c/WINDOWS/system32/curl
>> >>>> $ /cygdrive/c/WINDOWS/system32/curl -V
>> >>>> curl 7.55.1 (Windows) libcurl/7.55.1 WinSSL
>> >>>> Release-Date: 2017-11-14, security patched: 2019-11-05
>> >>>> Protocols: dict file ftp ftps http https imap imaps pop3 pop3s smtp
>> >> smtps
>> >>>> telnet tftp
>> >>>> Features: AsynchDNS IPv6 Largefile SSPI Kerberos SPNEGO NTLM SSL
>> >>>>
>> >>>> $ /usr/bin/curl -V
>> >>>> curl 7.66.0 (x86_64-pc-cygwin) libcurl/7.66.0 OpenSSL/1.1.1f
>> zlib/1.2.11
>> >>>> brotli/1.0.7 libidn2/2.2.0 libpsl/0.21.0 (+libidn2/2.0.4)
>> >>>> libssh/0.8.7/openssl/zlib nghttp2/1.37.0
>> >>>> Release-Date: 2019-09-11
>> >>>> Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps
>> >>>> pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp
>> >>>> Features: AsynchDNS brotli Debug GSS-API HTTP2 HTTPS-proxy IDN IPv6
>> >>>> Kerberos Largefile libz Metalink NTLM NTLM_WB PSL SPNEGO SSL TLS-SRP
>> >>>> TrackMemory UnixSockets
>> >>>> the support Forum https://discuss.kubernetes.io/
>> >>>> is probably the most indicate place for guidance
>> >>
>> >>> I thought it was obvious that it was not working because it was
>> calling
>> >> the
>> >>> Cygwin curl. I wouldn't have posted here if that wasn't obvious to me.
>> >>> And since I'm well aware of the k8s community, I already posted
>> questions
>> >>> about this in the appropriate place, before I posted here.
>> >>> What I was hoping to get here was some indication or thoughts on why a
>> >>> process using Windows curl doesn't have a problem, but does have a
>> >> problem
>> >>> when using Cygwin Curl. This isn't likely something that Cygwin curl
>> is
>> >>> doing "wrong", it's just that it's doing something different.
>> >>> If it matters, the following is an elided version of the resulting
>> curl
>> >>> command:
>> >>>     curl -k -v -XPOST  -H "User-Agent: kubectl.exe/v1.18.0
>> >> (windows/amd64)
>> >>> kubernetes/9e99141" -H "Authorization: Bearer ..." -H
>> >>> "X-Stream-Protocol-Version: v4.channel.k8s.io" -H
>> >>> "X-Stream-Protocol-Version: v3.channel.k8s.io" -H
>> >>> "X-Stream-Protocol-Version: v2.channel.k8s.io" -H
>> >>> "X-Stream-Protocol-Version: channel.k8s.io" 'https://
>> >>>
>> >>
>> .../api/v1/namespaces/.../pods/.../exec?command=%2Fbin%2Fls&container=...&stderr=true&stdin=true&stdout=true'
>> >>> I can't tell from the logging what request body it sent. It's
>> possible it
>> >>> didn't send any.
>> >>
>> >> We can't tell, not knowing who you are, and from what you are posting,
>> >> what may
>> >> or may not be obvious to you, where you are seeing "Upgrade request
>> >> required",
>> >> or where that may be coming from.
>> >> That you need diagnostic help indicates that, what may appear obvious
>> to
>> >> you,
>> >> may not be the case, as it is often our assumptions which lead us
>> astray,
>> >> and we
>> >> get daily proof that we are imperfect, which is why most of seek to
>> talk
>> >> over
>> >> and explain our issues to inanimate objects or colleagues e.g. talk to
>> the
>> >> rubber duck, teddy bear, plush hippo, etc. (My shelf holds two ducks
>> and a
>> >> pelican awarded by former projects.)
>> >>
>> >> For help with Cygwin, we need to see *whole* commands and all the
>> output
>> >> between
>> >> the shell prompts, preferably with context, in this case including PATH
>> >> under
>> >> Cygwin and MS Windows and/or program paths typed or invoked.
>> >>
>> >> You may retain privacy and security by substituting variables e.g. env
>> >> vars $VAR
>> >> to sanitize sensitive information: to avoid problems I use a script to
>> do
>> >> so
>> >> that logs are never written containing sensitive content.
>> >>
>> >
>> > Acknowledged. The "Upgrade request required" is coming from the
>> kubernetes
>> > server.  At this point, I think I'm going to need to understand exactly
>> > what that message means (other threads talking about this didn't make
>> that
>> > clear). Someone in that community did respond to my question, not with
>> an
>> > answer, but with a query for more information. I'll see what I can find
>> out.
>>
>> A short Google search turned up that the message is apparently an HTTP
>> 400 Bad
>> Request response with Upgrade request message, meaning you need to use the
>> websocket API to talk to a kubernetes pod:
>>
>>
>> https://stackoverflow.com/questions/49250370/kubernetes-pod-exec-api-upgrade-request-required
>> https://tools.ietf.org/html/rfc6455
>> https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/101
>
>
> Yes, I saw that thread in my research on this, but note that that
> conclusion about using WebSocket is only from the first sentence of the
> answer, whereas by the end of the answer, including the comment, it's clear
> that this should be mitigated by just using "kubectl exec", which is what
> I'm doing. In any case, it appears that even kubectl uses curl under the
> covers. I would assume that kubectl attempts to mitigate the issues around
> not using WebSocket for this, but it seems like it's not working at least
> in my case.
>
> Another thought I had is whether I can coerce kubectl to use the Windows
> curl by setting the PATH just before calling kubectl.  This didn't help. Is
> there possibly something happening under the covers that would still make
> this execute the Cygwin curl?
>

I have managed to resolve this.  The challenge here is finding the correct
version of kubectl to use for the corresponding cluster version.  The rule
is that you have to use a version of kubectl within one minor version of
the cluster version.  We're using version 1.13.5 in the cluster.  I
originally saw this problem with version 1.17, then moved to 1.18, then to
1.13.5.  Finally, version 1.14.0 resolves the problem. Version 1.13.5
should have worked.  I don't know why it didn't.  I also have no idea why
it would work in a non-cygwin shell.

Re: "kubectl exec" in Cygwin gets "Upgrade request required", but not in cmd shell

[David Karr]

On Mon, Jun 15, 2020 at 9:10 AM David Karr wrote:

>
> On Sun, Jun 14, 2020 at 12:32 PM David Karr wrote:
>
>>
>> On Sun, Jun 14, 2020 at 12:09 PM Brian Inglis <
>> Brian.Inglis@systematicsw.ab.ca> wrote:
>>
>>> On 2020-06-14 12:16, David Karr via Cygwin wrote:
>>> > On Sun, Jun 14, 2020 at 10:20 AM Brian Inglis <
>>> > Brian.Inglis@systematicsw.ab.ca> wrote:
>>> >
>>> >> On 2020-06-14 09:38, David Karr via Cygwin wrote:
>>> >>> On Sun, Jun 14, 2020 at 2:25 AM Marco Atzeri wrote:
>>> >>>> On 14.06.2020 08:12, David Karr wrote:
>>> >>>>> On Sat, Jun 13, 2020 at 10:31 PM Marco Atzeri via Cygwin wrote:
>>> >>>>>>     On 13.06.2020 20:53, David Karr via Cygwin wrote:
>>> >>>>>>> I've been using kubectl in Cygwin on Windows 10 for quite a
>>> while,
>>> >>>>>>> to communicate to our in-house k8s clusters. I often use "kubectl
>>> >>>>>>> exec" to open a shell in a container or directly execute a shell
>>> >>>>>>> command.
>>> >>>>>>> This has worked perfectly fine for a long time.
>>> >>>>>>> A couple of days ago, I discovered that all of these attempts
>>> were
>>> >>>>>>> failing with "Upgrade request required".  I hadn't upgraded
>>> kubectl
>>> >>>>>>> or Cygwin in quite a while. I doubt our clusters had a k8s
>>> upgrade,
>>> >>>>>>> but it's entirely possible.
>>> >>>>>>> A colleague of mine has a very similar desktop configuration
>>> >>>>>>> (Windows 10, Cygwin), and he's not seeing this symptom.
>>> >>>>>>> I noticed that when I ran "kubectl exec" with max verbosity, it
>>> shows
>>> >>>>>>> the resulting "curl" command that it runs. I tried that resulting
>>> >>>>>>> command, and it results in the same response. I then tried
>>> updating
>>> >>>>>>> my Cygwin tools and retesting, no change.>>>>> I then took the
>>> >> entire resulting "kubectl exec" command line and ran
>>> >>>>>>> it in a "cmd" shell.  No problem at all.  No error.
>>> >>>>>>> I know I haven't provided much useful information yet. I wanted
>>> to
>>> >>>>>>> get an initial response before I started providing those
>>> diagnostics.
>>> >>>>>>> Is there a clear issue here that I'm not aware of?
>>> >>
>>> >>>>> from where is kubectl coming from ?
>>> >>>>> In cygwin I found only a kubectl.py in the ansible package
>>> >>
>>> >>>>>> It's from here:
>>> >>>>>>
>>> >>
>>> https://kubernetes.io/docs/tasks/tools/install-kubectl/#install-kubectl-on-windows
>>> >>
>>> >>>> so it is NOT a cygwin program.
>>> >>>> If the warning is coming about curl, it is likely
>>> >>>> that using from cygwin you are using the cygwin curl
>>> >>>> and from CMD the windows one
>>> >>>> $ which -a curl
>>> >>>> /usr/bin/curl
>>> >>>> /cygdrive/c/WINDOWS/system32/curl
>>> >>>> $ /cygdrive/c/WINDOWS/system32/curl -V
>>> >>>> curl 7.55.1 (Windows) libcurl/7.55.1 WinSSL
>>> >>>> Release-Date: 2017-11-14, security patched: 2019-11-05
>>> >>>> Protocols: dict file ftp ftps http https imap imaps pop3 pop3s smtp
>>> >> smtps
>>> >>>> telnet tftp
>>> >>>> Features: AsynchDNS IPv6 Largefile SSPI Kerberos SPNEGO NTLM SSL
>>> >>>>
>>> >>>> $ /usr/bin/curl -V
>>> >>>> curl 7.66.0 (x86_64-pc-cygwin) libcurl/7.66.0 OpenSSL/1.1.1f
>>> zlib/1.2.11
>>> >>>> brotli/1.0.7 libidn2/2.2.0 libpsl/0.21.0 (+libidn2/2.0.4)
>>> >>>> libssh/0.8.7/openssl/zlib nghttp2/1.37.0
>>> >>>> Release-Date: 2019-09-11
>>> >>>> Protocols: dict file ftp ftps gopher http https imap imaps ldap
>>> ldaps
>>> >>>> pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp
>>> >>>> Features: AsynchDNS brotli Debug GSS-API HTTP2 HTTPS-proxy IDN IPv6
>>> >>>> Kerberos Largefile libz Metalink NTLM NTLM_WB PSL SPNEGO SSL TLS-SRP
>>> >>>> TrackMemory UnixSockets
>>> >>>> the support Forum https://discuss.kubernetes.io/
>>> >>>> is probably the most indicate place for guidance
>>> >>
>>> >>> I thought it was obvious that it was not working because it was
>>> calling
>>> >> the
>>> >>> Cygwin curl. I wouldn't have posted here if that wasn't obvious to
>>> me.
>>> >>> And since I'm well aware of the k8s community, I already posted
>>> questions
>>> >>> about this in the appropriate place, before I posted here.
>>> >>> What I was hoping to get here was some indication or thoughts on why
>>> a
>>> >>> process using Windows curl doesn't have a problem, but does have a
>>> >> problem
>>> >>> when using Cygwin Curl. This isn't likely something that Cygwin curl
>>> is
>>> >>> doing "wrong", it's just that it's doing something different.
>>> >>> If it matters, the following is an elided version of the resulting
>>> curl
>>> >>> command:
>>> >>>     curl -k -v -XPOST  -H "User-Agent: kubectl.exe/v1.18.0
>>> >> (windows/amd64)
>>> >>> kubernetes/9e99141" -H "Authorization: Bearer ..." -H
>>> >>> "X-Stream-Protocol-Version: v4.channel.k8s.io" -H
>>> >>> "X-Stream-Protocol-Version: v3.channel.k8s.io" -H
>>> >>> "X-Stream-Protocol-Version: v2.channel.k8s.io" -H
>>> >>> "X-Stream-Protocol-Version: channel.k8s.io" 'https://
>>> >>>
>>> >>
>>> .../api/v1/namespaces/.../pods/.../exec?command=%2Fbin%2Fls&container=...&stderr=true&stdin=true&stdout=true'
>>> >>> I can't tell from the logging what request body it sent. It's
>>> possible it
>>> >>> didn't send any.
>>> >>
>>> >> We can't tell, not knowing who you are, and from what you are posting,
>>> >> what may
>>> >> or may not be obvious to you, where you are seeing "Upgrade request
>>> >> required",
>>> >> or where that may be coming from.
>>> >> That you need diagnostic help indicates that, what may appear obvious
>>> to
>>> >> you,
>>> >> may not be the case, as it is often our assumptions which lead us
>>> astray,
>>> >> and we
>>> >> get daily proof that we are imperfect, which is why most of seek to
>>> talk
>>> >> over
>>> >> and explain our issues to inanimate objects or colleagues e.g. talk
>>> to the
>>> >> rubber duck, teddy bear, plush hippo, etc. (My shelf holds two ducks
>>> and a
>>> >> pelican awarded by former projects.)
>>> >>
>>> >> For help with Cygwin, we need to see *whole* commands and all the
>>> output
>>> >> between
>>> >> the shell prompts, preferably with context, in this case including
>>> PATH
>>> >> under
>>> >> Cygwin and MS Windows and/or program paths typed or invoked.
>>> >>
>>> >> You may retain privacy and security by substituting variables e.g. env
>>> >> vars $VAR
>>> >> to sanitize sensitive information: to avoid problems I use a script
>>> to do
>>> >> so
>>> >> that logs are never written containing sensitive content.
>>> >>
>>> >
>>> > Acknowledged. The "Upgrade request required" is coming from the
>>> kubernetes
>>> > server.  At this point, I think I'm going to need to understand exactly
>>> > what that message means (other threads talking about this didn't make
>>> that
>>> > clear). Someone in that community did respond to my question, not with
>>> an
>>> > answer, but with a query for more information. I'll see what I can
>>> find out.
>>>
>>> A short Google search turned up that the message is apparently an HTTP
>>> 400 Bad
>>> Request response with Upgrade request message, meaning you need to use
>>> the
>>> websocket API to talk to a kubernetes pod:
>>>
>>>
>>> https://stackoverflow.com/questions/49250370/kubernetes-pod-exec-api-upgrade-request-required
>>> https://tools.ietf.org/html/rfc6455
>>> https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/101
>>
>>
>> Yes, I saw that thread in my research on this, but note that that
>> conclusion about using WebSocket is only from the first sentence of the
>> answer, whereas by the end of the answer, including the comment, it's clear
>> that this should be mitigated by just using "kubectl exec", which is what
>> I'm doing. In any case, it appears that even kubectl uses curl under the
>> covers. I would assume that kubectl attempts to mitigate the issues around
>> not using WebSocket for this, but it seems like it's not working at least
>> in my case.
>>
>> Another thought I had is whether I can coerce kubectl to use the Windows
>> curl by setting the PATH just before calling kubectl.  This didn't help. Is
>> there possibly something happening under the covers that would still make
>> this execute the Cygwin curl?
>>
>
> I have managed to resolve this.  The challenge here is finding the correct
> version of kubectl to use for the corresponding cluster version.  The rule
> is that you have to use a version of kubectl within one minor version of
> the cluster version.  We're using version 1.13.5 in the cluster.  I
> originally saw this problem with version 1.17, then moved to 1.18, then to
> 1.13.5.  Finally, version 1.14.0 resolves the problem. Version 1.13.5
> should have worked.  I don't know why it didn't.  I also have no idea why
> it would work in a non-cygwin shell.
>
>
Actually, I was wrong about this being resolved. For some reason it was
working properly for a short period after I installed v1.14.0, but now it
is consistently failing with "Upgrade request required" again.

I've verified that the resulting kubectl call works fine from a cmd window.

Another thing that I didn't mention before is that I run a Linux VM on this
laptop, and I can run the same scripting layer, albeit with a Linux install
of kubectl, and it works perfectly fine.

Re: "kubectl exec" in Cygwin gets "Upgrade request required", but not in cmd shell

[Brian Inglis]

On 2020-06-17 10:39, David Karr via Cygwin wrote:
> On Mon, Jun 15, 2020 at 9:10 AM David Karr wrote:
>> On Sun, Jun 14, 2020 at 12:32 PM David Karr wrote:
>>> On Sun, Jun 14, 2020 at 12:09 PM Brian Inglis wrote:
>>>> On 2020-06-14 12:16, David Karr via Cygwin wrote:
>>>>> On Sun, Jun 14, 2020 at 10:20 AM Brian Inglis wrote:
>>>>>> On 2020-06-14 09:38, David Karr via Cygwin wrote:
>>>>>>> On Sun, Jun 14, 2020 at 2:25 AM Marco Atzeri wrote:
>>>>>>>> On 14.06.2020 08:12, David Karr wrote:
>>>>>>>>> On Sat, Jun 13, 2020 at 10:31 PM Marco Atzeri via Cygwin wrote:
>>>>>>>>>> On 13.06.2020 20:53, David Karr via Cygwin wrote:
>>>>>>>>>>> I've been using kubectl in Cygwin on Windows 10 for quite a
>>>> while,
>>>>>>>>>>> to communicate to our in-house k8s clusters. I often use "kubectl
>>>>>>>>>>> exec" to open a shell in a container or directly execute a shell
>>>>>>>>>>> command.
>>>>>>>>>>> This has worked perfectly fine for a long time.
>>>>>>>>>>> A couple of days ago, I discovered that all of these attempts
>>>> were
>>>>>>>>>>> failing with "Upgrade request required".  I hadn't upgraded
>>>> kubectl
>>>>>>>>>>> or Cygwin in quite a while. I doubt our clusters had a k8s
>>>> upgrade,
>>>>>>>>>>> but it's entirely possible.
>>>>>>>>>>> A colleague of mine has a very similar desktop configuration
>>>>>>>>>>> (Windows 10, Cygwin), and he's not seeing this symptom.
>>>>>>>>>>> I noticed that when I ran "kubectl exec" with max verbosity, it
>>>> shows
>>>>>>>>>>> the resulting "curl" command that it runs. I tried that resulting
>>>>>>>>>>> command, and it results in the same response. I then tried
>>>> updating
>>>>>>>>>>> my Cygwin tools and retesting, no change.>>>>> I then took the
>>>>>> entire resulting "kubectl exec" command line and ran
>>>>>>>>>>> it in a "cmd" shell.  No problem at all.  No error.
>>>>>>>>>>> I know I haven't provided much useful information yet. I wanted
>>>> to
>>>>>>>>>>> get an initial response before I started providing those
>>>> diagnostics.
>>>>>>>>>>> Is there a clear issue here that I'm not aware of?
>>>>>>
>>>>>>>>> from where is kubectl coming from ?
>>>>>>>>> In cygwin I found only a kubectl.py in the ansible package
>>>>>>
>>>>>>>>>> It's from here:
>>>>>>>>>>
>>>>>>
>>>> https://kubernetes.io/docs/tasks/tools/install-kubectl/#install-kubectl-on-windows
>>>>>>
>>>>>>>> so it is NOT a cygwin program.
>>>>>>>> If the warning is coming about curl, it is likely
>>>>>>>> that using from cygwin you are using the cygwin curl
>>>>>>>> and from CMD the windows one
>>>>>>>> $ which -a curl
>>>>>>>> /usr/bin/curl
>>>>>>>> /cygdrive/c/WINDOWS/system32/curl
>>>>>>>> $ /cygdrive/c/WINDOWS/system32/curl -V
>>>>>>>> curl 7.55.1 (Windows) libcurl/7.55.1 WinSSL
>>>>>>>> Release-Date: 2017-11-14, security patched: 2019-11-05
>>>>>>>> Protocols: dict file ftp ftps http https imap imaps pop3 pop3s smtp
>>>>>> smtps
>>>>>>>> telnet tftp
>>>>>>>> Features: AsynchDNS IPv6 Largefile SSPI Kerberos SPNEGO NTLM SSL
>>>>>>>>
>>>>>>>> $ /usr/bin/curl -V
>>>>>>>> curl 7.66.0 (x86_64-pc-cygwin) libcurl/7.66.0 OpenSSL/1.1.1f
>>>> zlib/1.2.11
>>>>>>>> brotli/1.0.7 libidn2/2.2.0 libpsl/0.21.0 (+libidn2/2.0.4)
>>>>>>>> libssh/0.8.7/openssl/zlib nghttp2/1.37.0
>>>>>>>> Release-Date: 2019-09-11
>>>>>>>> Protocols: dict file ftp ftps gopher http https imap imaps ldap
>>>> ldaps
>>>>>>>> pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp
>>>>>>>> Features: AsynchDNS brotli Debug GSS-API HTTP2 HTTPS-proxy IDN IPv6
>>>>>>>> Kerberos Largefile libz Metalink NTLM NTLM_WB PSL SPNEGO SSL TLS-SRP
>>>>>>>> TrackMemory UnixSockets
>>>>>>>> the support Forum https://discuss.kubernetes.io/
>>>>>>>> is probably the most indicate place for guidance
>>>>>>
>>>>>>> I thought it was obvious that it was not working because it was
>>>> calling
>>>>>> the
>>>>>>> Cygwin curl. I wouldn't have posted here if that wasn't obvious to
>>>> me.
>>>>>>> And since I'm well aware of the k8s community, I already posted
>>>> questions
>>>>>>> about this in the appropriate place, before I posted here.
>>>>>>> What I was hoping to get here was some indication or thoughts on why
>>>> a
>>>>>>> process using Windows curl doesn't have a problem, but does have a
>>>>>> problem
>>>>>>> when using Cygwin Curl. This isn't likely something that Cygwin curl
>>>> is
>>>>>>> doing "wrong", it's just that it's doing something different.
>>>>>>> If it matters, the following is an elided version of the resulting
>>>> curl
>>>>>>> command:
>>>>>>>     curl -k -v -XPOST  -H "User-Agent: kubectl.exe/v1.18.0
>>>>>> (windows/amd64)
>>>>>>> kubernetes/9e99141" -H "Authorization: Bearer ..." -H
>>>>>>> "X-Stream-Protocol-Version: v4.channel.k8s.io" -H
>>>>>>> "X-Stream-Protocol-Version: v3.channel.k8s.io" -H
>>>>>>> "X-Stream-Protocol-Version: v2.channel.k8s.io" -H
>>>>>>> "X-Stream-Protocol-Version: channel.k8s.io" 'https://
>>>>>>>
>>>>>>
>>>> .../api/v1/namespaces/.../pods/.../exec?command=%2Fbin%2Fls&container=...&stderr=true&stdin=true&stdout=true'
>>>>>>> I can't tell from the logging what request body it sent. It's
>>>> possible it
>>>>>>> didn't send any.
>>>>>>
>>>>>> We can't tell, not knowing who you are, and from what you are posting,
>>>>>> what may
>>>>>> or may not be obvious to you, where you are seeing "Upgrade request
>>>>>> required",
>>>>>> or where that may be coming from.
>>>>>> That you need diagnostic help indicates that, what may appear obvious
>>>> to
>>>>>> you,
>>>>>> may not be the case, as it is often our assumptions which lead us
>>>> astray,
>>>>>> and we
>>>>>> get daily proof that we are imperfect, which is why most of seek to
>>>> talk
>>>>>> over
>>>>>> and explain our issues to inanimate objects or colleagues e.g. talk
>>>> to the
>>>>>> rubber duck, teddy bear, plush hippo, etc. (My shelf holds two ducks
>>>> and a
>>>>>> pelican awarded by former projects.)
>>>>>>
>>>>>> For help with Cygwin, we need to see *whole* commands and all the
>>>> output
>>>>>> between
>>>>>> the shell prompts, preferably with context, in this case including
>>>> PATH
>>>>>> under
>>>>>> Cygwin and MS Windows and/or program paths typed or invoked.
>>>>>>
>>>>>> You may retain privacy and security by substituting variables e.g. env
>>>>>> vars $VAR
>>>>>> to sanitize sensitive information: to avoid problems I use a script
>>>> to do
>>>>>> so
>>>>>> that logs are never written containing sensitive content.
>>>>>>
>>>>>
>>>>> Acknowledged. The "Upgrade request required" is coming from the
>>>> kubernetes
>>>>> server.  At this point, I think I'm going to need to understand exactly
>>>>> what that message means (other threads talking about this didn't make
>>>> that
>>>>> clear). Someone in that community did respond to my question, not with
>>>> an
>>>>> answer, but with a query for more information. I'll see what I can
>>>> find out.
>>>>
>>>> A short Google search turned up that the message is apparently an HTTP
>>>> 400 Bad
>>>> Request response with Upgrade request message, meaning you need to use
>>>> the
>>>> websocket API to talk to a kubernetes pod:
>>>>
>>>>
>>>> https://stackoverflow.com/questions/49250370/kubernetes-pod-exec-api-upgrade-request-required
>>>> https://tools.ietf.org/html/rfc6455
>>>> https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/101
>>>
>>>
>>> Yes, I saw that thread in my research on this, but note that that
>>> conclusion about using WebSocket is only from the first sentence of the
>>> answer, whereas by the end of the answer, including the comment, it's clear
>>> that this should be mitigated by just using "kubectl exec", which is what
>>> I'm doing. In any case, it appears that even kubectl uses curl under the
>>> covers. I would assume that kubectl attempts to mitigate the issues around
>>> not using WebSocket for this, but it seems like it's not working at least
>>> in my case.
>>>
>>> Another thought I had is whether I can coerce kubectl to use the Windows
>>> curl by setting the PATH just before calling kubectl.  This didn't help. Is
>>> there possibly something happening under the covers that would still make
>>> this execute the Cygwin curl?
>>>
>>
>> I have managed to resolve this.  The challenge here is finding the correct
>> version of kubectl to use for the corresponding cluster version.  The rule
>> is that you have to use a version of kubectl within one minor version of
>> the cluster version.  We're using version 1.13.5 in the cluster.  I
>> originally saw this problem with version 1.17, then moved to 1.18, then to
>> 1.13.5.  Finally, version 1.14.0 resolves the problem. Version 1.13.5
>> should have worked.  I don't know why it didn't.  I also have no idea why
>> it would work in a non-cygwin shell.
>>
>>
> Actually, I was wrong about this being resolved. For some reason it was
> working properly for a short period after I installed v1.14.0, but now it
> is consistently failing with "Upgrade request required" again.
> 
> I've verified that the resulting kubectl call works fine from a cmd window.
> 
> Another thing that I didn't mention before is that I run a Linux VM on this
> laptop, and I can run the same scripting layer, albeit with a Linux install
> of kubectl, and it works perfectly fine.

Looks like curl has to send an Upgrade and other headers:
https://gist.github.com/htp/fbce19069187ec1cc486b594104f01d0
which kubectl exec should supply.

Maybe time to start looking at what is passing over the net at both ends, as
well as from the middle.

-- 
Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada

This email may be disturbing to some readers as it contains
too much technical detail. Reader discretion is advised.
[Data in IEC units and prefixes, physical quantities in SI.]

Re: "kubectl exec" in Cygwin gets "Upgrade request required", but not in cmd shell

[David Karr]

On Wed, Jun 17, 2020 at 6:06 PM Brian Inglis  wrote:

> On 2020-06-17 10:39, David Karr via Cygwin wrote:
> > On Mon, Jun 15, 2020 at 9:10 AM David Karr wrote:
> >> On Sun, Jun 14, 2020 at 12:32 PM David Karr wrote:
> >>> On Sun, Jun 14, 2020 at 12:09 PM Brian Inglis wrote:
> >>>> On 2020-06-14 12:16, David Karr via Cygwin wrote:
> >>>>> On Sun, Jun 14, 2020 at 10:20 AM Brian Inglis wrote:
> >>>>>> On 2020-06-14 09:38, David Karr via Cygwin wrote:
> >>>>>>> On Sun, Jun 14, 2020 at 2:25 AM Marco Atzeri wrote:
> >>>>>>>> On 14.06.2020 08:12, David Karr wrote:
> >>>>>>>>> On Sat, Jun 13, 2020 at 10:31 PM Marco Atzeri via Cygwin wrote:
> >>>>>>>>>> On 13.06.2020 20:53, David Karr via Cygwin wrote:
> >>>>>>>>>>> I've been using kubectl in Cygwin on Windows 10 for quite a
> >>>> while,
> >>>>>>>>>>> to communicate to our in-house k8s clusters. I often use
> "kubectl
> >>>>>>>>>>> exec" to open a shell in a container or directly execute a
> shell
> >>>>>>>>>>> command.
> >>>>>>>>>>> This has worked perfectly fine for a long time.
> >>>>>>>>>>> A couple of days ago, I discovered that all of these attempts
> >>>> were
> >>>>>>>>>>> failing with "Upgrade request required".  I hadn't upgraded
> >>>> kubectl
> >>>>>>>>>>> or Cygwin in quite a while. I doubt our clusters had a k8s
> >>>> upgrade,
> >>>>>>>>>>> but it's entirely possible.
> >>>>>>>>>>> A colleague of mine has a very similar desktop configuration
> >>>>>>>>>>> (Windows 10, Cygwin), and he's not seeing this symptom.
> >>>>>>>>>>> I noticed that when I ran "kubectl exec" with max verbosity, it
> >>>> shows
> >>>>>>>>>>> the resulting "curl" command that it runs. I tried that
> resulting
> >>>>>>>>>>> command, and it results in the same response. I then tried
> >>>> updating
> >>>>>>>>>>> my Cygwin tools and retesting, no change.>>>>> I then took the
> >>>>>> entire resulting "kubectl exec" command line and ran
> >>>>>>>>>>> it in a "cmd" shell.  No problem at all.  No error.
> >>>>>>>>>>> I know I haven't provided much useful information yet. I wanted
> >>>> to
> >>>>>>>>>>> get an initial response before I started providing those
> >>>> diagnostics.
> >>>>>>>>>>> Is there a clear issue here that I'm not aware of?
> >>>>>>
> >>>>>>>>> from where is kubectl coming from ?
> >>>>>>>>> In cygwin I found only a kubectl.py in the ansible package
> >>>>>>
> >>>>>>>>>> It's from here:
> >>>>>>>>>>
> >>>>>>
> >>>>
> https://kubernetes.io/docs/tasks/tools/install-kubectl/#install-kubectl-on-windows
> >>>>>>
> >>>>>>>> so it is NOT a cygwin program.
> >>>>>>>> If the warning is coming about curl, it is likely
> >>>>>>>> that using from cygwin you are using the cygwin curl
> >>>>>>>> and from CMD the windows one
> >>>>>>>> $ which -a curl
> >>>>>>>> /usr/bin/curl
> >>>>>>>> /cygdrive/c/WINDOWS/system32/curl
> >>>>>>>> $ /cygdrive/c/WINDOWS/system32/curl -V
> >>>>>>>> curl 7.55.1 (Windows) libcurl/7.55.1 WinSSL
> >>>>>>>> Release-Date: 2017-11-14, security patched: 2019-11-05
> >>>>>>>> Protocols: dict file ftp ftps http https imap imaps pop3 pop3s
> smtp
> >>>>>> smtps
> >>>>>>>> telnet tftp
> >>>>>>>> Features: AsynchDNS IPv6 Largefile SSPI Kerberos SPNEGO NTLM SSL
> >>>>>>>>
> >>>>>>>> $ /usr/bin/curl -V
> >>>>>>>> curl 7.66.0 (x86_64-pc-cygwin) libcurl/7.66.0 OpenSSL/1.1.1f
> >>>> zlib/1.2.11
> >>>>>>>> brotli/1.0.7 libidn2/2.2.0 libpsl/0.21.0 (+libidn2/2.0.4)
> >>>>>>>> libssh/0.8.7/openssl/zlib nghttp2/1.37.0
> >>>>>>>> Release-Date: 2019-09-11
> >>>>>>>> Protocols: dict file ftp ftps gopher http https imap imaps ldap
> >>>> ldaps
> >>>>>>>> pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp
> >>>>>>>> Features: AsynchDNS brotli Debug GSS-API HTTP2 HTTPS-proxy IDN
> IPv6
> >>>>>>>> Kerberos Largefile libz Metalink NTLM NTLM_WB PSL SPNEGO SSL
> TLS-SRP
> >>>>>>>> TrackMemory UnixSockets
> >>>>>>>> the support Forum https://discuss.kubernetes.io/
> >>>>>>>> is probably the most indicate place for guidance
> >>>>>>
> >>>>>>> I thought it was obvious that it was not working because it was
> >>>> calling
> >>>>>> the
> >>>>>>> Cygwin curl. I wouldn't have posted here if that wasn't obvious to
> >>>> me.
> >>>>>>> And since I'm well aware of the k8s community, I already posted
> >>>> questions
> >>>>>>> about this in the appropriate place, before I posted here.
> >>>>>>> What I was hoping to get here was some indication or thoughts on
> why
> >>>> a
> >>>>>>> process using Windows curl doesn't have a problem, but does have a
> >>>>>> problem
> >>>>>>> when using Cygwin Curl. This isn't likely something that Cygwin
> curl
> >>>> is
> >>>>>>> doing "wrong", it's just that it's doing something different.
> >>>>>>> If it matters, the following is an elided version of the resulting
> >>>> curl
> >>>>>>> command:
> >>>>>>>     curl -k -v -XPOST  -H "User-Agent: kubectl.exe/v1.18.0
> >>>>>> (windows/amd64)
> >>>>>>> kubernetes/9e99141" -H "Authorization: Bearer ..." -H
> >>>>>>> "X-Stream-Protocol-Version: v4.channel.k8s.io" -H
> >>>>>>> "X-Stream-Protocol-Version: v3.channel.k8s.io" -H
> >>>>>>> "X-Stream-Protocol-Version: v2.channel.k8s.io" -H
> >>>>>>> "X-Stream-Protocol-Version: channel.k8s.io" 'https://
> >>>>>>>
> >>>>>>
> >>>>
> .../api/v1/namespaces/.../pods/.../exec?command=%2Fbin%2Fls&container=...&stderr=true&stdin=true&stdout=true'
> >>>>>>> I can't tell from the logging what request body it sent. It's
> >>>> possible it
> >>>>>>> didn't send any.
> >>>>>>
> >>>>>> We can't tell, not knowing who you are, and from what you are
> posting,
> >>>>>> what may
> >>>>>> or may not be obvious to you, where you are seeing "Upgrade request
> >>>>>> required",
> >>>>>> or where that may be coming from.
> >>>>>> That you need diagnostic help indicates that, what may appear
> obvious
> >>>> to
> >>>>>> you,
> >>>>>> may not be the case, as it is often our assumptions which lead us
> >>>> astray,
> >>>>>> and we
> >>>>>> get daily proof that we are imperfect, which is why most of seek to
> >>>> talk
> >>>>>> over
> >>>>>> and explain our issues to inanimate objects or colleagues e.g. talk
> >>>> to the
> >>>>>> rubber duck, teddy bear, plush hippo, etc. (My shelf holds two ducks
> >>>> and a
> >>>>>> pelican awarded by former projects.)
> >>>>>>
> >>>>>> For help with Cygwin, we need to see *whole* commands and all the
> >>>> output
> >>>>>> between
> >>>>>> the shell prompts, preferably with context, in this case including
> >>>> PATH
> >>>>>> under
> >>>>>> Cygwin and MS Windows and/or program paths typed or invoked.
> >>>>>>
> >>>>>> You may retain privacy and security by substituting variables e.g.
> env
> >>>>>> vars $VAR
> >>>>>> to sanitize sensitive information: to avoid problems I use a script
> >>>> to do
> >>>>>> so
> >>>>>> that logs are never written containing sensitive content.
> >>>>>>
> >>>>>
> >>>>> Acknowledged. The "Upgrade request required" is coming from the
> >>>> kubernetes
> >>>>> server.  At this point, I think I'm going to need to understand
> exactly
> >>>>> what that message means (other threads talking about this didn't make
> >>>> that
> >>>>> clear). Someone in that community did respond to my question, not
> with
> >>>> an
> >>>>> answer, but with a query for more information. I'll see what I can
> >>>> find out.
> >>>>
> >>>> A short Google search turned up that the message is apparently an HTTP
> >>>> 400 Bad
> >>>> Request response with Upgrade request message, meaning you need to use
> >>>> the
> >>>> websocket API to talk to a kubernetes pod:
> >>>>
> >>>>
> >>>>
> https://stackoverflow.com/questions/49250370/kubernetes-pod-exec-api-upgrade-request-required
> >>>> https://tools.ietf.org/html/rfc6455
> >>>> https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/101
> >>>
> >>>
> >>> Yes, I saw that thread in my research on this, but note that that
> >>> conclusion about using WebSocket is only from the first sentence of the
> >>> answer, whereas by the end of the answer, including the comment, it's
> clear
> >>> that this should be mitigated by just using "kubectl exec", which is
> what
> >>> I'm doing. In any case, it appears that even kubectl uses curl under
> the
> >>> covers. I would assume that kubectl attempts to mitigate the issues
> around
> >>> not using WebSocket for this, but it seems like it's not working at
> least
> >>> in my case.
> >>>
> >>> Another thought I had is whether I can coerce kubectl to use the
> Windows
> >>> curl by setting the PATH just before calling kubectl.  This didn't
> help. Is
> >>> there possibly something happening under the covers that would still
> make
> >>> this execute the Cygwin curl?
> >>>
> >>
> >> I have managed to resolve this.  The challenge here is finding the
> correct
> >> version of kubectl to use for the corresponding cluster version.  The
> rule
> >> is that you have to use a version of kubectl within one minor version of
> >> the cluster version.  We're using version 1.13.5 in the cluster.  I
> >> originally saw this problem with version 1.17, then moved to 1.18, then
> to
> >> 1.13.5.  Finally, version 1.14.0 resolves the problem. Version 1.13.5
> >> should have worked.  I don't know why it didn't.  I also have no idea
> why
> >> it would work in a non-cygwin shell.
> >>
> >>
> > Actually, I was wrong about this being resolved. For some reason it was
> > working properly for a short period after I installed v1.14.0, but now it
> > is consistently failing with "Upgrade request required" again.
> >
> > I've verified that the resulting kubectl call works fine from a cmd
> window.
> >
> > Another thing that I didn't mention before is that I run a Linux VM on
> this
> > laptop, and I can run the same scripting layer, albeit with a Linux
> install
> > of kubectl, and it works perfectly fine.
>
> Looks like curl has to send an Upgrade and other headers:
> https://gist.github.com/htp/fbce19069187ec1cc486b594104f01d0
> which kubectl exec should supply.
>
> Maybe time to start looking at what is passing over the net at both ends,
> as
> well as from the middle.
>

I've managed to resolve this.  The key is the proxy, as you indicate here.
My first real indication that this was the problem was that I started to
see this happen from my LInux VM, just this morning.  It's very curious
that this hasn't been happening all along. However, there have been some
global changes in the firewall recently, but that mostly consisted of
retiring an obsolete proxy and setting to use a new one, which I've been
configured to use for quite a while.

My solution is simply to not set the proxy in my shell environment
variables.  This won't really be a problem, as anything that requires an
external connection has tool configurations. I'm now fully working again
both in Cygwin and Linux VM.

Our theory about why this was having trouble is that using WebSocket
results in a direct connection from the server to an IP address on the
client. I have a feeling the proxy doesn't like that.