From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from smtp-out-no.shaw.ca (smtp-out-no.shaw.ca [64.59.134.13]) by sourceware.org (Postfix) with ESMTPS id E2868393C89A for ; Thu, 18 Jun 2020 00:27:35 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.3.2 sourceware.org E2868393C89A Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=SystematicSw.ab.ca Authentication-Results: sourceware.org; spf=none smtp.mailfrom=brian.inglis@systematicsw.ab.ca Received: from [192.168.1.104] ([24.64.172.44]) by shaw.ca with ESMTP id liP0jq4d262brliP1jh3bv; Wed, 17 Jun 2020 18:27:35 -0600 X-Authority-Analysis: v=2.3 cv=LKf9vKe9 c=1 sm=1 tr=0 a=kiZT5GMN3KAWqtYcXc+/4Q==:117 a=kiZT5GMN3KAWqtYcXc+/4Q==:17 a=IkcTkHD0fZMA:10 a=0MaIOF3bAAAA:8 a=uPZiAMpXAAAA:8 a=48vgC7mUAAAA:8 a=pQs5aej7AAAA:8 a=J7xhV7lfAAAA:20 a=GwtC8ItWuLMr9VDuWoQA:9 a=QEXdDO2ut3YA:10 a=q-wEALBr1heszoC4fXDA:22 a=w1C3t2QeGrPiZgrLijVG:22 Subject: Re: "kubectl exec" in Cygwin gets "Upgrade request required", but not in cmd shell To: cygwin@cygwin.com References: <70ec2ece-c85e-c122-b4a0-3089da5c8a2c@SystematicSw.ab.ca> <3de6e80d-5ba5-3d12-f527-97b8ff3eaae7@SystematicSw.ab.ca> Reply-To: cygwin@cygwin.com From: Brian Inglis Autocrypt: addr=Brian.Inglis@SystematicSw.ab.ca; prefer-encrypt=mutual; keydata= mDMEXopx8xYJKwYBBAHaRw8BAQdAnCK0qv/xwUCCZQoA9BHRYpstERrspfT0NkUWQVuoePa0 LkJyaWFuIEluZ2xpcyA8QnJpYW4uSW5nbGlzQFN5c3RlbWF0aWNTdy5hYi5jYT6IlgQTFggA PhYhBMM5/lbU970GBS2bZB62lxu92I8YBQJeinHzAhsDBQkJZgGABQsJCAcCBhUKCQgLAgQW AgMBAh4BAheAAAoJEB62lxu92I8Y0ioBAI8xrggNxziAVmr+Xm6nnyjoujMqWcq3oEhlYGAO WacZAQDFtdDx2koSVSoOmfaOyRTbIWSf9/Cjai29060fsmdsDLg4BF6KcfMSCisGAQQBl1UB BQEBB0Awv8kHI2PaEgViDqzbnoe8B9KMHoBZLS92HdC7ZPh8HQMBCAeIfgQYFggAJhYhBMM5 /lbU970GBS2bZB62lxu92I8YBQJeinHzAhsMBQkJZgGAAAoJEB62lxu92I8YZwUBAJw/74rF IyaSsGI7ewCdCy88Lce/kdwX7zGwid+f8NZ3AQC/ezTFFi5obXnyMxZJN464nPXiggtT9gN5 RSyTY8X+AQ== Organization: Systematic Software Message-ID: Date: Wed, 17 Jun 2020 18:27:34 -0600 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101 Thunderbird/68.9.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-CA Content-Transfer-Encoding: 7bit X-CMAE-Envelope: MS4wfLo9hgqDMYjHCr/g5aP9Jm2vjy2rSEdIi9der+Y/S2dtjshPRBPD3Rs16H92Qk0LtRoM8E2Q/z4oFHzkQxp1jkgR85SubL9xiFhzsSGYLg8Uds5IGkre Q7K96oyCaFCTT0c2teeQO8cbF6E9Wl3LIp770yYrahTmgl47CcAOQvwH4bTDDdHniQ7vL2y2FhuqIQ== X-Spam-Status: No, score=-11.3 required=5.0 tests=BAYES_00, KAM_DMARC_STATUS, KAM_LAZY_DOMAIN_SECURITY, RCVD_IN_DNSWL_LOW, SPF_HELO_NONE, SPF_NONE, TXREP autolearn=no autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on server2.sourceware.org X-BeenThere: cygwin@cygwin.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: General Cygwin discussions and problem reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Jun 2020 00:27:39 -0000 On 2020-06-17 10:39, David Karr via Cygwin wrote: > On Mon, Jun 15, 2020 at 9:10 AM David Karr wrote: >> On Sun, Jun 14, 2020 at 12:32 PM David Karr wrote: >>> On Sun, Jun 14, 2020 at 12:09 PM Brian Inglis wrote: >>>> On 2020-06-14 12:16, David Karr via Cygwin wrote: >>>>> On Sun, Jun 14, 2020 at 10:20 AM Brian Inglis wrote: >>>>>> On 2020-06-14 09:38, David Karr via Cygwin wrote: >>>>>>> On Sun, Jun 14, 2020 at 2:25 AM Marco Atzeri wrote: >>>>>>>> On 14.06.2020 08:12, David Karr wrote: >>>>>>>>> On Sat, Jun 13, 2020 at 10:31 PM Marco Atzeri via Cygwin wrote: >>>>>>>>>> On 13.06.2020 20:53, David Karr via Cygwin wrote: >>>>>>>>>>> I've been using kubectl in Cygwin on Windows 10 for quite a >>>> while, >>>>>>>>>>> to communicate to our in-house k8s clusters. I often use "kubectl >>>>>>>>>>> exec" to open a shell in a container or directly execute a shell >>>>>>>>>>> command. >>>>>>>>>>> This has worked perfectly fine for a long time. >>>>>>>>>>> A couple of days ago, I discovered that all of these attempts >>>> were >>>>>>>>>>> failing with "Upgrade request required". I hadn't upgraded >>>> kubectl >>>>>>>>>>> or Cygwin in quite a while. I doubt our clusters had a k8s >>>> upgrade, >>>>>>>>>>> but it's entirely possible. >>>>>>>>>>> A colleague of mine has a very similar desktop configuration >>>>>>>>>>> (Windows 10, Cygwin), and he's not seeing this symptom. >>>>>>>>>>> I noticed that when I ran "kubectl exec" with max verbosity, it >>>> shows >>>>>>>>>>> the resulting "curl" command that it runs. I tried that resulting >>>>>>>>>>> command, and it results in the same response. I then tried >>>> updating >>>>>>>>>>> my Cygwin tools and retesting, no change.>>>>> I then took the >>>>>> entire resulting "kubectl exec" command line and ran >>>>>>>>>>> it in a "cmd" shell. No problem at all. No error. >>>>>>>>>>> I know I haven't provided much useful information yet. I wanted >>>> to >>>>>>>>>>> get an initial response before I started providing those >>>> diagnostics. >>>>>>>>>>> Is there a clear issue here that I'm not aware of? >>>>>> >>>>>>>>> from where is kubectl coming from ? >>>>>>>>> In cygwin I found only a kubectl.py in the ansible package >>>>>> >>>>>>>>>> It's from here: >>>>>>>>>> >>>>>> >>>> https://kubernetes.io/docs/tasks/tools/install-kubectl/#install-kubectl-on-windows >>>>>> >>>>>>>> so it is NOT a cygwin program. >>>>>>>> If the warning is coming about curl, it is likely >>>>>>>> that using from cygwin you are using the cygwin curl >>>>>>>> and from CMD the windows one >>>>>>>> $ which -a curl >>>>>>>> /usr/bin/curl >>>>>>>> /cygdrive/c/WINDOWS/system32/curl >>>>>>>> $ /cygdrive/c/WINDOWS/system32/curl -V >>>>>>>> curl 7.55.1 (Windows) libcurl/7.55.1 WinSSL >>>>>>>> Release-Date: 2017-11-14, security patched: 2019-11-05 >>>>>>>> Protocols: dict file ftp ftps http https imap imaps pop3 pop3s smtp >>>>>> smtps >>>>>>>> telnet tftp >>>>>>>> Features: AsynchDNS IPv6 Largefile SSPI Kerberos SPNEGO NTLM SSL >>>>>>>> >>>>>>>> $ /usr/bin/curl -V >>>>>>>> curl 7.66.0 (x86_64-pc-cygwin) libcurl/7.66.0 OpenSSL/1.1.1f >>>> zlib/1.2.11 >>>>>>>> brotli/1.0.7 libidn2/2.2.0 libpsl/0.21.0 (+libidn2/2.0.4) >>>>>>>> libssh/0.8.7/openssl/zlib nghttp2/1.37.0 >>>>>>>> Release-Date: 2019-09-11 >>>>>>>> Protocols: dict file ftp ftps gopher http https imap imaps ldap >>>> ldaps >>>>>>>> pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp >>>>>>>> Features: AsynchDNS brotli Debug GSS-API HTTP2 HTTPS-proxy IDN IPv6 >>>>>>>> Kerberos Largefile libz Metalink NTLM NTLM_WB PSL SPNEGO SSL TLS-SRP >>>>>>>> TrackMemory UnixSockets >>>>>>>> the support Forum https://discuss.kubernetes.io/ >>>>>>>> is probably the most indicate place for guidance >>>>>> >>>>>>> I thought it was obvious that it was not working because it was >>>> calling >>>>>> the >>>>>>> Cygwin curl. I wouldn't have posted here if that wasn't obvious to >>>> me. >>>>>>> And since I'm well aware of the k8s community, I already posted >>>> questions >>>>>>> about this in the appropriate place, before I posted here. >>>>>>> What I was hoping to get here was some indication or thoughts on why >>>> a >>>>>>> process using Windows curl doesn't have a problem, but does have a >>>>>> problem >>>>>>> when using Cygwin Curl. This isn't likely something that Cygwin curl >>>> is >>>>>>> doing "wrong", it's just that it's doing something different. >>>>>>> If it matters, the following is an elided version of the resulting >>>> curl >>>>>>> command: >>>>>>> curl -k -v -XPOST -H "User-Agent: kubectl.exe/v1.18.0 >>>>>> (windows/amd64) >>>>>>> kubernetes/9e99141" -H "Authorization: Bearer ..." -H >>>>>>> "X-Stream-Protocol-Version: v4.channel.k8s.io" -H >>>>>>> "X-Stream-Protocol-Version: v3.channel.k8s.io" -H >>>>>>> "X-Stream-Protocol-Version: v2.channel.k8s.io" -H >>>>>>> "X-Stream-Protocol-Version: channel.k8s.io" 'https:// >>>>>>> >>>>>> >>>> .../api/v1/namespaces/.../pods/.../exec?command=%2Fbin%2Fls&container=...&stderr=true&stdin=true&stdout=true' >>>>>>> I can't tell from the logging what request body it sent. It's >>>> possible it >>>>>>> didn't send any. >>>>>> >>>>>> We can't tell, not knowing who you are, and from what you are posting, >>>>>> what may >>>>>> or may not be obvious to you, where you are seeing "Upgrade request >>>>>> required", >>>>>> or where that may be coming from. >>>>>> That you need diagnostic help indicates that, what may appear obvious >>>> to >>>>>> you, >>>>>> may not be the case, as it is often our assumptions which lead us >>>> astray, >>>>>> and we >>>>>> get daily proof that we are imperfect, which is why most of seek to >>>> talk >>>>>> over >>>>>> and explain our issues to inanimate objects or colleagues e.g. talk >>>> to the >>>>>> rubber duck, teddy bear, plush hippo, etc. (My shelf holds two ducks >>>> and a >>>>>> pelican awarded by former projects.) >>>>>> >>>>>> For help with Cygwin, we need to see *whole* commands and all the >>>> output >>>>>> between >>>>>> the shell prompts, preferably with context, in this case including >>>> PATH >>>>>> under >>>>>> Cygwin and MS Windows and/or program paths typed or invoked. >>>>>> >>>>>> You may retain privacy and security by substituting variables e.g. env >>>>>> vars $VAR >>>>>> to sanitize sensitive information: to avoid problems I use a script >>>> to do >>>>>> so >>>>>> that logs are never written containing sensitive content. >>>>>> >>>>> >>>>> Acknowledged. The "Upgrade request required" is coming from the >>>> kubernetes >>>>> server. At this point, I think I'm going to need to understand exactly >>>>> what that message means (other threads talking about this didn't make >>>> that >>>>> clear). Someone in that community did respond to my question, not with >>>> an >>>>> answer, but with a query for more information. I'll see what I can >>>> find out. >>>> >>>> A short Google search turned up that the message is apparently an HTTP >>>> 400 Bad >>>> Request response with Upgrade request message, meaning you need to use >>>> the >>>> websocket API to talk to a kubernetes pod: >>>> >>>> >>>> https://stackoverflow.com/questions/49250370/kubernetes-pod-exec-api-upgrade-request-required >>>> https://tools.ietf.org/html/rfc6455 >>>> https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/101 >>> >>> >>> Yes, I saw that thread in my research on this, but note that that >>> conclusion about using WebSocket is only from the first sentence of the >>> answer, whereas by the end of the answer, including the comment, it's clear >>> that this should be mitigated by just using "kubectl exec", which is what >>> I'm doing. In any case, it appears that even kubectl uses curl under the >>> covers. I would assume that kubectl attempts to mitigate the issues around >>> not using WebSocket for this, but it seems like it's not working at least >>> in my case. >>> >>> Another thought I had is whether I can coerce kubectl to use the Windows >>> curl by setting the PATH just before calling kubectl. This didn't help. Is >>> there possibly something happening under the covers that would still make >>> this execute the Cygwin curl? >>> >> >> I have managed to resolve this. The challenge here is finding the correct >> version of kubectl to use for the corresponding cluster version. The rule >> is that you have to use a version of kubectl within one minor version of >> the cluster version. We're using version 1.13.5 in the cluster. I >> originally saw this problem with version 1.17, then moved to 1.18, then to >> 1.13.5. Finally, version 1.14.0 resolves the problem. Version 1.13.5 >> should have worked. I don't know why it didn't. I also have no idea why >> it would work in a non-cygwin shell. >> >> > Actually, I was wrong about this being resolved. For some reason it was > working properly for a short period after I installed v1.14.0, but now it > is consistently failing with "Upgrade request required" again. > > I've verified that the resulting kubectl call works fine from a cmd window. > > Another thing that I didn't mention before is that I run a Linux VM on this > laptop, and I can run the same scripting layer, albeit with a Linux install > of kubectl, and it works perfectly fine. Looks like curl has to send an Upgrade and other headers: https://gist.github.com/htp/fbce19069187ec1cc486b594104f01d0 which kubectl exec should supply. Maybe time to start looking at what is passing over the net at both ends, as well as from the middle. -- Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada This email may be disturbing to some readers as it contains too much technical detail. Reader discretion is advised. [Data in IEC units and prefixes, physical quantities in SI.]