From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-oi1-x22f.google.com (mail-oi1-x22f.google.com [IPv6:2607:f8b0:4864:20::22f]) by sourceware.org (Postfix) with ESMTPS id A9CDA3890409 for ; Wed, 17 Jun 2020 16:39:25 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.3.2 sourceware.org A9CDA3890409 Received: by mail-oi1-x22f.google.com with SMTP id k4so2353310oik.2 for ; Wed, 17 Jun 2020 09:39:25 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=L7eDHEGqZTKWdDiLAqWD3faanMpTQroM2Fiu6vHN8ac=; b=XIYeYzB25KwXmAv/U1wng6ByNw1LiXSMOqbtNzdbxJzrnj1Nteb8AXJBNHtHTwjLyN v4vdj43DMi0V5YZDLGoqL0qdR7wogl5VsSpsexKEzWH+cGX02rXec5BlGyez17lUzI9r 80e5AI73GFThfOpy50+6FEr683ryeyNfTs5h3P87zPE+i7NcYWyZe0vcT0o62JniDOO/ hGGGtlBK8zGia1/lr3ev+tmw/gRbw8n56iMEiPZdisVLXsAxXRaUvHQC8em28Dg5JiJc nrw40eQoqNUbbHTv2+xPooiSV5iFca74GuL+xq+EoLc6ZNuK/RWPNFNfq9kxC4T9EZgb CgcQ== X-Gm-Message-State: AOAM530X+028471STqO3xzZmDf1OIA5GZa9qJVE1q4gJg1LicX3xO9uC cJRT6wIWtH8x+M9rPRwNHrzTiTxY6niCgmC9n7e67qNZ X-Google-Smtp-Source: ABdhPJxchp44dbu8EFCXwJLyuLnTzaZ90oqsDwGHACKOZW4tjTnO5qSnjaqc/KCXBRfutaAoi3de2/fGSEGKFaflna8= X-Received: by 2002:a05:6808:4b:: with SMTP id v11mr7804867oic.31.1592411964513; Wed, 17 Jun 2020 09:39:24 -0700 (PDT) MIME-Version: 1.0 References: <70ec2ece-c85e-c122-b4a0-3089da5c8a2c@SystematicSw.ab.ca> <3de6e80d-5ba5-3d12-f527-97b8ff3eaae7@SystematicSw.ab.ca> In-Reply-To: From: David Karr Date: Wed, 17 Jun 2020 09:39:13 -0700 Message-ID: Subject: Re: "kubectl exec" in Cygwin gets "Upgrade request required", but not in cmd shell To: The Cygwin Mailing List X-Spam-Status: No, score=-1.7 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, FREEMAIL_FROM, HTML_MESSAGE, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on server2.sourceware.org Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: cygwin@cygwin.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: General Cygwin discussions and problem reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Jun 2020 16:39:28 -0000 On Mon, Jun 15, 2020 at 9:10 AM David Karr wrote: > > On Sun, Jun 14, 2020 at 12:32 PM David Karr wrote: > >> >> On Sun, Jun 14, 2020 at 12:09 PM Brian Inglis < >> Brian.Inglis@systematicsw.ab.ca> wrote: >> >>> On 2020-06-14 12:16, David Karr via Cygwin wrote: >>> > On Sun, Jun 14, 2020 at 10:20 AM Brian Inglis < >>> > Brian.Inglis@systematicsw.ab.ca> wrote: >>> > >>> >> On 2020-06-14 09:38, David Karr via Cygwin wrote: >>> >>> On Sun, Jun 14, 2020 at 2:25 AM Marco Atzeri wrote: >>> >>>> On 14.06.2020 08:12, David Karr wrote: >>> >>>>> On Sat, Jun 13, 2020 at 10:31 PM Marco Atzeri via Cygwin wrote: >>> >>>>>> On 13.06.2020 20:53, David Karr via Cygwin wrote: >>> >>>>>>> I've been using kubectl in Cygwin on Windows 10 for quite a >>> while, >>> >>>>>>> to communicate to our in-house k8s clusters. I often use "kubectl >>> >>>>>>> exec" to open a shell in a container or directly execute a shell >>> >>>>>>> command. >>> >>>>>>> This has worked perfectly fine for a long time. >>> >>>>>>> A couple of days ago, I discovered that all of these attempts >>> were >>> >>>>>>> failing with "Upgrade request required". I hadn't upgraded >>> kubectl >>> >>>>>>> or Cygwin in quite a while. I doubt our clusters had a k8s >>> upgrade, >>> >>>>>>> but it's entirely possible. >>> >>>>>>> A colleague of mine has a very similar desktop configuration >>> >>>>>>> (Windows 10, Cygwin), and he's not seeing this symptom. >>> >>>>>>> I noticed that when I ran "kubectl exec" with max verbosity, it >>> shows >>> >>>>>>> the resulting "curl" command that it runs. I tried that resulting >>> >>>>>>> command, and it results in the same response. I then tried >>> updating >>> >>>>>>> my Cygwin tools and retesting, no change.>>>>> I then took the >>> >> entire resulting "kubectl exec" command line and ran >>> >>>>>>> it in a "cmd" shell. No problem at all. No error. >>> >>>>>>> I know I haven't provided much useful information yet. I wanted >>> to >>> >>>>>>> get an initial response before I started providing those >>> diagnostics. >>> >>>>>>> Is there a clear issue here that I'm not aware of? >>> >> >>> >>>>> from where is kubectl coming from ? >>> >>>>> In cygwin I found only a kubectl.py in the ansible package >>> >> >>> >>>>>> It's from here: >>> >>>>>> >>> >> >>> https://kubernetes.io/docs/tasks/tools/install-kubectl/#install-kubectl-on-windows >>> >> >>> >>>> so it is NOT a cygwin program. >>> >>>> If the warning is coming about curl, it is likely >>> >>>> that using from cygwin you are using the cygwin curl >>> >>>> and from CMD the windows one >>> >>>> $ which -a curl >>> >>>> /usr/bin/curl >>> >>>> /cygdrive/c/WINDOWS/system32/curl >>> >>>> $ /cygdrive/c/WINDOWS/system32/curl -V >>> >>>> curl 7.55.1 (Windows) libcurl/7.55.1 WinSSL >>> >>>> Release-Date: 2017-11-14, security patched: 2019-11-05 >>> >>>> Protocols: dict file ftp ftps http https imap imaps pop3 pop3s smtp >>> >> smtps >>> >>>> telnet tftp >>> >>>> Features: AsynchDNS IPv6 Largefile SSPI Kerberos SPNEGO NTLM SSL >>> >>>> >>> >>>> $ /usr/bin/curl -V >>> >>>> curl 7.66.0 (x86_64-pc-cygwin) libcurl/7.66.0 OpenSSL/1.1.1f >>> zlib/1.2.11 >>> >>>> brotli/1.0.7 libidn2/2.2.0 libpsl/0.21.0 (+libidn2/2.0.4) >>> >>>> libssh/0.8.7/openssl/zlib nghttp2/1.37.0 >>> >>>> Release-Date: 2019-09-11 >>> >>>> Protocols: dict file ftp ftps gopher http https imap imaps ldap >>> ldaps >>> >>>> pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp >>> >>>> Features: AsynchDNS brotli Debug GSS-API HTTP2 HTTPS-proxy IDN IPv6 >>> >>>> Kerberos Largefile libz Metalink NTLM NTLM_WB PSL SPNEGO SSL TLS-SRP >>> >>>> TrackMemory UnixSockets >>> >>>> the support Forum https://discuss.kubernetes.io/ >>> >>>> is probably the most indicate place for guidance >>> >> >>> >>> I thought it was obvious that it was not working because it was >>> calling >>> >> the >>> >>> Cygwin curl. I wouldn't have posted here if that wasn't obvious to >>> me. >>> >>> And since I'm well aware of the k8s community, I already posted >>> questions >>> >>> about this in the appropriate place, before I posted here. >>> >>> What I was hoping to get here was some indication or thoughts on why >>> a >>> >>> process using Windows curl doesn't have a problem, but does have a >>> >> problem >>> >>> when using Cygwin Curl. This isn't likely something that Cygwin curl >>> is >>> >>> doing "wrong", it's just that it's doing something different. >>> >>> If it matters, the following is an elided version of the resulting >>> curl >>> >>> command: >>> >>> curl -k -v -XPOST -H "User-Agent: kubectl.exe/v1.18.0 >>> >> (windows/amd64) >>> >>> kubernetes/9e99141" -H "Authorization: Bearer ..." -H >>> >>> "X-Stream-Protocol-Version: v4.channel.k8s.io" -H >>> >>> "X-Stream-Protocol-Version: v3.channel.k8s.io" -H >>> >>> "X-Stream-Protocol-Version: v2.channel.k8s.io" -H >>> >>> "X-Stream-Protocol-Version: channel.k8s.io" 'https:// >>> >>> >>> >> >>> .../api/v1/namespaces/.../pods/.../exec?command=%2Fbin%2Fls&container=...&stderr=true&stdin=true&stdout=true' >>> >>> I can't tell from the logging what request body it sent. It's >>> possible it >>> >>> didn't send any. >>> >> >>> >> We can't tell, not knowing who you are, and from what you are posting, >>> >> what may >>> >> or may not be obvious to you, where you are seeing "Upgrade request >>> >> required", >>> >> or where that may be coming from. >>> >> That you need diagnostic help indicates that, what may appear obvious >>> to >>> >> you, >>> >> may not be the case, as it is often our assumptions which lead us >>> astray, >>> >> and we >>> >> get daily proof that we are imperfect, which is why most of seek to >>> talk >>> >> over >>> >> and explain our issues to inanimate objects or colleagues e.g. talk >>> to the >>> >> rubber duck, teddy bear, plush hippo, etc. (My shelf holds two ducks >>> and a >>> >> pelican awarded by former projects.) >>> >> >>> >> For help with Cygwin, we need to see *whole* commands and all the >>> output >>> >> between >>> >> the shell prompts, preferably with context, in this case including >>> PATH >>> >> under >>> >> Cygwin and MS Windows and/or program paths typed or invoked. >>> >> >>> >> You may retain privacy and security by substituting variables e.g. env >>> >> vars $VAR >>> >> to sanitize sensitive information: to avoid problems I use a script >>> to do >>> >> so >>> >> that logs are never written containing sensitive content. >>> >> >>> > >>> > Acknowledged. The "Upgrade request required" is coming from the >>> kubernetes >>> > server. At this point, I think I'm going to need to understand exactly >>> > what that message means (other threads talking about this didn't make >>> that >>> > clear). Someone in that community did respond to my question, not with >>> an >>> > answer, but with a query for more information. I'll see what I can >>> find out. >>> >>> A short Google search turned up that the message is apparently an HTTP >>> 400 Bad >>> Request response with Upgrade request message, meaning you need to use >>> the >>> websocket API to talk to a kubernetes pod: >>> >>> >>> https://stackoverflow.com/questions/49250370/kubernetes-pod-exec-api-upgrade-request-required >>> https://tools.ietf.org/html/rfc6455 >>> https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/101 >> >> >> Yes, I saw that thread in my research on this, but note that that >> conclusion about using WebSocket is only from the first sentence of the >> answer, whereas by the end of the answer, including the comment, it's clear >> that this should be mitigated by just using "kubectl exec", which is what >> I'm doing. In any case, it appears that even kubectl uses curl under the >> covers. I would assume that kubectl attempts to mitigate the issues around >> not using WebSocket for this, but it seems like it's not working at least >> in my case. >> >> Another thought I had is whether I can coerce kubectl to use the Windows >> curl by setting the PATH just before calling kubectl. This didn't help. Is >> there possibly something happening under the covers that would still make >> this execute the Cygwin curl? >> > > I have managed to resolve this. The challenge here is finding the correct > version of kubectl to use for the corresponding cluster version. The rule > is that you have to use a version of kubectl within one minor version of > the cluster version. We're using version 1.13.5 in the cluster. I > originally saw this problem with version 1.17, then moved to 1.18, then to > 1.13.5. Finally, version 1.14.0 resolves the problem. Version 1.13.5 > should have worked. I don't know why it didn't. I also have no idea why > it would work in a non-cygwin shell. > > Actually, I was wrong about this being resolved. For some reason it was working properly for a short period after I installed v1.14.0, but now it is consistently failing with "Upgrade request required" again. I've verified that the resulting kubectl call works fine from a cmd window. Another thing that I didn't mention before is that I run a Linux VM on this laptop, and I can run the same scripting layer, albeit with a Linux install of kubectl, and it works perfectly fine.