Re: OpenSSL failure in Cygwin: SSL_set_tlsext_host_name returns 1 (SSL_TLSEXT_ERR_ALERT_WARNING)

[Yuri <yuri@rawbw.com>]

On 5/8/23 08:31, Brian Inglis wrote:
> Which Cygwin, ssl/tls-devel libraries, and ca-certificates... packages 
> and versions are you using?

libssl-devel-1.1.1t-1

openssl-1.1.1t-1

ca-certificates-2021.2.60-1


>
>     $ man SSL_set_tlsext_host_name
>
> says SSL_set_tlsext_host_name etc. returns 1 for success, 0 for failure?

Yes. It looks like there is a coding error that they don't just check 
for 0 or 1, and check for 0 as a sign of success.

But this code, amazingly, works flawlessly on Linux/BSD.


>
> Web search TLS SNI and you will find that either the host presents a 
> list of certs none of which match the host name you are connecting to, 
> a matching cert cannot be validated, possibly due to a missing CA 
> chain, or one end could not handle the list presented or cert matched; 
> some hits offer diagnostic suggestions.
>


This program has a special variable no_check_cert_flag that allows to 
disable certificate check: 
https://github.com/proxytunnel/proxytunnel/blob/master/ptstream.c#L356

In my case the certificate is self-signed and this variable is activated.

On Linux the same invocation doesn't cause such failure.

Is this code incorrect?


I will report the incorrect use of SSL_set_tlsext_host_name to proxytunnel.



Yuri