Setup 2.917 fails to load mirror list

Setup 2.917 fails to load mirror list

[Vanda Vodkamilkevich]

Hi,
When behind a (corporate) proxy the mirror list is apparently fetched
without using the proxy configuration. This can only be seen in a fresh
install, I tried with 2.908 and it worked then the new version can use the
cached file...
I'll try to add the logs provided by '-v' but it's complicated to get them
out of my corporate network.
I can't debug more precisely as I have to reinstall my complete cygwin tree
before :-(

BTW (I know it's ugly to have 2 subjects in the same mail) I have updated
the French translations for setup and I don't really know how to proceed
with the new versions... Any hints?

Cheers...

Re: Setup 2.917 fails to load mirror list

[Brian Inglis]

On 2022-02-08 11:09, Vanda Vodkamilkevich wrote:
> I have updated the French translations for setup and I don't really
> know how to proceed with the new versions... Any hints?
Clone the git repo

	http://cygwin.com/git/cygwin-apps/setup.git

and submit a git format-patch against the French localization resource:

https://cygwin.com/git/?p=cygwin-apps/setup.git;a=blob;f=res/fr/res.rc

using git send-email to cygwin-apps AT ...

-- 
Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada

This email may be disturbing to some readers as it contains
too much technical detail. Reader discretion is advised.
[Data in binary units and prefixes, physical quantities in SI.]

Re: Setup 2.917 fails to load mirror list

[Jon Turney]

On 08/02/2022 18:09, Vanda Vodkamilkevich wrote:
> Hi,
> When behind a (corporate) proxy the mirror list is apparently fetched
> without using the proxy configuration. This can only be seen in a fresh
> install, I tried with 2.908 and it worked then the new version can use the

Only appearing on a fresh install makes sense, as a failure to fetch the 
mirror list is silent is we have a cached mirror list data.

> cached file...
> I'll try to add the logs provided by '-v' but it's complicated to get them
> out of my corporate network.
> I can't debug more precisely as I have to reinstall my complete cygwin tree
> before :-(

This is puzzling, since I don't see any changes which could cause this.

It might be helpful to bisect which version the problem first appears 
in.  Old setup versions are available at [1].

[1] https://cygwin.com/setup/

Re: Setup 2.917 fails to load mirror list

[Jon Turney]

> Le mer. 9 févr. 2022 à 12:11, Jon Turney a
> écrit :
> 
>> On 08/02/2022 18:09, Vanda Vodkamilkevich wrote:
>>> Hi,
>>> When behind a (corporate) proxy the mirror list is apparently fetched
>>> without using the proxy configuration. This can only be seen in a fresh
>>> install, I tried with 2.908 and it worked then the new version can use
>> the
>>
>> Only appearing on a fresh install makes sense, as a failure to fetch the
>> mirror list is silent is we have a cached mirror list data.
>>
>>> cached file...
>>> I'll try to add the logs provided by '-v' but it's complicated to get
>> them
>>> out of my corporate network.
>>> I can't debug more precisely as I have to reinstall my complete cygwin
>> tree
>>> before :-(
>>
>> This is puzzling, since I don't see any changes which could cause this.
>>

On 09/02/2022 15:35, Vanda Vodkamilkevich wrote:
 > If it helps, the output log when I saw the issues with setup

> ########### Try to download with proxy set
[...]
> Cached mirror list unavailable
[...]
> HTTP status 403 fetching https://cygwin.com/mirrors.lst

> ########### Using 2.908 version: it works
[...]
> Cached mirror list unavailable
[...]
> Fetched URL: http://cygwin.com/mirrors.lst

> ########### Rerun with new version
[...]
> Loaded cached mirror list
[...]> connection error: 12057 fetching https://cygwin.com/mirrors.lst
> Using cached mirror list

The significant change seems to be we now fetch the mirror list using 
https (since 2.892, but since you are using a self-built setup with 
local changes, you don't seem to have picked that up until now)

12057 is ERROR_INTERNET_SEC_CERT_REV_FAILED, which leads down quite a 
rabbit hole, but apparently this means something like 'certificate 
validity isn't checked in the process using wininet, but in a service, 
which doesn't have access to the proxy credentials we are using, so 
fails trying to fetch any CRL'.

You don't mention that your proxy actually needs any credentials.

Why we get a different error code the second time is mysterious.

How we can then go on to successfully fetch from a https:// mirror if it 
presents a CRL doesn't make a lot of sense.

I'm baffled.

Re: Setup 2.917 fails to load mirror list

[Vanda Vodkamilkevich]

Le jeu. 10 févr. 2022 à 14:54, Jon Turney <jon.turney@dronecode.org.uk> a
écrit :

> > Le mer. 9 févr. 2022 à 12:11, Jon Turney a
> > écrit :
> >
> >> On 08/02/2022 18:09, Vanda Vodkamilkevich wrote:
> >>> Hi,
> >>> When behind a (corporate) proxy the mirror list is apparently fetched
> >>> without using the proxy configuration. This can only be seen in a fresh
> >>> install, I tried with 2.908 and it worked then the new version can use
> >> the
> >>
> >> Only appearing on a fresh install makes sense, as a failure to fetch the
> >> mirror list is silent is we have a cached mirror list data.
> >>
> >>> cached file...
> >>> I'll try to add the logs provided by '-v' but it's complicated to get
> >> them
> >>> out of my corporate network.
> >>> I can't debug more precisely as I have to reinstall my complete cygwin
> >> tree
> >>> before :-(
> >>
> >> This is puzzling, since I don't see any changes which could cause this.
> >>
>
> On 09/02/2022 15:35, Vanda Vodkamilkevich wrote:
>  > If it helps, the output log when I saw the issues with setup
>
> > ########### Try to download with proxy set
> [...]
> > Cached mirror list unavailable
> [...]
> > HTTP status 403 fetching https://cygwin.com/mirrors.lst
>
> > ########### Using 2.908 version: it works
> [...]
> > Cached mirror list unavailable
> [...]
> > Fetched URL: http://cygwin.com/mirrors.lst
>
> > ########### Rerun with new version
> [...]
> > Loaded cached mirror list
> [...]> connection error: 12057 fetching https://cygwin.com/mirrors.lst
> > Using cached mirror list
>
> The significant change seems to be we now fetch the mirror list using
> https (since 2.892, but since you are using a self-built setup with
> local changes, you don't seem to have picked that up until now)
>
> 12057 is ERROR_INTERNET_SEC_CERT_REV_FAILED, which leads down quite a
> rabbit hole, but apparently this means something like 'certificate
> validity isn't checked in the process using wininet, but in a service,
> which doesn't have access to the proxy credentials we are using, so
> fails trying to fetch any CRL'.
>
> You don't mention that your proxy actually needs any credentials.
>
> Why we get a different error code the second time is mysterious.
>
> How we can then go on to successfully fetch from a https:// mirror if it
> presents a CRL doesn't make a lot of sense.
>
> I'm baffled.
>

You nailed it... My corporate proxy blocks the https to the mirror list.
And my old version of setup was using http.

Maybe if https failed you should retry with http?
Btw where is this mirror list file saved? I could cheat by fetching it with
http before using setup?

Re: Setup 2.917 fails to load mirror list

[Adam Dinwoodie]

On Thu, Feb 10, 2022 at 03:49:38PM +0100, Vanda Vodkamilkevich wrote:
> Le jeu. 10 févr. 2022 à 14:54, Jon Turney <jon.turney@dronecode.org.uk> a
> écrit :
> 
> > > Le mer. 9 févr. 2022 à 12:11, Jon Turney a
> > > écrit :
> > >
> > >> On 08/02/2022 18:09, Vanda Vodkamilkevich wrote:
> > >>> Hi,
> > >>> When behind a (corporate) proxy the mirror list is apparently fetched
> > >>> without using the proxy configuration. This can only be seen in a fresh
> > >>> install, I tried with 2.908 and it worked then the new version can use
> > >> the
> > >>
> > >> Only appearing on a fresh install makes sense, as a failure to fetch the
> > >> mirror list is silent is we have a cached mirror list data.
> > >>
> > >>> cached file...
> > >>> I'll try to add the logs provided by '-v' but it's complicated to get
> > >> them
> > >>> out of my corporate network.
> > >>> I can't debug more precisely as I have to reinstall my complete cygwin
> > >> tree
> > >>> before :-(
> > >>
> > >> This is puzzling, since I don't see any changes which could cause this.
> > >>
> >
> > On 09/02/2022 15:35, Vanda Vodkamilkevich wrote:
> >  > If it helps, the output log when I saw the issues with setup
> >
> > > ########### Try to download with proxy set
> > [...]
> > > Cached mirror list unavailable
> > [...]
> > > HTTP status 403 fetching https://cygwin.com/mirrors.lst
> >
> > > ########### Using 2.908 version: it works
> > [...]
> > > Cached mirror list unavailable
> > [...]
> > > Fetched URL: http://cygwin.com/mirrors.lst
> >
> > > ########### Rerun with new version
> > [...]
> > > Loaded cached mirror list
> > [...]> connection error: 12057 fetching https://cygwin.com/mirrors.lst
> > > Using cached mirror list
> >
> > The significant change seems to be we now fetch the mirror list using
> > https (since 2.892, but since you are using a self-built setup with
> > local changes, you don't seem to have picked that up until now)
> >
> > 12057 is ERROR_INTERNET_SEC_CERT_REV_FAILED, which leads down quite a
> > rabbit hole, but apparently this means something like 'certificate
> > validity isn't checked in the process using wininet, but in a service,
> > which doesn't have access to the proxy credentials we are using, so
> > fails trying to fetch any CRL'.
> >
> > You don't mention that your proxy actually needs any credentials.
> >
> > Why we get a different error code the second time is mysterious.
> >
> > How we can then go on to successfully fetch from a https:// mirror if it
> > presents a CRL doesn't make a lot of sense.
> >
> > I'm baffled.
> >
> 
> You nailed it... My corporate proxy blocks the https to the mirror list.
> And my old version of setup was using http.
> 
> Maybe if https failed you should retry with http?

This is generally considered A Bad Idea; it may well be a good plan to
make the error behaviour clearer (this is basically always true of any
software), but automatically falling back to HTTP is the sort of thing
that will make the installer much more vulnerable to man-in-the-middle
attacks.

> Btw where is this mirror list file saved? I could cheat by fetching it with
> http before using setup?

Re: Setup 2.917 fails to load mirror list

[Jon Turney]

On 10/02/2022 14:49, Vanda Vodkamilkevich wrote:
> Le jeu. 10 févr. 2022 à 14:54, Jon Turney a écrit :
>> On 09/02/2022 15:35, Vanda Vodkamilkevich wrote:
>>> If it helps, the output log when I saw the issues with setup
>> 
>>> ########### Try to download with proxy set
>> [...]
>>> Cached mirror list unavailable
>> [...]
>>> HTTP status 403 fetching https://cygwin.com/mirrors.lst
>> 
>>> ########### Using 2.908 version: it works
>> [...]
>>> Cached mirror list unavailable
>> [...]
>>> Fetched URL: http://cygwin.com/mirrors.lst
>> 
>>> ########### Rerun with new version
>> [...]
>>> Loaded cached mirror list
>> [...]> connection error: 12057 fetching
>> https://cygwin.com/mirrors.lst
>>> Using cached mirror list
>> 
>> The significant change seems to be we now fetch the mirror list
>> using https (since 2.892, but since you are using a self-built
>> setup with local changes, you don't seem to have picked that up
>> until now)
>> 
>> 12057 is ERROR_INTERNET_SEC_CERT_REV_FAILED, which leads down quite
>> a rabbit hole, but apparently this means something like
>> 'certificate validity isn't checked in the process using wininet,
>> but in a service, which doesn't have access to the proxy
>> credentials we are using, so fails trying to fetch any CRL'.
>> 
>> You don't mention that your proxy actually needs any credentials.
>> 
>> Why we get a different error code the second time is mysterious.
>> 
>> How we can then go on to successfully fetch from a https:// mirror
>> if it presents a CRL doesn't make a lot of sense.
>> 
>> I'm baffled.
> 
> You nailed it... My corporate proxy blocks the https to the mirror
> list. And my old version of setup was using http.

This could mean:
- https is blocked by the proxy (due to policy or misconfiguration)
- https to cygwin.com is blocked by the proxy (ditto)
- the setup code is doing something wrong so that the proxy is blocking 
it's attempt to use http here

> Maybe if https failed you should retry with http?

Nope, for the reasons already given by Adam.

I'd *maybe* consider a patch adding an '--no-https' option which causes 
plain http:// to be used (and probably turns off [1] as well) to allow 
setup to run in environments which are hostile to https.

[1] 
https://cygwin.com/git/?p=cygwin-apps/setup.git;a=commitdiff;h=b4947fb6db0cbd8b0c673dc49a18224c44da8116;hp=57ddb743c06996e93567a98c6de6694ddcc5d616

> Btw where is this mirror list file saved? I could cheat by fetching
> it with http before using setup?

The 'cached mirror list' referred to here is stored in the mirrors-lst 
key in /etc/setup/setup.rc