setup 2.878 only processes one site

setup 2.878 only processes one site

[David Rothenberger]

When I use the new Direct Connection option in setup 2.878, it appears 
that setup only downloads the setup.xz file from one of the three sites 
I have selected.

Switching back to the Direct Connection (Legacy) option fixes the problem.

-- 
David Rothenberger  ----  daveroth@acm.org

How apt the poor are to be proud.
                 -- William Shakespeare, "Twelfth-Night"

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: setup 2.878 only processes one site

[Jon Turney]

On 17/05/17 00:14, David Rothenberger wrote:
> When I use the new Direct Connection option in setup 2.878, it appears
> that setup only downloads the setup.xz file from one of the three sites
> I have selected.

Interesting.  I don't see anything in the changes since 2.877 that would 
have caused this.  Can you show a setup.log.full?

Something which might give this impression is that setup will now cache 
setup.xz, and only download it if it has changed.

I had no idea anyone actually used this feature of setup.  I'm somewhat 
sceptical it's making things faster or more reliable, rather than just 
downloading the same file 3 times... :)


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: setup 2.878 only processes one site

[David Rothenberger]

[-- Attachment #1: Type: text/plain, Size: 1716 bytes --]

Jon Turney wrote:
> On 17/05/17 00:14, David Rothenberger wrote:
>> When I use the new Direct Connection option in setup 2.878, it appears
>> that setup only downloads the setup.xz file from one of the three sites
>> I have selected.
> 
> Interesting.  I don't see anything in the changes since 2.877 that would 
> have caused this.  Can you show a setup.log.full?

Attached, although I didn't see any difference in the log between using 
the new Direct Connection option and the old one.

> Something which might give this impression is that setup will now cache 
> setup.xz, and only download it if it has changed.
> 
> I had no idea anyone actually used this feature of setup.  I'm somewhat 
> sceptical it's making things faster or more reliable, rather than just 
> downloading the same file 3 times... :)

I think you nailed it. I tried again this morning and the first time I 
ran setup, it did contact our local site, but it did not on subsequent 
runs (even to see if the file has changed). I then modified our web 
server to add a "Cache-Control: no-cache" header and cleared the Windows 
cache. After that, setup retrieved the setup.xz file from our server on 
every run.

For background, we host our own partial repository of locally built 
packages, so the setup.xz our server provides is not complete. It would 
be incorrect for setup to assume that the setup.xz downloaded from one 
site is identical to the ones available from the other selected sites.

-- 
David Rothenberger  ----  daveroth@acm.org

Unfair animal names:

-- tsetse fly                   -- bullhead
-- booby                        -- duck-billed platypus
-- sapsucker                    -- Clarence
                 -- Gary Larson

[-- Attachment #2: setup.log.full --]
[-- Type: text/plain, Size: 3243 bytes --]

2017/05/17 10:02:26 Starting cygwin install, version 2.878
2017/05/17 10:02:26 User has backup/restore rights
2017/05/17 10:02:26 Current Directory: //zora/downloads/development/cygwin
2017/05/17 10:02:26 Could not open service McShield for query, start and stop. McAfee may not be installed, or we don't have access.
2017/05/17 10:02:51 source: network install
2017/05/17 10:02:52 root: c:\cygwin system
2017/05/17 10:02:52 Selected local directory: //zora/downloads/development/cygwin
2017/05/17 10:02:53 net: Direct
Loaded cached mirror list
get_url_to_membuf http://cygwin.com/mirrors.lst
getUrlToStream http://cygwin.com/mirrors.lst
2017/05/17 10:02:54 site: http://trac-dev.entomo.com/docs/cygwin-release/
2017/05/17 10:02:54 site: http://www.daveroth.dyndns.org/cygwin-release/
2017/05/17 10:02:54 site: http://mirrors.kernel.org/sourceware/cygwin/
get_url_to_membuf http://trac-dev.entomo.com/docs/cygwin-release/x86_64/setup.xz.sig
getUrlToStream http://trac-dev.entomo.com/docs/cygwin-release/x86_64/setup.xz.sig
get_url_to_membuf http://trac-dev.entomo.com/docs/cygwin-release/x86_64/setup.xz
getUrlToStream http://trac-dev.entomo.com/docs/cygwin-release/x86_64/setup.xz
get_url_to_membuf http://trac-dev.entomo.com/docs/cygwin-release/daveroth-gpg-pubkey.gpg
getUrlToStream http://trac-dev.entomo.com/docs/cygwin-release/daveroth-gpg-pubkey.gpg
get_url_to_membuf http://cygwinports.org/ports.gpg
getUrlToStream http://cygwinports.org/ports.gpg
compress_xz::bid_xz: success: 48
compress_xz::bid_xz: success: 48
INSTALLED.DB version 3
get_url_to_membuf http://www.daveroth.dyndns.org/cygwin-release/x86_64/setup.xz.sig
getUrlToStream http://www.daveroth.dyndns.org/cygwin-release/x86_64/setup.xz.sig
get_url_to_membuf http://www.daveroth.dyndns.org/cygwin-release/x86_64/setup.xz
getUrlToStream http://www.daveroth.dyndns.org/cygwin-release/x86_64/setup.xz
get_url_to_membuf http://trac-dev.entomo.com/docs/cygwin-release/daveroth-gpg-pubkey.gpg
getUrlToStream http://trac-dev.entomo.com/docs/cygwin-release/daveroth-gpg-pubkey.gpg
get_url_to_membuf http://cygwinports.org/ports.gpg
getUrlToStream http://cygwinports.org/ports.gpg
compress_xz::bid_xz: success: 48
compress_xz::bid_xz: success: 48
get_url_to_membuf http://mirrors.kernel.org/sourceware/cygwin/x86_64/setup.xz.sig
getUrlToStream http://mirrors.kernel.org/sourceware/cygwin/x86_64/setup.xz.sig
get_url_to_membuf http://mirrors.kernel.org/sourceware/cygwin/x86_64/setup.xz
getUrlToStream http://mirrors.kernel.org/sourceware/cygwin/x86_64/setup.xz
get_url_to_membuf http://trac-dev.entomo.com/docs/cygwin-release/daveroth-gpg-pubkey.gpg
getUrlToStream http://trac-dev.entomo.com/docs/cygwin-release/daveroth-gpg-pubkey.gpg
get_url_to_membuf http://cygwinports.org/ports.gpg
getUrlToStream http://cygwinports.org/ports.gpg
compress_xz::bid_xz: success: 48
compress_xz::bid_xz: success: 48
package calm comparing versions 20170327-1 and 20170327e-1, result was -1
package calm comparing versions 20170215-1 and 20170215e-1, result was -1
package chere comparing versions 1.4-1 and 1.2-1, result was 1
package keychain comparing versions 2.7.1-1 and 2.8.1-0, result was -1
.ini setup_version is (null), our setup_version is 2.878
2017/05/17 10:03:02 Ending cygwin install


[-- Attachment #3: Type: text/plain, Size: 219 bytes --]


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: setup 2.878 only processes one site

[Jon Turney]

On 17/05/2017 18:06, David Rothenberger wrote:
> Jon Turney wrote:
>> On 17/05/17 00:14, David Rothenberger wrote:
>>> When I use the new Direct Connection option in setup 2.878, it appears
>>> that setup only downloads the setup.xz file from one of the three sites
>>> I have selected.
>>
>> Interesting.  I don't see anything in the changes since 2.877 that
>> would have caused this.  Can you show a setup.log.full?
>
> Attached, although I didn't see any difference in the log between using
> the new Direct Connection option and the old one.

Thanks.

Yes, we could do a better job of logging that a fetch was satisfied from 
the cache, but I hope that the fact that the setup.xz URLs are logged as 
fetched and used means if that they weren't actually fetched, they came 
from a cache.

>> Something which might give this impression is that setup will now
>> cache setup.xz, and only download it if it has changed.
>>
>> I had no idea anyone actually used this feature of setup.  I'm
>> somewhat sceptical it's making things faster or more reliable, rather
>> than just downloading the same file 3 times... :)
>
> I think you nailed it. I tried again this morning and the first time I
> ran setup, it did contact our local site, but it did not on subsequent
> runs (even to see if the file has changed). I then modified our web

This would be very bad, if true.  Are you sure that we aren't contacting 
the server at all? (such requests might not be logged in some 
configurations)

> server to add a "Cache-Control: no-cache" header and cleared the Windows
> cache. After that, setup retrieved the setup.xz file from our server on
> every run.
>
> For background, we host our own partial repository of locally built
> packages, so the setup.xz our server provides is not complete. It would
> be incorrect for setup to assume that the setup.xz downloaded from one
> site is identical to the ones available from the other selected sites.

Ignore what I wrote there.  Somehow "site" became "mirror" in my mind. 
Of course, using multiple package repositories should work as you expect.


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: setup 2.878 only processes one site

[David Rothenberger]

Jon Turney wrote:
> On 17/05/2017 18:06, David Rothenberger wrote:
>> Jon Turney wrote:
>>> On 17/05/17 00:14, David Rothenberger wrote:
>>>> When I use the new Direct Connection option in setup 2.878, it appears
>>>> that setup only downloads the setup.xz file from one of the three sites
>>>> I have selected.
>>>
>>> Something which might give this impression is that setup will now
>>> cache setup.xz, and only download it if it has changed.
>>
>> I think you nailed it. I tried again this morning and the first time I
>> ran setup, it did contact our local site, but it did not on subsequent
>> runs (even to see if the file has changed). I then modified our web
> 
> This would be very bad, if true.  Are you sure that we aren't contacting 
> the server at all? (such requests might not be logged in some 
> configurations)

Yes, I'm sure. I verified our web server logs requests with the 
If-Modified-Since header. Also, I used wireshark to confirm there was no 
request at all to our web server.

-- 
David Rothenberger  ----  daveroth@acm.org

Dirksen's Three Laws of Politics:
         (1) Get elected.
         (2) Get re-elected.
         (3) Don't get mad, get even.
                 -- Sen. Everett Dirksen

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: setup 2.878 only processes one site

[Jon Turney]

On 17/05/2017 21:45, David Rothenberger wrote:
> Jon Turney wrote:
>> On 17/05/2017 18:06, David Rothenberger wrote:
>>> Jon Turney wrote:
>>>> On 17/05/17 00:14, David Rothenberger wrote:
>>>>> When I use the new Direct Connection option in setup 2.878, it appears
>>>>> that setup only downloads the setup.xz file from one of the three
>>>>> sites
>>>>> I have selected.
>>>>
>>>> Something which might give this impression is that setup will now
>>>> cache setup.xz, and only download it if it has changed.
>>>
>>> I think you nailed it. I tried again this morning and the first time I
>>> ran setup, it did contact our local site, but it did not on subsequent
>>> runs (even to see if the file has changed). I then modified our web
>>
>> This would be very bad, if true.  Are you sure that we aren't
>> contacting the server at all? (such requests might not be logged in
>> some configurations)
>
> Yes, I'm sure. I verified our web server logs requests with the
> If-Modified-Since header. Also, I used wireshark to confirm there was no
> request at all to our web server.

Hmm... In my testing I saw requests with If-Modified-Since: and 
If-None-Match: headers being sent and a 304 Not Modified response being 
sent.

I added INTERNET_FLAG_RESYNCHRONIZE, which looks like it might be needed 
to do the right thing if the server doesn't send an ETag header, and 
made another build.

https://cygwin.com/setup/setup-2.878-4-g582a35.x86.exe
https://cygwin.com/setup/setup-2.878-4-g582a35.x86_64.exe

Perhaps you could try that and see if it improves things?


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: setup 2.878 only processes one site

[David Rothenberger]

Jon Turney wrote:
> On 17/05/2017 21:45, David Rothenberger wrote:
>> Jon Turney wrote:
>>> On 17/05/2017 18:06, David Rothenberger wrote:
>>>> Jon Turney wrote:
>>>>> On 17/05/17 00:14, David Rothenberger wrote:
>>>>>> When I use the new Direct Connection option in setup 2.878, it 
>>>>>> appears
>>>>>> that setup only downloads the setup.xz file from one of the three
>>>>>> sites
>>>>>> I have selected.
>>>>>
>>>>> Something which might give this impression is that setup will now
>>>>> cache setup.xz, and only download it if it has changed.
>>>>
>>>> I think you nailed it. I tried again this morning and the first time I
>>>> ran setup, it did contact our local site, but it did not on subsequent
>>>> runs (even to see if the file has changed). I then modified our web
>>>
>>> This would be very bad, if true.  Are you sure that we aren't
>>> contacting the server at all? (such requests might not be logged in
>>> some configurations)
>>
>> Yes, I'm sure. I verified our web server logs requests with the
>> If-Modified-Since header. Also, I used wireshark to confirm there was no
>> request at all to our web server.
> 
> Hmm... In my testing I saw requests with If-Modified-Since: and 
> If-None-Match: headers being sent and a 304 Not Modified response being 
> sent.
> 
> I added INTERNET_FLAG_RESYNCHRONIZE, which looks like it might be needed 
> to do the right thing if the server doesn't send an ETag header, and 
> made another build.

My server does send the ETag header. I'm using Apache 2.4.

> https://cygwin.com/setup/setup-2.878-4-g582a35.x86.exe
> https://cygwin.com/setup/setup-2.878-4-g582a35.x86_64.exe
> 
> Perhaps you could try that and see if it improves things?

The problem is indeed fixed with these builds. Thanks!

-- 
David Rothenberger  ----  daveroth@acm.org

Justice, n.:
         A decision in your favor.

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: setup 2.878 only processes one site

[Jon Turney]

On 18/05/2017 18:04, David Rothenberger wrote:
> Jon Turney wrote:
>> On 17/05/2017 21:45, David Rothenberger wrote:
>>> Jon Turney wrote:
>>>> On 17/05/2017 18:06, David Rothenberger wrote:
>>>>> Jon Turney wrote:
>>>>>> On 17/05/17 00:14, David Rothenberger wrote:
>>>>>>> When I use the new Direct Connection option in setup 2.878, it
>>>>>>> appears
>>>>>>> that setup only downloads the setup.xz file from one of the three
>>>>>>> sites
>>>>>>> I have selected.
>>>>>>
>>>>>> Something which might give this impression is that setup will now
>>>>>> cache setup.xz, and only download it if it has changed.
>>>>>
>>>>> I think you nailed it. I tried again this morning and the first time I
>>>>> ran setup, it did contact our local site, but it did not on subsequent
>>>>> runs (even to see if the file has changed). I then modified our web
>>>>
>>>> This would be very bad, if true.  Are you sure that we aren't
>>>> contacting the server at all? (such requests might not be logged in
>>>> some configurations)
>>>
>>> Yes, I'm sure. I verified our web server logs requests with the
>>> If-Modified-Since header. Also, I used wireshark to confirm there was no
>>> request at all to our web server.
>>
>> Hmm... In my testing I saw requests with If-Modified-Since: and
>> If-None-Match: headers being sent and a 304 Not Modified response
>> being sent.
>>
>> I added INTERNET_FLAG_RESYNCHRONIZE, which looks like it might be
>> needed to do the right thing if the server doesn't send an ETag
>> header, and made another build.
>
> My server does send the ETag header. I'm using Apache 2.4.

Well, I still don't know what's going on, then. :)

fwiw, I did some testing with an apache configured with 'FileETag None' 
and saw the the behaviour you described (doesn't even contact the 
server), so this flag is definitely needed.

>> https://cygwin.com/setup/setup-2.878-4-g582a35.x86.exe
>> https://cygwin.com/setup/setup-2.878-4-g582a35.x86_64.exe
>>
>> Perhaps you could try that and see if it improves things?
>
> The problem is indeed fixed with these builds. Thanks!

Great.  Thanks for testing.  I'll do a 2.879 with those changes.

And thank you for drawing attention to this problem.  I looks like it's 
probably existed since forever in ""Use Internet Explorer Proxy 
Settings" mode.


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple