Cygwin Setup - Choose Best Mirror Automatically for Automated / Silent Installs

Cygwin Setup - Choose Best Mirror Automatically for Automated / Silent Installs

[OwN-3m-All]

Hi All,

Is there a way to have the Cygwin setup utility pick a valid mirror
automatically from the list it downloads if the setup is run with the
following options:

setup-x86_64.exe --quiet-mode --root "C:\dir" --local-package-dir
"C:\dir\cygTemp" --packages "apache2"

If I run the setup utility with the parameters above, the user is
prompted to select a mirror.  This would be fine in itself if after
doing so the setup utility continued to operate in the quiet mode and
downloaded the selected packages (including dependencies for them)
automatically.  Instead, what happens is that the user selects a
mirror, and then the dialog is shown to confirm the packages even
though the --quiet-mode parameter was passed in.  So basically, the
way it works right now is not very consistent.  It should either
prompt a user to pick a mirror and then install packages quietly
without a confirmation dialog showing the packages that are going to
be installed, or a mirror should be automatically picked (select the
first one from the list the setup utility downloads), and the setup
should then continue quietly.

I'm trying to automate this as best as possible, and the only way to
do so right now is to specify the mirror the setup utility uses by
using something similar to:

setup-x86_64.exe --site "http://cygwin.mirror.constant.com/"
--quiet-mode --root "C:\dir" --local-package-dir "C:\dir\cygTemp"
--packages "apache2"

The problem with this is that it appears that mirror won't always work
with each version of Windows (at least I think that's the case based
on some strange behavior I'm seeing).

Is there a way to have the setup automatically pick a site from the
list of mirrors it downloads to use?

~
Thanks
OwN

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Cygwin Setup - Choose Best Mirror Automatically for Automated / Silent Installs

[Andrey Repin]

Greetings, OwN-3m-All!

> I'm trying to automate this as best as possible, and the only way to
> do so right now is to specify the mirror the setup utility uses by
> using something similar to:

> setup-x86_64.exe --site "http://cygwin.mirror.constant.com/"
> --quiet-mode --root "C:\dir" --local-package-dir "C:\dir\cygTemp"
> --packages "apache2"

> The problem with this is that it appears that mirror won't always work
> with each version of Windows (at least I think that's the case based
> on some strange behavior I'm seeing).

That can't be true. Simply because of how it works.

> Is there a way to have the setup automatically pick a site from the
> list of mirrors it downloads to use?

Just specify the mirror by hands.
If it doesn't work for some reason, find that reason and eliminate it.


-- 
With best regards,
Andrey Repin
Friday, November 18, 2016 06:14:06

Sorry for my terrible english...


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Cygwin Setup - Choose Best Mirror Automatically for Automated / Silent Installs

[OwN-3m-All]

> Just specify the mirror by hands.
> If it doesn't work for some reason, find that reason and eliminate it.

That's really not a good solution.  I don't have control over the
mirrors or would know why one isn't working.  I think a new
command-line argument should be added that makes Cygwin setup pick one
of the mirrors it knows is good from the list it downloads.

A command-line argument is the way to go, but a text file with a list
of known good mirrors that could be downloaded and parsed by a script
would also be of great help.  Cygwin already checks the status of its
mirros as far as I know.

And for some reason, using the mirror of
"http://cygwin.mirror.constant.com/" on Windows Server 2008 sometimes
causes it to randomly crash complaining about a runtime c++ error
without any good information output into the setup log.  Like I said,
when I run the setup manually, the list of mirrors it pulls does not
contain "http://cygwin.mirror.constant.com/".

I'd much rather rely on Cygwin's list than one hard coded site value
for a mirror which may or may not be up-to-date.

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Cygwin Setup - Choose Best Mirror Automatically for Automated / Silent Installs

[Brian Inglis]

On 2016-11-17 21:06, OwN-3m-All wrote:
>> Just specify the mirror by hands.
>> If it doesn't work for some reason, find that reason and eliminate it.
>
> That's really not a good solution.  I don't have control over the
> mirrors or would know why one isn't working.  I think a new
> command-line argument should be added that makes Cygwin setup pick one
> of the mirrors it knows is good from the list it downloads.
>
> A command-line argument is the way to go, but a text file with a list
> of known good mirrors that could be downloaded and parsed by a script
> would also be of great help.  Cygwin already checks the status of its
> mirros as far as I know.
>
> And for some reason, using the mirror of
> "http://cygwin.mirror.constant.com/" on Windows Server 2008 sometimes
> causes it to randomly crash complaining about a runtime c++ error
> without any good information output into the setup log.  Like I said,
> when I run the setup manually, the list of mirrors it pulls does not
> contain "http://cygwin.mirror.constant.com/".
>
> I'd much rather rely on Cygwin's list than one hard coded site value
> for a mirror which may or may not be up-to-date.

Setup gets its list from: https://cygwin.com/mirrors.lst

https://sourceware.org/git/?p=cygwin-apps%2Fsetup.git&a=search&h=HEAD&st=grep&s=MIRROR
Use the Source Luke!

-- 
Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Cygwin Setup - Choose Best Mirror Automatically for Automated / Silent Installs

[OwN-3m-All]

> Setup gets its list from: https://cygwin.com/mirrors.lst

Thanks.

I'm guessing the installer only shows a few of them though instead of
all of them?

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Cygwin Setup - Choose Best Mirror Automatically for Automated / Silent Installs

[OwN-3m-All]

See, this happens with the mirror I've provided in the setup (only
sometimes... sometimes it works):

https://s17.postimg.org/6xuvfaljz/setup_issue.png

The setup log only contains this:

Starting cygwin install, version 2.876
User has backup/restore rights
io_stream_cygfile: fopen(/etc/setup/setup.rc) failed 2 No such file or directory
Current Directory: C:\test\cygTemp
Could not open service McShield for query, start and stop. McAfee may
not be installed, or we don't have access.
root: C:\test system
Selected local directory: C:\test\cygTemp
net: Direct

How do I troubleshoot this further, and it is truly caused by the
mirror?  It appears to be because I haven't gotten it to crash yet
running it manually and using a mirror entry from the list it displays
to me.

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Cygwin Setup - Choose Best Mirror Automatically for Automated / Silent Installs

[Andrey Repin]

Greetings, OwN-3m-All!

> See, this happens with the mirror I've provided in the setup (only
> sometimes... sometimes it works):

> https://s17.postimg.org/6xuvfaljz/setup_issue.png

> The setup log only contains this:

> Starting cygwin install, version 2.876
> User has backup/restore rights
> io_stream_cygfile: fopen(/etc/setup/setup.rc) failed 2 No such file or directory
> Current Directory: C:\test\cygTemp
> Could not open service McShield for query, start and stop. McAfee may
> not be installed, or we don't have access.
> root: C:\test system
> Selected local directory: C:\test\cygTemp
> net: Direct

> How do I troubleshoot this further, and it is truly caused by the
> mirror?

It is not caused by the mirror. It is crashing long before it would try to
contact any mirror.

> It appears to be because I haven't gotten it to crash yet
> running it manually and using a mirror entry from the list it displays
> to me.

Again, this has nothing to do with it. It crashes on an attempt to retrieve
the list of mirrors.
If my guess is correct, it just crashes on first attempt to connect to the
internet.
Do you have any questionable measures employed? Like braindead AV/firewall
software?


-- 
With best regards,
Andrey Repin
Friday, November 18, 2016 09:31:17

Sorry for my terrible english...


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Cygwin Setup - Choose Best Mirror Automatically for Automated / Silent Installs

[Brian Inglis]

On 2016-11-17 23:10, OwN-3m-All wrote:
> See, this happens with the mirror I've provided in the setup (only
> sometimes... sometimes it works):
>
> https://s17.postimg.org/6xuvfaljz/setup_issue.png
>
> The setup log only contains this:
>
> Starting cygwin install, version 2.876
> User has backup/restore rights
> io_stream_cygfile: fopen(/etc/setup/setup.rc) failed 2 No such file or directory
> Current Directory: C:\test\cygTemp
> Could not open service McShield for query, start and stop. McAfee may
> not be installed, or we don't have access.
> root: C:\test system
> Selected local directory: C:\test\cygTemp
> net: Direct
>
> How do I troubleshoot this further, and it is truly caused by the
> mirror? It appears to be because I haven't gotten it to crash yet
> running it manually and using a mirror entry from the list it
> displays to me.

Check the permissions on the directory allow SYSTEM and
Administrators and your userid rwx access or full control.
Don't just use ls -l, use getfacl and icacls to be sure.

What Windows version; is it an insider, Education or Enterprise
build?
These all can have customizations that switch retail features
off, add unusual features and capabilities for extra control,
or run services that are unusual on retail systems.
Check particularly for any "security" or anti-* products:
these may be on the BLODA and cause random failures: see
https://cygwin.com/faq/faq.html#faq.using.bloda
The major retail products are just *BAD*!

You can still run setup in a script with stdout and stderr
redirection to your own log in case it produces some useful text
output when it crashes.

Install and try:
https://github.com/PhilipDaniels/dotfiles/blob/master/cygwin_mirror_test.pl
to find the lowest latency network mirror offering the fastest
downloads, but remember that doesn't tell you anything about update
frequency, availability, reliability, stability.

If I find my current mirror unavailable, I wait and hour and retry.
If it's still not working, I rerun the mirror test and switch.
I have to do this about every three months, probably because some
other mirrored distro has just come out with major updates that
everyone has to have right away! ;^>
It's worth while to set a quarterly or monthly reminder to recheck
your mirrors, or schedule a job that does so every few weeks, at
about the time and on the weekday you normally run setup.

-- 
Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Cygwin Setup - Choose Best Mirror Automatically for Automated / Silent Installs

[Achim Gratz]

OwN-3m-All writes:
> See, this happens with the mirror I've provided in the setup (only
> sometimes... sometimes it works):
>
> https://s17.postimg.org/6xuvfaljz/setup_issue.png

Since setup.exe isn't using that runtime library, I suggest you're at
the receiving end of some BLODA.  I've seen things like that before when
we were still using TrendMicro.  It might help to strip the downloaded
setup.exe of the information that it's from the Internet (most simply by
copying to some FAT formatted USB stick and then back, or just remove
the ADS that records that bit of information) or to rename it to
something else but "setup".  You may need to move and start the file
from some other directory as the one it was downloaded to.  There may be
restrictions implemented via group policy that interfere.  Lastly, some
scanners easy up a bit if you ask them to scan the file once before
starting it.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptations for Waldorf Q V3.00R3 and Q+ V3.54R2:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Cygwin Setup - Choose Best Mirror Automatically for Automated / Silent Installs

[OwN-3m-All]

> I'm afraid you will probably have to build setup from source to debug further

I tried that, but it wouldn't compile on my Ubuntu instance.  It
appears it's supposed to be compiled on Windows?  If that's the case,
I have no idea how.  I downloaded and installed mingw64 on my Windows
machine, but I'm not sure how I'm supposed to run the bootstrap.sh
bash file without bash on Windows.

Am I supposed to use Cygwin to compile Cygwin?

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Cygwin Setup - Choose Best Mirror Automatically for Automated / Silent Installs

[Corinna Vinschen]

[-- Attachment #1: Type: text/plain, Size: 654 bytes --]

On Nov 20 12:19, OwN-3m-All wrote:
> > I'm afraid you will probably have to build setup from source to debug further
> 
> I tried that, but it wouldn't compile on my Ubuntu instance.  It
> appears it's supposed to be compiled on Windows?  If that's the case,
> I have no idea how.  I downloaded and installed mingw64 on my Windows
> machine, but I'm not sure how I'm supposed to run the bootstrap.sh
> bash file without bash on Windows.

Use a mingw64 cross toolchain under linux.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Cygwin Setup - Choose Best Mirror Automatically for Automated / Silent Installs

[Andrey Repin]

Greetings, OwN-3m-All!

>> Setup gets its list from: https://cygwin.com/mirrors.lst

> Thanks.

> I'm guessing the installer only shows a few of them though instead of
> all of them?

It shows all of them. It's not like there's millions of them.


-- 
With best regards,
Andrey Repin
Friday, November 18, 2016 09:13:16

Sorry for my terrible english...


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Cygwin Setup - Choose Best Mirror Automatically for Automated / Silent Installs

[Andrey Repin]

Greetings, OwN-3m-All!

>> Just specify the mirror by hands.
>> If it doesn't work for some reason, find that reason and eliminate it.

> That's really not a good solution.  I don't have control over the
> mirrors or would know why one isn't working.

But you have control over the script that working your installation, and you
can choose a mirror that is likely working. I.e. kernel.org mirrors.

> I think a new command-line argument should be added that makes Cygwin setup
> pick one of the mirrors it knows is good from the list it downloads.

> A command-line argument is the way to go, but a text file with a list
> of known good mirrors that could be downloaded and parsed by a script
> would also be of great help.  Cygwin already checks the status of its
> mirros as far as I know.

It knows all of them and all of them are good for it.
If you know better, then it's up to you to tell it that.

> And for some reason, using the mirror of
> "http://cygwin.mirror.constant.com/" on Windows Server 2008 sometimes
> causes it to randomly crash complaining about a runtime c++ error
> without any good information output into the setup log.  Like I said,
> when I run the setup manually, the list of mirrors it pulls does not
> contain "http://cygwin.mirror.constant.com/".

Now, this is a more useful information. But still not quite enough to track
the problem.
If you are familiar with GDB, you can try debugging the execution and see why
it crashes.

> I'd much rather rely on Cygwin's list than one hard coded site value
> for a mirror which may or may not be up-to-date.

So, do that. I see no stopping for it.
As Brian pointed out, the list of mirrors is publicly available.


-- 
With best regards,
Andrey Repin
Friday, November 18, 2016 09:06:05

Sorry for my terrible english...


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Cygwin Setup - Choose Best Mirror Automatically for Automated / Silent Installs

[Achim Gratz]

OwN-3m-All writes:
> That's really not a good solution.  I don't have control over the
> mirrors or would know why one isn't working.

The only way to thoroughly solve that problem is to provide your own
mirror (which by definition you have control over).

> I think a new command-line argument should be added that makes Cygwin
> setup pick one of the mirrors it knows is good from the list it
> downloads.

That really doesn't work without quite a bit of server-side
infrastructure that currently simply isn't in place.  Personally I'd
prefer MirrorBrain, but SHTDI and all that.

> A command-line argument is the way to go, but a text file with a list
> of known good mirrors that could be downloaded and parsed by a script
> would also be of great help.  Cygwin already checks the status of its
> mirros as far as I know.

If you think you know which mirror is currently working (best) for you,
that file can easily be downloaded.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptation for Waldorf Blofeld V1.15B11:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Cygwin Setup - Choose Best Mirror Automatically for Automated / Silent Installs

[OwN-3m-All]

> What Windows version; is it an insider, Education or Enterprise
> build?

I'm testing on as many different operating systems as I can since I
don't know what my users will be running.  So far, I have it crashing
randomly as shown in my screenshot on Windows Server 2008 Enterprise
SP2 x86 and Windows Server 2008 R2 Standard x64.  It works perfectly
on Windows 7 x64 SP1.

I'm not running any antivirus or firewall software as far as I know.
I will disable Windows Defender on the Server OSes and see if that
helps.

I'll try to dig into this some more...

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Cygwin Setup - Choose Best Mirror Automatically for Automated / Silent Installs

[OwN-3m-All]

I'm not running Windows Defender or any software for that matter on my
test instance of Server 2008.

I tried running GDB, but the setup executable doesn't have any
debugging symbols.  Is there a download link somewhere for the latest
compiled cygwin setup that includes these symbols?

GNU gdb (GDB) 7.7.50.20140303-cvs
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This binary was built by Equation Solution <http://www.Equation.com>...
Reading symbols from setup-x86_64.exe...(no debugging symbols found)...done.
(gdb)

I won't be able to debug it further unless I have that... I guess.

The original log I posted is what you get when I redirect the output
to a file with this at the very end of the command " >
Cygwin64_Setup.log"

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Cygwin Setup - Choose Best Mirror Automatically for Automated / Silent Installs

[Brian Inglis]

On 2016-11-18 18:48, OwN-3m-All wrote:
> I'm not running Windows Defender or any software for that matter on
> my test instance of Server 2008.

There will be retail workstation components not installed on Server,
and all bets are off for Enterprise Server: there will be components
not installed and possibly extra components installed for remote
admin, configuration, and updating.
Have you asked the admins or checked on the systems for policy settings
that could cause issues.
Most corporate servers I have used have all internet access blocked
by default, and often components disabled or not installed, so admins
have to add policies or install components to allow each access.
Might want to run a regular cygcheck on the setup exe on each system.

> I tried running GDB, but the setup executable doesn't have any
> debugging symbols. Is there a download link somewhere for the latest
> compiled cygwin setup that includes these symbols?
>
> GNU gdb (GDB) 7.7.50.20140303-cvs
> Copyright (C) 2014 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
> This binary was built by Equation Solution <http://www.Equation.com>...
> Reading symbols from setup-x86_64.exe...(no debugging symbols found)...done.
> (gdb)
>
> I won't be able to debug it further unless I have that... I guess.
>
> The original log I posted is what you get when I redirect the output
> to a file with this at the very end of the command "> Cygwin64_Setup.log"

The original log you posted was what I would expect to see in
/var/log/setup.log{,.full}.
Retry with > Cygwin64_Setup.log 2>&1 to ensure you get stderr and stdout.

-- 
Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: Cygwin Setup - Choose Best Mirror Automatically for Automated / Silent Installs

[Jon Turney]

On 19/11/2016 01:48, OwN-3m-All wrote:
> I'm not running Windows Defender or any software for that matter on my
> test instance of Server 2008.
>
> I tried running GDB, but the setup executable doesn't have any
> debugging symbols.  Is there a download link somewhere for the latest
> compiled cygwin setup that includes these symbols?

Sorry, this isn't available...

> GNU gdb (GDB) 7.7.50.20140303-cvs
> Copyright (C) 2014 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
> This binary was built by Equation Solution <http://www.Equation.com>...
> Reading symbols from setup-x86_64.exe...(no debugging symbols found)...done.
> (gdb)

... and even if it was, this wouldn't work because the distributed setup 
executable is packed with UPX, and gdb doesn't understand what that does 
to the process memory map.

> I won't be able to debug it further unless I have that... I guess.

I'm afraid you will probably have to build setup from source to debug 
further (See [1])

[1] 
https://cygwin.com/git/gitweb.cgi?p=cygwin-apps/setup.git;a=blob;f=README

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple