RE: setup 2.194.2.24: Bug (?) in downloading from internet

RE: setup 2.194.2.24: Bug (?) in downloading from internet

[Robert Collins]

> -----Original Message-----
> From: Sam Edge [mailto:sam_edgeZZZ@hotmail.com] 
> Sent: Sunday, April 28, 2002 8:17 PM

> Every version of setup.exe I've used exhibits this annoying 
> behaviour but them I'm new here.
> 
> I agree with Daniele. I install different subsets of the 
> packages on several machines. I also like to keep a copy of 
> the most up-to-date versions of packages that I don't 
> currently use on any of them, in case I need them where I've 
> no (or poor) access to the 'Net. 

So use a mirroring tool! Setup.exe is -not- designed for this. If you
want a setup.ini parsing mirroring tool, take me up on my offer to help
someone leverage the setup.ini codebase to build a mirroring tool. 

Rob

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Bug reporting:         http://cygwin.com/bugs.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: setup 2.194.2.24: Bug (?) in downloading from internet

[Sam Edge]

Robert Collins wrote in
<FC169E059D1A0442A04C40F86D9BA7600C5F42@itdomain003.itdomain.net.au>
in gmane.os.cygwin on Sun, 28 Apr 2002 20:22:30 +1000:

> > I agree with Daniele. I install different subsets of the 
> > packages on several machines. I also like to keep a copy of 
> > the most up-to-date versions of packages that I don't 
> > currently use on any of them, in case I need them where I've 
> > no (or poor) access to the 'Net. 
> 
> So use a mirroring tool! Setup.exe is -not- designed for this. 

Surely if setup.exe isn't designed with at least a nod to this way of
working why does it have "download from Internet" and "install from
local directory" options, instead of just instisting on connecting to
a server every time?

> If you
> want a setup.ini parsing mirroring tool, take me up on my offer to help
> someone leverage the setup.ini codebase to build a mirroring tool. 

Although I've not looked at it, setup.exe must already have most of
the code required to operate in the way we'd like, surely?

If you tell me the module name(s) for setup.exe and any documentation
about setup.ini and the local and FTP/HTTP server directory structures
I'd certainly be interested to download them from the CVS server and
take a look.

-- 
Sam Edge

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Bug reporting:         http://cygwin.com/bugs.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

RE: setup 2.194.2.24: Bug (?) in downloading from internet

[Gary R. Van Sickle]

> > -----Original Message-----
> > From: Sam Edge [mailto:sam_edgeZZZ@hotmail.com]
> > Sent: Sunday, April 28, 2002 8:17 PM
>
> > Every version of setup.exe I've used exhibits this annoying
> > behaviour but them I'm new here.
> >
> > I agree with Daniele. I install different subsets of the
> > packages on several machines. I also like to keep a copy of
> > the most up-to-date versions of packages that I don't
> > currently use on any of them, in case I need them where I've
> > no (or poor) access to the 'Net.
>
> So use a mirroring tool! Setup.exe is -not- designed for this. If you
> want a setup.ini parsing mirroring tool, take me up on my offer to help
> someone leverage the setup.ini codebase to build a mirroring tool.
>
> Rob
>

What am I missing in all these posts?  Didn't somebody post a while ago that if
you set your "Local Package Directory" to a network share and do "Install From
Internet", you get all the functionality people are wanting here?  Now I admit
I've not tried it myself (though I have plenty of reason to do so and should,
maybe next week), but what more do people want here?

Maybe we should dump the "Download" option entirely.  Seems that all it serves
to do is confuse people that should really be looking at rsync.

--
Gary R. Van Sickle
Brewer.  Patriot.


--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Bug reporting:         http://cygwin.com/bugs.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: setup 2.194.2.24: Bug (?) in downloading from internet

[Sam Edge]

Gary R. Van Sickle wrote in
<NCBBIHCHBLCMLBLOBONKKEOLCNAA.g.r.vansickle@worldnet.att.net>
in gmane.os.cygwin on Sun, 28 Apr 2002 14:12:36 -0500:

> What am I missing in all these posts?  Didn't somebody post a while ago that if
> you set your "Local Package Directory" to a network share and do "Install From
> Internet", you get all the functionality people are wanting here?

No. You can use the same local package directory to install on
multiple machines very nicely. That aspect works fine.

The problem is that when you do "download from Internet" the list
tells you what you have installed in c:\cygwin on the machine where
you're running setup.exe not what you've previously downloaded to the
package directory. Add to this the lack of an option to skip
previously downloaded packages when they're manually selected (or
auto-selected by the dependency management) and it's a problem if
you're using a single local package directory for several different
machines that have different package sets installed.

On a single machine where you download and immediately install only
what'll be used locally it can still be a problem for those on slow,
unreliable or time-limited connections.

For example, many people in the UK use a 56kbps (or more usually
around 45kbps) connection. Many unmetered ISP packages disconnect
every two hours to prevent users hogging dial-in capacity they're not
using.

Downloading even a basic package with all its dependencies can run
over this time limit so you can't use the "install from Internet"
option.

But when you try to use "download from Internet" and then re-start
setup.exe after re-connecting it doesn't show you the packages you
already have. Worse, when you select your package again it
auto-selects all the dependencies and tries to download them all again
and so will never complete because you'll keep hitting the disconnect
timeout.

What you have to do is have an Explorer window open looking at the
package directory and manually go back and de-select all the ones
you've already got after auto-select has added them. This is prone to
error, especially if you've not used clean_setup.pl to flatten the
structure between each download session. It's also annoying because it
should be easy to automate into setup.exe itself.

> Maybe we should dump the "Download" option entirely.  Seems that all it serves
> to do is confuse people that should really be looking at rsync.

Please, no!

Apart from the reason I give above, on my main machine here I always
download first in my normal NT user account so I can continue to work
while it's going on. I then log in as ADMINISTRATOR to install because
I have the Cygwin directory hierarchy protected against write by
normal users. That way I'm running with potentially dangerous
privileges for as short a time as possible.

-- 
Sam Edge

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Bug reporting:         http://cygwin.com/bugs.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: setup 2.194.2.24: Bug (?) in downloading from internet

[Charles Wilson]

<begin rant>
Robert, you are a saint.

I am sick and tired of the whining about setup and downloads.

SETUP IS NOT A MIRRORING TOOL.

In order to make this clear, I am in favor of completely removing the 
download-only option, leaving just install-from-internet and 
install-from-localdir.  This will force the whiners to either shut up 
and use a fscking mirroring tool, or help out:

Robert has already said what the correct solution to all this is: create 
a separate program that leverages the setup.exe codebase, to do 
"multi-source merged mirroring".

This will require:

1) Add command-line options for setup.exe.  Robert has added a helpful 
set of examples.  He even created a whole new project, GetOpt++, to 
support this.  What more can Robert, all by himself, do?

2) Create a new cygmirror.cc as part of the setup module, that uses some 
(but not all) of the other classes in the setup module.  According to 
Robert:

 > The differences between setup and a mirroring tool include:
 > * All the install code isn't needed.
 > * Attention to local packages isn't needed.
 > * Persistent selection of packages irrespective of install status
 >   should be present.
 > * command line driven may be useful?

I'll add one more: reading options (or package lists) from a file.

Now, unless somebody who WANTS this functionality actually steps up to 
the plate and digs in, can we PLEASE stop discussing this issue?

Removing the download-only option will also have the salutory effect of 
forcing the whiners to either:

1) use a REAL mirroring tool, then use setup.exe to
    install-from-local-dir
   1a) heck, you could even use wget; I've posted a script
       to this mailing list that does exactly that.
2) help out with the coding.

Either way, it kills "setup download only mode doesn't do what I want" 
"What's up with the wacky %%%'ified directory names in my local setup 
directory" "Setup sucks" threads that NEVER DIE!.

--Chuck


--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Bug reporting:         http://cygwin.com/bugs.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: setup 2.194.2.24: Bug (?) in downloading from internet

[Sam Edge]

Charles Wilson wrote in <3CCC7290.706@ece.gatech.edu>
in gmane.os.cygwin on Sun, 28 Apr 2002 18:07:12 -0400:

> <begin rant>
> Robert, you are a saint.
> I am sick and tired of the whining about setup and downloads.
> SETUP IS NOT A MIRRORING TOOL.

LOL. No need to get your knickers in a twist! I was merely explaining
the limitations of setup.exe to Gary. ;-)

The problem with time limited slow connections is valid even when
using it purely for local setup on a single machine i.e. when NOT
using it as a mirroring tool. 

A simple checkbox option on the download-only page to skip
pre-downloaded packages even if they're selected in the list would
deal with that and I really don't see how this would compromise its
function in any way, as long as it were off by default.

Unless someone can give a good reason why it /would/ be a bad idea,
I'll look at the code and see if I can write a version that has this.

> In order to make this clear, I am in favor of completely removing the 
> download-only option, leaving just install-from-internet and 
> install-from-localdir.  This will force the whiners to either shut up 
> and use a fscking mirroring tool, or help out:

It will also stop users on slow time-limited connections from being
able to use setup.exe at all unless they first use a conventional
FTP/HTTP mirroring client and set up their own local FTP server to
which to connect setup.exe. They then have to work out which parts of
one of the current mirrors are required for their particular needs and
download those packages. They'll also end up with two copies of
everything on their hard discs.

This is hardly a good way of increasing the Cygwin user/developer
base!

-- 
Sam Edge

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Bug reporting:         http://cygwin.com/bugs.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: setup 2.194.2.24: Bug (?) in downloading from internet

[Christopher Faylor]

On Sun, Apr 28, 2002 at 06:07:12PM -0400, Charles Wilson wrote:
><begin rant>
>Robert, you are a saint.
>
>I am sick and tired of the whining about setup and downloads.
>
>SETUP IS NOT A MIRRORING TOOL.
>
>In order to make this clear, I am in favor of completely removing the 
>download-only option, leaving just install-from-internet and 
>install-from-localdir.  This will force the whiners to either shut up 
>and use a fscking mirroring tool, or help out:

I am really torn on this one.  I fully agree that setup.exe is not a
mirroring tool.  I've vigorously made that assertion on many occasions.

However, I think I may have been responsible for setup's current
behavior of basing what is downloaded on what is installed.  Even if
I was not, though, I think that the explanations about why some people
need this are rather convincing.

If you're on the other end of a slow connection and are apt to be
disconnected every couple of hours, you will *never* be able to do a
complete install since the setup.exe will start from scratch every time
you reconnect.  It doesn't matter if you chose "download from internet" or
not since setup.exe will only consider what's installed when considering
what to download.  That just seems like a bug to me.

If I have this right then even removing download from internet does not
eliminate this problem.

So, I think the right answer to this is the standard one "patches
gratefully accepted".

>Robert has already said what the correct solution to all this is: create 
>a separate program that leverages the setup.exe codebase, to do 
>"multi-source merged mirroring".

I do agree with this philosophy for anything beyond fixing the above, IMO,
glitch.  setup.exe should be a fairly simple tool.  If you find that it
isn't meeting your needs then you should be looking for a better tool.

cgf

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Bug reporting:         http://cygwin.com/bugs.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: setup 2.194.2.24: Bug (?) in downloading from internet

[Sam Edge]

Christopher Faylor wrote in <20020428223000.GB13849@redhat.com>
in gmane.os.cygwin on Sun, 28 Apr 2002 18:30:00 -0400:

> If you're on the other end of a slow connection and are apt to be
> disconnected every couple of hours, you will *never* be able to do a
> complete install since the setup.exe will start from scratch every time
> you reconnect.  It doesn't matter if you chose "download from internet" or
> not since setup.exe will only consider what's installed when considering
> what to download.  That just seems like a bug to me.

Thanks. Even something like the XFree86 "basic" install can take a
long time, let alone trying to install everything in one hit.
Sometimes people forget that there are many still out there on creaky
old dial-up connections.

> If I have this right then even removing download from internet does not
> eliminate this problem.

No, it makes it worse.

> So, I think the right answer to this is the standard one "patches
> gratefully accepted".

I'll look but I'm making no promises! ;-D

-- 
Sam Edge

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Bug reporting:         http://cygwin.com/bugs.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: setup 2.194.2.24: Bug (?) in downloading from internet

[Michael A Chase]

On Sun, 28 Apr 2002 18:30:00 -0400 Christopher Faylor <cgf@redhat.com> wrote:

> On Sun, Apr 28, 2002 at 06:07:12PM -0400, Charles Wilson wrote:

> If you're on the other end of a slow connection and are apt to be
> disconnected every couple of hours, you will *never* be able to do a
> complete install since the setup.exe will start from scratch every time
> you reconnect.  It doesn't matter if you chose "download from internet"
> or
> not since setup.exe will only consider what's installed when considering
> what to download.  That just seems like a bug to me.

That's what the discussion has been about for the last week or so.  After
lots of requests for this functionality, the position eventually hardened
to "it's not a download tool".

> If I have this right then even removing download from internet does not
> eliminate this problem.

That depends on how capable wget or rsync is.  I have not looked into it
but if we do remove the download only function from setup.exe, we (me?)
should probably add something under cygwin.com describing how to mirror
selected packages.

> So, I think the right answer to this is the standard one "patches
> gratefully accepted".

If download handling is going to change, at least Robert will have to buy
into it.  A request for patches can follow that.

-- 
Mac :})
** I normally forward private questions to the appropriate mail list. **
Ask Smarter: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Give a hobbit a fish and he eats fish for a day.
Give a hobbit a ring and he eats fish for an age.



--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Bug reporting:         http://cygwin.com/bugs.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: setup 2.194.2.24: Bug (?) in downloading from internet

[Christopher Faylor]

On Sun, Apr 28, 2002 at 04:04:36PM -0700, Michael A Chase wrote:
>On Sun, 28 Apr 2002 18:30:00 -0400 Christopher Faylor <cgf@redhat.com> wrote:
>
>> On Sun, Apr 28, 2002 at 06:07:12PM -0400, Charles Wilson wrote:
>
>>If you're on the other end of a slow connection and are apt to be
>>disconnected every couple of hours, you will *never* be able to do a
>>complete install since the setup.exe will start from scratch every time
>>you reconnect.  It doesn't matter if you chose "download from internet"
>>or not since setup.exe will only consider what's installed when
>>considering what to download.  That just seems like a bug to me.
>
>That's what the discussion has been about for the last week or so.
>After lots of requests for this functionality, the position eventually
>hardened to "it's not a download tool".

Right.  But it started with my suggesting this behavior was a bug and (I
thought) Robert agreeing.  So, I thought it was time to chime in again.

>> If I have this right then even removing download from internet does not
>> eliminate this problem.
>
>That depends on how capable wget or rsync is.  I have not looked into it
>but if we do remove the download only function from setup.exe, we (me?)
>should probably add something under cygwin.com describing how to mirror
>selected packages.

It's not a question of mirroring.  It's a question of setup.exe being
completely unusable for some people.  We can't say that setup.exe is the
only method we'll support but then tell people to use some other method
when it fails for them as they try to use it in a perfectly acceptable
way.

The bottom line is that if we have to tell someone to use wget or rsync
just to do a normal install, something is wrong.

It seems like this problem would be trivially fixed by not attempting to
redownload something that has already been downloaded.

cgf

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Bug reporting:         http://cygwin.com/bugs.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: setup 2.194.2.24: Bug (?) in downloading from internet

[Cliff Hones]

Christopher Faylor <cgf@redhat.com> wrote:
> The bottom line is that if we have to tell someone to use wget or rsync
> just to do a normal install, something is wrong.
> 
> It seems like this problem would be trivially fixed by not attempting to
> redownload something that has already been downloaded.

Hooray.  Seems lots of people have been trying to say this.  A week
ago when it was, for a moment, acknowledged as a bug, I downloaded
the source with ideas of trying to fix it.  But then the mood shifted
back to it being a feature, so I lost interest.

Now Sam Edge is taking the same path.  Good luck.

What would happen if someone did produce a version of setup identical
to the current, except for the removal of redownload?  Would it
be adopted?  One gets an unfortunate undercurrent from this list that
if it wasn't approved by the right people it might be suppressed
(and left to circulate on the black market - possibly the worst
scenario).

-- Cliff



--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Bug reporting:         http://cygwin.com/bugs.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: setup 2.194.2.24: Bug (?) in downloading from internet

[Max Bowsher]

Robert Collins <robert.collins@itdomain.com.au> wrote:
>> -----Original Message-----
>> From: Sam Edge [mailto:sam_edgeZZZ@hotmail.com]
>> Sent: Sunday, April 28, 2002 8:17 PM
>> Every version of setup.exe I've used exhibits this annoying
>> behaviour but them I'm new here.
>>
>> I agree with Daniele. I install different subsets of the
>> packages on several machines. I also like to keep a copy of
>> the most up-to-date versions of packages that I don't
>> currently use on any of them, in case I need them where I've
>> no (or poor) access to the 'Net.
>
> So use a mirroring tool! Setup.exe is -not- designed for this. If you
> want a setup.ini parsing mirroring tool, take me up on my offer to help
> someone leverage the setup.ini codebase to build a mirroring tool.
>
> Rob

But setup.exe has operated in the correct way in the past! Why write a new tool,
when fixing a bug in the current one would do all that needs to be done? NB: I'm
not saying 'go fix that now'. When (if? ;-) I find time to understand the setup
code, I will do this, if no one has beaten me to it.

Max.


--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Bug reporting:         http://cygwin.com/bugs.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: setup 2.194.2.24: Bug (?) in downloading from internet

[Robert Collins]

===
----- Original Message -----
From: "Max Bowsher" <maxb@ukf.net>
To: <cygwin@cygwin.com>

> But setup.exe has operated in the correct way in the past! Why write a
new tool,
> when fixing a bug in the current one would do all that needs to be
done?

Because the current tool is more complex than it should be, and a new
tool can be accurately target. The current tool can then be simplified
somewhat (i.e. download only could go away if we had a (non cygwin
linked) mirroring tool. Or a variation of this.

> NB: I'm
> not saying 'go fix that now'. When (if? ;-) I find time to understand
the setup
> code, I will do this, if no one has beaten me to it.

Thanks.

Rob


--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Bug reporting:         http://cygwin.com/bugs.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: setup 2.194.2.24: Bug (?) in downloading from internet

[Robert Collins]

===
----- Original Message -----
From: "Cliff Hones" <cliff@aonix.co.uk>
To: <cygwin@cygwin.com>

>
> Now Sam Edge is taking the same path.  Good luck.
>
> What would happen if someone did produce a version of setup identical
> to the current, except for the removal of redownload?  Would it
> be adopted?

As it stands, I don't know. I've asked everyone to write in and say
whether they use the redownload capability.

> One gets an unfortunate undercurrent from this list that
> if it wasn't approved by the right people it might be suppressed
> (and left to circulate on the black market - possibly the worst
> scenario).

Heh, this is open source, there is no black market :}. Realistically,
anyone can use anything they want to install, but 'you' the users have a
realistic expectation that 'we' the developers will provide a tool, and
support that tool. Conversely we expect that anyone wanting assistance
with install will use our provided tool.

Rob


--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Bug reporting:         http://cygwin.com/bugs.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Using a real mirroring tool...

[Charles Wilson]

Poor archive.progeny.com ...

First, rsync places a heavy load on the server, so please do not 
EVERYBODY run out and try this immediately.  I suggest, for initial 
rsyncs (when you need to download EVERYTHING), that folks follow this 
rotation:

Surname Begins With       Initial rysnc
ABC                          May 1
DEF                          May 2
GHI                          May 3
JKL                          May 4
MNO                          May 5
PQR                          May 6
STU                          May 7
VWX                          May 8
YZ                           May 9

Incremental rsync's don't place as much of a load on the server, so it 
won't matter as much after that...

Here is a script that will mirror the cygwin/release tree and 
setup.exe/setup.ini (you'll need to change LOCALDIR).  Also, you'll need 
to remove --dry-run from EXTRAOPTS.  If anybody can identify other 
mirrors that provide rsync: support, please post the appropriate MIRROR= 
line as a reply to this message.

---------------------
#!/bin/sh
MIRROR=rsync://archive.progeny.com/cygwin
LOCALDIR=/annex1/private/software/cygwin/
OPTS="-rlptgoDzu"
EXTRAOPTS="--delete -vv --dry-run"
RSYNC=/usr/bin/rsync

${RSYNC} ${OPTS} ${EXTRAOPTS} ${MIRROR}/release ${LOCALDIR}
${RSYNC} ${OPTS} ${EXTRAOPTS} ${MIRROR}/setup.* ${LOCALDIR}
---------------------

--Chuck


--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Bug reporting:         http://cygwin.com/bugs.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: Using a real mirroring tool...

[Charles Wilson]

Actually, this is a better set of options: we don't need to preserve 
owner, groups, or devices, and the modify-window helps with the limited 
timestamp granularity of the FAT filesystem.  Remember to fix LOCALDIR 
and remove --dry-run.

---------------------
#!/bin/sh
MIRROR=rsync://archive.progeny.com/cygwin
LOCALDIR=/annex1/private/software/cygwin/
OPTS="-rlptzu --modify-window=2"
EXTRAOPTS="--delete -vv --dry-run"
RSYNC=/usr/bin/rsync

${RSYNC} ${OPTS} ${EXTRAOPTS} ${MIRROR}/release ${LOCALDIR}
${RSYNC} ${OPTS} ${EXTRAOPTS} ${MIRROR}/setup.* ${LOCALDIR}
---------------------

--Chuck

Charles Wilson wrote:

> Poor archive.progeny.com ...
> 
> First, rsync places a heavy load on the server, so please do not 
> EVERYBODY run out and try this immediately.  I suggest, for initial 
> rsyncs (when you need to download EVERYTHING), that folks follow this 
> rotation:
> 
> Surname Begins With       Initial rysnc
> ABC                          May 1
> DEF                          May 2
> GHI                          May 3
> JKL                          May 4
> MNO                          May 5
> PQR                          May 6
> STU                          May 7
> VWX                          May 8
> YZ                           May 9
> 
> Incremental rsync's don't place as much of a load on the server, so it 
> won't matter as much after that...
> 
> Here is a script that will mirror the cygwin/release tree and 
> setup.exe/setup.ini (you'll need to change LOCALDIR).  Also, you'll need 
> to remove --dry-run from EXTRAOPTS.  If anybody can identify other 
> mirrors that provide rsync: support, please post the appropriate MIRROR= 
> line as a reply to this message.
> 
> ---------------------
> #!/bin/sh
> MIRROR=rsync://archive.progeny.com/cygwin
> LOCALDIR=/annex1/private/software/cygwin/
> OPTS="-rlptgoDzu"
> EXTRAOPTS="--delete -vv --dry-run"
> RSYNC=/usr/bin/rsync
> 
> ${RSYNC} ${OPTS} ${EXTRAOPTS} ${MIRROR}/release ${LOCALDIR}
> ${RSYNC} ${OPTS} ${EXTRAOPTS} ${MIRROR}/setup.* ${LOCALDIR}
> ---------------------
> 
> --Chuck
> 



--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Bug reporting:         http://cygwin.com/bugs.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: Using a real mirroring tool...

[Charles Wilson]

A few notes:

Using rsync does not help the following problem:  I have four favorite 
mirrors, plus a "private" setup-compatible website from my friend Bob. 
setup.exe will show me a merged view of the contents of all of those 
sites, insuring that I get the latest version of a package wherever it 
is found.  By merging in the "private" site, setup enables me to get my 
friend Bob's latest version of "King of Frobozz, Tyrant" game using the 
same setup tool/view, along with the latest apache package or binutils 
package from the "official" cygwin mirrors.

rsync can't merge.

I could rsync each of the four mirrors down to my machine into separate 
directories (assuming the mirrors all had anonymous rsync servers), and 
rsync Bob's private website (assuming his webhost provided rsync -- 
hah!).  But even then, setup can't merge multiple local directories, at 
least not right now.

So, we still need some variant of setup.exe that can merge multiple 
setup-compatible sites, and "mirror" only the latest stuff into a single 
  local "release" tree / setup.ini.

Also, rsync does some stuff we don't really care about: since most of 
the files are compressed, rsync's sophisticated "download changed 
chunks" algorithm is pretty useless.  In compressed files, the first 
byte that is different in the uncompressed file will cause a cascade of 
changes throughout the remainder of the compressed file.  So: 1 byte 
diff in uncompressed file == entire compressed file is different.

OTOH, since we typically (never?) upload a modified version of a given 
file USING THE SAME NAME, we don't really care too much about this. 
"Our" mirroring tool can just use the md5 hashes on the mirror and 
locally to determine the rare instance when a given file has had its 
contents changed -- but then we'd download the ENTIRE file, not "changed 
chunks".  (However, if this ever happens, isn't it a sign that something 
went wrong somewhere?)

Also, there's really no need to use rsync's compression engine, since 
most of the data to be transferred has already been compressed.

Finally, the local "setup.ini" will never(?) match the setup.ini at any 
one of the mirrors or private sites; it will be a merger of them.  So, 
rsync can't do that.

So, if someone cares enough to do the work, there is still a need for 
the "additional tool" that uses the setup.exe codebase to do 
"cygwin-specific distributed mirroring".  See my earlier rant.

But, the rsync method should enable folks with slow connections to limp 
along for a while...without trying to turn setup.exe ITSELF into a 
mirror tool.

--Chuck


--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Bug reporting:         http://cygwin.com/bugs.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: Using a real mirroring tool...

[Charles Wilson]

Charles Wilson wrote:

> So, if someone cares enough to do the work, there is still a need for 
> the "additional tool" that uses the setup.exe codebase to do 
> "cygwin-specific distributed mirroring".  See my earlier rant.


Oh, I left out the most obvious reason why rsync won't solve all of our 
problems: we have lots of ftp and http mirrors, but very few (ONE?) 
rsync mirrors.  That really kills the "distributed" nature of the mirror 
system; that poor guy in Thailand with the slow connection would be 
rsync'ing across the Pacific from Indiana...that can't be good.

A tool that leverages the setup.exe codebase would inherit the ability 
to use the existing ftp and http mirror sites.

--Chuck
</stating the obvious>

P.S. you might want to add --exclude *-src.tar.* to the option string, 
if you don't want the -src tarballs.


--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Bug reporting:         http://cygwin.com/bugs.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: Using a real mirroring tool...

[Sam Edge]

Charles Wilson wrote in <3CCDA73E.5000503@ece.gatech.edu>
in gmane.os.cygwin on Mon, 29 Apr 2002 16:04:14 -0400:

> P.S. you might want to add --exclude *-src.tar.* to the option string, 
> if you don't want the -src tarballs.

As I understand it you can also exclude the setup.hint and md5.sum
files if you're just populating a local package directory. They're
very small but every little helps and it reduces the clutter and can
save a few sectors on the local disc.

Or am I missing something? ("Again" do I hear from the chorus? Kindly
leave the auditorium, sir!) ;-)

-- 
Sam Edge

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Bug reporting:         http://cygwin.com/bugs.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: Using a real mirroring tool...

[Volker Quetschke]

[-- Attachment #1: Type: text/plain, Size: 615 bytes --]

Hi!

Charles Wilson wrote:

> A few notes:
> 
> ...
> 
> rsync can't merge.


Yes, but Michael A. Chases clean_setup.pl can! I allowed myself to add a 
new option to clean_setup.pl, it exports the list of missing files to a 
file. Then you can use wget to get all the missing files.

I attached the modified clean_setup.pl plus a shell scrip to get the 
missing files from a mirror. You have to modify the getmissing.sh to 
choose your mirror and your target download directory. The directory and 
both files must exist the same directory as your setup.exe. Just a quick 
hack, I guarantee nothing!

Bye
    Volker

[-- Attachment #2: getmissing.sh --]
[-- Type: text/plain, Size: 757 bytes --]

#!/bin/bash

# Mirror script for cygwin
# Uses Michael A. Chase "cleanup_setup.pl"
# in a modified version (V1.0303)
# and wget

# getmissing.sh and cleanup_setup.pl must reside in the same directory
# as setup.exe

# Files to download
filelist="missing.lst"
# Choose your mirror with directory of setup.ini
mirror="ftp://ftp.inf.tu-dresden.de/software/windows/cygwin32/"
# Cut directories                   1        2       3
cutdir=3
# Name your download directory
mirrordir="_mymirror"

wget --mirror --no-host-directories --cut-dirs=$cutdir --passive-ftp -P $mirrordir ${mirror}setup.ini
./clean_setup.pl -Arch -writelist
wget --mirror --no-host-directories --cut-dirs=$cutdir --passive-ftp -P $mirrordir -i $filelist -B $mirror
./clean_setup.pl -Setup

[-- Attachment #3: clean_setup.pl --]
[-- Type: text/plain, Size: 13836 bytes --]

#! /usr/bin/perl -w

require 5.005;

# Delete old files from Cygwin setup.exe archive subdirectories

# Name of file:  cleanup_setup.pl
#
# Type of file:  Perl script
#
# Author:        Michael A. Chase
#
# Purpose:       Clean out setup.exe archive subdirectories.

# Syntax:        See $HelpText Below
#--------------------------  MODIFICATION HISTORY -----------------------------
$VERSION = '1.0303';
# 020430 V. Quetschke   Add -writelist
#                       Writes list of missing files to missing.lst.
# 020423 M. Chase       Add -source, -install, -H options.
#                       Removed directory names from Missing Files list.
# $VERSION = '1.0301';
# 020417 M. Chase       Stop lc()ing file names during setup.ini parsing.
# $VERSION = '1.0300';
# 020416 M. Chase       Ignore setup.ini while collecting archive file names.
#                       Remove directories in reverse length order.
# $VERSION = '1.0202';
# 020416 M. Chase       Properly ignore directories in obsolete file search.
# $VERSION = '1.0201';
# 020404 M. Chase       Ignore files found more than once in move loop.
#                       Protect against attempts to move files from different
#                          trees over each other in the base tree.
# $VERSION = '1.0200';
# 020326 M. Chase       Properly handle directory case.
#                       Optionally move archives to base directory tree.
# $VERSION = '1.0100';
# 020212 M. Chase       Handle multiple setup.ini files and file sources.
# $VERSION = '1.0000';
# 010422 M. Chase       First draft.

use FindBin qw( $RealBin $RealScript );
use File::Basename qw( &basename &dirname &fileparse );
use File::Find;
use File::Spec;
use File::Path qw( &mkpath );
use File::Copy qw( &copy &move );
use Getopt::Long;

use strict;
use integer;
use vars qw( $VERSION );

# Initialize options
my $bArch    = 0;
my $bMove    = 0;
my $bSetup   = 0;
my $bCurr    = 1;
my $bPrev    = 1;
my $bTest    = 1;
my $bExp     = 1;
my $bInstall = 1;
my $bSource  = 0;
my $bWrite   = 0;
my @sHide    = ();
my @sIgnore  = ();
my $sDir0    = File::Spec -> rel2abs( "." );

# Syntax description
sub usage {
   my ( $sOpt, $sVal, @sMsg ) = @_;

   my $sHelpText = <<END_HELP_TEXT;
Cleanup Cygwin setup.exe package cache directories
syntax: $RealScript [opt]
Opt: ($VERSION)
   -[no]Move    = [Don't] Move archive files to base directory tree ($bMove)
   -[no]Arch    = [Don't] Delete obsolete archives and directories ($bArch)
   -[no]Setup   = [Don't] Delete obsolete setup.ini files ($bSetup), forces -Move
   -[no]curr    = [Don't] Report missing [curr] files ($bCurr)
   -[no]prev    = [Don't] Report missing [prev] files ($bPrev)
   -[no]test    = [Don't] Report missing [test] files ($bTest)
   -[no]exp     = [Don't] Report missing [exp]  files ($bExp)
   -[no]install = [Don't] Report missing install archives ($bInstall)
   -[no]source  = [Don't] Report missing source archives ($bSource)
   -writelist   = Write missing files report (missing.lst) to disk ($bWrite)
   -base dir    = Archive cache base directory ($sDir0)
   -I mask      = Ignore files and directories that match mask, multiple allowed
   -H mask      = Ignore packages that match mask, multiple allowed
      mask is a filename wildcard mask, not a regular expression
Arg:
END_HELP_TEXT
# Balance quotes in here document # ' # "

   my $nRet = 'help' eq $sOpt ? 0 : 0 + $sVal;
   select STDERR if $nRet;
   foreach ( @sMsg, $sHelpText ) { s/\s+$//; print "$_\n"; }
   exit $nRet;
}

# Parse command line
Getopt::Long::config( qw( no_ignore_case no_auto_abbrev require_order ) );
GetOptions(
   'Arch!'       => \$bArch,
   'Move!'       => \$bMove,
   'Setup!'      => \$bSetup,
   'curr!'       => \$bCurr,
   'prev!'       => \$bPrev,
   'test!'       => \$bTest,
   'exp!'        => \$bExp,
   'install!'    => \$bInstall,
   'source!'     => \$bSource,
   'writelist!'  => \$bWrite,
   'base=s'      => \$sDir0,
   'Hide|H=s@'   => \@sHide,
   'Ignore|I=s@' => \@sIgnore,
   'help|h' => \&usage ) or usage( 'die', 1 );
$bMove ||= $bSetup; # Don't remove setup files unless also moving archves
chdir $sDir0 or usage( 'die', 1, "Can't change directory to $sDir0, $!\n" );

# Provide default values for unset options and parameters

# Report arguments
$sDir0 = File::Spec -> rel2abs( "." );
$sDir0 =~ s,\\,/,g;
my $sOpt = '';
$sOpt .= "\nMoving archives to base directory tree" if $bMove;
$sOpt .= "\nDeleting unused files and empty directores" if $bArch;
$sOpt .= "\nDeleting obsolete setup.ini files" if $bSetup;
$sOpt .= "\nIgnoring files and directories: " . join ' ',  map { "'$_'" }
   @sIgnore if @sIgnore;
$sOpt .= "\nIgnoring packages: "   . join ' ',  map { "'$_'" } @sHide if @sHide;
print <<HERE;
Base Directory: $sDir0$sOpt
HERE

# Build file or directory name matcher
#    Adapted from Recipe 6.10 in Perl Cookbook
sub rfMatch {
   my ( $bDefault, $op, @sMask ) = @_; # @sMask must be a lexical array
   return sub { return $bDefault; } if 3 > @_;
   my $sExpr = join " $op\n",
      map {
         # Convert file expansion mask to regular expression
         $sMask[$_] =~ s/\./\\./g;
         $sMask[$_] =~ s/\?/.?/g;
         $sMask[$_] =~ s/\*/.*/g;
         "m:^\$sMask[$_]\$:oi";
         } 0 .. $#sMask;
   my $rfMatch = eval "sub { local \$_ = shift;\nreturn $sExpr; };";
   die $@ if $@;
   return $rfMatch;
}
local *bHide   = rfMatch( 0, '||', @sHide );
local *bIgnore = rfMatch( 0, '||', @sIgnore );

# Find and parse setup.ini files, collect other filenames at the same time
my ( %aSetup, $sRel, $sName, %sTarBall, %bDir );
my $wanted = sub {
   # Skip ., .., and files and directories in ignore list
   if ( '.' eq $_ || '..' eq $_ ) {
      $File::Find::prune = 1 if '..' eq $_;
      return;
   }
   if ( bIgnore( $_ ) ) {
      $File::Find::prune = 1 if -d $_; # Prune if a directory
      return;
   }

   # Handle directory or file
   $sRel = sRel( $File::Find::dir, $sDir0 );
   if ( -d $_ ) { $bDir{sRel( $File::Find::name, $sDir0 )} = 1; }
      # Remember directory name for possible removal
   elsif ( "setup.ini" eq $_ ) {
      # Parse setup.ini
      my @aSetup;

      # Get list of files to leave alone, includes subdirectory path
      # setup-timestamp: 1012849221
      # install: latest/bash/bash-2.05-1.tar.gz 576828
      # source:  latest/bash/bash-2.05-1-src.tar.gz 1792319
      my ( $bHide, $sGroup, $sType, $sFile, $sSize, $sVol, $sSubDir, $sName );
      my $sSetup = $File::Find::name;
      open( SETUP, $sSetup ) or usage( 'die', 1, "Can't open $sSetup, $!" );
      while ( <SETUP> ) {
         ( $sType, $sFile, $sSize )  = split /\s+/, $_;
         if    ( 'setup-timestamp:' eq $sType ) {
            $aSetup[0]         = $sFile; # Actually timestamp
         }
         elsif ( 'install:'         eq $sType ) {
            next if $bHide;
            ( $sVol, $sSubDir, $sName ) = File::Spec -> splitpath( $sFile );
            $sSubDir                    = File::Spec -> canonpath( $sSubDir );
            $aSetup[1]{$sName}          = [ $sGroup, $sSize, $sSubDir ];
         }
         elsif ( 'source:'          eq $sType ) {
            next if $bHide;
            ( $sVol, $sSubDir, $sName ) = File::Spec -> splitpath( $sFile );
            $sSubDir                    = File::Spec -> canonpath( $sSubDir );
            $aSetup[2]{$sName}          = [ $sGroup, $sSize, $sSubDir ];
         }
         elsif ( '[' eq substr( $_, 0, 1 ) )    { s/\s+$//; $sGroup = $_; } # ]
         elsif ( s/^@\s+// ) {
            s/\s+$//;
            $bHide  = bHide( $_ );
            $sGroup = '[curr]';
         }
      }
      close SETUP;
      usage( 'die', 1, "Nothing found in $sSetup" )
         if 3 != @aSetup || ! $aSetup[0];
      $aSetup{$sRel} = \@aSetup;
   }
   elsif ( ".tar.bz2" eq substr( $_, -8 ) || ".tar.gz" eq substr( $_, -7 ) ) {
      # Save name of archive file
      $sTarBall{$_}{File::Spec -> canonpath( $sRel )} = -s $_;
   }
   # Currently ignoring other types of files
};
find( $wanted, $sDir0 );
usage( "die", 1, "No setup.ini files found" ) if ! %aSetup;

# Pick newest setup.ini
my ( $tNewest, $sNewest, @sOldSetup ) = ( 0, "" );
foreach ( sort keys %aSetup ) {
   if ( $tNewest < $aSetup{$_}[0] ) {
      $tNewest = $aSetup{$_}[0];
      $sNewest = $_;
   }
}
print "Newest setup.ini files: ", scalar gmtime( $tNewest ), " GMT\n";
foreach ( sort keys %aSetup ) {
   print "   ", sUnPercent( sRel( $_ ) ), "\n" if $tNewest == $aSetup{$_}[0];
}

# Report or remove obsolete setup.ini files
foreach ( sort keys %aSetup ) {
   push @sOldSetup, $_ if $tNewest > $aSetup{$_}[0];
}
print "Obsolete setup.ini files:\n" if @sOldSetup;
foreach ( @sOldSetup ) {
   print "   ", scalar gmtime( $aSetup{$_}[0] ), " GMT ",
     sUnPercent( sRel( $_ ) ), "\n";
   unlink "$_/setup.ini" or print "      *** Can't remove, $!" if $bSetup;
}

# Check found files against those listed in latest setup.ini
my ( @sDir, $sDir, $sFile, @sDup, %sMove, @sRemove, @sWrongSize );
my %aInstall = ( %{$aSetup{$sNewest}[1]} );
my %aSource  = ( %{$aSetup{$sNewest}[2]} );
foreach $sName ( sort keys %sTarBall ) {
   @sDir  = sort keys %{$sTarBall{$sName}};
   $sDir  = $sDir[0];
   $sFile = File::Spec -> catfile( $sDir, $sName );
   if    ( 1 < @sDir ) {
      # More than one copy of a file was found
      push @sDup, "$sName: " . join( ", ", map { sUnPercent( $_ ) } @sDir );
   }
   elsif ( exists $aInstall{$sName} ) {
      if    ( $sDir eq $aInstall{$sName}[2] ) {} # Already in right place
      elsif ( $sTarBall{$sName}{$sDir} != $aInstall{$sName}[1] ) {
         # Wrong size
         push @sWrongSize,
            "$sFile: $sTarBall{$sName}{$sDir} != $aInstall{$sName}[1] in " .
            sUnPercent( $sDir );
      }
      else {
         $sMove{$sFile} = File::Spec -> catfile( $aInstall{$sName}[2], $sName );
      }
   }
   elsif ( exists $aSource{$sName} ) {
      if    ( $sDir eq $aSource{$sName}[2] ) {} # Already in right place
      elsif ( $sTarBall{$sName}{$sDir} != $aSource{$sName}[1] ) {
         # Wrong size
         push @sWrongSize,
            "$sFile: $sTarBall{$sName}{$sDir} != $aSource{$sName}[1] in " .
            sUnPercent( $sDir );
      }
      else {
         $sMove{$sFile} = File::Spec -> catfile( $aSource{$sName}[2], $sName );
      }
   }
   else {
      # File not in setup.ini
      push @sRemove, $sFile;
   }
}

# Check for missing files
my @sMissing = ();
my @sMissingList = ();

if ( $bInstall ) {
  foreach ( keys %aInstall ) {
	if ( ! exists $sTarBall{$_} &&
	  ( $bCurr || '[curr]' ne $aInstall{$_}[0] ) &&
	  ( $bPrev || '[prev]' ne $aInstall{$_}[0] ) &&
      ( $bTest || '[test]' ne $aInstall{$_}[0] ) &&
	  ( $bExp  || '[exp]'  ne $aInstall{$_}[0] ) ) {
		push @sMissing, "$aInstall{$_}[0] $aInstall{$_}[2]/$_" ;
		  # Alternative: $aInstall{$_}[0] . " " .
		  #              File::Spec -> catfile( $aInstall{$_}[2],  $_ ) }
		push @sMissingList, "$aInstall{$_}[2]/$_" ;
	  }
  }
}
if ( $bSource ) {
  foreach ( keys %aSource ) {
	if ( ! exists $sTarBall{$_} &&
	  ( $bCurr || '[curr]' ne $aSource{$_}[0] ) &&
      ( $bPrev || '[prev]' ne $aSource{$_}[0] ) &&
      ( $bTest || '[test]' ne $aSource{$_}[0] ) &&
      ( $bExp  || '[exp]'  ne $aSource{$_}[0] ) ) {
		push @sMissing, "$aSource{$_}[0] $_" ;
		  # Alternative: $aSource{$_}[0] . " " .
		  #              File::Spec -> catfile( $aSource{$_}[2],  $_ ) }
		push @sMissingList, "$aSource{$_}[2]/$_" ;
	  }
  }
}
@sMissing = sort @sMissing;

# Complain about errors
print join( "\n   ", "\nDuplicate Files",  @sDup ),       "\n" if @sDup;
print join( "\n   ", "\nWrong Size Files", @sWrongSize ), "\n" if @sWrongSize;
print join( "\n   ", "\nMissing Files:",   @sMissing ),   "\n" if @sMissing;

# Report list of missing files to disk
if ( $bWrite ) {
  @sMissingList = sort @sMissingList;
  open(REPORT, '>', 'missing.lst');
  print( REPORT join( "\n", @sMissingList ),"\n" ) if @sMissingList;
  close(REPORT);
}

# Move queued files to base directory tree
if ( %sMove ) {
   print $bMove ? "\n" : "\nNot ", "Moving files to base directory tree\n";
   my ( $sFrom, $sTo );
   foreach $sFrom ( sort keys %sMove ) {
      $sTo  = $sMove{$sFrom};
      $sDir = dirname( $sTo );
      print sUnPercent( "   $sFrom -> $sDir\n" );
      if ( -e $sTo ) { print "      *** Target already exists\n"; }
      elsif ( $bMove ) {
         mkpath( $sDir ) if ! -d $sDir;
         move( $sFrom, $sTo ) or print "      *** Can't move, $!\n";
      }
   }
}

# Remove files not listed in setup.ini
if ( @sRemove ) {
   print $bArch ? "\n" : "\nNot ", "Removing files not in setup.ini\n";
   foreach $sFile ( @sRemove ) {
      print sUnPercent( "   $sFile\n" );
      if ( $bArch ) {
         unlink( $sFile ) or print "      *** Can't remove, $!\n";
      }
   }
}

# Remove empty directories
my ( @g, $bFound );
foreach $sRel ( sort { length $b <=> length $a || $a cmp $b } keys %bDir ) {
   @g = glob( File::Spec -> catfile( $sRel, "*" ) );
   if ( ! @g ) {
      print $bArch ? "\n" : "\nNot ", "Removing Empty Directories\n"
         if ! $bFound++;
      print sUnPercent( "   $sRel\n" );
      if ( $bArch ) {
         rmdir $sRel or print "      *** Can't rmdir, $!\n";
      }
   }
}

# Copy latest setup.ini to base directory
if ( "." ne $sNewest ) {
   print sUnPercent( "\nCopying setup.ini from $sNewest to $sDir0\n" );
   copy( "$sNewest/setup.ini", "$sDir0/setup.ini" )
      or print "   *** Can't copy setup.ini, $!\n";
}

exit 0;

# Produce relative file or directory
sub sRel {
   my ( $sAbs, $sBase ) = @_;
   $sBase = "." if ! defined $sBase || ! length $sBase;
   my $sRel = File::Spec -> abs2rel( $sAbs, $sBase );
   $sRel =~ s,\\,/,g;
   $sRel =~ s/^\w://;
   $sRel =  "." if ! length $sRel;
   return $sRel;
}

# Convert %xx to characters
sub sUnPercent {
   local ( $_ ) = @_;

   s/\%([0-9a-f]{2})/chr(hex($1))/gie;
   return $_;
}


[-- Attachment #4: Type: text/plain, Size: 214 bytes --]

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Bug reporting:         http://cygwin.com/bugs.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: Using a real mirroring tool...

[Volker Quetschke]

[-- Attachment #1: Type: text/plain, Size: 753 bytes --]

Hi!

> Charles Wilson wrote:
> 
>> A few notes:
>> ...
>> rsync can't merge.
> 
> Yes, but Michael A. Chases clean_setup.pl can! I allowed myself to add a 
> new option to clean_setup.pl, it exports the list of missing files to a 
> file. Then you can use wget to get all the missing files.
> 
> I attached the modified clean_setup.pl plus a shell scrip to get the 
> missing files from a mirror. You have to modify the getmissing.sh to 
> choose your mirror and your target download directory. The directory and 
> both files must exist the same directory as your setup.exe. Just a quick 
> hack, I guarantee nothing!


Use the attached getmissing.sh instead, some options for wget were not 
correct, they left directory listing files.

Bye
    Volker

[-- Attachment #2: getmissing.sh --]
[-- Type: text/plain, Size: 878 bytes --]

#!/bin/bash

# Mirror script for cygwin
# Uses Michael A. Chase "cleanup_setup.pl"
# in a modified version (V1.0303)
# and wget

# getmissing.sh and cleanup_setup.pl must reside in the same directory
# as setup.exe

# Files to download
filelist="missing.lst"
# Choose your mirror with directory of setup.ini
#mirror="ftp://ftp.inf.tu-dresden.de/software/windows/cygwin32/"
# Cut directories                   1        2       3
mirror="ftp://mirrors.rcn.net/mirrors/sources.redhat.com/cygwin/"
# Cut directories             1       2                  3
cutdir=3
# Name your download directory
mirrordir="_mymirror"

wget -r -N --no-host-directories --cut-dirs=$cutdir --passive-ftp -P $mirrordir ${mirror}setup.ini
./clean_setup.pl -writelist
wget -r -N --no-host-directories --cut-dirs=$cutdir --passive-ftp -P $mirrordir -i $filelist -B $mirror
./clean_setup.pl -Arch -Setup


[-- Attachment #3: Type: text/plain, Size: 214 bytes --]

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Bug reporting:         http://cygwin.com/bugs.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: Using a real mirroring tool...

[Michael A Chase]

On Tue, 30 Apr 2002 16:00:52 +0200 Volker Quetschke <quetschke@scytek.de> wrote:

> I attached the modified clean_setup.pl plus a shell scrip to get the 
> missing files from a mirror. You have to modify the getmissing.sh to 
> choose your mirror and your target download directory. The directory and 
> both files must exist the same directory as your setup.exe. Just a quick 
> hack, I guarantee nothing!

You started with a little bit old version of the script.  I have uploaded a
new version 1.0401 which uses a '-misslist file' option to write the list
to the specified file.

The new version also allows you to exclude packages that you are not
interested in with the '-H mask' option.  This will allow your wget with
the list more selective.

Some of the other options have been changed as well, so be sure to run with
'-h' before using the script.

-- 
Mac :})
** I normally forward private questions to the appropriate mail list. **
Ask Smarter: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Give a hobbit a fish and he eats fish for a day.
Give a hobbit a ring and he eats fish for an age.



--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Bug reporting:         http://cygwin.com/bugs.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: Using a real mirroring tool...

[Earnie Boyd]

[-- Attachment #1: Type: text/plain, Size: 138 bytes --]

You might want to check out http://mirrors.rcn.net/pub/mirror/index.html
for a mirroring tool that uses perl for ftp/http access.

Earnie.

[-- Attachment #2: cygwin.49112 --]
[-- Type: message/rfc822, Size: 1209 bytes --]

From: Charles Wilson <cwilson@ece.gatech.edu>
To: Charles Wilson <cwilson@ece.gatech.edu>
Cc: cygwin@cygwin.com
Subject: Re: Using a real mirroring tool...
Date: Mon, 29 Apr 2002 16:04:14 -0400
Message-ID: <3CCDA73E.5000503@ece.gatech.edu>

Charles Wilson wrote:

> So, if someone cares enough to do the work, there is still a need for 
> the "additional tool" that uses the setup.exe codebase to do 
> "cygwin-specific distributed mirroring".  See my earlier rant.


Oh, I left out the most obvious reason why rsync won't solve all of our 
problems: we have lots of ftp and http mirrors, but very few (ONE?) 
rsync mirrors.  That really kills the "distributed" nature of the mirror 
system; that poor guy in Thailand with the slow connection would be 
rsync'ing across the Pacific from Indiana...that can't be good.

A tool that leverages the setup.exe codebase would inherit the ability 
to use the existing ftp and http mirror sites.

--Chuck
</stating the obvious>

P.S. you might want to add --exclude *-src.tar.* to the option string, 
if you don't want the -src tarballs.



[-- Attachment #3: Type: text/plain, Size: 214 bytes --]

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Bug reporting:         http://cygwin.com/bugs.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

RE: Using a real mirroring tool...

[Robert Collins]

> -----Original Message-----
> From: Sam Edge [mailto:sam_edgeZZZ@hotmail.com] 
> Sent: Tuesday, April 30, 2002 10:18 PM

> Or am I missing something? ("Again" do I hear from the 
> chorus? Kindly leave the auditorium, sir!) ;-)

As you may know, you can use the md5sum files to verify what was
downloaded via script, and until the md5sums are in setup.ini (coming
soon) the only place to get them is the mirror sites.

Rob

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Bug reporting:         http://cygwin.com/bugs.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/