* RE: setup 2.194.2.24: Bug (?) in downloading from internet @ 2002-04-28 4:47 Robert Collins 2002-04-28 8:08 ` Sam Edge ` (2 more replies) 0 siblings, 3 replies; 24+ messages in thread From: Robert Collins @ 2002-04-28 4:47 UTC (permalink / raw) To: Sam Edge, cygwin > -----Original Message----- > From: Sam Edge [mailto:sam_edgeZZZ@hotmail.com] > Sent: Sunday, April 28, 2002 8:17 PM > Every version of setup.exe I've used exhibits this annoying > behaviour but them I'm new here. > > I agree with Daniele. I install different subsets of the > packages on several machines. I also like to keep a copy of > the most up-to-date versions of packages that I don't > currently use on any of them, in case I need them where I've > no (or poor) access to the 'Net. So use a mirroring tool! Setup.exe is -not- designed for this. If you want a setup.ini parsing mirroring tool, take me up on my offer to help someone leverage the setup.ini codebase to build a mirroring tool. Rob -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Bug reporting: http://cygwin.com/bugs.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/ ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: setup 2.194.2.24: Bug (?) in downloading from internet 2002-04-28 4:47 setup 2.194.2.24: Bug (?) in downloading from internet Robert Collins @ 2002-04-28 8:08 ` Sam Edge 2002-04-28 15:04 ` Gary R. Van Sickle 2002-04-29 4:07 ` setup 2.194.2.24: Bug (?) in downloading from internet Max Bowsher 2 siblings, 0 replies; 24+ messages in thread From: Sam Edge @ 2002-04-28 8:08 UTC (permalink / raw) To: cygwin Robert Collins wrote in <FC169E059D1A0442A04C40F86D9BA7600C5F42@itdomain003.itdomain.net.au> in gmane.os.cygwin on Sun, 28 Apr 2002 20:22:30 +1000: > > I agree with Daniele. I install different subsets of the > > packages on several machines. I also like to keep a copy of > > the most up-to-date versions of packages that I don't > > currently use on any of them, in case I need them where I've > > no (or poor) access to the 'Net. > > So use a mirroring tool! Setup.exe is -not- designed for this. Surely if setup.exe isn't designed with at least a nod to this way of working why does it have "download from Internet" and "install from local directory" options, instead of just instisting on connecting to a server every time? > If you > want a setup.ini parsing mirroring tool, take me up on my offer to help > someone leverage the setup.ini codebase to build a mirroring tool. Although I've not looked at it, setup.exe must already have most of the code required to operate in the way we'd like, surely? If you tell me the module name(s) for setup.exe and any documentation about setup.ini and the local and FTP/HTTP server directory structures I'd certainly be interested to download them from the CVS server and take a look. -- Sam Edge -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Bug reporting: http://cygwin.com/bugs.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/ ^ permalink raw reply [flat|nested] 24+ messages in thread
* RE: setup 2.194.2.24: Bug (?) in downloading from internet 2002-04-28 4:47 setup 2.194.2.24: Bug (?) in downloading from internet Robert Collins 2002-04-28 8:08 ` Sam Edge @ 2002-04-28 15:04 ` Gary R. Van Sickle 2002-04-28 15:09 ` Sam Edge 2002-04-29 4:07 ` setup 2.194.2.24: Bug (?) in downloading from internet Max Bowsher 2 siblings, 1 reply; 24+ messages in thread From: Gary R. Van Sickle @ 2002-04-28 15:04 UTC (permalink / raw) To: cygwin > > -----Original Message----- > > From: Sam Edge [mailto:sam_edgeZZZ@hotmail.com] > > Sent: Sunday, April 28, 2002 8:17 PM > > > Every version of setup.exe I've used exhibits this annoying > > behaviour but them I'm new here. > > > > I agree with Daniele. I install different subsets of the > > packages on several machines. I also like to keep a copy of > > the most up-to-date versions of packages that I don't > > currently use on any of them, in case I need them where I've > > no (or poor) access to the 'Net. > > So use a mirroring tool! Setup.exe is -not- designed for this. If you > want a setup.ini parsing mirroring tool, take me up on my offer to help > someone leverage the setup.ini codebase to build a mirroring tool. > > Rob > What am I missing in all these posts? Didn't somebody post a while ago that if you set your "Local Package Directory" to a network share and do "Install From Internet", you get all the functionality people are wanting here? Now I admit I've not tried it myself (though I have plenty of reason to do so and should, maybe next week), but what more do people want here? Maybe we should dump the "Download" option entirely. Seems that all it serves to do is confuse people that should really be looking at rsync. -- Gary R. Van Sickle Brewer. Patriot. -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Bug reporting: http://cygwin.com/bugs.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/ ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: setup 2.194.2.24: Bug (?) in downloading from internet 2002-04-28 15:04 ` Gary R. Van Sickle @ 2002-04-28 15:09 ` Sam Edge 2002-04-28 15:30 ` Charles Wilson 0 siblings, 1 reply; 24+ messages in thread From: Sam Edge @ 2002-04-28 15:09 UTC (permalink / raw) To: cygwin Gary R. Van Sickle wrote in <NCBBIHCHBLCMLBLOBONKKEOLCNAA.g.r.vansickle@worldnet.att.net> in gmane.os.cygwin on Sun, 28 Apr 2002 14:12:36 -0500: > What am I missing in all these posts? Didn't somebody post a while ago that if > you set your "Local Package Directory" to a network share and do "Install From > Internet", you get all the functionality people are wanting here? No. You can use the same local package directory to install on multiple machines very nicely. That aspect works fine. The problem is that when you do "download from Internet" the list tells you what you have installed in c:\cygwin on the machine where you're running setup.exe not what you've previously downloaded to the package directory. Add to this the lack of an option to skip previously downloaded packages when they're manually selected (or auto-selected by the dependency management) and it's a problem if you're using a single local package directory for several different machines that have different package sets installed. On a single machine where you download and immediately install only what'll be used locally it can still be a problem for those on slow, unreliable or time-limited connections. For example, many people in the UK use a 56kbps (or more usually around 45kbps) connection. Many unmetered ISP packages disconnect every two hours to prevent users hogging dial-in capacity they're not using. Downloading even a basic package with all its dependencies can run over this time limit so you can't use the "install from Internet" option. But when you try to use "download from Internet" and then re-start setup.exe after re-connecting it doesn't show you the packages you already have. Worse, when you select your package again it auto-selects all the dependencies and tries to download them all again and so will never complete because you'll keep hitting the disconnect timeout. What you have to do is have an Explorer window open looking at the package directory and manually go back and de-select all the ones you've already got after auto-select has added them. This is prone to error, especially if you've not used clean_setup.pl to flatten the structure between each download session. It's also annoying because it should be easy to automate into setup.exe itself. > Maybe we should dump the "Download" option entirely. Seems that all it serves > to do is confuse people that should really be looking at rsync. Please, no! Apart from the reason I give above, on my main machine here I always download first in my normal NT user account so I can continue to work while it's going on. I then log in as ADMINISTRATOR to install because I have the Cygwin directory hierarchy protected against write by normal users. That way I'm running with potentially dangerous privileges for as short a time as possible. -- Sam Edge -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Bug reporting: http://cygwin.com/bugs.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/ ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: setup 2.194.2.24: Bug (?) in downloading from internet 2002-04-28 15:09 ` Sam Edge @ 2002-04-28 15:30 ` Charles Wilson 2002-04-28 16:05 ` Christopher Faylor ` (2 more replies) 0 siblings, 3 replies; 24+ messages in thread From: Charles Wilson @ 2002-04-28 15:30 UTC (permalink / raw) To: Sam Edge, cygwin <begin rant> Robert, you are a saint. I am sick and tired of the whining about setup and downloads. SETUP IS NOT A MIRRORING TOOL. In order to make this clear, I am in favor of completely removing the download-only option, leaving just install-from-internet and install-from-localdir. This will force the whiners to either shut up and use a fscking mirroring tool, or help out: Robert has already said what the correct solution to all this is: create a separate program that leverages the setup.exe codebase, to do "multi-source merged mirroring". This will require: 1) Add command-line options for setup.exe. Robert has added a helpful set of examples. He even created a whole new project, GetOpt++, to support this. What more can Robert, all by himself, do? 2) Create a new cygmirror.cc as part of the setup module, that uses some (but not all) of the other classes in the setup module. According to Robert: > The differences between setup and a mirroring tool include: > * All the install code isn't needed. > * Attention to local packages isn't needed. > * Persistent selection of packages irrespective of install status > should be present. > * command line driven may be useful? I'll add one more: reading options (or package lists) from a file. Now, unless somebody who WANTS this functionality actually steps up to the plate and digs in, can we PLEASE stop discussing this issue? Removing the download-only option will also have the salutory effect of forcing the whiners to either: 1) use a REAL mirroring tool, then use setup.exe to install-from-local-dir 1a) heck, you could even use wget; I've posted a script to this mailing list that does exactly that. 2) help out with the coding. Either way, it kills "setup download only mode doesn't do what I want" "What's up with the wacky %%%'ified directory names in my local setup directory" "Setup sucks" threads that NEVER DIE!. --Chuck -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Bug reporting: http://cygwin.com/bugs.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/ ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: setup 2.194.2.24: Bug (?) in downloading from internet 2002-04-28 15:30 ` Charles Wilson @ 2002-04-28 16:05 ` Christopher Faylor 2002-04-28 16:09 ` Sam Edge 2002-04-28 19:08 ` Michael A Chase 2002-04-28 16:05 ` Sam Edge 2002-04-29 13:01 ` Using a real mirroring tool Charles Wilson 2 siblings, 2 replies; 24+ messages in thread From: Christopher Faylor @ 2002-04-28 16:05 UTC (permalink / raw) To: cygwin On Sun, Apr 28, 2002 at 06:07:12PM -0400, Charles Wilson wrote: ><begin rant> >Robert, you are a saint. > >I am sick and tired of the whining about setup and downloads. > >SETUP IS NOT A MIRRORING TOOL. > >In order to make this clear, I am in favor of completely removing the >download-only option, leaving just install-from-internet and >install-from-localdir. This will force the whiners to either shut up >and use a fscking mirroring tool, or help out: I am really torn on this one. I fully agree that setup.exe is not a mirroring tool. I've vigorously made that assertion on many occasions. However, I think I may have been responsible for setup's current behavior of basing what is downloaded on what is installed. Even if I was not, though, I think that the explanations about why some people need this are rather convincing. If you're on the other end of a slow connection and are apt to be disconnected every couple of hours, you will *never* be able to do a complete install since the setup.exe will start from scratch every time you reconnect. It doesn't matter if you chose "download from internet" or not since setup.exe will only consider what's installed when considering what to download. That just seems like a bug to me. If I have this right then even removing download from internet does not eliminate this problem. So, I think the right answer to this is the standard one "patches gratefully accepted". >Robert has already said what the correct solution to all this is: create >a separate program that leverages the setup.exe codebase, to do >"multi-source merged mirroring". I do agree with this philosophy for anything beyond fixing the above, IMO, glitch. setup.exe should be a fairly simple tool. If you find that it isn't meeting your needs then you should be looking for a better tool. cgf -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Bug reporting: http://cygwin.com/bugs.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/ ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: setup 2.194.2.24: Bug (?) in downloading from internet 2002-04-28 16:05 ` Christopher Faylor @ 2002-04-28 16:09 ` Sam Edge 2002-04-28 19:08 ` Michael A Chase 1 sibling, 0 replies; 24+ messages in thread From: Sam Edge @ 2002-04-28 16:09 UTC (permalink / raw) To: cygwin Christopher Faylor wrote in <20020428223000.GB13849@redhat.com> in gmane.os.cygwin on Sun, 28 Apr 2002 18:30:00 -0400: > If you're on the other end of a slow connection and are apt to be > disconnected every couple of hours, you will *never* be able to do a > complete install since the setup.exe will start from scratch every time > you reconnect. It doesn't matter if you chose "download from internet" or > not since setup.exe will only consider what's installed when considering > what to download. That just seems like a bug to me. Thanks. Even something like the XFree86 "basic" install can take a long time, let alone trying to install everything in one hit. Sometimes people forget that there are many still out there on creaky old dial-up connections. > If I have this right then even removing download from internet does not > eliminate this problem. No, it makes it worse. > So, I think the right answer to this is the standard one "patches > gratefully accepted". I'll look but I'm making no promises! ;-D -- Sam Edge -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Bug reporting: http://cygwin.com/bugs.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/ ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: setup 2.194.2.24: Bug (?) in downloading from internet 2002-04-28 16:05 ` Christopher Faylor 2002-04-28 16:09 ` Sam Edge @ 2002-04-28 19:08 ` Michael A Chase 2002-04-29 0:06 ` Christopher Faylor 1 sibling, 1 reply; 24+ messages in thread From: Michael A Chase @ 2002-04-28 19:08 UTC (permalink / raw) To: cygwin On Sun, 28 Apr 2002 18:30:00 -0400 Christopher Faylor <cgf@redhat.com> wrote: > On Sun, Apr 28, 2002 at 06:07:12PM -0400, Charles Wilson wrote: > If you're on the other end of a slow connection and are apt to be > disconnected every couple of hours, you will *never* be able to do a > complete install since the setup.exe will start from scratch every time > you reconnect. It doesn't matter if you chose "download from internet" > or > not since setup.exe will only consider what's installed when considering > what to download. That just seems like a bug to me. That's what the discussion has been about for the last week or so. After lots of requests for this functionality, the position eventually hardened to "it's not a download tool". > If I have this right then even removing download from internet does not > eliminate this problem. That depends on how capable wget or rsync is. I have not looked into it but if we do remove the download only function from setup.exe, we (me?) should probably add something under cygwin.com describing how to mirror selected packages. > So, I think the right answer to this is the standard one "patches > gratefully accepted". If download handling is going to change, at least Robert will have to buy into it. A request for patches can follow that. -- Mac :}) ** I normally forward private questions to the appropriate mail list. ** Ask Smarter: http://www.tuxedo.org/~esr/faqs/smart-questions.html Give a hobbit a fish and he eats fish for a day. Give a hobbit a ring and he eats fish for an age. -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Bug reporting: http://cygwin.com/bugs.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/ ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: setup 2.194.2.24: Bug (?) in downloading from internet 2002-04-28 19:08 ` Michael A Chase @ 2002-04-29 0:06 ` Christopher Faylor 2002-04-29 0:57 ` Cliff Hones 0 siblings, 1 reply; 24+ messages in thread From: Christopher Faylor @ 2002-04-29 0:06 UTC (permalink / raw) To: cygwin On Sun, Apr 28, 2002 at 04:04:36PM -0700, Michael A Chase wrote: >On Sun, 28 Apr 2002 18:30:00 -0400 Christopher Faylor <cgf@redhat.com> wrote: > >> On Sun, Apr 28, 2002 at 06:07:12PM -0400, Charles Wilson wrote: > >>If you're on the other end of a slow connection and are apt to be >>disconnected every couple of hours, you will *never* be able to do a >>complete install since the setup.exe will start from scratch every time >>you reconnect. It doesn't matter if you chose "download from internet" >>or not since setup.exe will only consider what's installed when >>considering what to download. That just seems like a bug to me. > >That's what the discussion has been about for the last week or so. >After lots of requests for this functionality, the position eventually >hardened to "it's not a download tool". Right. But it started with my suggesting this behavior was a bug and (I thought) Robert agreeing. So, I thought it was time to chime in again. >> If I have this right then even removing download from internet does not >> eliminate this problem. > >That depends on how capable wget or rsync is. I have not looked into it >but if we do remove the download only function from setup.exe, we (me?) >should probably add something under cygwin.com describing how to mirror >selected packages. It's not a question of mirroring. It's a question of setup.exe being completely unusable for some people. We can't say that setup.exe is the only method we'll support but then tell people to use some other method when it fails for them as they try to use it in a perfectly acceptable way. The bottom line is that if we have to tell someone to use wget or rsync just to do a normal install, something is wrong. It seems like this problem would be trivially fixed by not attempting to redownload something that has already been downloaded. cgf -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Bug reporting: http://cygwin.com/bugs.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/ ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: setup 2.194.2.24: Bug (?) in downloading from internet 2002-04-29 0:06 ` Christopher Faylor @ 2002-04-29 0:57 ` Cliff Hones 2002-04-29 4:58 ` Robert Collins 0 siblings, 1 reply; 24+ messages in thread From: Cliff Hones @ 2002-04-29 0:57 UTC (permalink / raw) To: cygwin Christopher Faylor <cgf@redhat.com> wrote: > The bottom line is that if we have to tell someone to use wget or rsync > just to do a normal install, something is wrong. > > It seems like this problem would be trivially fixed by not attempting to > redownload something that has already been downloaded. Hooray. Seems lots of people have been trying to say this. A week ago when it was, for a moment, acknowledged as a bug, I downloaded the source with ideas of trying to fix it. But then the mood shifted back to it being a feature, so I lost interest. Now Sam Edge is taking the same path. Good luck. What would happen if someone did produce a version of setup identical to the current, except for the removal of redownload? Would it be adopted? One gets an unfortunate undercurrent from this list that if it wasn't approved by the right people it might be suppressed (and left to circulate on the black market - possibly the worst scenario). -- Cliff -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Bug reporting: http://cygwin.com/bugs.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/ ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: setup 2.194.2.24: Bug (?) in downloading from internet 2002-04-29 0:57 ` Cliff Hones @ 2002-04-29 4:58 ` Robert Collins 0 siblings, 0 replies; 24+ messages in thread From: Robert Collins @ 2002-04-29 4:58 UTC (permalink / raw) To: Cliff Hones, cygwin === ----- Original Message ----- From: "Cliff Hones" <cliff@aonix.co.uk> To: <cygwin@cygwin.com> > > Now Sam Edge is taking the same path. Good luck. > > What would happen if someone did produce a version of setup identical > to the current, except for the removal of redownload? Would it > be adopted? As it stands, I don't know. I've asked everyone to write in and say whether they use the redownload capability. > One gets an unfortunate undercurrent from this list that > if it wasn't approved by the right people it might be suppressed > (and left to circulate on the black market - possibly the worst > scenario). Heh, this is open source, there is no black market :}. Realistically, anyone can use anything they want to install, but 'you' the users have a realistic expectation that 'we' the developers will provide a tool, and support that tool. Conversely we expect that anyone wanting assistance with install will use our provided tool. Rob -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Bug reporting: http://cygwin.com/bugs.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/ ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: setup 2.194.2.24: Bug (?) in downloading from internet 2002-04-28 15:30 ` Charles Wilson 2002-04-28 16:05 ` Christopher Faylor @ 2002-04-28 16:05 ` Sam Edge 2002-04-29 13:01 ` Using a real mirroring tool Charles Wilson 2 siblings, 0 replies; 24+ messages in thread From: Sam Edge @ 2002-04-28 16:05 UTC (permalink / raw) To: cygwin Charles Wilson wrote in <3CCC7290.706@ece.gatech.edu> in gmane.os.cygwin on Sun, 28 Apr 2002 18:07:12 -0400: > <begin rant> > Robert, you are a saint. > I am sick and tired of the whining about setup and downloads. > SETUP IS NOT A MIRRORING TOOL. LOL. No need to get your knickers in a twist! I was merely explaining the limitations of setup.exe to Gary. ;-) The problem with time limited slow connections is valid even when using it purely for local setup on a single machine i.e. when NOT using it as a mirroring tool. A simple checkbox option on the download-only page to skip pre-downloaded packages even if they're selected in the list would deal with that and I really don't see how this would compromise its function in any way, as long as it were off by default. Unless someone can give a good reason why it /would/ be a bad idea, I'll look at the code and see if I can write a version that has this. > In order to make this clear, I am in favor of completely removing the > download-only option, leaving just install-from-internet and > install-from-localdir. This will force the whiners to either shut up > and use a fscking mirroring tool, or help out: It will also stop users on slow time-limited connections from being able to use setup.exe at all unless they first use a conventional FTP/HTTP mirroring client and set up their own local FTP server to which to connect setup.exe. They then have to work out which parts of one of the current mirrors are required for their particular needs and download those packages. They'll also end up with two copies of everything on their hard discs. This is hardly a good way of increasing the Cygwin user/developer base! -- Sam Edge -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Bug reporting: http://cygwin.com/bugs.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/ ^ permalink raw reply [flat|nested] 24+ messages in thread
* Using a real mirroring tool... 2002-04-28 15:30 ` Charles Wilson 2002-04-28 16:05 ` Christopher Faylor 2002-04-28 16:05 ` Sam Edge @ 2002-04-29 13:01 ` Charles Wilson 2002-04-29 17:17 ` Charles Wilson 2002-04-29 17:21 ` Charles Wilson 2 siblings, 2 replies; 24+ messages in thread From: Charles Wilson @ 2002-04-29 13:01 UTC (permalink / raw) To: cygwin Poor archive.progeny.com ... First, rsync places a heavy load on the server, so please do not EVERYBODY run out and try this immediately. I suggest, for initial rsyncs (when you need to download EVERYTHING), that folks follow this rotation: Surname Begins With Initial rysnc ABC May 1 DEF May 2 GHI May 3 JKL May 4 MNO May 5 PQR May 6 STU May 7 VWX May 8 YZ May 9 Incremental rsync's don't place as much of a load on the server, so it won't matter as much after that... Here is a script that will mirror the cygwin/release tree and setup.exe/setup.ini (you'll need to change LOCALDIR). Also, you'll need to remove --dry-run from EXTRAOPTS. If anybody can identify other mirrors that provide rsync: support, please post the appropriate MIRROR= line as a reply to this message. --------------------- #!/bin/sh MIRROR=rsync://archive.progeny.com/cygwin LOCALDIR=/annex1/private/software/cygwin/ OPTS="-rlptgoDzu" EXTRAOPTS="--delete -vv --dry-run" RSYNC=/usr/bin/rsync ${RSYNC} ${OPTS} ${EXTRAOPTS} ${MIRROR}/release ${LOCALDIR} ${RSYNC} ${OPTS} ${EXTRAOPTS} ${MIRROR}/setup.* ${LOCALDIR} --------------------- --Chuck -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Bug reporting: http://cygwin.com/bugs.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/ ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Using a real mirroring tool... 2002-04-29 13:01 ` Using a real mirroring tool Charles Wilson @ 2002-04-29 17:17 ` Charles Wilson 2002-04-29 17:21 ` Charles Wilson 1 sibling, 0 replies; 24+ messages in thread From: Charles Wilson @ 2002-04-29 17:17 UTC (permalink / raw) To: Charles Wilson; +Cc: cygwin Actually, this is a better set of options: we don't need to preserve owner, groups, or devices, and the modify-window helps with the limited timestamp granularity of the FAT filesystem. Remember to fix LOCALDIR and remove --dry-run. --------------------- #!/bin/sh MIRROR=rsync://archive.progeny.com/cygwin LOCALDIR=/annex1/private/software/cygwin/ OPTS="-rlptzu --modify-window=2" EXTRAOPTS="--delete -vv --dry-run" RSYNC=/usr/bin/rsync ${RSYNC} ${OPTS} ${EXTRAOPTS} ${MIRROR}/release ${LOCALDIR} ${RSYNC} ${OPTS} ${EXTRAOPTS} ${MIRROR}/setup.* ${LOCALDIR} --------------------- --Chuck Charles Wilson wrote: > Poor archive.progeny.com ... > > First, rsync places a heavy load on the server, so please do not > EVERYBODY run out and try this immediately. I suggest, for initial > rsyncs (when you need to download EVERYTHING), that folks follow this > rotation: > > Surname Begins With Initial rysnc > ABC May 1 > DEF May 2 > GHI May 3 > JKL May 4 > MNO May 5 > PQR May 6 > STU May 7 > VWX May 8 > YZ May 9 > > Incremental rsync's don't place as much of a load on the server, so it > won't matter as much after that... > > Here is a script that will mirror the cygwin/release tree and > setup.exe/setup.ini (you'll need to change LOCALDIR). Also, you'll need > to remove --dry-run from EXTRAOPTS. If anybody can identify other > mirrors that provide rsync: support, please post the appropriate MIRROR= > line as a reply to this message. > > --------------------- > #!/bin/sh > MIRROR=rsync://archive.progeny.com/cygwin > LOCALDIR=/annex1/private/software/cygwin/ > OPTS="-rlptgoDzu" > EXTRAOPTS="--delete -vv --dry-run" > RSYNC=/usr/bin/rsync > > ${RSYNC} ${OPTS} ${EXTRAOPTS} ${MIRROR}/release ${LOCALDIR} > ${RSYNC} ${OPTS} ${EXTRAOPTS} ${MIRROR}/setup.* ${LOCALDIR} > --------------------- > > --Chuck > -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Bug reporting: http://cygwin.com/bugs.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/ ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Using a real mirroring tool... 2002-04-29 13:01 ` Using a real mirroring tool Charles Wilson 2002-04-29 17:17 ` Charles Wilson @ 2002-04-29 17:21 ` Charles Wilson 2002-04-29 17:46 ` Charles Wilson 2002-04-30 10:09 ` Volker Quetschke 1 sibling, 2 replies; 24+ messages in thread From: Charles Wilson @ 2002-04-29 17:21 UTC (permalink / raw) To: Charles Wilson; +Cc: cygwin A few notes: Using rsync does not help the following problem: I have four favorite mirrors, plus a "private" setup-compatible website from my friend Bob. setup.exe will show me a merged view of the contents of all of those sites, insuring that I get the latest version of a package wherever it is found. By merging in the "private" site, setup enables me to get my friend Bob's latest version of "King of Frobozz, Tyrant" game using the same setup tool/view, along with the latest apache package or binutils package from the "official" cygwin mirrors. rsync can't merge. I could rsync each of the four mirrors down to my machine into separate directories (assuming the mirrors all had anonymous rsync servers), and rsync Bob's private website (assuming his webhost provided rsync -- hah!). But even then, setup can't merge multiple local directories, at least not right now. So, we still need some variant of setup.exe that can merge multiple setup-compatible sites, and "mirror" only the latest stuff into a single local "release" tree / setup.ini. Also, rsync does some stuff we don't really care about: since most of the files are compressed, rsync's sophisticated "download changed chunks" algorithm is pretty useless. In compressed files, the first byte that is different in the uncompressed file will cause a cascade of changes throughout the remainder of the compressed file. So: 1 byte diff in uncompressed file == entire compressed file is different. OTOH, since we typically (never?) upload a modified version of a given file USING THE SAME NAME, we don't really care too much about this. "Our" mirroring tool can just use the md5 hashes on the mirror and locally to determine the rare instance when a given file has had its contents changed -- but then we'd download the ENTIRE file, not "changed chunks". (However, if this ever happens, isn't it a sign that something went wrong somewhere?) Also, there's really no need to use rsync's compression engine, since most of the data to be transferred has already been compressed. Finally, the local "setup.ini" will never(?) match the setup.ini at any one of the mirrors or private sites; it will be a merger of them. So, rsync can't do that. So, if someone cares enough to do the work, there is still a need for the "additional tool" that uses the setup.exe codebase to do "cygwin-specific distributed mirroring". See my earlier rant. But, the rsync method should enable folks with slow connections to limp along for a while...without trying to turn setup.exe ITSELF into a mirror tool. --Chuck -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Bug reporting: http://cygwin.com/bugs.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/ ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Using a real mirroring tool... 2002-04-29 17:21 ` Charles Wilson @ 2002-04-29 17:46 ` Charles Wilson 2002-04-30 6:20 ` Sam Edge 2002-04-30 10:09 ` Volker Quetschke 1 sibling, 1 reply; 24+ messages in thread From: Charles Wilson @ 2002-04-29 17:46 UTC (permalink / raw) To: Charles Wilson; +Cc: cygwin Charles Wilson wrote: > So, if someone cares enough to do the work, there is still a need for > the "additional tool" that uses the setup.exe codebase to do > "cygwin-specific distributed mirroring". See my earlier rant. Oh, I left out the most obvious reason why rsync won't solve all of our problems: we have lots of ftp and http mirrors, but very few (ONE?) rsync mirrors. That really kills the "distributed" nature of the mirror system; that poor guy in Thailand with the slow connection would be rsync'ing across the Pacific from Indiana...that can't be good. A tool that leverages the setup.exe codebase would inherit the ability to use the existing ftp and http mirror sites. --Chuck </stating the obvious> P.S. you might want to add --exclude *-src.tar.* to the option string, if you don't want the -src tarballs. -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Bug reporting: http://cygwin.com/bugs.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/ ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Using a real mirroring tool... 2002-04-29 17:46 ` Charles Wilson @ 2002-04-30 6:20 ` Sam Edge 0 siblings, 0 replies; 24+ messages in thread From: Sam Edge @ 2002-04-30 6:20 UTC (permalink / raw) To: cygwin Charles Wilson wrote in <3CCDA73E.5000503@ece.gatech.edu> in gmane.os.cygwin on Mon, 29 Apr 2002 16:04:14 -0400: > P.S. you might want to add --exclude *-src.tar.* to the option string, > if you don't want the -src tarballs. As I understand it you can also exclude the setup.hint and md5.sum files if you're just populating a local package directory. They're very small but every little helps and it reduces the clutter and can save a few sectors on the local disc. Or am I missing something? ("Again" do I hear from the chorus? Kindly leave the auditorium, sir!) ;-) -- Sam Edge -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Bug reporting: http://cygwin.com/bugs.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/ ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Using a real mirroring tool... 2002-04-29 17:21 ` Charles Wilson 2002-04-29 17:46 ` Charles Wilson @ 2002-04-30 10:09 ` Volker Quetschke 2002-04-30 13:43 ` Volker Quetschke 2002-04-30 14:00 ` Michael A Chase 1 sibling, 2 replies; 24+ messages in thread From: Volker Quetschke @ 2002-04-30 10:09 UTC (permalink / raw) To: cygwin [-- Attachment #1: Type: text/plain, Size: 615 bytes --] Hi! Charles Wilson wrote: > A few notes: > > ... > > rsync can't merge. Yes, but Michael A. Chases clean_setup.pl can! I allowed myself to add a new option to clean_setup.pl, it exports the list of missing files to a file. Then you can use wget to get all the missing files. I attached the modified clean_setup.pl plus a shell scrip to get the missing files from a mirror. You have to modify the getmissing.sh to choose your mirror and your target download directory. The directory and both files must exist the same directory as your setup.exe. Just a quick hack, I guarantee nothing! Bye Volker [-- Attachment #2: getmissing.sh --] [-- Type: text/plain, Size: 757 bytes --] #!/bin/bash # Mirror script for cygwin # Uses Michael A. Chase "cleanup_setup.pl" # in a modified version (V1.0303) # and wget # getmissing.sh and cleanup_setup.pl must reside in the same directory # as setup.exe # Files to download filelist="missing.lst" # Choose your mirror with directory of setup.ini mirror="ftp://ftp.inf.tu-dresden.de/software/windows/cygwin32/" # Cut directories 1 2 3 cutdir=3 # Name your download directory mirrordir="_mymirror" wget --mirror --no-host-directories --cut-dirs=$cutdir --passive-ftp -P $mirrordir ${mirror}setup.ini ./clean_setup.pl -Arch -writelist wget --mirror --no-host-directories --cut-dirs=$cutdir --passive-ftp -P $mirrordir -i $filelist -B $mirror ./clean_setup.pl -Setup [-- Attachment #3: clean_setup.pl --] [-- Type: text/plain, Size: 13836 bytes --] #! /usr/bin/perl -w require 5.005; # Delete old files from Cygwin setup.exe archive subdirectories # Name of file: cleanup_setup.pl # # Type of file: Perl script # # Author: Michael A. Chase # # Purpose: Clean out setup.exe archive subdirectories. # Syntax: See $HelpText Below #-------------------------- MODIFICATION HISTORY ----------------------------- $VERSION = '1.0303'; # 020430 V. Quetschke Add -writelist # Writes list of missing files to missing.lst. # 020423 M. Chase Add -source, -install, -H options. # Removed directory names from Missing Files list. # $VERSION = '1.0301'; # 020417 M. Chase Stop lc()ing file names during setup.ini parsing. # $VERSION = '1.0300'; # 020416 M. Chase Ignore setup.ini while collecting archive file names. # Remove directories in reverse length order. # $VERSION = '1.0202'; # 020416 M. Chase Properly ignore directories in obsolete file search. # $VERSION = '1.0201'; # 020404 M. Chase Ignore files found more than once in move loop. # Protect against attempts to move files from different # trees over each other in the base tree. # $VERSION = '1.0200'; # 020326 M. Chase Properly handle directory case. # Optionally move archives to base directory tree. # $VERSION = '1.0100'; # 020212 M. Chase Handle multiple setup.ini files and file sources. # $VERSION = '1.0000'; # 010422 M. Chase First draft. use FindBin qw( $RealBin $RealScript ); use File::Basename qw( &basename &dirname &fileparse ); use File::Find; use File::Spec; use File::Path qw( &mkpath ); use File::Copy qw( © &move ); use Getopt::Long; use strict; use integer; use vars qw( $VERSION ); # Initialize options my $bArch = 0; my $bMove = 0; my $bSetup = 0; my $bCurr = 1; my $bPrev = 1; my $bTest = 1; my $bExp = 1; my $bInstall = 1; my $bSource = 0; my $bWrite = 0; my @sHide = (); my @sIgnore = (); my $sDir0 = File::Spec -> rel2abs( "." ); # Syntax description sub usage { my ( $sOpt, $sVal, @sMsg ) = @_; my $sHelpText = <<END_HELP_TEXT; Cleanup Cygwin setup.exe package cache directories syntax: $RealScript [opt] Opt: ($VERSION) -[no]Move = [Don't] Move archive files to base directory tree ($bMove) -[no]Arch = [Don't] Delete obsolete archives and directories ($bArch) -[no]Setup = [Don't] Delete obsolete setup.ini files ($bSetup), forces -Move -[no]curr = [Don't] Report missing [curr] files ($bCurr) -[no]prev = [Don't] Report missing [prev] files ($bPrev) -[no]test = [Don't] Report missing [test] files ($bTest) -[no]exp = [Don't] Report missing [exp] files ($bExp) -[no]install = [Don't] Report missing install archives ($bInstall) -[no]source = [Don't] Report missing source archives ($bSource) -writelist = Write missing files report (missing.lst) to disk ($bWrite) -base dir = Archive cache base directory ($sDir0) -I mask = Ignore files and directories that match mask, multiple allowed -H mask = Ignore packages that match mask, multiple allowed mask is a filename wildcard mask, not a regular expression Arg: END_HELP_TEXT # Balance quotes in here document # ' # " my $nRet = 'help' eq $sOpt ? 0 : 0 + $sVal; select STDERR if $nRet; foreach ( @sMsg, $sHelpText ) { s/\s+$//; print "$_\n"; } exit $nRet; } # Parse command line Getopt::Long::config( qw( no_ignore_case no_auto_abbrev require_order ) ); GetOptions( 'Arch!' => \$bArch, 'Move!' => \$bMove, 'Setup!' => \$bSetup, 'curr!' => \$bCurr, 'prev!' => \$bPrev, 'test!' => \$bTest, 'exp!' => \$bExp, 'install!' => \$bInstall, 'source!' => \$bSource, 'writelist!' => \$bWrite, 'base=s' => \$sDir0, 'Hide|H=s@' => \@sHide, 'Ignore|I=s@' => \@sIgnore, 'help|h' => \&usage ) or usage( 'die', 1 ); $bMove ||= $bSetup; # Don't remove setup files unless also moving archves chdir $sDir0 or usage( 'die', 1, "Can't change directory to $sDir0, $!\n" ); # Provide default values for unset options and parameters # Report arguments $sDir0 = File::Spec -> rel2abs( "." ); $sDir0 =~ s,\\,/,g; my $sOpt = ''; $sOpt .= "\nMoving archives to base directory tree" if $bMove; $sOpt .= "\nDeleting unused files and empty directores" if $bArch; $sOpt .= "\nDeleting obsolete setup.ini files" if $bSetup; $sOpt .= "\nIgnoring files and directories: " . join ' ', map { "'$_'" } @sIgnore if @sIgnore; $sOpt .= "\nIgnoring packages: " . join ' ', map { "'$_'" } @sHide if @sHide; print <<HERE; Base Directory: $sDir0$sOpt HERE # Build file or directory name matcher # Adapted from Recipe 6.10 in Perl Cookbook sub rfMatch { my ( $bDefault, $op, @sMask ) = @_; # @sMask must be a lexical array return sub { return $bDefault; } if 3 > @_; my $sExpr = join " $op\n", map { # Convert file expansion mask to regular expression $sMask[$_] =~ s/\./\\./g; $sMask[$_] =~ s/\?/.?/g; $sMask[$_] =~ s/\*/.*/g; "m:^\$sMask[$_]\$:oi"; } 0 .. $#sMask; my $rfMatch = eval "sub { local \$_ = shift;\nreturn $sExpr; };"; die $@ if $@; return $rfMatch; } local *bHide = rfMatch( 0, '||', @sHide ); local *bIgnore = rfMatch( 0, '||', @sIgnore ); # Find and parse setup.ini files, collect other filenames at the same time my ( %aSetup, $sRel, $sName, %sTarBall, %bDir ); my $wanted = sub { # Skip ., .., and files and directories in ignore list if ( '.' eq $_ || '..' eq $_ ) { $File::Find::prune = 1 if '..' eq $_; return; } if ( bIgnore( $_ ) ) { $File::Find::prune = 1 if -d $_; # Prune if a directory return; } # Handle directory or file $sRel = sRel( $File::Find::dir, $sDir0 ); if ( -d $_ ) { $bDir{sRel( $File::Find::name, $sDir0 )} = 1; } # Remember directory name for possible removal elsif ( "setup.ini" eq $_ ) { # Parse setup.ini my @aSetup; # Get list of files to leave alone, includes subdirectory path # setup-timestamp: 1012849221 # install: latest/bash/bash-2.05-1.tar.gz 576828 # source: latest/bash/bash-2.05-1-src.tar.gz 1792319 my ( $bHide, $sGroup, $sType, $sFile, $sSize, $sVol, $sSubDir, $sName ); my $sSetup = $File::Find::name; open( SETUP, $sSetup ) or usage( 'die', 1, "Can't open $sSetup, $!" ); while ( <SETUP> ) { ( $sType, $sFile, $sSize ) = split /\s+/, $_; if ( 'setup-timestamp:' eq $sType ) { $aSetup[0] = $sFile; # Actually timestamp } elsif ( 'install:' eq $sType ) { next if $bHide; ( $sVol, $sSubDir, $sName ) = File::Spec -> splitpath( $sFile ); $sSubDir = File::Spec -> canonpath( $sSubDir ); $aSetup[1]{$sName} = [ $sGroup, $sSize, $sSubDir ]; } elsif ( 'source:' eq $sType ) { next if $bHide; ( $sVol, $sSubDir, $sName ) = File::Spec -> splitpath( $sFile ); $sSubDir = File::Spec -> canonpath( $sSubDir ); $aSetup[2]{$sName} = [ $sGroup, $sSize, $sSubDir ]; } elsif ( '[' eq substr( $_, 0, 1 ) ) { s/\s+$//; $sGroup = $_; } # ] elsif ( s/^@\s+// ) { s/\s+$//; $bHide = bHide( $_ ); $sGroup = '[curr]'; } } close SETUP; usage( 'die', 1, "Nothing found in $sSetup" ) if 3 != @aSetup || ! $aSetup[0]; $aSetup{$sRel} = \@aSetup; } elsif ( ".tar.bz2" eq substr( $_, -8 ) || ".tar.gz" eq substr( $_, -7 ) ) { # Save name of archive file $sTarBall{$_}{File::Spec -> canonpath( $sRel )} = -s $_; } # Currently ignoring other types of files }; find( $wanted, $sDir0 ); usage( "die", 1, "No setup.ini files found" ) if ! %aSetup; # Pick newest setup.ini my ( $tNewest, $sNewest, @sOldSetup ) = ( 0, "" ); foreach ( sort keys %aSetup ) { if ( $tNewest < $aSetup{$_}[0] ) { $tNewest = $aSetup{$_}[0]; $sNewest = $_; } } print "Newest setup.ini files: ", scalar gmtime( $tNewest ), " GMT\n"; foreach ( sort keys %aSetup ) { print " ", sUnPercent( sRel( $_ ) ), "\n" if $tNewest == $aSetup{$_}[0]; } # Report or remove obsolete setup.ini files foreach ( sort keys %aSetup ) { push @sOldSetup, $_ if $tNewest > $aSetup{$_}[0]; } print "Obsolete setup.ini files:\n" if @sOldSetup; foreach ( @sOldSetup ) { print " ", scalar gmtime( $aSetup{$_}[0] ), " GMT ", sUnPercent( sRel( $_ ) ), "\n"; unlink "$_/setup.ini" or print " *** Can't remove, $!" if $bSetup; } # Check found files against those listed in latest setup.ini my ( @sDir, $sDir, $sFile, @sDup, %sMove, @sRemove, @sWrongSize ); my %aInstall = ( %{$aSetup{$sNewest}[1]} ); my %aSource = ( %{$aSetup{$sNewest}[2]} ); foreach $sName ( sort keys %sTarBall ) { @sDir = sort keys %{$sTarBall{$sName}}; $sDir = $sDir[0]; $sFile = File::Spec -> catfile( $sDir, $sName ); if ( 1 < @sDir ) { # More than one copy of a file was found push @sDup, "$sName: " . join( ", ", map { sUnPercent( $_ ) } @sDir ); } elsif ( exists $aInstall{$sName} ) { if ( $sDir eq $aInstall{$sName}[2] ) {} # Already in right place elsif ( $sTarBall{$sName}{$sDir} != $aInstall{$sName}[1] ) { # Wrong size push @sWrongSize, "$sFile: $sTarBall{$sName}{$sDir} != $aInstall{$sName}[1] in " . sUnPercent( $sDir ); } else { $sMove{$sFile} = File::Spec -> catfile( $aInstall{$sName}[2], $sName ); } } elsif ( exists $aSource{$sName} ) { if ( $sDir eq $aSource{$sName}[2] ) {} # Already in right place elsif ( $sTarBall{$sName}{$sDir} != $aSource{$sName}[1] ) { # Wrong size push @sWrongSize, "$sFile: $sTarBall{$sName}{$sDir} != $aSource{$sName}[1] in " . sUnPercent( $sDir ); } else { $sMove{$sFile} = File::Spec -> catfile( $aSource{$sName}[2], $sName ); } } else { # File not in setup.ini push @sRemove, $sFile; } } # Check for missing files my @sMissing = (); my @sMissingList = (); if ( $bInstall ) { foreach ( keys %aInstall ) { if ( ! exists $sTarBall{$_} && ( $bCurr || '[curr]' ne $aInstall{$_}[0] ) && ( $bPrev || '[prev]' ne $aInstall{$_}[0] ) && ( $bTest || '[test]' ne $aInstall{$_}[0] ) && ( $bExp || '[exp]' ne $aInstall{$_}[0] ) ) { push @sMissing, "$aInstall{$_}[0] $aInstall{$_}[2]/$_" ; # Alternative: $aInstall{$_}[0] . " " . # File::Spec -> catfile( $aInstall{$_}[2], $_ ) } push @sMissingList, "$aInstall{$_}[2]/$_" ; } } } if ( $bSource ) { foreach ( keys %aSource ) { if ( ! exists $sTarBall{$_} && ( $bCurr || '[curr]' ne $aSource{$_}[0] ) && ( $bPrev || '[prev]' ne $aSource{$_}[0] ) && ( $bTest || '[test]' ne $aSource{$_}[0] ) && ( $bExp || '[exp]' ne $aSource{$_}[0] ) ) { push @sMissing, "$aSource{$_}[0] $_" ; # Alternative: $aSource{$_}[0] . " " . # File::Spec -> catfile( $aSource{$_}[2], $_ ) } push @sMissingList, "$aSource{$_}[2]/$_" ; } } } @sMissing = sort @sMissing; # Complain about errors print join( "\n ", "\nDuplicate Files", @sDup ), "\n" if @sDup; print join( "\n ", "\nWrong Size Files", @sWrongSize ), "\n" if @sWrongSize; print join( "\n ", "\nMissing Files:", @sMissing ), "\n" if @sMissing; # Report list of missing files to disk if ( $bWrite ) { @sMissingList = sort @sMissingList; open(REPORT, '>', 'missing.lst'); print( REPORT join( "\n", @sMissingList ),"\n" ) if @sMissingList; close(REPORT); } # Move queued files to base directory tree if ( %sMove ) { print $bMove ? "\n" : "\nNot ", "Moving files to base directory tree\n"; my ( $sFrom, $sTo ); foreach $sFrom ( sort keys %sMove ) { $sTo = $sMove{$sFrom}; $sDir = dirname( $sTo ); print sUnPercent( " $sFrom -> $sDir\n" ); if ( -e $sTo ) { print " *** Target already exists\n"; } elsif ( $bMove ) { mkpath( $sDir ) if ! -d $sDir; move( $sFrom, $sTo ) or print " *** Can't move, $!\n"; } } } # Remove files not listed in setup.ini if ( @sRemove ) { print $bArch ? "\n" : "\nNot ", "Removing files not in setup.ini\n"; foreach $sFile ( @sRemove ) { print sUnPercent( " $sFile\n" ); if ( $bArch ) { unlink( $sFile ) or print " *** Can't remove, $!\n"; } } } # Remove empty directories my ( @g, $bFound ); foreach $sRel ( sort { length $b <=> length $a || $a cmp $b } keys %bDir ) { @g = glob( File::Spec -> catfile( $sRel, "*" ) ); if ( ! @g ) { print $bArch ? "\n" : "\nNot ", "Removing Empty Directories\n" if ! $bFound++; print sUnPercent( " $sRel\n" ); if ( $bArch ) { rmdir $sRel or print " *** Can't rmdir, $!\n"; } } } # Copy latest setup.ini to base directory if ( "." ne $sNewest ) { print sUnPercent( "\nCopying setup.ini from $sNewest to $sDir0\n" ); copy( "$sNewest/setup.ini", "$sDir0/setup.ini" ) or print " *** Can't copy setup.ini, $!\n"; } exit 0; # Produce relative file or directory sub sRel { my ( $sAbs, $sBase ) = @_; $sBase = "." if ! defined $sBase || ! length $sBase; my $sRel = File::Spec -> abs2rel( $sAbs, $sBase ); $sRel =~ s,\\,/,g; $sRel =~ s/^\w://; $sRel = "." if ! length $sRel; return $sRel; } # Convert %xx to characters sub sUnPercent { local ( $_ ) = @_; s/\%([0-9a-f]{2})/chr(hex($1))/gie; return $_; } [-- Attachment #4: Type: text/plain, Size: 214 bytes --] -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Bug reporting: http://cygwin.com/bugs.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/ ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Using a real mirroring tool... 2002-04-30 10:09 ` Volker Quetschke @ 2002-04-30 13:43 ` Volker Quetschke 2002-04-30 14:00 ` Michael A Chase 1 sibling, 0 replies; 24+ messages in thread From: Volker Quetschke @ 2002-04-30 13:43 UTC (permalink / raw) Cc: cygwin [-- Attachment #1: Type: text/plain, Size: 753 bytes --] Hi! > Charles Wilson wrote: > >> A few notes: >> ... >> rsync can't merge. > > Yes, but Michael A. Chases clean_setup.pl can! I allowed myself to add a > new option to clean_setup.pl, it exports the list of missing files to a > file. Then you can use wget to get all the missing files. > > I attached the modified clean_setup.pl plus a shell scrip to get the > missing files from a mirror. You have to modify the getmissing.sh to > choose your mirror and your target download directory. The directory and > both files must exist the same directory as your setup.exe. Just a quick > hack, I guarantee nothing! Use the attached getmissing.sh instead, some options for wget were not correct, they left directory listing files. Bye Volker [-- Attachment #2: getmissing.sh --] [-- Type: text/plain, Size: 878 bytes --] #!/bin/bash # Mirror script for cygwin # Uses Michael A. Chase "cleanup_setup.pl" # in a modified version (V1.0303) # and wget # getmissing.sh and cleanup_setup.pl must reside in the same directory # as setup.exe # Files to download filelist="missing.lst" # Choose your mirror with directory of setup.ini #mirror="ftp://ftp.inf.tu-dresden.de/software/windows/cygwin32/" # Cut directories 1 2 3 mirror="ftp://mirrors.rcn.net/mirrors/sources.redhat.com/cygwin/" # Cut directories 1 2 3 cutdir=3 # Name your download directory mirrordir="_mymirror" wget -r -N --no-host-directories --cut-dirs=$cutdir --passive-ftp -P $mirrordir ${mirror}setup.ini ./clean_setup.pl -writelist wget -r -N --no-host-directories --cut-dirs=$cutdir --passive-ftp -P $mirrordir -i $filelist -B $mirror ./clean_setup.pl -Arch -Setup [-- Attachment #3: Type: text/plain, Size: 214 bytes --] -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Bug reporting: http://cygwin.com/bugs.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/ ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Using a real mirroring tool... 2002-04-30 10:09 ` Volker Quetschke 2002-04-30 13:43 ` Volker Quetschke @ 2002-04-30 14:00 ` Michael A Chase 1 sibling, 0 replies; 24+ messages in thread From: Michael A Chase @ 2002-04-30 14:00 UTC (permalink / raw) To: Volker Quetschke, cygwin On Tue, 30 Apr 2002 16:00:52 +0200 Volker Quetschke <quetschke@scytek.de> wrote: > I attached the modified clean_setup.pl plus a shell scrip to get the > missing files from a mirror. You have to modify the getmissing.sh to > choose your mirror and your target download directory. The directory and > both files must exist the same directory as your setup.exe. Just a quick > hack, I guarantee nothing! You started with a little bit old version of the script. I have uploaded a new version 1.0401 which uses a '-misslist file' option to write the list to the specified file. The new version also allows you to exclude packages that you are not interested in with the '-H mask' option. This will allow your wget with the list more selective. Some of the other options have been changed as well, so be sure to run with '-h' before using the script. -- Mac :}) ** I normally forward private questions to the appropriate mail list. ** Ask Smarter: http://www.tuxedo.org/~esr/faqs/smart-questions.html Give a hobbit a fish and he eats fish for a day. Give a hobbit a ring and he eats fish for an age. -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Bug reporting: http://cygwin.com/bugs.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/ ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: setup 2.194.2.24: Bug (?) in downloading from internet 2002-04-28 4:47 setup 2.194.2.24: Bug (?) in downloading from internet Robert Collins 2002-04-28 8:08 ` Sam Edge 2002-04-28 15:04 ` Gary R. Van Sickle @ 2002-04-29 4:07 ` Max Bowsher 2002-04-29 4:24 ` Robert Collins 2 siblings, 1 reply; 24+ messages in thread From: Max Bowsher @ 2002-04-29 4:07 UTC (permalink / raw) To: cygwin Robert Collins <robert.collins@itdomain.com.au> wrote: >> -----Original Message----- >> From: Sam Edge [mailto:sam_edgeZZZ@hotmail.com] >> Sent: Sunday, April 28, 2002 8:17 PM >> Every version of setup.exe I've used exhibits this annoying >> behaviour but them I'm new here. >> >> I agree with Daniele. I install different subsets of the >> packages on several machines. I also like to keep a copy of >> the most up-to-date versions of packages that I don't >> currently use on any of them, in case I need them where I've >> no (or poor) access to the 'Net. > > So use a mirroring tool! Setup.exe is -not- designed for this. If you > want a setup.ini parsing mirroring tool, take me up on my offer to help > someone leverage the setup.ini codebase to build a mirroring tool. > > Rob But setup.exe has operated in the correct way in the past! Why write a new tool, when fixing a bug in the current one would do all that needs to be done? NB: I'm not saying 'go fix that now'. When (if? ;-) I find time to understand the setup code, I will do this, if no one has beaten me to it. Max. -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Bug reporting: http://cygwin.com/bugs.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/ ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: setup 2.194.2.24: Bug (?) in downloading from internet 2002-04-29 4:07 ` setup 2.194.2.24: Bug (?) in downloading from internet Max Bowsher @ 2002-04-29 4:24 ` Robert Collins 0 siblings, 0 replies; 24+ messages in thread From: Robert Collins @ 2002-04-29 4:24 UTC (permalink / raw) To: Max Bowsher, cygwin === ----- Original Message ----- From: "Max Bowsher" <maxb@ukf.net> To: <cygwin@cygwin.com> > But setup.exe has operated in the correct way in the past! Why write a new tool, > when fixing a bug in the current one would do all that needs to be done? Because the current tool is more complex than it should be, and a new tool can be accurately target. The current tool can then be simplified somewhat (i.e. download only could go away if we had a (non cygwin linked) mirroring tool. Or a variation of this. > NB: I'm > not saying 'go fix that now'. When (if? ;-) I find time to understand the setup > code, I will do this, if no one has beaten me to it. Thanks. Rob -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Bug reporting: http://cygwin.com/bugs.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/ ^ permalink raw reply [flat|nested] 24+ messages in thread
* RE: Using a real mirroring tool... @ 2002-04-30 6:50 Robert Collins 0 siblings, 0 replies; 24+ messages in thread From: Robert Collins @ 2002-04-30 6:50 UTC (permalink / raw) To: Sam Edge, cygwin > -----Original Message----- > From: Sam Edge [mailto:sam_edgeZZZ@hotmail.com] > Sent: Tuesday, April 30, 2002 10:18 PM > Or am I missing something? ("Again" do I hear from the > chorus? Kindly leave the auditorium, sir!) ;-) As you may know, you can use the md5sum files to verify what was downloaded via script, and until the md5sums are in setup.ini (coming soon) the only place to get them is the mirror sites. Rob -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Bug reporting: http://cygwin.com/bugs.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/ ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Using a real mirroring tool...
@ 2002-04-30 19:05 Earnie Boyd
0 siblings, 0 replies; 24+ messages in thread
From: Earnie Boyd @ 2002-04-30 19:05 UTC (permalink / raw)
To: Earnie Boyd
[-- Attachment #1: Type: text/plain, Size: 138 bytes --]
You might want to check out http://mirrors.rcn.net/pub/mirror/index.html
for a mirroring tool that uses perl for ftp/http access.
Earnie.
[-- Attachment #2: cygwin.49112 --]
[-- Type: message/rfc822, Size: 1209 bytes --]
From: Charles Wilson <cwilson@ece.gatech.edu>
To: Charles Wilson <cwilson@ece.gatech.edu>
Cc: cygwin@cygwin.com
Subject: Re: Using a real mirroring tool...
Date: Mon, 29 Apr 2002 16:04:14 -0400
Message-ID: <3CCDA73E.5000503@ece.gatech.edu>
Charles Wilson wrote:
> So, if someone cares enough to do the work, there is still a need for
> the "additional tool" that uses the setup.exe codebase to do
> "cygwin-specific distributed mirroring". See my earlier rant.
Oh, I left out the most obvious reason why rsync won't solve all of our
problems: we have lots of ftp and http mirrors, but very few (ONE?)
rsync mirrors. That really kills the "distributed" nature of the mirror
system; that poor guy in Thailand with the slow connection would be
rsync'ing across the Pacific from Indiana...that can't be good.
A tool that leverages the setup.exe codebase would inherit the ability
to use the existing ftp and http mirror sites.
--Chuck
</stating the obvious>
P.S. you might want to add --exclude *-src.tar.* to the option string,
if you don't want the -src tarballs.
[-- Attachment #3: Type: text/plain, Size: 214 bytes --]
--
Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Bug reporting: http://cygwin.com/bugs.html
Documentation: http://cygwin.com/docs.html
FAQ: http://cygwin.com/faq/
^ permalink raw reply [flat|nested] 24+ messages in thread
end of thread, other threads:[~2002-04-30 23:49 UTC | newest] Thread overview: 24+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2002-04-28 4:47 setup 2.194.2.24: Bug (?) in downloading from internet Robert Collins 2002-04-28 8:08 ` Sam Edge 2002-04-28 15:04 ` Gary R. Van Sickle 2002-04-28 15:09 ` Sam Edge 2002-04-28 15:30 ` Charles Wilson 2002-04-28 16:05 ` Christopher Faylor 2002-04-28 16:09 ` Sam Edge 2002-04-28 19:08 ` Michael A Chase 2002-04-29 0:06 ` Christopher Faylor 2002-04-29 0:57 ` Cliff Hones 2002-04-29 4:58 ` Robert Collins 2002-04-28 16:05 ` Sam Edge 2002-04-29 13:01 ` Using a real mirroring tool Charles Wilson 2002-04-29 17:17 ` Charles Wilson 2002-04-29 17:21 ` Charles Wilson 2002-04-29 17:46 ` Charles Wilson 2002-04-30 6:20 ` Sam Edge 2002-04-30 10:09 ` Volker Quetschke 2002-04-30 13:43 ` Volker Quetschke 2002-04-30 14:00 ` Michael A Chase 2002-04-29 4:07 ` setup 2.194.2.24: Bug (?) in downloading from internet Max Bowsher 2002-04-29 4:24 ` Robert Collins 2002-04-30 6:50 Using a real mirroring tool Robert Collins 2002-04-30 19:05 Earnie Boyd
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).