* RE: setup 2.194.2.24: Bug (?) in downloading from internet @ 2002-04-28 4:47 Robert Collins 2002-04-28 8:08 ` Sam Edge ` (2 more replies) 0 siblings, 3 replies; 42+ messages in thread From: Robert Collins @ 2002-04-28 4:47 UTC (permalink / raw) To: Sam Edge, cygwin > -----Original Message----- > From: Sam Edge [mailto:sam_edgeZZZ@hotmail.com] > Sent: Sunday, April 28, 2002 8:17 PM > Every version of setup.exe I've used exhibits this annoying > behaviour but them I'm new here. > > I agree with Daniele. I install different subsets of the > packages on several machines. I also like to keep a copy of > the most up-to-date versions of packages that I don't > currently use on any of them, in case I need them where I've > no (or poor) access to the 'Net. So use a mirroring tool! Setup.exe is -not- designed for this. If you want a setup.ini parsing mirroring tool, take me up on my offer to help someone leverage the setup.ini codebase to build a mirroring tool. Rob -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Bug reporting: http://cygwin.com/bugs.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/ ^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: setup 2.194.2.24: Bug (?) in downloading from internet 2002-04-28 4:47 setup 2.194.2.24: Bug (?) in downloading from internet Robert Collins @ 2002-04-28 8:08 ` Sam Edge 2002-04-28 15:04 ` Gary R. Van Sickle 2002-04-29 4:07 ` setup 2.194.2.24: Bug (?) in downloading from internet Max Bowsher 2 siblings, 0 replies; 42+ messages in thread From: Sam Edge @ 2002-04-28 8:08 UTC (permalink / raw) To: cygwin Robert Collins wrote in <FC169E059D1A0442A04C40F86D9BA7600C5F42@itdomain003.itdomain.net.au> in gmane.os.cygwin on Sun, 28 Apr 2002 20:22:30 +1000: > > I agree with Daniele. I install different subsets of the > > packages on several machines. I also like to keep a copy of > > the most up-to-date versions of packages that I don't > > currently use on any of them, in case I need them where I've > > no (or poor) access to the 'Net. > > So use a mirroring tool! Setup.exe is -not- designed for this. Surely if setup.exe isn't designed with at least a nod to this way of working why does it have "download from Internet" and "install from local directory" options, instead of just instisting on connecting to a server every time? > If you > want a setup.ini parsing mirroring tool, take me up on my offer to help > someone leverage the setup.ini codebase to build a mirroring tool. Although I've not looked at it, setup.exe must already have most of the code required to operate in the way we'd like, surely? If you tell me the module name(s) for setup.exe and any documentation about setup.ini and the local and FTP/HTTP server directory structures I'd certainly be interested to download them from the CVS server and take a look. -- Sam Edge -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Bug reporting: http://cygwin.com/bugs.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/ ^ permalink raw reply [flat|nested] 42+ messages in thread
* RE: setup 2.194.2.24: Bug (?) in downloading from internet 2002-04-28 4:47 setup 2.194.2.24: Bug (?) in downloading from internet Robert Collins 2002-04-28 8:08 ` Sam Edge @ 2002-04-28 15:04 ` Gary R. Van Sickle 2002-04-28 15:09 ` Sam Edge 2002-04-29 4:07 ` setup 2.194.2.24: Bug (?) in downloading from internet Max Bowsher 2 siblings, 1 reply; 42+ messages in thread From: Gary R. Van Sickle @ 2002-04-28 15:04 UTC (permalink / raw) To: cygwin > > -----Original Message----- > > From: Sam Edge [mailto:sam_edgeZZZ@hotmail.com] > > Sent: Sunday, April 28, 2002 8:17 PM > > > Every version of setup.exe I've used exhibits this annoying > > behaviour but them I'm new here. > > > > I agree with Daniele. I install different subsets of the > > packages on several machines. I also like to keep a copy of > > the most up-to-date versions of packages that I don't > > currently use on any of them, in case I need them where I've > > no (or poor) access to the 'Net. > > So use a mirroring tool! Setup.exe is -not- designed for this. If you > want a setup.ini parsing mirroring tool, take me up on my offer to help > someone leverage the setup.ini codebase to build a mirroring tool. > > Rob > What am I missing in all these posts? Didn't somebody post a while ago that if you set your "Local Package Directory" to a network share and do "Install From Internet", you get all the functionality people are wanting here? Now I admit I've not tried it myself (though I have plenty of reason to do so and should, maybe next week), but what more do people want here? Maybe we should dump the "Download" option entirely. Seems that all it serves to do is confuse people that should really be looking at rsync. -- Gary R. Van Sickle Brewer. Patriot. -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Bug reporting: http://cygwin.com/bugs.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/ ^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: setup 2.194.2.24: Bug (?) in downloading from internet 2002-04-28 15:04 ` Gary R. Van Sickle @ 2002-04-28 15:09 ` Sam Edge 2002-04-28 15:30 ` Charles Wilson 0 siblings, 1 reply; 42+ messages in thread From: Sam Edge @ 2002-04-28 15:09 UTC (permalink / raw) To: cygwin Gary R. Van Sickle wrote in <NCBBIHCHBLCMLBLOBONKKEOLCNAA.g.r.vansickle@worldnet.att.net> in gmane.os.cygwin on Sun, 28 Apr 2002 14:12:36 -0500: > What am I missing in all these posts? Didn't somebody post a while ago that if > you set your "Local Package Directory" to a network share and do "Install From > Internet", you get all the functionality people are wanting here? No. You can use the same local package directory to install on multiple machines very nicely. That aspect works fine. The problem is that when you do "download from Internet" the list tells you what you have installed in c:\cygwin on the machine where you're running setup.exe not what you've previously downloaded to the package directory. Add to this the lack of an option to skip previously downloaded packages when they're manually selected (or auto-selected by the dependency management) and it's a problem if you're using a single local package directory for several different machines that have different package sets installed. On a single machine where you download and immediately install only what'll be used locally it can still be a problem for those on slow, unreliable or time-limited connections. For example, many people in the UK use a 56kbps (or more usually around 45kbps) connection. Many unmetered ISP packages disconnect every two hours to prevent users hogging dial-in capacity they're not using. Downloading even a basic package with all its dependencies can run over this time limit so you can't use the "install from Internet" option. But when you try to use "download from Internet" and then re-start setup.exe after re-connecting it doesn't show you the packages you already have. Worse, when you select your package again it auto-selects all the dependencies and tries to download them all again and so will never complete because you'll keep hitting the disconnect timeout. What you have to do is have an Explorer window open looking at the package directory and manually go back and de-select all the ones you've already got after auto-select has added them. This is prone to error, especially if you've not used clean_setup.pl to flatten the structure between each download session. It's also annoying because it should be easy to automate into setup.exe itself. > Maybe we should dump the "Download" option entirely. Seems that all it serves > to do is confuse people that should really be looking at rsync. Please, no! Apart from the reason I give above, on my main machine here I always download first in my normal NT user account so I can continue to work while it's going on. I then log in as ADMINISTRATOR to install because I have the Cygwin directory hierarchy protected against write by normal users. That way I'm running with potentially dangerous privileges for as short a time as possible. -- Sam Edge -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Bug reporting: http://cygwin.com/bugs.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/ ^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: setup 2.194.2.24: Bug (?) in downloading from internet 2002-04-28 15:09 ` Sam Edge @ 2002-04-28 15:30 ` Charles Wilson 2002-04-28 16:05 ` Sam Edge ` (2 more replies) 0 siblings, 3 replies; 42+ messages in thread From: Charles Wilson @ 2002-04-28 15:30 UTC (permalink / raw) To: Sam Edge, cygwin <begin rant> Robert, you are a saint. I am sick and tired of the whining about setup and downloads. SETUP IS NOT A MIRRORING TOOL. In order to make this clear, I am in favor of completely removing the download-only option, leaving just install-from-internet and install-from-localdir. This will force the whiners to either shut up and use a fscking mirroring tool, or help out: Robert has already said what the correct solution to all this is: create a separate program that leverages the setup.exe codebase, to do "multi-source merged mirroring". This will require: 1) Add command-line options for setup.exe. Robert has added a helpful set of examples. He even created a whole new project, GetOpt++, to support this. What more can Robert, all by himself, do? 2) Create a new cygmirror.cc as part of the setup module, that uses some (but not all) of the other classes in the setup module. According to Robert: > The differences between setup and a mirroring tool include: > * All the install code isn't needed. > * Attention to local packages isn't needed. > * Persistent selection of packages irrespective of install status > should be present. > * command line driven may be useful? I'll add one more: reading options (or package lists) from a file. Now, unless somebody who WANTS this functionality actually steps up to the plate and digs in, can we PLEASE stop discussing this issue? Removing the download-only option will also have the salutory effect of forcing the whiners to either: 1) use a REAL mirroring tool, then use setup.exe to install-from-local-dir 1a) heck, you could even use wget; I've posted a script to this mailing list that does exactly that. 2) help out with the coding. Either way, it kills "setup download only mode doesn't do what I want" "What's up with the wacky %%%'ified directory names in my local setup directory" "Setup sucks" threads that NEVER DIE!. --Chuck -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Bug reporting: http://cygwin.com/bugs.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/ ^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: setup 2.194.2.24: Bug (?) in downloading from internet 2002-04-28 15:30 ` Charles Wilson @ 2002-04-28 16:05 ` Sam Edge 2002-04-28 16:05 ` Christopher Faylor 2002-04-29 13:01 ` Using a real mirroring tool Charles Wilson 2 siblings, 0 replies; 42+ messages in thread From: Sam Edge @ 2002-04-28 16:05 UTC (permalink / raw) To: cygwin Charles Wilson wrote in <3CCC7290.706@ece.gatech.edu> in gmane.os.cygwin on Sun, 28 Apr 2002 18:07:12 -0400: > <begin rant> > Robert, you are a saint. > I am sick and tired of the whining about setup and downloads. > SETUP IS NOT A MIRRORING TOOL. LOL. No need to get your knickers in a twist! I was merely explaining the limitations of setup.exe to Gary. ;-) The problem with time limited slow connections is valid even when using it purely for local setup on a single machine i.e. when NOT using it as a mirroring tool. A simple checkbox option on the download-only page to skip pre-downloaded packages even if they're selected in the list would deal with that and I really don't see how this would compromise its function in any way, as long as it were off by default. Unless someone can give a good reason why it /would/ be a bad idea, I'll look at the code and see if I can write a version that has this. > In order to make this clear, I am in favor of completely removing the > download-only option, leaving just install-from-internet and > install-from-localdir. This will force the whiners to either shut up > and use a fscking mirroring tool, or help out: It will also stop users on slow time-limited connections from being able to use setup.exe at all unless they first use a conventional FTP/HTTP mirroring client and set up their own local FTP server to which to connect setup.exe. They then have to work out which parts of one of the current mirrors are required for their particular needs and download those packages. They'll also end up with two copies of everything on their hard discs. This is hardly a good way of increasing the Cygwin user/developer base! -- Sam Edge -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Bug reporting: http://cygwin.com/bugs.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/ ^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: setup 2.194.2.24: Bug (?) in downloading from internet 2002-04-28 15:30 ` Charles Wilson 2002-04-28 16:05 ` Sam Edge @ 2002-04-28 16:05 ` Christopher Faylor 2002-04-28 16:09 ` Sam Edge 2002-04-28 19:08 ` Michael A Chase 2002-04-29 13:01 ` Using a real mirroring tool Charles Wilson 2 siblings, 2 replies; 42+ messages in thread From: Christopher Faylor @ 2002-04-28 16:05 UTC (permalink / raw) To: cygwin On Sun, Apr 28, 2002 at 06:07:12PM -0400, Charles Wilson wrote: ><begin rant> >Robert, you are a saint. > >I am sick and tired of the whining about setup and downloads. > >SETUP IS NOT A MIRRORING TOOL. > >In order to make this clear, I am in favor of completely removing the >download-only option, leaving just install-from-internet and >install-from-localdir. This will force the whiners to either shut up >and use a fscking mirroring tool, or help out: I am really torn on this one. I fully agree that setup.exe is not a mirroring tool. I've vigorously made that assertion on many occasions. However, I think I may have been responsible for setup's current behavior of basing what is downloaded on what is installed. Even if I was not, though, I think that the explanations about why some people need this are rather convincing. If you're on the other end of a slow connection and are apt to be disconnected every couple of hours, you will *never* be able to do a complete install since the setup.exe will start from scratch every time you reconnect. It doesn't matter if you chose "download from internet" or not since setup.exe will only consider what's installed when considering what to download. That just seems like a bug to me. If I have this right then even removing download from internet does not eliminate this problem. So, I think the right answer to this is the standard one "patches gratefully accepted". >Robert has already said what the correct solution to all this is: create >a separate program that leverages the setup.exe codebase, to do >"multi-source merged mirroring". I do agree with this philosophy for anything beyond fixing the above, IMO, glitch. setup.exe should be a fairly simple tool. If you find that it isn't meeting your needs then you should be looking for a better tool. cgf -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Bug reporting: http://cygwin.com/bugs.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/ ^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: setup 2.194.2.24: Bug (?) in downloading from internet 2002-04-28 16:05 ` Christopher Faylor @ 2002-04-28 16:09 ` Sam Edge 2002-04-28 19:08 ` Michael A Chase 1 sibling, 0 replies; 42+ messages in thread From: Sam Edge @ 2002-04-28 16:09 UTC (permalink / raw) To: cygwin Christopher Faylor wrote in <20020428223000.GB13849@redhat.com> in gmane.os.cygwin on Sun, 28 Apr 2002 18:30:00 -0400: > If you're on the other end of a slow connection and are apt to be > disconnected every couple of hours, you will *never* be able to do a > complete install since the setup.exe will start from scratch every time > you reconnect. It doesn't matter if you chose "download from internet" or > not since setup.exe will only consider what's installed when considering > what to download. That just seems like a bug to me. Thanks. Even something like the XFree86 "basic" install can take a long time, let alone trying to install everything in one hit. Sometimes people forget that there are many still out there on creaky old dial-up connections. > If I have this right then even removing download from internet does not > eliminate this problem. No, it makes it worse. > So, I think the right answer to this is the standard one "patches > gratefully accepted". I'll look but I'm making no promises! ;-D -- Sam Edge -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Bug reporting: http://cygwin.com/bugs.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/ ^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: setup 2.194.2.24: Bug (?) in downloading from internet 2002-04-28 16:05 ` Christopher Faylor 2002-04-28 16:09 ` Sam Edge @ 2002-04-28 19:08 ` Michael A Chase 2002-04-29 0:06 ` Christopher Faylor 1 sibling, 1 reply; 42+ messages in thread From: Michael A Chase @ 2002-04-28 19:08 UTC (permalink / raw) To: cygwin On Sun, 28 Apr 2002 18:30:00 -0400 Christopher Faylor <cgf@redhat.com> wrote: > On Sun, Apr 28, 2002 at 06:07:12PM -0400, Charles Wilson wrote: > If you're on the other end of a slow connection and are apt to be > disconnected every couple of hours, you will *never* be able to do a > complete install since the setup.exe will start from scratch every time > you reconnect. It doesn't matter if you chose "download from internet" > or > not since setup.exe will only consider what's installed when considering > what to download. That just seems like a bug to me. That's what the discussion has been about for the last week or so. After lots of requests for this functionality, the position eventually hardened to "it's not a download tool". > If I have this right then even removing download from internet does not > eliminate this problem. That depends on how capable wget or rsync is. I have not looked into it but if we do remove the download only function from setup.exe, we (me?) should probably add something under cygwin.com describing how to mirror selected packages. > So, I think the right answer to this is the standard one "patches > gratefully accepted". If download handling is going to change, at least Robert will have to buy into it. A request for patches can follow that. -- Mac :}) ** I normally forward private questions to the appropriate mail list. ** Ask Smarter: http://www.tuxedo.org/~esr/faqs/smart-questions.html Give a hobbit a fish and he eats fish for a day. Give a hobbit a ring and he eats fish for an age. -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Bug reporting: http://cygwin.com/bugs.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/ ^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: setup 2.194.2.24: Bug (?) in downloading from internet 2002-04-28 19:08 ` Michael A Chase @ 2002-04-29 0:06 ` Christopher Faylor 2002-04-29 0:57 ` Cliff Hones 0 siblings, 1 reply; 42+ messages in thread From: Christopher Faylor @ 2002-04-29 0:06 UTC (permalink / raw) To: cygwin On Sun, Apr 28, 2002 at 04:04:36PM -0700, Michael A Chase wrote: >On Sun, 28 Apr 2002 18:30:00 -0400 Christopher Faylor <cgf@redhat.com> wrote: > >> On Sun, Apr 28, 2002 at 06:07:12PM -0400, Charles Wilson wrote: > >>If you're on the other end of a slow connection and are apt to be >>disconnected every couple of hours, you will *never* be able to do a >>complete install since the setup.exe will start from scratch every time >>you reconnect. It doesn't matter if you chose "download from internet" >>or not since setup.exe will only consider what's installed when >>considering what to download. That just seems like a bug to me. > >That's what the discussion has been about for the last week or so. >After lots of requests for this functionality, the position eventually >hardened to "it's not a download tool". Right. But it started with my suggesting this behavior was a bug and (I thought) Robert agreeing. So, I thought it was time to chime in again. >> If I have this right then even removing download from internet does not >> eliminate this problem. > >That depends on how capable wget or rsync is. I have not looked into it >but if we do remove the download only function from setup.exe, we (me?) >should probably add something under cygwin.com describing how to mirror >selected packages. It's not a question of mirroring. It's a question of setup.exe being completely unusable for some people. We can't say that setup.exe is the only method we'll support but then tell people to use some other method when it fails for them as they try to use it in a perfectly acceptable way. The bottom line is that if we have to tell someone to use wget or rsync just to do a normal install, something is wrong. It seems like this problem would be trivially fixed by not attempting to redownload something that has already been downloaded. cgf -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Bug reporting: http://cygwin.com/bugs.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/ ^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: setup 2.194.2.24: Bug (?) in downloading from internet 2002-04-29 0:06 ` Christopher Faylor @ 2002-04-29 0:57 ` Cliff Hones 2002-04-29 4:58 ` Robert Collins 0 siblings, 1 reply; 42+ messages in thread From: Cliff Hones @ 2002-04-29 0:57 UTC (permalink / raw) To: cygwin Christopher Faylor <cgf@redhat.com> wrote: > The bottom line is that if we have to tell someone to use wget or rsync > just to do a normal install, something is wrong. > > It seems like this problem would be trivially fixed by not attempting to > redownload something that has already been downloaded. Hooray. Seems lots of people have been trying to say this. A week ago when it was, for a moment, acknowledged as a bug, I downloaded the source with ideas of trying to fix it. But then the mood shifted back to it being a feature, so I lost interest. Now Sam Edge is taking the same path. Good luck. What would happen if someone did produce a version of setup identical to the current, except for the removal of redownload? Would it be adopted? One gets an unfortunate undercurrent from this list that if it wasn't approved by the right people it might be suppressed (and left to circulate on the black market - possibly the worst scenario). -- Cliff -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Bug reporting: http://cygwin.com/bugs.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/ ^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: setup 2.194.2.24: Bug (?) in downloading from internet 2002-04-29 0:57 ` Cliff Hones @ 2002-04-29 4:58 ` Robert Collins 0 siblings, 0 replies; 42+ messages in thread From: Robert Collins @ 2002-04-29 4:58 UTC (permalink / raw) To: Cliff Hones, cygwin === ----- Original Message ----- From: "Cliff Hones" <cliff@aonix.co.uk> To: <cygwin@cygwin.com> > > Now Sam Edge is taking the same path. Good luck. > > What would happen if someone did produce a version of setup identical > to the current, except for the removal of redownload? Would it > be adopted? As it stands, I don't know. I've asked everyone to write in and say whether they use the redownload capability. > One gets an unfortunate undercurrent from this list that > if it wasn't approved by the right people it might be suppressed > (and left to circulate on the black market - possibly the worst > scenario). Heh, this is open source, there is no black market :}. Realistically, anyone can use anything they want to install, but 'you' the users have a realistic expectation that 'we' the developers will provide a tool, and support that tool. Conversely we expect that anyone wanting assistance with install will use our provided tool. Rob -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Bug reporting: http://cygwin.com/bugs.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/ ^ permalink raw reply [flat|nested] 42+ messages in thread
* Using a real mirroring tool... 2002-04-28 15:30 ` Charles Wilson 2002-04-28 16:05 ` Sam Edge 2002-04-28 16:05 ` Christopher Faylor @ 2002-04-29 13:01 ` Charles Wilson 2002-04-29 17:17 ` Charles Wilson 2002-04-29 17:21 ` Charles Wilson 2 siblings, 2 replies; 42+ messages in thread From: Charles Wilson @ 2002-04-29 13:01 UTC (permalink / raw) To: cygwin Poor archive.progeny.com ... First, rsync places a heavy load on the server, so please do not EVERYBODY run out and try this immediately. I suggest, for initial rsyncs (when you need to download EVERYTHING), that folks follow this rotation: Surname Begins With Initial rysnc ABC May 1 DEF May 2 GHI May 3 JKL May 4 MNO May 5 PQR May 6 STU May 7 VWX May 8 YZ May 9 Incremental rsync's don't place as much of a load on the server, so it won't matter as much after that... Here is a script that will mirror the cygwin/release tree and setup.exe/setup.ini (you'll need to change LOCALDIR). Also, you'll need to remove --dry-run from EXTRAOPTS. If anybody can identify other mirrors that provide rsync: support, please post the appropriate MIRROR= line as a reply to this message. --------------------- #!/bin/sh MIRROR=rsync://archive.progeny.com/cygwin LOCALDIR=/annex1/private/software/cygwin/ OPTS="-rlptgoDzu" EXTRAOPTS="--delete -vv --dry-run" RSYNC=/usr/bin/rsync ${RSYNC} ${OPTS} ${EXTRAOPTS} ${MIRROR}/release ${LOCALDIR} ${RSYNC} ${OPTS} ${EXTRAOPTS} ${MIRROR}/setup.* ${LOCALDIR} --------------------- --Chuck -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Bug reporting: http://cygwin.com/bugs.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/ ^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: Using a real mirroring tool... 2002-04-29 13:01 ` Using a real mirroring tool Charles Wilson @ 2002-04-29 17:17 ` Charles Wilson 2002-04-29 17:21 ` Charles Wilson 1 sibling, 0 replies; 42+ messages in thread From: Charles Wilson @ 2002-04-29 17:17 UTC (permalink / raw) To: Charles Wilson; +Cc: cygwin Actually, this is a better set of options: we don't need to preserve owner, groups, or devices, and the modify-window helps with the limited timestamp granularity of the FAT filesystem. Remember to fix LOCALDIR and remove --dry-run. --------------------- #!/bin/sh MIRROR=rsync://archive.progeny.com/cygwin LOCALDIR=/annex1/private/software/cygwin/ OPTS="-rlptzu --modify-window=2" EXTRAOPTS="--delete -vv --dry-run" RSYNC=/usr/bin/rsync ${RSYNC} ${OPTS} ${EXTRAOPTS} ${MIRROR}/release ${LOCALDIR} ${RSYNC} ${OPTS} ${EXTRAOPTS} ${MIRROR}/setup.* ${LOCALDIR} --------------------- --Chuck Charles Wilson wrote: > Poor archive.progeny.com ... > > First, rsync places a heavy load on the server, so please do not > EVERYBODY run out and try this immediately. I suggest, for initial > rsyncs (when you need to download EVERYTHING), that folks follow this > rotation: > > Surname Begins With Initial rysnc > ABC May 1 > DEF May 2 > GHI May 3 > JKL May 4 > MNO May 5 > PQR May 6 > STU May 7 > VWX May 8 > YZ May 9 > > Incremental rsync's don't place as much of a load on the server, so it > won't matter as much after that... > > Here is a script that will mirror the cygwin/release tree and > setup.exe/setup.ini (you'll need to change LOCALDIR). Also, you'll need > to remove --dry-run from EXTRAOPTS. If anybody can identify other > mirrors that provide rsync: support, please post the appropriate MIRROR= > line as a reply to this message. > > --------------------- > #!/bin/sh > MIRROR=rsync://archive.progeny.com/cygwin > LOCALDIR=/annex1/private/software/cygwin/ > OPTS="-rlptgoDzu" > EXTRAOPTS="--delete -vv --dry-run" > RSYNC=/usr/bin/rsync > > ${RSYNC} ${OPTS} ${EXTRAOPTS} ${MIRROR}/release ${LOCALDIR} > ${RSYNC} ${OPTS} ${EXTRAOPTS} ${MIRROR}/setup.* ${LOCALDIR} > --------------------- > > --Chuck > -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Bug reporting: http://cygwin.com/bugs.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/ ^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: Using a real mirroring tool... 2002-04-29 13:01 ` Using a real mirroring tool Charles Wilson 2002-04-29 17:17 ` Charles Wilson @ 2002-04-29 17:21 ` Charles Wilson 2002-04-29 17:46 ` Charles Wilson 2002-04-30 10:09 ` Volker Quetschke 1 sibling, 2 replies; 42+ messages in thread From: Charles Wilson @ 2002-04-29 17:21 UTC (permalink / raw) To: Charles Wilson; +Cc: cygwin A few notes: Using rsync does not help the following problem: I have four favorite mirrors, plus a "private" setup-compatible website from my friend Bob. setup.exe will show me a merged view of the contents of all of those sites, insuring that I get the latest version of a package wherever it is found. By merging in the "private" site, setup enables me to get my friend Bob's latest version of "King of Frobozz, Tyrant" game using the same setup tool/view, along with the latest apache package or binutils package from the "official" cygwin mirrors. rsync can't merge. I could rsync each of the four mirrors down to my machine into separate directories (assuming the mirrors all had anonymous rsync servers), and rsync Bob's private website (assuming his webhost provided rsync -- hah!). But even then, setup can't merge multiple local directories, at least not right now. So, we still need some variant of setup.exe that can merge multiple setup-compatible sites, and "mirror" only the latest stuff into a single local "release" tree / setup.ini. Also, rsync does some stuff we don't really care about: since most of the files are compressed, rsync's sophisticated "download changed chunks" algorithm is pretty useless. In compressed files, the first byte that is different in the uncompressed file will cause a cascade of changes throughout the remainder of the compressed file. So: 1 byte diff in uncompressed file == entire compressed file is different. OTOH, since we typically (never?) upload a modified version of a given file USING THE SAME NAME, we don't really care too much about this. "Our" mirroring tool can just use the md5 hashes on the mirror and locally to determine the rare instance when a given file has had its contents changed -- but then we'd download the ENTIRE file, not "changed chunks". (However, if this ever happens, isn't it a sign that something went wrong somewhere?) Also, there's really no need to use rsync's compression engine, since most of the data to be transferred has already been compressed. Finally, the local "setup.ini" will never(?) match the setup.ini at any one of the mirrors or private sites; it will be a merger of them. So, rsync can't do that. So, if someone cares enough to do the work, there is still a need for the "additional tool" that uses the setup.exe codebase to do "cygwin-specific distributed mirroring". See my earlier rant. But, the rsync method should enable folks with slow connections to limp along for a while...without trying to turn setup.exe ITSELF into a mirror tool. --Chuck -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Bug reporting: http://cygwin.com/bugs.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/ ^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: Using a real mirroring tool... 2002-04-29 17:21 ` Charles Wilson @ 2002-04-29 17:46 ` Charles Wilson 2002-04-30 6:20 ` Sam Edge 2002-04-30 10:09 ` Volker Quetschke 1 sibling, 1 reply; 42+ messages in thread From: Charles Wilson @ 2002-04-29 17:46 UTC (permalink / raw) To: Charles Wilson; +Cc: cygwin Charles Wilson wrote: > So, if someone cares enough to do the work, there is still a need for > the "additional tool" that uses the setup.exe codebase to do > "cygwin-specific distributed mirroring". See my earlier rant. Oh, I left out the most obvious reason why rsync won't solve all of our problems: we have lots of ftp and http mirrors, but very few (ONE?) rsync mirrors. That really kills the "distributed" nature of the mirror system; that poor guy in Thailand with the slow connection would be rsync'ing across the Pacific from Indiana...that can't be good. A tool that leverages the setup.exe codebase would inherit the ability to use the existing ftp and http mirror sites. --Chuck </stating the obvious> P.S. you might want to add --exclude *-src.tar.* to the option string, if you don't want the -src tarballs. -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Bug reporting: http://cygwin.com/bugs.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/ ^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: Using a real mirroring tool... 2002-04-29 17:46 ` Charles Wilson @ 2002-04-30 6:20 ` Sam Edge 0 siblings, 0 replies; 42+ messages in thread From: Sam Edge @ 2002-04-30 6:20 UTC (permalink / raw) To: cygwin Charles Wilson wrote in <3CCDA73E.5000503@ece.gatech.edu> in gmane.os.cygwin on Mon, 29 Apr 2002 16:04:14 -0400: > P.S. you might want to add --exclude *-src.tar.* to the option string, > if you don't want the -src tarballs. As I understand it you can also exclude the setup.hint and md5.sum files if you're just populating a local package directory. They're very small but every little helps and it reduces the clutter and can save a few sectors on the local disc. Or am I missing something? ("Again" do I hear from the chorus? Kindly leave the auditorium, sir!) ;-) -- Sam Edge -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Bug reporting: http://cygwin.com/bugs.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/ ^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: Using a real mirroring tool... 2002-04-29 17:21 ` Charles Wilson 2002-04-29 17:46 ` Charles Wilson @ 2002-04-30 10:09 ` Volker Quetschke 2002-04-30 13:43 ` Volker Quetschke 2002-04-30 14:00 ` Michael A Chase 1 sibling, 2 replies; 42+ messages in thread From: Volker Quetschke @ 2002-04-30 10:09 UTC (permalink / raw) To: cygwin [-- Attachment #1: Type: text/plain, Size: 615 bytes --] Hi! Charles Wilson wrote: > A few notes: > > ... > > rsync can't merge. Yes, but Michael A. Chases clean_setup.pl can! I allowed myself to add a new option to clean_setup.pl, it exports the list of missing files to a file. Then you can use wget to get all the missing files. I attached the modified clean_setup.pl plus a shell scrip to get the missing files from a mirror. You have to modify the getmissing.sh to choose your mirror and your target download directory. The directory and both files must exist the same directory as your setup.exe. Just a quick hack, I guarantee nothing! Bye Volker [-- Attachment #2: getmissing.sh --] [-- Type: text/plain, Size: 757 bytes --] #!/bin/bash # Mirror script for cygwin # Uses Michael A. Chase "cleanup_setup.pl" # in a modified version (V1.0303) # and wget # getmissing.sh and cleanup_setup.pl must reside in the same directory # as setup.exe # Files to download filelist="missing.lst" # Choose your mirror with directory of setup.ini mirror="ftp://ftp.inf.tu-dresden.de/software/windows/cygwin32/" # Cut directories 1 2 3 cutdir=3 # Name your download directory mirrordir="_mymirror" wget --mirror --no-host-directories --cut-dirs=$cutdir --passive-ftp -P $mirrordir ${mirror}setup.ini ./clean_setup.pl -Arch -writelist wget --mirror --no-host-directories --cut-dirs=$cutdir --passive-ftp -P $mirrordir -i $filelist -B $mirror ./clean_setup.pl -Setup [-- Attachment #3: clean_setup.pl --] [-- Type: text/plain, Size: 13836 bytes --] #! /usr/bin/perl -w require 5.005; # Delete old files from Cygwin setup.exe archive subdirectories # Name of file: cleanup_setup.pl # # Type of file: Perl script # # Author: Michael A. Chase # # Purpose: Clean out setup.exe archive subdirectories. # Syntax: See $HelpText Below #-------------------------- MODIFICATION HISTORY ----------------------------- $VERSION = '1.0303'; # 020430 V. Quetschke Add -writelist # Writes list of missing files to missing.lst. # 020423 M. Chase Add -source, -install, -H options. # Removed directory names from Missing Files list. # $VERSION = '1.0301'; # 020417 M. Chase Stop lc()ing file names during setup.ini parsing. # $VERSION = '1.0300'; # 020416 M. Chase Ignore setup.ini while collecting archive file names. # Remove directories in reverse length order. # $VERSION = '1.0202'; # 020416 M. Chase Properly ignore directories in obsolete file search. # $VERSION = '1.0201'; # 020404 M. Chase Ignore files found more than once in move loop. # Protect against attempts to move files from different # trees over each other in the base tree. # $VERSION = '1.0200'; # 020326 M. Chase Properly handle directory case. # Optionally move archives to base directory tree. # $VERSION = '1.0100'; # 020212 M. Chase Handle multiple setup.ini files and file sources. # $VERSION = '1.0000'; # 010422 M. Chase First draft. use FindBin qw( $RealBin $RealScript ); use File::Basename qw( &basename &dirname &fileparse ); use File::Find; use File::Spec; use File::Path qw( &mkpath ); use File::Copy qw( © &move ); use Getopt::Long; use strict; use integer; use vars qw( $VERSION ); # Initialize options my $bArch = 0; my $bMove = 0; my $bSetup = 0; my $bCurr = 1; my $bPrev = 1; my $bTest = 1; my $bExp = 1; my $bInstall = 1; my $bSource = 0; my $bWrite = 0; my @sHide = (); my @sIgnore = (); my $sDir0 = File::Spec -> rel2abs( "." ); # Syntax description sub usage { my ( $sOpt, $sVal, @sMsg ) = @_; my $sHelpText = <<END_HELP_TEXT; Cleanup Cygwin setup.exe package cache directories syntax: $RealScript [opt] Opt: ($VERSION) -[no]Move = [Don't] Move archive files to base directory tree ($bMove) -[no]Arch = [Don't] Delete obsolete archives and directories ($bArch) -[no]Setup = [Don't] Delete obsolete setup.ini files ($bSetup), forces -Move -[no]curr = [Don't] Report missing [curr] files ($bCurr) -[no]prev = [Don't] Report missing [prev] files ($bPrev) -[no]test = [Don't] Report missing [test] files ($bTest) -[no]exp = [Don't] Report missing [exp] files ($bExp) -[no]install = [Don't] Report missing install archives ($bInstall) -[no]source = [Don't] Report missing source archives ($bSource) -writelist = Write missing files report (missing.lst) to disk ($bWrite) -base dir = Archive cache base directory ($sDir0) -I mask = Ignore files and directories that match mask, multiple allowed -H mask = Ignore packages that match mask, multiple allowed mask is a filename wildcard mask, not a regular expression Arg: END_HELP_TEXT # Balance quotes in here document # ' # " my $nRet = 'help' eq $sOpt ? 0 : 0 + $sVal; select STDERR if $nRet; foreach ( @sMsg, $sHelpText ) { s/\s+$//; print "$_\n"; } exit $nRet; } # Parse command line Getopt::Long::config( qw( no_ignore_case no_auto_abbrev require_order ) ); GetOptions( 'Arch!' => \$bArch, 'Move!' => \$bMove, 'Setup!' => \$bSetup, 'curr!' => \$bCurr, 'prev!' => \$bPrev, 'test!' => \$bTest, 'exp!' => \$bExp, 'install!' => \$bInstall, 'source!' => \$bSource, 'writelist!' => \$bWrite, 'base=s' => \$sDir0, 'Hide|H=s@' => \@sHide, 'Ignore|I=s@' => \@sIgnore, 'help|h' => \&usage ) or usage( 'die', 1 ); $bMove ||= $bSetup; # Don't remove setup files unless also moving archves chdir $sDir0 or usage( 'die', 1, "Can't change directory to $sDir0, $!\n" ); # Provide default values for unset options and parameters # Report arguments $sDir0 = File::Spec -> rel2abs( "." ); $sDir0 =~ s,\\,/,g; my $sOpt = ''; $sOpt .= "\nMoving archives to base directory tree" if $bMove; $sOpt .= "\nDeleting unused files and empty directores" if $bArch; $sOpt .= "\nDeleting obsolete setup.ini files" if $bSetup; $sOpt .= "\nIgnoring files and directories: " . join ' ', map { "'$_'" } @sIgnore if @sIgnore; $sOpt .= "\nIgnoring packages: " . join ' ', map { "'$_'" } @sHide if @sHide; print <<HERE; Base Directory: $sDir0$sOpt HERE # Build file or directory name matcher # Adapted from Recipe 6.10 in Perl Cookbook sub rfMatch { my ( $bDefault, $op, @sMask ) = @_; # @sMask must be a lexical array return sub { return $bDefault; } if 3 > @_; my $sExpr = join " $op\n", map { # Convert file expansion mask to regular expression $sMask[$_] =~ s/\./\\./g; $sMask[$_] =~ s/\?/.?/g; $sMask[$_] =~ s/\*/.*/g; "m:^\$sMask[$_]\$:oi"; } 0 .. $#sMask; my $rfMatch = eval "sub { local \$_ = shift;\nreturn $sExpr; };"; die $@ if $@; return $rfMatch; } local *bHide = rfMatch( 0, '||', @sHide ); local *bIgnore = rfMatch( 0, '||', @sIgnore ); # Find and parse setup.ini files, collect other filenames at the same time my ( %aSetup, $sRel, $sName, %sTarBall, %bDir ); my $wanted = sub { # Skip ., .., and files and directories in ignore list if ( '.' eq $_ || '..' eq $_ ) { $File::Find::prune = 1 if '..' eq $_; return; } if ( bIgnore( $_ ) ) { $File::Find::prune = 1 if -d $_; # Prune if a directory return; } # Handle directory or file $sRel = sRel( $File::Find::dir, $sDir0 ); if ( -d $_ ) { $bDir{sRel( $File::Find::name, $sDir0 )} = 1; } # Remember directory name for possible removal elsif ( "setup.ini" eq $_ ) { # Parse setup.ini my @aSetup; # Get list of files to leave alone, includes subdirectory path # setup-timestamp: 1012849221 # install: latest/bash/bash-2.05-1.tar.gz 576828 # source: latest/bash/bash-2.05-1-src.tar.gz 1792319 my ( $bHide, $sGroup, $sType, $sFile, $sSize, $sVol, $sSubDir, $sName ); my $sSetup = $File::Find::name; open( SETUP, $sSetup ) or usage( 'die', 1, "Can't open $sSetup, $!" ); while ( <SETUP> ) { ( $sType, $sFile, $sSize ) = split /\s+/, $_; if ( 'setup-timestamp:' eq $sType ) { $aSetup[0] = $sFile; # Actually timestamp } elsif ( 'install:' eq $sType ) { next if $bHide; ( $sVol, $sSubDir, $sName ) = File::Spec -> splitpath( $sFile ); $sSubDir = File::Spec -> canonpath( $sSubDir ); $aSetup[1]{$sName} = [ $sGroup, $sSize, $sSubDir ]; } elsif ( 'source:' eq $sType ) { next if $bHide; ( $sVol, $sSubDir, $sName ) = File::Spec -> splitpath( $sFile ); $sSubDir = File::Spec -> canonpath( $sSubDir ); $aSetup[2]{$sName} = [ $sGroup, $sSize, $sSubDir ]; } elsif ( '[' eq substr( $_, 0, 1 ) ) { s/\s+$//; $sGroup = $_; } # ] elsif ( s/^@\s+// ) { s/\s+$//; $bHide = bHide( $_ ); $sGroup = '[curr]'; } } close SETUP; usage( 'die', 1, "Nothing found in $sSetup" ) if 3 != @aSetup || ! $aSetup[0]; $aSetup{$sRel} = \@aSetup; } elsif ( ".tar.bz2" eq substr( $_, -8 ) || ".tar.gz" eq substr( $_, -7 ) ) { # Save name of archive file $sTarBall{$_}{File::Spec -> canonpath( $sRel )} = -s $_; } # Currently ignoring other types of files }; find( $wanted, $sDir0 ); usage( "die", 1, "No setup.ini files found" ) if ! %aSetup; # Pick newest setup.ini my ( $tNewest, $sNewest, @sOldSetup ) = ( 0, "" ); foreach ( sort keys %aSetup ) { if ( $tNewest < $aSetup{$_}[0] ) { $tNewest = $aSetup{$_}[0]; $sNewest = $_; } } print "Newest setup.ini files: ", scalar gmtime( $tNewest ), " GMT\n"; foreach ( sort keys %aSetup ) { print " ", sUnPercent( sRel( $_ ) ), "\n" if $tNewest == $aSetup{$_}[0]; } # Report or remove obsolete setup.ini files foreach ( sort keys %aSetup ) { push @sOldSetup, $_ if $tNewest > $aSetup{$_}[0]; } print "Obsolete setup.ini files:\n" if @sOldSetup; foreach ( @sOldSetup ) { print " ", scalar gmtime( $aSetup{$_}[0] ), " GMT ", sUnPercent( sRel( $_ ) ), "\n"; unlink "$_/setup.ini" or print " *** Can't remove, $!" if $bSetup; } # Check found files against those listed in latest setup.ini my ( @sDir, $sDir, $sFile, @sDup, %sMove, @sRemove, @sWrongSize ); my %aInstall = ( %{$aSetup{$sNewest}[1]} ); my %aSource = ( %{$aSetup{$sNewest}[2]} ); foreach $sName ( sort keys %sTarBall ) { @sDir = sort keys %{$sTarBall{$sName}}; $sDir = $sDir[0]; $sFile = File::Spec -> catfile( $sDir, $sName ); if ( 1 < @sDir ) { # More than one copy of a file was found push @sDup, "$sName: " . join( ", ", map { sUnPercent( $_ ) } @sDir ); } elsif ( exists $aInstall{$sName} ) { if ( $sDir eq $aInstall{$sName}[2] ) {} # Already in right place elsif ( $sTarBall{$sName}{$sDir} != $aInstall{$sName}[1] ) { # Wrong size push @sWrongSize, "$sFile: $sTarBall{$sName}{$sDir} != $aInstall{$sName}[1] in " . sUnPercent( $sDir ); } else { $sMove{$sFile} = File::Spec -> catfile( $aInstall{$sName}[2], $sName ); } } elsif ( exists $aSource{$sName} ) { if ( $sDir eq $aSource{$sName}[2] ) {} # Already in right place elsif ( $sTarBall{$sName}{$sDir} != $aSource{$sName}[1] ) { # Wrong size push @sWrongSize, "$sFile: $sTarBall{$sName}{$sDir} != $aSource{$sName}[1] in " . sUnPercent( $sDir ); } else { $sMove{$sFile} = File::Spec -> catfile( $aSource{$sName}[2], $sName ); } } else { # File not in setup.ini push @sRemove, $sFile; } } # Check for missing files my @sMissing = (); my @sMissingList = (); if ( $bInstall ) { foreach ( keys %aInstall ) { if ( ! exists $sTarBall{$_} && ( $bCurr || '[curr]' ne $aInstall{$_}[0] ) && ( $bPrev || '[prev]' ne $aInstall{$_}[0] ) && ( $bTest || '[test]' ne $aInstall{$_}[0] ) && ( $bExp || '[exp]' ne $aInstall{$_}[0] ) ) { push @sMissing, "$aInstall{$_}[0] $aInstall{$_}[2]/$_" ; # Alternative: $aInstall{$_}[0] . " " . # File::Spec -> catfile( $aInstall{$_}[2], $_ ) } push @sMissingList, "$aInstall{$_}[2]/$_" ; } } } if ( $bSource ) { foreach ( keys %aSource ) { if ( ! exists $sTarBall{$_} && ( $bCurr || '[curr]' ne $aSource{$_}[0] ) && ( $bPrev || '[prev]' ne $aSource{$_}[0] ) && ( $bTest || '[test]' ne $aSource{$_}[0] ) && ( $bExp || '[exp]' ne $aSource{$_}[0] ) ) { push @sMissing, "$aSource{$_}[0] $_" ; # Alternative: $aSource{$_}[0] . " " . # File::Spec -> catfile( $aSource{$_}[2], $_ ) } push @sMissingList, "$aSource{$_}[2]/$_" ; } } } @sMissing = sort @sMissing; # Complain about errors print join( "\n ", "\nDuplicate Files", @sDup ), "\n" if @sDup; print join( "\n ", "\nWrong Size Files", @sWrongSize ), "\n" if @sWrongSize; print join( "\n ", "\nMissing Files:", @sMissing ), "\n" if @sMissing; # Report list of missing files to disk if ( $bWrite ) { @sMissingList = sort @sMissingList; open(REPORT, '>', 'missing.lst'); print( REPORT join( "\n", @sMissingList ),"\n" ) if @sMissingList; close(REPORT); } # Move queued files to base directory tree if ( %sMove ) { print $bMove ? "\n" : "\nNot ", "Moving files to base directory tree\n"; my ( $sFrom, $sTo ); foreach $sFrom ( sort keys %sMove ) { $sTo = $sMove{$sFrom}; $sDir = dirname( $sTo ); print sUnPercent( " $sFrom -> $sDir\n" ); if ( -e $sTo ) { print " *** Target already exists\n"; } elsif ( $bMove ) { mkpath( $sDir ) if ! -d $sDir; move( $sFrom, $sTo ) or print " *** Can't move, $!\n"; } } } # Remove files not listed in setup.ini if ( @sRemove ) { print $bArch ? "\n" : "\nNot ", "Removing files not in setup.ini\n"; foreach $sFile ( @sRemove ) { print sUnPercent( " $sFile\n" ); if ( $bArch ) { unlink( $sFile ) or print " *** Can't remove, $!\n"; } } } # Remove empty directories my ( @g, $bFound ); foreach $sRel ( sort { length $b <=> length $a || $a cmp $b } keys %bDir ) { @g = glob( File::Spec -> catfile( $sRel, "*" ) ); if ( ! @g ) { print $bArch ? "\n" : "\nNot ", "Removing Empty Directories\n" if ! $bFound++; print sUnPercent( " $sRel\n" ); if ( $bArch ) { rmdir $sRel or print " *** Can't rmdir, $!\n"; } } } # Copy latest setup.ini to base directory if ( "." ne $sNewest ) { print sUnPercent( "\nCopying setup.ini from $sNewest to $sDir0\n" ); copy( "$sNewest/setup.ini", "$sDir0/setup.ini" ) or print " *** Can't copy setup.ini, $!\n"; } exit 0; # Produce relative file or directory sub sRel { my ( $sAbs, $sBase ) = @_; $sBase = "." if ! defined $sBase || ! length $sBase; my $sRel = File::Spec -> abs2rel( $sAbs, $sBase ); $sRel =~ s,\\,/,g; $sRel =~ s/^\w://; $sRel = "." if ! length $sRel; return $sRel; } # Convert %xx to characters sub sUnPercent { local ( $_ ) = @_; s/\%([0-9a-f]{2})/chr(hex($1))/gie; return $_; } [-- Attachment #4: Type: text/plain, Size: 214 bytes --] -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Bug reporting: http://cygwin.com/bugs.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/ ^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: Using a real mirroring tool... 2002-04-30 10:09 ` Volker Quetschke @ 2002-04-30 13:43 ` Volker Quetschke 2002-04-30 14:00 ` Michael A Chase 1 sibling, 0 replies; 42+ messages in thread From: Volker Quetschke @ 2002-04-30 13:43 UTC (permalink / raw) Cc: cygwin [-- Attachment #1: Type: text/plain, Size: 753 bytes --] Hi! > Charles Wilson wrote: > >> A few notes: >> ... >> rsync can't merge. > > Yes, but Michael A. Chases clean_setup.pl can! I allowed myself to add a > new option to clean_setup.pl, it exports the list of missing files to a > file. Then you can use wget to get all the missing files. > > I attached the modified clean_setup.pl plus a shell scrip to get the > missing files from a mirror. You have to modify the getmissing.sh to > choose your mirror and your target download directory. The directory and > both files must exist the same directory as your setup.exe. Just a quick > hack, I guarantee nothing! Use the attached getmissing.sh instead, some options for wget were not correct, they left directory listing files. Bye Volker [-- Attachment #2: getmissing.sh --] [-- Type: text/plain, Size: 878 bytes --] #!/bin/bash # Mirror script for cygwin # Uses Michael A. Chase "cleanup_setup.pl" # in a modified version (V1.0303) # and wget # getmissing.sh and cleanup_setup.pl must reside in the same directory # as setup.exe # Files to download filelist="missing.lst" # Choose your mirror with directory of setup.ini #mirror="ftp://ftp.inf.tu-dresden.de/software/windows/cygwin32/" # Cut directories 1 2 3 mirror="ftp://mirrors.rcn.net/mirrors/sources.redhat.com/cygwin/" # Cut directories 1 2 3 cutdir=3 # Name your download directory mirrordir="_mymirror" wget -r -N --no-host-directories --cut-dirs=$cutdir --passive-ftp -P $mirrordir ${mirror}setup.ini ./clean_setup.pl -writelist wget -r -N --no-host-directories --cut-dirs=$cutdir --passive-ftp -P $mirrordir -i $filelist -B $mirror ./clean_setup.pl -Arch -Setup [-- Attachment #3: Type: text/plain, Size: 214 bytes --] -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Bug reporting: http://cygwin.com/bugs.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/ ^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: Using a real mirroring tool... 2002-04-30 10:09 ` Volker Quetschke 2002-04-30 13:43 ` Volker Quetschke @ 2002-04-30 14:00 ` Michael A Chase 1 sibling, 0 replies; 42+ messages in thread From: Michael A Chase @ 2002-04-30 14:00 UTC (permalink / raw) To: Volker Quetschke, cygwin On Tue, 30 Apr 2002 16:00:52 +0200 Volker Quetschke <quetschke@scytek.de> wrote: > I attached the modified clean_setup.pl plus a shell scrip to get the > missing files from a mirror. You have to modify the getmissing.sh to > choose your mirror and your target download directory. The directory and > both files must exist the same directory as your setup.exe. Just a quick > hack, I guarantee nothing! You started with a little bit old version of the script. I have uploaded a new version 1.0401 which uses a '-misslist file' option to write the list to the specified file. The new version also allows you to exclude packages that you are not interested in with the '-H mask' option. This will allow your wget with the list more selective. Some of the other options have been changed as well, so be sure to run with '-h' before using the script. -- Mac :}) ** I normally forward private questions to the appropriate mail list. ** Ask Smarter: http://www.tuxedo.org/~esr/faqs/smart-questions.html Give a hobbit a fish and he eats fish for a day. Give a hobbit a ring and he eats fish for an age. -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Bug reporting: http://cygwin.com/bugs.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/ ^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: setup 2.194.2.24: Bug (?) in downloading from internet 2002-04-28 4:47 setup 2.194.2.24: Bug (?) in downloading from internet Robert Collins 2002-04-28 8:08 ` Sam Edge 2002-04-28 15:04 ` Gary R. Van Sickle @ 2002-04-29 4:07 ` Max Bowsher 2002-04-29 4:24 ` Robert Collins 2 siblings, 1 reply; 42+ messages in thread From: Max Bowsher @ 2002-04-29 4:07 UTC (permalink / raw) To: cygwin Robert Collins <robert.collins@itdomain.com.au> wrote: >> -----Original Message----- >> From: Sam Edge [mailto:sam_edgeZZZ@hotmail.com] >> Sent: Sunday, April 28, 2002 8:17 PM >> Every version of setup.exe I've used exhibits this annoying >> behaviour but them I'm new here. >> >> I agree with Daniele. I install different subsets of the >> packages on several machines. I also like to keep a copy of >> the most up-to-date versions of packages that I don't >> currently use on any of them, in case I need them where I've >> no (or poor) access to the 'Net. > > So use a mirroring tool! Setup.exe is -not- designed for this. If you > want a setup.ini parsing mirroring tool, take me up on my offer to help > someone leverage the setup.ini codebase to build a mirroring tool. > > Rob But setup.exe has operated in the correct way in the past! Why write a new tool, when fixing a bug in the current one would do all that needs to be done? NB: I'm not saying 'go fix that now'. When (if? ;-) I find time to understand the setup code, I will do this, if no one has beaten me to it. Max. -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Bug reporting: http://cygwin.com/bugs.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/ ^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: setup 2.194.2.24: Bug (?) in downloading from internet 2002-04-29 4:07 ` setup 2.194.2.24: Bug (?) in downloading from internet Max Bowsher @ 2002-04-29 4:24 ` Robert Collins 0 siblings, 0 replies; 42+ messages in thread From: Robert Collins @ 2002-04-29 4:24 UTC (permalink / raw) To: Max Bowsher, cygwin === ----- Original Message ----- From: "Max Bowsher" <maxb@ukf.net> To: <cygwin@cygwin.com> > But setup.exe has operated in the correct way in the past! Why write a new tool, > when fixing a bug in the current one would do all that needs to be done? Because the current tool is more complex than it should be, and a new tool can be accurately target. The current tool can then be simplified somewhat (i.e. download only could go away if we had a (non cygwin linked) mirroring tool. Or a variation of this. > NB: I'm > not saying 'go fix that now'. When (if? ;-) I find time to understand the setup > code, I will do this, if no one has beaten me to it. Thanks. Rob -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Bug reporting: http://cygwin.com/bugs.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/ ^ permalink raw reply [flat|nested] 42+ messages in thread
* RE: setup 2.194.2.24: Bug (?) in downloading from internet @ 2002-04-29 2:35 Robert Collins 0 siblings, 0 replies; 42+ messages in thread From: Robert Collins @ 2002-04-29 2:35 UTC (permalink / raw) To: cygwin > -----Original Message----- > From: Christopher Faylor [mailto:cgf@redhat.com] > Sent: Monday, April 29, 2002 8:30 AM > However, I think I may have been responsible for setup's > current behavior of basing what is downloaded on what is > installed. Even if I was not, though, I think that the > explanations about why some people need this are rather convincing. Yup. > If I have this right then even removing download from > internet does not eliminate this problem. I don't actually like the idea of removing download from the net. It's such a useful get-this-for-me capacity that I feel it can stay. It's the redownload aspect that s***s me. Rob -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Bug reporting: http://cygwin.com/bugs.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/ ^ permalink raw reply [flat|nested] 42+ messages in thread
* RE: setup 2.194.2.24: Bug (?) in downloading from internet @ 2002-04-29 2:26 Robert Collins 2002-04-29 5:42 ` Sam Edge 0 siblings, 1 reply; 42+ messages in thread From: Robert Collins @ 2002-04-29 2:26 UTC (permalink / raw) To: cygwin > -----Original Message----- > From: Christopher Faylor [mailto:cgf@redhat.com] > Sent: Monday, April 29, 2002 3:07 PM > >That's what the discussion has been about for the last week or so. > >After lots of requests for this functionality, the position > eventually > >hardened to "it's not a download tool". > > Right. But it started with my suggesting this behavior was a > bug and (I > thought) Robert agreeing. So, I thought it was time to chime > in again. The *redownloading* is a bug, and will get fixed. The 'I want to mirror non-installed packages by using download' is not. > The bottom line is that if we have to tell someone to use > wget or rsync just to do a normal install, something is wrong. These folk aren't doing a normal install. They want to use download mode, and then *not* install what was downloaded. A 'normal' install - download and install - works fine, no problems. > It seems like this problem would be trivially fixed by not > attempting to redownload something that has already been downloaded. I've also expressed this opinion. But... redownloading is a current 'feature'. It's one I don't think we need, and that we can get rid of - which will correct the problem. A reference for this is http://sources.redhat.com/ml/cygwin-apps/2001-11/msg00271.html. Rob -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Bug reporting: http://cygwin.com/bugs.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/ ^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: setup 2.194.2.24: Bug (?) in downloading from internet 2002-04-29 2:26 Robert Collins @ 2002-04-29 5:42 ` Sam Edge 2002-04-29 5:49 ` Robert Collins 0 siblings, 1 reply; 42+ messages in thread From: Sam Edge @ 2002-04-29 5:42 UTC (permalink / raw) To: cygwin Robert Collins wrote in <FC169E059D1A0442A04C40F86D9BA7600C5F52@itdomain003.itdomain.net.au> in gmane.os.cygwin on Mon, 29 Apr 2002 17:47:16 +1000: > The *redownloading* is a bug, and will get fixed. Okay. Good. > The 'I want to mirror > > non-installed packages by using download' is not. Actually, the re-download fix would go 90% of the way to doing this anyway, since you could just reselect everything you want again and again and it'd eventually have the lot even on a slow link. > A 'normal' install - download and install - works fine, no problems. Read my and Christopher Faylor's posts especially <20020429050712.GC27298@redhat.com> where we explain where for some people download and install with the current version of setup.exe does not work fine. Even a single package such as the XFree86-fnts one can be a trial on dial-up connections, taking the best part of an hour. A clean minimum install of the Cygwin environment takes about the same, I think. A model that didn't allow multiple download sessions before a single install one would effectively bar many dial-up users from installing Cygwin and/or XFree86 at all. The selective mirroring and local FTP server route would simply be too much hassle for many just to evaluate something. > A reference for this is > http://sources.redhat.com/ml/cygwin-apps/2001-11/msg00271.html. Ah. Exactly. :-) So is it worth my while looking at the code with an eye to providing a patch for this (specifically the checkbox option) or is it work in progress elsewhere? -- Sam Edge -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Bug reporting: http://cygwin.com/bugs.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/ ^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: setup 2.194.2.24: Bug (?) in downloading from internet 2002-04-29 5:42 ` Sam Edge @ 2002-04-29 5:49 ` Robert Collins 2002-04-29 7:46 ` Sam Edge 2002-04-29 8:27 ` Christopher Faylor 0 siblings, 2 replies; 42+ messages in thread From: Robert Collins @ 2002-04-29 5:49 UTC (permalink / raw) To: Sam Edge, cygwin ----- Original Message ----- From: "Sam Edge" <sam_edgeZZZ@hotmail.com> To: <cygwin@cygwin.com> > > A 'normal' install - download and install - works fine, no problems. > > Read my and Christopher Faylor's posts especially > <20020429050712.GC27298@redhat.com> where we explain where for some > people download and install with the current version of setup.exe does > not work fine. Actually, the current functionality wil do the trick, as long as you install what you have downloaded. Combined with multi-mirror failure, unless you have a *really* bad phone line connection, you should be able to successfully install bit by bit. Notwithstanding that I do understand and (mostly) agree with whats been said though. > > A reference for this is > > http://sources.redhat.com/ml/cygwin-apps/2001-11/msg00271.html. > > Ah. Exactly. :-) > > So is it worth my while looking at the code with an eye to providing a > patch for this (specifically the checkbox option) or is it work in > progress elsewhere? Work is in progress. You (and everyone who's jumped in on this) might like to read Havoc Penningtons recent paper http://www106.pair.com/rhp/free-software-ui.html. What follows is not a critique of your suggestion, just an explanation as to why the checkbox is not appropriate IMO. Before reading the following, be sure to have read Havoc's paper above. We have a problem with setup.exe. * Corrupt package files are not always detected and automatically removed. * That's it. "Redownloading" is a workaround for that. (If in fact there are -other- reasons for "redownloading" then I will revise that statement.) Checkbox's, command line options, and 'special case code' and the redownload itself are all kludges around fixing the key problem. And thats why this has become a sudden issue. I *unknowningly* removed a bit of special case code that interfaced between good logic and the redownload capability to make it seem 'correct' to the user. (BTW: Chris, it was NOT your fault. Really.) The real solution is to positively identify corrupt archives and transparently remove them (perhaps asking the user whether we should delete, backup, or skip over the package). Rob -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Bug reporting: http://cygwin.com/bugs.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/ ^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: setup 2.194.2.24: Bug (?) in downloading from internet 2002-04-29 5:49 ` Robert Collins @ 2002-04-29 7:46 ` Sam Edge 2002-04-30 1:58 ` Gary R. Van Sickle 2002-04-29 8:27 ` Christopher Faylor 1 sibling, 1 reply; 42+ messages in thread From: Sam Edge @ 2002-04-29 7:46 UTC (permalink / raw) To: cygwin Robert Collins wrote in <016301c1ef7b$4d769f00$0200a8c0@lifelesswks> in gmane.os.cygwin on Mon, 29 Apr 2002 22:42:18 +1000: > Actually, the current functionality wil do the trick, as long as you > install what you have downloaded. Hmmm. I've just tried using "install from Internet" with bzip2, sharutils, unzip and zip, on a machine that doesn't have them installed and killing the connection after bzip2 has downloaded. When it says "download incomplete, try again" I say "No" and it attempts to do the install. I'm then getting "Can't open (null) for reading" errors during the attempted installation of sharutils (the sharutils sub-directory has been created but is empty) and a "installation incomplete" message box at the end but it /has/ installed bzip2. It's pretty ugly but it does seem to be working and if I reselect bzip2 when I try again it isn't attempting to download it again so yes, you could get there even on a slow link. (Not exactly good UI design though!) ;-) (While I'm here; BUG REPORT: When I get the "Download incomplete. Retry? Yes/No" message box in "install from Internet" mode and choose "No" without it then going on to install any packages I then get a final message box with "Download incomplete. Retry? Yes/No" again but only an "OK" button.) > http://www106.pair.com/rhp/free-software-ui.html. LOL. Yes. I think I can agree with most of this, especially the lack of UI consistency and the problems with debugging over-featured design. > Checkbox's, command line options, and 'special case code' and the > redownload itself are all kludges around fixing the key problem. I agree. The best solution is to make a bulletproof method of detecting whether the local copies are uncorrupted and to keep the UI uncluttered. But from the general tenor of the discussion at the beginning I was under the impression that there was some philosophical or technical reason why this wasn't appropriate. A manual option would therefore be the next best thing. Sorry if I got the wrong end of the stick. > The real solution is to positively identify corrupt archives and > transparently remove them (perhaps asking the user whether we should > delete, backup, or skip over the package). Yup. Agreed. Thesis, antithesis, synthesis! :-) Can I also suggest adding reget behaviour at some point, if it can be done reliably. Getting nine tenths of the way into a 15MB package when your line goes down is /very/ annoying! ;-( Finally, I stand by my wish to retain a separate download and installation invocation option to allow them to occur while logged on to NT/2k/XP with different permissions, again out of paranoia. Thanks. -- Sam Edge -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Bug reporting: http://cygwin.com/bugs.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/ ^ permalink raw reply [flat|nested] 42+ messages in thread
* RE: setup 2.194.2.24: Bug (?) in downloading from internet 2002-04-29 7:46 ` Sam Edge @ 2002-04-30 1:58 ` Gary R. Van Sickle 0 siblings, 0 replies; 42+ messages in thread From: Gary R. Van Sickle @ 2002-04-30 1:58 UTC (permalink / raw) To: cygwin > Robert Collins wrote in <016301c1ef7b$4d769f00$0200a8c0@lifelesswks> > in gmane.os.cygwin on Mon, 29 Apr 2002 22:42:18 +1000: > > > Actually, the current functionality wil do the trick, as long as you > > install what you have downloaded. > > Hmmm. I've just tried using "install from Internet" with bzip2, > sharutils, unzip and zip, on a machine that doesn't have them > installed and killing the connection after bzip2 has downloaded. > > When it says "download incomplete, try again" I say "No" and it > attempts to do the install. > > I'm then getting "Can't open (null) for reading" errors during the > attempted installation of sharutils (the sharutils sub-directory has > been created but is empty) and a "installation incomplete" message box > at the end but it /has/ installed bzip2. > > It's pretty ugly but it does seem to be working and if I reselect > bzip2 when I try again it isn't attempting to download it again so > yes, you could get there even on a slow link. (Not exactly good UI > design though!) ;-) > My ears are burning... ;-) -- Gary R. Van Sickle Brewer. Patriot. -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Bug reporting: http://cygwin.com/bugs.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/ ^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: setup 2.194.2.24: Bug (?) in downloading from internet 2002-04-29 5:49 ` Robert Collins 2002-04-29 7:46 ` Sam Edge @ 2002-04-29 8:27 ` Christopher Faylor 1 sibling, 0 replies; 42+ messages in thread From: Christopher Faylor @ 2002-04-29 8:27 UTC (permalink / raw) To: cygwin On Mon, Apr 29, 2002 at 10:42:18PM +1000, Robert Collins wrote: >Checkbox's, command line options, and 'special case code' and the >redownload itself are all kludges around fixing the key problem. That's for sure. The design that I proposed for setup's UI eliminated most of these. cgf -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Bug reporting: http://cygwin.com/bugs.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/ ^ permalink raw reply [flat|nested] 42+ messages in thread
* RE: setup 2.194.2.24: Bug (?) in downloading from internet @ 2002-04-28 22:34 Alec Mihailovs 2002-04-28 23:51 ` Charles Wilson 0 siblings, 1 reply; 42+ messages in thread From: Alec Mihailovs @ 2002-04-28 22:34 UTC (permalink / raw) To: Cygwin (E-mail) I have a perfect installation program for MiKTeX made by Christian Schenk, see http://www.miktex.org/ You people seem to be not able to make anything even close to it. Why don't you just copy it and use for cygwin instead of inventing a wheel yourself? Best wishes, Alec Mihailovs http://webpages.shepherd.edu/amihailo/ -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Bug reporting: http://cygwin.com/bugs.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/ ^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: setup 2.194.2.24: Bug (?) in downloading from internet 2002-04-28 22:34 Alec Mihailovs @ 2002-04-28 23:51 ` Charles Wilson 0 siblings, 0 replies; 42+ messages in thread From: Charles Wilson @ 2002-04-28 23:51 UTC (permalink / raw) To: cygwin Alec Mihailovs wrote: > I have a perfect installation program for MiKTeX made > by Christian Schenk, see http://www.miktex.org/ > > You people seem to be not able to make anything even close to it. > Why don't you just copy it and use for cygwin instead of inventing > a wheel yourself? I'll ignore your rudeness, and pretend that you actually are trying to help. I don't believe it for a minute, but I'm a decent hobbyist actor... 1) We are providing an entire development environment, that includes tex as just one small part. MikTeX, which is a fine program that I use for all my document creation, is MUCH more narrowly focused -- includes less than 50 executables -- and thus places far fewer demands on its installer. To be sure, there are some similarities: you might be tempted to draw a parallel between MikTeX's packages, and Cygwin's package system; but that parallel is false. The caption2 package and the fancyhdr package are both just modification scripts for the same engine; they both help make pretty documents. The cvs package and the tetex-beta package are quite different; one is a source revision control program and the other includes all of MikTeX's functionality. It's a bit more complicated to install cygwin stuff. 2) We predate the MikTeX installer by quite some time. 3) Christian provides CVS access to his source code -- but the installer is restricted in use, and is not licensed under the GPL. We cannot take it and adapt it without his permission. Call us old-fashioned, but we prefer truly free (as in speech) software. 4) To claim that MikTeX's installation program is "perfect" -- e.g. bug free -- is utter lunacy. Check the MikTeX forums... 5) Christian's installer, I believe, is built using MSVC and seems to actually leverage the MS installer code from Microsoft's installer SDK. That's a fine idea -- but is probably off limits in a truly free (speech) project. We require that our tools -- including the installer -- are built using free (speech) tools. Now, perhaps you could make the argument that this proves that free tools aren't as good as the MS stuff; I disagree and believe it's just another example of MS leveraging an existing monopoly (OS) to take over another market (SoftwareInstallers). In any case, the "free software vs. microsoft" debate belongs somewhere other than the main cygwin mailing list. > Best wishes, > Alec Mihailovs Somehow, I doubt that you really have the "best wishes" of the cygwin project at heart, Dr. Mihailovs, since you began your message accusing us of incompetence. --Charles Wilson -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Bug reporting: http://cygwin.com/bugs.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/ ^ permalink raw reply [flat|nested] 42+ messages in thread
* RE: setup 2.194.2.24: Bug (?) in downloading from internet @ 2002-04-28 10:15 Robert Collins 0 siblings, 0 replies; 42+ messages in thread From: Robert Collins @ 2002-04-28 10:15 UTC (permalink / raw) To: Kurt Roeckx, cygwin Kurt, you've jumped in half way through a discussion. I've addressed every point you made, as have other people, in various emails in this list over the last few months. Rob -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Bug reporting: http://cygwin.com/bugs.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/ ^ permalink raw reply [flat|nested] 42+ messages in thread
* RE: setup 2.194.2.24: Bug (?) in downloading from internet @ 2002-04-28 8:14 Robert Collins 2002-04-28 9:56 ` Kurt Roeckx 0 siblings, 1 reply; 42+ messages in thread From: Robert Collins @ 2002-04-28 8:14 UTC (permalink / raw) To: Sam Edge, cygwin > -----Original Message----- > From: Sam Edge [mailto:sam_edgeZZZ@hotmail.com] > Sent: Sunday, April 28, 2002 9:32 PM > > So use a mirroring tool! Setup.exe is -not- designed for this. > > Surely if setup.exe isn't designed with at least a nod to > this way of working why does it have "download from Internet" > and "install from local directory" options, instead of just > instisting on connecting to a server every time? It's designed to allow an off-line mode, as opposed to having spare copies of potentially desired pacakges around. The redownloading behaviour is a result of a kludge to allow deliberate redownloading of packages in 'download only' mode. If we remove that, then packages won't get downloaded again and again and again - but setup will be no more suited to being a mirroring tool. The cache directory is shareable with multiple cygwin installs (say over a SMB share). So yes, there is a nod towards what you do, but that's abou tit. > > If you > > want a setup.ini parsing mirroring tool, take me up on my offer to > > help someone leverage the setup.ini codebase to build a mirroring > > tool. > > Although I've not looked at it, setup.exe must already have > most of the code required to operate in the way we'd like, surely? It has a significant amount of code, that is getting more flexible and paramterised all the time. If you look at http://sources.redhat.com/cygwin-apps/ you can access the setup.exe developers page. It's a but rough at the moment, but it does have the CVS location amongst other things. > If you tell me the module name(s) for setup.exe and any > documentation about setup.ini and the local and FTP/HTTP > server directory structures I'd certainly be interested to > download them from the CVS server and take a look. Setup.ini is documented at http://www.cygwin.com/setup.html. There is no ftp/http data structure - setup.ini determines everything. Setup's multiple setup.ini capability allows for advanced merging and per-organisation tweaking. The differences between setup and a mirroring tool include: * All the install code isn't needed. * Attention to local packages isn't needed. * Persistent selection of packages irrespective of install status should be present. * command line driven may be useful? Anyway, there's some notes, have a look, and we can talk further. Rob -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Bug reporting: http://cygwin.com/bugs.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/ ^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: setup 2.194.2.24: Bug (?) in downloading from internet 2002-04-28 8:14 Robert Collins @ 2002-04-28 9:56 ` Kurt Roeckx 0 siblings, 0 replies; 42+ messages in thread From: Kurt Roeckx @ 2002-04-28 9:56 UTC (permalink / raw) To: cygwin On Sun, Apr 28, 2002 at 09:47:14PM +1000, Robert Collins wrote: > > -----Original Message----- > > From: Sam Edge [mailto:sam_edgeZZZ@hotmail.com] > > Sent: Sunday, April 28, 2002 9:32 PM > > > > So use a mirroring tool! Setup.exe is -not- designed for this. > > > > Surely if setup.exe isn't designed with at least a nod to > > this way of working why does it have "download from Internet" > > and "install from local directory" options, instead of just > > instisting on connecting to a server every time? > > It's designed to allow an off-line mode, as opposed to having spare > copies of potentially desired pacakges around. The redownloading > behaviour is a result of a kludge to allow deliberate redownloading of > packages in 'download only' mode. Why is the redownloading a "good thing" in case of download only mode? If I want to redownload something, I could just remove it and then say to download it again. > If we remove that, then packages won't > get downloaded again and again and again - but setup will be no more > suited to being a mirroring tool. The "no more suited to being a mirroring tool" really confuses me! How is redownloading good for a mirror tool? The only case I can see to redownload a certain file is when the time of the file has changed (or the size). If it *looks* the same, don't download it again. The only file that really should need that, afaik, is setup.ini, and I think you download that anyway? > > > If you > > > want a setup.ini parsing mirroring tool, take me up on my offer to > > > help someone leverage the setup.ini codebase to build a mirroring > > > tool. Why do you even need to parse setup.ini to be able to mirror something? Using something as wget should work, I just didn't try it yet. Kurt -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Bug reporting: http://cygwin.com/bugs.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/ ^ permalink raw reply [flat|nested] 42+ messages in thread
* setup 2.194.2.24: Bug (?) in downloading from internet @ 2002-04-28 3:17 Daniele Mezzetti 2002-04-28 4:31 ` Sam Edge 0 siblings, 1 reply; 42+ messages in thread From: Daniele Mezzetti @ 2002-04-28 3:17 UTC (permalink / raw) To: cygwin (I hope this will show up in the correct thread.. excuse me if not) refers to msg01259.html msg01481.html After dutifully reading the mailing list discussion on the subject, I would like to point out that this behavior - redownloading uninstalled packages - was absent in previous versions of setup. In my humble opinion, the downloaded packages and the installed packages should be treated independently by setup for the simple reason that you may download a set of packages for installing on multiple machines, and install a subset of it on the downloading machine. In other words should be: download from internet: compares repository with packages install from internet: compares repository with installation install from local: comapres packages with installation that is, previous behaviour. -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Bug reporting: http://cygwin.com/bugs.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/ ^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: setup 2.194.2.24: Bug (?) in downloading from internet 2002-04-28 3:17 Daniele Mezzetti @ 2002-04-28 4:31 ` Sam Edge 0 siblings, 0 replies; 42+ messages in thread From: Sam Edge @ 2002-04-28 4:31 UTC (permalink / raw) To: cygwin Daniele wrote in <Pine.A41.3.96.1020428003038.145792A-100000@egeo.unipg.it> in gmane.os.cygwin on Sun, 28 Apr 2002 00:44:25 +0200 (MDT): > After dutifully reading the mailing list discussion on the subject, > I would like to point out that this behavior - redownloading uninstalled > packages - was absent > in previous versions of setup. In my humble opinion, the downloaded > packages and the installed packages should be treated independently by > setup for the simple reason that you may download a set of packages for > installing on multiple machines, and install a subset of it on the > downloading machine. In other words should be: > download from internet: compares repository with packages > install from internet: compares repository with installation > install from local: comapres packages with installation > that is, previous behaviour. Every version of setup.exe I've used exhibits this annoying behaviour but them I'm new here. I agree with Daniele. I install different subsets of the packages on several machines. I also like to keep a copy of the most up-to-date versions of packages that I don't currently use on any of them, in case I need them where I've no (or poor) access to the 'Net. Doing this is something of a headache and requires manual checking in the local package directory (now even more difficult because of its fragmented nature even when mitigated by clean_setup.pl - thanks Michael) and in the setup.exe list. -- Sam Edge -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Bug reporting: http://cygwin.com/bugs.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/ ^ permalink raw reply [flat|nested] 42+ messages in thread
* RE: setup 2.194.2.24: Bug (?) in downloading from internet @ 2002-04-26 8:34 Robert Collins 0 siblings, 0 replies; 42+ messages in thread From: Robert Collins @ 2002-04-26 8:34 UTC (permalink / raw) To: Alexei Lioubimov, cygwin, Larry Hall (RFK Partners, Inc) > -----Original Message----- > From: Alexei Lioubimov [mailto:e-complex@mtu-net.ru] > Sent: Saturday, April 27, 2002 12:32 AM > To: Robert Collins; cygwin@cygwin.com; Larry Hall (RFK Partners, Inc) > Subject: Re: setup 2.194.2.24: Bug (?) in downloading from internet > > > Hello Robert, > Excuse me, but my English is not so good, and i didn't > understand one sentence from your answer: "...which is > something I'm seriously considering nuking." What does the > word "nuking" mean? Removing/destroying/eliminating. With a sense of anger/frustration. > What is the re-download functionality you are speaking about? When you select 'download from internet', if you select a package that is already cached, it deletes the cached copy and downloads the file again. As it doesn't keep corrupt files, I see no value in this - and it is the reason you see setup downloading stuff again and again. Rob -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Bug reporting: http://cygwin.com/bugs.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/ ^ permalink raw reply [flat|nested] 42+ messages in thread
* RE: setup 2.194.2.24: Bug (?) in downloading from internet
@ 2002-04-26 7:10 Robert Collins
2002-04-26 7:57 ` Alexei Lioubimov
0 siblings, 1 reply; 42+ messages in thread
From: Robert Collins @ 2002-04-26 7:10 UTC (permalink / raw)
To: Alexei Lioubimov, cygwin, Larry Hall (RFK Partners, Inc)
> -----Original Message-----
> From: Alexei Lioubimov [mailto:e-complex@mtu-net.ru]
> Sent: Friday, April 26, 2002 11:44 PM
> To: cygwin@cygwin.com; Larry Hall (RFK Partners, Inc)
> Subject: Re: setup 2.194.2.24: Bug (?) in downloading from internet
>
>
> Hello Larry,
> I know, that this problem has already been reported
> (msg01259.html). And i've read your answer to use some
> mirroring tool (wget) instead before i sent my message to the
> forum. It seemes to me that you are wrong proposing such
> solution and it is not the answer to the question.
The solution you propose is also less than satisfactory. It means that
setup won't be looking at what you need, but rather at what you have
'available'. The real cause of the problem is setup's re-download
functionality, which is something I'm seriously considering nuking.
Rob
--
Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Bug reporting: http://cygwin.com/bugs.html
Documentation: http://cygwin.com/docs.html
FAQ: http://cygwin.com/faq/
^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: setup 2.194.2.24: Bug (?) in downloading from internet 2002-04-26 7:10 Robert Collins @ 2002-04-26 7:57 ` Alexei Lioubimov 0 siblings, 0 replies; 42+ messages in thread From: Alexei Lioubimov @ 2002-04-26 7:57 UTC (permalink / raw) To: Robert Collins, cygwin, Larry Hall (RFK Partners, Inc) Hello Robert, Excuse me, but my English is not so good, and i didn't understand one sentence from your answer: "...which is something I'm seriously considering nuking." What does the word "nuking" mean? What is the re-download functionality you are speaking about? Alexei Lioubimov ----- Original Message ----- From: "Robert Collins" <robert.collins@itdomain.com.au> To: "Alexei Lioubimov" <e-complex@mtu-net.ru>; <cygwin@cygwin.com>; "Larry Hall (RFK Partners, Inc)" <lhall@rfk.com> Sent: Friday, April 26, 2002 5:57 PM Subject: RE: setup 2.194.2.24: Bug (?) in downloading from internet > -----Original Message----- > From: Alexei Lioubimov [mailto:e-complex@mtu-net.ru] > Sent: Friday, April 26, 2002 11:44 PM > To: cygwin@cygwin.com; Larry Hall (RFK Partners, Inc) > Subject: Re: setup 2.194.2.24: Bug (?) in downloading from internet > > > Hello Larry, > I know, that this problem has already been reported > (msg01259.html). And i've read your answer to use some > mirroring tool (wget) instead before i sent my message to the > forum. It seemes to me that you are wrong proposing such > solution and it is not the answer to the question. The solution you propose is also less than satisfactory. It means that setup won't be looking at what you need, but rather at what you have 'available'. The real cause of the problem is setup's re-download functionality, which is something I'm seriously considering nuking. Rob -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Bug reporting: http://cygwin.com/bugs.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/ ^ permalink raw reply [flat|nested] 42+ messages in thread
* setup 2.194.2.24: Bug (?) in downloading from internet @ 2002-04-25 8:55 Alexei Lioubimov 2002-04-25 11:00 ` Larry Hall (RFK Partners, Inc) 0 siblings, 1 reply; 42+ messages in thread From: Alexei Lioubimov @ 2002-04-25 8:55 UTC (permalink / raw) To: cygwin Hello, I have a problem using setup.exe. Here is the description setup.exe: 2.194.2.24; mode: Download from Inernet; keyword: "skip" flag; The problem: if i've downloaded some tarballs and didn't install them - then during the following updates the SETUP program doesn't report me about that. It simply marks those files as "skip" by default and allows me to change this to the current version. But i may have already downloaded the stuff... So i caught myself downloading the same tarballs twice or triple times. If you take into account the slow speed of dial-up connection, that i have, then you can imagine what i'm feeling. The suggestion: may be it's possible force SETUP consider "current" be "downloaded" but not "installed" stuff during "Download from Internet" process? Thank you, Alexei Lioubimov -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Bug reporting: http://cygwin.com/bugs.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/ ^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: setup 2.194.2.24: Bug (?) in downloading from internet 2002-04-25 8:55 Alexei Lioubimov @ 2002-04-25 11:00 ` Larry Hall (RFK Partners, Inc) 2002-04-26 7:03 ` Alexei Lioubimov 0 siblings, 1 reply; 42+ messages in thread From: Larry Hall (RFK Partners, Inc) @ 2002-04-25 11:00 UTC (permalink / raw) To: Alexei Lioubimov, cygwin At 11:08 AM 4/25/2002, Alexei Lioubimov wrote: >Hello, >I have a problem using setup.exe. Here is the description > >setup.exe: 2.194.2.24; >mode: Download from Inernet; >keyword: "skip" flag; > >The problem: if i've downloaded some tarballs and didn't install them - then >during the following updates the SETUP program doesn't report me about that. >It simply marks those files as "skip" by default and allows me to change >this to the current version. But i may have already downloaded the stuff... >So i caught myself downloading the same tarballs twice or triple times. If >you take into account the slow speed of dial-up connection, that i have, >then you can imagine what i'm feeling. > >The suggestion: may be it's possible force SETUP consider "current" be >"downloaded" but not "installed" stuff during "Download from Internet" >process? This has already been reported. If you're interested in the discussion on this subject, please see the email list archives. Larry Hall lhall@rfk.com RFK Partners, Inc. http://www.rfk.com 838 Washington Street (508) 893-9779 - RFK Office Holliston, MA 01746 (508) 893-9889 - FAX -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Bug reporting: http://cygwin.com/bugs.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/ ^ permalink raw reply [flat|nested] 42+ messages in thread
* Re: setup 2.194.2.24: Bug (?) in downloading from internet 2002-04-25 11:00 ` Larry Hall (RFK Partners, Inc) @ 2002-04-26 7:03 ` Alexei Lioubimov 0 siblings, 0 replies; 42+ messages in thread From: Alexei Lioubimov @ 2002-04-26 7:03 UTC (permalink / raw) To: cygwin, Larry Hall (RFK Partners, Inc) Hello Larry, I know, that this problem has already been reported (msg01259.html). And i've read your answer to use some mirroring tool (wget) instead before i sent my message to the forum. It seemes to me that you are wrong proposing such solution and it is not the answer to the question. Of course i can download everything "by hand" - without the SETUP program, but why should i? SETUP "knows" more about the mirror sites, directory structure, package versions etc. I'm only asking about one small bug in SETUP -- since it doesn't show already downloaded (but NOT installed)files, it's a bit confusing. Alexei Lioubimov ----- Original Message ----- From: "Larry Hall (RFK Partners, Inc)" <lhall@rfk.com> To: "Alexei Lioubimov" <e-complex@mtu-net.ru>; <cygwin@cygwin.com> Sent: Thursday, April 25, 2002 7:24 PM Subject: Re: setup 2.194.2.24: Bug (?) in downloading from internet > At 11:08 AM 4/25/2002, Alexei Lioubimov wrote: > >Hello, > >I have a problem using setup.exe. Here is the description > > > >setup.exe: 2.194.2.24; > >mode: Download from Inernet; > >keyword: "skip" flag; > > > >The problem: if i've downloaded some tarballs and didn't install them - then > >during the following updates the SETUP program doesn't report me about that. > >It simply marks those files as "skip" by default and allows me to change > >this to the current version. But i may have already downloaded the stuff... > >So i caught myself downloading the same tarballs twice or triple times. If > >you take into account the slow speed of dial-up connection, that i have, > >then you can imagine what i'm feeling. > > > >The suggestion: may be it's possible force SETUP consider "current" be > >"downloaded" but not "installed" stuff during "Download from Internet" > >process? > > > > This has already been reported. If you're interested in the discussion on > this subject, please see the email list archives. > > > > Larry Hall lhall@rfk.com > RFK Partners, Inc. http://www.rfk.com > 838 Washington Street (508) 893-9779 - RFK Office > Holliston, MA 01746 (508) 893-9889 - FAX > > -- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Bug reporting: http://cygwin.com/bugs.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/ ^ permalink raw reply [flat|nested] 42+ messages in thread
end of thread, other threads:[~2002-04-30 17:38 UTC | newest] Thread overview: 42+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2002-04-28 4:47 setup 2.194.2.24: Bug (?) in downloading from internet Robert Collins 2002-04-28 8:08 ` Sam Edge 2002-04-28 15:04 ` Gary R. Van Sickle 2002-04-28 15:09 ` Sam Edge 2002-04-28 15:30 ` Charles Wilson 2002-04-28 16:05 ` Sam Edge 2002-04-28 16:05 ` Christopher Faylor 2002-04-28 16:09 ` Sam Edge 2002-04-28 19:08 ` Michael A Chase 2002-04-29 0:06 ` Christopher Faylor 2002-04-29 0:57 ` Cliff Hones 2002-04-29 4:58 ` Robert Collins 2002-04-29 13:01 ` Using a real mirroring tool Charles Wilson 2002-04-29 17:17 ` Charles Wilson 2002-04-29 17:21 ` Charles Wilson 2002-04-29 17:46 ` Charles Wilson 2002-04-30 6:20 ` Sam Edge 2002-04-30 10:09 ` Volker Quetschke 2002-04-30 13:43 ` Volker Quetschke 2002-04-30 14:00 ` Michael A Chase 2002-04-29 4:07 ` setup 2.194.2.24: Bug (?) in downloading from internet Max Bowsher 2002-04-29 4:24 ` Robert Collins -- strict thread matches above, loose matches on Subject: below -- 2002-04-29 2:35 Robert Collins 2002-04-29 2:26 Robert Collins 2002-04-29 5:42 ` Sam Edge 2002-04-29 5:49 ` Robert Collins 2002-04-29 7:46 ` Sam Edge 2002-04-30 1:58 ` Gary R. Van Sickle 2002-04-29 8:27 ` Christopher Faylor 2002-04-28 22:34 Alec Mihailovs 2002-04-28 23:51 ` Charles Wilson 2002-04-28 10:15 Robert Collins 2002-04-28 8:14 Robert Collins 2002-04-28 9:56 ` Kurt Roeckx 2002-04-28 3:17 Daniele Mezzetti 2002-04-28 4:31 ` Sam Edge 2002-04-26 8:34 Robert Collins 2002-04-26 7:10 Robert Collins 2002-04-26 7:57 ` Alexei Lioubimov 2002-04-25 8:55 Alexei Lioubimov 2002-04-25 11:00 ` Larry Hall (RFK Partners, Inc) 2002-04-26 7:03 ` Alexei Lioubimov
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).