* CT_PREFER_MIRROR working? @ 2011-08-01 15:21 ANDY KENNEDY 2011-08-01 17:08 ` Yann E. MORIN 0 siblings, 1 reply; 11+ messages in thread From: ANDY KENNEDY @ 2011-08-01 15:21 UTC (permalink / raw) To: crossgcc Yann, All, I select the following settings: CT_USE_MIRROR=y CT_PREFER_MIRROR=y CT_MIRROR_BASE_URL="ftp://akennedy_lin/pub/dl" with the expectation that for each package needed, the build will go look at that location (which _is_ a valid name in our network). Unfortunately, there seems to be "custom" downloads for eglibc. I modified the scripts to avoid that custom download if the MIRRO_BASE_URL is set, however, I was wondering if there is already a fix for this? Should _all_ packages attempt to download from the PREFER_MIRROR base if it is set? If not, I'll send what I've got, however, it is a really UGLY hack. Thanks, Andy -- For unsubscribe information see http://sourceware.org/lists.html#faq ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: CT_PREFER_MIRROR working? 2011-08-01 15:21 CT_PREFER_MIRROR working? ANDY KENNEDY @ 2011-08-01 17:08 ` Yann E. MORIN [not found] ` <F9C551623D2CBB4C9488801D14F864C604B493@ex-mb1.corp.adtran.com> 0 siblings, 1 reply; 11+ messages in thread From: Yann E. MORIN @ 2011-08-01 17:08 UTC (permalink / raw) To: crossgcc; +Cc: ANDY KENNEDY [-- Attachment #1: Type: Text/Plain, Size: 1699 bytes --] Andy, All, On Monday 01 August 2011 17:20:42 ANDY KENNEDY wrote: > I select the following settings: > > CT_USE_MIRROR=y > CT_PREFER_MIRROR=y > CT_MIRROR_BASE_URL="ftp://akennedy_lin/pub/dl" > > with the expectation that for each package needed, the build will go > look at that location (which _is_ a valid name in our network). > Unfortunately, there seems to be "custom" downloads for eglibc. Indeed, eglibc primarily fetches its sources via svn. It first looks in the local tarballs dir to see if it was already retrieved, and if not, reverts to using svn. The mirror stuff is only for upstreams that provide tarballs. > I > modified the scripts to avoid that custom download if the > MIRRO_BASE_URL is set, however, I was wondering if there is already a > fix for this? It might be possible to try the mirror for SVN as well. Care to try the attached patch, please? > Should _all_ packages attempt to download from the > PREFER_MIRROR base if it is set? That was not intended for all packages at first, as it seemed complex to do so. But if the attached patch works as expected, then it was not that complex, in fact. It could be made to work for cvs (newlib) as well. Regards, Yann E. MORIN. -- .-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------' [-- Attachment #2: ct-ng-1.12.0-svn-tries-mirror-first.patch --] [-- Type: text/x-patch, Size: 825 bytes --] diff --git a/scripts/functions b/scripts/functions --- a/scripts/functions +++ b/scripts/functions @@ -556,6 +556,7 @@ for ext in ${first_ext} .tar.bz2 .tar.gz .tgz .tar ''; do # Try all urls in turn for url in ${URLS}; do + [ -n "${url}" ] || continue CT_DoLog DEBUG "Trying '${url}/${file}${ext}'" CT_DoGetFile "${url}/${file}${ext}" if [ -f "${CT_TARBALLS_DIR}/${file}${ext}" ]; then @@ -628,9 +629,11 @@ local uri="$2" local rev="$3" - # Does it exist localy? - CT_GetLocal "${basename}" && return 0 || true - # No, it does not... + # First try locally, then the mirror + if CT_GetFile "${basename}"; then + # Got it! Return early! :-) + return 0 + fi CT_DoLog EXTRA "Retrieving '${basename}'" [-- Attachment #3: Type: text/plain, Size: 71 bytes --] -- For unsubscribe information see http://sourceware.org/lists.html#faq ^ permalink raw reply [flat|nested] 11+ messages in thread
[parent not found: <F9C551623D2CBB4C9488801D14F864C604B493@ex-mb1.corp.adtran.com>]
* Re: CT_PREFER_MIRROR working? [not found] ` <F9C551623D2CBB4C9488801D14F864C604B493@ex-mb1.corp.adtran.com> @ 2011-08-01 18:37 ` Yann E. MORIN 2011-08-01 19:24 ` ANDY KENNEDY 0 siblings, 1 reply; 11+ messages in thread From: Yann E. MORIN @ 2011-08-01 18:37 UTC (permalink / raw) To: crossgcc; +Cc: ANDY KENNEDY Andy, All, [Please keep the list CCed.] On Monday 01 August 2011 19:51:27 ANDY KENNEDY wrote: > > It might be possible to try the mirror for SVN as well. Care to try > > the attached patch, please? > > Yes. I'll try the patch. Will the do_eglibc_get() still get called? Yes, it will. The patch is against the current repo, where the direct calls to svn have been replaced by calls to the SVN helper. It's the helper that now: - tries to search for a local tarball in ${CT_LOCAL_TARBALLS_DIR} - then tries the mirror - falls back to downloading via svn So it goes a bit like that, now: eglibc::do_libc_get() -> CT_GetSVN "basename" "svn_url" "revision" -> [ -f "${local_dir}/${basename}" ] -> wget "${mirror_base}/${basename}" -> svn co -r "${revision}" "${svn_url}" ... stopping at the first which works. The patch will not work on ct-ng-1.11.x, you'll have to update to 1.12.0 Now that the 1.12 branch is live, maintenance only happens there. The old 1.11 branch is closed, and no changes will get backported onto 1.11. > The reason I ask is that I cannot fully test whether this works, since > I have svn installed on my box. If it still gets called, svn is > required by do_eglibc_get() which will break my build on uncontrolled- > by-me boxes that don't have svn. I can do a host-build of svn, if > that is what I need to do, but I'm trying to avoid more work ;). It will (hopefully!) work with 1.12.0. Regards, Yann E. MORIN. -- .-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------' -- For unsubscribe information see http://sourceware.org/lists.html#faq ^ permalink raw reply [flat|nested] 11+ messages in thread
* RE: CT_PREFER_MIRROR working? 2011-08-01 18:37 ` Yann E. MORIN @ 2011-08-01 19:24 ` ANDY KENNEDY 2011-08-01 19:44 ` Yann E. MORIN 0 siblings, 1 reply; 11+ messages in thread From: ANDY KENNEDY @ 2011-08-01 19:24 UTC (permalink / raw) To: crossgcc; +Cc: Yann E. MORIN Yann, All, > Andy, All, > > [Please keep the list CCed.] I Do, _every_ time (except when I forget or click the wrong button) ;). Though I did at least address it to "Yann, all," showing the intent; so I got it half right :D. > > On Monday 01 August 2011 19:51:27 ANDY KENNEDY wrote: > > > It might be possible to try the mirror for SVN as well. Care to > try > > > the attached patch, please? > > > > Yes. I'll try the patch. Will the do_eglibc_get() still get > called? > > Yes, it will. The patch is against the current repo, where the Nope, it won't even get past ./configure as it sees svn as a prereq. So, I guess I'll have to build svn as apart of my build system. . . Andy -- For unsubscribe information see http://sourceware.org/lists.html#faq ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: CT_PREFER_MIRROR working? 2011-08-01 19:24 ` ANDY KENNEDY @ 2011-08-01 19:44 ` Yann E. MORIN 2011-08-02 17:35 ` ANDY KENNEDY 0 siblings, 1 reply; 11+ messages in thread From: Yann E. MORIN @ 2011-08-01 19:44 UTC (permalink / raw) To: ANDY KENNEDY; +Cc: crossgcc Andy, All, On Monday 01 August 2011 21:24:38 ANDY KENNEDY wrote: > Nope, it won't even get past ./configure as it sees svn as a > prereq. So, I guess I'll have to build svn as apart of my > build system. . . Ah, yes. Just remove the svn check from ./configure, jsut to see if the patch works... Regards, Yann E. MORIN. -- .-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------' -- For unsubscribe information see http://sourceware.org/lists.html#faq ^ permalink raw reply [flat|nested] 11+ messages in thread
* RE: CT_PREFER_MIRROR working? 2011-08-01 19:44 ` Yann E. MORIN @ 2011-08-02 17:35 ` ANDY KENNEDY 2011-08-02 18:43 ` Yann E. MORIN 0 siblings, 1 reply; 11+ messages in thread From: ANDY KENNEDY @ 2011-08-02 17:35 UTC (permalink / raw) To: Yann E. MORIN; +Cc: crossgcc > Andy, All, > > On Monday 01 August 2011 21:24:38 ANDY KENNEDY wrote: > > Nope, it won't even get past ./configure as it sees svn as a > > prereq. So, I guess I'll have to build svn as apart of my > > build system. . . > > Ah, yes. Just remove the svn check from ./configure, jsut to see if > the patch works... > Works. How about we add a config option --mirror-dl-only or something like that to configure so that we can skip the svn check if we _know_ we don't want to go to the web for _anything_. If you want me to make that patch, I can (though it is so trivial it would take you longer to port the change in than to do it yourself). Andy -- For unsubscribe information see http://sourceware.org/lists.html#faq ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: CT_PREFER_MIRROR working? 2011-08-02 17:35 ` ANDY KENNEDY @ 2011-08-02 18:43 ` Yann E. MORIN 2011-08-02 18:51 ` ANDY KENNEDY 0 siblings, 1 reply; 11+ messages in thread From: Yann E. MORIN @ 2011-08-02 18:43 UTC (permalink / raw) To: crossgcc; +Cc: ANDY KENNEDY Andy, All, On Tuesday 02 August 2011 19:33:35 ANDY KENNEDY wrote: > > On Monday 01 August 2011 21:24:38 ANDY KENNEDY wrote: > > > Nope, it won't even get past ./configure as it sees svn as a > > > prereq. So, I guess I'll have to build svn as apart of my > > > build system. . . > > > > Ah, yes. Just remove the svn check from ./configure, jsut to see if > > the patch works... > > Works. Good! Thanks for testing! :-) > How about we add a config option --mirror-dl-only or > something like that to configure so that we can skip the svn check > if we _know_ we don't want to go to the web for _anything_. We already have FORBID_DOWNLOAD, which CT_GetSVN et al. should follow. > If you want me to make that patch, I can (though it is so trivial it > would take you longer to port the change in than to do it yourself). Yes, I'll handle that. I already have a small series here than does related changes, so it'll be better that I enqueue that change as well. Thanks for testing, reporting and suggesting! ;-) Regards, Yann E. MORIN. -- .-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------' -- For unsubscribe information see http://sourceware.org/lists.html#faq ^ permalink raw reply [flat|nested] 11+ messages in thread
* RE: CT_PREFER_MIRROR working? 2011-08-02 18:43 ` Yann E. MORIN @ 2011-08-02 18:51 ` ANDY KENNEDY 2011-08-02 20:39 ` Yann E. MORIN 0 siblings, 1 reply; 11+ messages in thread From: ANDY KENNEDY @ 2011-08-02 18:51 UTC (permalink / raw) To: Yann E. MORIN, crossgcc Yann, All, Perhaps I missed something, so questions follow: > > How about we add a config option --mirror-dl-only or > > something like that to configure so that we can skip the svn > check > > if we _know_ we don't want to go to the web for _anything_. > > We already have FORBID_DOWNLOAD, which CT_GetSVN et al. should > follow. But, I don't want to FORBID_DOWNLOAD, I want to _CONTROL_ where I download from -- in this case, I want to ONLY go to the mirror, which is a wget for the file, but not out to the web for it. So, will that still allow me to "download" from my internal repo? > > > If you want me to make that patch, I can (though it is so trivial > it > > would take you longer to port the change in than to do it > yourself). > > Yes, I'll handle that. I already have a small series here than does > related changes, so it'll be better that I enqueue that change as > well. Cool. I eagerly anticipate your patch. Thanks! Andy -- For unsubscribe information see http://sourceware.org/lists.html#faq ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: CT_PREFER_MIRROR working? 2011-08-02 18:51 ` ANDY KENNEDY @ 2011-08-02 20:39 ` Yann E. MORIN 2011-08-02 21:26 ` ANDY KENNEDY 0 siblings, 1 reply; 11+ messages in thread From: Yann E. MORIN @ 2011-08-02 20:39 UTC (permalink / raw) To: crossgcc; +Cc: ANDY KENNEDY Andy, All, On Tuesday 02 August 2011 20:51:02 ANDY KENNEDY wrote: > > We already have FORBID_DOWNLOAD, which CT_GetSVN et al. should > > follow. > > But, I don't want to FORBID_DOWNLOAD, I want to _CONTROL_ where I > download from -- in this case, I want to ONLY go to the mirror, which > is a wget for the file, but not out to the web for it. So, will that > still allow me to "download" from my internal repo? Currently, either downloads are allowed, in which case CT_Get* behave so: - first, try from local tqarballs dir - then try from the mirror (which svn, cvs and git do not honor so far) - then try from upstream locations - finally fail if not found The mirror option is labelled the "LAN mirror", but really nothing prevents it from being any where on the network, local or not. I already provide a mirror for a few tarballs (eg. uClibc snapshots, or a few mpfr versions as upstream is sometime unreachable). It is even suggested to use that mirror in the CT_USE_MIRROR option. So how would one differentiate between: - I don't want to call to the internet, but still use the mirror and: - I don't want to call to the internet at all when there is no way to know whether the mirror is on the LAN or not? But anyway, here's what I was planning: - if downloads are forbidden, and no mirror is set - just look in local tarballs dir (if set) - if downloads are forbidden, and a mirror has been set - look in local tarballs dir (if set) - then try the mirror - if downloads are allowed - look in local tarballs dir (if set) - then try the mirror (if set) - then try provided URLs Selecting the mirror would be de-correlated from the FORBID_DOWNLOAD option, so it is always available. Would that match your use-case? That's what I was intending to do. ;-) I believe that's about all we should support. FORBID_DOWNLOAD is just here as a fast-path when roaming, and network is not available. But nothing prevents you to be connection-less and still attempt downloads; it's just that it will take some time before the failure occurs. FORBID_DOWNLOAD merely makes it happen sooner. Regards, Yann E. MORIN. -- .-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------' -- For unsubscribe information see http://sourceware.org/lists.html#faq ^ permalink raw reply [flat|nested] 11+ messages in thread
* RE: CT_PREFER_MIRROR working? 2011-08-02 20:39 ` Yann E. MORIN @ 2011-08-02 21:26 ` ANDY KENNEDY 2011-08-02 21:20 ` Yann E. MORIN 0 siblings, 1 reply; 11+ messages in thread From: ANDY KENNEDY @ 2011-08-02 21:26 UTC (permalink / raw) To: Yann E. MORIN, crossgcc Yann, All, > Currently, either downloads are allowed, in which case CT_Get* > behave so: > - first, try from local tqarballs dir > - then try from the mirror (which svn, cvs and git do not honor so > far) > - then try from upstream locations > - finally fail if not found > > The mirror option is labelled the "LAN mirror", but really nothing > prevents > it from being any where on the network, local or not. I already > provide a > mirror for a few tarballs (eg. uClibc snapshots, or a few mpfr > versions as > upstream is sometime unreachable). It is even suggested to use that > mirror > in the CT_USE_MIRROR option. > > So how would one differentiate between: > - I don't want to call to the internet, but still use the mirror > and: > - I don't want to call to the internet at all > when there is no way to know whether the mirror is on the LAN or > not? > > But anyway, here's what I was planning: > - if downloads are forbidden, and no mirror is set > - just look in local tarballs dir (if set) > - if downloads are forbidden, and a mirror has been set > - look in local tarballs dir (if set) > - then try the mirror > - if downloads are allowed > - look in local tarballs dir (if set) > - then try the mirror (if set) > - then try provided URLs > > Selecting the mirror would be de-correlated from the > FORBID_DOWNLOAD option, > so it is always available. > Great explanation, that helped. > Would that match your use-case? That's what I was intending to do. > ;-) Yes, that gets me. > > I believe that's about all we should support. FORBID_DOWNLOAD is > just here > as a fast-path when roaming, and network is not available. But > nothing > prevents you to be connection-less and still attempt downloads; > it's just > that it will take some time before the failure occurs. > FORBID_DOWNLOAD > merely makes it happen sooner. Okay, BUT, there is one problem that I didn't see you mention: ./configure looks for svn, and fails if it is not there. That was what I meant by --mirror-dl-only. A switch for the configure script to NOT check to see that we have svn installed. Did you already account for that in the above? If so, sorry, I missed it. Andy -- For unsubscribe information see http://sourceware.org/lists.html#faq ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: CT_PREFER_MIRROR working? 2011-08-02 21:26 ` ANDY KENNEDY @ 2011-08-02 21:20 ` Yann E. MORIN 0 siblings, 0 replies; 11+ messages in thread From: Yann E. MORIN @ 2011-08-02 21:20 UTC (permalink / raw) To: ANDY KENNEDY; +Cc: crossgcc On Tuesday 02 August 2011 22:53:51 ANDY KENNEDY wrote: [--SNIP--] > > Would that match your use-case? That's what I was intending to do. > > ;-) > Yes, that gets me. Good! :-) > Okay, BUT, there is one problem that I didn't see you mention: > > ./configure looks for svn, and fails if it is not there. That > was what I meant by --mirror-dl-only. A switch for the configure > script to NOT check to see that we have svn installed. > > Did you already account for that in the above? If so, sorry, I > missed it. Yes, it's been taken care of. svn is now longer a hard-dependency, and if not available, svn downloads will just not be possible. svn is now tested with has_or_warn, instead of has_or_abort. See configure how both work. No, it's not been explained in the above, indeed. ;-) It's already queued here (needs polish, though). Regards, Yann E. MORIN. -- .-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------' -- For unsubscribe information see http://sourceware.org/lists.html#faq ^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2011-08-02 21:26 UTC | newest] Thread overview: 11+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2011-08-01 15:21 CT_PREFER_MIRROR working? ANDY KENNEDY 2011-08-01 17:08 ` Yann E. MORIN [not found] ` <F9C551623D2CBB4C9488801D14F864C604B493@ex-mb1.corp.adtran.com> 2011-08-01 18:37 ` Yann E. MORIN 2011-08-01 19:24 ` ANDY KENNEDY 2011-08-01 19:44 ` Yann E. MORIN 2011-08-02 17:35 ` ANDY KENNEDY 2011-08-02 18:43 ` Yann E. MORIN 2011-08-02 18:51 ` ANDY KENNEDY 2011-08-02 20:39 ` Yann E. MORIN 2011-08-02 21:26 ` ANDY KENNEDY 2011-08-02 21:20 ` Yann E. MORIN
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).