CT_PREFER_MIRROR working?

CT_PREFER_MIRROR working?

[ANDY KENNEDY]

Yann, All,

I select the following settings:

CT_USE_MIRROR=y
CT_PREFER_MIRROR=y
CT_MIRROR_BASE_URL="ftp://akennedy_lin/pub/dl"

with the expectation that for each package needed, the build will go
look at that location (which _is_ a valid name in our network).
Unfortunately, there seems to be "custom" downloads for eglibc.  I
modified the scripts to avoid that custom download if the
MIRRO_BASE_URL is set, however, I was wondering if there is already a
fix for this?  Should _all_ packages attempt to download from the
PREFER_MIRROR base if it is set?

If not, I'll send what I've got, however, it is a really UGLY hack.

Thanks,
Andy

--
For unsubscribe information see http://sourceware.org/lists.html#faq

Re: CT_PREFER_MIRROR working?

[Yann E. MORIN]

[-- Attachment #1: Type: Text/Plain, Size: 1699 bytes --]

Andy, All,

On Monday 01 August 2011 17:20:42 ANDY KENNEDY wrote:
> I select the following settings:
> 
> CT_USE_MIRROR=y
> CT_PREFER_MIRROR=y
> CT_MIRROR_BASE_URL="ftp://akennedy_lin/pub/dl"
> 
> with the expectation that for each package needed, the build will go
> look at that location (which _is_ a valid name in our network).
> Unfortunately, there seems to be "custom" downloads for eglibc.

Indeed, eglibc primarily fetches its sources via svn. It first looks in
the local tarballs dir to see if it was already retrieved, and if not,
reverts to using svn.

The mirror stuff is only for upstreams that provide tarballs.

> I
> modified the scripts to avoid that custom download if the
> MIRRO_BASE_URL is set, however, I was wondering if there is already a
> fix for this?

It might be possible to try the mirror for SVN as well. Care to try
the attached patch, please?

> Should _all_ packages attempt to download from the
> PREFER_MIRROR base if it is set?

That was not intended for all packages at first, as it seemed complex to
do so. But if the attached patch works as expected, then it was not that
complex, in fact.

It could be made to work for cvs (newlib) as well.

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[-- Attachment #2: ct-ng-1.12.0-svn-tries-mirror-first.patch --]
[-- Type: text/x-patch, Size: 825 bytes --]

diff --git a/scripts/functions b/scripts/functions
--- a/scripts/functions
+++ b/scripts/functions
@@ -556,6 +556,7 @@
     for ext in ${first_ext} .tar.bz2 .tar.gz .tgz .tar ''; do
         # Try all urls in turn
         for url in ${URLS}; do
+            [ -n "${url}" ] || continue
             CT_DoLog DEBUG "Trying '${url}/${file}${ext}'"
             CT_DoGetFile "${url}/${file}${ext}"
             if [ -f "${CT_TARBALLS_DIR}/${file}${ext}" ]; then
@@ -628,9 +629,11 @@
     local uri="$2"
     local rev="$3"
 
-    # Does it exist localy?
-    CT_GetLocal "${basename}" && return 0 || true
-    # No, it does not...
+    # First try locally, then the mirror
+    if CT_GetFile "${basename}"; then
+        # Got it! Return early! :-)
+        return 0
+    fi
 
     CT_DoLog EXTRA "Retrieving '${basename}'"
 


[-- Attachment #3: Type: text/plain, Size: 71 bytes --]

--
For unsubscribe information see http://sourceware.org/lists.html#faq

Re: CT_PREFER_MIRROR working?

[Yann E. MORIN]

Andy, All,

[Please keep the list CCed.]

On Monday 01 August 2011 19:51:27 ANDY KENNEDY wrote:
> > It might be possible to try the mirror for SVN as well. Care to try
> > the attached patch, please?
> 
> Yes.  I'll try the patch.  Will the do_eglibc_get() still get called?

Yes, it will. The patch is against the current repo, where the direct calls
to svn have been replaced by calls to the SVN helper. It's the helper that
now:
 - tries to search for a local tarball in ${CT_LOCAL_TARBALLS_DIR}
 - then tries the mirror
 - falls back to downloading via svn

So it goes a bit like that, now:
  eglibc::do_libc_get()
  -> CT_GetSVN "basename" "svn_url" "revision"
     -> [ -f "${local_dir}/${basename}" ]
     -> wget "${mirror_base}/${basename}"
     -> svn co -r "${revision}" "${svn_url}"
... stopping at the first which works.

The patch will not work on ct-ng-1.11.x, you'll have to update to 1.12.0
Now that the 1.12 branch is live, maintenance only happens there. The old
1.11 branch is closed, and no changes will get backported onto 1.11.

> The reason I ask is that I cannot fully test whether this works, since
> I have svn installed on my box.  If it still gets called, svn is
> required by do_eglibc_get() which will break my build on uncontrolled-
> by-me boxes that don't have svn.  I can do a host-build of svn, if
> that is what I need to do, but I'm trying to avoid more work ;).

It will (hopefully!) work with 1.12.0.

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

--
For unsubscribe information see http://sourceware.org/lists.html#faq

RE: CT_PREFER_MIRROR working?

[ANDY KENNEDY]

Yann, All,

> Andy, All,
> 
> [Please keep the list CCed.]

I Do, _every_ time (except when I forget or click the wrong
button) ;). Though I did at least address it to "Yann, all,"
showing the intent; so I got it half right :D.

> 
> On Monday 01 August 2011 19:51:27 ANDY KENNEDY wrote:
> > > It might be possible to try the mirror for SVN as well. Care to
> try
> > > the attached patch, please?
> >
> > Yes.  I'll try the patch.  Will the do_eglibc_get() still get
> called?
> 
> Yes, it will. The patch is against the current repo, where the

Nope, it won't even get past ./configure as it sees svn as a
prereq.  So, I guess I'll have to build svn as apart of my
build system. . . 

Andy

--
For unsubscribe information see http://sourceware.org/lists.html#faq

Re: CT_PREFER_MIRROR working?

[Yann E. MORIN]

Andy, All,

On Monday 01 August 2011 21:24:38 ANDY KENNEDY wrote:
> Nope, it won't even get past ./configure as it sees svn as a
> prereq.  So, I guess I'll have to build svn as apart of my
> build system. . . 

Ah, yes. Just remove the svn check from ./configure, jsut to see if
the patch works...

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

--
For unsubscribe information see http://sourceware.org/lists.html#faq

RE: CT_PREFER_MIRROR working?

[ANDY KENNEDY]

> Andy, All,
> 
> On Monday 01 August 2011 21:24:38 ANDY KENNEDY wrote:
> > Nope, it won't even get past ./configure as it sees svn as a
> > prereq.  So, I guess I'll have to build svn as apart of my
> > build system. . .
> 
> Ah, yes. Just remove the svn check from ./configure, jsut to see if
> the patch works...
> 

Works.  How about we add a config option --mirror-dl-only or
something like that to configure so that we can skip the svn check
if we _know_ we don't want to go to the web for _anything_.

If you want me to make that patch, I can (though it is so trivial it
would take you longer to port the change in than to do it yourself).

Andy

--
For unsubscribe information see http://sourceware.org/lists.html#faq

Re: CT_PREFER_MIRROR working?

[Yann E. MORIN]

Andy, All,

On Tuesday 02 August 2011 19:33:35 ANDY KENNEDY wrote:
> > On Monday 01 August 2011 21:24:38 ANDY KENNEDY wrote:
> > > Nope, it won't even get past ./configure as it sees svn as a
> > > prereq.  So, I guess I'll have to build svn as apart of my
> > > build system. . .
> > 
> > Ah, yes. Just remove the svn check from ./configure, jsut to see if
> > the patch works...
> 
> Works.

Good! Thanks for testing! :-)

> How about we add a config option --mirror-dl-only or
> something like that to configure so that we can skip the svn check
> if we _know_ we don't want to go to the web for _anything_.

We already have FORBID_DOWNLOAD, which CT_GetSVN et al. should follow.

> If you want me to make that patch, I can (though it is so trivial it
> would take you longer to port the change in than to do it yourself).

Yes, I'll handle that. I already have a small series here than does
related changes, so it'll be better that I enqueue that change as well.

Thanks for testing, reporting and suggesting! ;-)

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

--
For unsubscribe information see http://sourceware.org/lists.html#faq

RE: CT_PREFER_MIRROR working?

[ANDY KENNEDY]

Yann, All,

Perhaps I missed something, so questions follow:

> > How about we add a config option --mirror-dl-only or
> > something like that to configure so that we can skip the svn
> check
> > if we _know_ we don't want to go to the web for _anything_.
> 
> We already have FORBID_DOWNLOAD, which CT_GetSVN et al. should
> follow.

But, I don't want to FORBID_DOWNLOAD, I want to _CONTROL_ where I
download from -- in this case, I want to ONLY go to the mirror, which
is a wget for the file, but not out to the web for it.  So, will that
still allow me to "download" from my internal repo?

> 
> > If you want me to make that patch, I can (though it is so trivial
> it
> > would take you longer to port the change in than to do it
> yourself).
> 
> Yes, I'll handle that. I already have a small series here than does
> related changes, so it'll be better that I enqueue that change as
> well.

Cool.  I eagerly anticipate your patch.

Thanks!
Andy

--
For unsubscribe information see http://sourceware.org/lists.html#faq

Re: CT_PREFER_MIRROR working?

[Yann E. MORIN]

Andy, All,

On Tuesday 02 August 2011 20:51:02 ANDY KENNEDY wrote:
> > We already have FORBID_DOWNLOAD, which CT_GetSVN et al. should
> > follow.
> 
> But, I don't want to FORBID_DOWNLOAD, I want to _CONTROL_ where I
> download from -- in this case, I want to ONLY go to the mirror, which
> is a wget for the file, but not out to the web for it.  So, will that
> still allow me to "download" from my internal repo?

Currently, either downloads are allowed, in which case CT_Get* behave so:
 - first, try from local tqarballs dir
 - then try from the mirror (which svn, cvs and git do not honor so far)
 - then try from upstream locations
 - finally fail if not found

The mirror option is labelled the "LAN mirror", but really nothing prevents
it from being any where on the network, local or not. I already provide a
mirror for a few tarballs (eg. uClibc snapshots, or a few mpfr versions as
upstream is sometime unreachable). It is even suggested to use that mirror
in the CT_USE_MIRROR option.

So how would one differentiate between:
 - I don't want to call to the internet, but still use the mirror
and:
 - I don't want to call to the internet at all
when there is no way to know whether the mirror is on the LAN or not?

But anyway, here's what I was planning:
 - if downloads are forbidden, and no mirror is set
   - just look in local tarballs dir (if set)
 - if downloads are forbidden, and a mirror has been set
   - look in local tarballs dir (if set)
   - then try the mirror
 - if downloads are allowed
   - look in local tarballs dir (if set)
   - then try the mirror (if set)
   - then try provided URLs

Selecting the mirror would be de-correlated from the FORBID_DOWNLOAD option,
so it is always available.

Would that match your use-case? That's what I was intending to do. ;-)

I believe that's about all we should support. FORBID_DOWNLOAD is just here
as a fast-path when roaming, and network is not available. But nothing
prevents you to be connection-less and still attempt downloads; it's just
that it will take some time before the failure occurs. FORBID_DOWNLOAD
merely makes it happen sooner.

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

--
For unsubscribe information see http://sourceware.org/lists.html#faq

Re: CT_PREFER_MIRROR working?

[Yann E. MORIN]

On Tuesday 02 August 2011 22:53:51 ANDY KENNEDY wrote:
[--SNIP--]
> > Would that match your use-case? That's what I was intending to do.
> > ;-)
> Yes, that gets me.

Good! :-)

> Okay, BUT, there is one problem that I didn't see you mention:
> 
> ./configure looks for svn, and fails if it is not there.  That
> was what I meant by --mirror-dl-only.  A switch for the configure
> script to NOT check to see that we have svn installed.
> 
> Did you already account for that in the above?  If so, sorry, I
> missed it.

Yes, it's been taken care of. svn is now longer a hard-dependency, and if
not available, svn downloads will just not be possible. svn is now tested
with has_or_warn, instead of has_or_abort. See configure how both work.

No, it's not been explained in the above, indeed. ;-)

It's already queued here (needs polish, though).

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

--
For unsubscribe information see http://sourceware.org/lists.html#faq

RE: CT_PREFER_MIRROR working?

[ANDY KENNEDY]

Yann, All,

> Currently, either downloads are allowed, in which case CT_Get*
> behave so:
>  - first, try from local tqarballs dir
>  - then try from the mirror (which svn, cvs and git do not honor so
> far)
>  - then try from upstream locations
>  - finally fail if not found
> 
> The mirror option is labelled the "LAN mirror", but really nothing
> prevents
> it from being any where on the network, local or not. I already
> provide a
> mirror for a few tarballs (eg. uClibc snapshots, or a few mpfr
> versions as
> upstream is sometime unreachable). It is even suggested to use that
> mirror
> in the CT_USE_MIRROR option.
> 
> So how would one differentiate between:
>  - I don't want to call to the internet, but still use the mirror
> and:
>  - I don't want to call to the internet at all
> when there is no way to know whether the mirror is on the LAN or
> not?
> 
> But anyway, here's what I was planning:
>  - if downloads are forbidden, and no mirror is set
>    - just look in local tarballs dir (if set)
>  - if downloads are forbidden, and a mirror has been set
>    - look in local tarballs dir (if set)
>    - then try the mirror
>  - if downloads are allowed
>    - look in local tarballs dir (if set)
>    - then try the mirror (if set)
>    - then try provided URLs
> 
> Selecting the mirror would be de-correlated from the
> FORBID_DOWNLOAD option,
> so it is always available.
> 

Great explanation, that helped.

> Would that match your use-case? That's what I was intending to do.
> ;-)

Yes, that gets me.

> 
> I believe that's about all we should support. FORBID_DOWNLOAD is
> just here
> as a fast-path when roaming, and network is not available. But
> nothing
> prevents you to be connection-less and still attempt downloads;
> it's just
> that it will take some time before the failure occurs.
> FORBID_DOWNLOAD
> merely makes it happen sooner.

Okay, BUT, there is one problem that I didn't see you mention:

./configure looks for svn, and fails if it is not there.  That
was what I meant by --mirror-dl-only.  A switch for the configure
script to NOT check to see that we have svn installed.

Did you already account for that in the above?  If so, sorry, I
missed it.

Andy

--
For unsubscribe information see http://sourceware.org/lists.html#faq