PATCH: Test the just built windres/dlltool only

PATCH: Test the just built windres/dlltool only

[H. J. Lu]

windres/dlltool may not be built for all platforms. But we are picking
up them from PATH even if they weren't built at all. This patch
changes it.


H.J.
----
2005-02-18  H.J. Lu  <hongjiu.lu@intel.com>

	* config/default.exp: Set WINDRES only if windres is built. Set
	DLLTOOL only if dlltool is built.

--- binutils/testsuite/config/default.exp.win	2004-11-05 14:41:25.000000000 -0800
+++ binutils/testsuite/config/default.exp	2005-02-18 14:30:17.384437504 -0800
@@ -62,11 +62,11 @@ if ![info exists READELF] then {
 if ![info exists READELFFLAGS] then {
     set READELFFLAGS ""
 }
-if ![info exists WINDRES] then {
-    set WINDRES [findfile $base_dir/windres]
+if {![info exists WINDRES] && [file exists $base_dir/windres]} then {
+    set WINDRES $base_dir/windres
 }
-if ![info exists DLLTOOL] then {
-    set DLLTOOL [findfile $base_dir/dlltool]
+if {![info exists DLLTOOL] && [file exists $base_dir/dlltool] } then {
+    set DLLTOOL $base_dir/dlltool
 }
 
 if ![file isdirectory tmpdir] {catch "exec mkdir tmpdir" status}

Re: PATCH: Test the just built windres/dlltool only

[Daniel Jacobowitz]

On Fri, Feb 18, 2005 at 02:59:35PM -0800, H. J. Lu wrote:
> windres/dlltool may not be built for all platforms. But we are picking
> up them from PATH even if they weren't built at all. This patch
> changes it.

If they are on your $PATH, why aren't we building them?  This patch
will disable the tests when testing an installed binutils, I think.

> 
> 
> H.J.
> ----
> 2005-02-18  H.J. Lu  <hongjiu.lu@intel.com>
> 
> 	* config/default.exp: Set WINDRES only if windres is built. Set
> 	DLLTOOL only if dlltool is built.
> 
> --- binutils/testsuite/config/default.exp.win	2004-11-05 14:41:25.000000000 -0800
> +++ binutils/testsuite/config/default.exp	2005-02-18 14:30:17.384437504 -0800
> @@ -62,11 +62,11 @@ if ![info exists READELF] then {
>  if ![info exists READELFFLAGS] then {
>      set READELFFLAGS ""
>  }
> -if ![info exists WINDRES] then {
> -    set WINDRES [findfile $base_dir/windres]
> +if {![info exists WINDRES] && [file exists $base_dir/windres]} then {
> +    set WINDRES $base_dir/windres
>  }
> -if ![info exists DLLTOOL] then {
> -    set DLLTOOL [findfile $base_dir/dlltool]
> +if {![info exists DLLTOOL] && [file exists $base_dir/dlltool] } then {
> +    set DLLTOOL $base_dir/dlltool
>  }
>  
>  if ![file isdirectory tmpdir] {catch "exec mkdir tmpdir" status}
> 

-- 
Daniel Jacobowitz
CodeSourcery, LLC

Re: PATCH: Test the just built windres/dlltool only

[H. J. Lu]

On Fri, Feb 18, 2005 at 06:01:00PM -0500, Daniel Jacobowitz wrote:
> On Fri, Feb 18, 2005 at 02:59:35PM -0800, H. J. Lu wrote:
> > windres/dlltool may not be built for all platforms. But we are picking
> > up them from PATH even if they weren't built at all. This patch
> > changes it.
> 
> If they are on your $PATH, why aren't we building them?  This patch

It is configured for a different target.

> will disable the tests when testing an installed binutils, I think.
> 

How do you test the installed binutils with the binutils build tree?
I thought you should set them in environment if you wanted to test the
installed ones.


H.J.

Re: PATCH: Test the just built windres/dlltool only

[Nick Clifton]

Hi H. J.

>>>windres/dlltool may not be built for all platforms. But we are picking
>>>up them from PATH even if they weren't built at all. This patch
>>>changes it.
>>
>>If they are on your $PATH, why aren't we building them?  This patch

> It is configured for a different target.

>>will disable the tests when testing an installed binutils, I think.

> How do you test the installed binutils with the binutils build tree?
> I thought you should set them in environment if you wanted to test the
> installed ones.

I think that the point is that you should set the PATH in your 
environment to only include the built tools for your current target, not 
for other targets.  Otherwise you could end up testing the entirely 
wrong set of tools.

Cheers
   Nick

Re: PATCH: Test the just built windres/dlltool only

[H. J. Lu]

On Mon, Feb 21, 2005 at 05:44:43PM +0000, Nick Clifton wrote:
> Hi H. J.
> 
> >>>windres/dlltool may not be built for all platforms. But we are picking
> >>>up them from PATH even if they weren't built at all. This patch
> >>>changes it.
> >>
> >>If they are on your $PATH, why aren't we building them?  This patch
> 
> >It is configured for a different target.
> 
> >>will disable the tests when testing an installed binutils, I think.
> 
> >How do you test the installed binutils with the binutils build tree?
> >I thought you should set them in environment if you wanted to test the
> >installed ones.
> 
> I think that the point is that you should set the PATH in your 
> environment to only include the built tools for your current target, not 
> for other targets.  Otherwise you could end up testing the entirely 
> wrong set of tools.
> 

My PATH has binaries for my host machine, which are irrelevant to the
target for my binutils built tree. I don't think

# configure
# make
# make check

should test any of my install binaries. "make check" should assume
my installed binaries are OK. I still haven't seen a reasonable reason
for "make check" to test the installed binaries.


H.J.

Re: PATCH: Test the just built windres/dlltool only

[Nick Clifton]

Hi H. J.

> My PATH has binaries for my host machine, which are irrelevant to the
> target for my binutils built tree. I don't think
> 
> # configure
> # make
> # make check
> 
> should test any of my install binaries. "make check" should assume
> my installed binaries are OK. 

But by this argument, your patch ought to be extended to all of the 
programs checked in the default.exp file, not just windres and dlltool.

Plus I do not think that it is always reasonable to assume that 
installed binaries are OK.  Users should be able to test them if they 
are worried that they contain bugs, or if they want to compare the 
behaviour of an installed set of tools against another set.

> I still haven't seen a reasonable reason
> for "make check" to test the installed binaries.

Perhaps what you really want is a way to disable the testsuite from 
searching through $PATH for binaries to test.  So that if, and only if, 
you are sure that you do not want to check any installed binaries, then 
you could enable this magic option and the testsuite would never check 
anything but binaries in build trees.

I would suggest creating your own version of findfile which when it 
cannot find the file checks to see whether names-in-PATH is OK, and if 
not then just returns "" and otherwise returns the transformed name. 
Then replace all uses of findfile (in default.exp) with this version.

Cheers
   Nick

Re: PATCH: Test the just built windres/dlltool only

[H. J. Lu]

On Tue, Feb 22, 2005 at 05:00:31PM +0000, Nick Clifton wrote:
> Hi H. J.
> 
> >My PATH has binaries for my host machine, which are irrelevant to the
> >target for my binutils built tree. I don't think
> >
> ># configure
> ># make
> ># make check
> >
> >should test any of my install binaries. "make check" should assume
> >my installed binaries are OK. 
> 
> But by this argument, your patch ought to be extended to all of the 
> programs checked in the default.exp file, not just windres and dlltool.
> 

windres and dlltool are only enabled for certain targets. It isn't
impossible that I have windres or dlltool in PATH for entirely
different purposes. I asked for windres nor dlltool not to be
built. It is kind of odd for "make check" to check my windres or
dlltool behind my back.

> Plus I do not think that it is always reasonable to assume that 
> installed binaries are OK.  Users should be able to test them if they 
> are worried that they contain bugs, or if they want to compare the 
> behaviour of an installed set of tools against another set.

But I don't think it is the best way to test the installed binutils.
Have you tried to test the installed binutils in the build directory?
Can you can show me how my patch prenvents you from testing the
installed binutils?


H.J.

Re: PATCH: Test the just built windres/dlltool only

[Nick Clifton]

Hi H. J.

>>But by this argument, your patch ought to be extended to all of the 
>>programs checked in the default.exp file, not just windres and dlltool.

> windres and dlltool are only enabled for certain targets. It isn't
> impossible that I have windres or dlltool in PATH for entirely
> different purposes. I asked for windres nor dlltool not to be
> built. It is kind of odd for "make check" to check my windres or
> dlltool behind my back.

I agree - especially if "make check" is going to check in-build-tree 
versions of the other commands.  It certainly ought to be consistent, 
either checking all in-build-tree commands or all in-installed-tree 
commands.  The point I was making was that your patch appeared to be 
treating windres and dlltool as special cases when really this ought to 
be an all or nothing approach as to which tools are tested.

> But I don't think it is the best way to test the installed binutils.
> Have you tried to test the installed binutils in the build directory?

Err well I do not install binaries into a build directory, I install 
them into a completely separate directory.  But I have tested installed 
binaries that were created from a build tree.  (Usually because there is 
a bug in the install process somewhere and I want the testsuite to help 
me track down the problem).

> Can you can show me how my patch prevents you from testing the
> installed binutils?

Sure, try this:

   * Build a mingw32-pe targeted toolchain (including gcc).
   * Install at least gcc and the binutils somewhere.
   * Go to the binutils build directory and run "make check".  You will 
test the built executables in the build directory.  Save the 
binutils.log for later comparison.
   * Then run "make clean" to get rid of the built executables and add 
the installed bin directory to your search path.  Rerun "make check". 
You will test the installed executables using the build directory as a 
test framework.  Save the binutils.log again.  It should be essentially 
the same as the results from the in-build-tree test.
   * Now apply your patch to default.exp and rerun the "make check". 
This time the installed windres and dlltool will not be tested.

Cheers
   Nick

Re: PATCH: Test the just built windres/dlltool only

[H. J. Lu]

On Wed, Feb 23, 2005 at 02:31:49PM +0000, Nick Clifton wrote:
> Hi H. J.
> 
> >>But by this argument, your patch ought to be extended to all of the 
> >>programs checked in the default.exp file, not just windres and dlltool.
> 
> >windres and dlltool are only enabled for certain targets. It isn't
> >impossible that I have windres or dlltool in PATH for entirely
> >different purposes. I asked for windres nor dlltool not to be
> >built. It is kind of odd for "make check" to check my windres or
> >dlltool behind my back.
> 
> I agree - especially if "make check" is going to check in-build-tree 
> versions of the other commands.  It certainly ought to be consistent, 
> either checking all in-build-tree commands or all in-installed-tree 
> commands.  The point I was making was that your patch appeared to be 
> treating windres and dlltool as special cases when really this ought to 
> be an all or nothing approach as to which tools are tested.
> 
> >But I don't think it is the best way to test the installed binutils.
> >Have you tried to test the installed binutils in the build directory?
> 
> Err well I do not install binaries into a build directory, I install 
> them into a completely separate directory.  But I have tested installed 
> binaries that were created from a build tree.  (Usually because there is 
> a bug in the install process somewhere and I want the testsuite to help 
> me track down the problem).
> 
> >Can you can show me how my patch prevents you from testing the
> >installed binutils?
> 
> Sure, try this:
> 
>   * Build a mingw32-pe targeted toolchain (including gcc).
>   * Install at least gcc and the binutils somewhere.
>   * Go to the binutils build directory and run "make check".  You will 
> test the built executables in the build directory.  Save the 
> binutils.log for later comparison.
>   * Then run "make clean" to get rid of the built executables and add 
> the installed bin directory to your search path.  Rerun "make check". 
> You will test the installed executables using the build directory as a 
> test framework.  Save the binutils.log again.  It should be essentially 
> the same as the results from the in-build-tree test.
>   * Now apply your patch to default.exp and rerun the "make check". 
> This time the installed windres and dlltool will not be tested.
> 

I put the following commands in my ~/.dejagnurc. I don't have to do
"make clean" to test the installed binutils. If anything goes wrong,
I don't have to do a "make" again.


H.J.
----
set AR ar
set ar ar
set AS as
set as as
set NM nm
set NM nm
set DLLTOOL dlltool
set dlltool dlltool
set LD ld
set ld ld
set OBJCOPY objcopy
set objcopy objcopy
set OBJDUMP objdump
set objdump objdump
set READELF readelf
set readelf readelf
set SIZE size
set size size
set STRIP strip
set strip strip
set WINDRES windres
set windres windres