PATCH: Avoid command line length limit building tools.zip (PR libgcj/38251)

PATCH: Avoid command line length limit building tools.zip (PR libgcj/38251)

[Rainer Orth]

I've finally looked into PR libgcj/38251 again where building tools.zip
in classpath/tools fails on systems with low limits on command line
lenght.  I think the solution is obvious on second thought: instead of
passing all file names except those below to .svn to jar/zip explicitly,
just prune the copies of the classes and asm directories of .svn subdirs
and then archive the whole remaining hierarchy.  The following patch
(only lightly tested so far) does just that.

A few caveats, though:

* The test for the .svn dir must check in asm (or classes), since in a
  VPATH build, there will be no .svn directory otherwise.

* find | xargs is a problem for filenames containing whitespace, but
  find -print0/xargs -0 or Sun's find -exec \{\} + aren't portable.

Ok for mainline anyway?

	Rainer

-- 
-----------------------------------------------------------------------------
Rainer Orth, Center for Biotechnology, Bielefeld University


2010-01-15  Rainer Orth  <ro@CeBiTec.Uni-Bielefeld.DE>

	PR libgcj/38251
	* tools/Makefile.am (dist-hook): Prune .svn directories in asm and
	classes copies.
	* tools/Makefile.in: Regenerate.

	Revert:
	2008-11-05  Andrew Haley  <aph@redhat.com>

	* tools/Makefile.am (UPDATE_TOOLS_ZIP, CREATE_TOOLS_ZIP): Exclude
	.svn direcories.

Index: libjava/classpath/tools/Makefile.am
===================================================================
--- libjava/classpath/tools/Makefile.am	(revision 155837)
+++ libjava/classpath/tools/Makefile.am	(working copy)
@@ -373,6 +373,7 @@
 ## BEGIN GCJ LOCAL
 	cp -pR $(srcdir)/asm .
 	cp -pR $(srcdir)/classes .
+	[ -d asm/.svn ] && find asm classes -type d -name .svn -print | xargs rm -rf
 ## END GCJ LOCAL
 if CREATE_GJDOC
 ## Copy over gjdoc resource files.
@@ -385,11 +386,11 @@
 endif
 
 if WITH_JAR
-CREATE_TOOLS_ZIP=$(JAR) cf ../$(TOOLS_ZIP) `find . -name .svn -prune -o -type f -print`
-UPDATE_TOOLS_ZIP=$(JAR) uf ../$(TOOLS_ZIP) `find . -name .svn -prune -o -type f -print`
+CREATE_TOOLS_ZIP=$(JAR) cf ../$(TOOLS_ZIP) .
+UPDATE_TOOLS_ZIP=$(JAR) uf ../$(TOOLS_ZIP) .
 else
-CREATE_TOOLS_ZIP=$(ZIP) -r ../$(TOOLS_ZIP) `find . -name .svn -prune -o -type f -print`
-UPDATE_TOOLS_ZIP=$(ZIP) -u -r ../$(TOOLS_ZIP) `find . -name .svn -prune -o -type f -print`
+CREATE_TOOLS_ZIP=$(ZIP) -r ../$(TOOLS_ZIP) .
+UPDATE_TOOLS_ZIP=$(ZIP) -u -r ../$(TOOLS_ZIP) .
 endif
 
 ## First add classpath tools stuff.

Re: PATCH: Avoid command line length limit building tools.zip (PR libgcj/38251)

[Rainer Orth]

Rainer Orth <ro@CeBiTec.Uni-Bielefeld.DE> writes:

> I've finally looked into PR libgcj/38251 again where building tools.zip
> in classpath/tools fails on systems with low limits on command line
> lenght.  I think the solution is obvious on second thought: instead of
> passing all file names except those below to .svn to jar/zip explicitly,
> just prune the copies of the classes and asm directories of .svn subdirs
> and then archive the whole remaining hierarchy.  The following patch
> (only lightly tested so far) does just that.
>
> A few caveats, though:
>
> * The test for the .svn dir must check in asm (or classes), since in a
>   VPATH build, there will be no .svn directory otherwise.
>
> * find | xargs is a problem for filenames containing whitespace, but
>   find -print0/xargs -0 or Sun's find -exec \{\} + aren't portable.
>
> Ok for mainline anyway?

Proper testing revealed a couple of problem, fixed by this patch which
survived a full bootstrap on alpha-dec-osf4.0f and alpha-dec-osf5.1b.
The ChangeLog is unchanged, but the patch itself needed some
adjustments:

* The [ -d asm/.svn ] && find ... construct doesn't work because if the
  .svn directory doesn't exist (like in a hg checkout), make aborts
  since the command exits with exit status 1.  Thus I wrap the find in
  if [ -d asm/.svn ] instead.

* Even so, find complaints

  $ find asm -type d -name .svn -exec rm -rf \{\} \;
  find: asm/.svn: No such file or directory

  Using find -depth instead cures this.

* Last but not least, it occured to me that the problematic (in the case
  of directory names with whitespace) find | xargs construct isn't
  necessary at all since it is just a performance optimization to avoid
  creating too many processes.  Since the number of .svn directories in
  those trees isn't that large, I think it's better to err on the side
  of caution and accept the possible performance penalty by using find
  -exec rm instead.

The patch below does just that.

Ok for mainline now?  Tested as above, and it fixes a regression in GCC
4.4/4.5.

	Rainer

-- 
-----------------------------------------------------------------------------
Rainer Orth, Center for Biotechnology, Bielefeld University


2010-01-15  Rainer Orth  <ro@CeBiTec.Uni-Bielefeld.DE>

	PR libgcj/38251
	* tools/Makefile.am (dist-hook): Prune .svn directories in asm and
	classes copies.
	* tools/Makefile.in: Regenerate.

	Revert:
	2008-11-05  Andrew Haley  <aph@redhat.com>

	* tools/Makefile.am (UPDATE_TOOLS_ZIP, CREATE_TOOLS_ZIP): Exclude
	.svn direcories.

diff -r da043a0a9cee libjava/classpath/tools/Makefile.am
--- a/libjava/classpath/tools/Makefile.am	Mon Jan 11 04:28:36 2010 +0000
+++ b/libjava/classpath/tools/Makefile.am	Wed Jan 20 17:49:17 2010 +0100
@@ -373,6 +373,9 @@
 ## BEGIN GCJ LOCAL
 	cp -pR $(srcdir)/asm .
 	cp -pR $(srcdir)/classes .
+	if [ -d asm/.svn ]; then \
+	  find asm classes -depth -type d -name .svn -exec rm -rf \{\} \;; \
+	fi
 ## END GCJ LOCAL
 if CREATE_GJDOC
 ## Copy over gjdoc resource files.
@@ -385,11 +388,11 @@
 endif
 
 if WITH_JAR
-CREATE_TOOLS_ZIP=$(JAR) cf ../$(TOOLS_ZIP) `find . -name .svn -prune -o -type f -print`
-UPDATE_TOOLS_ZIP=$(JAR) uf ../$(TOOLS_ZIP) `find . -name .svn -prune -o -type f -print`
+CREATE_TOOLS_ZIP=$(JAR) cf ../$(TOOLS_ZIP) .
+UPDATE_TOOLS_ZIP=$(JAR) uf ../$(TOOLS_ZIP) .
 else
-CREATE_TOOLS_ZIP=$(ZIP) -r ../$(TOOLS_ZIP) `find . -name .svn -prune -o -type f -print`
-UPDATE_TOOLS_ZIP=$(ZIP) -u -r ../$(TOOLS_ZIP) `find . -name .svn -prune -o -type f -print`
+CREATE_TOOLS_ZIP=$(ZIP) -r ../$(TOOLS_ZIP) .
+UPDATE_TOOLS_ZIP=$(ZIP) -u -r ../$(TOOLS_ZIP) .
 endif
 
 ## First add classpath tools stuff.

Re: PATCH: Avoid command line length limit building tools.zip (PR libgcj/38251)

[Ralf Wildenhues]

Hello Rainer,

* Rainer Orth wrote on Wed, Jan 20, 2010 at 05:59:37PM CET:
> Ok for mainline now?  Tested as above, and it fixes a regression in GCC
> 4.4/4.5.

> 2010-01-15  Rainer Orth  <ro@CeBiTec.Uni-Bielefeld.DE>
> 
>         PR libgcj/38251
>         * tools/Makefile.am (dist-hook): Prune .svn directories in asm and
>         classes copies.
>         * tools/Makefile.in: Regenerate.

> --- a/libjava/classpath/tools/Makefile.am       Mon Jan 11 04:28:36 2010 +0000
> +++ b/libjava/classpath/tools/Makefile.am       Wed Jan 20 17:49:17 2010 +0100
> @@ -373,6 +373,9 @@
>  ## BEGIN GCJ LOCAL
>         cp -pR $(srcdir)/asm .
>         cp -pR $(srcdir)/classes .
> +       if [ -d asm/.svn ]; then \
> +         find asm classes -depth -type d -name .svn -exec rm -rf \{\} \;; \
> +       fi
>  ## END GCJ LOCAL

FWIW, this has the potential to corrupt a SVN source tree iff the build
tree and the source trees happen to coincide.

I have to admit that I haven't tested building GCC in-tree for a few
months now, but after fixing a couple of nasty bugs in this area around
09/2009, it used to work well for me.  Don't remember if I built java
for those tests, but I usually do.

Cheers,
Ralf

Re: PATCH: Avoid command line length limit building tools.zip (PR libgcj/38251)

[Rainer Orth]

Hi Ralf,

>> --- a/libjava/classpath/tools/Makefile.am       Mon Jan 11 04:28:36 2010 +0000
>> +++ b/libjava/classpath/tools/Makefile.am       Wed Jan 20 17:49:17 2010 +0100
>> @@ -373,6 +373,9 @@
>>  ## BEGIN GCJ LOCAL
>>         cp -pR $(srcdir)/asm .
>>         cp -pR $(srcdir)/classes .
>> +       if [ -d asm/.svn ]; then \
>> +         find asm classes -depth -type d -name .svn -exec rm -rf \{\} \;; \
>> +       fi
>>  ## END GCJ LOCAL
>
> FWIW, this has the potential to corrupt a SVN source tree iff the build
> tree and the source trees happen to coincide.

right, that occured to me some time ago.

> I have to admit that I haven't tested building GCC in-tree for a few
> months now, but after fixing a couple of nasty bugs in this area around
> 09/2009, it used to work well for me.  Don't remember if I built java
> for those tests, but I usually do.

I've no idea if any developer does in-tree builds anymore.  If so, I'm
not sure even the cp -pR part is guaranteed to work.  And end-users who
do are probably building from a tarball and don't care about a SVN
checkout anyway.

Thoughts?
	Rainer

-- 
-----------------------------------------------------------------------------
Rainer Orth, Center for Biotechnology, Bielefeld University

Re: PATCH: Avoid command line length limit building tools.zip (PR libgcj/38251)

[Andrew Haley]

On 03/02/2010 11:06 AM, Rainer Orth wrote:
> Hi Ralf,
> 
>>> --- a/libjava/classpath/tools/Makefile.am       Mon Jan 11 04:28:36 2010 +0000
>>> +++ b/libjava/classpath/tools/Makefile.am       Wed Jan 20 17:49:17 2010 +0100
>>> @@ -373,6 +373,9 @@
>>>  ## BEGIN GCJ LOCAL
>>>         cp -pR $(srcdir)/asm .
>>>         cp -pR $(srcdir)/classes .
>>> +       if [ -d asm/.svn ]; then \
>>> +         find asm classes -depth -type d -name .svn -exec rm -rf \{\} \;; \
>>> +       fi
>>>  ## END GCJ LOCAL
>>
>> FWIW, this has the potential to corrupt a SVN source tree iff the build
>> tree and the source trees happen to coincide.
> 
> right, that occured to me some time ago.
> 
>> I have to admit that I haven't tested building GCC in-tree for a few
>> months now, but after fixing a couple of nasty bugs in this area around
>> 09/2009, it used to work well for me.  Don't remember if I built java
>> for those tests, but I usually do.
> 
> I've no idea if any developer does in-tree builds anymore.  If so, I'm
> not sure even the cp -pR part is guaranteed to work.  And end-users who
> do are probably building from a tarball and don't care about a SVN
> checkout anyway.
> 
> Thoughts?

Either  a) don't do this, or b) issue an error if a user tries
to configure in-tree.

Andrew.

Re: PATCH: Avoid command line length limit building tools.zip (PR libgcj/38251)

[Ralf Wildenhues]

* Rainer Orth wrote on Tue, Mar 02, 2010 at 12:06:34PM CET:
> >> --- a/libjava/classpath/tools/Makefile.am       Mon Jan 11 04:28:36 2010 +0000
> >> +++ b/libjava/classpath/tools/Makefile.am       Wed Jan 20 17:49:17 2010 +0100
> >> @@ -373,6 +373,9 @@
> >>  ## BEGIN GCJ LOCAL
> >>         cp -pR $(srcdir)/asm .
> >>         cp -pR $(srcdir)/classes .
> >> +       if [ -d asm/.svn ]; then \
> >> +         find asm classes -depth -type d -name .svn -exec rm -rf \{\} \;; \
> >> +       fi
> >>  ## END GCJ LOCAL
> >
> > FWIW, this has the potential to corrupt a SVN source tree iff the build
> > tree and the source trees happen to coincide.

> I've no idea if any developer does in-tree builds anymore.  If so, I'm
> not sure even the cp -pR part is guaranteed to work.

Yes, that will likely fail.

> And end-users who
> do are probably building from a tarball and don't care about a SVN
> checkout anyway.

Yep.

> Thoughts?

Since due to the cp -pR this isn't a regression, I suggest not doing any
further changes now, but please open a low-priority PR about this
in-tree build bug and CC: me or assign to me, so that it is not
forgotten.

Thanks,
Ralf

Re: PATCH: Avoid command line length limit building tools.zip (PR libgcj/38251)

[Rainer Orth]

Ralf Wildenhues <Ralf.Wildenhues@gmx.de> writes:

> Since due to the cp -pR this isn't a regression, I suggest not doing any
> further changes now, but please open a low-priority PR about this
> in-tree build bug and CC: me or assign to me, so that it is not
> forgotten.

Done:

libgcj/43258	In-tree build may fail in libjava/classpath/tools

Thanks.
        Rainer

-- 
-----------------------------------------------------------------------------
Rainer Orth, Center for Biotechnology, Bielefeld University