* [PATCH] libjava: faster 'make install'
@ 2008-03-09 14:58 Ralf Wildenhues
2008-03-09 15:50 ` Tom Tromey
0 siblings, 1 reply; 3+ messages in thread
From: Ralf Wildenhues @ 2008-03-09 14:58 UTC (permalink / raw)
To: gcc-patches, java-patches
[-- Attachment #1: Type: text/plain, Size: 4820 bytes --]
There's quite a bit of unnecessary overhead of 'make install' in
libjava. Two improvements:
1) Avoid one shell fork per file in write_entries_to_file, instead
use 4 forks per 50 files, plus some constant overhead.
Timings for `make install' in libjava on a GNU/Linux x86:
Pre: 155s, post: 120s. (This should marginally benefit creation of
lib-gnu-awt-xlib.la as well.)
2) Use awk, and use `install' with multiple files at once.
Both of these uses are sanctioned by the GNU Coding Standards now.
However, it needs a fixed AC_PROG_INSTALL from Autoconf mainline,
to ensure that the `install' program is powerful enough.
Pre: 120s, post: 44s. With both patches, `make install-data-local'
takes less than 10s, from 130s previously.
The two separate (and independent) limitations to 50 files are
- expected to suffice to stay below command line limits for any system
(with `../gcc/configure', the longest line I measured has roughly
3600 bytes),
- but still be only a few percent off the optimal speed on modern
systems,
- in the second patch, avoid quadratic scaling due to shell string
concatenation.
This patch should be pretty portable (I'm adding stuff similar to the
second patch into Automake[1]).
For testing the first patch, I have verified that
cd $host/libjava
make write-entries-to-file-check
produces an identical libgcj.objectlist file; for both patches, I have
verified that, on i686-unknown-linux-gnu,
make install
produces an identical installed image (apart from the inevitable
timestamp differences in produced static libraries and binary programs).
Both patches are shown without the (mechanical) changes to the
regenerated files.
OK for mainline? I suppose some more testing would not hurt, but I
don't have access to a system with a really low command line length
limit. If approved I can provide a patch to sync src.
Thanks,
Ralf
PS: Of course, libstdc++/include/Makefile.am (and maybe others) would
benefit from a similar patch as the second one (which I intend to submit
if this approach is deemed desirable).
[1] <http://thread.gmane.org/gmane.comp.sysutils.automake.patches/2945/focus=3088>
First patch:
libjava/ChangeLog:
2008-03-08 Ralf Wildenhues <Ralf.Wildenhues@gmx.de>
* Makefile.am (write_entries_to_file_split): New variable.
(write_entries_to_file): Write out that many entries at once,
for speed.
* Makefile.in: Regenerate.
Second patch:
ChangeLog:
2008-03-08 Ralf Wildenhues <Ralf.Wildenhues@gmx.de>
* configure.ac: m4_include config/proginstall.m4.
* configure: Regenerate.
config/ChangeLog:
2008-03-08 Ralf Wildenhues <Ralf.Wildenhues@gmx.de>
* proginstall.m4: New file, with fixed AC_PROG_INSTALL.
libjava/ChangeLog:
2008-03-08 Ralf Wildenhues <Ralf.Wildenhues@gmx.de>
* Makefile.am (install-data-local): Rewrite to be more efficient,
using awk, and `install' with multiple files at once. Do not gather
more than 50 files in one go to avoid hitting command line limits.
(install_data_local_split): New variable.
* aclocal.m4: Regenerate.
* configure: Likewise.
* Makefile.in: Likewise.
* gcj/Makefile.in: Likewise.
* include/Makefile.in: Likewise.
* testsuite/Makefile.in: Likewise.
boehm-gc/ChangeLog:
2008-03-08 Ralf Wildenhues <Ralf.Wildenhues@gmx.de>
* aclocal.m4: Regenerate.
* configure: Likewise.
* Makefile.in: Likewise.
* include/Makefile.in: Likewise.
libffi/ChangeLog:
2008-03-08 Ralf Wildenhues <Ralf.Wildenhues@gmx.de>
* aclocal.m4: Regenerate.
* configure: Likewise.
* Makefile.in: Likewise.
* include/Makefile.in: Likewise.
* testsuite/Makefile.in: Likewise.
libgfortran/ChangeLog:
2008-03-08 Ralf Wildenhues <Ralf.Wildenhues@gmx.de>
* aclocal.m4: Regenerate.
* configure: Likewise.
* Makefile.in: Likewise.
libgomp/ChangeLog:
2008-03-08 Ralf Wildenhues <Ralf.Wildenhues@gmx.de>
* aclocal.m4: Regenerate.
* configure: Likewise.
* Makefile.in: Likewise.
* testsuite/Makefile.in: Likewise.
libmudflap/ChangeLog:
2008-03-08 Ralf Wildenhues <Ralf.Wildenhues@gmx.de>
* aclocal.m4: Regenerate.
* configure: Likewise.
* Makefile.in: Likewise.
* testsuite/Makefile.in: Likewise.
libssp/ChangeLog:
2008-03-08 Ralf Wildenhues <Ralf.Wildenhues@gmx.de>
* aclocal.m4: Regenerate.
* configure: Likewise.
* Makefile.in: Likewise.
libstdc++-v3/ChangeLog:
2008-03-08 Ralf Wildenhues <Ralf.Wildenhues@gmx.de>
* aclocal.m4: Regenerate.
* configure: Likewise.
* Makefile.in: Likewise.
* doc/Makefile.in: Likewise.
* include/Makefile.in: Likewise.
* libmath/Makefile.in: Likewise.
* libsupc++/Makefile.in: Likewise.
* po/Makefile.in: Likewise.
* src/Makefile.in: Likewise.
* testsuite/Makefile.in: Likewise.
zlib/ChangeLog:
2008-03-08 Ralf Wildenhues <Ralf.Wildenhues@gmx.de>
* aclocal.m4: Regenerate.
* configure: Likewise.
* Makefile.in: Likewise.
[-- Attachment #2: p1 --]
[-- Type: text/plain, Size: 1527 bytes --]
Speedup write_entries_to_file.
libjava/ChangeLog:
2008-03-08 Ralf Wildenhues <Ralf.Wildenhues@gmx.de>
* Makefile.am (write_entries_to_file_split): New variable.
(write_entries_to_file): Write out that many entries at once,
for speed.
* Makefile.in: Regenerate.
diff --git a/libjava/Makefile.am b/libjava/Makefile.am
index d840782..5b84c17 100644
--- a/libjava/Makefile.am
+++ b/libjava/Makefile.am
@@ -13,13 +13,20 @@ SUBDIRS += testsuite
endif
# write_entries_to_file - writes each entry in a list
-# to the specified file. Each entry is written individually
-# to accomodate systems with severe command-line-length
-# limitations.
+# to the specified file. Entries are written in chunks of
+# $(write_entries_to_file_split) to accomodate systems with
+# severe command-line-length limitations.
# Parameters:
# $(1): variable containing entries to iterate over
# $(2): output file
-write_entries_to_file = $(shell rm -f $(2) || :) $(shell touch $(2)) $(foreach object,$(1),$(shell echo $(object) >> $(2)))
+write_entries_to_file_split = 50
+write_entries_to_file = $(shell rm -f $(2) || :) $(shell touch $(2)) \
+ $(foreach range, \
+ $(shell i=1; while test $$i -le $(words $(1)); do \
+ echo $$i; i=`expr $$i + $(write_entries_to_file_split)`; done), \
+ $(shell echo $(wordlist $(range), \
+ $(shell expr $(range) + $(write_entries_to_file_split) - 1), $(1)) \
+ | tr ' ' '\n' >> $(2)))
## ################################################################
[-- Attachment #3: p2 --]
[-- Type: text/plain, Size: 8589 bytes --]
ChangeLog:
2008-03-08 Ralf Wildenhues <Ralf.Wildenhues@gmx.de>
* configure.ac: m4_include config/proginstall.m4.
* configure: Regenerate.
config/ChangeLog:
2008-03-08 Ralf Wildenhues <Ralf.Wildenhues@gmx.de>
* proginstall.m4: New file, with fixed AC_PROG_INSTALL.
libjava/ChangeLog:
2008-03-08 Ralf Wildenhues <Ralf.Wildenhues@gmx.de>
* Makefile.am (install-data-local): Rewrite to be more efficient,
using awk, and `install' with multiple files at once. Do not gather
more than 50 files in one go to avoid hitting command line limits.
(install_data_local_split): New variable.
* aclocal.m4: Regenerate.
* configure: Likewise.
* Makefile.in: Likewise.
* gcj/Makefile.in: Likewise.
* include/Makefile.in: Likewise.
* testsuite/Makefile.in: Likewise.
boehm-gc/ChangeLog:
2008-03-08 Ralf Wildenhues <Ralf.Wildenhues@gmx.de>
* aclocal.m4: Regenerate.
* configure: Likewise.
* Makefile.in: Likewise.
* include/Makefile.in: Likewise.
libffi/ChangeLog:
2008-03-08 Ralf Wildenhues <Ralf.Wildenhues@gmx.de>
* aclocal.m4: Regenerate.
* configure: Likewise.
* Makefile.in: Likewise.
* include/Makefile.in: Likewise.
* testsuite/Makefile.in: Likewise.
libgfortran/ChangeLog:
2008-03-08 Ralf Wildenhues <Ralf.Wildenhues@gmx.de>
* aclocal.m4: Regenerate.
* configure: Likewise.
* Makefile.in: Likewise.
libgomp/ChangeLog:
2008-03-08 Ralf Wildenhues <Ralf.Wildenhues@gmx.de>
* aclocal.m4: Regenerate.
* configure: Likewise.
* Makefile.in: Likewise.
* testsuite/Makefile.in: Likewise.
libmudflap/ChangeLog:
2008-03-08 Ralf Wildenhues <Ralf.Wildenhues@gmx.de>
* aclocal.m4: Regenerate.
* configure: Likewise.
* Makefile.in: Likewise.
* testsuite/Makefile.in: Likewise.
libssp/ChangeLog:
2008-03-08 Ralf Wildenhues <Ralf.Wildenhues@gmx.de>
* aclocal.m4: Regenerate.
* configure: Likewise.
* Makefile.in: Likewise.
libstdc++-v3/ChangeLog:
2008-03-08 Ralf Wildenhues <Ralf.Wildenhues@gmx.de>
* aclocal.m4: Regenerate.
* configure: Likewise.
* Makefile.in: Likewise.
* doc/Makefile.in: Likewise.
* include/Makefile.in: Likewise.
* libmath/Makefile.in: Likewise.
* libsupc++/Makefile.in: Likewise.
* po/Makefile.in: Likewise.
* src/Makefile.in: Likewise.
* testsuite/Makefile.in: Likewise.
zlib/ChangeLog:
2008-03-08 Ralf Wildenhues <Ralf.Wildenhues@gmx.de>
* aclocal.m4: Regenerate.
* configure: Likewise.
* Makefile.in: Likewise.
diff --git a/config/proginstall.m4 b/config/proginstall.m4
new file mode 100644
index 0000000..a78a3d4
--- /dev/null
+++ b/config/proginstall.m4
@@ -0,0 +1,98 @@
+# AC_PROG_INSTALL
+# ---------------
+#
+# This macro is stolen from Autoconf 2.61a-341.
+# It requires `install' to be able to install multiple files at once.
+# This file will be obsolete when GCC moves to Autoconf 2.62.
+
+m4_version_prereq([2.62], [],
+[
+AC_DEFUN([AC_PROG_INSTALL],
+[AC_REQUIRE([AC_CONFIG_AUX_DIR_DEFAULT])dnl
+dnl AC_REQUIRE_AUX_FILE([install-sh])dnl This line does not work with 2.59.
+# Find a good install program. We prefer a C program (faster),
+# so one script is as good as another. But avoid the broken or
+# incompatible versions:
+# SysV /etc/install, /usr/sbin/install
+# SunOS /usr/etc/install
+# IRIX /sbin/install
+# AIX /bin/install
+# AmigaOS /C/install, which installs bootblocks on floppy discs
+# AIX 4 /usr/bin/installbsd, which doesn't work without a -g flag
+# AFS /usr/afsws/bin/install, which mishandles nonexistent args
+# SVR4 /usr/ucb/install, which tries to use the nonexistent group "staff"
+# OS/2's system install, which has a completely different semantic
+# ./install, which can be erroneously created by make from ./install.sh.
+# Reject install programs that cannot install multiple files.
+AC_MSG_CHECKING([for a BSD-compatible install])
+if test -z "$INSTALL"; then
+AC_CACHE_VAL(ac_cv_path_install,
+[_AS_PATH_WALK([$PATH],
+[# Account for people who put trailing slashes in PATH elements.
+case $as_dir/ in
+ ./ | .// | /[cC]/* | \
+ /etc/* | /usr/sbin/* | /usr/etc/* | /sbin/* | /usr/afsws/bin/* | \
+ ?:[\\/]os2[\\/]install[\\/]* | ?:[\\/]OS2[\\/]INSTALL[\\/]* | \
+ /usr/ucb/* ) ;;
+ *)
+ # OSF1 and SCO ODT 3.0 have their own names for install.
+ # Don't use installbsd from OSF since it installs stuff as root
+ # by default.
+ for ac_prog in ginstall scoinst install; do
+ for ac_exec_ext in '' $ac_executable_extensions; do
+ if AS_EXECUTABLE_P(["$as_dir/$ac_prog$ac_exec_ext"]); then
+ if test $ac_prog = install &&
+ grep dspmsg "$as_dir/$ac_prog$ac_exec_ext" >/dev/null 2>&1; then
+ # AIX install. It has an incompatible calling convention.
+ :
+ elif test $ac_prog = install &&
+ grep pwplus "$as_dir/$ac_prog$ac_exec_ext" >/dev/null 2>&1; then
+ # program-specific install script used by HP pwplus--don't use.
+ :
+ else
+ rm -rf conftest.one conftest.two conftest.dir
+ echo one > conftest.one
+ echo two > conftest.two
+ mkdir conftest.dir
+ if "$as_dir/$ac_prog$ac_exec_ext" -c conftest.one conftest.two "`pwd`/conftest.dir" &&
+ test -s conftest.one && test -s conftest.two &&
+ test -s conftest.dir/conftest.one &&
+ test -s conftest.dir/conftest.two
+ then
+ ac_cv_path_install="$as_dir/$ac_prog$ac_exec_ext -c"
+ break 3
+ fi
+ fi
+ fi
+ done
+ done
+ ;;
+esac])
+rm -rf conftest.one conftest.two conftest.dir
+])dnl
+ if test "${ac_cv_path_install+set}" = set; then
+ INSTALL=$ac_cv_path_install
+ else
+ # As a last resort, use the slow shell script. Don't cache a
+ # value for INSTALL within a source directory, because that will
+ # break other packages using the cache if that directory is
+ # removed, or if the value is a relative name.
+ INSTALL=$ac_install_sh
+ fi
+fi
+dnl Do special magic for INSTALL instead of AC_SUBST, to get
+dnl relative names right.
+AC_MSG_RESULT([$INSTALL])
+
+# Use test -z because SunOS4 sh mishandles braces in ${var-val}.
+# It thinks the first close brace ends the variable substitution.
+test -z "$INSTALL_PROGRAM" && INSTALL_PROGRAM='${INSTALL}'
+AC_SUBST(INSTALL_PROGRAM)dnl
+
+test -z "$INSTALL_SCRIPT" && INSTALL_SCRIPT='${INSTALL}'
+AC_SUBST(INSTALL_SCRIPT)dnl
+
+test -z "$INSTALL_DATA" && INSTALL_DATA='${INSTALL} -m 644'
+AC_SUBST(INSTALL_DATA)dnl
+])# AC_PROG_INSTALL
+])
diff --git a/configure.ac b/configure.ac
index 315fa3b..d908cac 100644
--- a/configure.ac
+++ b/configure.ac
@@ -19,6 +19,7 @@
### WARNING: this file contains embedded tabs. Do not run untabify on this file.
sinclude(config/acx.m4)
+m4_include(config/proginstall.m4)
AC_INIT(move-if-change)
AC_PREREQ(2.59)
diff --git a/libjava/Makefile.am b/libjava/Makefile.am
index 5b84c17..ec05b19 100644
--- a/libjava/Makefile.am
+++ b/libjava/Makefile.am
@@ -484,6 +484,7 @@ endif BUILD_ECJ1
## Install the headers. It is fairly ugly that we have to do this by
## hand.
+install_data_local_split = 50
install-data-local:
$(PRE_INSTALL)
## Install the .pc file.
@@ -496,12 +497,23 @@ install-data-local:
## length limit of some shells.
@echo Creating list of headers to install...
@: $(call write_entries_to_file,$(nat_headers_install) $(extra_headers),tmp-ilist)
- @cat tmp-ilist | while read f; do \
- d="`echo $$f | sed -e 's,/[^/]*$$,,'`"; \
- $(mkinstalldirs) $(DESTDIR)$(gxx_include_dir)/$$d; \
- if test -f $(srcdir)/$$f; then p=$(srcdir)/$$f; else p=$$f; fi; \
- echo " $(INSTALL_DATA) $$p $(DESTDIR)$(gxx_include_dir)/$$f"; \
- $(INSTALL_DATA) $$p $(DESTDIR)$(gxx_include_dir)/$$f; \
+ @srcdirstrip=`echo "$(srcdir)" | sed 's/[].[^$$\\*]/\\\\&/g'`; \
+ while read file; do echo "$$file $$file"; done < tmp-ilist | \
+ sed "s| $$srcdirstrip/| |;"' / .*\//!s/ .*/ ./; s,\( .*\)/[^/]*$$,\1,' | \
+ $(AWK) 'BEGIN { files["."] = "" } \
+ { files[$$2] = files[$$2] " " $$1; \
+ if (++n[$$2] == $(install_data_local_split)) { print $$2, files[$$2]; n[$$2] = 0; files[$$2] = "" } } \
+ END { for (dir in files) print dir, files[dir] }' | \
+ while read dir files; do \
+ xfiles=; for file in $$files; do \
+ if test -f "$$file"; then xfiles="$$xfiles $$file"; \
+ else xfiles="$$xfiles $(srcdir)/$$file"; fi; done; \
+ test -z "$$xfiles" || { \
+ test "x$$dir" = x. || { \
+ echo "$(mkinstalldirs) $(DESTDIR)$(gxx_include_dir)/$$dir"; \
+ $(mkinstalldirs) $(DESTDIR)$(gxx_include_dir)/$$dir; }; \
+ echo " $(INSTALL_DATA) $$xfiles $(DESTDIR)$(gxx_include_dir)/$$dir"; \
+ $(INSTALL_DATA) $$xfiles $(DESTDIR)$(gxx_include_dir)/$$dir; }; \
done
-@rm -f tmp-ilist
## Install inner class headers.
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: [PATCH] libjava: faster 'make install'
2008-03-09 14:58 [PATCH] libjava: faster 'make install' Ralf Wildenhues
@ 2008-03-09 15:50 ` Tom Tromey
2008-03-09 19:22 ` Paolo Bonzini
0 siblings, 1 reply; 3+ messages in thread
From: Tom Tromey @ 2008-03-09 15:50 UTC (permalink / raw)
To: Ralf Wildenhues; +Cc: gcc-patches, java-patches
>>>>> "Ralf" == Ralf Wildenhues <Ralf.Wildenhues@gmx.de> writes:
Ralf> OK for mainline?
The libjava (and related: boehm-gc, libffi, zlib) parts are ok, thanks
very much. Someone else will have to approve the other bits in the
second patch.
Tom
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: [PATCH] libjava: faster 'make install'
2008-03-09 15:50 ` Tom Tromey
@ 2008-03-09 19:22 ` Paolo Bonzini
0 siblings, 0 replies; 3+ messages in thread
From: Paolo Bonzini @ 2008-03-09 19:22 UTC (permalink / raw)
To: Tom Tromey; +Cc: Ralf Wildenhues, gcc-patches, java-patches
Tom Tromey wrote:
>>>>>> "Ralf" == Ralf Wildenhues <Ralf.Wildenhues@gmx.de> writes:
>
> Ralf> OK for mainline?
>
> The libjava (and related: boehm-gc, libffi, zlib) parts are ok, thanks
> very much. Someone else will have to approve the other bits in the
> second patch.
They are ok.
Paolo
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2008-03-09 19:22 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2008-03-09 14:58 [PATCH] libjava: faster 'make install' Ralf Wildenhues
2008-03-09 15:50 ` Tom Tromey
2008-03-09 19:22 ` Paolo Bonzini
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).