Re: Upated memory hog patch for make

Re: Upated memory hog patch for make

[Dan Kegel]

Gerald wrote:
>On Wed, 1 Feb 2006, H. J. Lu wrote:
>> My memory hog patch for make has 2 typos. This patch fixes them.
>Thanks, H. J.  What's the upstream status of your patches?

I think they're in upstream (hopefully H.J. will confirm that).
I know that the O(N^2) bug that Jeff Evarts found and fixed
is in upstream.

If you want to try, there's a release candidate out now:

> From: psmith@...
> Sent: 20 February 2006 04:25
> To: bug-make@...; help-make@...; make-w32@...
> Subject: GNU make 3.81rc1 released -- please test.
>
> Please find the first release candidate for GNU make 3.81, 3.81rc1,
> available now for download from ftp://alpha.gnu.org/gnu/make:
>
>  c907a044ebe7dff19f56f8dbb829cd3f  ftp://alpha.gnu.org/gnu/make/make-3.81rc1.tar.bz2

- Dan

--
Wine for Windows ISVs: http://kegel.com/wine/isv

Re: Upated memory hog patch for make

[H. J. Lu]

On Mon, Feb 20, 2006 at 10:51:13AM -0800, Dan Kegel wrote:
> Gerald wrote:
> >On Wed, 1 Feb 2006, H. J. Lu wrote:
> >> My memory hog patch for make has 2 typos. This patch fixes them.
> >Thanks, H. J.  What's the upstream status of your patches?
> 
> I think they're in upstream (hopefully H.J. will confirm that).
> I know that the O(N^2) bug that Jeff Evarts found and fixed
> is in upstream.
> 
> > Sent: 20 February 2006 04:25
> > To: bug-make@...; help-make@...; make-w32@...
> > Subject: GNU make 3.81rc1 released -- please test.
> >
> > Please find the first release candidate for GNU make 3.81, 3.81rc1,
> > available now for download from ftp://alpha.gnu.org/gnu/make:
> >
> >  c907a044ebe7dff19f56f8dbb829cd3f  ftp://alpha.gnu.org/gnu/make/make-3.81rc1.tar.bz2

3.81rc1 works much better than 3.80. But my patch for 3.80 no longer
applies cleanly. Paul is aware of the libjava build problem and is
working on it.


H.J.

Re: Upated memory hog patch for make

[Gerald Pfeifer]

On Mon, 20 Feb 2006, H. J. Lu wrote:
> 3.81rc1 works much better than 3.80. But my patch for 3.80 no longer
> applies cleanly. Paul is aware of the libjava build problem and is
> working on it.

That's good news, thanks!  I'll give 3.81 RC1 a try, and hope we'll
have a release soon...

Gerald

Re: Upated memory hog patch for make

[Gerald Pfeifer]

On Wed, 1 Feb 2006, H. J. Lu wrote:
> My memory hog patch for make has 2 typos. This patch fixes them.

Thanks, H. J.  What's the upstream status of your patches?  Did you
submit your updated patch there as well?

(As long as this patch, or a similar change, has not become part of a 
released version of GNU make, some distributions like SUSE will carry your 
patch and thus reduce the overall pain point to address the problem in 
libgcj itself, whereas many users will still run into the problem.)

Gerald

Upated memory hog patch for make

[H. J. Lu]

[-- Attachment #1: Type: text/plain, Size: 437 bytes --]

On Wed, Feb 01, 2006 at 06:33:40PM +0100, Klaus Singvogel wrote:
> 
> Hello Mr. H. J. Lu,
> I just want to inform you, that we got a problem report regarding your
> "memory hog" patch from 2005-Dec-08 for gmake-3.80. If compilation of
> "binutils" is started with "MALLOC_CHECK_=2", a crash in gmake occurs.
> 
> I digged a bit deeper into it and noticed this:
> 

My memory hog patch for make has 2 typos. This patch fixes them.


H.J.

[-- Attachment #2: make-3.80-memory-2.patch --]
[-- Type: text/plain, Size: 6379 bytes --]

--- make-3.80/file.c.memory	2002-10-03 19:13:42.000000000 -0700
+++ make-3.80/file.c	2006-02-01 10:45:32.000000000 -0800
@@ -434,7 +434,7 @@ snap_deps ()
 	    if (d->file == 0)
 	      d->file = enter_file (d->name);
 	    else
-	      free (d->name);
+	      hash_strfree (d->name);
 	    d->name = 0;
 	  }
   free (file_slot_0);
--- make-3.80/implicit.c.memory	2002-09-04 00:26:19.000000000 -0700
+++ make-3.80/implicit.c	2006-02-01 10:45:32.000000000 -0800
@@ -539,7 +539,7 @@ pattern_search (file, archive, depth, re
 	      dep->file = enter_file (dep->name);
               /* enter_file uses dep->name _if_ we created a new file.  */
               if (dep->name != dep->file->name)
-                free (dep->name);
+                hash_strfree (dep->name);
 	      dep->name = 0;
 	      dep->file->tried_implicit |= dep->changed;
 	    }
--- make-3.80/main.c.memory	2002-08-09 18:27:17.000000000 -0700
+++ make-3.80/main.c	2006-02-01 10:45:32.000000000 -0800
@@ -501,6 +501,7 @@ initialize_global_hash_tables ()
   init_hash_files ();
   hash_init_directories ();
   hash_init_function_table ();
+  init_hash_strings ();
 }
 
 static struct file *
--- make-3.80/make.h.memory	2002-09-11 09:55:44.000000000 -0700
+++ make-3.80/make.h	2006-02-01 10:45:32.000000000 -0800
@@ -427,6 +427,11 @@ extern char *find_char_unquote PARAMS ((
 extern char *find_percent PARAMS ((char *));
 extern FILE *open_tmpfile PARAMS ((char **, const char *));
 
+extern void init_hash_strings PARAMS ((void));
+extern char *hash_strdup PARAMS ((const char *));
+extern char *hash_savestring PARAMS ((const char *, unsigned int));
+extern void hash_strfree PARAMS ((char *));
+
 #ifndef NO_ARCHIVES
 extern int ar_name PARAMS ((char *));
 extern void ar_parse_name PARAMS ((char *, char **, char **));
--- make-3.80/misc.c.memory	2002-09-12 15:15:58.000000000 -0700
+++ make-3.80/misc.c	2006-02-01 11:05:44.000000000 -0800
@@ -18,8 +18,10 @@ along with GNU Make; see the file COPYIN
 the Free Software Foundation, Inc., 59 Temple Place - Suite 330,
 Boston, MA 02111-1307, USA.  */
 
+#include <assert.h>
 #include "make.h"
 #include "dep.h"
+#include "hash.h"
 #include "debug.h"
 
 /* Variadic functions.  We go through contortions to allow proper function
@@ -564,7 +566,7 @@ copy_dep_chain (d)
       c = (struct dep *) xmalloc (sizeof (struct dep));
       bcopy ((char *) d, (char *) c, sizeof (struct dep));
       if (c->name != 0)
-	c->name = xstrdup (c->name);
+	c->name = hash_strdup (c->name);
       c->next = 0;
       if (firstnew == 0)
 	firstnew = lastnew = c;
@@ -891,3 +893,154 @@ atomic_readdir(dir)
 }
 
 #endif  /* HAVE_BROKEN_RESTART */
+
+/* Hash table of duplicated strings.  */
+
+struct hash_string
+{
+  char *string;
+  unsigned int count;
+};
+
+static unsigned long
+string_hash_1 (key)
+    const void *key;
+{
+  return_ISTRING_HASH_1 (((const struct hash_string *) key)->string);
+}
+
+static unsigned long
+string_hash_2 (key)
+    const void *key;
+{
+  return_ISTRING_HASH_2 (((const struct hash_string *) key)->string);
+}
+
+static int
+string_hash_cmp (x, y)
+    const void *x;
+    const void *y;
+{
+  return_ISTRING_COMPARE (((const struct hash_string *) x)->string,
+			  ((const struct hash_string *) y)->string);
+}
+
+static struct hash_table strings;
+
+void
+init_hash_strings ()
+{
+  hash_init (&strings, 1000, string_hash_1, string_hash_2,
+	     string_hash_cmp);
+}
+
+/* Keep track duplicated string and return the old one if exists.  */
+
+char *
+hash_strdup (ptr)
+     const char *ptr;
+{
+  struct hash_string *h, key;
+  
+  if (*ptr == '\0')
+    return "";
+
+  key.string = (char *) ptr;
+  key.count = 0;
+  h = (struct hash_string *) hash_find_item (&strings, &key);
+  if (h == NULL)
+    {
+      char *result = (char *) malloc (strlen (ptr) + 1);
+
+      if (result == NULL)
+	fatal (NILF, _("virtual memory exhausted"));
+
+      strcpy (result, ptr);
+
+      h = (struct hash_string *) malloc (sizeof (struct hash_string));
+      if (h == NULL)
+	fatal (NILF, _("virtual memory exhausted"));
+
+      h->string = result;
+      h->count = 1;
+      hash_insert (&strings, h);
+    }
+  else
+    {
+      h->count++;
+      assert (h->count != 0);
+    }
+  
+  return h->string;
+}
+
+char *
+hash_savestring (str, length)
+     const char *str;
+     unsigned int length;
+{
+  struct hash_string *h, key;
+  
+  if (length == 0 || *str == '\0')
+    return "";
+
+  key.string = alloca (length + 1);
+  key.count = 0;
+  bcopy (str, key.string, length);
+  key.string [length] = '\0';
+
+  h = (struct hash_string *) hash_find_item (&strings, &key);
+  if (h == NULL)
+    {
+      char *out = (char *) xmalloc (length + 1);
+      bcopy (str, out, length);
+      out[length] = '\0';
+
+      h = (struct hash_string *) malloc (sizeof (struct hash_string));
+      if (h == NULL)
+	fatal (NILF, _("virtual memory exhausted"));
+
+      h->string = out;
+      h->count = 1;
+      hash_insert (&strings, h);
+    }
+  else
+    {
+      h->count++;
+      assert (h->count != 0);
+    }
+  
+  return h->string;
+}
+
+void
+hash_strfree (ptr)
+     char *ptr;
+{
+  struct hash_string *h, key;
+  
+  if (*ptr == '\0')
+    return;
+
+  key.string = ptr;
+  key.count = 0;
+  h = (struct hash_string *) hash_find_item (&strings, &key);
+
+  /* Check if string comes from hash_strdup or hash_savestring.  */
+  if (h == NULL || h->string != ptr)
+    {
+      free (ptr);
+      return;
+    }
+
+  h->count--;
+  if (h->count == 0)
+    {
+      struct hash_string *d;
+
+      d = hash_delete (&strings, h);
+      assert (d == h);
+      free (h->string);
+      free (h);
+    }
+}
--- make-3.80/read.c.memory	2006-02-01 10:45:32.000000000 -0800
+++ make-3.80/read.c	2006-02-01 10:45:32.000000000 -0800
@@ -1871,8 +1871,8 @@ record_files (filenames, pattern, patter
                     fatal (flocp,
                            _("target `%s' leaves prerequisite pattern empty"),
                            name);
-		  free (d->name);
-		  d->name = savestring (buffer, o - buffer);
+		  hash_strfree (d->name);
+		  d->name = hash_savestring (buffer, o - buffer);
 		}
 	    }
 	}
@@ -2017,7 +2017,7 @@ record_files (filenames, pattern, patter
 	      while (d != 0)
 		{
 		  struct dep *nextd = d->next;
- 		  free (d->name);
+ 		  hash_strfree (d->name);
  		  free ((char *)d);
 		  d = nextd;
 		}