Call for testers: libiberty/physmem.c

Call for testers: libiberty/physmem.c

[Kaveh R. Ghazi]

Now that the libiberty bits are installed, I'd like to see how many
systems physmem.c works on.

So far these are my results:

Works?	Platform
YES	sparc-solaris2.7
YES	hpux11
YES	x86-linux-gnu
NO	mips-irix6.5

I've already worked out something with the *utils maintainer for
irix6 which I'll import from upstream.

Given that x86-linux-gnu works, I'm assuming all linux-gnu does and
probably all glibc systems in general.  But you never know.

If people would please link the code below against the most recent
libiberty.a from the trunk and/or 3.3 and tell me the results I would
appreciate it.  (Zeros are bad.)

I'm especially interested in the evaluation platforms listed in here.
http://gcc.gnu.org/gcc-3.1/criteria.html
E.g. *bsd, aix, cygwin,

plus alpha-osf, hpux10, darwin and anywhere else you host gcc.


If your favorite platform yields zero, don't panic, the GC heuristics
will simply back down to the old default values.  But if you can
provide suggestions on how to improve physmem.c for your platform I'd
very much appreciate it.  Note physmem.c must run as non-root, so
peeking into /dev/kmem etc. is not appropriate.

		Thanks,
		--Kaveh


#include <stdio.h>

extern double physmem_total(void);
extern double physmem_available(void);

int main(void)
{
  printf ("total=%fMb avail=%fMb\n",
	  physmem_total() / (1024 * 1024),
	  physmem_available() / (1024 * 1024));
  
  return 0;
}

Re: Call for testers: libiberty/physmem.c

[Andrew Pinski]

On `OpenBSD grendel 3.1 GENERIC#4 i386' aka i?86-unknown-openbsd3.1:
total=0.000000Mb avail=0.000000Mb

but `sysctl -A' gives:
hw.physmem = 536059904
hw.usermem = 535629824
hw.pagesize = 4096

I think this might be the same problem on all *BSD (including darwin).

Thanks,
Andrew Pinski


On Friday, Feb 21, 2003, at 06:28 US/Pacific, Kaveh R. Ghazi wrote:

> Now that the libiberty bits are installed, I'd like to see how many
> systems physmem.c works on.
>
> So far these are my results:
>
> Works?	Platform
> YES	sparc-solaris2.7
> YES	hpux11
> YES	x86-linux-gnu
> NO	mips-irix6.5
>
> I've already worked out something with the *utils maintainer for
> irix6 which I'll import from upstream.
>
> Given that x86-linux-gnu works, I'm assuming all linux-gnu does and
> probably all glibc systems in general.  But you never know.
>
> If people would please link the code below against the most recent
> libiberty.a from the trunk and/or 3.3 and tell me the results I would
> appreciate it.  (Zeros are bad.)
>
> I'm especially interested in the evaluation platforms listed in here.
> http://gcc.gnu.org/gcc-3.1/criteria.html
> E.g. *bsd, aix, cygwin,
>
> plus alpha-osf, hpux10, darwin and anywhere else you host gcc.
>
>
> If your favorite platform yields zero, don't panic, the GC heuristics
> will simply back down to the old default values.  But if you can
> provide suggestions on how to improve physmem.c for your platform I'd
> very much appreciate it.  Note physmem.c must run as non-root, so
> peeking into /dev/kmem etc. is not appropriate.
>
> 		Thanks,
> 		--Kaveh
>
>
> #include <stdio.h>
>
> extern double physmem_total(void);
> extern double physmem_available(void);
>
> int main(void)
> {
>   printf ("total=%fMb avail=%fMb\n",
> 	  physmem_total() / (1024 * 1024),
> 	  physmem_available() / (1024 * 1024));
>
>   return 0;
> }
>

Re: Call for testers: libiberty/physmem.c

[Karel Gardas]

On Fri, 21 Feb 2003, Kaveh R. Ghazi wrote:

> Now that the libiberty bits are installed, I'd like to see how many
> systems physmem.c works on.

I would like to give it a try, but my gcc-3_3-branch build unfortunatelly
fails... I'm tuned to see if it really speedup C++ compilation :-)

> If people would please link the code below against the most recent
> libiberty.a from the trunk and/or 3.3 and tell me the results I would
> appreciate it.  (Zeros are bad.)

I have quite common x86 Debian GNU/Linux 3.0, but anyway, here is numbers
for you:

thinkpad:/tmp/physmem$ ./pmem
total=503.476562Mb avail=24.187500Mb
thinkpad:/tmp/physmem$

Cheers,

Karel
--
Karel Gardas                  kgardas@objectsecurity.com
ObjectSecurity Ltd.           http://www.objectsecurity.com

Re: Call for testers: libiberty/physmem.c

[Andreas Tobler]

Kaveh R. Ghazi wrote:

> plus alpha-osf, hpux10, darwin and anywhere else you host gcc.
> 
Hm, don't have sys/pstat.h

Andreas

[titanium:~/gcc-work] andreast% ./phymem
total=0.000000Mb avail=0.000000Mb

titanium:~/gcc-work] andreast% hostinfo
Mach kernel version:
          Darwin Kernel Version 6.3:

Kernel configured for up to 2 processors.
1 processor is physically available.
Processor type: ppc7400 (PowerPC 7400)
Processor active: 0
Primary memory available: 512.00 megabytes.

Re: Call for testers: libiberty/physmem.c

[Krzysztof Parzyszek]

On Fri, Feb 21, 2003 at 09:28:52AM -0500, Kaveh R. Ghazi wrote:
> Now that the libiberty bits are installed, I'd like to see how many
> systems physmem.c works on.
> 
> So far these are my results:
> 
> Works?	Platform
> YES	sparc-solaris2.7
> YES	hpux11
> YES	x86-linux-gnu
> NO	mips-irix6.5

On FreeBSD 4.8-PRERELEASE it doesn't work (prints zeros).  FreeBSD
doesn't provide physical memory information via sysconf, you can use
sysctl instead.

Krzysztof
 

Re: Call for testers: libiberty/physmem.c

[Kaveh R. Ghazi]

 > From: Andrew Pinski <apinski@apple.com>
 > 
 > On `OpenBSD grendel 3.1 GENERIC#4 i386' aka i?86-unknown-openbsd3.1:
 > total=0.000000Mb avail=0.000000Mb
 > 
 > but `sysctl -A' gives:
 > hw.physmem = 536059904
 > hw.usermem = 535629824
 > hw.pagesize = 4096
 > 
 > I think this might be the same problem on all *BSD (including darwin).

Is `sysctl' a binary program or a system call?  (Or both?)

What would the C interface look like?  What autoconf magic does it need?

		Thanks,
		--Kaveh
--
Kaveh R. Ghazi			ghazi@caip.rutgers.edu

Re: Call for testers: libiberty/physmem.c

[Kaveh R. Ghazi]

 > From: Andreas Tobler <toa@pop.agri.ch>
 > 
 > Kaveh R. Ghazi wrote:
 > 
 > > plus alpha-osf, hpux10, darwin and anywhere else you host gcc.
 > > 
 > Hm, don't have sys/pstat.h
 > Andreas

I think pstat.h is an hpux thing.


 > [titanium:~/gcc-work] andreast% ./phymem
 > total=0.000000Mb avail=0.000000Mb
 > 
 > titanium:~/gcc-work] andreast% hostinfo
 > Mach kernel version:
 >           Darwin Kernel Version 6.3:

Do you have sysctl?  Is it usable for this?

--
Kaveh R. Ghazi			ghazi@caip.rutgers.edu

Re: Call for testers: libiberty/physmem.c

[Kaveh R. Ghazi]

 > From: Krzysztof Parzyszek <kparz@iastate.edu>
 > 
 > On Fri, Feb 21, 2003 at 09:28:52AM -0500, Kaveh R. Ghazi wrote:
 > > Now that the libiberty bits are installed, I'd like to see how many
 > > systems physmem.c works on.
 > > 
 > > So far these are my results:
 > > 
 > > Works?	Platform
 > > YES	sparc-solaris2.7
 > > YES	hpux11
 > > YES	x86-linux-gnu
 > > NO	mips-irix6.5
 > 
 > On FreeBSD 4.8-PRERELEASE it doesn't work (prints zeros).  FreeBSD
 > doesn't provide physical memory information via sysconf, you can use
 > sysctl instead.
 > Krzysztof

How exactly?  Thanks.

--
Kaveh R. Ghazi			ghazi@caip.rutgers.edu

Re: Call for testers: libiberty/physmem.c

[Andreas Tobler]

Kaveh R. Ghazi wrote:

>  > [titanium:~/gcc-work] andreast% ./phymem
>  > total=0.000000Mb avail=0.000000Mb
>  > 
>  > titanium:~/gcc-work] andreast% hostinfo
>  > Mach kernel version:
>  >           Darwin Kernel Version 6.3:
> 
> Do you have sysctl?  Is it usable for this?

'sysctl -A'

hw.physmem = 536870912
hw.usermem = 484876288
hw.pagesize = 4096

Same as Andrew reported for *BSD.

Let me know for further tests.
Andreas

Re: Call for testers: libiberty/physmem.c

[Richard Earnshaw]

>  > From: Andrew Pinski <apinski@apple.com>
>  > 
>  > On `OpenBSD grendel 3.1 GENERIC#4 i386' aka i?86-unknown-openbsd3.1:
>  > total=0.000000Mb avail=0.000000Mb
>  > 
>  > but `sysctl -A' gives:
>  > hw.physmem = 536059904
>  > hw.usermem = 535629824
>  > hw.pagesize = 4096
>  > 
>  > I think this might be the same problem on all *BSD (including darwin).
> 
> Is `sysctl' a binary program or a system call?  (Or both?)
> 
> What would the C interface look like?  What autoconf magic does it need?
> 
> 		Thanks,
> 		--Kaveh
> --
> Kaveh R. Ghazi			ghazi@caip.rutgers.edu

Both.  See:

http://netbsd.gw.com/cgi-bin/man-cgi?sysctl+3+NetBSD-1.6

For NetBSD you would need CTL_HW and HW_PHYSMEM or HW_USERMEM.  

R.

Re: Call for testers: libiberty/physmem.c

[Andreas Tobler]

Kaveh R. Ghazi wrote:
> Is `sysctl' a binary program or a system call?  (Or both?)
> 
> What would the C interface look like?  What autoconf magic does it need?

As I understand both sysctl(3) and sysctl(8).

The below is from the darwin man pages top.

HISTORY
      sysctl first appeared in 4.4BSD.

Andreas

 From the man page 3:
SYSCTL(3)               System Library Functions Manual

NAME
      sysctl, sysctlbyname, sysctlnametomib - get or set system information

LIBRARY
      Standard C Library (libc, -lc)

SYNOPSIS
      #include <sys/types.h>
      #include <sys/sysctl.h>

      int
      sysctl(int *name, u_int namelen, void *oldp, size_t *oldlenp, void 
*newp,
              size_t newlen);
.....

SYSCTL(8)                   System Manager's Manual

NAME
      sysctl - get or set kernel state

SYNOPSIS
      sysctl [-n] name ...
      sysctl [-n] -w name=value ...
      sysctl [-n] -aA

...

Re: Call for testers: libiberty/physmem.c

[Krzysztof Parzyszek]

On Fri, Feb 21, 2003 at 12:00:15PM -0500, Kaveh R. Ghazi wrote:
>  > From: Krzysztof Parzyszek <kparz@iastate.edu>
>  > 
>  > On FreeBSD 4.8-PRERELEASE it doesn't work (prints zeros).  FreeBSD
>  > doesn't provide physical memory information via sysconf, you can use
>  > sysctl instead.
>  > Krzysztof
> 
> How exactly?  Thanks.

Here is a sample code that works on FreeBSD-STABLE.  The values returned
by sysctl are said to be in bytes and so I don't know what happens when
you have >= 2G of available memory.

If you want to check out manual pages for sysctl(3) for various *BSD
systems, you can find them on http://www.freebsd.org/docs.html#man

Krzysztof



#include <stdio.h>
#include <sys/types.h>
#include <sys/sysctl.h>

int main() {
    int mib[2];
    int physmem, usermem, pagesize;
    size_t len;

    mib[0] = CTL_HW;

    mib[1] = HW_PHYSMEM;
    len = sizeof(int);
    if (sysctl(mib, 2, &physmem, &len, NULL, 0) != 0) perror("sysctl");

    mib[1] = HW_USERMEM;
    len = sizeof(int);
    if (sysctl(mib, 2, &usermem, &len, NULL, 0) != 0) perror("sysctl");

    mib[1] = HW_PAGESIZE;
    len = sizeof(int);
    if (sysctl(mib, 2, &pagesize, &len, NULL, 0) != 0) perror("sysctl");

    printf("physmem = %d bytes\nusermem = %d bytes\npagesize = %d bytes\n",
        physmem, usermem, pagesize);

}

Re: Call for testers: libiberty/physmem.c

[Gerald Pfeifer]

On Fri, 21 Feb 2003, Krzysztof Parzyszek wrote:
> Here is a sample code that works on FreeBSD-STABLE.  The values returned
> by sysctl are said to be in bytes and so I don't know what happens when
> you have >= 2G of available memory.

  physmem = -140066816 bytes
  usermem = -291561472 bytes
  pagesize = 4096 bytes

With the patch at the end, it works as expected:

  physmem = 4154900480 bytes
  usermem = 4003393536 bytes
  pagesize = 4096 bytes

Gerald

--- x.c	Fri Feb 21 18:54:11 2003
+++ y.c	Fri Feb 21 18:55:10 2003
@@ -5,5 +5,5 @@
 int main() {
     int mib[2];
-    int physmem, usermem, pagesize;
+    unsigned physmem, usermem, pagesize;
     size_t len;

@@ -22,5 +22,5 @@
     if (sysctl(mib, 2, &pagesize, &len, NULL, 0) != 0) perror("sysctl");

-    printf("physmem = %d bytes\nusermem = %d bytes\npagesize = %d bytes\n",
+    printf("physmem = %u bytes\nusermem = %u bytes\npagesize = %u bytes\n",
         physmem, usermem, pagesize);

Re: Call for testers: libiberty/physmem.c

[Krzysztof Parzyszek]

On Fri, Feb 21, 2003 at 06:58:43PM +0100, Gerald Pfeifer wrote:
> On Fri, 21 Feb 2003, Krzysztof Parzyszek wrote:
> > Here is a sample code that works on FreeBSD-STABLE.  The values returned
> > by sysctl are said to be in bytes and so I don't know what happens when
> > you have >= 2G of available memory.
> 
> With the patch at the end, it works as expected:

[patch changes `int' to `unsigned int']

Funny thing is that the variable from which (I believe) the hw.physmem
is extracted was declared as `int' in the kernel.

I looked at /usr/src/sys/i386/i386/machdep.c, variable physmem.
This applies to -STABLE, I think -CURRENT does it differently.

Anyway, if it works, we're good.


Krzysztof

Re: Call for testers: libiberty/physmem.c

[Kelley Cook]

In article <200302211428.h1LESqP17581@caip.rutgers.edu>, 
ghazi@caip.rutgers.edu says...
> Now that the libiberty bits are installed, I'd like to see how many
> systems physmem.c works on.
> 
> So far these are my results:
> 
> Works?	Platform
> YES	sparc-solaris2.7
> YES	hpux11
> YES	x86-linux-gnu
> NO	mips-irix6.5
> 

On a 512MB Windows XP/SP1 machine 

under Cygwin 1.3.20

$ ./kavtest.exe
total=511.484375Mb avail=511.929688Mb

under mingw32 (1.0.7)

$ ./kavtest.exe 
total=0.000000Mb avail=0.000000Mb

Re: Call for testers: libiberty/physmem.c

[Joel Sherrill]

Kelley Cook wrote:
> 
> In article <200302211428.h1LESqP17581@caip.rutgers.edu>,
> ghazi@caip.rutgers.edu says...
> > Now that the libiberty bits are installed, I'd like to see how many
> > systems physmem.c works on.
> >
> > So far these are my results:
> >
> > Works?        Platform
> > YES   sparc-solaris2.7
> > YES   hpux11
> > YES   x86-linux-gnu
> > NO    mips-irix6.5
> >
> 
> On a 512MB Windows XP/SP1 machine
> 
> under Cygwin 1.3.20
> 
> $ ./kavtest.exe
> total=511.484375Mb avail=511.929688Mb

The available is higher than the total. Weird.

> under mingw32 (1.0.7)
> 
> $ ./kavtest.exe
> total=0.000000Mb avail=0.000000Mb

-- 
Joel Sherrill, Ph.D.             Director of Research & Development
joel@OARcorp.com                 On-Line Applications Research
Ask me about RTEMS: a free RTOS  Huntsville AL 35805
Support Available                (256) 722-9985

Re: Call for testers: libiberty/physmem.c

[Kaveh R. Ghazi]

 > From: Joel Sherrill <joel dot sherrill at OARcorp dot com> 
 > 
 > Kelley Cook wrote:
 > 
 > > On a 512MB Windows XP/SP1 machine
 > > 
 > > under Cygwin 1.3.20
 > > 
 > > $ ./kavtest.exe
 > > total=511.484375Mb avail=511.929688Mb
 > 
 > The available is higher than the total. Weird.

Yeah, I got the same cygwin report privately from Roger.  It's either
a bug in cygwin or they consider "available" mem to be
e.g. RAM+swap. (?)  For our (GCC) purposes it doesn't matter since the
"total" mem figure is correct enough for our purposes.

 > > under mingw32 (1.0.7)
 > > 
 > > $ ./kavtest.exe
 > > total=0.000000Mb avail=0.000000Mb

Is there a way to fix this?

--
Kaveh R. Ghazi			ghazi@caip.rutgers.edu

Re: Call for testers: libiberty/physmem.c

[Richard Earnshaw]

One of our Linux compute servers is reporting

$ tmp/phys
total=7820.195312Mb avail=933.585938Mb

Which is correct, but not necessarily useful (an x86 processor can't 
address more then 4G in any one process).

R.

Re: Call for testers: libiberty/physmem.c

[Kaveh R. Ghazi]

 > From: Richard Earnshaw <rearnsha@arm.com>
 > 
 > One of our Linux compute servers is reporting
 > 
 > $ tmp/phys
 > total=7820.195312Mb avail=933.585938Mb
 > 
 > Which is correct, but not necessarily useful (an x86 processor can't 
 > address more then 4G in any one process).
 > R.

The heuristic currently tops out at RAM==1G, more than that doesn't
(shouldn't) affect the algorithm.

On a 16G solaris2 box, my values are expand=100%, heap=128M (== 1G/8).

Try compiling any file with -v and examine the GC heuristic values it
outputs.

		--Kaveh
--
Kaveh R. Ghazi			ghazi@caip.rutgers.edu

Re: Call for testers: libiberty/physmem.c

[Nix]

[moved from gcc-patches]
On Fri, 21 Feb 2003, Kaveh R. Ghazi moaned:
> Now that the libiberty bits are installed, I'd like to see how many
> systems physmem.c works on.
> 
> So far these are my results:
> 
> Works?	Platform
> YES	sparc-solaris2.7
> YES	hpux11
> YES	x86-linux-gnu
> NO	mips-irix6.5
> 
> I've already worked out something with the *utils maintainer for
> irix6 which I'll import from upstream.
> 
> Given that x86-linux-gnu works, I'm assuming all linux-gnu does and
> probably all glibc systems in general.  But you never know.
> 
> If people would please link the code below against the most recent
> libiberty.a from the trunk and/or 3.3 and tell me the results I would
> appreciate it.  (Zeros are bad.)

Well, on sparc-unknown-linux-gnu (UltraSPARC II with a 32-bit userspace
and 512Mb of RAM), I see:

nix@amaterasu 325 /tmp% ./physmemtest 
total=497.445312Mb avail=6.593750Mb

so it seems to work indeed.

As expected, `avail' is almost totally useless, but the total is
certainly correct (modulo amount consumed by the kernel).

Now if someone will test on an uncommon platform (S/390? IA64?
MIPS/glibc, if that still works?) we can be sure...

-- 
2003-02-01: the day the STS died.

Re: Call for testers: libiberty/physmem.c

[Marc Espie]

In article <20030221174040.GA80208@antares.student.iastate.edu> you write:
>Here is a sample code that works on FreeBSD-STABLE.  The values returned
>by sysctl are said to be in bytes and so I don't know what happens when
>you have >= 2G of available memory.
>
>If you want to check out manual pages for sysctl(3) for various *BSD
>systems, you can find them on http://www.freebsd.org/docs.html#man
>
>Krzysztof
>


>#include <stdio.h>
>#include <sys/types.h>
#include <sys/param.h>
>#include <sys/sysctl.h>
>
>int main() {
>    int mib[2];
>    int physmem, usermem, pagesize;
>    size_t len;
>
>    mib[0] = CTL_HW;
>
>    mib[1] = HW_PHYSMEM;
>    len = sizeof(int);
>    if (sysctl(mib, 2, &physmem, &len, NULL, 0) != 0) perror("sysctl");
>
>    mib[1] = HW_USERMEM;
>    len = sizeof(int);
>    if (sysctl(mib, 2, &usermem, &len, NULL, 0) != 0) perror("sysctl");
>
>    mib[1] = HW_PAGESIZE;
>    len = sizeof(int);
>    if (sysctl(mib, 2, &pagesize, &len, NULL, 0) != 0) perror("sysctl");
>
>    printf("physmem = %d bytes\nusermem = %d bytes\npagesize = %d bytes\n",
>        physmem, usermem, pagesize);
>
>}
>
>


And it's good on OpenBSD as well (and almost certainly NetBSD)

Re: Call for testers: libiberty/physmem.c

[Mike Stump]

On Friday, February 21, 2003, at 06:28 AM, Kaveh R. Ghazi wrote:
> Now that the libiberty bits are installed, I'd like to see how many
> systems physmem.c works on.

On darwin, I am getting:

ulimit -a:
datasize        6144 kbytes

though, I am not sure what is causing it.  The causes gcc to be really 
conservative about memory on darwin (4MB on 512MB machine), normally it 
would use 64MB or so.

Does anyone know where this comes from?

[ I send this here, just so that people can know about the issue.  I'll 
ask around internally as well.  ]

Re: Call for testers: libiberty/physmem.c

[Kaveh R. Ghazi]

 > From: Mike Stump <mstump@apple.com>
 > 
 > On Friday, February 21, 2003, at 06:28 AM, Kaveh R. Ghazi wrote:
 > > Now that the libiberty bits are installed, I'd like to see how many
 > > systems physmem.c works on.
 > 
 > On darwin, I am getting:
 > 
 > ulimit -a:
 > datasize        6144 kbytes
 > 
 > though, I am not sure what is causing it.  The causes gcc to be really 
 > conservative about memory on darwin (4MB on 512MB machine), normally it 
 > would use 64MB or so.
 > 
 > Does anyone know where this comes from?
 > 
 > [ I send this here, just so that people can know about the issue.  I'll 
 > ask around internally as well.  ]

That's really strange.  I don't know how you ever got gcc to work on
that particular box.  Just the xmalloc'ed stuff alone will eat more
than 6M on decent sized input file, let alone any additional heap used
for GC memory.

--
Kaveh R. Ghazi			ghazi@caip.rutgers.edu

Re: Call for testers: libiberty/physmem.c

[Mike Stump]

On Monday, February 24, 2003, at 06:06 PM, Kaveh R. Ghazi wrote:
> That's really strange.  I don't know how you ever got gcc to work on
> that particular box.

Not the box, the OS.  Conclusion, it isn't actually enforced.  Hum...  
Do we want to add code to gcc to ignore DATA on darwin?

The reason is that all boxes, all users currently have this set 
artifically low, everyone else just ignores it, maybe gcc should as 
well?

:-(

Even if I could get a change to the OS to set this better, for the next 
1-6 years, users will have the `old' value.

Another possibility would be for the compiler to bump it up by itself 
if it is too small, also icky, but that mirrors what we do with the 
stack currently.

And last, we could just leave everything as is, and tell users to put 
ulimit -d unlimited in their .profile.

Re: Call for testers: libiberty/physmem.c

[Zack Weinberg]

Mike Stump <mstump@apple.com> writes:

> On Monday, February 24, 2003, at 06:06 PM, Kaveh R. Ghazi wrote:
>> That's really strange.  I don't know how you ever got gcc to work on
>> that particular box.
>
> Not the box, the OS.  Conclusion, it isn't actually enforced.  Hum...
> Do we want to add code to gcc to ignore DATA on darwin?
>
> The reason is that all boxes, all users currently have this set
> artifically low, everyone else just ignores it, maybe gcc should as
> well?

I have a theory.

My theory is that, on Darwin, RLIMIT_DATA is applied only to memory
allocation via brk(), but both malloc() and ggc-page.c are getting
their memory from mmap() or equivalent.  Possibly malloc uses the real
Mach memory-allocation primitive, whose name I do not remember?

One way to confirm or falsify this theory would be to write a mini
program that allocates and dirties memory, using malloc, and see if
it maxes out at RLIMIT_DATA or RLIMIT_AS.

And one plausible thing to do about it, if the theory proves true,
is to ignore RLIMIT_DATA on systems that define RLIMIT_AS.

zw

Re: Call for testers: libiberty/physmem.c

[Jason R Thorpe]

On Mon, Feb 24, 2003 at 06:34:31PM -0800, Zack Weinberg wrote:

 > I have a theory.
 > 
 > My theory is that, on Darwin, RLIMIT_DATA is applied only to memory
 > allocation via brk(), but both malloc() and ggc-page.c are getting
 > their memory from mmap() or equivalent.  Possibly malloc uses the real
 > Mach memory-allocation primitive, whose name I do not remember?

This is a good theory.  Anonymous mmap pages were mis-accounted in Mach VM
and in NetBSD's UVM for a while.

-- 
        -- Jason R. Thorpe <thorpej@wasabisystems.com>

Re: Call for testers: libiberty/physmem.c

[Richard Earnshaw]

> On Monday, February 24, 2003, at 06:06 PM, Kaveh R. Ghazi wrote:
> > That's really strange.  I don't know how you ever got gcc to work on
> > that particular box.
> 
> Not the box, the OS.  Conclusion, it isn't actually enforced.  Hum...  
> Do we want to add code to gcc to ignore DATA on darwin?
> 
> The reason is that all boxes, all users currently have this set 
> artifically low, everyone else just ignores it, maybe gcc should as 
> well?
> 
> :-(
> 
> Even if I could get a change to the OS to set this better, for the next 
> 1-6 years, users will have the `old' value.
> 
> Another possibility would be for the compiler to bump it up by itself 
> if it is too small, also icky, but that mirrors what we do with the 
> stack currently.
> 
> And last, we could just leave everything as is, and tell users to put 
> ulimit -d unlimited in their .profile.
> 

We could add code to get gcc to raise the limit to the hard limit.  
There's already code in the kernel to do this for the core size, eg 
toplev.c:

#if defined(HAVE_SETRLIMIT)
  {
    struct rlimit rlim;
    if (getrlimit (RLIMIT_CORE, &rlim) != 0)
      fatal_io_error ("getting core file size maximum limit");
    rlim.rlim_cur = rlim.rlim_max;
    if (setrlimit (RLIMIT_CORE, &rlim) != 0)
      fatal_io_error ("setting core file size limit to maximum");
  }
#endif

R.

Re: Call for testers: libiberty/physmem.c

[Kaveh R. Ghazi]

 > From: Mike Stump <mstump@apple.com>
 > 
 > On Monday, February 24, 2003, at 06:06 PM, Kaveh R. Ghazi wrote:
 > > That's really strange.  I don't know how you ever got gcc to work on
 > > that particular box.
 > 
 > Not the box, the OS.  Conclusion, it isn't actually enforced.  Hum...  
 > Do we want to add code to gcc to ignore DATA on darwin?
 > 
 > The reason is that all boxes, all users currently have this set 
 > artifically low, everyone else just ignores it, maybe gcc should as 
 > well?
 > 
 > :-(
 > 
 > Even if I could get a change to the OS to set this better, for the next 
 > 1-6 years, users will have the `old' value.
 > 
 > Another possibility would be for the compiler to bump it up by itself 
 > if it is too small, also icky, but that mirrors what we do with the 
 > stack currently.

If we do this, I'd prefer to do it only for Darwin.  If someone set's
it intentionally, we should honor it if possible.  Since Darwin
ignores the limit, resetting it doesn't hurt there.

What's the default hard limit shipped on Darwin?  I.e. is it high
enough such that resetting rlim_cur to rlim_max fixes the issue?

		--Kaveh
--
Kaveh R. Ghazi			ghazi@caip.rutgers.edu

Re: Call for testers: libiberty/physmem.c

[Mike Stump]

On Monday, February 24, 2003, at 06:34 PM, Zack Weinberg wrote:
> I have a theory.
>
> My theory is that, on Darwin, RLIMIT_DATA is applied only to memory
> allocation via brk(),

A multitude of bad behavior stops me from testing your theory.  sbrk 
wraps and never fails.  Touching memory dies way too soon, not sure 
why.  [ trying harder ]  Ok, no, it doesn't apply in that case either.  
It is ignored.

> One way to confirm or falsify this theory would be to write a mini
> program that allocates and dirties memory, using malloc, and see if
> it maxes out at RLIMIT_DATA or RLIMIT_AS.

malloc swicthes over to mmap for larger things.  When I use things that 
are small enough, neither stop it from allocating more than these 
limits.

Re: Call for testers: libiberty/physmem.c

[Mike Stump]

On Tuesday, February 25, 2003, at 08:07 AM, Kaveh R. Ghazi wrote:
> If we do this, I'd prefer to do it only for Darwin.  If someone set's
> it intentionally, we should honor it if possible.  Since Darwin
> ignores the limit, resetting it doesn't hurt there.
>
> What's the default hard limit shipped on Darwin?  I.e. is it high
> enough such that resetting rlim_cur to rlim_max fixes the issue?

8 [ tilt your head for that ]

Yes.

Re: Call for testers: libiberty/physmem.c

[Michael Matz]

Hi,

On Thu, 27 Feb 2003, Mike Stump wrote:

> >>> total=496.429688Mb avail=8.648438Mb
> >
> > It's on s390-ibm-linux, the values look pretty good.
>
> Hum, I'd expect more than 8 available.  If you like it, you like it...
> But, do you like it?

That's on linux, and there this number is based on MemFree, which is
excluding buffers and cache.  The actual number of free bytes is much
higher (therefore we should never do anything based on _that_ avail
number).


Ciao,
Michael.

Re: Call for testers: libiberty/physmem.c

[Mike Stump]

On Thursday, February 27, 2003, at 06:17 AM, Gerhard Tonn wrote:
>>> total=496.429688Mb avail=8.648438Mb
>
> It's on s390-ibm-linux, the values look pretty good.

Hum, I'd expect more than 8 available.  If you like it, you like it...  
But, do you like it?

Re: Call for testers: libiberty/physmem.c

[Gerhard Tonn]

On Thursday 27 February 2003 14:59, you wrote:
> > >Now if someone will test on an uncommon platform (S/390? IA64?
> > >
>  > >MIPS/glibc, if that still works?) we can be sure...
>  >
>  > I get the following on S/390
>  >
>  > total=496.429688Mb avail=8.648438Mb
>  >
>  > Gerhard
>
> Thanks.  Which S/390 OS did you get this on?  (What's your target
> triplet?)  Is 496Mb roughly accurate given your hardware?
>

It's on s390-ibm-linux, the values look pretty good.

Gerhard

Re: Call for testers: libiberty/physmem.c

[Kaveh R. Ghazi]

 > >Now if someone will test on an uncommon platform (S/390? IA64?
 > >MIPS/glibc, if that still works?) we can be sure...
 > 
 > I get the following on S/390
 > 
 > total=496.429688Mb avail=8.648438Mb
 > 
 > Gerhard

Thanks.  Which S/390 OS did you get this on?  (What's your target
triplet?)  Is 496Mb roughly accurate given your hardware?

--
Kaveh R. Ghazi			ghazi@caip.rutgers.edu

Re: Call for testers: libiberty/physmem.c

[Gerhard Tonn]

>On Fri, 21 Feb 2003, Kaveh R. Ghazi moaned:
>> Now that the libiberty bits are installed, I'd like to see how many
>> systems physmem.c works on.
>> 
>> So far these are my results:
>> 
>> Works?        Platform
>> YES   sparc-solaris2.7
>> YES   hpux11
>> YES   x86-linux-gnu
>> NO    mips-irix6.5
>> 
>> I've already worked out something with the *utils maintainer for
>> irix6 which I'll import from upstream.
>> 
>> Given that x86-linux-gnu works, I'm assuming all linux-gnu does and
>> probably all glibc systems in general.  But you never know.
>> 
>> If people would please link the code below against the most recent
>> libiberty.a from the trunk and/or 3.3 and tell me the results I would
>> appreciate it.  (Zeros are bad.)

>Now if someone will test on an uncommon platform (S/390? IA64?
>MIPS/glibc, if that still works?) we can be sure...

I get the following on S/390

total=496.429688Mb avail=8.648438Mb

Gerhard

Re: Call for testers: libiberty/physmem.c

[Marco Morandini]

> Now if someone will test on an uncommon platform (S/390? IA64?
> MIPS/glibc, if that still works?) we can be sure...
> 

Platform (8 Gb RAM):
ia64-unknown-linux-gnu

Result:
total=8106.156250Mb avail=6938.234375Mb

Regards,

Marco Morandini