malloc problem

malloc problem

[wakamiya]

My name is Wakamiya from Japan.

I use Cygwin 1.1.6 with WindowsNT Japanese Edition and 
I have had a problem on malloc.
As I get a memory block larger than 0x8000000 using malloc,
malloc returns NULL pointer.
How do I get a memory block larger than 0x8000000 ?
Or is it a Windows limitation ?


Yours faithfully,

wakamiya@osaka.email.ne.jp

--
Want to unsubscribe from this list?
Check out: http://cygwin.com/ml/#unsubscribe-simple

Re: malloc problem

[Kazuhiro Fujieda]

>>> On Mon, 25 Dec 2000 09:51:40 +0900
>>> wakamiya <wakamiya@osaka.email.ne.jp> said:

> I use Cygwin 1.1.6 with WindowsNT Japanese Edition and 
> I have had a problem on malloc.
> As I get a memory block larger than 0x8000000 using malloc,
> malloc returns NULL pointer.
> How do I get a memory block larger than 0x8000000 ?
> Or is it a Windows limitation ?

It is a configurable limitation of the Cygwin DLL. You can
control it through a registry entry.

After you incorporate the following registry file, you can
allocate a memory block up to 256MB. Take care that this entry
isn't available until you terminate all applications of Cygwin.

Re: malloc problem

[wakamiya]

Hello Fujieda san.

Kazuhiro Fujieda <fujieda@jaist.ac.jp> wrote:
> It is a configurable limitation of the Cygwin DLL. You can
> control it through a registry entry.
I have modified registry entry using attached registry file and 
I am able to allocate a large memory block.
Thank you for your quick reply.

Yours faithfully,




--
Want to unsubscribe from this list?
Check out: http://cygwin.com/ml/#unsubscribe-simple

Re: malloc() problem

[Greg Smith]

Christopher Faylor wrote:
> I've set the threshold up to a larger number in malloc and built a
> new snapshot.  This should work around the problem until we can figure
> out what is going wrong with mmap.  I've alerted our mmap maintainer
> to the problem.

Many thanks to you and Corinna for fixing this problem so promptly
(1.3.20-1).

Greg Smith


--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Bug reporting:         http://cygwin.com/bugs.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

Re: malloc() problem

[Christopher Faylor]

On Thu, Feb 06, 2003 at 05:35:01PM -0500, Greg Smith wrote:
>>>There's also a bug right now that causes allocation of three times as
>>>much memory as Hercules actually calls for.  (I believe that one's
>>>already been reported; it really hits Hercules hard, though, as it's
>>>not uncommon to have Hercules allocate 256 MB of memory for the
>>>emulated system's central storage, and a 768 MB allocation will drive
>>>most Windows boxes to their knees.)
>
>>I'm not aware of any bug report along these lines.  I must have missed
>>it.
>>
>>A test case would be welcome.
>
>One of our developers has come up with a test case for this problem
>which I have pasted below.  He's basically a windows programmer and
>I don't think he can write a short program to save his life ;-)

A test case is not a 218 line program.  Sorry.

I did write a very small test case which malloced 4M and saw the problem
you described by watching memory size go up in task manager.  The
problem seems to be due to the fact that the new version of malloc uses
mmap for requests over 256K and mmap is, for some reason, doubly
allocating private regions.

I've set the threshold up to a larger number in malloc and built a
new snapshot.  This should work around the problem until we can figure
out what is going wrong with mmap.  I've alerted our mmap maintainer
to the problem.

So, try the snapshot and see if it fixes your problem.

For your edification, the below is what I mean by a "test case".  A
"test case" should be source code which demonstrates the problem in
the smallest amount of code possible.  The below has an extra bell
or whistle but it at least should be understandable without a lot
of study.

cgf

#include <stdlib.h>
#include <unistd.h>

int
main(int argc, char **argv)
{
  char *p;
  unsigned size = strtoul (*++argv ?: "4194304", &p, 0);
  int i;
  printf ("pid %d\n", getpid());
  for (i = 0; i < 1000; i++)
    {
      char buf[16];
      malloc (size);
      puts ("ding");
      gets (buf);
    }
}
--
Please use the resources at cygwin.com rather than sending personal email.
Special for spam email harvesters: send email to aaaspam@sourceware.org
and be permanently blocked from mailing lists at sources.redhat.com

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Bug reporting:         http://cygwin.com/bugs.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/

malloc() problem

[Greg Smith]

 >>There's also a bug right now that causes allocation of three times as
 >>much memory as Hercules actually calls for.  (I believe that one's
 >>already been reported; it really hits Hercules hard, though, as it's
 >>not uncommon to have Hercules allocate 256 MB of memory for the
 >>emulated system's central storage, and a 768 MB allocation will drive
 >>most Windows boxes to their knees.)

 >I'm not aware of any bug report along these lines.  I must have missed
 >it.
 >
 >A test case would be welcome.

One of our developers has come up with a test case for this problem
which I have pasted below.  He's basically a windows programmer and
I don't think he can write a short program to save his life ;-)

The gist is, if you compile the program under cygwin, and use it
to malloc() say 4M then PageFileUsage from GetProcessMemoryInfo()
will be a little over 8M.  If you compile the program using a
windows compiler then the same value is reported back as a little
over 4M (using windows malloc).  Somehow, it seems, memory is
getting doubly committed for large requests.

I've been going through the code (last night's snapshot) but reading
malloc code is like trying to swim in molasses ;-)  I have figured
out that malloc calls mALLOc() in winsup/cygwin/malloc.cc... I haven't
yet determined which sbrk() is being called yet.  Once I do, I plan
to trace the VirtualAlloc() calls.

Thanks,
Greg Smith



/////////////////////////////////////////////////////////////////////////////////////////
// Cygwin double malloc research...
/////////////////////////////////////////////////////////////////////////////////////////

#include <stdio.h>			// (need printf)
#include <windows.h>		// (need Windows (duh!))
#include <Psapi.h>			// (need GetProcessMemoryInfo)

#ifdef WIN32
#pragma comment(lib,"Psapi.lib")
#else
#include <unistd.h>			// (need sleep)
#endif

/////////////////////////////////////////////////////////////////////////////////////////

int format()
{
	printf
	(
		"\n"

		"Format:\n\n"

		"  \"fishtest  <malloc_size>  <num_mallocs>  <sleep_secs>\"\n\n"

		"Where:\n\n"

		"  <malloc_size>  =  size of each malloc in 'K' (default is 1024)\n"
		"  <num_mallocs>  =  how many mallocs to do (default is 1, max 100)\n"
		"  <sleep_secs>   =  seconds to sleep before allocating, after allocating,\n"
		"                    and before freeing memory to give you time to manually\n"
		"                    examine system memory usage (default = 1)\n"
	);

	return 0;
}

/////////////////////////////////////////////////////////////////////////////////////////

void mysleep(int secs)
{
#ifdef WIN32
	Sleep(secs*1000);
#else
	sleep(secs);
#endif
}

/////////////////////////////////////////////////////////////////////////////////////////

void mem_info()
{
	// From SDK docs:

	/*
		The working set is the amount of memory physically mapped to
		the process context at a given time. Memory in the paged pool
		is system memory that can be transferred to the paging file on
		disk (paged) when it is not being used. Memory in the nonpaged
		pool is system memory that cannot be paged to disk as long as
		the corresponding objects are allocated. The pagefile usage
		represents how much memory is set aside for the process in the
		system paging file. When memory usage is too high, the virtual
		memory manager pages selected memory to disk. When a thread
		needs a page that is not in memory, the memory manager reloads
		it from the paging file.
	*/

	// Thus, all we're really interested in is the PagefileUsage value.

	PROCESS_MEMORY_COUNTERS  pmc;

	pmc.cb = sizeof(pmc);

	if (!GetProcessMemoryInfo(GetCurrentProcess(),&pmc,sizeof(pmc)))
		return;		// (must be running on Win9x)

	printf("\nCurrent PagefileUsage = %ld K   (peak: %ld K)\n\n",
		pmc.PagefileUsage/1024,pmc.PeakPagefileUsage/1024);
}

/////////////////////////////////////////////////////////////////////////////////////////

void virt_info (unsigned long *mem_free, unsigned long *mem_reserved, unsigned long *mem_committed)
{
	MEMORY_BASIC_INFORMATION memory_info;
	memory_info.BaseAddress = 0;
	*mem_free = *mem_reserved = *mem_committed = 0;
	while (VirtualQuery (memory_info.BaseAddress, &memory_info, sizeof (memory_info)))
	{
		switch (memory_info.State)
		{
			case MEM_FREE:
				*mem_free += memory_info.RegionSize;
				break;
			case MEM_RESERVE:
				*mem_reserved += memory_info.RegionSize;
				break;
			case MEM_COMMIT:
				*mem_reserved  += memory_info.RegionSize;
				*mem_committed += memory_info.RegionSize;
				break;
			default:
				printf("LOGIC ERROR IN virt_info FUNCTION\n");
				abort();
		}

		memory_info.BaseAddress = (char *) memory_info.BaseAddress + memory_info.RegionSize;
	}
}

/////////////////////////////////////////////////////////////////////////////////////////

int main (int argc, char* argv[])
{
	unsigned long mem_free, mem_reserved, mem_committed;

	int nMallocSizeInK = 1024;
	int nNumMallocs    = 1;
	int nSleepSecs     = 1;

	void*  pMemAddrs[100];
	int    i = 0;

	if (argc != 4)
		return format();

	nMallocSizeInK = atoi(argv[1]);
	nNumMallocs    = atoi(argv[2]);
	nSleepSecs     = atoi(argv[3]);

	if (0
		|| nMallocSizeInK <= 0
		|| nNumMallocs    <= 0
		|| nNumMallocs    > 100
		|| nSleepSecs     <= 0
	)
		return format();

	printf("%s %s %s %s\n\n",argv[0],argv[1],argv[2],argv[3]);

	// Report starting virtual memory usage...

	virt_info (           &mem_free,    &mem_reserved,    &mem_committed);
	printf ("start:         free=%7ld K,  reserved=%7ld K,  committed=%7ld K\n",
			            mem_free/1024,mem_reserved/1024,mem_committed/1024);
	mem_info();

	mysleep(nSleepSecs);

	// allocate the memory...

	for (i = 0; i < nNumMallocs; i++)
	{
		pMemAddrs[i] = malloc(nMallocSizeInK * 1024);

		if (!pMemAddrs[i])
		{
			printf("malloc #%d of %dK failed!\n",i,nMallocSizeInK);
		}
		else
			memset(pMemAddrs[i],0xCD,nMallocSizeInK * 1024);
	}

	// Report virtual memory usage after all memory allocations...

	virt_info (           &mem_free,    &mem_reserved,    &mem_committed);
	printf ("after mallocs: free=%7ld K,  reserved=%7ld K,  committed=%7ld K\n",
			            mem_free/1024,mem_reserved/1024,mem_committed/1024);
	mem_info();

	mysleep(nSleepSecs);

	// free the memory...

	for (i = 0; i < nNumMallocs; i++)
	{
		if (pMemAddrs[i])
			free(pMemAddrs[i]);
	}

	// Report virtual memory usage after freeing all memory allocations...

	virt_info (           &mem_free,    &mem_reserved,    &mem_committed);
	printf ("after frees:   free=%7ld K,  reserved=%7ld K,  committed=%7ld K\n",
			            mem_free/1024,mem_reserved/1024,mem_committed/1024);
	mem_info();

	mysleep(nSleepSecs);

	return 0;
}

/////////////////////////////////////////////////////////////////////////////////////////


--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Bug reporting:         http://cygwin.com/bugs.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/