Too much memory in chan/select2.go used

Too much memory in chan/select2.go used

[Uros Bizjak]

On Wed, Feb 1, 2012 at 10:27 PM, Ian Lance Taylor <iant@google.com> wrote:

>> (BTW: Do you have any idea on what to do with excessive memory usage
>> in chan/select2.go? )
>
> At this point I don't.  It's sort of peculiar.  Sending an int on a
> channel should not use any memory.  The test is careful to only measure
> the memory allocated for sending and receiving, and as far as I can see
> nothing else should be allocated during the test.  You reported that the
> test was allocating 2098576 bytes.  When I run it I see it allocating
> 1408 bytes on x86_64, 640 bytes on i386.  2098576 is much larger than
> either number.  What is allocating that memory?
>
> In other words, there appears to be a real bug here.  You can probably
> track it down by setting a breakpoint on runtime_mallocgc after the line
>        runtime.MemStats.Alloc = 0
> What is calling runtime_mallocgc?

The backtrace says:

After a couple of calls from __go_new_channel, __go_new,__go_alloc,
the test starts to loop with allocations from newselect:


Breakpoint 2, runtime_mallocgc (size=156, flag=0, dogc=1, zeroed=1) at
../../../gcc-svn/trunk/libgo/runtime/malloc.goc:41
41              m = runtime_m();
(gdb) bt
#0  runtime_mallocgc (size=156, flag=0, dogc=1, zeroed=1) at
../../../gcc-svn/trunk/libgo/runtime/malloc.goc:41
#1  0x0000020000510364 in runtime_mal (n=156) at
../../../gcc-svn/trunk/libgo/runtime/malloc.goc:414
#2  0x0000020000506424 in newselect (selp=<synthetic pointer>, size=2)
at ../../../gcc-svn/trunk/libgo/runtime/chan.c:644
#3  runtime.newselect (size=<optimized out>) at
../../../gcc-svn/trunk/libgo/runtime/chan.c:c630
#4  0x0000000120001634 in main.receiver (c=0xf84020e000,
dummy=0xf84020e7c0, n=100000)
    at /space/uros/gcc-svn/trunk/gcc/testsuite/go.test/test/chan/select2.go:19
#5  0x0000000120001824 in main.main () at
/space/uros/gcc-svn/trunk/gcc/testsuite/go.test/test/chan/select2.go:36

I didn't check where memory gets de-allocated, but there are really a
lot of above calls.

Uros.

Re: Too much memory in chan/select2.go used

[Uros Bizjak]

On Wed, Feb 1, 2012 at 10:59 PM, Uros Bizjak <ubizjak@gmail.com> wrote:
> On Wed, Feb 1, 2012 at 10:27 PM, Ian Lance Taylor <iant@google.com> wrote:
>
>>> (BTW: Do you have any idea on what to do with excessive memory usage
>>> in chan/select2.go? )
>>
>> At this point I don't.  It's sort of peculiar.  Sending an int on a
>> channel should not use any memory.  The test is careful to only measure
>> the memory allocated for sending and receiving, and as far as I can see
>> nothing else should be allocated during the test.  You reported that the
>> test was allocating 2098576 bytes.  When I run it I see it allocating
>> 1408 bytes on x86_64, 640 bytes on i386.  2098576 is much larger than
>> either number.  What is allocating that memory?
>>
>> In other words, there appears to be a real bug here.  You can probably
>> track it down by setting a breakpoint on runtime_mallocgc after the line
>>        runtime.MemStats.Alloc = 0
>> What is calling runtime_mallocgc?

Some more debugging reveal the difference between alpha and x86_64.
Alpha does not implement split stacks, so soon after
"runtime.MemStats.Alloc = 0" line, we allocate exactly 2MB fake stack
via:

Breakpoint 5, runtime_mallocgc (size=2097152, flag=6, dogc=0,
zeroed=0) at ../../../gcc-svn/trunk/libgo/runtime/malloc.goc:41
41              m = runtime_m();
(gdb) bt
#0  runtime_mallocgc (size=2097152, flag=6, dogc=0, zeroed=0) at
../../../gcc-svn/trunk/libgo/runtime/malloc.goc:41
#1  0x000002000050d4b0 in runtime_malg (stacksize=<optimized out>,
ret_stack=0xf840205f70, ret_stacksize=0xf840205f68)
    at ../../../gcc-svn/trunk/libgo/runtime/proc.c:1166
#2  0x000002000050e3b8 in __go_go (fn=0x1200016b0 <main.$thunk1>,
arg=0xf8400001f0) at ../../../gcc-svn/trunk/libgo/runtime/proc.c:1218
#3  0x0000000120001968 in main.main () at select2.go:46
#4  0x000002000050e800 in runtime_main () at
../../../gcc-svn/trunk/libgo/runtime/proc.c:371

So, short of XFAILing the test on non-split stack targets, I have no
other idea how to handle this testcase failure.

Uros.

Re: Too much memory in chan/select2.go used

[Ian Lance Taylor]

[-- Attachment #1: Type: text/plain, Size: 840 bytes --]

Uros Bizjak <ubizjak@gmail.com> writes:

> Some more debugging reveal the difference between alpha and x86_64.
> Alpha does not implement split stacks, so soon after
> "runtime.MemStats.Alloc = 0" line, we allocate exactly 2MB fake stack
> via:

Thanks for tracking this down.

> So, short of XFAILing the test on non-split stack targets, I have no
> other idea how to handle this testcase failure.

I thought about it for a while, and I can't think of any better approach
than to skip this test case.  A simple xfail doesn't work because it
xfails the compilation of the test case rather than the execution,
leading to an XPASS and a FAIL.

I have committed this patch.

Ian


2012-02-08  Ian Lance Taylor  <iant@google.com>

	* go.test/go-test.exp (go-gc-tests): Don't run chan/select2.go on
	systems which don't support -fsplit-stack.



[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: patch --]
[-- Type: text/x-diff, Size: 782 bytes --]

Index: gcc/testsuite/go.test/go-test.exp
===================================================================
--- gcc/testsuite/go.test/go-test.exp	(revision 184032)
+++ gcc/testsuite/go.test/go-test.exp	(working copy)
@@ -33,6 +33,7 @@
 
 load_lib go-dg.exp
 load_lib go-torture.exp
+load_lib target-supports.exp
 
 # Compare two files
 proc filecmp { file1 file2 testname } {
@@ -329,6 +330,15 @@ proc go-gc-tests { } {
 	    continue
 	}
 
+	if { [file tail $test] == "select2.go" && \
+		 ! [check_effective_target_split_stack] } {
+	    # chan/select2.go fails on targets without split stack,
+	    # because they allocate a large stack segment that blows
+	    # out the memory calculations.
+	    untested $name
+	    continue
+	}
+
 	set fd [open $test r]
 
 	set lines_ok 1

Re: Too much memory in chan/select2.go used

[Uros Bizjak]

On Thu, Feb 9, 2012 at 7:43 AM, Ian Lance Taylor <iant@google.com> wrote:
> Uros Bizjak <ubizjak@gmail.com> writes:
>
>> Some more debugging reveal the difference between alpha and x86_64.
>> Alpha does not implement split stacks, so soon after
>> "runtime.MemStats.Alloc = 0" line, we allocate exactly 2MB fake stack
>> via:
>
> Thanks for tracking this down.
>
>> So, short of XFAILing the test on non-split stack targets, I have no
>> other idea how to handle this testcase failure.
>
> I thought about it for a while, and I can't think of any better approach
> than to skip this test case.  A simple xfail doesn't work because it
> xfails the compilation of the test case rather than the execution,
> leading to an XPASS and a FAIL.

Thanks!

BTW: I'd like to point out that we can use go-execute-xfail in this
case, so the case is compiled but is known to fail execution for some
reason.

Uros.

Re: Too much memory in chan/select2.go used

[Ian Lance Taylor]

Uros Bizjak <ubizjak@gmail.com> writes:

> BTW: I'd like to point out that we can use go-execute-xfail in this
> case, so the case is compiled but is known to fail execution for some
> reason.

Thanks, I didn't know about that one.

I think I'll leave it as is for now because I'm treating stack.go the
same way, and stack.go does sometimes pass even without -fsplit-stack.

Ian