[Bug other/55517] New: [ASAN] ASAN doesn't work with (soft) ulimit on virtual memory

[Bug other/55517] New: [ASAN] ASAN doesn't work with (soft) ulimit on virtual memory

[rguenth at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=55517

             Bug #: 55517
           Summary: [ASAN] ASAN doesn't work with (soft) ulimit on virtual
                    memory
    Classification: Unclassified
           Product: gcc
           Version: 4.8.0
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: other
        AssignedTo: unassigned@gcc.gnu.org
        ReportedBy: rguenth@gcc.gnu.org


The asan runtime should raise any soft limit on virtual memory by the amount
of shadow virtual mappings it creates.  Those are not accounted for in any
limit calculation.

If a hard limit is present it should probably report that instead of failing
with a cryptic message that it cannot allocate memory:
==10720== ERROR: Failed to allocate 0x20000001000 (2199023259648) bytes at
address 0x0ffffffff000 (12)

[Bug other/55517] [ASAN] ASAN doesn't work with (soft) ulimit on virtual memory

[konstantin.s.serebryany at gmail dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=55517

Konstantin Serebryany <konstantin.s.serebryany at gmail dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |konstantin.s.serebryany at
                   |                            |gmail dot com

--- Comment #1 from Konstantin Serebryany <konstantin.s.serebryany at gmail dot com> 2012-11-28 12:43:32 UTC ---
I am quite sure that asan should not mess with the limits itself.
It gets too messy too soon. (e.g. in tsan we try to reexec if the stack is
unlimited, but it's not reliable). 

I agree, the messages could be less cryptic.

[Bug other/55517] [ASAN] ASAN doesn't work with (soft) ulimit on virtual memory

[rguenth at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=55517

--- Comment #2 from Richard Biener <rguenth at gcc dot gnu.org> 2012-11-28 12:48:56 UTC ---
(In reply to comment #1)
> I am quite sure that asan should not mess with the limits itself.
> It gets too messy too soon. (e.g. in tsan we try to reexec if the stack is
> unlimited, but it's not reliable). 

Well, raising a soft virtual limit should be straight-forward if you
create such large mappings.  SLES ships with a default soft virtual limit,
and only asan knows how much extra shadow memory it needs.

> I agree, the messages could be less cryptic.

It btw causes c-c++-common/asan/memcmp-1.c to FAIL in all my tests
which is quite annoying because the addresses printed in the message
vary and thus testresults are not comparable.

[Bug other/55517] [ASAN] ASAN doesn't work with (soft) ulimit on virtual memory

[konstantin.s.serebryany at gmail dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=55517

--- Comment #3 from Konstantin Serebryany <konstantin.s.serebryany at gmail dot com> 2012-11-28 12:50:09 UTC ---
[The component for such bugs should be 'sanitizer' but for some reason I can't
change it]

[Bug sanitizer/55517] [ASAN] ASAN doesn't work with (soft) ulimit on virtual memory

[rguenth at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=55517

Richard Biener <rguenth at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |dodji at gcc dot gnu.org,
                   |                            |dvyukov at gcc dot gnu.org,
                   |                            |jakub at gcc dot gnu.org,
                   |                            |kcc at gcc dot gnu.org
          Component|other                       |sanitizer

--- Comment #4 from Richard Biener <rguenth at gcc dot gnu.org> 2012-11-28 12:52:55 UTC ---
(In reply to comment #3)
> [The component for such bugs should be 'sanitizer' but for some reason I can't
> change it]

Ah, I searched for libasan and didn't find it.

You can manage bugs if you login with your gcc.gnu.org account (or in the
first place create one with your gcc.gnu.org e-mail address).

[Bug sanitizer/55517] [ASAN] ASAN doesn't work with (soft) ulimit on virtual memory

[konstantin.s.serebryany at gmail dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=55517

--- Comment #5 from Konstantin Serebryany <konstantin.s.serebryany at gmail dot com> 2012-11-28 12:56:53 UTC ---
We try to minimize the number of syscalls we make in asan run-time. 
One reason for that is that asan may run in a sanbox which disallows some of
them. (Another is just the code bloat)

Why simply removing all the limits before running asan doesn't work? 
Better warning messages will make it move obvious.

[Bug sanitizer/55517] [ASAN] ASAN doesn't work with (soft) ulimit on virtual memory

[jakub at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=55517

--- Comment #6 from Jakub Jelinek <jakub at gcc dot gnu.org> 2012-11-28 13:00:27 UTC ---
I think raising soft limit is a standard approach done in multiple places, even
gcc itself does that, and is far better than just crashing.  Unlike
RLIMIT_STACK, it doesn't cause significant problems if you enlarge RLIMIT_AS
during the runtime of a program as opposed to making sure it is big enough
already at process start.

You don't need to call getrlimit unconditionally, all you can do is upon mmap
MAP_FAILED on these 3 large mmap calls you try to do getrlimit and if hard
limit is bigger than what is needed and soft limit smaller, increase, setrlimit
and retry the mmap.

[Bug sanitizer/55517] [ASAN] ASAN doesn't work with (soft) ulimit on virtual memory

[kcc at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=55517

--- Comment #7 from kcc at gcc dot gnu.org 2012-11-28 13:17:21 UTC ---
BTW, the messages are actually quite nice: 

==22487== ERROR: Failed to allocate 0x20000001000 (2199023259648) bytes at
address 0x0ffffffff000 (12)
==22487== ReserveShadowMemoryRange failed while trying to map 0x20000001000
bytes. Perhaps you're using ulimit -v

[Bug sanitizer/55517] [ASAN] ASAN doesn't work with (soft) ulimit on virtual memory

[anlauf at gmx dot de]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=55517

Harald Anlauf <anlauf at gmx dot de> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |anlauf at gmx dot de

--- Comment #8 from Harald Anlauf <anlauf at gmx dot de> ---
(In reply to Kostya Serebryany from comment #7)
> BTW, the messages are actually quite nice: 
> 
> ==22487== ERROR: Failed to allocate 0x20000001000 (2199023259648) bytes at
> address 0x0ffffffff000 (12)
> ==22487== ReserveShadowMemoryRange failed while trying to map 0x20000001000
> bytes. Perhaps you're using ulimit -v

I'd like to try ASAN on a system where the sysadmins enforce
a hard limit.  Is this possible?

I've set the soft limit to the same value as the hard limit,
but keep getting similar errors.

[Bug sanitizer/55517] [ASAN] ASAN doesn't work with (soft) ulimit on virtual memory

[kcc at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=55517

--- Comment #9 from Kostya Serebryany <kcc at gcc dot gnu.org> ---
> I'd like to try ASAN on a system where the sysadmins enforce
> a hard limit.  Is this possible?

No. asan requires 20Tb (+ a bit) of *virtual* memory to properly function. 
talk to your sysadmins to relax their limits 

> 
> I've set the soft limit to the same value as the hard limit,
> but keep getting similar errors.

[Bug sanitizer/55517] [ASAN] ASAN doesn't work with (soft) ulimit on virtual memory

[kcc at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=55517

--- Comment #11 from Kostya Serebryany <kcc at gcc dot gnu.org> ---
> That will not happen.  So ASAN will be unusable for me unless
> it can be changed to work with a sane amount of virtual memory.

Correct.
I don't know any way to make ASAN work in the presence of hard VM limit.
It's simpler to change your sisadmin :)

[Bug sanitizer/55517] [ASAN] ASAN doesn't work with (soft) ulimit on virtual memory

[mrs at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=55517

mrs at gcc dot gnu.org <mrs at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |mrs at gcc dot gnu.org

--- Comment #12 from mrs at gcc dot gnu.org <mrs at gcc dot gnu.org> ---
This is annoying as it causes a large amount of chaff in the gcc/g++ testsuite
runs if one has not previously done a ulimit -v unlimited.

[Bug sanitizer/55517] [ASAN] ASAN doesn't work with (soft) ulimit on virtual memory

[kcc at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=55517

--- Comment #13 from Kostya Serebryany <kcc at gcc dot gnu.org> ---
(In reply to mrs@gcc.gnu.org from comment #12)
> This is annoying as it causes a large amount of chaff in the gcc/g++
> testsuite runs if one has not previously done a ulimit -v unlimited.

Patches (upstream) are welcome.