From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 23920 invoked by alias); 4 Mar 2014 10:06:04 -0000 Mailing-List: contact gcc-help-help@gcc.gnu.org; run by ezmlm Precedence: bulk List-Id: List-Archive: List-Post: List-Help: Sender: gcc-help-owner@gcc.gnu.org Received: (qmail 23908 invoked by uid 89); 4 Mar 2014 10:06:03 -0000 Authentication-Results: sourceware.org; auth=none X-Virus-Found: No X-Spam-SWARE-Status: No, score=-1.9 required=5.0 tests=AWL,BAYES_00,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.2 X-HELO: mail-pd0-f169.google.com Received: from mail-pd0-f169.google.com (HELO mail-pd0-f169.google.com) (209.85.192.169) by sourceware.org (qpsmtpd/0.93/v0.84-503-g423c35a) with (AES128-SHA encrypted) ESMTPS; Tue, 04 Mar 2014 10:06:02 +0000 Received: by mail-pd0-f169.google.com with SMTP id fp1so4952756pdb.14 for ; Tue, 04 Mar 2014 02:06:00 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:message-id:date:from:user-agent:mime-version :newsgroups:to:cc:subject:references:in-reply-to:content-type :content-transfer-encoding; bh=jREShEZkI1P/XsW5PtGcAYlhNoin6kVUDZz9sqCx12Q=; b=ECELNPHZEnJaZyDbf2DZxyB6iFdUsOzsyjdaV3+NbiV+JzXZUHvT+WgvaZTrjlRtZC gHQw48bDwIac/jgmRVT5LH6Qnz5dU/9f5JnBD24GfmuESRaF672ycBgfmKnj+P1YFbDQ LJcnxW2ZveZPQy8ggfBsU+EKr1GZc6fHdQI/AOYHVoosz/AzeczWEPWHG/D4pJou5kR+ uie7+N6j25axUG9QAdsIqgXeGHS9D1QnDNEOE+3F9StD1ycYJEM8SS8H2iCURO5VcCuj HFsGyUklxzlkvj7w0HgtxD2gkLGrDhUrsIl1kq3+Xsvv1h63HuavWthR+zFUuNlyTdQ+ UrQw== X-Gm-Message-State: ALoCoQkk/DCHteP8qoD1TSjNOordNKFW9OWUpbDTroHHk4fS2p+Fyu+mZJ+8uCRvDqLjR93mH/x/ X-Received: by 10.66.8.169 with SMTP id s9mr4970635paa.101.1393927560484; Tue, 04 Mar 2014 02:06:00 -0800 (PST) Received: from [192.168.1.6] (ppp118-210-209-19.lns20.adl6.internode.on.net. [118.210.209.19]) by mx.google.com with ESMTPSA id ix2sm45987871pbc.45.2014.03.04.02.05.58 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Tue, 04 Mar 2014 02:05:59 -0800 (PST) Message-ID: <5315A582.7070600@toojays.net> Date: Tue, 04 Mar 2014 10:06:00 -0000 From: John Steele Scott User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.3.0 MIME-Version: 1.0 Newsgroups: gmane.comp.gcc.help To: Ian Lance Taylor CC: gcc-help@gcc.gnu.org Subject: Re: C++ exceptions with many threads and no memory. References: In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-SW-Source: 2014-03/txt/msg00013.txt.bz2 On 04/03/14 17:05, Ian Lance Taylor wrote: > On Mon, Mar 3, 2014 at 3:53 AM, John Steele Scott wrote: >> >> When comparing this to GCC's exception handling, I find that libstdc++ calls malloc() whenever an exception is thrown, and falls back to statically allocated memory only when malloc fails. Only 64 statically allocated exceptions may be in flight at once, and if this limit is violated, std::terminate() is called. >> >> This seems like quite a brittle behaviour to me. I guess not often seen as a problem in practice because Linux overcommit usually ensures malloc() won't fail (although if it does, then you have to take your chances with the oom-killer anyway). >> >> There doesn't seem to be any way around this, is there? Do GCC developers consider it a bug? > > If you have a good proposal for how to handle this situation, I'm sure > that GCC developers would consider it. > > It seems pretty tough to handle large numbers of exceptions when > malloc fails. And since it is such an unusual condition, I don't > think it would be a good idea to address it in a manner that penalizes > performance in ordinary conditions. Thanks Ian for your reply. I agree, it's tough to find a suitable general solution which scales with the number of threads, and I don't have a solution for the general case. In my case, we have a thread budget, it's just that the number of threads we use is much larger than the 64 which GCC can guarantee to safely handle. I'd settle for being able to supply my own pair of allocation/deallocation functions for this specific use case, and having enough documentation to be able to confidently estimate the amount (size and shape, i.e. N blocks of M bytes each) of memory I'd need to reserve for this. >From what I've seen so far, it seems that as long as I know the maximum number of threads in my program, and the maximum size of the exceptions in my program, this should be doable, if I could override the memory allocator just for __cxa_allocate_exception() and friends. I admit to not having a deep understanding of GCC/libstdc++ implementation details or C++ exceptions. If there are other reasons as to why this wouldn't work, I'd appreciate knowing about them. Cheers, John From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 24996 invoked by alias); 4 Mar 2014 10:06:15 -0000 Mailing-List: contact gcc-help-help@gcc.gnu.org; run by ezmlm Precedence: bulk List-Id: List-Archive: List-Post: List-Help: Sender: gcc-help-owner@gcc.gnu.org Received: (qmail 24926 invoked by uid 89); 4 Mar 2014 10:06:15 -0000 Authentication-Results: sourceware.org; auth=none X-Virus-Found: No X-Spam-SWARE-Status: No, score=-1.9 required=5.0 tests=AWL,BAYES_00,RCVD_IN_DNSWL_NONE,RP_MATCHES_RCVD,SPF_HELO_PASS,SPF_PASS autolearn=ham version=3.3.2 X-HELO: plane.gmane.org Received: from plane.gmane.org (HELO plane.gmane.org) (80.91.229.3) by sourceware.org (qpsmtpd/0.93/v0.84-503-g423c35a) with (AES256-SHA encrypted) ESMTPS; Tue, 04 Mar 2014 10:06:13 +0000 Received: from list by plane.gmane.org with local (Exim 4.69) (envelope-from ) id 1WKmEk-0007cf-3h for gcc-help@gcc.gnu.org; Tue, 04 Mar 2014 11:06:10 +0100 Received: from ppp118-210-209-19.lns20.adl6.internode.on.net ([118.210.209.19]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Tue, 04 Mar 2014 11:06:10 +0100 Received: from toojays by ppp118-210-209-19.lns20.adl6.internode.on.net with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Tue, 04 Mar 2014 11:06:10 +0100 To: gcc-help@gcc.gnu.org From: John Steele Scott Subject: Re: C++ exceptions with many threads and no memory. Date: Wed, 05 Mar 2014 08:35:00 -0000 Message-ID: <5315A582.7070600@toojays.net> References: Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Cc: gcc-help@gcc.gnu.org User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.3.0 In-Reply-To: X-IsSubscribed: yes X-SW-Source: 2014-03/txt/msg00014.txt.bz2 Message-ID: <20140305083500.4YAZq2MJvU9KRysfL4lAhR7oP1rZ0r2e5LgSw5X6hNM@z> On 04/03/14 17:05, Ian Lance Taylor wrote: > On Mon, Mar 3, 2014 at 3:53 AM, John Steele Scott wrote: >> >> When comparing this to GCC's exception handling, I find that libstdc++ calls malloc() whenever an exception is thrown, and falls back to statically allocated memory only when malloc fails. Only 64 statically allocated exceptions may be in flight at once, and if this limit is violated, std::terminate() is called. >> >> This seems like quite a brittle behaviour to me. I guess not often seen as a problem in practice because Linux overcommit usually ensures malloc() won't fail (although if it does, then you have to take your chances with the oom-killer anyway). >> >> There doesn't seem to be any way around this, is there? Do GCC developers consider it a bug? > > If you have a good proposal for how to handle this situation, I'm sure > that GCC developers would consider it. > > It seems pretty tough to handle large numbers of exceptions when > malloc fails. And since it is such an unusual condition, I don't > think it would be a good idea to address it in a manner that penalizes > performance in ordinary conditions. Thanks Ian for your reply. I agree, it's tough to find a suitable general solution which scales with the number of threads, and I don't have a solution for the general case. In my case, we have a thread budget, it's just that the number of threads we use is much larger than the 64 which GCC can guarantee to safely handle. I'd settle for being able to supply my own pair of allocation/deallocation functions for this specific use case, and having enough documentation to be able to confidently estimate the amount (size and shape, i.e. N blocks of M bytes each) of memory I'd need to reserve for this. >From what I've seen so far, it seems that as long as I know the maximum number of threads in my program, and the maximum size of the exceptions in my program, this should be doable, if I could override the memory allocator just for __cxa_allocate_exception() and friends. I admit to not having a deep understanding of GCC/libstdc++ implementation details or C++ exceptions. If there are other reasons as to why this wouldn't work, I'd appreciate knowing about them. Cheers, John