From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 13606 invoked by alias); 7 Jan 2015 22:33:24 -0000 Mailing-List: contact gdb-patches-help@sourceware.org; run by ezmlm Precedence: bulk List-Id: List-Subscribe: List-Archive: List-Post: List-Help: , Sender: gdb-patches-owner@sourceware.org Received: (qmail 13592 invoked by uid 89); 7 Jan 2015 22:33:22 -0000 Authentication-Results: sourceware.org; auth=none X-Virus-Found: No X-Spam-SWARE-Status: No, score=-2.7 required=5.0 tests=AWL,BAYES_00,RCVD_IN_DNSWL_LOW,SPF_PASS,T_RP_MATCHES_RCVD autolearn=ham version=3.3.2 X-HELO: mail-ob0-f169.google.com Received: from mail-ob0-f169.google.com (HELO mail-ob0-f169.google.com) (209.85.214.169) by sourceware.org (qpsmtpd/0.93/v0.84-503-g423c35a) with (AES128-SHA encrypted) ESMTPS; Wed, 07 Jan 2015 22:33:20 +0000 Received: by mail-ob0-f169.google.com with SMTP id vb8so5521117obc.0 for ; Wed, 07 Jan 2015 14:33:18 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=YaSQ/d6sRgpAeEmSQM/vVk/SgXwUrDqd86V8yNG//bs=; b=FkhiqaCruVrfzcZKBLnXlxszaaogVhX5Z03XvIiZ+Bci/xCIUWCiQhO52u3oX7X3b2 AefLXjO24XXZ2X7dD1aDdtUK4ceSgTNz5FSZpYglUYndcf79ne6FD7lJqQMCKgAZva40 xFQ/QvuUT3LRvve3TZWDPNn5aoI2HikuhdISehtpuGeKKcYzRqpJIC2PwVMarNKaBeeh lSPU4CPaEjgejgnvGSN1smtX4YTj6ulEH9nHRdUh/76QPpW1YkBPk3R/79YJ9K03YUvS 4cF8u5HUZ9GUSU4fZirV8q/RscbLSICfKL0jcRGbQ4J0bveerngh0dLgWuR0kXITRtw0 XRNg== X-Gm-Message-State: ALoCoQkhQ//UGqV5/M4XYMfqffd8JJf66BP4r8iZDgnBi9zrawvfUD+llIqvoSwdIFQbMP1IdJKz MIME-Version: 1.0 X-Received: by 10.182.227.161 with SMTP id sb1mr3664620obc.80.1420669998711; Wed, 07 Jan 2015 14:33:18 -0800 (PST) Received: by 10.182.222.98 with HTTP; Wed, 7 Jan 2015 14:33:18 -0800 (PST) In-Reply-To: <871tn7udyt.fsf@codesourcery.com> References: <87mw5xuzdc.fsf@codesourcery.com> <871tn7udyt.fsf@codesourcery.com> Date: Wed, 07 Jan 2015 22:33:00 -0000 Message-ID: Subject: Re: [RFC] Monster testcase generator for performance testsuite From: Doug Evans To: Yao Qi Cc: gdb-patches Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-IsSubscribed: yes X-SW-Source: 2015-01/txt/msg00147.txt.bz2 On Wed, Jan 7, 2015 at 1:39 AM, Yao Qi wrote: > Doug Evans writes: > >> If a change to gdb increases the time it takes to run a particular comma= nd >> by one second is that ok? Maybe. And if my users see the increase >> become ten seconds is that still ok? Also maybe, but I'd like to make the >> case that it'd be preferable to have mechanisms in place to find out soo= ner >> than later. >> > > Yeah, I agree that it is better to find out problems sooner than later. > That is why we create perf test cases. If one second time increase is > sufficient to find the performance problem, isn't it good? Why do we > still need to run a bigger version which demonstrated ten seconds increas= e? Some performance problems only present themselves at scale. We need a perf test framework that lets us explore such things. The point of the 1 second vs 10 second scenario is that the community may find that 1 second is acceptable (IOW *not* a performance problem significant enough to address). It'll depend on the situation. But at scale the performance may be untenable, causing one to want to rethink one's algorithm or data structure or whatever. Similar issues arise elsewhere btw. E.g., gdb may handle 10 or 100 threads ok, but how about 1000 threads? >> Similarly, if a change to gdb increases memory usage by 40MB is that ok? >> Maybe. And if my users see that increase become 400MB is that still ok? >> Possibly (depending on the nature of the change). But, again, one of my >> goals here is to have in place mechanisms to find out sooner than later. >> > > Similarly, if 40MB memory usage increase is sufficient to show the > performance problem, why do we still have to use a bigger one? > > Perf test case is used to demonstrate the real performance problems in > some super large programs, but it doesn't mean the perf test case should > be as big as these super large programs. One may think 40MB is a reasonable price to pay for some change or some new feature. But at scale that price may become unbearable. So, yes, we do need perf testcases that let one exercise gdb at scale. >>>> These tests currently require separate build-perf and check-perf steps, >>>> which is different from normal perf tests. However, due to the time >>>> it takes to build the program I've added support for building the piec= es >>>> of the test in parallel, and hooking this parallel build support into >>>> the existing framework required some pragmatic compromise. >>> >>> ... so the parallel build part may not be needed. >> >> I'm not sure what the hangup is on supporting parallel builds here. >> Can you elaborate? It's really not that much code, and while I could > > I'd like keep gdb perf test simple. How simple? What about parallel builds adds too much complexity? make check-parallel adds complexity, but I'm guessing no one is advocating removing it, or was advocating against checking it in. >>> It looks like a monster rather than a perf test case :) >> >> Depends. How long do your users still wait for gdb to do something? >> My users are still waiting too long for several things (e.g., startup ti= me). >> And I want to be able to measure what my users see. >> And I want to be able to provide upstream with demonstrations of that. >> > > IMO, your expectation is beyond the scope or the purpose perf test > case. The purpose of each perf test case is to make sure there is no > performance regression and to expose performance problems as code > evolves. It's precisely within the scope and purpose of the perf testsuite! We need to measure how well gdb will work on real programs, and make sure changes introduced don't adversely affect such programs. How do you know a feature/change/improvement will work at scale unless you test it at scale? > It is not reasonable to me that we measure what users see by > running our perf test cases. Perf test cases aren't an end unto themselves. They exist to help serve our users. If we're not able to measure what our users see, how do we know what their gdb experience is? > Each perf test case is to measure the > performance on gdb on a certain path, so it doesn't have to behave > exactly the same as the application users are debugging. > >>> It is good to >>> have a small version enabled by default, which requires less than 1 G, >>> for example, to run it under GDB. How much time it takes to compile >>> (sequential build) and run the small version? >> >> There are mechanisms in place to control the amount of parallelism. >> One could make it part of the test spec, but I'm not sure it'd be useful >> enough. Thus I think there's no need to compile small testcases >> serially. >> > > Is it possible (or necessary) that we divide it to two parts, 1) perf > test case generator and 2) parallel build? As we increase the size > generated perf test cases, the long compilation time can justify having > parallel build. I'm not sure what you're advocating for here. Can you rephrase/elaborate? >> As for what upstream wants the "default" to be, I don't have >> a strong opinion, beyond it being minimally useful. If the default isn't >> useful to me, it's easy enough to tweak the test with a local change >> to cover what I need. >> >> Note that I'm not expecting the default to be these >> super long times, which I noted in my original email. OTOH, I do want >> the harness to be able to usefully handle (as in not wait an hour for the >> testcase to be built) the kind of large programs that I need to run the >> tests on. Thus my plan is to have a harness that can handle what >> I need, but have defaults that don't impose that on everyone. >> Given appropriate knobs it will be easy enough to have useful >> defaults and still be able to run the tests with larger programs. >> And then if my runs find a problem, it will be straightforward for >> me to provide a demonstration of what I'm seeing (which is part >> of what I want to accomplish here). > > Yeah, I agree. > > -- > Yao (=E9=BD=90=E5=B0=A7)