From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by sourceware.org (Postfix) with ESMTPS id 5A3D03858C83 for ; Mon, 27 Mar 2023 19:04:23 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 5A3D03858C83 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=oracle.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=oracle.com Received: from pps.filterd (m0333521.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 32RIoKi1028609; Mon, 27 Mar 2023 19:04:21 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : references : in-reply-to : content-type : content-id : content-transfer-encoding : mime-version; s=corp-2022-7-12; bh=u1LePwOY69c4Q5TqzOs0t4BujKpo9L9r87Nnw2w6i4o=; b=XMBTtXujBioccPhW+1jVH9W0NtgaesELoGLWhvDzmKSClcTEcndHKxV74t557nE3Ft+P pLeVIuv2k0Wc+0fo8piG/HzT0rvwW7KqMSSCw8jTVrChAe+1woaMHpR4CFrJh/4n+pyc 7t96Y+tWlmBP44QTYVDTZVFIFAtEzujigetzZ8m2UkxshKh5eT2IHERCAK8EPW8qhZzg n1K5Zev9WOkBj9RhYUYGX8cfHgEOSnEXw2Q7uwPiPLIeg0DkRt7P0VSVurNauCAjmmpE DInoGzwRyyu/AVk7QtnmIIaA03eUO2/OCvCcGySlkZLd/Q+r1O3wXEp+Qat7/Z0CUGj2 TA== Received: from iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta02.appoci.oracle.com [147.154.18.20]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3pkgsa028s-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 27 Mar 2023 19:04:21 +0000 Received: from pps.filterd (iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.5/8.17.1.5) with ESMTP id 32RHrF4p032248; Mon, 27 Mar 2023 19:04:19 GMT Received: from nam12-dm6-obe.outbound.protection.outlook.com (mail-dm6nam12lp2171.outbound.protection.outlook.com [104.47.59.171]) by iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 3phqdd57j4-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 27 Mar 2023 19:04:18 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=mluyMaRS8j9m6IpH29BWu/19h0ZfnZiJAkhvMVM7XeBM7Qn5SvN/PRqSISaQpNoTijMtVgbjsz6dgsAKyzRxxKjrzS9kx6ZJ8fTGPeRGJd6bfH6Ml/iA47tiofutZXYTtHHI/Jo2vZhWgJgTxOLCRI/8ySj99ZyF4mvTOepKlDc6kenr1La+tsPR/xXLJGUkblDi1gjMMsafWozGJAfgGLqZiuP3XbrvkBivGYyZ0J+r7jpOlALh0fK+d60akP09Tve11bXru9GafdvaPl9YCTRXnWQ09eQmnjAQKixymetmWe+J7hVtHjj/g6nlCBAEyvPDE8rpJluTV3dncTF7ew== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=u1LePwOY69c4Q5TqzOs0t4BujKpo9L9r87Nnw2w6i4o=; b=XHBvI4ZT4bMBwcPC9fxnecxnF5VEZ1XMZQnq96NNY0wsEr9Er0MKVAnEXyhl5VAKtK7JlgcvlAj+oZFAZTe7kfxQbMoJvjpbfCVe46Hc2Cpn3U7UTT1edJ5EJB0tXxTOQhNJxjtnm57g1YOFSPgzrcmEmTtgQuIHPBkneAZIvLk4r6IhiVYTnSOCQWfHy6CtjslMe7RK/rhEk9zgGt/K5IOSl+lMoSl8BweNCvL9kCpjw+6qCUXgU/AbjiqbPcLB+MtJlooD1OC40la5kkM8z/60WoK5YyZRJ+iqRmAoWx4mxGlVPkbLTPHQNRxU+exDso/Hs0ZUyOo089v1T/yZ5A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=u1LePwOY69c4Q5TqzOs0t4BujKpo9L9r87Nnw2w6i4o=; b=NsSbsX+DY+oBSpv3nWdZGnLu8o0dutm+tVZgfDSpwIW5BI9/2KYgz9+kVdpL6hs9d4nC4OIjjEbuiq1IK6sw1qhjxGIuFgD1tVxFjDVLBLhy8+XH30RYJS0JuOYNBcpmq5a+2x2UKn1UU8fy5drhMfAvymexcZ7b3n8oIRbo6ws= Received: from IA0PR10MB7327.namprd10.prod.outlook.com (2603:10b6:208:40e::22) by MN2PR10MB4174.namprd10.prod.outlook.com (2603:10b6:208:1dd::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6222.32; Mon, 27 Mar 2023 19:04:17 +0000 Received: from IA0PR10MB7327.namprd10.prod.outlook.com ([fe80::7557:2abe:7fb6:b735]) by IA0PR10MB7327.namprd10.prod.outlook.com ([fe80::7557:2abe:7fb6:b735%4]) with mapi id 15.20.6178.041; Mon, 27 Mar 2023 19:04:17 +0000 From: Ruud van der Pas To: "mikedunlavey44@gmail.com" CC: "binutils@sourceware.org" Subject: Re: feedback on gprofng Thread-Topic: feedback on gprofng Thread-Index: Adlgt6LkBEfJlp8KQ8aaliwyYhtikQAJ0okA Date: Mon, 27 Mar 2023 19:04:16 +0000 Message-ID: <6B579F47-5F70-496A-BB47-7DD8DC693445@oracle.com> References: <001a01d960b9$d56bfa60$8043ef20$@gmail.com> In-Reply-To: <001a01d960b9$d56bfa60$8043ef20$@gmail.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-mailer: Apple Mail (2.3696.120.41.1.2) x-ms-publictraffictype: Email x-ms-traffictypediagnostic: IA0PR10MB7327:EE_|MN2PR10MB4174:EE_ x-ms-office365-filtering-correlation-id: 2933227d-ed12-4c5a-662b-08db2ef6101a x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: cOxuMKfO8WKJjTQ9nNWX1V6F+hVPayvZXNpxOLiuEEGY8MfNQEC7B7RMre3Zd+Omb9grpmCsxvF6tSaB07ludyzLsCKnG5b2nomSANZgz6ZCMlpgu02CLGULUuVlXD8QMUCrNfgLYSuI2Dj2DlFcoeerQifiRwElBi+E7zI4AsLcV4bULPeVbiCY7c2Y4LoptGwIQ9eXqj2QGREcGULyMjhEQtmdgdLgaAP+wbUaLojVy2BoUs2pefiBcxYp6dEqKUy2UDrIb5Z66lg7WkhLYRxRYS0tRi17zE8+HOzR0THVQiMjZnElzEzlf8D7odi2J30jBoV2UpBqJU08ScKndkxXx1Dg9ggvQVEoJ15t5vQWgIySBsv1ZoLd8P3Kz9Cr0vlMythIF3naA6JXWTgWfOfwTelwZY/0fm5WL0n9li7+bEu9stp9uY4BaviuxpdeKCL22YO4e79d4IVgoC65NA9wwJzEmpwY9fdzKdLq+eAkSzj/WmEwnu3uNo6AoDDVW7GNGCWFI8ZqOHPMuuFG1j6NHn2dbKL0RVJS3nBjvFifcOisLVmXHbOZLZlO7f5DWX0MHKZYQPeki+BNIiNCn5WTRklDNT7/HO7zBPj6139OYGTEEl6l8XQ8z/Gd1q2ZQnQQcEsoTEtiRnh7X8P4eQ== x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:IA0PR10MB7327.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230028)(366004)(376002)(39860400002)(346002)(396003)(136003)(451199021)(3480700007)(66556008)(66476007)(64756008)(66446008)(76116006)(6916009)(4326008)(66946007)(8676002)(91956017)(122000001)(316002)(8936002)(7116003)(41300700001)(5660300002)(6506007)(186003)(83380400001)(71200400001)(2616005)(6486002)(478600001)(6512007)(38100700002)(86362001)(2906002)(33656002)(36756003)(38070700005)(45980500001);DIR:OUT;SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?NHTBnYkYYyWvwQfaSFmsmQO+HerRgrRcwDkNhpSO/gr50sIkMUMBTePKs1T0?= =?us-ascii?Q?Mbbm+nD12a0w/mks3EAJS+9grB8VNsMnFv4p0dvrexMAXtxgwLkwV/VjBSmK?= =?us-ascii?Q?afWFNZYiUG0snOAJV5VqcjzbByEB/Y7DrtKxGA6RcSNNdGpU/WGRw+gQ/mj3?= =?us-ascii?Q?rHe6FpCIBkzjNS+UHeVjXFXaFfByyyE5evdjvxl+2HsEnLQimJs86LdSMD9f?= =?us-ascii?Q?CQXzONmnpdEbPLURCK7WPPHoDUzjvKYz6iBaXqj16ZpqwgFUNPkwWueYQZYj?= =?us-ascii?Q?i7zCzl3030hu4Ufrh+yypAZYojkNPv/agc+/jv1+faFCBsbC81Kjy/7/c8iV?= =?us-ascii?Q?mtmW+TG9FwyoUtoA+cHClUNYkecg5rhC9NyJzN7rwbIztTc9VbPEEnKNVv7h?= =?us-ascii?Q?WyEb2RyFmfg3Qj4S5t5DE+5SxoX3/kuoFQtOOWy0+T74bF/VcfeScE40SLr1?= =?us-ascii?Q?46Q1azsPPJbWxPMhvBKFSjaj1Qf9O652tDmYQLTxa3WT7LbdwPPXOQ1BsWRa?= =?us-ascii?Q?ZkKE1rrB6BwG5bX5Lb+dqwd9AUIEsimVCwSxLnmZIbjs/rsDcNy8SYLvkIH0?= =?us-ascii?Q?gpf0aluuT+M75qVOij929MvFBiT8FdSLNW0Jo79Zwi1Jva+yXt0tCUfoptxW?= =?us-ascii?Q?jA1+Nt70CzSRl3UVcx9rAGyJvi/SHQHxKl2jGlBiES17bnuQHsm6unjWmWYq?= =?us-ascii?Q?O/6OcvvemqCaTJ16GI6vLM/PAoVe4nUwgBw0dLNep0p1LjJXhWwBlo5OH7WY?= =?us-ascii?Q?pNejRkcFBwhcP7XBYWAiiC6M57CFHKHN8vG0ZvVhwSgcugbzn8NgHhD0DJ3p?= =?us-ascii?Q?5NsVb4Vk3YrErzALDBCSRu+uHKMGFhTVXhNuSGU+XDIEE0rvn4/Zi7W6mXhs?= =?us-ascii?Q?cO9o8aZv/P9Hi0LZ1ptOwsFZgGrwNpt5nyGKrzfByzMnh6cBMcYJIi8QyW+N?= =?us-ascii?Q?MtekePqYBRRCL2bETKeC9lA8C9qsa/I4N5epzE0q2V35orS/hVATYzeMpGcT?= =?us-ascii?Q?+SIa8D7hgPqJ9Bkvv0iRS7h/Dvmftxm5NUERL/pNKaErbc2Roq4vIfcwQd4S?= =?us-ascii?Q?O235iFaSbuOdtwcIWQNkUqYPsrykqV4/gTYMSXPXjMeksP72QELoBPaOPoGU?= =?us-ascii?Q?m0LjX+a5lX4zEuH6uRlPWV3sdIpRJSP1PgBoHIc7PQTAdazcNV5qEHFjYh+X?= =?us-ascii?Q?t7qKge9zrMQ8t6gNZIPs6OyuttAAVyR4ud2tFBHfXKx7j5loDDHguf8GRTCl?= =?us-ascii?Q?efPMAiUV4InvXJ5D4hrEeewMELLcKqb8Cs0na6mIlFK+Aqix5sZFJTe5Oh4c?= =?us-ascii?Q?WUE7mAF2FJWb8Hqrwu5g9Z0jGDQ3nars9Niu7uUQrVSnvgdhn9bbHFTajbyn?= =?us-ascii?Q?XPawspDG7RYjtDqQOwEAOPhEFuclmnh1b8M3xdi9SyzxumXsmCjr5jl86hMG?= =?us-ascii?Q?noxJPutPqQ/IJ3NzrSAMerKnb5U+ConqgZOYyEN9eGQZ8jcaLa9YNTrtJJmj?= =?us-ascii?Q?Pb0Z7Ci2HL5fcgK0z7jgqc/JDd1l1d5VN9RHCvCg2fNT5uvgovAzep1UmGDd?= =?us-ascii?Q?n3yBfy90R/9EH84hePy1IhtDOrKb2nnkLKuo15OFRLLDhMvRl/4LDoy4m4QI?= =?us-ascii?Q?ARmfR1p9K7BVTBwbDnOWTx96V0s6UTKYWPTbT1y8NShgXV4vgjZuaDaCA7Bp?= =?us-ascii?Q?bsGIpg=3D=3D?= Content-Type: text/plain; charset="us-ascii" Content-ID: <3BCEB79875B3554B92980078B4B803BF@namprd10.prod.outlook.com> Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: /MvtLatNM3PbTVrC74crWJxelJ0mLThsq8tNpQEilP46W9VSQxrEib8F6hmetwFPesga00p0OpLGqc2NUPyxAfIh6+ARNvPuk+VVpoVyIl/5zLAvfkLQxINSUvvwqsg0BJudQ/prDZHon+/BaZnpJOeRTxpNTZzCRKRQT6U1PmwaKUi0papZjsH/fmlcSkD503wjTzKXThfZnxjCeqd+2V6k7jH33CTkc3RAO5yILVp6U8KHzJEUOGdShzX047sK2ZsUY3WXhcs7L6Cfv1um7X4NsC1SWhv4ztLuxUZPUbbAdli3vg7Fvrl9j5u5fM1X2e1lzvK//vgRmHFUrLac2CDz/noar2pCs+VwJTbweVDn0fIM6xktmz12dSBWriMdTiHaCcevcU/kpnsXQn+15wrz6QHAYMZDnuClWmqwO+4Bo2iS6gIrf7eV9y+LfuLxHK4FBKNkqtZOEKutIdVA+TZhptxi+Noc6vDbixWMNSy2kA7l8MXMMILehAfW8356P4kqonZfAdWImdnGwV1qmw4ZDPErQKcHYJyOXR95Y1untpuUMXubMHw1dzvPYQBuY/p0NAh/s1by82/q/cfw+FpNwCUVPULFYC3KfNCfH128A457WV018ew4taGJ3HC2xYx/V9HawOar3Izal6wqo3La2+1zIk9s0Il8hz09+VI1GN+WIZL+hRs3MbirSnIRY/g2nymsDoNJPiF7ad8ugZJ4ne9wDP3oFjhrKuc5Hx7Duep8hAqDmIGznQazeTgZDl6y48n0Xr66RAU/rjj4mX1BjBvk/prM7rtXbqOVUxc= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: IA0PR10MB7327.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 2933227d-ed12-4c5a-662b-08db2ef6101a X-MS-Exchange-CrossTenant-originalarrivaltime: 27 Mar 2023 19:04:16.9430 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: Irn+FQIIbMTLpE10+LkVYtjWPeDlSJ115m1Qxxv8NJEViFkc2HmAQCL/rF+opvEIa0+uwXXXqsN7YBfKL/V7rrKUrQNYEfhunsLU9x4AkJ8= X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR10MB4174 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.942,Hydra:6.0.573,FMLib:17.11.170.22 definitions=2023-03-24_11,2023-03-27_02,2023-02-09_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 bulkscore=0 spamscore=0 phishscore=0 mlxscore=0 mlxlogscore=687 suspectscore=0 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2303200000 definitions=main-2303270156 X-Proofpoint-GUID: LAooz0I5PsY2Oazpn2Z6aqJYtN7sndbZ X-Proofpoint-ORIG-GUID: LAooz0I5PsY2Oazpn2Z6aqJYtN7sndbZ X-Spam-Status: No, score=-3.8 required=5.0 tests=BAYES_00,DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_LOW,RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE,TXREP autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org List-Id: Hi Mike, Thanks for your email and suggestion. > I suggest you have another output from gprofng: We actually use sampling. It is the cornerstone of our approach. > (I assume the sampling is on wall-clock time, so it has visibility into > I/O.) Yes, we do and indeed have visibility into I/O. We actually just heard of a use case where gprofng shows that the (formatte= d) I/O is the sequential bottleneck in a multithreaded application. We record call stacks, also for individual threads, using sampling. The user can control the sampling granularity, either symbolically (e.g. "high", or "low"), or through a sampling rate. There are also filters to select call stacks, a window in time, threads, etc. We also just released a GUI that, among other things, has a timeline where we show color coded call stacks. Each function we see gets assigned a color and we show these call stacks as a function of time. This is=20 also done for the threads in a multithreaded application. In this timeline, it is really easy to identify gaps in the execution. You literally see them and by inspecting the call stacks, you can find out where the execution was when it happened. Kind regards, Ruud >=20 >=20 >=20 > Let the user choose a small number N, like 10 or 20, and then select N > stacks at random (with source code line info) and display them, in a tree= or > in raw form. The point is - any performance problem consists of activity > that isn't necessary, and if it accounts for fraction F of time, then it > will show up on NF samples. High precision of measurement is not necessar= y, > but precision of insight is.=20 >=20 >=20 >=20 > If there are multiple threads, let each sample be from all running thread= s > at the same time, so the user can see which threads are waiting for which > other threads at the point in time. >=20 >=20 >=20 > Let me know if this makes sense, or maybe you've already done it. >=20 >=20 >=20 > Thanks, >=20 > Mike Dunlavey >=20 >=20 >=20 > P.S. I've been advocating this for years on StackOverflow. People who've > tried it agree that it works. I've also got a YouTube video about it. >=20 >=20 >=20