From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by sourceware.org (Postfix) with ESMTPS id 1EFE0385AC25 for ; Wed, 1 Sep 2021 18:45:11 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org 1EFE0385AC25 Received: from pps.filterd (m0246627.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.1.2/8.16.1.2) with SMTP id 181I4674008965 for ; Wed, 1 Sep 2021 18:45:09 GMT Received: from aserp3020.oracle.com (aserp3020.oracle.com [141.146.126.70]) by mx0b-00069f02.pphosted.com with ESMTP id 3atdw0g7j8-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Wed, 01 Sep 2021 18:45:09 +0000 Received: from pps.filterd (aserp3020.oracle.com [127.0.0.1]) by aserp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 181IffT6177204 for ; Wed, 1 Sep 2021 18:45:08 GMT Received: from nam11-bn8-obe.outbound.protection.outlook.com (mail-bn8nam11lp2174.outbound.protection.outlook.com [104.47.58.174]) by aserp3020.oracle.com with ESMTP id 3atdyub1nd-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Wed, 01 Sep 2021 18:45:08 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=IuLIFq+gYxLyNaQsIwLuwcB6lehXauS0XtME0O+Pbhadvr2Q+PhQLw3pbj4KpPd+rCg4G8xLCVoQ+2WV8JTAXYY92lNTCMzV93G36D0nmDuof+7ISAM6pwlC4RAsVHomkWAuIf2fn3i0+uP8RorAT49qeCJrJ0ToRTfenxQ+MjsU8i+d4y2rFZ0kz1k5jmqhbw6JUykFM3frWeO319NlDBrFg5fewErqVFP0qGPwaUhJ1ZSOhFFsnnOySkYMPbL+NhKVJ+n3tqRKZD+hwfLbYgZT0XXtfcJWYADZ3nKJpiO3R2cSXcPDKQjeNVVJXh5L5ExU6k2SB2xhwwlgeJMfvA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=7MOAUW0fAENFO1fnLJ5gGMmVKmteGUVszNXdqDEhBIM=; b=Aa26iQq5TYPCQcwitGDrfDw9FEG7lsLMjc8SUzxXeczp9y4O7Y7/fumvPN4b2HNHv17pqsUqXfqjYR22jLEC8AqqEelGe+Thq0kgaUfXBLsI4Qc7qooPntsQ0KxTqUXZHuMQnRMMlBdaMjvk9t7bDOK2c9piB7vDWa+iPpXvRI2QA1R7aB8HAyXu1U9oLKOyyqOnq7aGsbxKnt2URb356oIuGUj2rIXw3xkdztpnqSyMvFljfV9C2gNuFPHZAqL6y7sCe+9WUm35/8GmAu5BZpqCLRqBhaTHBMOZtObM3S38UNTiw2XF5c33QqzHclBiQc/ZuYZwp/VPJ4h9N879iA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none Received: from SA2PR10MB4636.namprd10.prod.outlook.com (2603:10b6:806:11e::10) by SN6PR10MB2671.namprd10.prod.outlook.com (2603:10b6:805:46::29) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4478.19; Wed, 1 Sep 2021 18:45:06 +0000 Received: from SA2PR10MB4636.namprd10.prod.outlook.com ([fe80::4d60:5d4c:a74c:c9d9]) by SA2PR10MB4636.namprd10.prod.outlook.com ([fe80::4d60:5d4c:a74c:c9d9%4]) with mapi id 15.20.4457.024; Wed, 1 Sep 2021 18:45:06 +0000 Subject: [PATCH V3] gprofng: a new GNU profiler References: <047143ae-be5e-9eec-237f-ad9ffe3795c0@oracle.com> To: binutils@sourceware.org From: Vladimir Mezentsev X-Forwarded-Message-Id: <047143ae-be5e-9eec-237f-ad9ffe3795c0@oracle.com> Message-ID: <52ead579-44ab-4d05-67d6-4a09f7a1693d@oracle.com> Date: Wed, 1 Sep 2021 11:45:01 -0700 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.12.0 In-Reply-To: <047143ae-be5e-9eec-237f-ad9ffe3795c0@oracle.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US X-ClientProxiedBy: BYAPR05CA0062.namprd05.prod.outlook.com (2603:10b6:a03:74::39) To SA2PR10MB4636.namprd10.prod.outlook.com (2603:10b6:806:11e::10) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from [IPv6:2606:b400:400:7446:8000::519] (2606:b400:8301:1010::16aa) by BYAPR05CA0062.namprd05.prod.outlook.com (2603:10b6:a03:74::39) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4500.4 via Frontend Transport; Wed, 1 Sep 2021 18:45:05 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 1d8f5176-0fc8-4ef0-2910-08d96d789dd0 X-MS-TrafficTypeDiagnostic: SN6PR10MB2671: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:10000; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: rJW97OHsx4IYSCJSiGaEuJ1AluV+bfroSJWcigxjD0q9T3z8S5ZrfhL7FFsSSK8TW9p8vISbz3sDKqmVthA6tEFvDHbGTkmEgrsJBkLhSh9/6NBlrxJk1hZ0UaQANGh39jnt7j9RPAIZZuyYdNfmzfFuq/NwEqFS+Uvu0v5uOPNCGE2ATgt6/NR2T8TVco8rsAx7GCjq1lB3gOPkUXXJhPbcl+Xxd0oa+rNPIFhjV8HzPjsSx8tooeSfTYtqI22EIe/TexBaJhC+cGp8LooC2GIq75NbqO/Qn3gKSP3vbZmPCYVqg305/tFntFdrGuWXFMnMf6cHRuw/Rc/b99NkMETy2EJil7lzd4ZxniUxzH3zw81dGK2CyQ7N/bN56QMPQHXHAt8i4POU9egn8pVcMj6ZwVzpRqmL5qok5yXBAIDUp/vSQmBBel5L406476Wnr2gTznP1rd58H1KV8efhf6CywOOK8PUPHv8BZHMKlFeyWey7X0+GaZAQ78kFxrcD4BU9FFAWjNZ45jpBdEDXcGAAGPCWy1lcGDAvzc+oPr5rdbCWAYfUUSorj4t8MIMGx7MMIcHAgfghywwvtqoh15CyYSMKrWErB8VHxOuKKVbYrPS8fggqBNYurxBa4fFXaui+VTkPg9SaPrcyihQs7H62xOtBnyQ8TkpGtD1mJRRm/G8lzZdRY+mzPcD/5snX8NiFe89WBQW2e1aAUUjtrrvPe1B4NocXrUtVgKPl7Vj/HDshIXmWA+y6wheV6zdQ3+ObBU5q8aAACzek0en4ln0XHaHwxXnGpzxSGNdkLwwjKV7g2mgZ4BqlhbycVjQC X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:SA2PR10MB4636.namprd10.prod.outlook.com; PTR:; CAT:NONE; SFS:(366004)(346002)(39860400002)(136003)(396003)(376002)(5660300002)(478600001)(316002)(66476007)(66556008)(6666004)(38100700002)(83380400001)(66946007)(31686004)(31696002)(2906002)(86362001)(8936002)(44832011)(2616005)(6486002)(186003)(8676002)(36756003)(6916009)(966005)(43740500002)(45980500001); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?dkdmd2Irei9zaTF0YnpwUnpCQ1JNMGxWY3JTZ202bThoR05xM1cwOFpPR3ZY?= =?utf-8?B?NkM5WDBmSHJXbFJpMlJMeGZ2aThBSGttQnZGejRzSTF0NEJWQmxFTU9YVGtr?= =?utf-8?B?WVRWb0dLWE5UUHdNUm1CdFJHVE1tQjd3UHB6ME80dzhvUW1uR0FlNy9zRVNV?= =?utf-8?B?OVJEQWMrN3gyV1FpSVQ2TnI2QnQ2ME5kZjBiZGlPVWxGanQza0hwYjRKRE9W?= =?utf-8?B?bE5KTDZDOTg1eDF3RUNqaDZuVTJLQVJOTEhiT0VseCtGb1NFd2ZydUxnTTdR?= =?utf-8?B?ZXFrWUFRdXJQcGdka3BjNktEOFFYN2xTSk5RakZGTWlNcVdsZjVLV3VvY3Iv?= =?utf-8?B?TExiTDJnaDl4SjcxVWRJYVM1YVdYaCtWTk9lc2VJL095bkMwMGV5Smt1Ujla?= =?utf-8?B?M2RYYm10T3FKcDF4R3ROUCtwUzRBZy9DS09yTjhaRVdJZC9UbndnZ2szajlP?= =?utf-8?B?US96NCtGTmdNN1dTUmxGZ1loeVNMMWNjZzdYQjg2eDBQRE53QUpvc1VyREJW?= =?utf-8?B?cW5ockZvUGxCRWp2UXFIM1MrTjlMQnplb3d1Sk4rT1R2QTJJdFdGNlZhNWhQ?= =?utf-8?B?VlQ1d1E1ZGhQcXZOSlVzVGJjZm42QUZCWHQyNWR3b1VSVWhhSEkzdjBmbkRQ?= =?utf-8?B?Rm5GZEkxMGlRUjZENGpHbHdPSXFLZDdVMG1KK1hPYUZYWUlPOFgxMUZhQW1B?= =?utf-8?B?ZVVWK1hTWnJCNWRtdWltTldZaUlrYTcwZ1pxQ1VSeSsvTUhVQ2ZycFc0YVBY?= =?utf-8?B?Vk4yUG02NWtMd0VlQmNKRGJrZ1NIcFhVdnYyK3JxSllZOWFTK0luZHdCNExw?= =?utf-8?B?VUtsbnh2NVpBQ3JMUnNkMGJjWU5hWld5MC9kVTVUWkhLTjhPYXMvakRudnRR?= =?utf-8?B?MXMxNnUyakgrMUZVWS9EOHdBOG1zNENDZUpPK3Aza2ZLUDlERVBjaXo1K25J?= =?utf-8?B?bEFlcWpLOWhvYkVMRm41NSs5SUdjUUh4L05UNmdORGlTRk0xaS9JN1laWDlW?= =?utf-8?B?OWpCNGpQcjRmcUxNaVU0M01xbU5MUmYxL1VoVW8xZHI2bDMzYXFFSGxLcENq?= =?utf-8?B?S09zWEtDeWdUNkFIWm9yUzBKNVI1Mks4MjNldUM1M1ZMV0syZ3g2VSs4S1lw?= =?utf-8?B?MHQ2RDY4SDNqdHFYMVhJYkxsK0xobEExaTkyU2psdWFNMUtJQVorakVIWmZi?= =?utf-8?B?Z21ZNkpGQ0JQUTJkRmxOTnZXUU9HaEh6eGJyc0VJazRhaW03ZE5QQ0ZLQXhq?= =?utf-8?B?WDFJeWJONlFSZzRQRjZOU2dIUTM0dDlXMlowaEV5WWlHdHFKbnFEVUNXdWZO?= =?utf-8?B?TkpSRWZmUjRhaVJaOFp6SUhhYVh2YjY4SlVZcjZ0UXFJOTk5aVVXZkgvR01K?= =?utf-8?B?cVcrOG9YTW1wWFpJTFpTcGswWHpFdjJ0SnRLZzlnVE9NYkN5QnIrVStaTXhU?= =?utf-8?B?bW92WmJnYlE2U2JIYVk3T2VEd1RuQXRrZmFTd2tiZDZMWEVkM2NOamZEWmRB?= =?utf-8?B?VlhMQS8xNE5qSUtSME1JdHJWdjFrUlFOMjZjR0taUTE0aGZ6VWRwWGFIZW5P?= =?utf-8?B?RGhlWkZ4SEJmSVdoUWlzUzdQTWtjcU5zcm9RbTlHK0thZGdWeDlTR2UrNnQ3?= =?utf-8?B?ZEZSeHhUUkw2eTBFd3VBZzJEcVVUQWdwbjM3a0RndFl1QW1MUnRVa3E4Rm0r?= =?utf-8?B?bmpJR2dsZEl5d2RCSi9QM3FyeStsajRZbyt5ejluYUVkODR3NW5sWVkwTGlD?= =?utf-8?B?YklhVVJvc210cFNyV002VEt1bWRCMi9iOHd1czlxdm9sbWZUV0JOcFVUL09L?= =?utf-8?B?b0NPSW5yK1Qwb1l0VkRuQT09?= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 1d8f5176-0fc8-4ef0-2910-08d96d789dd0 X-MS-Exchange-CrossTenant-AuthSource: SA2PR10MB4636.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 Sep 2021 18:45:06.2546 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: rezE/+Ik+D7LkMymxgCuZ+bmBpO9WfoZsGkyj+9eC6DvWpKJF5C3i4pDedWxdOKUZYDavzV/HID6mQ843jFeOsBirj1vZJLohASOz455C7Q= X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN6PR10MB2671 X-Proofpoint-Virus-Version: vendor=nai engine=6300 definitions=10094 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 malwarescore=0 bulkscore=0 suspectscore=0 phishscore=0 adultscore=0 mlxscore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2108310000 definitions=main-2109010107 X-Proofpoint-GUID: mr8cqt9D3qCpYeA64iXNQNQiPYgvF_H_ X-Proofpoint-ORIG-GUID: mr8cqt9D3qCpYeA64iXNQNQiPYgvF_H_ X-Spam-Status: No, score=-4.6 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, KAM_ASCII_DIVIDERS, MSGID_FROM_MTA_HEADER, RCVD_IN_DNSWL_LOW, SPF_HELO_NONE, SPF_NONE, TXREP autolearn=ham autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on server2.sourceware.org X-BeenThere: binutils@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Binutils mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 01 Sep 2021 18:45:21 -0000 Hi people! 09/01/2021: Changes from V2 ===========================   - Renamed experiment.h to gp-experiment.h (Hannes Domani reported problem on Windows).   - Fixed regression in build on ARM.   - Improved gprofng/testsuite. Where to get gprofng ==================== Due to the size of the contribution, we thought it would be better to submit it in the form of a git branch, instead of a regular emailed patch series. Repository: https://www.github.com/oracle/binutils-gdb Branch:     oracle/gprofng-v3 Build gprofng ============= For x86_64/i686, gprofng should be built twice to support profiling for both 32-bit and 64-bit applications:    configure --enable-shared --libdir=PREFIX/lib64 && make install    configure --enable-shared CC='gcc -m32' CXX='g++ -m32' --target=i686-pc-linux-gnu --disable-gprofng-tools && make install For ARM:    configure --enable-shared && make install To support java profiling, configure build with -with-jdk=PATH_TO_JDK if javac is not in your PATH. o8/25/2021: Changes from V1 =========================== We addressed all of Joseph Myers's remarks about gprofng Makefiles and configure scripts:   - Report an error when bison is absent, or a version of bison is too old.   - Added preload_libdirs in ${prefix}/etc/gprofng.rc to configure the location of the     gprofng libraries (this was hard-coded previously). We also took note of Jim Wilson’s comment about the naming of files in src/machinemodels. In particular, file m5.ermm and his suggestion to make it clear this is for a SPARC processor. For example sparc-m5.ermm. This is going to be part of a bigger overhaul of the naming convention for these files. The most obvious choice is to include the product name, but we definitely appreciate any suggestions this group may have. 08/11/2021: a new GNU profiler ============================== In this submission we are contributing a new profiler to the GNU binary utilities, called gprofng (for GNU profiler, next generation). Why a new profiler? =================== The GNU profiler, gprof, works well enough in many cases. However, it hasn't aged well and it is not that very well suited for profiling modern-world applications. Examples of its limitations are lack of support for profiling multithreaded programs, and shared objects. Both are ubiquitous nowadays. Main characteristics of gprofng =============================== gprofng supports profiling C, C++ and Java programs. Unlike the old gprof, it doesn't require to build annotated versions of the programs. Profiling "production" binaries should work just fine. Another distinguishing feature of gprofng is the support for various filters that allow the user to easily drill deeper into an area of interest. The profiler is commanded through a driver program called `gprofng'. This driver supports the following sub-commands: gpronfg collect app EXECUTABLE This runs EXECUTABLE and collects application performance data. gprofng display text EXPERIMENT This runs a client command-line interface that provides access to the collected performance data stored in the experiment directory. gprofng display html EXPERIMENT This generates an HTML report from the collected performance data. stored in the experiment directory. gprofng display src OBJECT-FILE This displays source (if available) or disassembly interleaved with the source code. gprofng archive EXPERIMENT Archive the associated application binaries, load objects and source files in an existing experiment directory to make it self contained. There is also an extensive graphical user interface (written in Java) that displays and analyzes gprofng collected data in a very sophisticated way. We plan to release this GUI as a separate project. While WIP, we would like to share some screenshots of the current development version. These show the following: pic1.png - a flame graph: https://jemarch.net/gprofng-pics/pic1.png pic2.png - color coded call stacks as a function of time ("the timeline"): https://jemarch.net/gprofng-pics/pic2.png pic3.png - zoom in on the timeline and adapt colors to identify details: https://jemarch.net/gprofng-pics/pic3.png pic4.png - compare two mulithreaded profiles: https://jemarch.net/gprofng-pics/pic4.png Some notes on the implementation ================================ - The gp-display-html tool is written in Perl. All other components are written in C/C++. - gprofng sources are mostly contained in a new top-level directory gprofng/ that in turn contains: + src/ contains the source code of the gp-* programs and libgprofng. + libcollector/ contains the sources of libcollector. + common/ contains a few source files that are used by both the gp-* utilities and libcollector. + doc/ contains the Texinfo sources for the gprofng manual. + testsuite/ contains the gprofng testsuite. Three installed header files are distributed in the top-level include/ directory. These are libcollector.h, libfcollector.h, and collectorAPI.h. - Currently gprofng supports profiling programs in GNU/Linux systems running on x86_64 and aarch64 hardware. It is possible to add support for additional architectures. - The tools come with a set of man pages. They are generated upon installation and can be found in the installation directory under share/man/man1. Platform support ================ The basic profiling features are supported on most processors from Intel. Regarding AMD we did not yet test on their recent EPYC processors, but do not expect serious issues. We also support the Arm processors as used in systems from Ampere. Hardware event counters, which are optional and used by gprofng in advanced profiling, are supported for many modern Intel and AMD processors. If a particular processor is not supported, a warning message will be issued when trying to run an event counter experiment. This code has been developed and tested on Oracle Linux 8 with the latest GNU toolchain from the sourceware git repos. Structure of the patch series ============================= The first patch is preparatory and makes the x86 disassembler in opcodes to be thread-safe. This is so it can be used by gprofng. The second patch is the implementation of gprofng proper. This includes source code for the libraries (libcollector, libgprofng) and the utilities (gp-collect, etc). In this patch there are also updates to the corresponding build machinery (e.g. configure.ac, Makefile.def, plus binutils/MAINTAINERS to cover gprofng) The third patch adds a testsuite in gprofng/testsuite. The fourth patch adds a Texinfo manual for gprofng. The manual is still WIP but already provides a tutorial-like introduction to the tools. Limitations =========== The gp-display-html tool is present, and can be executed, but it is not functional yet. Full support for this tool is expected to be delivered in a future patch. Requirements =========== In order to successfully build gprofng, the following versions of external components are required: - Bison 3.7.5, or higher - Texinfo 6.7, or higher - Java include files (--with-jdk=PATH) if java profiling should be enabled Maintenance =========== We are of course volunteering to maintain gprofng once it is incorporated into the main binutils distribution. We suggest having feedback and discussion in this mail thread.