From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 123245 invoked by alias); 21 Dec 2016 15:27:39 -0000 Mailing-List: contact libc-alpha-help@sourceware.org; run by ezmlm Precedence: bulk List-Id: List-Subscribe: List-Archive: List-Post: List-Help: , Sender: libc-alpha-owner@sourceware.org Received: (qmail 123230 invoked by uid 89); 21 Dec 2016 15:27:38 -0000 Authentication-Results: sourceware.org; auth=none X-Virus-Found: No X-Spam-SWARE-Status: No, score=-1.9 required=5.0 tests=AWL,BAYES_00,RCVD_IN_DNSWL_NONE,SPF_HELO_PASS,SPF_PASS autolearn=ham version=3.3.2 spammy=blocked, tm, remembered, consists X-HELO: EUR01-VE1-obe.outbound.protection.outlook.com Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=Szabolcs.Nagy@arm.com; Message-ID: <585A9F58.8040603@arm.com> Date: Wed, 21 Dec 2016 15:27:00 -0000 From: Szabolcs Nagy User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.8.0 MIME-Version: 1.0 To: Torvald Riegel CC: , GNU C Library Subject: Re: [RFC][BZ #19329] high level description of pthread_create vs dlopen races References: <583C7F76.3000601@arm.com> <1480965875.14990.199.camel@redhat.com> <5846F3EB.2060003@arm.com> <1481119669.14990.511.camel@redhat.com> In-Reply-To: <1481119669.14990.511.camel@redhat.com> Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit X-ClientProxiedBy: HE1PR06CA0065.eurprd06.prod.outlook.com (10.164.28.161) To AM5PR0802MB2484.eurprd08.prod.outlook.com (10.175.42.151) X-MS-Office365-Filtering-Correlation-Id: 9c02914e-3ea1-4491-ac72-08d429b5ddd4 X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:(22001);SRVR:AM5PR0802MB2484; X-Microsoft-Exchange-Diagnostics: 1;AM5PR0802MB2484;3:g/us1ZgNZVjyqHIHsh8iMmfFq3zFSul/HUQZx8XRvQ3FFxOmqZu1LVbWKx5gGb4e7A5NzgHqo3KOTQd1/g53Bf+HaZ8vpScE65IiKOijtq9spNoMj4+99kNjRjwPWZeJa2gfSh2Ks7yGnBw/4LCZSP8eM+EpGo2N05KCBR2d15P+fs3fEoSwnuELLQ9Nb27FPq6WIHHP6vCTx0OxNcqOM32C9fdrizESoia7vpH1kzeOGOSaph+LmGWbpZ6cxZRq/n0KyWC03uFBkRXcneQD2g==;25:F8Ss68OoPFlE2Jr70wsyc8MzC0gTOfY4jmz7Og124zxAWw0yvfbeo6uonuPIMkpvMy/QViL6kal2xYTTuHgQZn8oOFjK/sEyKcv5y7RdOrLCLOOtcS040V33szIsRP+CbgODYEKfuB0fvcWd3jMggmouONMWmXZUVYF6japNO9hfMa3xPiRZdKI1CtscLq5ZOWaRT3OHeRj2d41P2KSWqjfQcQgIZgiokvQtvomorB4zs4bkQ/M7RRfbCwkbxVrjtAB01Ofld6NfJskRMq940UkpKWEbkYZ9nBgbzX8fSCeuo6Kw8DRVpKPzpIuBlibK0sL7gkP6N4zZ9EhDPDS3k0y/plD/QVg22ZGYaaovqRuZ9GdeugsyArE0KEaIY2MhwIdrAOozIRKyyL1mL7rdkD8ugqfZIyZW+YLU5MlgFQPvIj2gV40I9UXqEmRuQAPwSgh61EvYTR36sV381nD4jA== X-Microsoft-Exchange-Diagnostics: 1;AM5PR0802MB2484;31:0Aobh8G3rCqTv+Oanlv3dSRcFMKPq0yCCj871WFINJ/Uec3fd/+d07b7/b/INW14ugtteQ1ZFAzWC6PANbIUwnAUzwANfeylPS+91Vy7lUPqYOLR6p8A4p2d7WSFXlbZYfWqOetpPQfV9JQcD+Luhjwom8gR8njlNj2vOft1Z5f3acslnxNpbe74xQG9AylVjONLQ++ulBuxvO52eT0bvfY04Kvr/SaLGjdKv5xTtGB6OZnpsVVE1W0I+Aglkz+y;20:YjdUMoHNax5gnQQuVu9MQO7yNqD/FotFmufLKM2tYA1UI8fDyvdn6sQZn9iG88z5WIcZ1LXEokS3JkBjQ2M8xoCdZKrESPWIYljbfT/skXs6ZuMEX8PoYfaioekMGq7NDy1daG9khXroN6OGM6qcYAW7LheaTJQrQCGNlt1GEsk= NoDisclaimer: True X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(6040375)(601004)(2401047)(8121501046)(5005006)(3002001)(10201501046)(6055026)(6041248)(20161123562025)(20161123560025)(20161123555025)(20161123564025)(6072148);SRVR:AM5PR0802MB2484;BCL:0;PCL:0;RULEID:;SRVR:AM5PR0802MB2484; X-Microsoft-Exchange-Diagnostics: 1;AM5PR0802MB2484;4:W3zfdOUJxEu7GHBltvoBXJLuQuyzcolq4UlzPgbot3WQSv+fEtK9g/INaGDDgJn6n13lh7xDgZXy3jCSIuJtaAx/v9DrSpBMVGwpCOmTz/f2H/ufzwWzbqz2FhtrfU0dp12U2FZQ0uZXwCnQNi4+xVMIXTKA67JiYfXpr8h5va5x18GLM4fBcSaRau25oe6n3oz8ZiL+4W2WpDRgIV8WCOBKJm17p6ZnIUkALmH0bGKt4YZ7t+XfXMymwX/DfqRkZPEHlIcCX/sZ18FoxEgnU0VH8kuPGpsPKi9yvLKa6eZViPflB81t9iRwYmkO9no2Jq1VgWW2A8Hzq9vDXcbyBWBkLxdBuqi3mSR0bSI1oF/BcPymp26fNQZF1RCIqlU71cFKbq+bce10ovhRfxjED9JJGQXo/EeKxPKbKuEwlE2Cuh7gb3PVhMHLslPQLzpPz0NjlpVsIIGvcrywg6qtJBwpa5QtSc9GFFbRIZLVVuaXs+jrM+di29g3lSo+YgyX5G7SZR4JuecqZ8F+hbjsC5KfeA+EaD6P6AjmvOjKVPG/8ieb2N8RL7v3iHq3y3QOBrJtoGsd0rxmrVqzqoNV2g== X-Forefront-PRVS: 01630974C0 X-Forefront-Antispam-Report: SFV:NSPM;SFS:(10009020)(4630300001)(6009001)(6049001)(7916002)(39850400002)(39860400002)(39840400002)(39410400002)(39450400003)(377424004)(24454002)(51444003)(199003)(189002)(189998001)(2906002)(42186005)(305945005)(7736002)(33656002)(3846002)(97736004)(101416001)(64126003)(6116002)(4001150100001)(4001350100001)(66066001)(4326007)(50986999)(80316001)(65956001)(65806001)(76176999)(87266999)(54356999)(47776003)(65816999)(6666003)(229853002)(92566002)(23676002)(6916009)(2950100002)(83506001)(59896002)(8676002)(38730400001)(105586002)(81166006)(81156014)(90366009)(36756003)(230700001)(68736007)(5660300001)(25786008)(6486002)(50466002)(106356001)(77096006)(110136003)(93886004)(86362001);DIR:OUT;SFP:1101;SCL:1;SRVR:AM5PR0802MB2484;H:[10.2.206.73];FPR:;SPF:None;PTR:InfoNoRecords;A:1;MX:1;LANG:en; Received-SPF: None (protection.outlook.com: arm.com does not designate permitted sender hosts) X-Microsoft-Exchange-Diagnostics: =?utf-8?B?MTtBTTVQUjA4MDJNQjI0ODQ7MjM6RFBvdC9sS0x3MWNzZHJacUUybk1Uek9Y?= =?utf-8?B?NGVxODBqclhsRE5ia01EY01oRjVCUHdxSGVGOWRZL3c3eWsrOXRYVXc0Y1Nk?= =?utf-8?B?QU9UcWQzcnN4S3lXTGVhRVl6bWV5TWxOTHVTSE1YaDJkV3dDRGp4dHEwZzRp?= =?utf-8?B?NG9SVGdSVXlqWll3OEI4MFlSWDFlbVpONCtURXZFTmNVNFhkMkJVWmhpVzQr?= =?utf-8?B?aDhoWi9pR2czTzgvYllGVEcyUG11dExDcVB0VyttSHB5TnhrT0M5akxHQzVu?= =?utf-8?B?TGlIMlRWRFhIbWtxMDI5bFZyTlNRRTlXMVJzYjhta2JUeXFVVldKa3orR29B?= =?utf-8?B?N0VSZ3ZNeHlISUtUdStCemxQT2doVUZqNHBjaUM2ZCtScmZjV0ZrZmZvL2p5?= =?utf-8?B?RWJ3NVFHRHRCc3dERjZHeHYyditrMkFxVUdiV2YxMjViWmZyL1dCSkFvSVZK?= =?utf-8?B?WlBuN3NQNHM3aGdBRi9yMDdQNDV3ekdmNXZhcUNsODJmU1VLUWVIcmRkZk5j?= =?utf-8?B?dEx4blNPdS9Fd1VEb0VrSDU1M0xUYkVkMHJiRkpCVGNBdGFRVFBoNndkc0Zj?= =?utf-8?B?UXFiZ2VYQUNQdHdMQlUzekY1MGVRMmYxSDI1a0t2Z0dDMmNzYklRREx3YkJQ?= =?utf-8?B?NXArcWovRVZuTnNDMGorTlhtNVNiSjBxTERjd0VzdnY0Y2QySjNXUFZETms1?= =?utf-8?B?Yk9EVXV5NjR0MzJIM3A2SlJOc1pmTHRCQ2NsL0Y5d1BjOHdBZCt2ZHdjRzE2?= =?utf-8?B?ZTJVdW16SUljWlpBUXpOMTZ0b3g2RmVvbDZ2NEdNRXJMSXErWlFGUnRycVlp?= =?utf-8?B?OHBTelByWkdBTWVtaks0d3NGVzRiWE4zdTRWYzdlR0J3cEludDBKdWJqTzhJ?= =?utf-8?B?V3FuYWcxUGVqbWJtRVJsclA2aHhNM3JLWDJ2SGtJMXcrVGkvVE9zTlNEODMv?= =?utf-8?B?Qjd6aU1kTHZQSEVlblE4QkZpMTlKNDQ1VmRRODVzNjgvUmptSUtGRE1JWkhk?= =?utf-8?B?ZE1nR0ZIQWd6aDFJMVpOYnBoYi9wV1VTOE1OSVFYcE1sSFk5cU1UdkFSbzgx?= =?utf-8?B?MGZVenBSMFY2SW5TNUhJKzF0dkRsUUd5SXJ2a2xTRVBQUUNrUnZtUWV3TUt0?= =?utf-8?B?ZWU1R24zRDFLTC9BMHNHKy9YVitEeVQzbE9BQm9MeG1WTkxrTWlYWWk2b2o3?= =?utf-8?B?aFRaVjIrYVNuVS9PMnY5NkVkck1CRzFMbGZDMmwva0FHTzNpMkxlbjVFcURk?= =?utf-8?B?T1NiV05MSGx4NVd1aWYvSGxaZ25Zai9JbnJtamwyM3czSUZ6UVQ0eFVQZk9B?= =?utf-8?B?eXJOdkpabWpFRHFBNEZIM3FUS2wyMTRrMnJUR2pzalJ3Vjlva1pSTXFyMGR6?= =?utf-8?B?NE10Yzh1Wmp6YnFTUUk1Z1BvMkozdjhuek56WWhXcUMyTG9iT0V5dDNXTGxj?= =?utf-8?B?MEFDbFNUSUl1b25HbHlMV1lRanZ3RGV0aklIR0d3Rmw4ZDhTZlJXRFh1dHE2?= =?utf-8?B?c25ubHVpT0M2WmZLV09ONmxwL05yNUpZb2tuQVNSNkhiZE8zcm1GdDFLVTcy?= =?utf-8?B?MXYvSTVBellRaGs1QmFjVUg3TWYyVVVJT0hrdHF6UHF5RVBnc1dPSFJqS0lu?= =?utf-8?B?RWlkWW5oRXhzd2N5THR0c2dRZUcyWGZWNVkzN0MvejNGRCtidXlHK2JCaVI5?= =?utf-8?B?SDY3cld3anJEYzhvRzA4Q0dLZjNVV21kSmlhQzBoVFNQUTNyNnVaQmhUTzM2?= =?utf-8?B?ZjVxVzRMQXVOY0pKbmVjZWlsYXMydCtoYkpDL3U0QkVhYzJsVVpkOWZLWmd4?= =?utf-8?B?SVh1ekRVRlo0TmRFMDJET21VWkxnMjJlWDdRTjl1Y1dhV0FCNVpUUVZvSk9k?= =?utf-8?B?S2hncWVpYTV6QXJoSkt1M1ZiQzh4VnJpS252NmRKQkxURlphYmRzcTZnd3l0?= =?utf-8?B?Rjl2OXZlNlZuSjZtZFErbjhlQkFtRnVadXliNnBhZFdERlpGTjUrQktjNU5m?= =?utf-8?B?RTY3WWFMOExvTEMzUmtPQlpiTnJVRDhWOS9LaW5JbDViMkF3VXVTaXVOc0hh?= =?utf-8?B?d0s1anpKdlVBaEtmcFY5dTBkODU4S3h6VHAxZFNFV3VuZUljT3p5Yk0xRVNL?= =?utf-8?B?cnB4dz09?= X-Microsoft-Exchange-Diagnostics: 1;AM5PR0802MB2484;6:IThY3lsDfaIfOTQatY/sCZ5/VyweNoze2rINOwwOpClKT472S1kWA/iYl3GldkIaIvWKedc1rrFY4QRA6x+U2AOWO93KXUTCPHZRamWwLETKA6noX1tL7pwNbv4gyOi2H43QnDpNjsvVBWbC1ODIGjDWjSAn2oL+Ie6HB/VnDfPx89O9bAC/8Cu4qB95/P9AjqHsUc/EwPW59UrCNPJWLGwIj9ENpoIxtsAAn5H1nnl92B8kJoD+tX5M2xQvaozPQwEQs4u1QJR8Puf9/I+1KbsTHRGLObacuXJC0mRPaWjPqDZ7qpeerztFy2CVvZyLf90W3+KQt8UO/Cnz9G93jP5z2YqriXkrpHvMALggZ/1QfBlVUswh0JJVont5cmfeB3EpR3MfsldxgB5HW3ecCpc7c4z4u+Jm5f3olyZB/PTOhn5x/NqrRaZm+DRNCgjaSfxREynr+JARu34HNyYPbw==;5:XbXQD3CJnvp6NHzqTgt9nXs8vBo2IN0qKup/B+vtCyanbVGL4ZCfcprYQ67lCeh+SRDJyJGpk1mT1i1n9QZTKv9mZr8a7b6VAl8mJ6NBw+gPICFN0TAnkrIGIlGc1NYlbA0B9NdGs2ybX4WdysxXMQ==;24:mEwXaTLaoUUl6RQZmR6bwcvcrVc8kftS/RHb+3/E1tUoqDXUFQApSWRjSm/13bxsNIVpbwxwnFS9AyHUYoHQXwxDb0e9bl+nenK8eAxcReI= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1;AM5PR0802MB2484;7:PBxJy4Rn7mKMWVoXW4bDj38T45BEUJa5tcvLIkm5jVCMfsrxkzxrnzayDbECVFCkmzBS2y7ItC1EDTpudVUlW0X4u5f4A3LKt21+vQyynRiQnews4OJckdMF59OwAmHN5C/KW7E6GSb+zmyfN6uBGLcTvDC8wp5e3/vj8olVc4utz8gcIGxBqjOQn0HmRXtRpJIauznIoL5si7btHLMlrMHDBoKiFibzKJQ4SzNdXwJ7OpmftMQG2TLO5s6uvCAE0kpgJzlTMQQ4l63siO0iruX7frOuJqk4Cxe4YY1n/Lid60SpSbaJ/Lst/N4uSh4Q6daU+PXlokmC8tD9JXbPsbTswtY0yLxxPBlj8KZoPsLW6gaNGi7AgkMLGVZLT7m4sv6+EPOAdb9hhPLOT15aO6qyA7xXIueKH+Ucgslp13KNjGtGYAmeCgmw28lYPloDGhYlJd48HfBwwvELpk0Klw== X-OriginatorOrg: arm.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 21 Dec 2016 15:27:24.9671 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM5PR0802MB2484 X-SW-Source: 2016-12/txt/msg00815.txt.bz2 On 07/12/16 14:07, Torvald Riegel wrote: > On Tue, 2016-12-06 at 17:22 +0000, Szabolcs Nagy wrote: >> On 05/12/16 19:24, Torvald Riegel wrote: >>> On Mon, 2016-11-28 at 19:03 +0000, Szabolcs Nagy wrote: >>>> (5) GL(dl_tls_generation) is not checked for overflow consistently and >>>> slotinfo entries can get assigned an overflowed generation count, its type >>>> is size_t and both dlopen and dlclose increment it, so overflow may be a >>>> concern on 32bit targets. >>> >>> Can the size be extended on 32b targets? If so, we can build a >>> monotonically increasing, larger atomic counter on 32b targets too. >> >> the generation count up to which a dtv is set up in a >> thread is stored in dtv[0] which is 32bit on a 32bit >> system. > > The other 32b half could be somewhere else. It's more efficient if it's > in the same cacheline of course, but it's not required to be adjacent. i was wrong about this: a dtv entry has 2 pointers already so it is 64bit on a 32bit target so a 64bit counter fits. >>> Before focusing on taking the lock, I'd suggest to look at the bigger >>> picture (ie, the abstract operations and atomicity and ordering >>> requirements). >>> It can be possible that a clean redesign is actually simpler than trying >>> to fix a system that is already complex and which we don't understand >>> 100%. >> >> i think that's optimistic but i can write up what i think >> the requirements are. > > Thanks. Better documentation of what the requirements are would help > even if we a rewrite is not what we decide to do. i tried to collect the requirements, the current implementation is far from it: as-safety of tls access and ctor/dtor without locks are difficult to fix. if as-safe tls access is not required then the concurrency fixes are much simpler. simplified model: m: module. m.i: modid. t: thread. t.dtv: dtv pointer. t.dtv[i]: dtv entry i (tls for t,m where m.i==i). constraints: tls of a currently loaded module m in thread t is at t.dtv[m.i]. m.i is unique during the lifetime of m: it is assigned before ctors of m are entered and may be reused after dtors of m return. tls access to m in thread t is only valid during the lifetime of t and m (after ctors of m start and before dtors of m end). during the lifetime of a thread its dtv may be updated: t.dtv may need to be resized (if an m is loaded with larger m.i). t.dtv[i] may need to be freed (if an m is dlclosed). t.dtv[i] may need to be allocated (if an m is dlopened). if dtv updates are not immediate for all threads at dlopen and dlclose time, then the threads need a way to tell if t.dtv[i] is out-of-date in case modid i is reused by the time the dtv update happens. (this can be done by counting dlopen/dlclose calls and remembering the counter of the last dtv update and globally tracking the last counter for each modid i, if global_counter[i] > t.dtv_counter then t.dtv[i] is out-of-date. such counter should be 64bit). dtv update consists of three kind of operations: 1) allocate dtv and dtv entries (malloc) 2) unset out-of-date entries (free) 3) resize dtv, set dtv entries (memcpy and ptr update) 1) alloc: pthread_create and dlopen needs to be synchronized such that either sync in dlopen or sync in pthread_create happens before the other (for all dlopen/pthrea_create pairs), the one that happens later should do the allocation. (this is needed because right after dlopen and pthread_create tls access is valid, but must be as-safe so it cannot easily allocate). 2) free: t.dtv[m.i] should be freed eventually after dlclose of m or after t exits. this is difficult because t.dtv[m.i] need to be updated if m.i is reused and the tls of the new module is accessed, but tls access cannot do the free (not as-safe). so the options are - dlclose of m frees t.dtv[m.i] for all t (non-trivial). - allocated dtv entry pointers are remembered somewhere and garbage collected in some way (such that overwriting t.dtv[i] does not leak memory). 3) update dtv pointer and dtv entry: either dlopen stops all threads and takes care of dtv updates or it has to be done during tls access lazily in which case signals may need to be blocked. an m with static tls is a special case: 1) if m is already loaded when t is created, then pthread_create needs to do setups (copy tls init image) that requires accessing m, so either pthread_create needs to sync with dlclose or it is invalid to unload an m with static tls, in the later case pthread_create should be able to walk the currently loaded modules and tell if they have static tls without accessing the module structures that are freed during dlclose. 2) if m is loaded after t is created, then dlopen should do the setup for the current thread, but i think it has to do the setup for other threads as well (?). (in principle static tls cannot be dynamically loaded/unloaded but i'm not sure what are the requirements if glibc internal libraries are dlopened.) indirectly related to tls: ctor,dtor handling should be callback safe: dlopen and dlclose must not hold internal locks otherwise correct ctor/dtor code may deadlock. dlopen and pthread_create should roll back state changes on allocation failure and report the errors (some state changes may be harmless this needs further analysis).