From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-pf1-x42a.google.com (mail-pf1-x42a.google.com [IPv6:2607:f8b0:4864:20::42a]) by sourceware.org (Postfix) with ESMTPS id B6E10385743E for ; Tue, 8 Jun 2021 04:45:37 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org B6E10385743E Received: by mail-pf1-x42a.google.com with SMTP id h12so11942547pfe.2 for ; Mon, 07 Jun 2021 21:45:37 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:subject:to:cc:references:in-reply-to :mime-version:message-id:content-transfer-encoding; bh=shODUYCaNFDiwECqxn3wjQe6HiQJ4Xmfy6/QLkqCTj0=; b=CCOBBlnO2sg7I1bzz3NGmFCZ8145sWxIh80uIwZco7Kb7dAxO6To/vU3EJ5wO9jtiK J+NTJQDG3h2nbBzmJCqATDdqc9Qv5zoirBcvxfmPh3QWWVNPMoT/IxcyIjSaY5Y2aNVp XFjarFDKF7Cs8RT4QYnPn68s9qdGvFQ9yGj6NfTBTW9XPk1+TIuQt4MFxWeZDxM+lNcp F6D/9QdvsUZgU5pJWDWtDZ6BBOztA92LnR7Kkh0+xWy7fT7C2gls5SzpsbFkaFBph0Rx EvFVvIYxQ6AVHaZDtAY7+sYUISNnGAZqJBb9UtscOf5EFvrj+cqgk3J7Eow3fqXvH06j Af+g== X-Gm-Message-State: AOAM531buh4swNQdHa8LNH8GIlTfKcEDV0ySKQFoUy1YgNEJ8aSw+9dr dFNCzu877KAWKtjg/zi5DDo= X-Google-Smtp-Source: ABdhPJzo/Il7mH42sxJ9pqb+2MljKEc8Fi+ttQDpWQawgD1x5JDzvIavvyCaTIekl88MdUSoxOAf3g== X-Received: by 2002:a62:60c4:0:b029:2ca:ebf7:cd0d with SMTP id u187-20020a6260c40000b02902caebf7cd0dmr20270348pfb.71.1623127536730; Mon, 07 Jun 2021 21:45:36 -0700 (PDT) Received: from localhost (60-242-147-73.tpgi.com.au. [60.242.147.73]) by smtp.gmail.com with ESMTPSA id v11sm950423pju.27.2021.06.07.21.45.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 07 Jun 2021 21:45:36 -0700 (PDT) Date: Tue, 08 Jun 2021 14:45:31 +1000 From: Nicholas Piggin Subject: Re: [PATCH v4 00/15] Add futex2 syscalls To: =?iso-8859-1?b?QW5kcu+/vQ==?= Almeida , Davidlohr Bueso Cc: acme@kernel.org, Andrey Semashev , Sebastian Andrzej Siewior , corbet@lwn.net, Darren Hart , fweimer@redhat.com, joel@joelfernandes.org, kernel@collabora.com, krisman@collabora.com, libc-alpha@sourceware.org, linux-api@vger.kernel.org, linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, malteskarupke@fastmail.fm, Ingo Molnar , Peter Zijlstra , pgriffais@valvesoftware.com, Peter Oskolkov , Steven Rostedt , shuah@kernel.org, Thomas Gleixner , z.figura12@gmail.com References: <20210603195924.361327-1-andrealmeid@collabora.com> <1622799088.hsuspipe84.astroid@bobo.none> <1622853816.mokf23xgnt.astroid@bobo.none> <22137ccd-c5e6-9fcc-a176-789558e9ab1e@collabora.com> <20210608023302.34yzrm5ktf3qvxhq@offworld> In-Reply-To: <20210608023302.34yzrm5ktf3qvxhq@offworld> MIME-Version: 1.0 Message-Id: <1623125574.l2jo2w0x2v.astroid@bobo.none> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Spam-Status: No, score=-1.7 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, FREEMAIL_FROM, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on server2.sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Jun 2021 04:45:48 -0000 Excerpts from Davidlohr Bueso's message of June 8, 2021 12:33 pm: > On Mon, 07 Jun 2021, Andr=C3=AF=C2=BF=C2=BD Almeida wrote: >=20 >>=C3=80s 22:09 de 04/06/21, Nicholas Piggin escreveu: >>> Actually one other scalability thing while I remember it: >>> >>> futex_wait currently requires that the lock word is tested under the >>> queue spin lock (to avoid consuming a wakeup). The problem with this is >>> that the lock word can be a very hot cache line if you have a lot of >>> concurrency, so accessing it under the queue lock can increase queue >>> lock hold time. >>> >>> I would prefer if the new API was relaxed to avoid this restriction >>> (e.g., any wait call may consume a wakeup so it's up to userspace to >>> avoid that if it is a problem). >> >>Maybe I'm wrong, but AFAIK the goal of checking the lock word inside the >>spin lock is to avoid sleeping forever (in other words, wrongly assuming >>that the lock is taken and missing a wakeup call), not to avoid >>consuming wakeups. Or at least this is my interpretation of this long >>comment in futex.c: >> >>https://elixir.bootlin.com/linux/v5.12.9/source/kernel/futex.c#L51 >=20 > I think what Nick is referring to is that futex_wait() could return 0 > instead of EAGAIN upon a uval !=3D val condition if the check is done > without the hb lock. The value could have changed between when userspace > did the condition check and called into futex(2) to block in the slowpath= . I just mean the check could be done after queueing ourselves on the wait=20 queue (and unlocking the waitqueue lock, not checking while holding the lock). That is the standard pattern used everywhere else by the kernel: prepare_to_wait() /* -> lock; add_wait_queue; unlock; */ check_again(); schedule(); It can still return EAGAIN if there is a reasonable use for it, but I'd be wary about user code that cares about this -- it's racy you could=20 arrive right before the value changes or right after it changes, so any user code checking this I would be suspicious of (I'm willing to see a use case that really cares). >=20 > But such spurious scenarios should be pretty rare, and while I agree that > the cacheline can be hot, I'm not sure how much of a performance issue th= is > really is(?), It's not a spurious anything. The problem is the contention on the lock word cacheline means it can take a relatively long time just to perfor= m that one load instruction. Mandating that it must be done while holding the lock translates to increased lock hold times. This matters particularly in situations that have lock stealing,=20 optimistic spinning, reader-writer locks or more exotic kind of things=20 that allow some common types of critical section to go through while others are blocking. And partiuclarly when such things hash collide on other futexes that share the same hash lock. > compared to other issues, certainly not to govern futex2 > design. Changing such semantics would be a _huge_ difference between fute= x1 > and futex2. futex1 behaviour should not govern futex2 design. That's the only nice=20 thing you get with an API change, so we should take full advantage of=20 it. I'm not saying make changes for no reason, but I gave a reason, so=20 that should be countered with a better reason to not change. Thanks, Nick >=20 > At least compared, for example, to the hb collisions serializing independ= ent > futexes, affecting both performance and determinism. And I agree that a n= ew > interface should address this problem - albeit most of the workloads I ha= ve > seen in production use but a handful of futexes and larger thread counts. > One thing that crossed my mind (but have not actually sat down to look at= ) > would be to use rlhastables for the dynamic resizing, but of course that = would > probably add a decent amount of overhead to the simple hashing we current= ly have.