From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-qv1-xf2d.google.com (mail-qv1-xf2d.google.com [IPv6:2607:f8b0:4864:20::f2d]) by sourceware.org (Postfix) with ESMTPS id B75983858D33 for ; Wed, 7 Apr 2021 18:36:52 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.3.2 sourceware.org B75983858D33 Received: by mail-qv1-xf2d.google.com with SMTP id fn8so4692721qvb.5 for ; Wed, 07 Apr 2021 11:36:52 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=ia8pSapQlKwqPK9U0rvGgEkeIsRRNW8e2F4hS3wsOpk=; b=WRkrfRp1mAAQvrs3Ve42BiguGJY5NsYueXyz/ZD4ZIhslK1kPnETdcFGVcxu8wGQJW +CjMnEMICbMmWOcdPZQAT5lRc5NxuqEh3QvXL8vjogUH4FHKhQbkULT8Shb+j87Z7hvU +vAYIfwVnZCrClwznqWbAYNRmFMXxSRcqBpxzrp1zVjAOpvZosZJ3woIuq+SvwU+APAw X3DWVVc5Rx0Kj1FMH49dv608QHhRUbrcHussu75Adx0JPmKDdiNYrdWXGR8HkGVPoK8g GH2mK2i9IJjwkZVmVulxj40MW14d61P0KpRLwI2LfPZl64aX42TNIt8gFdC7repuqraU kEWQ== X-Gm-Message-State: AOAM533q9baoTALW/j+OdEt5haJovbcmXm+hInvBFs+yibxYF8dQwzFP cvg1MpyBEldrjbD2PdMZVKqvfA== X-Google-Smtp-Source: ABdhPJwBMLHPekPzUVau5rzmsZheLNj3RoK/bOixxbrsfv5GfNWOg7RMTlBTAx3uznMIM8rwJUEU7w== X-Received: by 2002:ad4:57a5:: with SMTP id g5mr4892605qvx.60.1617820612224; Wed, 07 Apr 2021 11:36:52 -0700 (PDT) Received: from [192.168.1.132] ([177.194.41.149]) by smtp.gmail.com with ESMTPSA id w5sm18859763qkc.85.2021.04.07.11.36.50 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 07 Apr 2021 11:36:51 -0700 (PDT) Subject: Re: [PATCH] linux: sysconf: limit _SC_MAX_ARG to 6 MiB [BZ #25305] To: Michal Nazarewicz , libc-alpha@sourceware.org, carlos@systemhalted.org, Florian Weimer References: <20210407151058.1176364-1-mina86@mina86.com> From: Adhemerval Zanella Message-ID: <6c926d3b-3094-f220-7777-91dacf975275@linaro.org> Date: Wed, 7 Apr 2021 15:36:48 -0300 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.7.1 MIME-Version: 1.0 In-Reply-To: <20210407151058.1176364-1-mina86@mina86.com> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit X-Spam-Status: No, score=-12.5 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, NICE_REPLY_A, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on server2.sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Apr 2021 18:36:54 -0000 On 07/04/2021 12:10, Michal Nazarewicz wrote: > Since Linux 4.13, kernel limits the maximum command line arguments > length to 6 MiB.¹ Normally the limit is still quarter of the maximum > stack size but if that limit exceeds 6 MiB it’s clamped down. > > glibc’s __sysconf implementation for Linux platform is not aware of > this limitation and for stack sizes of over 24 MiB it returns higher > ARG_MAX than Linux will actually accept. This can be verified by > executing the following application on Linux 4.13 or newer: > > #include > #include > #include > #include > #include > > int main(void) { > const struct rlimit rlim = { 40 * 1024 * 1024, > 40 * 1024 * 1024 }; > if (setrlimit(RLIMIT_STACK, &rlim) < 0) { > perror("setrlimit: RLIMIT_STACK"); > return 1; > } > > printf("ARG_MAX : %8ld\n", sysconf(_SC_ARG_MAX)); > printf("63 * 100 KiB: %8ld\n", 63L * 100 * 1024); > printf("6 MiB : %8ld\n", 6L * 1024 * 1024); > > char str[100 * 1024], *argv[64], *envp[1]; > memset(&str, 'A', sizeof str); > str[sizeof str - 1] = '\0'; > for (size_t i = 0; i < sizeof argv / sizeof *argv - 1; ++i) { > argv[i] = str; > } > argv[sizeof argv / sizeof *argv - 1] = envp[0] = 0; > > execve("/bin/true", argv, envp); > perror("execve"); > return 1; > } > > On affected systems the program will report ARG_MAX as 10 MiB but > despite that executing /bin/true with a bit over 6 MiB of command line > arguments will fail with E2BIG error. Expected result is that ARG_MAX > is reported as 6 MiB. > > Update the __sysconf function to clamp ARG_MAX value to 6 MiB if it > would otherwise exceed it. This resolves bug #25305 which was market > WONTFIX as suggested solution was to cap ARG_MAX at 128 KiB. > > As an aside and point of comparison, bionic (a libc implementation for > Android systems) decided to resolve this issue by always returning 128 > KiB ignoring any potential xargs regressions.² > > On older kernels this results in returning overly conservative value > but that’s a safer option than being aggressive and returning invalid > value on recent systems. It’s also worth noting that at this point > all supported Linux releases have the 6 MiB barrier so only someone > running an unsupported kernel version would get incorrectly truncated > result. > > ¹ See https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=da029c11e6b12f321f36dac8771e833b65cec962 > ² See https://android.googlesource.com/platform/bionic/+/baed51ee3a13dae4b87b11870bdf7f10bdc9efc1 It is unfortunate that Linux did not provide a dynamic way to obtain this value, specially since it has changed over versions. So we will need to continue using heuristics. And I am not sure by what Carlos has stated in last comment from BZ#25305 since the only information we have is the maximum stack from getrlimit: | You need to implement dynamic detection of the usable value between | the two extreme values depending on your application requirements | and kernel behaviour. There simply isn't a portable way to optimize | for the largest value. I don't see how we can't really get the maximum upper limit with current kernel interfaces unless we parse the kernel version and start to mimic the kernel heuristics (which has its own issues). And I am not sure if this characterizes as a performance regression for process like xargs: it seems to use a conservative value for internal arg_max (131Kb) and limits to about 2MB (1/4 of current 8MB stack limit): $ ./xargs/xargs --show-limits Your environment variables take up 4599 bytes POSIX upper limit on argument length (this system): 2090505 POSIX smallest allowable upper limit on argument length (all systems): 4096 Maximum length of command we could actually use: 2085906 Size of command buffer we are actually using: 131072 Maximum parallelism (--max-procs must be no greater): 2147483647 It would be a problem only uses start to play with -s option, but I don't see a better way to handle. Same for programs that tries to use large _SC_MAX_ARG, with current behavior they will need to handle possible E2BIG from execve on newer kernels. Carlos and Florian, what possible issues do you see by limiting _SC_MAX_ARG to what kernel now expects? IMHO this should align with what Linux supports now. > --- > sysdeps/unix/sysv/linux/sysconf.c | 5 ++++- > 1 file changed, 4 insertions(+), 1 deletion(-) > > diff --git a/sysdeps/unix/sysv/linux/sysconf.c b/sysdeps/unix/sysv/linux/sysconf.c > index 366fcef01e..bd711795c7 100644 > --- a/sysdeps/unix/sysv/linux/sysconf.c > +++ b/sysdeps/unix/sysv/linux/sysconf.c > @@ -55,7 +55,10 @@ __sysconf (int name) > struct rlimit rlimit; > /* Use getrlimit to get the stack limit. */ > if (__getrlimit (RLIMIT_STACK, &rlimit) == 0) > - return MAX (legacy_ARG_MAX, rlimit.rlim_cur / 4); > + { > + const long int limit = MAX (legacy_ARG_MAX, rlimit.rlim_cur / 4); > + return MIN (limit, 6 << 10 << 10); > + } > > return legacy_ARG_MAX; > } >