Fwd: gawk core dumped on too many input values

Fwd: gawk core dumped on too many input values

[Ed Morton]

[-- Attachment #1: Type: text/plain, Size: 1745 bytes --]

This (original email below) turned out to be a general cygwin issue, not 
a gawk issue:

$ LC_ALL=C sed 's/x/y/' $(seq 1000000)
Segmentation fault (core dumped)

$ LC_ALL=C grep 'foo' $(seq 1000000)
Segmentation fault (core dumped)

Regards,

     Ed.

-------- Forwarded Message --------
Subject: 	gawk core dumped on too many input values
Date: 	Sun, 27 Aug 2023 08:09:54 -0500
From: 	Ed Morton <mortoneccc@comcast.net>
To: 	bug-gawk@gnu.org <bug-gawk@gnu.org>



Configuration Information [Automatically generated, do not change]:
Machine: x86_64
OS: cygwin
Compiler: gcc
Compilation CFLAGS: -ggdb -O2 -pipe -Wall -Werror=format-security 
-Wp,-D_FORTIFY_SOURCE=2 -fstack-protector-strong 
--param=ssp-buffer-size=4 
-fdebug-prefix-map=/cygdrive/d/a/scallywag/gawk/gawk-5.2.2-1.x86_64/build=/usr/src/debug/gawk-5.2.2-1 
-fdebug-prefix-map=/cygdrive/d/a/scallywag/gawk/gawk-5.2.2-1.x86_64/src/gawk-5.2.2=/usr/src/debug/gawk-5.2.2-1 
-DNDEBUG
uname output: CYGWIN_NT-10.0-22621 TournaMart_2023 3.4.8-1.x86_64 
2023-08-17 17:02 UTC x86_64 Cygwin
Machine Type: x86_64-pc-cygwin

Gawk Version: 5.2.2

Attestation 1:
         I have read 
https://www.gnu.org/software/gawk/manual/html_node/Bugs.html.
         Yes

Attestation 2:
         I have not modified the sources before building gawk.
         True

Description:
         I was trying to test something related to ARG_MAX when I ran the
         awk script below and it core dumped instead of reporting an error
         and exiting gracefully. In case it's useful getconf ARG_MAX outputs
         32000.

Repeat-By:
         $ LC_ALL=C awk 'BEGIN{print ARGC}' $(seq 1000000)
         Segmentation fault (core dumped)

Re: gawk core dumped on too many input values

[Jeremy Hetzler]

[-- Attachment #1: Type: text/plain, Size: 2780 bytes --]

On Sun, Aug 27, 2023 at 2:25 PM Ed Morton via Cygwin <cygwin@cygwin.com>
wrote:
>
> This (original email below) turned out to be a general cygwin issue, not
> a gawk issue:
>
> $ LC_ALL=C sed 's/x/y/' $(seq 1000000)
> Segmentation fault (core dumped)
>
> $ LC_ALL=C grep 'foo' $(seq 1000000)
> Segmentation fault (core dumped)
>

Seems that all commands linked with cygwin1.dll will fault if you pass them
a long enough arglist.

For me, /bin/true faults on {1..258231} but not {1..258230}.

> $ /bin/true {1..258230}
>


> $ /bin/true {1..258231}
> Segmentation fault (core dumped)


strace, which is not linked with cygwin1.dll, exits cleanly.

> $ /bin/strace {1..300000}
> -bash: /bin/strace: Argument list too long


See this page [1] on maximum argument lengths.

It would be nice to document this limit, whatever it is.

It would also be nice to return an error to the shell on too-long arglist.

$ cat true.exe.stackdump
>
Exception: STATUS_STACK_OVERFLOW at rip=7FFD41BD4646
> rax=0000000000051F10 rbx=0000000800009991 rcx=00000007FFE03C50
> rdx=00007FFD41BF58A0 rsi=0000000000000000 rdi=0000000000000000
> r8 =FEFEFEFEFEFEFF42 r9 =00007FFD41BF5820 r10=FEFEFEFEFEFEFEFF
> r11=FEFEFEFEFEFEFEFF r12=000000080000998D r13=00000007FFFFCDF0
> r14=0000000000000000 r15=0000000000000000
> rbp=00000007FFFFCD30 rsp=00000007FFFFCC38
> program=C:\cygwin64\bin\true.exe, pid 44496, thread
> cs=0033 ds=002B es=002B fs=0053 gs=002B ss=002B
> Stack trace:
> Frame         Function      Args
> 0007FFFFCD30  7FFD41BD4646 (7FFD41A08035, 7FFD41A06F80, 000000000000,
> 000000000000) cygwin1.dll+0x1D4646
> 0007FFFFCD30  000000249F10 (7FFD41A06F80, 000000000000, 000000000000,
> 000000000000)
> 0007FFFFCD30  7FFD41BF5800 (000000000000, 000000000000, 000000000000,
> 000000000000) cygwin1.dll+0x1F5800
> 0007FFFFCD30  7FFD41A08035 (000000000000, 000000000000, 000000000000,
> 000000000000) cygwin1.dll+0x8035
> 0007FFFFFFF0  7FFD41A05C86 (000000000000, 000000000000, 000000000000,
> 000000000000) cygwin1.dll+0x5C86
> 0007FFFFFFF0  7FFD41A05D34 (000000000000, 000000000000, 000000000000,
> 000000000000) cygwin1.dll+0x5D34
> End of stack trace
> Loaded modules:
> 000100400000 true.exe
> 7FFD52F30000 ntdll.dll
> 7FFD52250000 KERNEL32.DLL
> 7FFD50940000 KERNELBASE.dll
> 0005EE2D0000 cygintl-8.dll
> 7FFD41A00000 cygwin1.dll
> 0003F9F70000 cygiconv-2.dll
> 7FFD51C70000 advapi32.dll
> 7FFD51FD0000 msvcrt.dll
> 7FFD525A0000 sechost.dll
> 7FFD52650000 RPCRT4.dll
> 7FFD4F3A0000 CRYPTBASE.DLL
> 7FFD50380000 bcryptPrimitives.dll



> $ uname -a
> CYGWIN_NT-10.0-22621 nzxt 3.4.8-1.x86_64 2023-08-17 17:02 UTC x86_64 Cygwin


Thanks,
Jeremy Hetzler

[1] https://www.in-ulm.de/~mascheck/various/argmax/

Re: Fwd: gawk core dumped on too many input values

[Takashi Yano]

On Sun, 27 Aug 2023 13:24:55 -0500
Ed Morton wrote:
> This (original email below) turned out to be a general cygwin issue, not 
> a gawk issue:
> 
> $ LC_ALL=C sed 's/x/y/' $(seq 1000000)
> Segmentation fault (core dumped)
> 
> $ LC_ALL=C grep 'foo' $(seq 1000000)
> Segmentation fault (core dumped)
> 
> Regards,
> 
>      Ed.
> 
> -------- Forwarded Message --------
> Subject: 	gawk core dumped on too many input values
> Date: 	Sun, 27 Aug 2023 08:09:54 -0500
> From: 	Ed Morton <mortoneccc@comcast.net>
> To: 	bug-gawk@gnu.org <bug-gawk@gnu.org>
> 
> 
> 
> Configuration Information [Automatically generated, do not change]:
> Machine: x86_64
> OS: cygwin
> Compiler: gcc
> Compilation CFLAGS: -ggdb -O2 -pipe -Wall -Werror=format-security 
> -Wp,-D_FORTIFY_SOURCE=2 -fstack-protector-strong 
> --param=ssp-buffer-size=4 
> -fdebug-prefix-map=/cygdrive/d/a/scallywag/gawk/gawk-5.2.2-1.x86_64/build=/usr/src/debug/gawk-5.2.2-1 
> -fdebug-prefix-map=/cygdrive/d/a/scallywag/gawk/gawk-5.2.2-1.x86_64/src/gawk-5.2.2=/usr/src/debug/gawk-5.2.2-1 
> -DNDEBUG
> uname output: CYGWIN_NT-10.0-22621 TournaMart_2023 3.4.8-1.x86_64 
> 2023-08-17 17:02 UTC x86_64 Cygwin
> Machine Type: x86_64-pc-cygwin
> 
> Gawk Version: 5.2.2
> 
> Attestation 1:
>          I have read 
> https://www.gnu.org/software/gawk/manual/html_node/Bugs.html.
>          Yes
> 
> Attestation 2:
>          I have not modified the sources before building gawk.
>          True
> 
> Description:
>          I was trying to test something related to ARG_MAX when I ran the
>          awk script below and it core dumped instead of reporting an error
>          and exiting gracefully. In case it's useful getconf ARG_MAX outputs
>          32000.
> 
> Repeat-By:
>          $ LC_ALL=C awk 'BEGIN{print ARGC}' $(seq 1000000)
>          Segmentation fault (core dumped)

Thanks for the report.
I will submit a patch for this issue shortly.

-- 
Takashi Yano <takashi.yano@nifty.ne.jp>

Re: gawk core dumped on too many input values

[Joshuah Hurst]

On Mon, Aug 28, 2023 at 1:08 AM Jeremy Hetzler via Cygwin
<cygwin@cygwin.com> wrote:
>
> On Sun, Aug 27, 2023 at 2:25 PM Ed Morton via Cygwin <cygwin@cygwin.com>
> wrote:
> >
> > This (original email below) turned out to be a general cygwin issue, not
> > a gawk issue:
> >
> > $ LC_ALL=C sed 's/x/y/' $(seq 1000000)
> > Segmentation fault (core dumped)
> >
> > $ LC_ALL=C grep 'foo' $(seq 1000000)
> > Segmentation fault (core dumped)
> >
>
> Seems that all commands linked with cygwin1.dll will fault if you pass them
> a long enough arglist.
>
> For me, /bin/true faults on {1..258231} but not {1..258230}.
>
> > $ /bin/true {1..258230}
> >
>
>
> > $ /bin/true {1..258231}
> > Segmentation fault (core dumped)
>
>
> strace, which is not linked with cygwin1.dll, exits cleanly.
>
> > $ /bin/strace {1..300000}
> > -bash: /bin/strace: Argument list too long
>
>
> See this page [1] on maximum argument lengths.
>
> It would be nice to document this limit, whatever it is.

Is this limit?

$ getconf -a | grep -E 'ARG_MAX'
_POSIX_ARG_MAX                      4096
ARG_MAX                             32000

>
> It would also be nice to return an error to the shell on too-long arglist.

+1
-- 
Josh

Re: gawk core dumped on too many input values

[Brian Inglis]

On 2023-08-28 05:47, Joshuah Hurst via Cygwin wrote:
> On Mon, Aug 28, 2023 at 1:08 AM Jeremy Hetzler via Cygwin
> <cygwin@cygwin.com> wrote:
>>
>> On Sun, Aug 27, 2023 at 2:25 PM Ed Morton via Cygwin <cygwin@cygwin.com>
>> wrote:
>>>
>>> This (original email below) turned out to be a general cygwin issue, not
>>> a gawk issue:
>>>
>>> $ LC_ALL=C sed 's/x/y/' $(seq 1000000)
>>> Segmentation fault (core dumped)
>>>
>>> $ LC_ALL=C grep 'foo' $(seq 1000000)
>>> Segmentation fault (core dumped)
>>>
>>
>> Seems that all commands linked with cygwin1.dll will fault if you pass them
>> a long enough arglist.
>>
>> For me, /bin/true faults on {1..258231} but not {1..258230}.
>>
>>> $ /bin/true {1..258230}
>>>
>>
>>
>>> $ /bin/true {1..258231}
>>> Segmentation fault (core dumped)
>>
>>
>> strace, which is not linked with cygwin1.dll, exits cleanly.
>>
>>> $ /bin/strace {1..300000}
>>> -bash: /bin/strace: Argument list too long
>>
>>
>> See this page [1] on maximum argument lengths.
>>
>> It would be nice to document this limit, whatever it is.
> 
> Is this limit?
> 
> $ getconf -a | grep -E 'ARG_MAX'
> _POSIX_ARG_MAX                      4096
> ARG_MAX                             32000

On my system, /bin/true and /bin/echo get to between 250K and 256K before seg 
faulting, requiring between digits + separator + ptr (10+8)*(250-256)KB == 
3750-3840KB;

current Cygwin and `bash` `:` and `echo` internal commands get to between 128M 
and 256M before `bash` fork fails, and Windows gets unhappy with memory full, 
paging high, and page thrashing on both paging devices: calmed eventually with a 
few interrupts then `exec bash`;
requiring between digits + separator + ptr (10+8)*(128-256)MB == 2.25-4.5GB;

This seems to be extremely conservative:

$ xargs -r --show-limits <<< ' '
Your environment variables take up 9246 bytes
POSIX upper limit on argument length (this system): 20706
POSIX smallest allowable upper limit on argument length (all systems): 4096
Maximum length of command we could actually use: 11460
Size of command buffer we are actually using: 20706
Maximum parallelism (--max-procs must be no greater): 2147483647
$ seq $((256*1024)) | xargs | wc -lwcL
     102  262144 1723903   18094

Once a patch is in a test version of Cygwin, I can rebuild and release a test 
version of findutils, which includes xargs, with updated results in the test 
release announcement.

-- 
Take care. Thanks, Brian Inglis              Calgary, Alberta, Canada

La perfection est atteinte                   Perfection is achieved
non pas lorsqu'il n'y a plus rien à ajouter  not when there is no more to add
mais lorsqu'il n'y a plus rien à retirer     but when there is no more to cut
                                 -- Antoine de Saint-Exupéry

Re: gawk core dumped on too many input values

[Corinna Vinschen]

On Aug 28 12:20, Brian Inglis via Cygwin wrote:
> On 2023-08-28 05:47, Joshuah Hurst via Cygwin wrote:
> > On Mon, Aug 28, 2023 at 1:08 AM Jeremy Hetzler via Cygwin
> > <cygwin@cygwin.com> wrote:
> > > 
> > > On Sun, Aug 27, 2023 at 2:25 PM Ed Morton via Cygwin <cygwin@cygwin.com>
> > > wrote:
> > > > 
> > > > This (original email below) turned out to be a general cygwin issue, not
> > > > a gawk issue:
> > > > 
> > > > $ LC_ALL=C sed 's/x/y/' $(seq 1000000)
> > > > Segmentation fault (core dumped)
> > > > 
> > > > $ LC_ALL=C grep 'foo' $(seq 1000000)
> > > > Segmentation fault (core dumped)

This is fixed in current git and can be tested with the next test
release cygwin-3.5.0-0.404.gca2a4ec24362, which is just being built
and uploaded in a few mins.

> > [...]
> > Is this limit?
> > 
> > $ getconf -a | grep -E 'ARG_MAX'
> > _POSIX_ARG_MAX                      4096
> > ARG_MAX                             32000

This isn't the real limit.

ARG_MAX has been chosen at one point to be 32000, because that's a safe
size for the Windows command line length.  Therefore this is a hard limit
if you start non-Cygwin executables.  Cygwin executables don't have this
limit.  In fact, the limit is defined only by the amount of memory the
parent process has available when creating the argv and environment lists
for the child.

We fixed that in git.  As a result, sysconf(_SC_ARG_MAX) will now return
-1.  I. e., ARG_MAX has an indeterminate limit:

  $ getconf -a | grep -E 'ARG_MAX'
  _POSIX_ARG_MAX                      4096
  ARG_MAX
  $ getconf ARG_MAX
  undefined

However! limits.h still defines ARG_MAX as 32000, and we'll stick to
this, on account it being a safe value.

This has a precedent on Linux, where getconf returns something big, but
ARG_MAX is still 131072:

  $ grep ARG_MAX /usr/include/linux/limits.h
  #define ARG_MAX       131072	/* # bytes of args + environ for exec() */
  $ getconf ARG_MAX
  2097152

The limits.h limit of 131072 is historical (32 pages for argv and envp).
The getconf value is a quarter of the stack which is reserved for argv
and envp.

I hope that explains things sufficiently.

The patches will be backported to 3.4.9.


Thanks,
Corinna