* BASH 4.4 mapfile/readarray/read builtins mis-behaving with pipe @ 2018-07-17 12:57 BloomingAzaleas 2018-07-17 12:57 ` Steven Penny 2018-07-18 11:17 ` BASH 4.4 mapfile/readarray/read builtins mis-behaving with pipe [edit] documentation bug BloomingAzaleas 0 siblings, 2 replies; 7+ messages in thread From: BloomingAzaleas @ 2018-07-17 12:57 UTC (permalink / raw) To: cygwin Windows 10 Pro 10.0.17134 N/A Build 17134 patched through 15 July 2018 Cygwin 2.10.0(0.325/5/3) Bash 4.4.12(3) Cygwin man pages show: mapfile [-d delim] [-n count] [-O origin] [-s count] [-t] [-u fd] [-C callback] [-c quantum] [array] readarray [-d delim] [-n count] [-O origin] [-s count] [-t] [-u fd] [-C callback] [-c quantum] [array] Read lines from the standard input into the indexed array variable array, or ... read [-ers] [-a aname] [-d delim] [-i text] [-n nchars] [-N nchars] [-p prompt] [-t timeout] [-u fd] [name ...] One line is read from the standard input, or ... So the expectation is that all bash stdin re-directions such as pipes and left chevs should work. For mapfile/readarry, the output array variable MAPFILE is described as being created if not array_var argument is given to mapfile. A) Searched cygwin email list with terms 'mapfile', 'readarray' and 'read builtin'. No obvious hits of recent vintage in summary result list. B) Confirm, not using mapfile/readarry/read, that bash stdin redirs work as expected. echo multi-line_arg | /bin/cat /bin/cat < some_unix_fmt_file /bin/cat <<END some lines END all work as expected. C) Test mapfile/readarray/read in various stdin redir situations. Basic harnesses are : mapfile/readarry HARNESS: stdin or fd 0 feed to mapfile then : "${MAPFILE[@]:?MAPFILE null or unset}" if get past the ${:?} test then dump MAPFILE with a simple /for /loop as: for ent in "${MAPFILE[@]}" ; do echo ="${ent}"= ; done read HARNESS: stdin feed to 'read foo' then : "${foo:?foo null or unset}" if get past the ${:?} test then dump variable foo with echo ="${foo}"= CASES: echo multi-line_arg | mapfile cat multi-line_unix_fmt_file | mapfile RESULT: Fail with MAPFILE null or unset If 'for' loop forced, "${MAPFILE[@]}" expands to zero words. CASE: echo multi-line_arg | read foo cat multi-line_unix_fmt_file | read foo RESULT: Fail with variable foo null or unset CASES: left chev re-directions mapfile < some_unix_fmt_file mapfile <<END multi-line_here_doc read < some_unix_fmt_file read <<END multi-line_here_doc RESULT: Work as expected CASES: mapfile or read in a sub-shell echo multi-line_arg | ( mapfile ; stuff ) echo multi-line_arg | ( read foo; stuff ) RESULT: Work as expected. However, since mapfile/read result vars are not accessible in parent shell context, can force a notable re-structuring of a script to use sub-shells. CONCLUSIONS: For bash builtins mapfile/readarray and read, stdin pipe re-direction directly to the builtin fails as if the builtin read()/fread() sees a silent pre-mature EOF on the pipe. In the sub-shell tests, the pipe re-direction happens for the sub-shell as a whole before commands internal to the sub-shell get processed for individual re-dirs, so no re-dir processing directly for the builtin. -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: BASH 4.4 mapfile/readarray/read builtins mis-behaving with pipe 2018-07-17 12:57 BASH 4.4 mapfile/readarray/read builtins mis-behaving with pipe BloomingAzaleas @ 2018-07-17 12:57 ` Steven Penny 2018-07-18 11:17 ` BASH 4.4 mapfile/readarray/read builtins mis-behaving with pipe [edit] documentation bug BloomingAzaleas 1 sibling, 0 replies; 7+ messages in thread From: Steven Penny @ 2018-07-17 12:57 UTC (permalink / raw) To: cygwin On Sun, 15 Jul 2018 19:01:21, BloomingAzaleas wrote: > CASE: > echo multi-line_arg | read foo > cat multi-line_unix_fmt_file | read foo > =C2=A0RESULT: > Fail with variable foo null or unset > > CASES: left chev re-directions > mapfile < some_unix_fmt_file > mapfile <<END multi-line_here_doc > read < some_unix_fmt_file > read <<END multi-line_here_doc > RESULT: > Work as expected > > CASES: mapfile or read in a sub-shell > echo multi-line_arg | ( mapfile ; stuff ) > echo multi-line_arg | ( read foo; stuff ) > RESULT: > Work as expected.=C2=A0 However, since mapfile/read result vars are=20 > not accessible in parent shell context, can force a notable=20 > re-structuring of a script to use sub-shells. no mis-behaving: this is intended behavior - you yourself have given workarounds: either redirect output to a file that can be later read, or pipe to command grouping ala {} or () and read stdin from inside the subshell im not sure what you are asking here - it seems you have a grasp already of the "problem" and "solution" - so nothing more needs to be said really, as nothing needs to be fixed - another option is "lastpipe" http://mywiki.wooledge.org/BashFAQ/024 -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: BASH 4.4 mapfile/readarray/read builtins mis-behaving with pipe [edit] documentation bug 2018-07-17 12:57 BASH 4.4 mapfile/readarray/read builtins mis-behaving with pipe BloomingAzaleas 2018-07-17 12:57 ` Steven Penny @ 2018-07-18 11:17 ` BloomingAzaleas 2018-07-21 1:19 ` Eric Blake 1 sibling, 1 reply; 7+ messages in thread From: BloomingAzaleas @ 2018-07-18 11:17 UTC (permalink / raw) To: cygwin Reply to Steven Penny <svnpenn at gmail dot com>: no mis-behaving: this is intended behavior - you yourself have given workarounds: either redirect output to a file that can be later read, or pipe to command grouping ala {} or () and read stdin from inside the subshell im not sure what you are asking here - it seems you have a grasp already of the "problem" and "solution" - so nothing more needs to be said really, as nothing needs to be fixed - another option is "lastpipe" If the reported behavior is the intended behavior, then this complaint is now a documentation complaint. Neither the Cygwin man pages (quoted in the original report) nor the upstream GNU documentation indicate any exception to stdin redirect with pipe in the doc paragraphs for the subject builtins. What confuses the situation is that some stdin re-directions work for these builtins in a parent shell context (left-chevs) but not others (pipes) UNLESS performed strictly at mid-night under a full moon using obscure incantation hidden in pieces throughout a 178-page pdf. In the GNU doc, the sub-shell for builtins in a pipeline rule involved is obscured by burial in paragraph six of section 3.7.3 Command Execution Environment. I suggest the following adjustment to the man pages inserting a parenthetical cue regards behavior in pipes: mapfile [-d delim] [-n count] [-O origin] [-s count] [-t] [-u fd] [-C callback] [-c quantum] [array] readarray [-d delim] [-n count] [-O origin] [-s count] [-t] [-u fd] [-C callback] [-c quantum] [array] Read lines from the standard input (see pipeline restriction) into the indexed array variable array, or ... read [-ers] [-a aname] [-d delim] [-i text] [-n nchars] [-N nchars] [-p prompt] [-t timeout] [-u fd] [name ...] One line is read from the standard input (see pipeline restriction), or ... At least the reader gets a cue to modify expectations and research the "pipeline restriction" compared to all other commands that work just peachy with stdin pipe re-direction. Regards the shopt lastpipe doc single-sentence paragraph in GNU section 4.3.2 The Shopt Builtin, it could be improved by adding: "This option will change the parent shell context behavior of shell builtins mapfile, readarray and read when they are used as the last command of a pipeline." As to the left-chev work-arounds, yes they work, but that would require some script re-structuring. In any case, "what you are asking here" is that there appears to be a difference between a plain reading of the doc paragraphs for the builtins and the builtins behavior and so a bug against one or the other. If this were a unix/linux kernel environment, the triage emphasis would be documentation wording rather than implementation bug. Given the Cygwin context of necessary hidden emulation chicanery on Windows, triage emphasis is not that obvious. I exerted the effort to construct a reasonably complete report with multiple illustration cases contrasted to the doc to improve the FOSS, be it implementation or documentation. I didn't need too. Regards. On 7/15/2018 7:01 PM, BloomingAzaleas wrote: > Windows 10 Pro 10.0.17134 N/A Build 17134 patched through 15 > July 2018 > Cygwin 2.10.0(0.325/5/3) > Bash 4.4.12(3) > > Cygwin man pages show: > > mapfile [-d delim] [-n count] [-O origin] [-s count] [-t] [-u > fd] [-C callback] [-c quantum] [array] > readarray [-d delim] [-n count] [-O origin] [-s count] [-t] > [-u fd] [-C callback] [-c quantum] [array] > Read lines from the standard input into the indexed array > variable array, or ... > > read [-ers] [-a aname] [-d delim] [-i text] [-n nchars] [-N > nchars] [-p prompt] [-t timeout] [-u fd] [name ...] > One line is read from the standard input, or ... > > So the expectation is that all bash stdin re-directions such as > pipes and left chevs should work. For mapfile/readarry, the > output array variable MAPFILE is described as being created if > not array_var argument is given to mapfile. > > A) Searched cygwin email list with terms 'mapfile', 'readarray' > and 'read builtin'. No obvious hits of recent vintage in > summary result list. > > B) Confirm, not using mapfile/readarry/read, that bash stdin > redirs work as expected. > > echo multi-line_arg | /bin/cat > > /bin/cat < some_unix_fmt_file > > /bin/cat <<END > some > lines > END > > all work as expected. > > C) Test mapfile/readarray/read in various stdin redir > situations. Basic harnesses are : > > mapfile/readarry HARNESS: > stdin or fd 0 feed to mapfile > then > : "${MAPFILE[@]:?MAPFILE null or unset}" > if get past the ${:?} test then dump MAPFILE with a simple /for > /loop as: for ent in "${MAPFILE[@]}" ; do echo ="${ent}"= ; done > > read HARNESS: > stdin feed to 'read foo' > then > : "${foo:?foo null or unset}" > if get past the ${:?} test then dump variable foo with echo > ="${foo}"= > > CASES: > echo multi-line_arg | mapfile > cat multi-line_unix_fmt_file | mapfile > RESULT: > Fail with MAPFILE null or unset > If 'for' loop forced, "${MAPFILE[@]}" expands to zero words. > > CASE: > echo multi-line_arg | read foo > cat multi-line_unix_fmt_file | read foo > RESULT: > Fail with variable foo null or unset > > CASES: left chev re-directions > mapfile < some_unix_fmt_file > mapfile <<END multi-line_here_doc > read < some_unix_fmt_file > read <<END multi-line_here_doc > RESULT: > Work as expected > > CASES: mapfile or read in a sub-shell > echo multi-line_arg | ( mapfile ; stuff ) > echo multi-line_arg | ( read foo; stuff ) > RESULT: > Work as expected. However, since mapfile/read result vars are > not accessible in parent shell context, can force a notable > re-structuring of a script to use sub-shells. > > CONCLUSIONS: > For bash builtins mapfile/readarray and read, stdin pipe > re-direction directly to the builtin fails as if the builtin > read()/fread() sees a silent pre-mature EOF on the pipe. In > the sub-shell tests, the pipe re-direction happens for the > sub-shell as a whole before commands internal to the sub-shell > get processed for individual re-dirs, so no re-dir processing > directly for the builtin. > -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: BASH 4.4 mapfile/readarray/read builtins mis-behaving with pipe [edit] documentation bug 2018-07-18 11:17 ` BASH 4.4 mapfile/readarray/read builtins mis-behaving with pipe [edit] documentation bug BloomingAzaleas @ 2018-07-21 1:19 ` Eric Blake 2018-07-22 5:42 ` BloomingAzaleas 0 siblings, 1 reply; 7+ messages in thread From: Eric Blake @ 2018-07-21 1:19 UTC (permalink / raw) To: cygwin, rdbingham On 07/17/2018 08:52 PM, BloomingAzaleas wrote: > Reply to Steven Penny <svnpenn at gmail dot com>: > > Â Â no mis-behaving: this is intended behavior - you yourself have given > Â Â workarounds: either redirect output to a file that can be later > read, or pipe to > Â Â command grouping ala {} or () and read stdin from inside the subshell > > I suggest the following adjustment to the man pages inserting a > parenthetical cue regards behavior in pipes: Is the behavior you are complaining about unique to Cygwin, or can it be reproduced on a GNU/Linux box? If the latter, then an upstream bug report is better than asking for a cygwin-specific patch. [Hint - as the maintainer of the cygwin bash port, I don't recall adding any cygwin-specific tweaks for mapfile - and a quick test on Fedora shows the same behaviors] -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: BASH 4.4 mapfile/readarray/read builtins mis-behaving with pipe [edit] documentation bug 2018-07-21 1:19 ` Eric Blake @ 2018-07-22 5:42 ` BloomingAzaleas 2018-07-23 2:15 ` Steven Penny 0 siblings, 1 reply; 7+ messages in thread From: BloomingAzaleas @ 2018-07-22 5:42 UTC (permalink / raw) To: cygwin Reply to Eric Blake, At this time, I do not have a Linux image available to me. If you saw the same behavior on Fedora, then I suggest the behavior originates upstream at or close the the GNU source-code level. Mr. Penny's response asserted the observed behavior "is intended behavior", in which case there should exist a GNU specification document describing the intended pipe STDIN re-direction restrictions for 'mapfile', 'read' and possibly other "builtins". Lacking such a reference nothing can be said about intent. Implementation-as-intent implies errors and conceptual and behavioral inconsistencies in the implementation were intended. I decline to think that of the distributed BASH maintenance team. He provided a reference to http://mywiki.wooledge.org/BashFAQ/024, where, if you read the page, the behavior inconsistency I reported is shown under the heading of "More broken stuff:". I take that "More broken stuff:" opinion to mean yet others read the same surface discrepancy between doc and behavior as I did. I believe what we have at this point is: A) GNU doc lacking nuance, attention to consistent terminology, and helpful rule-statement to code example/counter-example illustrations adjacent to rule statements. For a doc counter-example, the Open Group doc at http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html does make an effort to distinguish behaviors between "2.14 Special Built-In Utilities" and "2.9.1 Simple Commands" whereas the GNU doc indiscriminately mixes the words "commands" (section 3.2) and "builtin" and "command" and the phrase "builtin command" (section 4) as if their behaviors are identical under the section 4 title of "Shell Builtin Commands" (https://www.gnu.org/software/bash/manual/html_node/Shell-Builtin-Commands.html#Shell-Builtin-Commands). The GNU doc for shopt lastpipeis fairly opaque unless you have deep knowledge and that knowledge is cued when considering the possible meanings of "current shell environment" for built-ins (same process) vs. external child-process executables. B) However conceptually inconsistent, an obsessive BASH doc reader could imply the observed bash built-ins behavior by integrating multiple hints and rule statements scattered across the GNU doc and then, crucially, doubting the plain meaning of the unqualified doc statements of "Read lines from the standard input..." . Thank you for your response. Regards, UN*X Since '85 On 7/20/2018 12:08 PM, Eric Blake wrote: > On 07/17/2018 08:52 PM, BloomingAzaleas wrote: >> Reply to Steven Penny <svnpenn at gmail dot com>: >> >> no mis-behaving: this is intended behavior - you yourself >> have given >> workarounds: either redirect output to a file that can be >> later read, or pipe to >> command grouping ala {} or () and read stdin from inside >> the subshell >> > >> I suggest the following adjustment to the man pages inserting >> a parenthetical cue regards behavior in pipes: > > Is the behavior you are complaining about unique to Cygwin, or > can it be reproduced on a GNU/Linux box? If the latter, then > an upstream bug report is better than asking for a > cygwin-specific patch. [Hint - as the maintainer of the cygwin > bash port, I don't recall adding any cygwin-specific tweaks for > mapfile - and a quick test on Fedora shows the same behaviors] > -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: BASH 4.4 mapfile/readarray/read builtins mis-behaving with pipe [edit] documentation bug 2018-07-22 5:42 ` BloomingAzaleas @ 2018-07-23 2:15 ` Steven Penny 2018-07-23 6:32 ` Brian Inglis 0 siblings, 1 reply; 7+ messages in thread From: Steven Penny @ 2018-07-23 2:15 UTC (permalink / raw) To: cygwin On Sat, 21 Jul 2018 19:36:38, BloomingAzaleas wrote: > At this time, I do not have a Linux image available to me. its not that hard to get: 1. http://virtualbox.org 2. http://cdimage.debian.org/debian-cd/current-live/amd64/iso-hybrid > you saw the same behavior on Fedora, then I suggest the behavior > originates upstream at or close the the GNU source-code level. the behavior is documented by POSIX. Command grouping example: $ bash -c 'q=2; (q=3); echo "$q"' 2 and reference: > Execute compound-list in a subshell environment [...] Variable assignments and > built-in commands that affect the environment shall not remain in effect after > the list finishes. http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_09_04_01 pipe example: $ bash -c 'q=2; echo 3 | read q; echo "$q"' 2 and reference: > Changes made to the subshell environment shall not affect the shell > environment. [...] Additionally, each command of a multi-command pipeline is > in a subshell environment http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_12 > For a doc counter-example, the Open Group doc at > http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html its funny that you list this page, as thats exactly where i got my references. thanks. > Thank you for your response. > > Regards, > > UN*X Since '85 finally i would say to stop top posting. for someone who is self declared unix for 30 years, it seems odd that you cant even run fedora, or adhere to defacto mailing list etiquette. Cheers. -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: BASH 4.4 mapfile/readarray/read builtins mis-behaving with pipe [edit] documentation bug 2018-07-23 2:15 ` Steven Penny @ 2018-07-23 6:32 ` Brian Inglis 0 siblings, 0 replies; 7+ messages in thread From: Brian Inglis @ 2018-07-23 6:32 UTC (permalink / raw) To: cygwin On 2018-07-21 18:37, Steven Penny wrote: > On Sat, 21 Jul 2018 19:36:38, BloomingAzaleas wrote: >> At this time, I do not have a Linux image available to me. > its not that hard to get: > 1. http://virtualbox.org > 2. http://cdimage.debian.org/debian-cd/current-live/amd64/iso-hybrid Even easier to install one (or more) of the WSL distros from the MS Store after enabling developer mode in Settings. -- Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple ^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2018-07-22 5:42 UTC | newest] Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2018-07-17 12:57 BASH 4.4 mapfile/readarray/read builtins mis-behaving with pipe BloomingAzaleas 2018-07-17 12:57 ` Steven Penny 2018-07-18 11:17 ` BASH 4.4 mapfile/readarray/read builtins mis-behaving with pipe [edit] documentation bug BloomingAzaleas 2018-07-21 1:19 ` Eric Blake 2018-07-22 5:42 ` BloomingAzaleas 2018-07-23 2:15 ` Steven Penny 2018-07-23 6:32 ` Brian Inglis
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).