[Bug localedata/15831] New: Bad $expect passed from tst-fmon.sh to tst-fmon

[Bug localedata/15831] New: Bad $expect passed from tst-fmon.sh to tst-fmon

[slicer_ghent at hushmail dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=15831

            Bug ID: 15831
           Summary: Bad $expect passed from tst-fmon.sh to tst-fmon
           Product: glibc
           Version: 2.17
            Status: NEW
          Keywords: glibc_2.17
          Severity: normal
          Priority: P2
         Component: localedata
          Assignee: unassigned at sourceware dot org
          Reporter: slicer_ghent at hushmail dot com
                CC: libc-locales at sourceware dot org
              Host: x86_64-unknown-linux-gnu
             Build: x86_64-unknown-linux-gnu

uname -m = x86_64
uname -r = 3.0.0-12-generic
uname -s = Linux
uname -v = #20-Ubuntu SMP Fri Oct 7 14:56:25 UTC 2011

Working from SRCDIR/glibc-build/, as per INSTALL instructions:

$ ../glibc-2.17/configure --prefix=/usr --enable-add-ons --enable-shared
--enable-kernel=3.0.0 --with-headers=/usr/src/linux-headers-3.0.0-32/include

Completes successfully. Next:

$ make -j

Completes successfully. Next:

$ make check
[...]
/bin/sh tst-fmon.sh SRCDIR/glibc-build/ ' 
SRCDIR/glibc-build/elf/ld-linux-x86-64.so.2 --library-path
SRCDIR/glibc-build:SRCDIR/glibc-build/math:SRCDIR/glibc-build/elf:SRCDIR/glibc-build/dlfcn:SRCDIR/glibc-build/nss:SRCDIR/glibc-build/nis:SRCDIR/glibc-build/rt:SRCDIR/glibc-build/resolv:SRCDIR/glibc-build/crypt:SRCDIR/glibc-build/nptl'
tst-fmon.data \
      > SRCDIR/glibc-build/localedata/tst-fmon.out
make[2]: *** [SRCDIR/glibc-build/localedata/tst-fmon.out] Error 1
make[2]: Leaving directory `SRCDIR/glibc-2.17/localedata'
make[1]: *** [localedata/tests] Error 2
make[1]: Leaving directory `SRCDIR/glibc-2.17'
make: *** [check] Error 2

Contents of tst-fmon.out:

Locale: "de_DE.ISO-8859-1" Format: "%n" Value: "1.23" Received: "1,23 EUR"
Expected: "  1,23 EUR" => false
Locale: "de_DE.ISO-8859-1" Format: "%n" Value: "-1.23" Received: "-1,23 EUR"
Expected: "        -1,23 EUR" => false
Locale: "de_DE.ISO-8859-1" Format: "%n" Value: "1234.56" Received: "1.234,56
EUR" Expected: "   1.234,56 EUR" => false
Locale: "de_DE.ISO-8859-1" Format: "%12n" Value: "123.45" Received: "  123,45
EUR" Expected: "    123,45 EUR" => false
Locale: "de_DE.ISO-8859-1" Format: "%12n" Value: "-123.45" Received: " -123,45
EUR" Expected: "  -123,45 EUR" => false
Locale: "de_DE.ISO-8859-1" Format: "%^n" Value: "1234.56" Received: "1234,56
EUR" Expected: "   1234,56 EUR" => false
Locale: "de_DE.ISO-8859-1" Format: "%+n" Value: "1234.56" Received: "1.234,56
EUR" Expected: "  1.234,56 EUR" => false
Locale: "de_DE.ISO-8859-1" Format: "%(n" Value: "1234.56" Received: "1.234,56
EUR" Expected: "  1.234,56 EUR" => false
Locale: "de_DE.ISO-8859-1" Format: "%^n" Value: "1234.56" Received: "1234,56
EUR" Expected: "   1234,56 EUR" => false
Locale: "de_DE.ISO-8859-1" Format: "%i" Value: "1.23" Received: "1,23 EUR"
Expected: "  1,23 EUR" => false
Locale: "de_DE.ISO-8859-1" Format: "%i" Value: "-1.23" Received: "-1,23 EUR"
Expected: "        -1,23 EUR" => false
Locale: "de_DE.ISO-8859-1" Format: "%i" Value: "1234.56" Received: "1.234,56
EUR" Expected: "   1.234,56 EUR" => false
Locale: "de_DE.ISO-8859-1" Format: "%^i" Value: "1234.56" Received: "1234,56
EUR" Expected: "   1234,56 EUR" => false
Locale: "de_DE.ISO-8859-1" Format: "%+i" Value: "1234.56" Received: "1.234,56
EUR" Expected: "  1.234,56 EUR" => false
Locale: "de_DE.ISO-8859-1" Format: "%(i" Value: "1234.56" Received: "1.234,56
EUR" Expected: "  1.234,56 EUR" => false
Locale: "de_DE.ISO-8859-1" Format: "%^i" Value: "1234.56" Received: "1234,56
EUR" Expected: "   1234,56 EUR" => false
Locale: "de_DE.ISO-8859-1" Format: "%#5n" Value: "123.45" Received: "    123,45
EUR" Expected: "            123,45 EUR" => false
Locale: "de_DE.ISO-8859-1" Format: "%#5n" Value: "-123.45" Received: "-  
123,45 EUR" Expected: "       -   123,45 EUR" => false
Locale: "de_DE.ISO-8859-1" Format: "%=*#5n" Value: "123.45" Received: "
***123,45 EUR" Expected: "       ***123,45 EUR" => false
Locale: "de_DE.ISO-8859-1" Format: "%=*#5n" Value: "-123.45" Received:
"-***123,45 EUR" Expected: "     -***123,45 EUR" => false
Locale: "de_DE.ISO-8859-1" Format: "%=0#5n" Value: "123.45" Received: "
000123,45 EUR" Expected: "       000123,45 EUR" => false
Locale: "de_DE.ISO-8859-1" Format: "%=0#5n" Value: "-123.45" Received:
"-000123,45 EUR" Expected: "     -000123,45 EUR" => false
Locale: "de_DE.ISO-8859-1" Format: "%^#5n" Value: "123.45" Received: "   123,45
EUR" Expected: "           123,45 EUR" => false
Locale: "de_DE.ISO-8859-1" Format: "%^#5n" Value: "-123.45" Received: "- 
123,45 EUR" Expected: "       -  123,45 EUR" => false
Locale: "de_DE.ISO-8859-1" Format: "%^#5.0n" Value: "123.45" Received: "   123
EUR" Expected: "    123 EUR" => false
Locale: "de_DE.ISO-8859-1" Format: "%^#5.0n" Value: "-123.45" Received: "-  123
EUR" Expected: "        -  123 EUR" => false
Locale: "de_DE.ISO-8859-1" Format: "%^#5.4n" Value: "123.45" Received: "  
123,4500 EUR" Expected: "       123,4500 EUR" => false
Locale: "de_DE.ISO-8859-1" Format: "%^#5.4n" Value: "-123.45" Received: "- 
123,4500 EUR" Expected: "   -  123,4500 EUR" => false
Locale: "de_DE.ISO-8859-1" Format: "%(#5n" Value: "123.45" Received: "   
123,45 EUR" Expected: "           123,45 EUR" => false
Locale: "de_DE.ISO-8859-1" Format: "%(#5n" Value: "-123.45" Received: "(  
123,45 EUR)" Expected: "     (   123,45 EUR)" => false
Locale: "de_DE.ISO-8859-1" Format: "%!(#5n" Value: "123.45" Received: "   
123,45" Expected: "      123,45" => false
Locale: "de_DE.ISO-8859-1" Format: "%!(#5n" Value: "-123.45" Received: "(  
123,45)" Expected: "        (   123,45)" => false
Locale: "en_US.ISO-8859-1" Format: "%n" Value: "123.45" Received: "$123.45"
Expected: " $123.45" => false
Locale: "en_US.ISO-8859-1" Format: "%n" Value: "-123.45" Received: "-$123.45"
Expected: "       -$123.45" => false
Locale: "en_US.ISO-8859-1" Format: "%i" Value: "123.45" Received: "USD 123.45"
Expected: "      USD 123.45" => false
Locale: "en_US.ISO-8859-1" Format: "%i" Value: "-123.45" Received: "-USD
123.45" Expected: "    -USD 123.45" => false
Locale: "en_US.ISO-8859-1" Format: "%11n" Value: "123.45" Received: "   
$123.45" Expected: "       $123.45" => false
Locale: "en_US.ISO-8859-1" Format: "%11n" Value: "-123.45" Received: "  
-$123.45" Expected: "     -$123.45" => false
Locale: "en_US.ISO-8859-1" Format: "%=*#5n" Value: "123.45" Received: "
$***123.45" Expected: "  $***123.45" => false
Locale: "en_US.ISO-8859-1" Format: "%=*#5n" Value: "-123.45" Received:
"-$***123.45" Expected: "        -$***123.45" => false
Locale: "en_US.ISO-8859-1" Format: "%=0#5n" Value: "123.45" Received: "
$000123.45" Expected: "  $000123.45" => false
Locale: "en_US.ISO-8859-1" Format: "%=0#5n" Value: "-123.45" Received:
"-$000123.45" Expected: "        -$000123.45" => false
Locale: "en_US.ISO-8859-1" Format: "%^#5n" Value: "123.45" Received: " $ 
123.45" Expected: "    $  123.45" => false
Locale: "en_US.ISO-8859-1" Format: "%^#5n" Value: "-123.45" Received: "-$ 
123.45" Expected: "  -$  123.45" => false
Locale: "en_US.ISO-8859-1" Format: "%^#5.0n" Value: "123.45" Received: " $ 
123" Expected: "     $  123" => false
Locale: "en_US.ISO-8859-1" Format: "%^#5.0n" Value: "-123.45" Received: "-$ 
123" Expected: "   -$  123" => false
Locale: "en_US.ISO-8859-1" Format: "%^#5.4n" Value: "123.45" Received: " $ 
123.4500" Expected: "        $  123.4500" => false
Locale: "en_US.ISO-8859-1" Format: "%^#5.4n" Value: "-123.45" Received: "-$ 
123.4500" Expected: "      -$  123.4500" => false
Locale: "en_US.ISO-8859-1" Format: "%(#5n" Value: "123.45" Received: " $  
123.45" Expected: "   $   123.45" => false
Locale: "en_US.ISO-8859-1" Format: "%(#5n" Value: "-123.45" Received: "($  
123.45)" Expected: "        ($   123.45)" => false
Locale: "en_US.ISO-8859-1" Format: "%!(#5n" Value: "123.45" Received: "   
123.45" Expected: "      123.45" => false
Locale: "en_US.ISO-8859-1" Format: "%!(#5n" Value: "-123.45" Received: "(  
123.45)" Expected: "        (   123.45)" => false
Locale: "en_US.ISO-8859-1" Format: "%#5n" Value: "123.45" Received: " $  
123.45" Expected: "    $   123.45" => false
Locale: "en_US.ISO-8859-1" Format: "%#5n" Value: "-123.45" Received: "-$  
123.45" Expected: "  -$   123.45" => false

Began inspecting tst-fmon.sh since it was the script called in the last command
reported. Eventually worked through tst-fmon.c and then strfmon.c. Feeling in
over my head, I found documentation for strfmon.c at
http://www.gnu.org/software/libc/manual/html_mono/libc.html in Section 7.7.
Returning to tst-fmon.out, I realized the "Received" values were correct, as
per the documentation I had just read, and the "Expected" values were not.

I now opened tst-fmon.data for inspection. What is weird to me is that extra
white space appears to be added to the left of the expectation, but this is not
consistent because, for instance, the line:

de_DE.ISO-8859-1        %12n    3456.781        3.456,78 EUR

passes the test, as it should if all intervening whitespace were removed, but
the line:

de_DE.ISO-8859-1        %12n    -123.45          -123,45 EUR

fails because an extra space has been added to the left side giving the
expectation a total width of 13 characters, rather than 12, as seen in the
format specifier.

Why the extra space would be added to the second example and not the first is
not a question I am able to answer. I thought it might have been an issue with
the read builtin. On my system:

$ ll /bin/sh
lrwxrwxrwx 1 root root 4 2011-12-24 01:00 /bin/sh -> dash
$ ll /bin/dash
-rwxr-xr-x 1 root root 109768 2011-05-03 08:04 /bin/dash

So I included a line in tst-fmon.sh to print the values read in from
tst-fmon.data. The results appeared to be correct, so I modified tst-fmon.c to
print out the values it was passed and found that extra whitespace had then
been added.

For example, the following line from tst-fmon.data:

de_DE.ISO-8859-1        %=*#5n  123.45           ***123,45 EUR

is stored as such in tst-fmon.sh:

de_DE.ISO-8859-1
%=*#5n
123.45
 ***123,45 EUR

but appears in tst-fmon as:

de_DE.ISO-8859-1
%=*#5n
123.45
        ***123,45 EUR

Hopefully this is helpful. If you require any more information, let me know.
Also, from the lines I did inspect in tst-fmon.data, there appear to be
redundant tests at lines 59/64 and 68/73 and there is a typo in tst-fmon.c
which states that it expects three parameters and then describes four.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug localedata/15831] Bad $expect passed from tst-fmon.sh to tst-fmon

[schwab@linux-m68k.org]

http://sourceware.org/bugzilla/show_bug.cgi?id=15831

Andreas Schwab <schwab@linux-m68k.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |WAITING

--- Comment #1 from Andreas Schwab <schwab@linux-m68k.org> ---
Try running the test with /bin/bash instead of /bin/sh.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug localedata/15831] Bad $expect passed from tst-fmon.sh to tst-fmon

[slicer_ghent at hushmail dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=15831

--- Comment #2 from slicer_ghent at hushmail dot com ---
(In reply to Andreas Schwab from comment #1)
> Try running the test with /bin/bash instead of /bin/sh.

That was an inspired suggestion, thank you. It worked, but it wasn't
necessarily because of running with /bin/bash instead of /bin/sh, but instead
of /bin/dash.

First, I changed both the first line of tst-fmon.sh from "#! /bin/sh" to
"#!/bin/bash" and the symlink at /bin/sh (originally "-> dash") to point to
"/bin/bash". Removing the file glibc-build/localedata/tst-fmon.out and
re-running "make check" resulted in a successful pass of the fmon test
(glibc-buld/localedata/tst-fmon.out was empty). Other things failed later in
rt/, but that's a story for another day.

Then I changed the symlink /bin/sh to point back to dash but left the header of
tst-fmon.sh as "#!/bin/bash" and re-ran make check. This failed the fmon test
and resulted in similar (possibly identical, didn't bother to diff) output in
tst-fmon.out as in previous failures.

Next, I changed the symlink /bin/sh to point back to /bin/bash and changed the
header in tst-fmon.sh back to "#! /bin/sh", as it was before I modified it,
removed tst-fmon.out and ran "make check" again. These conditions gave another
successful test with failures occurring later in the rt/ tests.

I suppose this isn't necessarily a bug in glibc then because it appears to be a
shell issue. Why dash would read a variable in just fine but pass it wrong
seems odd, but at least you know there is a shell one of your tests doesn't
play well with.

Thank you for your help.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug localedata/15831] Bad $expect passed from tst-fmon.sh to tst-fmon

[carlos at redhat dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=15831

Carlos O'Donell <carlos at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|WAITING                     |RESOLVED
                 CC|                            |carlos at redhat dot com
         Resolution|---                         |INVALID

--- Comment #3 from Carlos O'Donell <carlos at redhat dot com> ---
(In reply to slicer_ghent from comment #2)
> (In reply to Andreas Schwab from comment #1)
> > Try running the test with /bin/bash instead of /bin/sh.
> 
> That was an inspired suggestion, thank you. It worked, but it wasn't
> necessarily because of running with /bin/bash instead of /bin/sh, but
> instead of /bin/dash.
> 
> First, I changed both the first line of tst-fmon.sh from "#! /bin/sh" to
> "#!/bin/bash" and the symlink at /bin/sh (originally "-> dash") to point to
> "/bin/bash". Removing the file glibc-build/localedata/tst-fmon.out and
> re-running "make check" resulted in a successful pass of the fmon test
> (glibc-buld/localedata/tst-fmon.out was empty). Other things failed later in
> rt/, but that's a story for another day.
> 
> Then I changed the symlink /bin/sh to point back to dash but left the header
> of tst-fmon.sh as "#!/bin/bash" and re-ran make check. This failed the fmon
> test and resulted in similar (possibly identical, didn't bother to diff)
> output in tst-fmon.out as in previous failures.
> 
> Next, I changed the symlink /bin/sh to point back to /bin/bash and changed
> the header in tst-fmon.sh back to "#! /bin/sh", as it was before I modified
> it, removed tst-fmon.out and ran "make check" again. These conditions gave
> another successful test with failures occurring later in the rt/ tests.
> 
> I suppose this isn't necessarily a bug in glibc then because it appears to
> be a shell issue. Why dash would read a variable in just fine but pass it
> wrong seems odd, but at least you know there is a shell one of your tests
> doesn't play well with.
> 
> Thank you for your help.

There are patches outstanding to glibc to allow it to build with a POSIX shell
and not require bash-specific extensions.

I suggest looking through back months in libc-alpha, finding the patches that
were posted and adding them to the `Pending Reviews' list so we don't loose
track of them and get around to reviewing them:
http://sourceware.org/glibc/wiki/Pending%20Reviews

Note, you need to be in EditorGroup to edit the wiki:
http://sourceware.org/glibc/wiki/EditorGroup

This would go a long way to fixing this problem.

At present we only support /bin/bash, thus closing as RESOLVED/INVALID.

Feel free to open an enhancement request to support just POSIX shell and we can
use that as a holding place to list all of the patches that have been posted
for this kind of support.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug localedata/15831] Bad $expect passed from tst-fmon.sh to tst-fmon

[joseph at codesourcery dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=15831

--- Comment #4 from joseph at codesourcery dot com <joseph at codesourcery dot com> ---
On Wed, 14 Aug 2013, carlos at redhat dot com wrote:

> There are patches outstanding to glibc to allow it to build with a POSIX shell
> and not require bash-specific extensions.
> 
> I suggest looking through back months in libc-alpha, finding the patches that
> were posted and adding them to the `Pending Reviews' list so we don't loose
> track of them and get around to reviewing them:
> http://sourceware.org/glibc/wiki/Pending%20Reviews
> 
> Note, you need to be in EditorGroup to edit the wiki:
> http://sourceware.org/glibc/wiki/EditorGroup
> 
> This would go a long way to fixing this problem.
> 
> At present we only support /bin/bash, thus closing as RESOLVED/INVALID.

I don't believe your answer justifies closing this bug, or is in any way 
relevant to this bug.  Such patches as there are are about making 
installed bash scripts such as ldd into installed POSIX shell scripts.  
They are not about supporting building or testing glibc without bash 
available for scripts that require bash.  And they are not about cases 
where scripts use /bin/sh but in fact require non-POSIX shell features - 
such cases are all bugs, for which two fixes are possible in each case 
(either explicitly require bash for the script, or make it work with the 
POSIX shell).

This script (tst-fmon.sh) is an sh script.  So, if it doesn't work with a 
POSIX shell, that's a bug, not an enhancement request, and not invalid, 
and should be fixed in one of those two ways.

Now, the analysis in this bug does not appear to disclose whether in fact 
the script is doing anything non-POSIX, or whether the problem comes from 
a dash bug where dash fails to follow POSIX requirements - if the latter, 
the bug would indeed be invalid.  But such an analysis of whether the 
differences between bash and dash reflect a dash bug or the script 
requiring something beyond POSIX is needed before the bug can be 
considered invalid.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug localedata/15831] Bad $expect passed from tst-fmon.sh to tst-fmon

[carlos at redhat dot com]

http://sourceware.org/bugzilla/show_bug.cgi?id=15831

--- Comment #5 from Carlos O'Donell <carlos at redhat dot com> ---
(In reply to joseph@codesourcery.com from comment #4)
> On Wed, 14 Aug 2013, carlos at redhat dot com wrote:
> 
> > There are patches outstanding to glibc to allow it to build with a POSIX shell
> > and not require bash-specific extensions.
> > 
> > I suggest looking through back months in libc-alpha, finding the patches that
> > were posted and adding them to the `Pending Reviews' list so we don't loose
> > track of them and get around to reviewing them:
> > http://sourceware.org/glibc/wiki/Pending%20Reviews
> > 
> > Note, you need to be in EditorGroup to edit the wiki:
> > http://sourceware.org/glibc/wiki/EditorGroup
> > 
> > This would go a long way to fixing this problem.
> > 
> > At present we only support /bin/bash, thus closing as RESOLVED/INVALID.
> 
> I don't believe your answer justifies closing this bug, or is in any way 
> relevant to this bug.  Such patches as there are are about making 
> installed bash scripts such as ldd into installed POSIX shell scripts. 
> They are not about supporting building or testing glibc without bash 
> available for scripts that require bash.  And they are not about cases 
> where scripts use /bin/sh but in fact require non-POSIX shell features - 
> such cases are all bugs, for which two fixes are possible in each case 
> (either explicitly require bash for the script, or make it work with the 
> POSIX shell).

If that's the case then I've misremembered the patches that have been posted.

> This script (tst-fmon.sh) is an sh script.  So, if it doesn't work with a 
> POSIX shell, that's a bug, not an enhancement request, and not invalid, 
> and should be fixed in one of those two ways.

Good point.

> Now, the analysis in this bug does not appear to disclose whether in fact 
> the script is doing anything non-POSIX, or whether the problem comes from 
> a dash bug where dash fails to follow POSIX requirements - if the latter, 
> the bug would indeed be invalid.  But such an analysis of whether the 
> differences between bash and dash reflect a dash bug or the script 
> requiring something beyond POSIX is needed before the bug can be 
> considered invalid.

I have completed a review of tst-fmon.sh and I see nothing that is non-POSIX in
the shell. I've looked for common bash-isms (function, case, numeric loops,
expand sequences, extended glob, select, ==) and fond none. I also ran Debian's
checkbashisms perl script and it found no bash-isms in the script.

Therefore I'm leaving it closed as resolved/invalid and leaving it up to the
submitter to work out the dash bug.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Bug localedata/15831] Bad $expect passed from tst-fmon.sh to tst-fmon

[fweimer at redhat dot com]

https://sourceware.org/bugzilla/show_bug.cgi?id=15831

Florian Weimer <fweimer at redhat dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
              Flags|                            |security-

-- 
You are receiving this mail because:
You are on the CC list for the bug.