[Bug libstdc++/15733] New: getline(wifstream, wstring) seems to be broken with UTF-8

[Bug libstdc++/15733] New: getline(wifstream, wstring) seems to be broken with UTF-8

[ahlgren at lysator dot liu dot se]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 1221 bytes --]

To following code doesn't work with non-ascii characters in UTF-8: 
 
#include <iostream> 
#include <string> 
#include <locale> 
#include <clocale> 
#include <fstream> 
 
using namespace std; 
 
int main() 
{ 
  setlocale(LC_ALL, ""); 
  locale loc(""); 
  wcin.imbue(loc); 
  wcout.imbue(loc); 
 
  wifstream fin("test.txt"); // contains the UTF-8 string "ett smörgåsbord\n" 
 
  wstring s; 
  while (fin >> s) { 
    wcout << s << L" "; 
  } 
 
} 
 
This will print: "ett sm ". 
Now substitute "while (fin >> s)" for "while (wcin >> s)" and do: 
"echo 'ett smörgåsbord' | ./test", this will work as expected, printing: 
"ett smörgåsbord ". The problems seems to be in wifstream (the same problem is 
present in wfstream).

-- 
           Summary: getline(wifstream, wstring) seems to be broken with UTF-
                    8
           Product: gcc
           Version: 3.4.0
            Status: UNCONFIRMED
          Severity: normal
          Priority: P2
         Component: libstdc++
        AssignedTo: unassigned at gcc dot gnu dot org
        ReportedBy: ahlgren at lysator dot liu dot se
                CC: gcc-bugs at gcc dot gnu dot org


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=15733

[Bug libstdc++/15733] getline(wifstream, wstring) seems to be broken with UTF-8

[ahlgren at lysator dot liu dot se]

------- Additional Comments From ahlgren at lysator dot liu dot se  2004-05-30 13:53 -------
Created an attachment (id=6433)
 --> (http://gcc.gnu.org/bugzilla/attachment.cgi?id=6433&action=view)
The file test.txt

You'll need this file to confirm the bug.

-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=15733

[Bug libstdc++/15733] getline(wifstream, wstring) seems to be broken with UTF-8

[pinskia at gcc dot gnu dot org]

------- Additional Comments From pinskia at gcc dot gnu dot org  2004-05-30 13:56 -------
Again what is your env set to and are you sure that your libc is UTF-8 compable?

-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=15733

[Bug libstdc++/15733] getline(wifstream, wstring) seems to be broken with UTF-8

[ahlgren at lysator dot liu dot se]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 2393 bytes --]


------- Additional Comments From ahlgren at lysator dot liu dot se  2004-05-30 13:57 -------
Created an attachment (id=6434)
 --> (http://gcc.gnu.org/bugzilla/attachment.cgi?id=6434&action=view)
the ii file, I don't know what it is :-)

$ g++34 -v --save-temps wstring.cpp
Reading specs from /usr/lib/gcc/i386-redhat-linux/3.4.0/specs
Configured with: ../configure --prefix=/usr --mandir=/usr/share/man
--infodir=/usr/share/info --enable-shared --enable-threads=posix
--disable-checking --with-system-zlib --enable-__cxa_atexit
--disable-libunwind-exceptions --enable-languages=c,c++,objc,java,f77
--host=i386-redhat-linux
Thread model: posix
gcc version 3.4.0 (Red Hat Linux 3.4.0-1)
 /usr/libexec/gcc/i386-redhat-linux/3.4.0/cc1plus -E -quiet -v -D_GNU_SOURCE
wstring.cpp -o wstring.ii
ignorerar ej existerande katalog
"/usr/lib/gcc/i386-redhat-linux/3.4.0/../../../../i386-redhat-linux/include"
#include "..." sökning startar här:
#include <...> sökning startar här:
 /usr/lib/gcc/i386-redhat-linux/3.4.0/../../../../include/c++/3.4.0

/usr/lib/gcc/i386-redhat-linux/3.4.0/../../../../include/c++/3.4.0/i386-redhat-linux
 /usr/lib/gcc/i386-redhat-linux/3.4.0/../../../../include/c++/3.4.0/backward
 /usr/local/include
 /usr/lib/gcc/i386-redhat-linux/3.4.0/include
 /usr/include
Slut på söklistan.
 /usr/libexec/gcc/i386-redhat-linux/3.4.0/cc1plus -fpreprocessed wstring.ii
-quiet -dumpbase wstring.cpp -auxbase wstring -version -o wstring.s
GNU C++ version 3.4.0 (Red Hat Linux 3.4.0-1) (i386-redhat-linux)
	compiled by GNU C version 3.4.0 (Red Hat Linux 3.4.0-1).
GGC heuristics: --param ggc-min-expand=64 --param ggc-min-heapsize=64645
 as -V -Qy -o wstring.o wstring.s
GNU assembler version 2.15.90.0.3 (i386-redhat-linux) using BFD version
2.15.90.0.3 20040415
 /usr/libexec/gcc/i386-redhat-linux/3.4.0/collect2 --eh-frame-hdr -m elf_i386
-dynamic-linker /lib/ld-linux.so.2
/usr/lib/gcc/i386-redhat-linux/3.4.0/../../../crt1.o
/usr/lib/gcc/i386-redhat-linux/3.4.0/../../../crti.o
/usr/lib/gcc/i386-redhat-linux/3.4.0/crtbegin.o
-L/usr/lib/gcc/i386-redhat-linux/3.4.0 -L/usr/lib/gcc/i386-redhat-linux/3.4.0
-L/usr/lib/gcc/i386-redhat-linux/3.4.0/../../.. wstring.o -lstdc++ -lm -lgcc_s
-lgcc -lc -lgcc_s -lgcc /usr/lib/gcc/i386-redhat-linux/3.4.0/crtend.o
/usr/lib/gcc/i386-redhat-linux/3.4.0/../../../crtn.o


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=15733

[Bug libstdc++/15733] getline(wifstream, wstring) seems to be broken with UTF-8

[ahlgren at lysator dot liu dot se]

------- Additional Comments From ahlgren at lysator dot liu dot se  2004-05-30 13:59 -------
(In reply to comment #2) 
> Again what is your env set to and are you sure that your libc is UTF-8 
compable? 
 
Yes my libc should be UTF-8 compatible, it works fine using "wcin" rather than 
"fin" as explained above. My env: 
 
KDE_MULTIHEAD=false 
SSH_AGENT_PID=2354 
HOSTNAME=skynet 
TERM=xterm 
SHELL=/bin/bash 
XDM_MANAGED=/var/run/xdmctl/xdmctl-:0,maysd,mayfn,sched,rsvd,method=classic 
HISTSIZE=1000 
GTK2_RC_FILES=/etc/gtk-2.0/gtkrc:/home/ahlgren/.gtkrc-2.0:/home/ahlgren/.kde/share/config/gtkrc 
GTK_RC_FILES=/etc/gtk/gtkrc:/home/ahlgren/.gtkrc:/home/ahlgren/.kde/share/config/gtkrc 
GS_LIB=/home/ahlgren/.fonts 
WINDOWID=46137352 
QTDIR=/usr/lib/qt-3.3 
KDE_FULL_SESSION=true 
USER=ahlgren 
XCURSOR_SIZE= 
LS_COLORS=no=00:fi=00:di=00;34:ln=00;36:pi=40;33:so=00;35:bd=40;33;01:cd=40;33;01:or=01;05;37;41:mi=01;05;37;41:ex=00;32:*.cmd=00;32:*.exe=00;32:*.com=00;32:*.btm=00;32:*.bat=00;32:*.sh=00;32:*.csh=00;32:*.tar=00;31:*.tgz=00;31:*.arj=00;31:*.taz=00;31:*.lzh=00;31:*.zip=00;31:*.z=00;31:*.Z=00;31:*.gz=00;31:*.bz2=00;31:*.bz=00;31:*.tz=00;31:*.rpm=00;31:*.cpio=00;31:*.jpg=00;35:*.gif=00;35:*.bmp=00;35:*.xbm=00;35:*.xpm=00;35:*.png=00;35:*.tif=00;35: 
LD_LIBRARY_PATH=/usr/X11R6/lib:/usr/X11R6/lib/modules 
SSH_AUTH_SOCK=/tmp/ssh-JnnJ2311/agent.2311 
KDEDIR=/usr 
SESSION_MANAGER=local/skynet:/tmp/.ICE-unix/2450 
KONSOLE_DCOP=DCOPRef(konsole-2494,konsole) 
DESKTOP_SESSION=default 
PATH=/usr/kerberos/bin:/usr/local/bin:/usr/bin:/bin:/usr/X11R6/bin:/usr/games:/home/ahlgren/bin 
MAIL=/var/spool/mail/ahlgren 
KONSOLE_DCOP_SESSION=DCOPRef(konsole-2494,session-3) 
PWD=/home/ahlgren/dev/examples 
INPUTRC=/etc/inputrc 
LANG=sv_SE.UTF-8 
SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass 
SHLVL=3 
HOME=/home/ahlgren 
XCURSOR_THEME=default 
CHARMAPS=/usr/lib/utrans/charmaps 
LOGNAME=ahlgren 
LESSOPEN=|/usr/bin/lesspipe.sh %s 
DISPLAY=:0 
G_BROKEN_FILENAMES=1 
COLORTERM= 
_=/usr/bin/env 
OLDPWD=/home/ahlgren 
 

-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=15733

[Bug libstdc++/15733] getline(wifstream, wstring) seems to be broken with UTF-8

[pcarlini at suse dot de]

------- Additional Comments From pcarlini at suse dot de  2004-05-30 14:02 -------
Strictly speaking, without 'test.txt' we cannot reproduce the PR and cannot do
much about it.

However, you are reporting that things work ok with 'fin', *not* with 'wcin',
right? This is expected, since, 'wcin' and the other predefined streams are
by default synched with stdio and *cannot* support encoding and decoding.


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=15733

[Bug libstdc++/15733] getline(wifstream, wstring) seems to be broken with UTF-8

[ahlgren at lysator dot liu dot se]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 704 bytes --]


------- Additional Comments From ahlgren at lysator dot liu dot se  2004-05-30 14:04 -------
(In reply to comment #5) 
> Strictly speaking, without 'test.txt' we cannot reproduce the PR and cannot 
do 
> much about it. 
>  
> However, you are reporting that things work ok with 'fin', *not* with 'wcin', 
> right? This is expected, since, 'wcin' and the other predefined streams are 
> by default synched with stdio and *cannot* support encoding and decoding. 
>  
 
No it's the other way around. It works fin with 'wcin', but not with 'fin'. fin 
doesn't recognize the the 'ö' in 'smörgåsbord' as a valid character, while wcin 
does. 

-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=15733

[Bug libstdc++/15733] getline(wifstream, wstring) seems to be broken with UTF-8

[pinskia at gcc dot gnu dot org]

------- Additional Comments From pinskia at gcc dot gnu dot org  2004-05-30 14:04 -------
This works for me on powerpc-apple-darwin. What is the error you get.

-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=15733

[Bug libstdc++/15733] getline(wifstream, wstring) seems to be broken with UTF-8

[ahlgren at lysator dot liu dot se]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 838 bytes --]


------- Additional Comments From ahlgren at lysator dot liu dot se  2004-05-30 14:05 -------
(In reply to comment #6) 
> (In reply to comment #5)  
> > Strictly speaking, without 'test.txt' we cannot reproduce the PR and cannot  
> do  
> > much about it.  
> >   
> > However, you are reporting that things work ok with 'fin', *not* with 
'wcin',  
> > right? This is expected, since, 'wcin' and the other predefined streams are  
> > by default synched with stdio and *cannot* support encoding and decoding.  
> >   
>   
> No it's the other way around. It works fin with 'wcin', but not with 'fin'. 
fin  
> doesn't recognize the the 'ö' in 'smörgåsbord' as a valid character, while 
wcin  
> does.  
 
Oh, and test.txt is sent as an attachment in my first reply. 
 

-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=15733

[Bug libstdc++/15733] getline(wifstream, wstring) seems to be broken with UTF-8

[ahlgren at lysator dot liu dot se]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 431 bytes --]


------- Additional Comments From ahlgren at lysator dot liu dot se  2004-05-30 14:06 -------
(In reply to comment #7) 
> This works for me on powerpc-apple-darwin. What is the error you get. 
 
Using wifstream it doesn't print 'ett smörgåsbord', only 'ett sm'! With wcin 
however, everything works fine, it prints 'ett smörgåsbord' (or whatever I 
echo). 
 
 

-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=15733

[Bug libstdc++/15733] getline(wifstream, wstring) seems to be broken with UTF-8

[pcarlini at suse dot de]

------- Additional Comments From pcarlini at suse dot de  2004-05-30 14:10 -------
To be clear: the predefined streams 'wcin', 'wcout', etc, are synced with
stdio by default and for this reason *cannot* do any encoding/decoding.

In order to obtain sensible results you either need to use always wfstreams
(in input and output) or call 'std::ios::sync_with_stdio(false);' before
doing any I/O.

If wcin appear to work is only because is *not* doing any conversion.

You need to imbue loc to the ifstream fin!

Do things work in this case?

-- 
           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|UNCONFIRMED                 |WAITING


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=15733

[Bug libstdc++/15733] getline(wifstream, wstring) seems to be broken with UTF-8

[ahlgren at lysator dot liu dot se]

------- Additional Comments From ahlgren at lysator dot liu dot se  2004-05-30 14:17 -------
(In reply to comment #10) 
> To be clear: the predefined streams 'wcin', 'wcout', etc, are synced with 
> stdio by default and for this reason *cannot* do any encoding/decoding. 
>  
> In order to obtain sensible results you either need to use always wfstreams 
> (in input and output) or call 'std::ios::sync_with_stdio(false);' before 
> doing any I/O. 
 
Oh, does that explain why I need 'setenv(LC_ALL, "")' to make the output work 
with wcin/wcout? Removing that line will cripple the output with wcin/wcout. On 
the other hand, removing the imbue calls doesn't affect anything, but this 
should be fixed by calling sync_with_stdio(false), did I get it right? 
 
>  
> If wcin appear to work is only because is *not* doing any conversion. 
>  
> You need to imbue loc to the ifstream fin! 
>  
> Do things work in this case? 
 
Yes things work fine adding 'fin.imbue(loc);' 
Thanks. The reason for not trying this is that the only call that actually sets 
the proper locale is 'setlocale(LC_ALL, "")' so far, but that was apparently as 
misunderstanding :-( 
 
 

-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=15733

[Bug libstdc++/15733] getline(wifstream, wstring) seems to be broken with UTF-8

[pcarlini at suse dot de]

------- Additional Comments From pcarlini at suse dot de  2004-05-30 14:26 -------
> Oh, does that explain why I need 'setenv(LC_ALL, "")' to make the output work 
> with wcin/wcout? Removing that line will cripple the output with wcin/wcout. 
> On the other hand, removing the imbue calls doesn't affect anything, but this 
> should be fixed by calling sync_with_stdio(false), did I get it right?

Basically, yes.

>Yes things work fine adding 'fin.imbue(loc);' 
>Thanks. The reason for not trying this is that the only call that actually sets 
>the proper locale is 'setlocale(LC_ALL, "")' so far, but that was apparently as 
>misunderstanding :-(

Yes. You can think of 'setlocale(LC_ALL, "")' as the 'C', not the 'C++' way
to deal with localization. In C++, every different converting stream (i.e.,
fstream, basically or the predefined but only if un-synced, which leads to
fstream under the hood anyway) has its own locale used for doing conversions:
you may have thousands of those, each one different, at the same time. You
select each locale via 'imbue'.

Notice that my explanation above is simplified and rough. In order to know
the details I suggest a good book, such as Langer & Kreft.

Thanks.

-- 
           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|WAITING                     |RESOLVED
         Resolution|                            |INVALID


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=15733

[Bug libstdc++/15733] getline(wifstream, wstring) seems to be broken with UTF-8

[ahlgren at lysator dot liu dot se]

------- Additional Comments From ahlgren at lysator dot liu dot se  2004-05-30 14:35 -------
Thanks a lot! 
That cleared up the basic details at least. 
 

-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=15733