libstdc++/9875: filebuf doesn't handle codecvt::encoding() > 1

libstdc++/9875: filebuf doesn't handle codecvt::encoding() > 1

[peturr02]

>Number:         9875
>Category:       libstdc++
>Synopsis:       filebuf doesn't handle codecvt::encoding() > 1
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    unassigned
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Thu Feb 27 10:36:01 UTC 2003
>Closed-Date:
>Last-Modified:
>Originator:     peturr02@ru.is
>Release:        gcc-3.2.1
>Organization:
>Environment:
Red Hat Linux 8.0
>Description:
Many basic_filebuf members (at least seekoff, seekpos, sync, overflow, underflow) can't handle fixed width multibyte encodings (where codecvt<>::encoding() > 1). It seems that every place _M_file.seekoff is called, it is assumed that codecvt<>::encoding() == 1.
>How-To-Repeat:
See attachment. Note that this test is incomplete because the bug in seekoff shadows some of the other bugs.
>Fix:

>Release-Note:
>Audit-Trail:
>Unformatted:
----gnatsweb-attachment----
Content-Type: text/plain; name="seekoffbug.cc"
Content-Disposition: inline; filename="seekoffbug.cc"

#include <fstream>
#include <locale>
#include <algorithm>
#include <cstring>

#undef NDEBUG
#include <cassert>

class Cvt : public std::codecvt<wchar_t, char, mbstate_t>
{
protected:
	virtual std::codecvt_base::result
	do_out(std::mbstate_t&, const wchar_t* from, const wchar_t* from_end,
	       const wchar_t*& from_next, char* to, char* to_end,
	       char*& to_next) const
		{
			std::size_t from_len = from_end - from;
			std::size_t to_len = (to_end - to) / sizeof(wchar_t);
			std::size_t len = std::min(from_len, to_len);
			std::memcpy(to, from, len * sizeof(wchar_t));
			from_next = from + len;
			to_next = to + len * sizeof(wchar_t);
			return from_next == from_end ? std::codecvt_base::ok :
				std::codecvt_base::partial;
		}

	virtual std::codecvt_base::result
	do_in(std::mbstate_t&, const char* from, const char* from_end,
	      const char*& from_next, wchar_t* to, wchar_t* to_end,
	      wchar_t*& to_next) const
		{
			std::size_t from_len =
				(from_end - from) / sizeof(wchar_t);
			std::size_t to_len = to_end - to;
			std::size_t len = std::min(from_len, to_len);
			std::memcpy(to, from, len * sizeof(wchar_t));
			from_next = from + len * sizeof(wchar_t);
			to_next = to + len;
			return from_next == from_end ? std::codecvt_base::ok :
				std::codecvt_base::partial;
		}

	virtual std::codecvt_base::result
	do_unshift(std::mbstate_t&, char*, char*, char*&) const
		{ return std::codecvt_base::noconv; }

	virtual int do_encoding() const throw() { return sizeof(wchar_t); }
	virtual bool do_always_noconv() const throw() { return false; }

	virtual int
	do_length(std::mbstate_t&, const char* from, const char* end,
		  std::size_t max)
		{
			std::size_t len = (end - from) / sizeof(wchar_t);
			return std::min(len, max) * sizeof(wchar_t);
		}

	virtual int do_max_length() const throw() { return sizeof(wchar_t); }
};

void test1()
{
	using namespace std;

	// seekoff
	wfilebuf fb;
	fb.pubimbue(locale(locale::classic(), new Cvt));
	fb.open("tmp", ios_base::out | ios_base::in | ios_base::trunc);
	fb.sputn(L"0123456789", 10);
	fb.pubseekoff(-3, ios_base::cur);
	assert(fb.sgetc() == L'7');
	fb.pubseekoff(-3, ios_base::cur);
	assert(fb.sgetc() == L'4');
	fb.close();
}

void test2()
{
	using namespace std;

	// seekpos
	wfilebuf fb;
	fb.pubimbue(locale(locale::classic(), new Cvt));
	fb.open("tmp", ios_base::out | ios_base::in | ios_base::trunc);
	fb.sputn(L"0123456789", 10);

	streampos p1 = fb.pubseekoff(0, ios_base::cur);
	assert(p1 != streampos(-1));
	fb.sputc(L'a');

	streampos p2 = fb.pubseekpos(p1);
	assert(p2 != streampos(-1));
	assert(p2 == p1);
	assert(fb.sgetc() == L'a');

	fb.pubseekoff(0, ios_base::beg);
	wchar_t buf[11];
	streamsize s1 = fb.sgetn(buf, 11);
	assert(s1 == 11);
	assert(!wmemcmp(buf, L"0123456789a", 11));

	fb.close();
}

void test3()
{
	using namespace std;

	// overflow
	locale loc (locale::classic(), new Cvt);
	wfilebuf fb1;
	fb1.pubimbue(loc);
	fb1.open("tmp", ios_base::out | ios_base::trunc);
	fb1.sputn(L"0123456789", 10);
	fb1.close();

	wfilebuf fb2;
	fb2.pubimbue(loc);
	fb2.open("tmp", ios_base::in | ios_base::out);
	fb2.sputc(L'a');
	fb2.close();

	wfilebuf fb3;
	fb3.pubimbue(loc);
	fb3.open("tmp", ios_base::in);
	assert(fb3.is_open());
	wchar_t buf[10];
	streamsize s1 = fb3.sgetn(buf, 10);
	assert(s1 == 10);
	assert(!wmemcmp(buf, L"a123456789", 10));
	fb3.close();
}

int main()
{
	test1();
	test2();
	test3();

	return 0;
}

RE: libstdc++/9875: filebuf doesn't handle codecvt::encoding() > 1

[Pétur Runólfsson]

The following reply was made to PR libstdc++/9875; it has been noted by GNATS.

From: =?iso-8859-1?Q?P=E9tur_Run=F3lfsson?= <peturr02@ru.is>
To: "Paolo Carlini" <pcarlini@unitus.it>
Cc: <paolo@gcc.gnu.org>,
	<gcc-bugs@gcc.gnu.org>,
	<gcc-gnats@gcc.gnu.org>
Subject: RE: libstdc++/9875: filebuf doesn't handle codecvt::encoding() > 1
Date: Mon, 3 Mar 2003 18:38:27 -0000

 > >>    The underflow bits are still missing, however. Are you
 > >>    willing to provide some tests for it too?
 > >>   =20
 > >>
 > >Will do.
 > >
 > Thanks P=E9tur.
 > By the way, I would appreciate also some actual ;) tests for=20
 > overflow,=20
 > since those
 > present in the PR didn't really lead to overflow for the default 8192=20
 > chars wide buffer.
 > (temporarily I have added some using setbuf to shorten the buffer)
 
 These are a bit hairy, but that is by design. The seekoff in
 _M_really_overflow is only called when _M_filepos !=3D _M_out_beg
 which only happens when switching modes, and the seekoff in
 _M_underflow_common only happens when _M_in_cur !=3D _M_filepos,
 which never happens when underflow is called from any public
 members of streambuf or filebuf.
 
 Petur
 
 void test4()
 {
 	using namespace std;
 
 	// overflow
 	locale loc (locale::classic(), new Cvt);
 	wfilebuf fb1;
 	fb1.pubimbue(loc);
 	fb1.open("tmp", ios_base::out | ios_base::trunc);
 	fb1.sputn(L"0123456789", 10);
 	fb1.close();
 
 	wfilebuf fb2;
 	fb2.pubimbue(loc);
 	fb2.open("tmp", ios_base::in | ios_base::out);
 
 	fb2.sgetc();
 =09
 	int written =3D 0;
 	for (int i =3D 0; i < BUFSIZ; ++i)
 		written +=3D fb2.sputn(L"abc", 3);
 
 	fb2.close();
 
 	wfilebuf fb3;
 	fb3.pubimbue(loc);
 	fb3.open("tmp", ios_base::in);
 
 	for (int j =3D 0; j < BUFSIZ; ++j)
 	{
 		wchar_t buf[] =3D L"xxx\n";
 		int n =3D fb3.sgetn(buf, 3);
 		assert(n =3D=3D 3);
 		fputws(buf, stderr);
 		assert(!wmemcmp(buf, L"abc", 3));
 	}
 =09
 	fb3.close();
 }
 
 class Buf : public std::wfilebuf
 {
 public:
 	int_type pub_underflow()
 		{ return underflow(); }
 };
 
 void test5()
 {
 	using namespace std;
 
 	// underflow
 	const wchar_t* strlit =3D L"0123456789";
 	locale loc (locale::classic(), new Cvt);
 	wfilebuf fb1;
 	fb1.pubimbue(loc);
 	fb1.open("tmp", ios_base::out | ios_base::trunc);
 	fb1.sputn(strlit, 10);
 	fb1.close();
 
 	Buf fb2;
 	fb2.pubimbue(loc);
 	fb2.open("tmp", ios_base::in);
 
 	for (int i =3D 0; i < 10; ++i)
 	{
 		assert(fb2.pub_underflow() =3D=3D strlit[i]);
 		fb2.sbumpc();
 	}
 
 	fb2.close();
 }

Re: libstdc++/9875: filebuf doesn't handle codecvt::encoding() > 1

[Paolo Carlini]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 806 bytes --]

The following reply was made to PR libstdc++/9875; it has been noted by GNATS.

From: Paolo Carlini <pcarlini@unitus.it>
To: =?ISO-8859-1?Q?P=E9tur_Run=F3lfsson?= <peturr02@ru.is>
Cc: paolo@gcc.gnu.org,  gcc-bugs@gcc.gnu.org,  gcc-gnats@gcc.gnu.org
Subject: Re: libstdc++/9875: filebuf doesn't handle codecvt::encoding() >
 1
Date: Mon, 03 Mar 2003 11:57:05 +0100

 Pétur Runólfsson wrote:
 
 >>    The underflow bits are still missing, however. Are you
 >>    willing to provide some tests for it too?
 >>    
 >>
 >Will do.
 >
 Thanks Pétur.
 By the way, I would appreciate also some actual ;) tests for overflow, 
 since those
 present in the PR didn't really lead to overflow for the default 8192 
 chars wide buffer.
 (temporarily I have added some using setbuf to shorten the buffer)
 
 Paolo.
 

RE: libstdc++/9875: filebuf doesn't handle codecvt::encoding() > 1

[Pétur Runólfsson]

The following reply was made to PR libstdc++/9875; it has been noted by GNATS.

From: =?iso-8859-1?Q?P=E9tur_Run=F3lfsson?= <peturr02@ru.is>
To: <paolo@gcc.gnu.org>,
	<gcc-bugs@gcc.gnu.org>,
	=?iso-8859-1?Q?P=E9tur_Run=F3lfsson?= <peturr02@ru.is>,
	<gcc-gnats@gcc.gnu.org>
Cc:  
Subject: RE: libstdc++/9875: filebuf doesn't handle codecvt::encoding() > 1
Date: Mon, 3 Mar 2003 10:51:01 -0000

 >     The underflow bits are still missing, however. Are you
 >     willing to provide some tests for it too?
 
 Will do.
 
 Petur

Re: libstdc++/9875: filebuf doesn't handle codecvt::encoding() > 1

[paolo]

Synopsis: filebuf doesn't handle codecvt::encoding() > 1

State-Changed-From-To: closed->analyzed
State-Changed-By: paolo
State-Changed-When: Sat Mar  1 10:44:16 2003
State-Changed-Why:
    Mistakenly closed ;)
    
    Anyway, I have a patch for it, quite simple indeed, which
    already passes all your tests.
    
    The underflow bits are still missing, however. Are you
    willing to provide some tests for it too?
    
    Thanks, Paolo.

http://gcc.gnu.org/cgi-bin/gnatsweb.pl?cmd=view%20audit-trail&database=gcc&pr=9875

Re: libstdc++/9875: filebuf doesn't handle codecvt::encoding() > 1

[paolo]

Synopsis: filebuf doesn't handle codecvt::encoding() > 1

Responsible-Changed-From-To: unassigned->paolo
Responsible-Changed-By: paolo
Responsible-Changed-When: Fri Feb 28 22:55:07 2003
Responsible-Changed-Why:
    Patching.
State-Changed-From-To: open->closed
State-Changed-By: paolo
State-Changed-When: Fri Feb 28 22:55:07 2003
State-Changed-Why:
    Confirmed.

http://gcc.gnu.org/cgi-bin/gnatsweb.pl?cmd=view%20audit-trail&database=gcc&pr=9875