Signedness of wchar_t and wint_t leads to problems with gcc -Wsign-conversion

Signedness of wchar_t and wint_t leads to problems with gcc -Wsign-conversion

[Michael Kerrisk (man-pages)]

I was looking at a Debian bug report[1] that left me puzzled on a
certain point. At the very least, I want to document a solution to the
reporter's (Igor) problem, but I wonder also if there is an
implementation bug.

The issue concerns iwslower() and simialr functions that take a wint_t argument:

     int iswlower(wint_t wc);

Suppose that the argument to be passed is a 'wchar_t' (as is common,
AFAIK). If one compiles with

    gcc -Wsign-conversion

then the following warning results:

warning: conversion to ‘wint_t {aka unsigned int}’ from ‘wchar_t {aka
int}’ may change the sign of the result [-Wsign-conversion]

What should one do to remove this warning?

There is an analogous situation with islower() and similar functions,
where the solution is described by an update I recently added for the
upcoming man-pages release:

       The standards require that the argument c for these  functions  is
       either  EOF  or a value that is representable in the type unsigned
       char.  If the argument c is of type  char,  it  must  be  cast  to
       unsigned char, as in the following example:

           char c;
           ...
           res = toupper((unsigned char) c);

       This  is  necessary  because  char may be the equivalent of signed
       char, in which case a byte where the top bit is set would be  sign
       extended  when converting to int, yielding a value that is outside
       the range of unsigned char.

However, we don't have a similar solution for iswlower(), because
there is no "(unsigned wchar_t)" cast. And casting to (wint_t) seems
incorrect to me, because if wchar_t is a signed type smaller than
wint_t, then sign extension could occur. (POSIX allows wint_t and
wchar_t to be either signed or unsigned integer types.)

I could be wrong, but it seems like an implementation bug that one of
these types is signed and the other is unsigned. (My brief inspection
of some other systems suggests that on those systems, wchar_t and
wint_t have the same signedness.)

Anyone have any insights here, or did I miss something obvious?

Cheers,

Michael


[1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=845172
-- 
Michael Kerrisk
Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/
Linux/UNIX System Programming Training: http://man7.org/training/

Re: Signedness of wchar_t and wint_t leads to problems with gcc -Wsign-conversion

[Andreas Schwab]

On Nov 21 2016, "Michael Kerrisk (man-pages)" <mtk.manpages@gmail.com> wrote:

> However, we don't have a similar solution for iswlower(), because
> there is no "(unsigned wchar_t)" cast. And casting to (wint_t) seems
> incorrect to me, because if wchar_t is a signed type smaller than
> wint_t, then sign extension could occur.

wint_t is "an integer type capable of storing any valid value of wchar_t
or WEOF".  If the sign-extended value of a wchar_t isn't a valid value
then you have undefined behaviour.  WEOF must be defined such that no
conflict can occur with any valid wchar_t value.

Andreas.

-- 
Andreas Schwab, SUSE Labs, schwab@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

Re: Signedness of wchar_t and wint_t leads to problems with gcc -Wsign-conversion

[Michael Kerrisk (man-pages)]

Hello Andreas,

On 21 November 2016 at 13:45, Andreas Schwab <schwab@suse.de> wrote:
> On Nov 21 2016, "Michael Kerrisk (man-pages)" <mtk.manpages@gmail.com> wrote:
>
>> However, we don't have a similar solution for iswlower(), because
>> there is no "(unsigned wchar_t)" cast. And casting to (wint_t) seems
>> incorrect to me, because if wchar_t is a signed type smaller than
>> wint_t, then sign extension could occur.
>
> wint_t is "an integer type capable of storing any valid value of wchar_t
> or WEOF".

Yes.

> If the sign-extended value of a wchar_t isn't a valid value
> then you have undefined behaviour.

See below.

> WEOF must be defined such that no
> conflict can occur with any valid wchar_t value.

Yes.

So, I'm not quite clear on the point that you are making in your
second point. Given code such as:

    wchar_t w;
    ...
    if (iswlower((wint_t) w) ....

Do you mean that the standards are saying that casting to wint_t here
is guaranteed to be correct? I can't see the line of reasoning that
leads there.

Cheers,

Michael



-- 
Michael Kerrisk
Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/
Linux/UNIX System Programming Training: http://man7.org/training/

Re: Signedness of wchar_t and wint_t leads to problems with gcc -Wsign-conversion

[Andreas Schwab]

On Nov 21 2016, "Michael Kerrisk (man-pages)" <mtk.manpages@gmail.com> wrote:

> So, I'm not quite clear on the point that you are making in your
> second point. Given code such as:
>
>     wchar_t w;
>     ...
>     if (iswlower((wint_t) w) ....
>
> Do you mean that the standards are saying that casting to wint_t here
> is guaranteed to be correct? I can't see the line of reasoning that
> leads there.

Any valid value of wchar_t is representable by wint_t.  I cannot find
any more convincing argument.

Andreas.

-- 
Andreas Schwab, SUSE Labs, schwab@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

Re: Signedness of wchar_t and wint_t leads to problems with gcc -Wsign-conversion

[Szabolcs Nagy]

On 21/11/16 12:54, Michael Kerrisk (man-pages) wrote:
> So, I'm not quite clear on the point that you are making in your
> second point. Given code such as:
> 
>     wchar_t w;
>     ...
>     if (iswlower((wint_t) w) ....
> 
> Do you mean that the standards are saying that casting to wint_t here
> is guaranteed to be correct? I can't see the line of reasoning that
> leads there.

an implementation is allowed to support wide
characters with >=0 value only (independently
of signedness of wchar_t).

in which case (wint_t)w is ok.

Re: Signedness of wchar_t and wint_t leads to problems with gcc -Wsign-conversion

[Michael Kerrisk (man-pages)]

On 11/21/2016 02:01 PM, Szabolcs Nagy wrote:
> On 21/11/16 12:54, Michael Kerrisk (man-pages) wrote:
>> So, I'm not quite clear on the point that you are making in your
>> second point. Given code such as:
>>
>>     wchar_t w;
>>     ...
>>     if (iswlower((wint_t) w) ....
>>
>> Do you mean that the standards are saying that casting to wint_t here
>> is guaranteed to be correct? I can't see the line of reasoning that
>> leads there.
> 
> an implementation is allowed to support wide
> characters with >=0 value only (independently
> of signedness of wchar_t).

Yes, it's allowed to, but it may also do otherwise, as in there
may be implementations where wchar_t runes are negative (right?).

> in which case (wint_t)w is ok.

Yes, but from the perspective of portable code, it's still
not clear to me that the cast is okay on every possible
implementation of 'wchar_t' and 'wint_t'. 

Cheers,

Michael



-- 
Michael Kerrisk
Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/
Linux/UNIX System Programming Training: http://man7.org/training/

Re: Signedness of wchar_t and wint_t leads to problems with gcc -Wsign-conversion

[Michael Kerrisk (man-pages)]

On 11/21/2016 02:01 PM, Andreas Schwab wrote:
> On Nov 21 2016, "Michael Kerrisk (man-pages)" <mtk.manpages@gmail.com> wrote:
> 
>> So, I'm not quite clear on the point that you are making in your
>> second point. Given code such as:
>>
>>     wchar_t w;
>>     ...
>>     if (iswlower((wint_t) w) ....
>>
>> Do you mean that the standards are saying that casting to wint_t here
>> is guaranteed to be correct? I can't see the line of reasoning that
>> leads there.
> 
> Any valid value of wchar_t is representable by wint_t.  I cannot find
> any more convincing argument.

Yes, but is a sign-extended wchar_t a valid value?

To put things another way, why is this scenario different from
the islower() example, where the case to '(unsigned char)' is
needed in portable code? I'm being dense about something, I guess.

Cheers,

Michael



-- 
Michael Kerrisk
Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/
Linux/UNIX System Programming Training: http://man7.org/training/

Re: Signedness of wchar_t and wint_t leads to problems with gcc -Wsign-conversion

[Szabolcs Nagy]

On 21/11/16 13:06, Michael Kerrisk (man-pages) wrote:
> On 11/21/2016 02:01 PM, Szabolcs Nagy wrote:
>> On 21/11/16 12:54, Michael Kerrisk (man-pages) wrote:
>>> So, I'm not quite clear on the point that you are making in your
>>> second point. Given code such as:
>>>
>>>     wchar_t w;
>>>     ...
>>>     if (iswlower((wint_t) w) ....
>>>
>>> Do you mean that the standards are saying that casting to wint_t here
>>> is guaranteed to be correct? I can't see the line of reasoning that
>>> leads there.
>>
>> an implementation is allowed to support wide
>> characters with >=0 value only (independently
>> of signedness of wchar_t).
> 
> Yes, it's allowed to, but it may also do otherwise, as in there
> may be implementations where wchar_t runes are negative (right?).

on those systems wint_t must be signed.

Re: Signedness of wchar_t and wint_t leads to problems with gcc -Wsign-conversion

[Andreas Schwab]

On Nov 21 2016, "Michael Kerrisk (man-pages)" <mtk.manpages@gmail.com> wrote:

> On 11/21/2016 02:01 PM, Andreas Schwab wrote:
>> On Nov 21 2016, "Michael Kerrisk (man-pages)" <mtk.manpages@gmail.com> wrote:
>> 
>>> So, I'm not quite clear on the point that you are making in your
>>> second point. Given code such as:
>>>
>>>     wchar_t w;
>>>     ...
>>>     if (iswlower((wint_t) w) ....
>>>
>>> Do you mean that the standards are saying that casting to wint_t here
>>> is guaranteed to be correct? I can't see the line of reasoning that
>>> leads there.
>> 
>> Any valid value of wchar_t is representable by wint_t.  I cannot find
>> any more convincing argument.
>
> Yes, but is a sign-extended wchar_t a valid value?

A valid value of wchar_t is a valid value.

Andreas.

-- 
Andreas Schwab, SUSE Labs, schwab@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

Re: Signedness of wchar_t and wint_t leads to problems with gcc -Wsign-conversion

[Michael Kerrisk (man-pages)]

On 11/21/2016 02:28 PM, Andreas Schwab wrote:
> On Nov 21 2016, "Michael Kerrisk (man-pages)" <mtk.manpages@gmail.com> wrote:
> 
>> On 11/21/2016 02:01 PM, Andreas Schwab wrote:
>>> On Nov 21 2016, "Michael Kerrisk (man-pages)" <mtk.manpages@gmail.com> wrote:
>>>
>>>> So, I'm not quite clear on the point that you are making in your
>>>> second point. Given code such as:
>>>>
>>>>     wchar_t w;
>>>>     ...
>>>>     if (iswlower((wint_t) w) ....
>>>>
>>>> Do you mean that the standards are saying that casting to wint_t here
>>>> is guaranteed to be correct? I can't see the line of reasoning that
>>>> leads there.
>>>
>>> Any valid value of wchar_t is representable by wint_t.  I cannot find
>>> any more convincing argument.
>>
>> Yes, but is a sign-extended wchar_t a valid value?
> 
> A valid value of wchar_t is a valid value.

Sorry, that makes me no wiser :-(. You snipped the question
whose answer might help me:

To put things another way, why is this scenario different from
the islower() example, where the cast to '(unsigned char)' is
needed in portable code?

Cheers,

Michael

-- 
Michael Kerrisk
Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/
Linux/UNIX System Programming Training: http://man7.org/training/

Re: Signedness of wchar_t and wint_t leads to problems with gcc -Wsign-conversion

[Andreas Schwab]

On Nov 21 2016, "Michael Kerrisk (man-pages)" <mtk.manpages@gmail.com> wrote:

> To put things another way, why is this scenario different from
> the islower() example, where the cast to '(unsigned char)' is
> needed in portable code?

Because char is weird.

Andreas.

-- 
Andreas Schwab, SUSE Labs, schwab@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

Re: Signedness of wchar_t and wint_t leads to problems with gcc -Wsign-conversion

[Zack Weinberg]

On Mon, Nov 21, 2016 at 8:46 AM, Michael Kerrisk (man-pages)
<mtk.manpages@gmail.com> wrote:
> On 11/21/2016 02:28 PM, Andreas Schwab wrote:
>> On Nov 21 2016, "Michael Kerrisk (man-pages)" <mtk.manpages@gmail.com> wrote:
>>> On 11/21/2016 02:01 PM, Andreas Schwab wrote:
>>>> On Nov 21 2016, "Michael Kerrisk (man-pages)" <mtk.manpages@gmail.com> wrote:
>>>>
>>>>> So, I'm not quite clear on the point that you are making in your
>>>>> second point. Given code such as:
>>>>>
>>>>>     wchar_t w;
>>>>>     ...
>>>>>     if (iswlower((wint_t) w) ....
>>>>>
>>>>> Do you mean that the standards are saying that casting to wint_t here
>>>>> is guaranteed to be correct? I can't see the line of reasoning that
>>>>> leads there.
>>>>
>>>> Any valid value of wchar_t is representable by wint_t.  I cannot find
>>>> any more convincing argument.
>>>
>>> Yes, but is a sign-extended wchar_t a valid value?
>>
>> A valid value of wchar_t is a valid value.
>
> Sorry, that makes me no wiser :-(. You snipped the question
> whose answer might help me:
>
> To put things another way, why is this scenario different from
> the islower() example, where the cast to '(unsigned char)' is
> needed in portable code?

It's clear to me that Andreas believes that `iswlower((wint_t) w)` is
required to do the Right Thing, and off the top of my head, that
interpretation of the standard makes more sense than any alternative.

However, it is not clear to me that actual implementations honor that
requirement.  Consider the case where wchar_t is signed short (this
hypothetical implementation is limited to the BMP), wint_t is signed
int, and WEOF == ((wint_t)-1).  Then (wint_t)U+FFFF is
indistinguishable from WEOF.  Yes, U+FFFF is a reserved codepoint, but
it's still wrong for it to collide.

I would not want to swear that this never happens in real life without
an exhaustive audit of existing implementations.

zw

Re: Signedness of wchar_t and wint_t leads to problems with gcc -Wsign-conversion

[Florian Weimer]

On 11/21/2016 03:05 PM, Zack Weinberg wrote:

> However, it is not clear to me that actual implementations honor that
> requirement.  Consider the case where wchar_t is signed short (this
> hypothetical implementation is limited to the BMP), wint_t is signed
> int, and WEOF == ((wint_t)-1).  Then (wint_t)U+FFFF is
> indistinguishable from WEOF.  Yes, U+FFFF is a reserved codepoint, but
> it's still wrong for it to collide.

That would be consistent with the situation with signed chars and EOF. 
Everyone in a while, someone complains why isalpha ('ÿ') is 0.

Florian

Re: Signedness of wchar_t and wint_t leads to problems with gcc -Wsign-conversion

[Andreas Schwab]

On Nov 21 2016, Zack Weinberg <zackw@panix.com> wrote:

> However, it is not clear to me that actual implementations honor that
> requirement.  Consider the case where wchar_t is signed short (this
> hypothetical implementation is limited to the BMP), wint_t is signed
> int, and WEOF == ((wint_t)-1).  Then (wint_t)U+FFFF is
> indistinguishable from WEOF.  Yes, U+FFFF is a reserved codepoint, but
> it's still wrong for it to collide.

WEOF must be defined "to a constant expression of type wint_t whose
value does not correspond to any member of the extended character set."
This guarantees that there cannot be a conflict with valid values of
wchar_t.

Andreas.

-- 
Andreas Schwab, SUSE Labs, schwab@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

Re: Signedness of wchar_t and wint_t leads to problems with gcc -Wsign-conversion

[Zack Weinberg]

On Mon, Nov 21, 2016 at 9:37 AM, Andreas Schwab <schwab@suse.de> wrote:
> On Nov 21 2016, Zack Weinberg <zackw@panix.com> wrote:
>
>> However, it is not clear to me that actual implementations honor that
>> requirement.  Consider the case where wchar_t is signed short (this
>> hypothetical implementation is limited to the BMP), wint_t is signed
>> int, and WEOF == ((wint_t)-1).  Then (wint_t)U+FFFF is
>> indistinguishable from WEOF.  Yes, U+FFFF is a reserved codepoint, but
>> it's still wrong for it to collide.
>
> WEOF must be defined "to a constant expression of type wint_t whose
> value does not correspond to any member of the extended character set."
> This guarantees that there cannot be a conflict with valid values of
> wchar_t.

 "It is not clear to me that actual implementations honor that requirement."

Re: Signedness of wchar_t and wint_t leads to problems with gcc -Wsign-conversion

[Igor Liferenko]

Hi all,

Can you please say if the typecasts in the following two programs are valid?
(I just replaced wint_t and wchar_t to their actual meaning in glibc)

    #include <locale.h>
    #include <wchar.h>
    int main(void)
    {
      setlocale(LC_CTYPE, "en_US.UTF-8");
      unsigned int wc;
      wc = getwchar();
      putwchar((int) wc);
    }

--

    #include <locale.h>
    #include <wchar.h>
    #include <wctype.h>
    int main(void)
    {
      setlocale(LC_CTYPE, "en_US.UTF-8");
      int wc;
      wc = L'ÿ';
      if (iswlower((unsigned int) wc)) return 0;
      return 1;
    }


Igor

Re: Signedness of wchar_t and wint_t leads to problems with gcc -Wsign-conversion

[Szabolcs Nagy]

On 23/11/16 05:50, Igor Liferenko wrote:
> Can you please say if the typecasts in the following two programs are valid?
> (I just replaced wint_t and wchar_t to their actual meaning in glibc)
> 

that is not valid (to replace the types) as they are
target dependent, so the code is no longer portable
and you didnt say what targets you are interested in.

>     #include <locale.h>
>     #include <wchar.h>
>     int main(void)
>     {
>       setlocale(LC_CTYPE, "en_US.UTF-8");
>       unsigned int wc;
>       wc = getwchar();
>       putwchar((int) wc);

getwchar may return WEOF, you need to check for that
before passing wc to putwchar.

if the check is performed, then the cast is unnecessary
(but valid) if the right types are used.

>     }
> 
> --
> 
>     #include <locale.h>
>     #include <wchar.h>
>     #include <wctype.h>
>     int main(void)
>     {
>       setlocale(LC_CTYPE, "en_US.UTF-8");
>       int wc;
>       wc = L'ÿ';
>       if (iswlower((unsigned int) wc)) return 0;

the cast is unnecessary, it does not change the value
(assuming the right types are used).

>       return 1;
>     }
> 
> 
> Igor
>