Re: proposed Opengroup action for c99 command (XCU ERN 76)

Re: proposed Opengroup action for c99 command (XCU ERN 76)

[Ross Ridge]

>The POSIXy way to do that would be to refer to the LC_CHARSET
>environment variable, but then consider
>
>LC_CHARSET=UTF-16 c99 foo.c
>
>where 'foo.c' is in UTF-16 and contains '#include <stdio.h>',

Not really a problem for a number of reasons.  First, it's LC_CTYPE
you're thinking of.  Second, the narrow character set can only be 16-bits
wide if "char" is 16-bits.  Thirdly, if the character set that LC_CTYPE
selects isn't superset of the POSIX portable character set then result
is undefined.  So if <stdio.h> happens to writen using characters only
from C basic character set (which is a subset of the portable character
set), there isn't a problem.

						Ross Ridge

Re: proposed Opengroup action for c99 command (XCU ERN 76)

[Geoffrey Keating]

rridge@csclub.uwaterloo.ca (Ross Ridge) writes:

> >The POSIXy way to do that would be to refer to the LC_CHARSET
> >environment variable, but then consider
> >
> >LC_CHARSET=UTF-16 c99 foo.c
> >
> >where 'foo.c' is in UTF-16 and contains '#include <stdio.h>',
> 
> Not really a problem for a number of reasons.  First, it's LC_CTYPE
> you're thinking of.  Second, the narrow character set can only be 16-bits
> wide if "char" is 16-bits.  Thirdly, if the character set that LC_CTYPE
> selects isn't superset of the POSIX portable character set then result
> is undefined.  So if <stdio.h> happens to writen using characters only
> from C basic character set (which is a subset of the portable character
> set), there isn't a problem.

My concern would be that it would be written using only characters
"from the basic C character set", but the actual byte values that
represent those characters might differ.  For instance, if POSIX
allows LC_CTYPE=EBCDIC (or equivalent).

Re: proposed Opengroup action for c99 command (XCU ERN 76)

[Ross Ridge]

> > Not really a problem for a number of reasons.  First, it's LC_CTYPE
> > you're thinking of.  Second, the narrow character set can only be 16-bits
> > wide if "char" is 16-bits.  Thirdly, if the character set that LC_CTYPE
> > selects isn't superset of the POSIX portable character set then result
> > is undefined.  So if <stdio.h> happens to writen using characters only
> > from C basic character set (which is a subset of the portable character
> > set), there isn't a problem.
> 
> My concern would be that it would be written using only characters
> "from the basic C character set", but the actual byte values that
> represent those characters might differ.  For instance, if POSIX
> allows LC_CTYPE=EBCDIC (or equivalent).

If setting LC_CTYPE=EBCDIC selects a character set that is a superset of
the POSIX portable character set on that POSIX system then there isn't
a problem.  If it selects a character set that isn't a superset, say
because the POSIX portable character set is encoded using ASCII on that
system, then the effect is undefined, and so there's also isn't a problem.

						Ross Ridge

-- 
 l/  //	  Ross Ridge -- The Great HTMU
[oo][oo]  rridge@csclub.uwaterloo.ca
-()-/()/  http://www.csclub.uwaterloo.ca/u/rridge/ 
 db  //	  

Re: proposed Opengroup action for c99 command (XCU ERN 76)

[Geoffrey Keating]

Paul Eggert <eggert@CS.UCLA.EDU> writes:

> Thanks, everybody, for writing about this.
> 
> The standardization process is one of consensus, and if the GCC
> developers find some areas of disagreement here I think it unlikely
> that the other POSIX implementers will agree with the proposed action.
> Hence I am thinking of weakening it.
> 
> Currently the action proposes to insert the following text:
> 
>    It is implementation-defined whether trailing white-space characters
>    in each C-language source line are ignored.  Otherwise, the
>    multibyte characters of each source line are mapped on a one-to-one
>    basis to the C source character set.
> 
> How about if I propose to insert the following text instead?
> 
>    The multibyte characters of each source file are mapped to the C
>    source character set on a one-to-one basis, with the following
>    exceptions:
> 
>      * It is implementation-defined whether trailing white-space
>        characters in each input line are ignored.
> 
>      * Each extended source character, and each sequence of characters
>        that would otherwise be recognized as a universal character
>        name, is mapped to an implementation-defined extended source
>        character or universal character name.  If a universal
>        character name is continued by a backslash-newline across a
>        line boundary, the mapped output sequence contains the same
>        number of backslash-newlines as the the input, but their
>        location in the output sequence is unspecified.
> 
> Would this weaker action pose an undue burden on GCC?  My sense from
> the discussion is "no", but I'd like to double-check with the experts.

I think this is no problem for GCC, at least as far as UCNs go.


I am less sure about the things that this is missing.  For instance,
it doesn't really define 'extended source character' in POSIX terms.
The POSIXy way to do that would be to refer to the LC_CHARSET
environment variable, but then consider

LC_CHARSET=UTF-16 c99 foo.c

where 'foo.c' is in UTF-16 and contains '#include <stdio.h>', which is
in /usr/include but is in some other character set.  So, there's some
complication here that I'm not sure is completely solved, or at least
that I don't fully understand.  But, I'm pretty sure GCC would be just
as hurt by that as any other compiler, so at least there are no
GCC-specific problems.

Re: proposed Opengroup action for c99 command (XCU ERN 76)

[Kai Henningsen]

joseph@codesourcery.com (Joseph S. Myers)  wrote on 16.09.05 in <Pine.LNX.4.61.0509161151570.8721@digraph.polyomino.org.uk>:

> C++ requires (A) and provides examples of valid programs where it can be
> told whether a normalisation of UCNs is part of the implementation-defined
> phase 1 transformation.  As I gave in a previous discussion,
>
> #include <stdlib.h>
> #include <string.h>
> #define h(s) #s
> #define str(s) h(s)
> int
> main()
> {
>   if (strcmp(str(str(\u00c1)), "\"\\u00c1\"")) abort ();
>   if (strcmp(str(str(\u00C1)), "\"\\u00C1\"")) abort ();
> }

Incidentally, gcc 3.3.5 passes this test.

However, I'm far from convinced that this is reasonable for a compiler  
supporting UCNs. It seems to me they really ought to be handled more like  
trigraphs, so that this would be

  if (strcmp(str(str(\u00C1)), "\"\u00C1\"")) abort ();

(note one less backslash), and the case of the C would be irrelevant - in  
fact, *anything* that makes it relevant feels like a bug to me.

I think comparing \u00C1 to \xC1 is a false friend here - \xC1 is only  
valid inside a string. Instead look at identifiers.

If str(str(ab\00C1cd)) == "\"ab\\00C1cd\"", then I'll certainly be  
unpleasantly surprised - that is almost certainly not what I wanted.

MfG Kai

Re: proposed Opengroup action for c99 command (XCU ERN 76)

[Ross Ridge]

Paul Eggert writes:
> Would this weaker action pose an undue burden on GCC?  My sense from
> the discussion is "no", but I'd like to double-check with the experts

I'd say "no", but I think the experts might see it as posing no burden
at all on GCC.  The burden would be on whomever wants to use GCC to
implement the c99 command.  

						Ross Ridge

Re: proposed Opengroup action for c99 command (XCU ERN 76)

[Gabriel Dos Reis]

"Zack Weinberg" <zackw@panix.com> writes:

[...]

| Also: why do you care so much about this corner case?  I only care from the
| implementation perspective, since I doubt it matters to any real software that
| GCC might compile.  I'm pointing out an approach to the problem which would
| avoid having to change GCC at all, saving someone a great deal of coding
| effort.  You are arguing strongly for not changing this one tiny piece of the
| standard, and I don't understand why.

I think standards should be changed for valid or convincing reasons.  So
far, I've not seen any reason that convinced me that it should be
changed.  I've heard that it is a bug in the standard, but for reasons
that it did not fit GCC current implementation or "diverged" from something
that came after.  It is not like people did not want to contribute
patch to align GCC behaviour on standard semantics.  Going through the
audit trail, the impression I've gotten is you're blocking patches on
the grounds that I have trouble finding words to describe.

-- Gaby

Re: proposed Opengroup action for c99 command (XCU ERN 76)

[Zack Weinberg]

Gabriel Dos Reis said:
[...]
> Good.  It seems to me like those who would be spending a great deal of
> time and money are not sufficiently convinced by your arguments.
> Consequently, it appears that they are not in position to explain your
> strong opinion to the committees -- personally, I'm not convinced
> enough to take committee scare resource (e.g. time) to explain them
> that ZW strongly believes that the C++ spec has bug.  If you do have
> people who would defend your position; please let us know.  Otherwise,
> it would seem like you're agitating that argument for no useful purpose.

Based on their statements elsewhere in this thread, I believe Joe Buck and
Geoff Keating share my opinions.  I am so far the only person to have said out
loud that this should be addressed by changing the C++ standard, but I don't
think I am the only one to consider it.

Also: why do you care so much about this corner case?  I only care from the
implementation perspective, since I doubt it matters to any real software that
GCC might compile.  I'm pointing out an approach to the problem which would
avoid having to change GCC at all, saving someone a great deal of coding
effort.  You are arguing strongly for not changing this one tiny piece of the
standard, and I don't understand why.

zw

Re: proposed Opengroup action for c99 command (XCU ERN 76)

[Gabriel Dos Reis]

Mike Stump <mrs@apple.com> writes:

| On Sep 17, 2005, at 3:33 PM, Gabriel Dos Reis wrote:
| > C++98 came before C99, so who diverged from whom?
| 
| You seem to not not how the C++ standard was made.

Thank you very much.

| In fact, it come before C99, like it or not.

Oh really?  What did I say?

| The intention was that C++ would come up
| with a follow on standard that tracked C99, in next revision, as the
| 98 revision just didn't allow for it time wise.

You seem not to have been following development on C++ since late Nov
1997. 

| It is entirely reasonable to pressfit those aspects of C99 that makes
| sense in the confines of the C++ standard, into C++. 

Please be my guest.

| long long is my favorite example of this.

I closely followed all the revisions to the paper that finally got
"long long" accepted for C++0x at the last meeting -- which, BTW, I 
voted for (and encouraged my National Body to do so).

| We wanted to do it, but, the C99 standard
| wasn't done enough for our tastes.

Aha?

-- Gaby

Re: proposed Opengroup action for c99 command (XCU ERN 76)

[Mike Stump]

On Sep 17, 2005, at 3:33 PM, Gabriel Dos Reis wrote:
> C++98 came before C99, so who diverged from whom?

You seem to not not how the C++ standard was made.  In fact, it come  
before C99, like it or not.  The intention was that C++ would come up  
with a follow on standard that tracked C99, in next revision, as the  
98 revision just didn't allow for it time wise.

It is entirely reasonable to pressfit those aspects of C99 that makes  
sense in the confines of the C++ standard, into C++.  long long is my  
favorite example of this.  We wanted to do it, but, the C99 standard  
wasn't done enough for our tastes.

Re: proposed Opengroup action for c99 command (XCU ERN 76)

[Gabriel Dos Reis]

"Zack Weinberg" <zackw@panix.com> writes:

| Gabriel Dos Reis said:
| > "Zack Weinberg" <zackw@panix.com> writes:
| >
| > | Gabriel Dos Reis said:
| > | > C++98 came before C99, so who diverged from whom?
| > |
| > | It doesn't matter.
| >
| > Yet, you're you were construeing it as an argument to support your position.
| 
| I'm only bringing up the divergence as an argument that at least one of the
| standards should be changed.

except that it was at best misleading.

| 
| Now my position _is_ that C++ should change, and for rationale look no further
| than Geoff's observation that implementing (A) or (B) rather than (C)
| semantics
| serves primarily to increase the set of programs that are ill-formed.  Since I
| don't think anyone but a conformance-test author would ever code something like
| Joseph's example, I think our users are best served by sticking to (C) and making
| the standards match.
| 
| > | > If you do feel so strongly about this, why don't you invest time in
| > | > sorting this with the committees?
| > |
| > | I am not in a position to do that.  Others on this list are.
| >
| > Please be more explicit.
| 
| Participating in the standards committees takes a great deal of time and
| money (for travel), neither of which I have right now.  I don't care to
| discuss my precise circumstances.

Good.  It seems to me like those who would be spending a great deal of
time and money are not sufficiently convinced by your arguments.
Consequently, it appears that they are not in position to explain your
strong opinion to the committees -- personally, I'm not convinced
enough to take committee scare resource (e.g. time) to explain them
that ZW strongly believes that the C++ spec has bug.  If you do have
people who would defend your position; please let us know.  Otherwise,
it would seem like you're agitating that argument for no useful purpose.

-- Gaby

Re: proposed Opengroup action for c99 command (XCU ERN 76)

[Zack Weinberg]

Gabriel Dos Reis said:
> "Zack Weinberg" <zackw@panix.com> writes:
>
> | Gabriel Dos Reis said:
> | > C++98 came before C99, so who diverged from whom?
> |
> | It doesn't matter.
>
> Yet, you're you were construeing it as an argument to support your position.

I'm only bringing up the divergence as an argument that at least one of the
standards should be changed.

Now my position _is_ that C++ should change, and for rationale look no further
than Geoff's observation that implementing (A) or (B) rather than (C)
semantics
serves primarily to increase the set of programs that are ill-formed.  Since I
don't think anyone but a conformance-test author would ever code something like
Joseph's example, I think our users are best served by sticking to (C) and making
the standards match.

> | > If you do feel so strongly about this, why don't you invest time in
> | > sorting this with the committees?
> |
> | I am not in a position to do that.  Others on this list are.
>
> Please be more explicit.

Participating in the standards committees takes a great deal of time and
money (for travel), neither of which I have right now.  I don't care to
discuss my precise circumstances.

zw

Re: proposed Opengroup action for c99 command (XCU ERN 76)

[Gabriel Dos Reis]

"Zack Weinberg" <zackw@panix.com> writes:

| Gabriel Dos Reis said:
| > "Zack Weinberg" <zackw@panix.com> writes:
| > | When the standard is arguably buggy -- if nothing else, it diverges from C
| >
| > C++98 came before C99, so who diverged from whom?
| 
| It doesn't matter.

Yet, you're you were construeing it as an argument to support your position.

[...]

| > If you do feel so strongly about this, why don't you invest time in
| > sorting this with the committees?
| 
| I am not in a position to do that.  Others on this list are.

Please be more explicit.

-- Gaby

Re: proposed Opengroup action for c99 command (XCU ERN 76)

[Zack Weinberg]

Gabriel Dos Reis said:
> "Zack Weinberg" <zackw@panix.com> writes:
> | When the standard is arguably buggy -- if nothing else, it diverges from C
>
> C++98 came before C99, so who diverged from whom?

It doesn't matter.  The divergence should be resolved in favor of whichever
standard has it right, not whichever came first.  I happen to think that the
C standard, not the C++ standard, has it right, so I argue as I do.

> If you do feel so strongly about this, why don't you invest time in
> sorting this with the committees?

I am not in a position to do that.  Others on this list are.

zw

Re: proposed Opengroup action for c99 command (XCU ERN 76)

[Gabriel Dos Reis]

"Zack Weinberg" <zackw@panix.com> writes:

| Joseph S. Myers said:
| > On Fri, 16 Sep 2005, Zack Weinberg wrote:
| >> I am with Joe Buck in the opinion that even a 1% performance penalty for
| >> implementing (A) [or (B)] would be too much -- I suggest this be fixed by
| >> convincing the C++ committee to allow (C) and not just by phase 1
| >> transformations, thus allowing the existing implementation to conform.
| >
| > I don't think solutions starting with convincing the committee to fix a
| > working part of the standard are generally that practical!
| 
| When the standard is arguably buggy -- if nothing else, it diverges from C

C++98 came before C99, so who diverged from whom?
If you do feel so strongly about this, why don't you invest time in
sorting this with the committees?

-- Gaby

Re: proposed Opengroup action for c99 command (XCU ERN 76)

[Zack Weinberg]

Joseph S. Myers said:
> On Fri, 16 Sep 2005, Zack Weinberg wrote:
>> I am with Joe Buck in the opinion that even a 1% performance penalty for
>> implementing (A) [or (B)] would be too much -- I suggest this be fixed by
>> convincing the C++ committee to allow (C) and not just by phase 1
>> transformations, thus allowing the existing implementation to conform.
>
> I don't think solutions starting with convincing the committee to fix a
> working part of the standard are generally that practical!

When the standard is arguably buggy -- if nothing else, it diverges from C
and there is no good reason for divergence -- I think working with the committee
to improve the standard should always be an option considered.

When a simple change to the standard would get you out of having to make
a large, complicated change to GCC, and when the standard-mandated semantics
are of dubious utility to real users, I think you should try to change the
standard before you even start thinking about changing the compiler.

I don't disagree strongly with any of your other points, but I do want to
mention that foreclosing future improvements in another domain is something
that should be
considered when making implementation choices, that proper statistical methods
*can* detect 1% performance degradations amid the noise, and that -- as
pointed out by others -- a pile of 1% degradations adds up to a slow compiler.

zw

Re: proposed Opengroup action for c99 command (XCU ERN 76)

[Joseph S. Myers]

On Fri, 16 Sep 2005, Zack Weinberg wrote:

> Especially for C++ which constructs a cpp_token array (sort of) representing
> the entire translation unit, it is desirable to make cpp_token *smaller* --
> and it would be relatively easy to win back that 'whole word not used in the
> case of identifiers', so I do not like a solution which starts using that word
> for identifiers.  Note that identifiers and punctuators are vastly more common
> than numbers or strings, in all C-family languages.

As a general principle I'm doubtful of blocking fixes on the basis that 
some other hypothetical future fix conflicts.  If UCN handling is refined 
first then subsequent changes to cpp_token should take account of that and 
avoid regressing; if cpp_token is reduced in size first then subsequent 
UCN handling changes should avoid losing significant performance 
improvements from such a reduction.

Additional information is only relevant for preprocessing tokens before 
they are converted to tokens, so in principle the front end could use a 
smaller version of the structure if necessary.

> I am with Joe Buck in the opinion that even a 1% performance penalty for
> implementing (A) [or (B)] would be too much -- I suggest this be fixed by
> convincing the C++ committee to allow (C) and not just by phase 1
> transformations, thus allowing the existing implementation to conform.

I don't think solutions starting with convincing the committee to fix a 
working part of the standard are generally that practical!

My experience is that 1% compile-time performance differences are in 
practice not distinguishable from noise; even if times are reliably 
measured with under 1% variation they still have a greater than 1% 
dependence on random factors and the 1% difference may no longer show 
given a trivially different compiler or test data.

-- 
Joseph S. Myers               http://www.srcf.ucam.org/~jsm28/gcc/
    jsm@polyomino.org.uk (personal mail)
    joseph@codesourcery.com (CodeSourcery mail)
    jsm28@gcc.gnu.org (Bugzilla assignments and CCs)

Re: proposed Opengroup action for c99 command (XCU ERN 76)

[Joseph S. Myers]

On Fri, 16 Sep 2005, Ross Ridge wrote:

> Joe Buck writes:
> > To me, even a 1% performance hit to fix this would be excessive.
> 
> I think any performance hit to support UCNs or extended characters outside
> of strings and comments is undesirable.  GCC should have an option like
> "-trigraphs" for the few programs that need this feature.

I have indeed just submitted a patch to add such an option, 
-fextended-identifiers.  This is however in recognition of the 
*implementation* being experimental rather than an intention permanently 
to exclude it from the -std options (including -std=gnu99) which should in 
principle enable this feature.

Naturally the idea would be to avoid performance cost *in the normal case 
where extended identifiers are not used* from extended identifiers, with 
all complications from such identifiers being a slow case, just as lexing

a??/U??/
??/
0??/
000000??/
??/
z

(as the three tokens a, \, U0000000z, because the UCN syntax is not met) 
is a slow and rare case (which GCC correctly implements).

I do not however see any problem with a 1% (or 10%) slowdown for programs 
*where a significant proportion of identifiers are extended identifiers* 
in order to refine the handling of such identifiers.

-- 
Joseph S. Myers               http://www.srcf.ucam.org/~jsm28/gcc/
    jsm@polyomino.org.uk (personal mail)
    joseph@codesourcery.com (CodeSourcery mail)
    jsm28@gcc.gnu.org (Bugzilla assignments and CCs)

Re: proposed Opengroup action for c99 command (XCU ERN 76)

[Paul Eggert]

Thanks, everybody, for writing about this.

The standardization process is one of consensus, and if the GCC
developers find some areas of disagreement here I think it unlikely
that the other POSIX implementers will agree with the proposed action.
Hence I am thinking of weakening it.

Currently the action proposes to insert the following text:

   It is implementation-defined whether trailing white-space characters
   in each C-language source line are ignored.  Otherwise, the
   multibyte characters of each source line are mapped on a one-to-one
   basis to the C source character set.

How about if I propose to insert the following text instead?

   The multibyte characters of each source file are mapped to the C
   source character set on a one-to-one basis, with the following
   exceptions:

     * It is implementation-defined whether trailing white-space
       characters in each input line are ignored.

     * Each extended source character, and each sequence of characters
       that would otherwise be recognized as a universal character
       name, is mapped to an implementation-defined extended source
       character or universal character name.  If a universal
       character name is continued by a backslash-newline across a
       line boundary, the mapped output sequence contains the same
       number of backslash-newlines as the the input, but their
       location in the output sequence is unspecified.

Would this weaker action pose an undue burden on GCC?  My sense from
the discussion is "no", but I'd like to double-check with the experts.

Thanks again.

Re: proposed Opengroup action for c99 command (XCU ERN 76)

[Ross Ridge]

Joe Buck writes:
> To me, even a 1% performance hit to fix this would be excessive.

I think any performance hit to support UCNs or extended characters outside
of strings and comments is undesirable.  GCC should have an option like
"-trigraphs" for the few programs that need this feature.

						Ross Ridge

Re: proposed Opengroup action for c99 command (XCU ERN 76)

[Gabriel Dos Reis]

Joe Buck <Joe.Buck@synopsys.COM> writes:

| On Fri, Sep 16, 2005 at 08:58:12PM +0200, Gabriel Dos Reis wrote:
| > Joe Buck <Joe.Buck@synopsys.COM> writes:
| > 
| > | To me, even a 1% performance hit to fix this would be excessive.
| > 
| > My opinion is that is an excessive statement.
| 
| Really, Gaby?  Consider that there are hundreds of other open bugs.
| Would you accept a 1% hit to fix each?

That wasn't my statement.  E.g. I did not say that it was OK for
*every* bug-fix to introduce a 1% preformance loss.  That also is very
different from saying that no fix should introduce a 1% loss.

-- Gaby

Re: proposed Opengroup action for c99 command (XCU ERN 76)

[Joe Buck]

On Fri, Sep 16, 2005 at 08:58:12PM +0200, Gabriel Dos Reis wrote:
> Joe Buck <Joe.Buck@synopsys.COM> writes:
> 
> | To me, even a 1% performance hit to fix this would be excessive.
> 
> My opinion is that is an excessive statement.

Really, Gaby?  Consider that there are hundreds of other open bugs.
Would you accept a 1% hit to fix each?  GCC is slower than it used
to be because of dozens and dozens of 1-2% performance regressions.

Why is this problem so important that a 1% speed regression for *all*
programs would be acceptable?

Re: proposed Opengroup action for c99 command (XCU ERN 76)

[Zack Weinberg]

[I apologize for breaking the thread; I am currently stuck using a web-mail
client that does not permit manual insertion of References: headers.  Please
don't take this comment as a sign that I am resuming participation in GCC
development in general.]

Joseph Myers:
> There are plenty of spare bits in cpp_token to flag extended identifiers
> and handle them specially (as a slow path, marked as such with
> __builtin_expect).  There's one bit in the flags byte, two unused bytes
> after it and a whole word not used in the case of identifiers (identifiers
> use a cpp_hashnode * where strings and numbers use a struct cpp_string
> which is bigger)

Especially for C++ which constructs a cpp_token array (sort of) representing
the entire translation unit, it is desirable to make cpp_token *smaller* --
and it would be relatively easy to win back that 'whole word not used in the
case of identifiers', so I do not like a solution which starts using that word
for identifiers.  Note that identifiers and punctuators are vastly more common
than numbers or strings, in all C-family languages.

Geoff Keating:
> > Adding salt to the wound, of course, is that for C the only difference
> > between an (A) or (B) and a (C) implementation is that a (C)
> > implementation is less expressive: there are some programs, all of
> > which are erroneous and require a diagnostic, that can't be written.
> > So you lose compiler performance just so users have another bullet
> > to shoot their feet with.

Joseph:
> C++ requires (A) and provides examples of valid programs where it can be
> told whether a normalisation of UCNs is part of the implementation-defined
> phase 1 transformation.

I am with Joe Buck in the opinion that even a 1% performance penalty for
implementing (A) [or (B)] would be too much -- I suggest this be fixed by
convincing the C++ committee to allow (C) and not just by phase 1
transformations, thus allowing the existing implementation to conform.

zw

Re: proposed Opengroup action for c99 command (XCU ERN 76)

[Gabriel Dos Reis]

Joe Buck <Joe.Buck@synopsys.COM> writes:

| To me, even a 1% performance hit to fix this would be excessive.

My opinion is that is an excessive statement.

-- Gaby

Re: proposed Opengroup action for c99 command (XCU ERN 76)

[Geoff Keating]

[-- Attachment #1: Type: text/plain, Size: 2848 bytes --]


On 16/09/2005, at 5:12 AM, Joseph S. Myers wrote:

> On Fri, 16 Sep 2005, Geoffrey Keating wrote:
>
>
>> What this means in practise, I think, is that the structure that
>> represents a token, 'struct cpp_token' will grow from 16 bytes to 20
>> bytes, which makes it 2 cache lines rather than 1, and a subsequent
>> memory use increase and compiler performance decrease.  It might be
>> that someone will think of some clever way to avoid this, but I
>> couldn't think of any that would be likely to be a win overall, since
>> a significant proportion of tokens are identifiers.  (I especially
>> didn't like the alternative that required a second hash lookup for
>> every identifier.)
>>
>
> There are plenty of spare bits in cpp_token to flag extended  
> identifiers
> and handle them specially (as a slow path, marked as such with
> __builtin_expect).  There's one bit in the flags byte, two unused  
> bytes
> after it and a whole word not used in the case of identifiers  
> (identifiers
> use a cpp_hashnode * where strings and numbers use a struct cpp_string
> which is bigger) which could store a canonical form of an  
> identifier (or
> could store the noncanonical spelling for the use of the specific  
> places
> which care about the original spelling).

Yes, I think this can be made to work efficiently.

>> Adding salt to the wound, of course, is that for C the only  
>> difference
>> between an (A) or (B) and a (C) implementation is that a (C)
>> implementation is less expressive: there are some programs, all of
>> which are erroneous and require a diagnostic, that can't be written.
>> So you lose compiler performance just so users have another bullet
>> to shoot their feet with.
>>
>
> C++ requires (A)

This is true, but only in the sense that C requires (B).  Either  
language can be supported by any of the three implementations with an  
appropriate phase 1 rule.

> Implementation of (A) could start by a (slow path, if there are  
> extended
> characters present) conversion of the whole input to UCNs, or a more
> efficient conversion that avoids the need to convert within comments.

Although UCNs would be the most convenient form for the preprocessor,  
the backend would like strings to be in UTF-8, to avoid the need for  
conversion when outputting names to the assembler.

> But if any normalisation of UCNs is documented for C++ it does need  
> to be
> documented in the form of transforming UCNs to other UCNs (not to  
> UTF-8).

Yes; but this is not a difficult problem.  For C++, you would just  
say (following my proposed wording) that after they're converted to  
UTF-8, they are converted back to some canonical form of UCN ('the  
version with the most lower-case characters', for instance).  Then,  
when stringifying, you would convert UTF-8 characters in identifiers  
to that canonical UCN.

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 2410 bytes --]

Re: proposed Opengroup action for c99 command (XCU ERN 76)

[Joe Buck]

On Fri, 16 Sep 2005, Geoffrey Keating wrote:
> > Adding salt to the wound, of course, is that for C the only difference
> > between an (A) or (B) and a (C) implementation is that a (C)
> > implementation is less expressive: there are some programs, all of
> > which are erroneous and require a diagnostic, that can't be written.
> > So you lose compiler performance just so users have another bullet
> > to shoot their feet with.

On Fri, Sep 16, 2005 at 12:12:34PM +0000, Joseph S. Myers wrote:
> C++ requires (A) and provides examples of valid programs where it can be 
> told whether a normalisation of UCNs is part of the implementation-defined 
> phase 1 transformation.  As I gave in a previous discussion,
> 
> #include <stdlib.h>
> #include <string.h>
> #define h(s) #s
> #define str(s) h(s)
> int
> main()
> {
>   if (strcmp(str(str(\u00c1)), "\"\\u00c1\"")) abort ();
>   if (strcmp(str(str(\u00C1)), "\"\\u00C1\"")) abort ();
> }

OK, Joseph, you've given us a valid program where normalization of UCNs
makes a difference.

But the question on my mind is, is it worth the effort to get this right?
Rigorous standards compliance is a good target to aim for.  But if there
is a cost in compiler performance, every user has to pay, for every
program, just so that weird cases like the one you describe come out
correctly.

To me, even a 1% performance hit to fix this would be excessive.  We're
in a constant losing battle to keep each gcc release from being slower
than the last one; sometimes we gain a bit of ground but more often we
lose ground.

Re: proposed Opengroup action for c99 command (XCU ERN 76)

[Ross Ridge]

Ross Ridge wrote:
> Thinking semantically is irrelevent because the question isn't whether GCC
> conforms to C99 or POSIX.  It clearly doesn't.  GCC fails the as-if rule.
> The question is one of implementation burden, which can only be answered
> by examining GCC's implementation.
 
Robert Dewar wrote:
> Once again we are not discussing general conformance here, just 
> conformance on this specific issue.

Wrong.  The question that was asked and I answered was about
implementation burden, not conformance.

						Ross Ridge

-- 
 l/  //	  Ross Ridge -- The Great HTMU
[oo][oo]  rridge@csclub.uwaterloo.ca
-()-/()/  http://www.csclub.uwaterloo.ca/u/rridge/ 
 db  //	  

Re: proposed Opengroup action for c99 command (XCU ERN 76)

[Joseph S. Myers]

On Fri, 16 Sep 2005, Geoffrey Keating wrote:

> What this means in practise, I think, is that the structure that
> represents a token, 'struct cpp_token' will grow from 16 bytes to 20
> bytes, which makes it 2 cache lines rather than 1, and a subsequent
> memory use increase and compiler performance decrease.  It might be
> that someone will think of some clever way to avoid this, but I
> couldn't think of any that would be likely to be a win overall, since
> a significant proportion of tokens are identifiers.  (I especially
> didn't like the alternative that required a second hash lookup for
> every identifier.)

There are plenty of spare bits in cpp_token to flag extended identifiers 
and handle them specially (as a slow path, marked as such with 
__builtin_expect).  There's one bit in the flags byte, two unused bytes 
after it and a whole word not used in the case of identifiers (identifiers 
use a cpp_hashnode * where strings and numbers use a struct cpp_string 
which is bigger) which could store a canonical form of an identifier (or 
could store the noncanonical spelling for the use of the specific places 
which care about the original spelling).

> Adding salt to the wound, of course, is that for C the only difference
> between an (A) or (B) and a (C) implementation is that a (C)
> implementation is less expressive: there are some programs, all of
> which are erroneous and require a diagnostic, that can't be written.
> So you lose compiler performance just so users have another bullet
> to shoot their feet with.

C++ requires (A) and provides examples of valid programs where it can be 
told whether a normalisation of UCNs is part of the implementation-defined 
phase 1 transformation.  As I gave in a previous discussion,

#include <stdlib.h>
#include <string.h>
#define h(s) #s
#define str(s) h(s)
int
main()
{
  if (strcmp(str(str(\u00c1)), "\"\\u00c1\"")) abort ();
  if (strcmp(str(str(\u00C1)), "\"\\u00C1\"")) abort ();
}

Implementation of (A) could start by a (slow path, if there are extended 
characters present) conversion of the whole input to UCNs, or a more 
efficient conversion that avoids the need to convert within comments.  
But if any normalisation of UCNs is documented for C++ it does need to be 
documented in the form of transforming UCNs to other UCNs (not to UTF-8).

-- 
Joseph S. Myers               http://www.srcf.ucam.org/~jsm28/gcc/
    jsm@polyomino.org.uk (personal mail)
    joseph@codesourcery.com (CodeSourcery mail)
    jsm28@gcc.gnu.org (Bugzilla assignments and CCs)

Re: proposed Opengroup action for c99 command (XCU ERN 76)

[Geoffrey Keating]

Paul Eggert <eggert@CS.UCLA.EDU> writes:

Hi Paul,

> I proposed to insert the following paragraph after XCU page 213 line
> 8366 (i.e, at the end of the INPUT FILES section of the c99 spec
> <http://www.opengroup.org/onlinepubs/009695399/utilities/c99.html>):
> 
>    It is implementation-defined whether trailing white-space characters
>    in each C-language source line are ignored.  Otherwise, the
>    multibyte characters of each source line are mapped on a one-to-one
>    basis to the C source character set.
> 
> In response Joseph S. Myers pointed out that this action would require
> c99 to use interpretation B of section 5.2.1 (page 20) of the C99 Rationale
> <http://www.open-std.org/jtc1/sc22/wg14/www/C99RationaleV5.10.pdf>.
> The Rationale says C preprocessors can be implemented in three ways:
> 
>   A.  Convert everything to UCNs in basic source characters as soon
>       as possible, that is, in translation phase 1.  (This is what
>       C++ requires, apparently.)
> 
>   B.  Use native encodings where possible, UCNs otherwise.
> 
>   C.  Convert everything to wide characters as soon as possible
>       using an internal encoding that encompasses the entire source
>       character set and all UCNs.
> 
> The C99 standardizers chose (B), but said implementations could also
> use (A) or (C) because the C99 standard gives almost unlimited freedom
> in translation phase 1 for compilers to do whatever transformations
> they like.
> 
> However, the proposed action for the c99 command would close this
> escape hatch, forcing interpretation (B) for c99 implementations.
> 
> So my question is: Is it a burden on GCC to require interpretation (B)?
> 
> My understanding is that GCC already uses (B), and that the answer is
> "no, it's no problem", but if I'm wrong please let me know.

I believe that GCC currently uses (C), from a language standpoint.

Following the discussion in
<http://gcc.gnu.org/bugzilla/show_bug.cgi?id=9449>, you can tell the
difference between an (A) or (B) and a (C) implementation by writing:

#define foo \u00c1
#define foo \u00C1
int x;

in a text file and seeing if the compiler accepts it.  If the compiler
accepts it, it's a type (C) implementation.  GCC presently accepts
this testcase.

The actual internal implementation does not exactly follow any of
these models.  (For instance, it really uses UTF-8 rather than wide
characters; and strings aren't handled in quite the same way as
identifiers.)

There is a difference in terms of implementation cost.  A type (C)
implementation can immediately unify all identifiers that refer to the
same name, even if they are spelt differently, while an (A) or (B)
implementation must initially distinguish identifiers that are the
same identifier but spelt differently, and will have to keep
distinguishing them until the end of preprocessing.  Yet, it will
eventually need to unify them so that the rest of the compiler doesn't
have to do string comparisons, and when determining whether a token
has a macro replacement it must treat same-but-different identifiers
as the same.

What this means in practise, I think, is that the structure that
represents a token, 'struct cpp_token' will grow from 16 bytes to 20
bytes, which makes it 2 cache lines rather than 1, and a subsequent
memory use increase and compiler performance decrease.  It might be
that someone will think of some clever way to avoid this, but I
couldn't think of any that would be likely to be a win overall, since
a significant proportion of tokens are identifiers.  (I especially
didn't like the alternative that required a second hash lookup for
every identifier.)

Adding salt to the wound, of course, is that for C the only difference
between an (A) or (B) and a (C) implementation is that a (C)
implementation is less expressive: there are some programs, all of
which are erroneous and require a diagnostic, that can't be written.
So you lose compiler performance just so users have another bullet
to shoot their feet with.

Re: proposed Opengroup action for c99 command (XCU ERN 76)

[Robert Dewar]

Ross Ridge wrote:

> Thinking semantically is irrelevent because the question isn't whether GCC
> conforms to C99 or POSIX.  It clearly doesn't.  GCC fails the as-if rule.
> The question is one of implementation burden, which can only be answered
> by examining GCC's implementation.

Once again we are not discussing general conformance here, just 
conformance on this specific issue.

Re: proposed Opengroup action for c99 command (XCU ERN 76)

[Ross Ridge]

> You are thinking operationally, when you should think semantically.
> Remember that as-if applies here. The rules as stated give ways to
> achieve certain effects, the question is not whether we are following
> the operational rules, but whether we are following the effects.
 
Thinking semantically is irrelevent because the question isn't whether GCC
conforms to C99 or POSIX.  It clearly doesn't.  GCC fails the as-if rule.
The question is one of implementation burden, which can only be answered
by examining GCC's implementation.

> Indeed I am not sure I understand that the three options are in fact
> distinct semantically.

The aren't in C99, as Paul Eggert's original message made clear, but
they are in an environment that defines a command like "c99" that makes
preprocessed output visable.

						Ross Ridge

-- 
 l/  //	  Ross Ridge -- The Great HTMU
[oo][oo]  rridge@csclub.uwaterloo.ca
-()-/()/  http://www.csclub.uwaterloo.ca/u/rridge/ 
 db  //	  

Re: proposed Opengroup action for c99 command (XCU ERN 76)

[Joe Buck]

Ross Ridge wrote:
> >Well, maybe I'm missing something, but it never converts input characters
> >to UCNs so that means it doesn't do (A) or (B), and the only thing it
> >converts to wide characters are wide string literals, so it doesn't do
> >(C).

On Thu, Sep 15, 2005 at 07:24:47AM -0400, Robert Dewar wrote:
> You are thinking operationally, when you should think semantically.
> Remember that as-if applies here. The rules as stated give ways to
> achieve certain effects, the question is not whether we are following
> the operational rules, but whether we are following the effects.

This was exactly my point when I asked for an example.  Joseph Myers
pointed to some, in the discussion for PR 9449 (particularly comment #39).

The examples, however, are obscure enough that users might prefer having
a few corner-case bugs to a rigorously correct, but slower,
implementation.  For example, if the cost of correctly handling

#define foo \u00b1
#define foo \u00B1   /* invalid: different spelling in replacement */
#define \u00c1 foo
#define \u00C1 foo   /* valid: spelling of macro *name* can vary */

is a 10% slowdown in preprocessor speed, I'd rather leave it broken
and document the breakage.  On the other hand, if a correct implementation
of these rules does not impose a speed penalty, wonderful.

It is truly annoying that the C and C++ committees created different
rules, BTW.

Re: proposed Opengroup action for c99 command (XCU ERN 76)

[Joseph S. Myers]

On Wed, 14 Sep 2005, Paul Eggert wrote:

> So my question is: Is it a burden on GCC to require interpretation (B)?

Yes, but my position remains that we *should* aim to implement (B) and 
this is a proper burden to impose.

For how the interpretations can be distinguished, see bug 9449 comments 21 
et seq (especially the examples in comment 39).

-- 
Joseph S. Myers               http://www.srcf.ucam.org/~jsm28/gcc/
    jsm@polyomino.org.uk (personal mail)
    joseph@codesourcery.com (CodeSourcery mail)
    jsm28@gcc.gnu.org (Bugzilla assignments and CCs)

Re: proposed Opengroup action for c99 command (XCU ERN 76)

[Robert Dewar]

Ross Ridge wrote:

> Well, maybe I'm missing something, but it never converts input characters
> to UCNs so that means it doesn't do (A) or (B), and the only thing it
> converts to wide characters are wide string literals, so it doesn't do
> (C).

You are thinking operationally, when you should think semantically.
Remember that as-if applies here. The rules as stated give ways to
achieve certain effects, the question is not whether we are following
the operational rules, but whether we are following the effects.

Indeed I am not sure I understand that the three options are in fact
distinct semantically. Can someone who understands this better address
this point, and explain what the observable semantic effects of the
three approaches are and how they differ.

Re: proposed Opengroup action for c99 command (XCU ERN 76)

[Ross Ridge]

> >   A.  Convert everything to UCNs in basic source characters as soon
> >       as possible, that is, in translation phase 1.  (This is what
> >       C++ requires, apparently.)
> > 
> >   B.  Use native encodings where possible, UCNs otherwise.
> > 
> >   C.  Convert everything to wide characters as soon as possible
> >       using an internal encoding that encompasses the entire source
> >       character set and all UCNs.
> 
> Now, see libcpp/charset.c.  See the -finput-charset= option.  To me
> that looks like code which does something related to (A), (B), or (C).

Well, maybe I'm missing something, but it never converts input characters
to UCNs so that means it doesn't do (A) or (B), and the only thing it
converts to wide characters are wide string literals, so it doesn't do
(C).

Hmm... maybe it's doing (B) if UCNs aren't ever necessary, though it
doesn't seem to support extended characters in indentifiers so it's not
quite using native encodings where possible.  Though the intent does
seem to be to go with option (B).

						Ross Ridge

Re: proposed Opengroup action for c99 command (XCU ERN 76)

[Eric Christopher]

>
> I am the opposite of an expert on this topic.  But in fact gcc does
> appear to have code related to (A), (B), and (C).  I repeat those
> choices here from Paul's original e-mail:
>
>
>>   A.  Convert everything to UCNs in basic source characters as soon
>>       as possible, that is, in translation phase 1.  (This is what
>>       C++ requires, apparently.)
>>
>>   B.  Use native encodings where possible, UCNs otherwise.
>>
>>   C.  Convert everything to wide characters as soon as possible
>>       using an internal encoding that encompasses the entire source
>>       character set and all UCNs.
>>
>
> Now, see libcpp/charset.c.  See the -finput-charset= option.  To me
> that looks like code which does something related to (A), (B), or (C).

It does. I think the best bet would be (A) for the code that we have in
libcpp at the moment. Right now we translate upon getting characters
into an intermediate format that does encompass as much as possible
(IIRC). That, and it'd make sure that we handle what c++ requires.

I'm also not as much of an expert as I'd have liked to be when dealing
with this in the first place.

-eric

Re: proposed Opengroup action for c99 command (XCU ERN 76)

[Ian Lance Taylor]

rridge@csclub.uwaterloo.ca (Ross Ridge) writes:

> > I was not asking the general question, I was asking how it fails
> > to conform wrt the particular technical issue at hand.
> 
> Since GCC doesn't have any code that does (A), (B), or (C) it doesn't
> place a burden on GCC to require it to do (B).  That's sufficient to
> answer the techinical issue at hand.  While that implies GCC doesn't
> conform, I said so explictly because Paul Eggert said that c99 is often
> implemented using GCC.

I am the opposite of an expert on this topic.  But in fact gcc does
appear to have code related to (A), (B), and (C).  I repeat those
choices here from Paul's original e-mail:

>   A.  Convert everything to UCNs in basic source characters as soon
>       as possible, that is, in translation phase 1.  (This is what
>       C++ requires, apparently.)
> 
>   B.  Use native encodings where possible, UCNs otherwise.
> 
>   C.  Convert everything to wide characters as soon as possible
>       using an internal encoding that encompasses the entire source
>       character set and all UCNs.

Now, see libcpp/charset.c.  See the -finput-charset= option.  To me
that looks like code which does something related to (A), (B), or (C).

Ian

Re: proposed Opengroup action for c99 command (XCU ERN 76)

[Ross Ridge]

> I was not asking the general question, I was asking how it fails
> to conform wrt the particular technical issue at hand.

Since GCC doesn't have any code that does (A), (B), or (C) it doesn't
place a burden on GCC to require it to do (B).  That's sufficient to
answer the techinical issue at hand.  While that implies GCC doesn't
conform, I said so explictly because Paul Eggert said that c99 is often
implemented using GCC.

						Ross Ridge

-- 
 l/  //	  Ross Ridge -- The Great HTMU
[oo][oo]  rridge@csclub.uwaterloo.ca
-()-/()/  http://www.csclub.uwaterloo.ca/u/rridge/ 
 db  //	  

Re: proposed Opengroup action for c99 command (XCU ERN 76)

[Robert Dewar]

Ross Ridge wrote:

> GCC doesn't claim C99 conformance.  The following URL lists a number of
> different areas in which GCC is known not to conform:

I was not asking the general question, I was asking how it fails
to conform wrt the particular technical issue at hand.

Re: proposed Opengroup action for c99 command (XCU ERN 76)

[Ross Ridge]

Ross Ridge wrote:
> GCC doesn't use (A), (B) or (C).  GCC doesn't conform to C99 and
> any implementation of "c99" that uses GCC would presumably also be
> non-conforming.
  
Robert Dewar wrote:
> What exactly is the observable non-conformance?

Ross Ridge wrote:
> GCC doesn't claim C99 conformance.  The following URL lists a number of
> different areas in which GCC is known not to conform:
> 
> 	http://gcc.gnu.org/c99status.html
 
Ian Lance Taylor wrote:
> How does the fact that gcc does not currently conform to C99 imply
> that gcc doesn't use (A), (B), or (C)?
 
It doesn't, the implication is the other way around.  But if you're
asking for "observable non-conformance" then there are lot more obvious
ways to observe it than by showing that GCC doesn't use (A), (B) or (C).

> In any case, the general goal is to conform to C99, so it still makes
> sense to discuss this.

Maybe, but there's no implementation of (A) or (C) in GCC that would
make requiring (B) a burden.  So I think Paul Eggert's question has
been answered.

						Ross Ridge

Re: proposed Opengroup action for c99 command (XCU ERN 76)

[Neil Booth]

Paul Eggert wrote:-

> Here's the problem.  Currently, POSIX places almost no requirements on
> how c99 transforms the physical source file into C source-language
> characters.  For example, c99 is free to treat CR as LF, ignore
> trailing white space, convert tabs to spaces, or even (perversely)
> require that input files all start with line numbers that are
> otherwise ignored.  This lack of specification was not intended, and
> I'm trying to help nail down the intent of what c99 is allowed to do.

I suggest we simply remove the GCC extension.  I never liked it,
it's awkward to code, and it slows down CPP a bit.

Neil.

Re: proposed Opengroup action for c99 command (XCU ERN 76)

[Ian Lance Taylor]

rridge@csclub.uwaterloo.ca (Ross Ridge) writes:

> Ross Ridge wrote:
> > GCC doesn't use (A), (B) or (C).  GCC doesn't conform to C99 and
> > any implementation of "c99" that uses GCC would presumably also be
> > non-conforming.
>  
> Robert Dewar wrote:
> > What exactly is the observable non-conformance?
> 
> GCC doesn't claim C99 conformance.  The following URL lists a number of
> different areas in which GCC is known not to conform:
> 
> 	http://gcc.gnu.org/c99status.html

How does the fact that gcc does not currently conform to C99 imply
that gcc doesn't use (A), (B), or (C)?

In any case, the general goal is to conform to C99, so it still makes
sense to discuss this.

Ian

Re: proposed Opengroup action for c99 command (XCU ERN 76)

[Ross Ridge]

Ross Ridge wrote:
> GCC doesn't use (A), (B) or (C).  GCC doesn't conform to C99 and
> any implementation of "c99" that uses GCC would presumably also be
> non-conforming.
 
Robert Dewar wrote:
> What exactly is the observable non-conformance?

GCC doesn't claim C99 conformance.  The following URL lists a number of
different areas in which GCC is known not to conform:

	http://gcc.gnu.org/c99status.html

						Ross Ridge

-- 
 l/  //	  Ross Ridge -- The Great HTMU
[oo][oo]  rridge@csclub.uwaterloo.ca
-()-/()/  http://www.csclub.uwaterloo.ca/u/rridge/ 
 db  //	  

Re: proposed Opengroup action for c99 command (XCU ERN 76)

[Robert Dewar]

Ross Ridge wrote:

> GCC doesn't use (A), (B) or (C).  GCC doesn't conform to C99 and
> any implementation of "c99" that uses GCC would presumably also be
> non-conforming.

What exactly is the observable non-conformance?

Re: proposed Opengroup action for c99 command (XCU ERN 76)

[Ross Ridge]

Paul Eggert wrote:
>So my question is: Is it a burden on GCC to require interpretation (B)?
>
>My understanding is that GCC already uses (B), and that the answer is
>"no, it's no problem", but if I'm wrong please let me know.

GCC doesn't use (A), (B) or (C).  GCC doesn't conform to C99 and
any implementation of "c99" that uses GCC would presumably also be
non-conforming.

					Ross Ridge

Re: proposed Opengroup action for c99 command (XCU ERN 76)

[Joe Buck]

On Wed, Sep 14, 2005 at 01:50:19PM -0700, Paul Eggert wrote:
> In response Joseph S. Myers pointed out that this action would require
> c99 to use interpretation B of section 5.2.1 (page 20) of the C99 Rationale
> <http://www.open-std.org/jtc1/sc22/wg14/www/C99RationaleV5.10.pdf>.
> The Rationale says C preprocessors can be implemented in three ways:
> 
>   A.  Convert everything to UCNs in basic source characters as soon
>       as possible, that is, in translation phase 1.  (This is what
>       C++ requires, apparently.)
> 
>   B.  Use native encodings where possible, UCNs otherwise.
> 
>   C.  Convert everything to wide characters as soon as possible
>       using an internal encoding that encompasses the entire source
>       character set and all UCNs.
>
> The C99 standardizers chose (B), but said implementations could also
> use (A) or (C) because the C99 standard gives almost unlimited freedom
> in translation phase 1 for compilers to do whatever transformations
> they like.
> 
> However, the proposed action for the c99 command would close this
> escape hatch, forcing interpretation (B) for c99 implementations.
> 
> So my question is: Is it a burden on GCC to require interpretation (B)?

This is written in terms of internal implementation rather than visible
behavior.  Can you provide an example that lets us distinguish a compiler
that uses method A, from a compiler that uses method B?  If "where
possible" in B simply means "in cases where otherwise the user could 
tell that we aren't doing A", then there would be no observable
difference.

proposed Opengroup action for c99 command (XCU ERN 76)

[Paul Eggert]

I recently proposed to the Open Group an action that would modify the
POSIX specification for the c99 command that is often implemented
using GCC.  I thought the action would not affect GCC's conformance,
but Joseph S. Myers raised the issue of UCNs and multibyte characters
and I'd like to double-check that GCC is OK.  If the action does
affect GCC I'd like to modify the action before it's too late.

Here's the problem.  Currently, POSIX places almost no requirements on
how c99 transforms the physical source file into C source-language
characters.  For example, c99 is free to treat CR as LF, ignore
trailing white space, convert tabs to spaces, or even (perversely)
require that input files all start with line numbers that are
otherwise ignored.  This lack of specification was not intended, and
I'm trying to help nail down the intent of what c99 is allowed to do.

I proposed to insert the following paragraph after XCU page 213 line
8366 (i.e, at the end of the INPUT FILES section of the c99 spec
<http://www.opengroup.org/onlinepubs/009695399/utilities/c99.html>):

   It is implementation-defined whether trailing white-space characters
   in each C-language source line are ignored.  Otherwise, the
   multibyte characters of each source line are mapped on a one-to-one
   basis to the C source character set.

In response Joseph S. Myers pointed out that this action would require
c99 to use interpretation B of section 5.2.1 (page 20) of the C99 Rationale
<http://www.open-std.org/jtc1/sc22/wg14/www/C99RationaleV5.10.pdf>.
The Rationale says C preprocessors can be implemented in three ways:

  A.  Convert everything to UCNs in basic source characters as soon
      as possible, that is, in translation phase 1.  (This is what
      C++ requires, apparently.)

  B.  Use native encodings where possible, UCNs otherwise.

  C.  Convert everything to wide characters as soon as possible
      using an internal encoding that encompasses the entire source
      character set and all UCNs.

The C99 standardizers chose (B), but said implementations could also
use (A) or (C) because the C99 standard gives almost unlimited freedom
in translation phase 1 for compilers to do whatever transformations
they like.

However, the proposed action for the c99 command would close this
escape hatch, forcing interpretation (B) for c99 implementations.

So my question is: Is it a burden on GCC to require interpretation (B)?

My understanding is that GCC already uses (B), and that the answer is
"no, it's no problem", but if I'm wrong please let me know.

For more details, please see Shell and Utilities Enhancement Request
Number 76 (XCU ERN 76), which you can find in
<http://www.opengroup.org/austin/aardvark/latest/xcubug2.txt>.
Also please see the followup email discussion at
<http://www.opengroup.org/austin/mailarchives/ag/>
(look for messages whose subject lines contain "XCU ERN 76").

Thanks.