RFC: c++ diagnostics

RFC: c++ diagnostics

[Benjamin Kosnik]

Hello all! 

I've put up a short diagnostics comparison between gcc, icc, and
clang. It is my plan to update this with major revisions to individual
compilers. 

Included are most of the outstanding bugzilla requests with the
"diagnostic" keyword. However, I am looking for help! Please send me
code samples that frustrate, obfuscate, and annoy. 

In particular, I am looking for template instantiation issues such as
c++/41884, but hopefully something in a deliciously small snippet. No
doubt other C++ hackers have particular annoyances.

I'm also looking for guidance on how to rank the priority of these
issues. Is there some way to tell what the biggest annoyance is?

http://people.redhat.com/bkoz/diagnostics/diagnostics.html

best,
benjamin

Re: RFC: c++ diagnostics

[Chris Lattner]

On Apr 5, 2010, at 8:20 AM, Benjamin Kosnik wrote:

> 
> Hello all! 
> 
> I've put up a short diagnostics comparison between gcc, icc, and
> clang. It is my plan to update this with major revisions to individual
> compilers. 
> 
> Included are most of the outstanding bugzilla requests with the
> "diagnostic" keyword. However, I am looking for help! Please send me
> code samples that frustrate, obfuscate, and annoy. 
> 
> In particular, I am looking for template instantiation issues such as
> c++/41884, but hopefully something in a deliciously small snippet. No
> doubt other C++ hackers have particular annoyances.
> 
> I'm also looking for guidance on how to rank the priority of these
> issues. Is there some way to tell what the biggest annoyance is?
> 
> http://people.redhat.com/bkoz/diagnostics/diagnostics.html

This is a great resource Benjamin, thanks for putting it together!

Some random thoughts if you ever regenerate this:

1) the caret diagnostics would be easier to understand in the output if formatted with a <pre> or <tt> tag.  

2) The clang invocations don't need -fcaret-diagnostics -fshow-source-location -fdiagnostics-fixit-info because they are the default.

3) It's best to not pass -fdiagnostics-print-source-range-info unless you're looking for machine interpretable output.  This flag adds things like {3:29-3:32} which are useful to a machine, but otherwise just clutter the output up.

4) It looks like ICC defaults to a number of coding standards types of checks, e.g. "access control not specified".  I don't know if it is possible to turn those off, but they seem to just be noise for the purposes of this comparison.

5) There are a couple cases of GCC rejecting valid code (e.g. 19377), or which there may be some debate about (19538) it might be worth pointing this out. *shrug*

6) My understanding was that GCC's complex extension in C++ mode is supposed to work like C99 _Complex. If so, 41895 looks like a GCC bug.  I don't know if C++'0x affects this though.

7) There are some clang bugs here.  Access control is not yet enabled by default (affects 20397, 39728), and a variety of other bugs (affects 14283, 38612).  I file Clang PR#6782/6783 to track these.

Thanks again for putting this together,

-Chris

Re: RFC: c++ diagnostics

[Benjamin Kosnik]

> 2) The clang invocations don't need -fcaret-diagnostics
> -fshow-source-location -fdiagnostics-fixit-info because they are the
> default.
> 
> 3) It's best to not pass -fdiagnostics-print-source-range-info unless
> you're looking for machine interpretable output.  This flag adds
> things like {3:29-3:32} which are useful to a machine, but otherwise
> just clutter the output up.
> 
> 4) It looks like ICC defaults to a number of coding standards types
> of checks, e.g. "access control not specified".  I don't know if it
> is possible to turn those off, but they seem to just be noise for the
> purposes of this comparison.

I'll look into this. Some of this is over-zealousness on my part and
not wanting to miss a diagnostics flag. If I could make the diagnostics
more verbose I tried to not miss an opportunity to twist a knob.

The actual flags I compiled the data with are in this script and may
vary from the total listing at the top of the webpage.

 http://people.redhat.com/bkoz/diagnostics/make-diagnostic-output.sh

> 5) There are a couple cases of GCC rejecting valid code (e.g. 19377),
> or which there may be some debate about (19538) it might be worth
> pointing this out. *shrug*

One of the goals was to measure the output when the input is
truncated, or obviously flawed (no semicolons is very
common!). Certainly if you can think of other (obvious) issues where
refactoring or haste make general classes of errors I'm very interested
in this particular type of pathology.

The valid code issues I can flag in the existing bug reports.

thanks,
benjamin

Re: RFC: c++ diagnostics

[Chris Lattner]

On Apr 5, 2010, at 12:51 PM, Benjamin Kosnik wrote:

>> 
>> 5) There are a couple cases of GCC rejecting valid code (e.g. 19377),
>> or which there may be some debate about (19538) it might be worth
>> pointing this out. *shrug*
> 
> One of the goals was to measure the output when the input is
> truncated, or obviously flawed (no semicolons is very
> common!). Certainly if you can think of other (obvious) issues where
> refactoring or haste make general classes of errors I'm very interested
> in this particular type of pathology.

Absolutely, this is one of the reasons I'm particularly interested in common errors like missing semi colons, . vs ->, use of things like <::foo>, explaining overload set failures, etc.

> The valid code issues I can flag in the existing bug reports.

Ok, thanks again.

-Chris

Re: RFC: c++ diagnostics

[Manuel López-Ibáñez]

On 5 April 2010 17:20, Benjamin Kosnik <bkoz@redhat.com> wrote:
>
> Hello all!
>
> I've put up a short diagnostics comparison between gcc, icc, and
> clang. It is my plan to update this with major revisions to individual
> compilers.

Awesome!

How to contribute? patches against the html? I see there are some
examples without output. Also, it would be nicer if the page linked to
each PR in bugzilla.

On the one hand, it is strange that you use
-fdiagnostics-show-location=once  without -fmessage-length= because
that doesn't have any effect. On the other hand,
-fdiagnostics-show-option will give an output more similar to the one
of clang.

> Included are most of the outstanding bugzilla requests with the
> "diagnostic" keyword. However, I am looking for help! Please send me
> code samples that frustrate, obfuscate, and annoy.

I guess most of the ones listed here:

http://clang.llvm.org/diagnostics.html

Also, posting this to kde-devel@kde.org may get some further feedback.

> I'm also looking for guidance on how to rank the priority of these
> issues. Is there some way to tell what the biggest annoyance is?

Let people vote in GCC bugzilla? Hum, perhaps not...

Cheers,

Manuel.

Re: RFC: c++ diagnostics

[Benjamin Kosnik]

> How to contribute? patches against the html? I see there are some
> examples without output. Also, it would be nicer if the page linked to
> each PR in bugzilla.

Well, the html is auto-generated so that isn't really the way to go.
Should I just check in the tests + xml into some gcc repository? There
is a README in the enclosing directory in the original URL that tries to
explain the way the sources are generated.

Certainly I can add a notes section to the HTML output and put in links
to specific bug reports.

> On the one hand, it is strange that you use
> -fdiagnostics-show-location=once  without -fmessage-length= because
> that doesn't have any effect. On the other hand,
> -fdiagnostics-show-option will give an output more similar to the one
> of clang.

Thanks for pointing this out. 
 
> > Included are most of the outstanding bugzilla requests with the
> > "diagnostic" keyword. However, I am looking for help! Please send me
> > code samples that frustrate, obfuscate, and annoy.
> 
> I guess most of the ones listed here:
> 
> http://clang.llvm.org/diagnostics.html
> 
> Also, posting this to kde-devel@kde.org may get some further feedback.

Oh, good idea. 
 
> > I'm also looking for guidance on how to rank the priority of these
> > issues. Is there some way to tell what the biggest annoyance is?
> 
> Let people vote in GCC bugzilla? Hum, perhaps not...

That's an idea, to be able to vote up diagnostics things in bugzilla.
The Red Hat bugzilla lets me do that for os bugs.

One of the other areas I really want to explore are some of the
verbose diagnostics that deal with missed optimizations, looping, or
vectorization. I see a lot more options for complex diagnostic/analysis
passes in the current round of compilers and would like to figure out
some way of tracking this for GCC. Any ideas? Even the basics would be
helpful.

-benjamin

Re: RFC: c++ diagnostics

[Chris Lattner]

On Apr 5, 2010, at 8:20 AM, Benjamin Kosnik wrote:

> 
> Hello all! 
> 
> I've put up a short diagnostics comparison between gcc, icc, and
> clang. It is my plan to update this with major revisions to individual
> compilers. 
> 
> Included are most of the outstanding bugzilla requests with the
> "diagnostic" keyword. However, I am looking for help! Please send me
> code samples that frustrate, obfuscate, and annoy. 
> 
> In particular, I am looking for template instantiation issues such as
> c++/41884, but hopefully something in a deliciously small snippet. No
> doubt other C++ hackers have particular annoyances.
> 

Hi Benjamin,

I wrote a little blog post that shows off some of the things that Clang can do.  It would be great to improve some of GCC/G++'s diagnostics in a similar way:

http://blog.llvm.org/2010/04/amazing-feats-of-clang-error-recovery.html

-Chris

Re: RFC: c++ diagnostics

[Joe Buck]

On Tue, Apr 06, 2010 at 09:00:16AM -0700, Chris Lattner wrote:
> I wrote a little blog post that shows off some of the things that Clang can do.  It would be great to improve some of GCC/G++'s diagnostics in a similar way:
> 
> http://blog.llvm.org/2010/04/amazing-feats-of-clang-error-recovery.html

Some of Chris's examples appear to be regressions, in that gcc's old
bison-based parser did a better job than the current parser.  In
particular, around the time of the 2->3 transition, a rule was added to
catch the fact that

foo bar;

where neither is defined as a type, almost certainly is because of a
missing definition of the type foo, or a mis-spelling (this is what is
going on in several of Chris's examples).  I pushed to get that rule
added because I tried a lot of "C++" code that had only been compiled
with g++ 2.9x, and it was filled with uses of STL without std:: because
back then, the standard library wasn't in a namespace and std:: was
magic that was ignored.

Re: RFC: c++ diagnostics

[Manuel López-Ibáñez]

On 6 April 2010 18:00, Chris Lattner <clattner@apple.com> wrote:
>
> On Apr 5, 2010, at 8:20 AM, Benjamin Kosnik wrote:
>
>>
>> Hello all!
>>
>> I've put up a short diagnostics comparison between gcc, icc, and
>> clang. It is my plan to update this with major revisions to individual
>> compilers.
>>
>> Included are most of the outstanding bugzilla requests with the
>> "diagnostic" keyword. However, I am looking for help! Please send me
>> code samples that frustrate, obfuscate, and annoy.
>>
>> In particular, I am looking for template instantiation issues such as
>> c++/41884, but hopefully something in a deliciously small snippet. No
>> doubt other C++ hackers have particular annoyances.
>>
>
> Hi Benjamin,
>
> I wrote a little blog post that shows off some of the things that Clang can do.  It would be great to improve some of GCC/G++'s diagnostics in a similar way:
>
> http://blog.llvm.org/2010/04/amazing-feats-of-clang-error-recovery.html

Unfortunately, you mix C and C++ examples. I know it is because clang
parses both with the same parser but this thread is only discussing
improving C++ diagnostics in GCC, and there is no plan or project for
C diagnostics. As it happens, some C++ diagnostics are better than the
same diagnostic for C and viceversa.

I love the merge conflict detector, btw. :-)  If I am allowed, I would
like to implement that one in C/C++ parsers. Joseph? Jason?

Cheers,

Manuel.

Re: RFC: c++ diagnostics

[Chris Lattner]

On Apr 6, 2010, at 9:29 AM, Manuel López-Ibáñez wrote:
>> Hi Benjamin,
>> 
>> I wrote a little blog post that shows off some of the things that Clang can do.  It would be great to improve some of GCC/G++'s diagnostics in a similar way:
>> 
>> http://blog.llvm.org/2010/04/amazing-feats-of-clang-error-recovery.html
> 
> Unfortunately, you mix C and C++ examples. I know it is because clang
> parses both with the same parser but this thread is only discussing
> improving C++ diagnostics in GCC, and there is no plan or project for
> C diagnostics. As it happens, some C++ diagnostics are better than the
> same diagnostic for C and viceversa.

I think all the C examples are also valid C++ code, they should apply equally well, but I admit that I didn't try those on g++ to see how it does.  I figured it also didn't matter much because there has surely been significant progress since gcc 4.2.

-Chris

Re: RFC: c++ diagnostics

[Joe Buck]

> >> http://blog.llvm.org/2010/04/amazing-feats-of-clang-error-recovery.html
> > 
> > ...As it happens, some C++ diagnostics are better than the
> > same diagnostic for C and viceversa.

On Tue, Apr 06, 2010 at 09:45:11AM -0700, Chris Lattner wrote:
> I think all the C examples are also valid C++ code, they should apply equally well, but I admit that I didn't try those on g++ to see how it does.  I figured it also didn't matter much because there has surely been significant progress since gcc 4.2.

Yes, g++ does a better job for some of Chris's examples than gcc does.

For the second example we get

t.c:1: error: 'pid_t' has not been declared

For the third example:
t.c:2: error: 'int64' does not name a type

However, most of the criticisms do apply, and the spell checker is a
very good idea.

Re: RFC: c++ diagnostics

[Laurent GUERBY]

On Tue, 2010-04-06 at 12:02 -0700, Joe Buck wrote:
> > >> http://blog.llvm.org/2010/04/amazing-feats-of-clang-error-recovery.html
> > > 
> > > ...As it happens, some C++ diagnostics are better than the
> > > same diagnostic for C and viceversa.
> 
> On Tue, Apr 06, 2010 at 09:45:11AM -0700, Chris Lattner wrote:
> > I think all the C examples are also valid C++ code, they should apply equally well, but I admit that I didn't try those on g++ to see how it does.  I figured it also didn't matter much because there has surely been significant progress since gcc 4.2.
> 
> Yes, g++ does a better job for some of Chris's examples than gcc does.
> 
> For the second example we get
> 
> t.c:1: error: 'pid_t' has not been declared
> 
> For the third example:
> t.c:2: error: 'int64' does not name a type
> 
> However, most of the criticisms do apply, and the spell checker is a
> very good idea.

The Ada FE does it FWIW:

$ cat p.ads
pakage P is
  subtype My_Count is Intger
end P;
$ gcc -c -gnatq p.ads
p.ads:1:01: incorrect spelling of keyword "package"
p.ads:2:23: "Intger" is undefined
p.ads:2:23: possible misspelling of "Integer"
p.ads:2:29: missing ";"
$

The checker is in gcc/ada/g-spchge.adb

Laurent

Re: RFC: c++ diagnostics

[Robert Dewar]

Laurent GUERBY wrote:
> On Tue, 2010-04-06 at 12:02 -0700, Joe Buck wrote:
>>>>> http://blog.llvm.org/2010/04/amazing-feats-of-clang-error-recovery.html
>>>> ...As it happens, some C++ diagnostics are better than the
>>>> same diagnostic for C and viceversa.
>> On Tue, Apr 06, 2010 at 09:45:11AM -0700, Chris Lattner wrote:
>>> I think all the C examples are also valid C++ code, they should apply equally well, but I admit that I didn't try those on g++ to see how it does.  I figured it also didn't matter much because there has surely been significant progress since gcc 4.2.
>> Yes, g++ does a better job for some of Chris's examples than gcc does.
>>
>> For the second example we get
>>
>> t.c:1: error: 'pid_t' has not been declared
>>
>> For the third example:
>> t.c:2: error: 'int64' does not name a type
>>
>> However, most of the criticisms do apply, and the spell checker is a
>> very good idea.
> 
> The Ada FE does it FWIW:
> 
> $ cat p.ads
> pakage P is
>   subtype My_Count is Intger
> end P;
> $ gcc -c -gnatq p.ads
> p.ads:1:01: incorrect spelling of keyword "package"
> p.ads:2:23: "Intger" is undefined
> p.ads:2:23: possible misspelling of "Integer"
> p.ads:2:29: missing ";"

The algorithm used in the Ada compiler is in
g-speche.ads and it seems to be just right for
use by the compiler.
> $
> 
> The checker is in gcc/ada/g-spchge.adb
> 
> Laurent
> 

Re: RFC: c++ diagnostics

[Manuel López-Ibáñez]

On 5 April 2010 17:20, Benjamin Kosnik <bkoz@redhat.com> wrote:
>
> Included are most of the outstanding bugzilla requests with the
> "diagnostic" keyword. However, I am looking for help! Please send me
> code samples that frustrate, obfuscate, and annoy.

Some PRs missing in this list:

986
13452
13657
15766
16663
17459
23263
31423
36888
37200
39858
39859
43275


And these are only about errors. There are a lot more about warnings.

On the other hand, PR 43431 is marked as fixed, it is not a parsing
problem and i think it should be removed. Again, links in the PR
numbers would help to check the status of the PRs listed in the
webpage.

Cheers,

Manuel.

Re: RFC: c++ diagnostics

[Manuel López-Ibáñez]

This may be of interest to KDE developers. So adding them to the CC list.

Manuel.

On 5 April 2010 17:20, Benjamin Kosnik <bkoz@redhat.com> wrote:
>
> Hello all!
>
> I've put up a short diagnostics comparison between gcc, icc, and
> clang. It is my plan to update this with major revisions to individual
> compilers.
>
> Included are most of the outstanding bugzilla requests with the
> "diagnostic" keyword. However, I am looking for help! Please send me
> code samples that frustrate, obfuscate, and annoy.
>
> In particular, I am looking for template instantiation issues such as
> c++/41884, but hopefully something in a deliciously small snippet. No
> doubt other C++ hackers have particular annoyances.
>
> I'm also looking for guidance on how to rank the priority of these
> issues. Is there some way to tell what the biggest annoyance is?
>
> http://people.redhat.com/bkoz/diagnostics/diagnostics.html
>
> best,
> benjamin
>