[Bug c++/49973] New: Column numbers count special characters as multiple columns

[Bug c++/49973] New: Column numbers count special characters as multiple columns

[timothy003 at msn dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=49973

           Summary: Column numbers count special characters as multiple
                    columns
           Product: gcc
           Version: 4.5.2
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: c++
        AssignedTo: unassigned@gcc.gnu.org
        ReportedBy: timothy003@msn.com


int main()
{
    /* 中 */ asdf;
}

g++ -finput-charset=utf8 hello.cpp

hello.cpp: In function ‘int main()’:
hello.cpp:3:12: error: ‘asdf’ was not declared in this scope

The column number should be 10, not 12.

[Bug c++/49973] Column numbers count special characters as multiple columns

[jakub at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=49973

Jakub Jelinek <jakub at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |jakub at gcc dot gnu.org

--- Comment #1 from Jakub Jelinek <jakub at gcc dot gnu.org> 2011-08-04 10:27:53 UTC ---
Depends on how the column numbers are defined.  I think gcc uses bytes from the
beginning of the line, then 12 is correct (and e.g. for tab characters gcc
counts them as one instead of 1-8 depending on position too).

[Bug c++/49973] Column numbers count special characters as multiple columns

[timothy003 at msn dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=49973

--- Comment #2 from Timothy Liang <timothy003 at msn dot com> 2011-08-04 10:36:53 UTC ---
(In reply to comment #1)
> Depends on how the column numbers are defined.  I think gcc uses bytes from the
> beginning of the line, then 12 is correct (and e.g. for tab characters gcc
> counts them as one instead of 1-8 depending on position too).

That isn't the case here.  Substituting the '中' for another character makes the
column number 10.  Setting -finput-charset=latin1 makes the column number 15.

[Bug c++/49973] Column numbers count special characters as multiple columns

[schwab@linux-m68k.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=49973

--- Comment #3 from Andreas Schwab <schwab@linux-m68k.org> 2011-08-04 10:55:13 UTC ---
Why 10? "    /* 中 */ " has 12 characters (and 14 bytes as utf8).

[Bug c++/49973] Column numbers count special characters as multiple columns

[timothy003 at msn dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=49973

--- Comment #4 from Timothy Liang <timothy003 at msn dot com> 2011-08-04 11:03:32 UTC ---
(In reply to comment #3)
> Why 10? "    /* 中 */ " has 12 characters (and 14 bytes as utf8).

The four spaces is supposed to be a tab.  Also, the column number starts from
one.  So:

 /* 中 */ asdf
         |
1234567890

Since I set the input charset as UTF-8, g++ should count the '中' as one
character, not three.  And when I set it to latin1, g++ should count the '中' as
three, not six.

[Bug c++/49973] Column numbers count special characters as multiple columns

[manu at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=49973

Manuel López-Ibáñez <manu at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |manu at gcc dot gnu.org

--- Comment #5 from Manuel López-Ibáñez <manu at gcc dot gnu.org> 2011-08-04 12:18:19 UTC ---
GNU Emacs 23.2.1 counts it as two, and puts the cursor at s.

For the simpler case of:

/* ñ */ asdf;

we print 

test.c:3:10: error: ‘asdf’ was not declared in this scope

whereas emacs counts only 1 char, so it again puts the cursor at s. I am not
sure whether Emacs is following some GNU standard, but the case of ñ versus n,
should at least produce the same result.

Unfortunately, I don't have time to work on this.

[Bug c++/49973] Column numbers count special characters as multiple columns

[joseph at codesourcery dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=49973

--- Comment #6 from joseph at codesourcery dot com <joseph at codesourcery dot com> 2011-08-04 14:38:16 UTC ---
The GCS says "column numbers should start from 1 at the beginning of the 
line ... Calculate column numbers assuming that space and all ASCII 
printing characters have equal width, and assuming tab stops every 8 
columns.".  This doesn't say how other characters should be counted, 
although for the results of wcswidth seem appropriate.

[Bug c++/49973] Column numbers count special characters as multiple columns

[manu at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=49973

Manuel López-Ibáñez <manu at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|UNCONFIRMED                 |NEW
   Last reconfirmed|                            |2011.08.04 15:18:20
     Ever Confirmed|0                           |1

--- Comment #7 from Manuel López-Ibáñez <manu at gcc dot gnu.org> 2011-08-04 15:18:20 UTC ---
(In reply to comment #6)
> The GCS says "column numbers should start from 1 at the beginning of the 
> line ... Calculate column numbers assuming that space and all ASCII 
> printing characters have equal width, and assuming tab stops every 8 
> columns.".  This doesn't say how other characters should be counted, 
> although for the results of wcswidth seem appropriate.

Then GCC is not using wcswidth to count or it is setting the locale
inappropriately because it is counting 2 for ñ and 3 for 中, while it should be
1 and 2.

[Bug c++/49973] Column numbers count special characters as multiple columns

[timothy003 at msn dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=49973

--- Comment #8 from Timothy Liang <timothy003 at msn dot com> 2011-08-04 19:52:31 UTC ---
(In reply to comment #7)
> (In reply to comment #6)
> > The GCS says "column numbers should start from 1 at the beginning of the 
> > line ... Calculate column numbers assuming that space and all ASCII 
> > printing characters have equal width, and assuming tab stops every 8 
> > columns.".  This doesn't say how other characters should be counted, 
> > although for the results of wcswidth seem appropriate.
> 
> Then GCC is not using wcswidth to count or it is setting the locale
> inappropriately because it is counting 2 for ñ and 3 for 中, while it should be
> 1 and 2.

I'm confused.  Shouldn't 中 be 1?

[Bug preprocessor/49973] Column numbers count special characters as multiple columns

[tromey at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=49973

Tom Tromey <tromey at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |tromey at gcc dot gnu.org
          Component|c++                         |preprocessor

--- Comment #9 from Tom Tromey <tromey at gcc dot gnu.org> 2011-12-07 17:59:52 UTC ---
(In reply to comment #6)
> The GCS says "column numbers should start from 1 at the beginning of the 
> line ... Calculate column numbers assuming that space and all ASCII 
> printing characters have equal width, and assuming tab stops every 8 
> columns.".  This doesn't say how other characters should be counted, 
> although for the results of wcswidth seem appropriate.

Note that GCC also handles the tab case incorrectly here.
This shows up if you M-x next-error in Emacs in the case where
gcc emits column numbers and your source code includes tabs.\

Refiling this to preprocessor.

[Bug preprocessor/49973] Column numbers count special characters as multiple columns

[joseph at codesourcery dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=49973

--- Comment #10 from joseph at codesourcery dot com <joseph at codesourcery dot com> 2011-12-07 20:56:01 UTC ---
On Wed, 7 Dec 2011, tromey at gcc dot gnu.org wrote:

> Note that GCC also handles the tab case incorrectly here.

Yes, GCC should be fixed to follow the GCS there as well.

The GCS now explicitly say "For non-ASCII characters, Unicode character 
widths should be used when in a UTF-8 locale; GNU libc and GNU gnulib 
provide suitable @code{wcwidth} functions."

[Bug preprocessor/49973] Column numbers count multibyte characters as multiple columns

[cvs-commit at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=49973

--- Comment #22 from CVS Commits <cvs-commit at gcc dot gnu.org> ---
The master branch has been updated by Lewis Hyatt <lhyatt@gcc.gnu.org>:

https://gcc.gnu.org/g:004bb936d6d5f177af26ad4905595e843d5665a5

commit r11-2092-g004bb936d6d5f177af26ad4905595e843d5665a5
Author: Lewis Hyatt <lhyatt@gmail.com>
Date:   Tue Jul 14 12:05:56 2020 -0400

    diagnostics: Support conversion of tabs to spaces [PR49973] [PR86904]

    Supports conversion of tabs to spaces when outputting diagnostics. Also
    adds -fdiagnostics-column-unit and -fdiagnostics-column-origin options to
    control how the column number is output, thereby resolving the two PRs.

    gcc/c-family/ChangeLog:

            PR other/86904
            * c-indentation.c (should_warn_for_misleading_indentation): Get
            global tabstop from the new source.
            * c-opts.c (c_common_handle_option): Remove handling of -ftabstop,
which
            is now a common option.
            * c.opt: Likewise.

    gcc/ChangeLog:

            PR preprocessor/49973
            PR other/86904
            * common.opt: Handle -ftabstop here instead of in c-family
            options.  Add -fdiagnostics-column-unit= and
            -fdiagnostics-column-origin= options.
            * opts.c (common_handle_option): Handle the new options.
            * diagnostic-format-json.cc (json_from_expanded_location): Add
            diagnostic_context argument.  Use it to convert column numbers as
per
            the new options.
            (json_from_location_range): Likewise.
            (json_from_fixit_hint): Likewise.
            (json_end_diagnostic): Pass the new context argument to helper
            functions above.  Add "column-origin" field to the output.
            (test_unknown_location): Add the new context argument to calls to
            helper functions.
            (test_bad_endpoints): Likewise.
            * diagnostic-show-locus.c
            (exploc_with_display_col::exploc_with_display_col): Support
            tabstop parameter.
            (layout_point::layout_point): Make use of class
            exploc_with_display_col.
            (layout_range::layout_range): Likewise.
            (struct line_bounds): Clarify that the units are now always
            display columns.  Rename members accordingly.  Add constructor.
            (layout::print_source_line): Add support for tab expansion.
            (make_range): Adapt to class layout_range changes.
            (layout::maybe_add_location_range): Likewise.
            (layout::layout): Adapt to class exploc_with_display_col changes.
            (layout::calculate_x_offset_display): Support tabstop parameter.
            (layout::print_annotation_line): Adapt to struct line_bounds
changes.
            (layout::print_line): Likewise.
            (line_label::line_label): Add diagnostic_context argument.
            (get_affected_range): Likewise.
            (get_printed_columns): Likewise.
            (layout::print_any_labels): Adapt to struct line_label changes.
            (class correction): Add m_tabstop member.
            (correction::correction): Add tabstop argument.
            (correction::compute_display_cols): Use m_tabstop.
            (class line_corrections): Add m_context member.
            (line_corrections::line_corrections): Add diagnostic_context
argument.
            (line_corrections::add_hint): Use m_context to handle tabstops.
            (layout::print_trailing_fixits): Adapt to class line_corrections
            changes.
            (test_layout_x_offset_display_utf8): Support tabstop parameter.
            (test_layout_x_offset_display_tab): New selftest.
            (test_one_liner_colorized_utf8): Likewise.
            (test_tab_expansion): Likewise.
            (test_diagnostic_show_locus_one_liner_utf8): Call the new tests.
            (diagnostic_show_locus_c_tests): Likewise.
            (test_overlapped_fixit_printing): Adapt to helper class and
            function changes.
            (test_overlapped_fixit_printing_utf8): Likewise.
            (test_overlapped_fixit_printing_2): Likewise.
            * diagnostic.h (enum diagnostics_column_unit): New enum.
            (struct diagnostic_context): Add members for the new options.
            (diagnostic_converted_column): Declare.
            (json_from_expanded_location): Add new context argument.
            * diagnostic.c (diagnostic_initialize): Initialize new members.
            (diagnostic_converted_column): New function.
            (maybe_line_and_column): Be willing to output a column of 0.
            (diagnostic_get_location_text): Convert column number as per the
new
            options.
            (diagnostic_report_current_module): Likewise.
            (assert_location_text): Add origin and column_unit arguments for
            testing the new functionality.
            (test_diagnostic_get_location_text): Test the new functionality.
            * doc/invoke.texi: Document the new options and behavior.
            * input.h (location_compute_display_column): Add tabstop argument.
            * input.c (location_compute_display_column): Likewise.
            (test_cpp_utf8): Add selftests for tab expansion.
            * tree-diagnostic-path.cc (default_tree_make_json_for_path): Pass
the
            new context argument to json_from_expanded_location().

    libcpp/ChangeLog:

            PR preprocessor/49973
            PR other/86904
            * include/cpplib.h (struct cpp_options):  Removed support for
-ftabstop,
            which is now handled by diagnostic_context.
            (class cpp_display_width_computation): New class.
            (cpp_byte_column_to_display_column): Add optional tabstop argument.
            (cpp_display_width): Likewise.
            (cpp_display_column_to_byte_column): Likewise.
            * charset.c
            (cpp_display_width_computation::cpp_display_width_computation): New
            function.
            (cpp_display_width_computation::advance_display_cols): Likewise.
            (compute_next_display_width): Removed and implemented this
            functionality in a new function...
            (cpp_display_width_computation::process_next_codepoint): ...here.
            (cpp_byte_column_to_display_column): Added tabstop argument.
            Reimplemented in terms of class cpp_display_width_computation.
            (cpp_display_column_to_byte_column): Likewise.
            * init.c (cpp_create_reader): Remove handling of -ftabstop, which
is now
            handled by diagnostic_context.

    gcc/testsuite/ChangeLog:

            PR preprocessor/49973
            PR other/86904
            * c-c++-common/Wmisleading-indentation-3.c: Adjust expected output
            for new defaults.
            * c-c++-common/Wmisleading-indentation.c: Likewise.
            * c-c++-common/diagnostic-format-json-1.c: Likewise.
            * c-c++-common/diagnostic-format-json-2.c: Likewise.
            * c-c++-common/diagnostic-format-json-3.c: Likewise.
            * c-c++-common/diagnostic-format-json-4.c: Likewise.
            * c-c++-common/diagnostic-format-json-5.c: Likewise.
            * c-c++-common/missing-close-symbol.c: Likewise.
            * g++.dg/diagnostic/bad-binary-ops.C: Likewise.
            * g++.dg/parse/error4.C: Likewise.
            * g++.old-deja/g++.brendan/crash11.C: Likewise.
            * g++.old-deja/g++.pt/overload2.C: Likewise.
            * g++.old-deja/g++.robertl/eb109.C: Likewise.
            * gcc.dg/analyzer/malloc-paths-9.c: Likewise.
            * gcc.dg/bad-binary-ops.c: Likewise.
            * gcc.dg/format/branch-1.c: Likewise.
            * gcc.dg/format/pr79210.c: Likewise.
            * gcc.dg/plugin/diagnostic-test-expressions-1.c: Likewise.
            * gcc.dg/plugin/diagnostic-test-string-literals-1.c: Likewise.
            * gcc.dg/redecl-4.c: Likewise.
            * gfortran.dg/diagnostic-format-json-1.F90: Likewise.
            * gfortran.dg/diagnostic-format-json-2.F90: Likewise.
            * gfortran.dg/diagnostic-format-json-3.F90: Likewise.
            * go.dg/arrayclear.go: Add a comment explaining why adding a
            comment was necessary to work around a dejagnu bug.
            * c-c++-common/diagnostic-units-1.c: New test.
            * c-c++-common/diagnostic-units-2.c: New test.
            * c-c++-common/diagnostic-units-3.c: New test.
            * c-c++-common/diagnostic-units-4.c: New test.
            * c-c++-common/diagnostic-units-5.c: New test.
            * c-c++-common/diagnostic-units-6.c: New test.
            * c-c++-common/diagnostic-units-7.c: New test.
            * c-c++-common/diagnostic-units-8.c: New test.

[Bug preprocessor/49973] Column numbers count multibyte characters as multiple columns

[lhyatt at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=49973

--- Comment #23 from Lewis Hyatt <lhyatt at gmail dot com> ---
Fixed for GCC 11, may I ask someone please to close the PR?

[Bug preprocessor/49973] Column numbers count multibyte characters as multiple columns

[manu at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=49973

Manuel López-Ibáñez <manu at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
         Resolution|---                         |FIXED
             Status|NEW                         |RESOLVED

--- Comment #24 from Manuel López-Ibáñez <manu at gcc dot gnu.org> ---
Great!

(If you login into bugzilla with your @gcc.gnu.org account, you should be able
to admin PRs)