Re: [PATCH] c-family: Implement pragma_lex () for preprocess-only mode

[Lewis Hyatt <lhyatt@gmail.com>]

On Wed, Jul 26, 2023 at 5:36 PM Jason Merrill <jason@redhat.com> wrote:
>
> On 6/30/23 18:59, Lewis Hyatt wrote:
> > In order to support processing #pragma in preprocess-only mode (-E or
> > -save-temps for gcc/g++), we need a way to obtain the #pragma tokens from
> > libcpp. In full compilation modes, this is accomplished by calling
> > pragma_lex (), which is a symbol that must be exported by the frontend, and
> > which is currently implemented for C and C++. Neither of those frontends
> > initializes its parser machinery in preprocess-only mode, and consequently
> > pragma_lex () does not work in this case.
> >
> > Address that by adding a new function c_init_preprocess () for the frontends
> > to implement, which arranges for pragma_lex () to work in preprocess-only
> > mode, and adjusting pragma_lex () accordingly.
> >
> > In preprocess-only mode, the preprocessor is accustomed to controlling the
> > interaction with libcpp, and it only knows about tokens that it has called
> > into libcpp itself to obtain. Since it still needs to see the tokens
> > obtained by pragma_lex () so that they can be streamed to the output, also
> > add a new libcpp callback, on_token_lex (), that ensures the preprocessor
> > sees these tokens too.
> >
> > Currently, there is one place where we are already supporting #pragma in
> > preprocess-only mode, namely the handling of `#pragma GCC diagnostic'.  That
> > was done by directly interfacing with libcpp, rather than making use of
> > pragma_lex (). Now that pragma_lex () works, that code is no longer
> > necessary; remove it.
> >
> > gcc/c-family/ChangeLog:
> >
> >       * c-common.h (c_init_preprocess): Declare new function.
> >       * c-opts.cc (c_common_init): Call it.
> >       * c-pragma.cc (pragma_diagnostic_lex_normal): Rename to...
> >       (pragma_diagnostic_lex): ...this.
> >       (pragma_diagnostic_lex_pp): Remove.
> >       (handle_pragma_diagnostic_impl): Call pragma_diagnostic_lex () in
> >       all modes.
> >       (c_pp_invoke_early_pragma_handler): Adapt to support pragma_lex ()
> >       usage.
> >       * c-pragma.h (pragma_lex_discard_to_eol): Declare new function.
> >
> > gcc/c/ChangeLog:
> >
> >       * c-parser.cc (pragma_lex): Support preprocess-only mode.
> >       (pragma_lex_discard_to_eol): New function.
> >       (c_init_preprocess): New function.
> >
> > gcc/cp/ChangeLog:
> >
> >       * parser.cc (c_init_preprocess): New function.
> >       (maybe_read_tokens_for_pragma_lex): New function.
> >       (pragma_lex): Support preprocess-only mode.
> >       (pragma_lex_discard_to_eol): New funtion.
> >
> > libcpp/ChangeLog:
> >
> >       * include/cpplib.h (struct cpp_callbacks): Add new callback
> >       on_token_lex.
> >       * macro.cc (cpp_get_token_1): Support new callback.
> > ---
> >
> > Notes:
> >      Hello-
> >
> >      In r13-1544, I added support for processing `#pragma GCC diagnostic' in
> >      preprocess-only mode. Because pragma_lex () doesn't work in that mode, in
> >      that patch I called into libcpp directly to obtain the tokens needed to
> >      process the pragma. As part of the review, Jason noted that it would
> >      probably be better to make pragma_lex () usable in preprocess-only mode, and
> >      we decided just to add a comment about that for the time being, and to go
> >      ahead and implement that in the future, if it became necessary to support
> >      other pragmas during preprocessing.
> >
> >      I think now is a good time to proceed with that plan, because I would like
> >      to fix PR87299, which is about another pragma (#pragma GCC target) not
> >      working in preprocess-only mode. This patch makes the necessary changes for
> >      pragma_lex () to work in preprocess-only mode.
> >
> >      I have also added a new callback, on_token_lex (), to libcpp. This is so the
> >      preprocessor can see and stream out all the tokens that pragma_lex () gets
> >      from libcpp, since it won't otherwise see them.  This seemed the simplest
> >      approach to me. Another possibility would be to add a wrapper function in
> >      c-family/c-lex.cc, which would call cpp_get_token_with_location(), and then
> >      also stream the token in preprocess-only mode, and then change all calls
> >      into libcpp in that file to use the wrapper function.  The libcpp callback
> >      seemed cleaner to me FWIW.
>
> I think the other way sounds better to me; there are only three calls to
> cpp_get_... in c_lex_with_flags.
>
> The rest of the patch looks good.

Thank you very much for the feedback. I will test it this way and send
the updated version.

-Lewis