LSP based on GCC

LSP based on GCC

[Eli Zaretskii]

Dear GCC developers,

Emacs 29, to be released soon, will come with a built-in client for
the LSP protocol.  This allows to enhance important Emacs features,
such as at-point documentation, on-the-fly diagnostic annotations,
finding definitions and uses of program identifiers, enhanced
completion of symbols and code, etc., based on capabilities of LSP
servers.

The Emacs LSP client comes with support for many popular LSP servers
OOTB and for all the programming languages supported by Emacs.
However, all the available servers for C and C++ languages are based
on Clang.  AFAIU, this is because GCC does not yet have its own
implementation of the LSP.  I found this message posted to gcc-patches
in 2017:

  https://gcc.gnu.org/legacy-ml/gcc-patches/2017-07/msg01448.html

which described the initial implementation of LSP in GCC, but I seem
to be unable to find out what happened with that since then.

Are there plans for implementing the LSP in GCC?  If so, which GCC
version is expected to have this included?

If there are no current plans for implementing LSP, I hope someone
will work on that soon, given that Emacs can now use it, and because
having a GCC-based LSP implementation will allow people to use their
installed GCC as the basis for LSP features, instead of having to
install yet another compiler.

TIA

Re: LSP based on GCC

[David Malcolm]

On Wed, 2023-05-17 at 17:28 +0300, Eli Zaretskii via Gcc wrote:
> Dear GCC developers,

[CCing Frank, re the systemtap LSP implementation]

Hi Eli

> Emacs 29, to be released soon, will come with a built-in client for
> the LSP protocol.  This allows to enhance important Emacs features,
> such as at-point documentation, on-the-fly diagnostic annotations,
> finding definitions and uses of program identifiers, enhanced
> completion of symbols and code, etc., based on capabilities of LSP
> servers.

Excellent.  Do you have a handy link to the specific LSP methods that
Emacs consumes?

> 
> The Emacs LSP client comes with support for many popular LSP servers
> OOTB and for all the programming languages supported by Emacs.
> However, all the available servers for C and C++ languages are based
> on Clang.  AFAIU, this is because GCC does not yet have its own
> implementation of the LSP.  I found this message posted to gcc-
> patches
> in 2017:
> 
>   https://gcc.gnu.org/legacy-ml/gcc-patches/2017-07/msg01448.html
> 
> which described the initial implementation of LSP in GCC, but I seem
> to be unable to find out what happened with that since then.
> 
> Are there plans for implementing the LSP in GCC?  If so, which GCC
> version is expected to have this included?

I wrote that prototype, but I haven't touched it since 2017, and I
already have more than enough other work, alas.  I'm happy to help if
someone wants to pick up the work and finish it.

That patch kit did several things:
(a) adds a new "on the side" representation of source code locations
(b) adds a JSON implementation to gcc
(c) implements an LSP server on a port, implementing only the
"textDocument/definition" method.

Not having quite enough source code location is a pet peeve of mine
within GCC's intermediate representation, as we lose a fair bit of
location information as we go from frontends to the tree/generic
representation (e.g. "exactly where was each brace?").  Unfortunately
the particular approach I came up with in (a) was rejected by frontend
maintainers, so I abandoned that part of the work.

The part of (b) for storing JSON trees in memory and writing them out
is in GCC's source tree; the JSON-parsing code isn't yet, but I have a
relatively up-to-date refreshed version of that code in one of my
working copies which I can post to the list if it will be helpful.

As for (c), doing it on a port is probably wrong, and working with
stdin/stdout seems a much better approach.


I note that systemtap recently gained an LSP server [1], where the
release notes say:

[begin quote]
* Added a "--language-server" option to stap, which switches it into
  an LSP server on stdin/stdout.  This allows users of a variety of
  editors to trigger symbol-completions and similar operations in
  context.  This code includes a small reusable jsonrpc server library
  implementation in C++.
[end quote]

The code in question appears to be in
https://sourceware.org/git/?p=systemtap.git;a=tree;f=language-server

though I'm not sure to what extent it's compatible with GCC (it's GPLv2
or later, but it seems to use exceptions; also I'm not sure which C++
dialect it uses - we can't go later than C++11 for our implementation
code).


> If there are no current plans for implementing LSP, I hope someone
> will work on that soon, given that Emacs can now use it, and because
> having a GCC-based LSP implementation will allow people to use their
> installed GCC as the basis for LSP features, instead of having to
> install yet another compiler.

Agreed.

BTW, in GCC 13 I implemented SARIF support for capturing diagnostics in
a standardized JSON-based format [2]; see e.g. the "Diagnostic
Serialization" slides in [3] which shows VS Code showing GCC
diagnostics with squiggly underlines.  Has anyone implemented SARIF
support for Emacs yet?  (FWIW I'm now on the SARIF technical committee)

Hope this is helpful
Dave


[1] https://lwn.net/ml/linux-kernel/ZEw5pdqi7y%2FjYqeA@elastic.org/
[2] https://sarifweb.azurewebsites.net/
[3]https://gcc.gnu.org/wiki/cauldron2022?action=AttachFile&do=get&target=2022-Cauldron-analyzer-talk.pdf

Re: LSP based on GCC

[Eli Zaretskii]

> From: David Malcolm <dmalcolm@redhat.com>
> Cc: "Frank Ch. Eigler" <fche@elastic.org>
> Date: Wed, 17 May 2023 11:46:58 -0400
> 
> > Emacs 29, to be released soon, will come with a built-in client for
> > the LSP protocol.  This allows to enhance important Emacs features,
> > such as at-point documentation, on-the-fly diagnostic annotations,
> > finding definitions and uses of program identifiers, enhanced
> > completion of symbols and code, etc., based on capabilities of LSP
> > servers.
> 
> Excellent.  Do you have a handy link to the specific LSP methods that
> Emacs consumes?

IIUC what you mean (I'm not an expert on LSP), then you will find the
answers here:

  https://git.savannah.gnu.org/cgit/emacs.git/tree/lisp/progmodes/eglot.el#n775
  https://git.savannah.gnu.org/cgit/emacs.git/tree/lisp/progmodes/eglot.el#n1721

> >   https://gcc.gnu.org/legacy-ml/gcc-patches/2017-07/msg01448.html
> > 
> > which described the initial implementation of LSP in GCC, but I seem
> > to be unable to find out what happened with that since then.
> > 
> > Are there plans for implementing the LSP in GCC?  If so, which GCC
> > version is expected to have this included?
> 
> I wrote that prototype, but I haven't touched it since 2017, and I
> already have more than enough other work, alas.  I'm happy to help if
> someone wants to pick up the work and finish it.

I hope someone will, thanks.

> BTW, in GCC 13 I implemented SARIF support for capturing diagnostics in
> a standardized JSON-based format [2]; see e.g. the "Diagnostic
> Serialization" slides in [3] which shows VS Code showing GCC
> diagnostics with squiggly underlines.  Has anyone implemented SARIF
> support for Emacs yet?

AFAIK, not yet.  I'll ask on the Emacs devel list if anyone is
interested.

Re: LSP based on GCC

[Paul Smith]

On Wed, 2023-05-17 at 17:28 +0300, Eli Zaretskii via Gcc wrote:
> If there are no current plans for implementing LSP, I hope someone
> will work on that soon, given that Emacs can now use it, and because
> having a GCC-based LSP implementation will allow people to use their
> installed GCC as the basis for LSP features, instead of having to
> install yet another compiler.

Just to note, existing LSP servers are stand-alone binaries: you don't
need to install the compiler.  The two main C/C++ LSP servers I'm aware
of are clangd and ccls.  Both of them are built from libclang, as you
suggest, but you don't need a full clang installation to get them.

You do need to install some Clang "resource" header files for
intrinsics etc. but not the entire compiler (for example you don't need
the STL or anything like that: they do work fine with the GCC STL).

Nevertheless I wholeheartedly agree with your hopes Eli because using
Clang-based LSP is annoying for people developing with GCC:

First, when you're viewing code that is using #ifdefs to choose between
compilers you always see the code for Clang as "active", even though
you're using GCC as the compiler, since it's using the Clang built-in
macros.

More frustratingly, Clang has made some poor decisions around
"compatibility": they tried to leverage the GNU ecosystem by emulating
GCC features and arguments but sometimes break things.  The most
egregious example I'm aware of is that they look for GCC-named
precompiled headers (.gch), even though the Clang PCH format is
completely different.  So if Clang (and the LSP servers built on it)
find a .gch header file they will try to read it, fail, and give an
error.  I filed a bug about this in 2019 but it's been ignored.

This means you need to modify your LSP server arguments to omit any PCH
compiler command line arguments; for environments based on auto-
generated definitions like compile_commands.json this is frustrating.

Re: LSP based on GCC

[Arsen Arsenović]

[-- Attachment #1: Type: text/plain, Size: 2352 bytes --]


David Malcolm <dmalcolm@redhat.com> writes:

> [...snip...]
> I wrote that prototype, but I haven't touched it since 2017, and I
> already have more than enough other work, alas.  I'm happy to help if
> someone wants to pick up the work and finish it.
>
> That patch kit did several things:
> (a) adds a new "on the side" representation of source code locations
> (b) adds a JSON implementation to gcc
> (c) implements an LSP server on a port, implementing only the
> "textDocument/definition" method.
>
> Not having quite enough source code location is a pet peeve of mine
> within GCC's intermediate representation, as we lose a fair bit of
> location information as we go from frontends to the tree/generic
> representation (e.g. "exactly where was each brace?").

You mentioned 'cousin' tools in your original post, and I largely agree
with your sentiment.  Many parts of the job of an FE can be reused for
many other purposes, evidently.  Even beyond an LSP implementation.

> Unfortunately the particular approach I came up with in (a) was
> rejected by frontend maintainers, so I abandoned that part of the
> work.

I couldn't find the relevant messages.  Mind sharing a message-id or
archive link?

> The part of (b) for storing JSON trees in memory and writing them out
> is in GCC's source tree; the JSON-parsing code isn't yet, but I have a
> relatively up-to-date refreshed version of that code in one of my
> working copies which I can post to the list if it will be helpful.
>
> As for (c), doing it on a port is probably wrong, and working with
> stdin/stdout seems a much better approach.

AIUI, many editors (including Emacs' Eglot) also expect this (but I
suspect many can leverage other methods of connecting too).


ISTR Alexandre Oliva (CC added) mentioning leveraging GDB to implement
various bits of LSP functionality, such as handling multiple TUs.  This
sounds like a good idea to me (at least at a high level), as it could
lead to the hypothetical GNU toolchain LSP implementation being
partially language-agnostic (naturally, some things like candidate lists
would still need language support, as well as documentation parsing,
...), which would be quite handy.

Do you happen to have any memory of that?

Thanks in advance, have a lovely evening.
-- 
Arsen Arsenović

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 381 bytes --]

Re: LSP based on GCC

[Ben Boeckel]

On Wed, May 17, 2023 at 15:48:09 -0400, Paul Smith wrote:
> More frustratingly, Clang has made some poor decisions around
> "compatibility": they tried to leverage the GNU ecosystem by emulating
> GCC features and arguments but sometimes break things.  The most

Alas, the cost of trying to make a compiler that can injest in-the-wild
code. It's the reason "every" compiler claims various GCC things: too
many projects ended up with `#error "Unknown compiler"` in their
detections and fixing them when you're just trying to get off the ground
is annoying. As far as I know, GCC is locked into never providing a
single uniquely identifiable trait because other compilers would end up
having to emulate it too once it gets used by projects. CMake basically
just ends up with "it's GCC" if `__GNUC__` is defined and none of the
other, more specific, preprocessor markers are present.

We're kind of getting this again with the variety of different frontends
available on top of Clang these days (Apple's Xcode build, upstream
itself, Intel's frontend, IBM's LLVM-based frontend, the XL-alike Clang
build, Fujitsu has one, ARM's, and who knows how many others are out
there). Sometimes they've…forgotten to make something distinctive so
that it can be detected reliably.

> egregious example I'm aware of is that they look for GCC-named
> precompiled headers (.gch), even though the Clang PCH format is
> completely different.  So if Clang (and the LSP servers built on it)
> find a .gch header file they will try to read it, fail, and give an
> error.  I filed a bug about this in 2019 but it's been ignored.
> 
> This means you need to modify your LSP server arguments to omit any PCH
> compiler command line arguments; for environments based on auto-
> generated definitions like compile_commands.json this is frustrating.

FWIW, this is only going to get worse with C++ modules.

--Ben

Re: LSP based on GCC

[Paul Smith]

On Wed, 2023-05-17 at 18:38 -0400, Ben Boeckel wrote:
> > egregious example I'm aware of is that they look for GCC-named
> > precompiled headers (.gch), even though the Clang PCH format is
> > completely different.  So if Clang (and the LSP servers built on
> > it) find a .gch header file they will try to read it, fail, and
> > give an error.  I filed a bug about this in 2019 but it's been
> > ignored.
> > 
> > This means you need to modify your LSP server arguments to omit any
> > PCH compiler command line arguments; for environments based on
> > auto-generated definitions like compile_commands.json this is
> > frustrating.
> 
> FWIW, this is only going to get worse with C++ modules.

There's no reason it should.  Of course the right answer is to tell
people to fix their build systems and if they want to use a different
compiler AND use PCH, they use the appropriate suffix for that
compiler.

But even if you don't want to do that the fix in this case is trivial.
I even sent a patch (although since I don't know the clang code there's
no doubt that it was not done "the right way" and needed to be
massaged), they just never cared about it.

The GCC PCH files use a special 4-byte prefix in every file; all you
have to do in clang is, if you find a .gch file open the file and read
the first 4 bytes and if it's a real GCC PCH file you ignore it and if
it's actually a Clang PCH with a malformed name you complain bitterly
and dump core.... er, I mean, you read it silently as if it had the
right name.

One would hope that, if the GCC module files have a similar compiler-
specific format (I'm not too familiar with modules) they also use a
similar magic number at the beginning of the file.

But anyway this is losing the thread of Eli's hopeful request.

Re: LSP based on GCC

[Jason Merrill]

[-- Attachment #1: Type: text/plain, Size: 1730 bytes --]

On Wed, May 17, 2023 at 11:47 AM David Malcolm via Gcc <gcc@gcc.gnu.org>
wrote:

> On Wed, 2023-05-17 at 17:28 +0300, Eli Zaretskii via Gcc wrote:
> > Dear GCC developers,
>
> [CCing Frank, re the systemtap LSP implementation]
>
> Hi Eli
>
> > Emacs 29, to be released soon, will come with a built-in client for
> > the LSP protocol.  This allows to enhance important Emacs features,
> > such as at-point documentation, on-the-fly diagnostic annotations,
> > finding definitions and uses of program identifiers, enhanced
> > completion of symbols and code, etc., based on capabilities of LSP
> > servers.
>
> Excellent.  Do you have a handy link to the specific LSP methods that
> Emacs consumes?
>
> >
> > The Emacs LSP client comes with support for many popular LSP servers
> > OOTB and for all the programming languages supported by Emacs.
> > However, all the available servers for C and C++ languages are based
> > on Clang.  AFAIU, this is because GCC does not yet have its own
> > implementation of the LSP.  I found this message posted to gcc-
> > patches
> > in 2017:
> >
> >   https://gcc.gnu.org/legacy-ml/gcc-patches/2017-07/msg01448.html
> >
> > which described the initial implementation of LSP in GCC, but I seem
> > to be unable to find out what happened with that since then.
> >
> > Are there plans for implementing the LSP in GCC?  If so, which GCC
> > version is expected to have this included?
>
> I wrote that prototype, but I haven't touched it since 2017, and I
> already have more than enough other work, alas.  I'm happy to help if
> someone wants to pick up the work and finish it.
>

Maybe create a BZ and gcc.gnu.org/projects entry for this work?

Jason

Re: LSP based on GCC

[Ben Boeckel]

On Thu, May 18, 2023 at 09:25:04 -0400, Paul Smith wrote:
> On Wed, 2023-05-17 at 18:38 -0400, Ben Boeckel wrote:
> > FWIW, this is only going to get worse with C++ modules.
> 
> There's no reason it should.  Of course the right answer is to tell
> people to fix their build systems and if they want to use a different
> compiler AND use PCH, they use the appropriate suffix for that
> compiler.
> 
> But even if you don't want to do that the fix in this case is trivial.
> I even sent a patch (although since I don't know the clang code there's
> no doubt that it was not done "the right way" and needed to be
> massaged), they just never cared about it.
> 
> The GCC PCH files use a special 4-byte prefix in every file; all you
> have to do in clang is, if you find a .gch file open the file and read
> the first 4 bytes and if it's a real GCC PCH file you ignore it and if
> it's actually a Clang PCH with a malformed name you complain bitterly
> and dump core.... er, I mean, you read it silently as if it had the
> right name.

PCH files can "be ignored" in some sense because they can be
recalculated from `#include` files pretty easily. Module files, however,
cannot.

> One would hope that, if the GCC module files have a similar compiler-
> specific format (I'm not too familiar with modules) they also use a
> similar magic number at the beginning of the file.

GCC module files are use ELF containers, so there's plenty of metadata
to know it's not-for-Clang. But Clang will need to make its own version
of these module files to know what, if anything, is provided by it by
sources that import it to make any kind of useful suggestions.

> But anyway this is losing the thread of Eli's hopeful request.

Agreed. A GCC-based LSP will help immensely with GCC-using projects
(whether it be Emacs or Vim on the other end of the LSP pipe ;) ).

--Ben

Re: LSP based on GCC

[Paul Smith]

On Thu, 2023-05-18 at 10:52 -0400, Ben Boeckel wrote:
> PCH files can "be ignored" in some sense because they can be
> recalculated from `#include` files pretty easily. Module files,
> however, cannot.

This makes it even more important that there be a GCC-based LSP server,
if a Clang-based one can't read GCC module files AND module files are
required to exist.  Without a GCC-based LSP server, it means you have
to build your modules for _both_ GCC (for compilation) _and_ Clang (for
LSP) which would be extremely frustrating.

It would also likely make Eli's initial contention that you need to
install a full Clang compiler to use a Clang-based LSP server, into a
reality.

Re: LSP based on GCC

[Alexandre Oliva]

On May 17, 2023, Arsen Arsenović <arsen@aarsen.me> wrote:

> ISTR Alexandre Oliva (CC added) mentioning leveraging GDB to implement
> various bits of LSP functionality, such as handling multiple TUs.  This
> sounds like a good idea to me (at least at a high level), as it could
> lead to the hypothetical GNU toolchain LSP implementation being
> partially language-agnostic (naturally, some things like candidate lists
> would still need language support, as well as documentation parsing,
> ...), which would be quite handy.

> Do you happen to have any memory of that?

I recall advancing that suggestion, reasoning that GDB was capable of
combining information from multiple translation units and of reloading
debug information, which GCC doesn't.  But that was mainly a hunch that
it could work, not something based on extensive knowledge of GDB or LSP.
Details were all yet to be figured out.  I expected there'd be a need
for additional debug information to be emitted.  I wasn't sure how to
approach the issue of translation units that wouldn't compile or link
successfully yet.  There are many thorny issues to be sorted out.

-- 
Alexandre Oliva, happy hacker                https://FSFLA.org/blogs/lxo/
   Free Software Activist                       GNU Toolchain Engineer
Disinformation flourishes because many people care deeply about injustice
but very few check the facts.  Ask me about <https://stallmansupport.org>

Re: LSP based on GCC

[Paul Smith]

On Mon, 2023-05-29 at 17:16 -0300, Alexandre Oliva via Gcc wrote:
> On May 17, 2023, Arsen Arsenović <arsen@aarsen.me> wrote:
> 
> > ISTR Alexandre Oliva (CC added) mentioning leveraging GDB to
> > implement various bits of LSP functionality, such as handling
> > multiple TUs.
> 
> I recall advancing that suggestion, reasoning that GDB was capable of
> combining information from multiple translation units and of
> reloading debug information, which GCC doesn't.

I'm not sure this will work well.  The information LSP servers need is
quite a bit more detailed (I believe) than what is found in the object
file debug sections.

Typically LSP servers generate their own index files, which are
completely separate from any compiler-generated output.  When the
server starts it will proceed to parse and index the entire codebase
(which is unfortunately slow) then they try to keep that index updated
over time as code changes.  So the LSP server will use the compiler
front-end to parse the code and keep information about symbols and
locations separately.

LSP servers are not intended to be limited to dealing with the code
that has already been compiled: not only do you want to be able to edit
code before it's been compiled, but you want to be able to query new
code as it's written in the editor, before it's even saved to disk.

I recommend that anyone who is interested in this project, examine the
LSP spec to understand what information the server needs to deal with:

https://microsoft.github.io/language-server-protocol/specifications/lsp/3.17/specification/

Re: LSP based on GCC

[Alexandre Oliva]

On May 30, 2023, Paul Smith <paul@mad-scientist.net> wrote:

> On Mon, 2023-05-29 at 17:16 -0300, Alexandre Oliva via Gcc wrote:
>> On May 17, 2023, Arsen Arsenović <arsen@aarsen.me> wrote:
>> 
>> > ISTR Alexandre Oliva (CC added) mentioning leveraging GDB to
>> > implement various bits of LSP functionality, such as handling
>> > multiple TUs.
>> 
>> I recall advancing that suggestion, reasoning that GDB was capable of
>> combining information from multiple translation units and of
>> reloading debug information, which GCC doesn't.

> I'm not sure this will work well.  The information LSP servers need is
> quite a bit more detailed (I believe) than what is found in the object
> file debug sections.

It was well understood that augmenting debug information for LSP
purposes would likely be needed, and that GDB's ability to combine debug
information from multiple units would be advantageous to this end as
well.

An mode similar to thin LTO, that outputs internal representation along
with debug information, rather than compiling all the way to machine
code, was also in my mind.  That's similar to the indexing you mention.

> LSP servers are not intended to be limited to dealing with the code
> that has already been compiled: not only do you want to be able to edit
> code before it's been compiled, but you want to be able to query new
> code as it's written in the editor, before it's even saved to disk.

ISTM there's some miscommunication here.  Using the compiler's parser
and debug information generator to index a project is no more
"compilation" than using the linker's ability to integrate multiple
translation units and the debugger's ability to take debug information
from them all to aid editing and other LSP activities amounts to
"debugging", it's just using each project's strengths to achieve the
desired goal.

-- 
Alexandre Oliva, happy hacker                https://FSFLA.org/blogs/lxo/
   Free Software Activist                       GNU Toolchain Engineer
Disinformation flourishes because many people care deeply about injustice
but very few check the facts.  Ask me about <https://stallmansupport.org>