How to look up where a structure is defined?

How to look up where a structure is defined?

[Peng Yu]

Hi,

I would like to lookup types and constants in libc. It would be most
efficient to have a database lookup for this purpose instead of using
google search or code search.

Is there such a database available? Thanks.

-- 
Regards,
Peng

Re: How to look up where a structure is defined?

[Konstantin Kharlamov]

On Wed, 2021-03-03 at 08:12 -0600, Peng Yu via Libc-help wrote:
> Hi,
> 
> I would like to lookup types and constants in libc. It would be most
> efficient to have a database lookup for this purpose instead of using
> google search or code search.
> 
> Is there such a database available? Thanks.
> 

I assume you're referring to the usual "go to definition" functional in code editors/IDE. In that case I don't think glibc is too different from other projects. The simplest (but not very good) way is using Universal Ctags, and generating a tags file from the source tree, and then feeding that into your editor/IDE. I don't know if it's good enough for you: it should work for basic navigation, but wouldn't provide you with navigation inside functions, or for various complex constructions out of macros.

Nowadays most popular way is lsp-server, should be supported by most existing modern IDEs/editors, including vim/emacs (you might need to install a plugin). For C/C++ projects I usually use `clangd` as the lsp-server. You'll stumble upon a gotcha though: Glibc is a very old project, that is still using autotools/make. So, unlike Meson-based projects, you'll need to generate `compile_commands.json` file yourself. It should be easy though: there's `bear` utility, which takes a make command as input, and parses it output while it builds the project, and stores the output into `compile_commands.json` file. So for example: if you usually build glibc with `make`, then to generate the file you have to use a `bear -- make` instead.

Re: How to look up where a structure is defined?

[Konstantin Kharlamov]

On Wed, 2021-03-03 at 17:26 +0300, Konstantin Kharlamov wrote:
> there's `bear` utility, which takes a make command as input, and parses it
> output while it builds the project, and stores the output into
> `compile_commands.json` file. So for example: if you usually build glibc with
> `make`, then to generate the file you have to use a `bear -- make` instead.

To clarify: if you already built the project you'll need somehow to re-build it
with bear. So, for example, you can call a `find -type f -exec touch {} \;` in
the source tree to cause timestamps of all files to get bumped, and then running
a make command will trigger a full rebuild.

Re: How to look up where a structure is defined?

[Peng Yu]

This seems to be a complicated solution. I just want to get a database
(a TSV file should be fine) of types and the header they appear. I
don't want to build the project just to get this info.

It seems that this database could be made and hosted online for easy
download or lookup.

On Wed, Mar 3, 2021 at 8:55 AM Konstantin Kharlamov <hi-angel@yandex.ru> wrote:
>
> On Wed, 2021-03-03 at 17:26 +0300, Konstantin Kharlamov wrote:
> > there's `bear` utility, which takes a make command as input, and parses it
> > output while it builds the project, and stores the output into
> > `compile_commands.json` file. So for example: if you usually build glibc with
> > `make`, then to generate the file you have to use a `bear -- make` instead.
>
> To clarify: if you already built the project you'll need somehow to re-build it
> with bear. So, for example, you can call a `find -type f -exec touch {} \;` in
> the source tree to cause timestamps of all files to get bumped, and then running
> a make command will trigger a full rebuild.

-- 
Regards,
Peng

Re: How to look up where a structure is defined?

[Konstantin Kharlamov]

On Wed, 2021-03-03 at 13:51 -0600, Peng Yu wrote:
> This seems to be a complicated solution. I just want to get a database
> (a TSV file should be fine) of types and the header they appear. I
> don't want to build the project just to get this info.

I see, well, the Universal Ctags I mentioned should work for you. It doesn't
require building the project: you just run `ctags -R` or `ctags -Re` (first for
vim-style tags file, second one for emacs-style) over the repository, and you
get a `tags` or `TAGS` file with a list of definitions.

Possible drawbacks on ctags I mentioned in the other email. Basically it's that
it doesn't take context into consideration.

Regarding usage: the tags file it generates, although can be read for human,
supposed to be read by text editors/IDEs. Since you mention a CSV file, I assume
you might want something human-readable. Please see option --output-format= in
`man ctags` for details: I think you might want the `xref` format. (I never
tried it myself, just reading the man it seems like it what you're after). 

> It seems that this database could be made and hosted online for easy
> download or lookup.

Yeah, something like a https://elixir.bootlin.com/linux/v5.12-rc1/source site
(which allows to navigate Linux kernel sources online), but for glibc — I agree,
would be nice!

> On Wed, Mar 3, 2021 at 8:55 AM Konstantin Kharlamov <hi-angel@yandex.ru> wrote:
> > 
> > On Wed, 2021-03-03 at 17:26 +0300, Konstantin Kharlamov wrote:
> > > there's `bear` utility, which takes a make command as input, and parses it
> > > output while it builds the project, and stores the output into
> > > `compile_commands.json` file. So for example: if you usually build glibc
> > > with
> > > `make`, then to generate the file you have to use a `bear -- make` instead.
> > 
> > To clarify: if you already built the project you'll need somehow to re-build
> > it
> > with bear. So, for example, you can call a `find -type f -exec touch {} \;` in
> > the source tree to cause timestamps of all files to get bumped, and then
> > running
> > a make command will trigger a full rebuild.
> 

Re: How to look up where a structure is defined?

[Konstantin Kharlamov]

On Wed, 2021-03-03 at 23:09 +0300, Konstantin Kharlamov wrote:
> On Wed, 2021-03-03 at 13:51 -0600, Peng Yu wrote:
> > This seems to be a complicated solution. I just want to get a database
> > (a TSV file should be fine) of types and the header they appear. I
> > don't want to build the project just to get this info.
> 
> I see, well, the Universal Ctags I mentioned should work for you. It doesn't
> require building the project: you just run `ctags -R` or `ctags -Re` (first
> for
> vim-style tags file, second one for emacs-style) over the repository, and you
> get a `tags` or `TAGS` file with a list of definitions.
> 
> Possible drawbacks on ctags I mentioned in the other email. Basically it's
> that
> it doesn't take context into consideration.
> 
> Regarding usage: the tags file it generates, although can be read for human,
> supposed to be read by text editors/IDEs. Since you mention a CSV file, I
> assume
> you might want something human-readable. Please see option --output-format= in
> `man ctags` for details: I think you might want the `xref` format. (I never
> tried it myself, just reading the man it seems like it what you're after). 

Although, I wouldn't hold my breath that reading a resulting xref file would be easy ☺ The reason being is that I expect a tags file created from glibc repo to be some hundreds of megabytes. For reference, a TAGS file I generated long ago for libreoffice project is sized at 183M.

So yeah, you will probably want to use the file from an IDE or text editor, rather than reading it manually.

Re: How to look up where a structure is defined?

[Peng Yu]

On Wed, Mar 3, 2021 at 2:23 PM Konstantin Kharlamov <hi-angel@yandex.ru> wrote:
>
> On Wed, 2021-03-03 at 23:09 +0300, Konstantin Kharlamov wrote:
> > On Wed, 2021-03-03 at 13:51 -0600, Peng Yu wrote:
> > > This seems to be a complicated solution. I just want to get a database
> > > (a TSV file should be fine) of types and the header they appear. I
> > > don't want to build the project just to get this info.
> >
> > I see, well, the Universal Ctags I mentioned should work for you. It doesn't
> > require building the project: you just run `ctags -R` or `ctags -Re` (first
> > for
> > vim-style tags file, second one for emacs-style) over the repository, and you
> > get a `tags` or `TAGS` file with a list of definitions.
> >
> > Possible drawbacks on ctags I mentioned in the other email. Basically it's
> > that
> > it doesn't take context into consideration.
> >
> > Regarding usage: the tags file it generates, although can be read for human,
> > supposed to be read by text editors/IDEs. Since you mention a CSV file, I
> > assume
> > you might want something human-readable. Please see option --output-format= in
> > `man ctags` for details: I think you might want the `xref` format. (I never
> > tried it myself, just reading the man it seems like it what you're after).
>
> Although, I wouldn't hold my breath that reading a resulting xref file would be easy The reason being is that I expect a tags file created from glibc repo to be some hundreds of megabytes. For reference, a TAGS file I generated long ago for libreoffice project is sized at 183M.
>
> So yeah, you will probably want to use the file from an IDE or text editor, rather than reading it manually.

ctags can partially solve the problem. The declaration and definition
of a struct are all in the header. So finding the definition is OK.
But it can not return function declaration. Is there a way to show
function declaration as well? (For example, ./socket/sys/socket.h for
setsockopt().)

$ grep '^icmphdr\>' tags
icmphdr sysdeps/gnu/netinet/ip_icmp.h /^struct icmphdr$/;" s
$ grep ^setsockopt tags
setsockopt sysdeps/unix/sysv/linux/setsockopt.c /^setsockopt (int fd,
int level, int optname, const void *optval, socklen_t len)$/;" f

./socket/sys/socket.h
215:extern int setsockopt (int __fd, int __level, int __optname,

-- 
Regards,
Peng

Re: How to look up where a structure is defined?

[Konstantin Kharlamov]

On Wed, 2021-03-03 at 16:24 -0600, Peng Yu wrote:
> On Wed, Mar 3, 2021 at 2:23 PM Konstantin Kharlamov <hi-angel@yandex.ru>
> wrote:
> > 
> > On Wed, 2021-03-03 at 23:09 +0300, Konstantin Kharlamov wrote:
> > > On Wed, 2021-03-03 at 13:51 -0600, Peng Yu wrote:
> > > > This seems to be a complicated solution. I just want to get a database
> > > > (a TSV file should be fine) of types and the header they appear. I
> > > > don't want to build the project just to get this info.
> > > 
> > > I see, well, the Universal Ctags I mentioned should work for you. It
> > > doesn't
> > > require building the project: you just run `ctags -R` or `ctags -Re`
> > > (first
> > > for
> > > vim-style tags file, second one for emacs-style) over the repository, and
> > > you
> > > get a `tags` or `TAGS` file with a list of definitions.
> > > 
> > > Possible drawbacks on ctags I mentioned in the other email. Basically it's
> > > that
> > > it doesn't take context into consideration.
> > > 
> > > Regarding usage: the tags file it generates, although can be read for
> > > human,
> > > supposed to be read by text editors/IDEs. Since you mention a CSV file, I
> > > assume
> > > you might want something human-readable. Please see option --output-
> > > format= in
> > > `man ctags` for details: I think you might want the `xref` format. (I
> > > never
> > > tried it myself, just reading the man it seems like it what you're after).
> > 
> > Although, I wouldn't hold my breath that reading a resulting xref file would
> > be easy The reason being is that I expect a tags file created from glibc
> > repo to be some hundreds of megabytes. For reference, a TAGS file I
> > generated long ago for libreoffice project is sized at 183M.
> > 
> > So yeah, you will probably want to use the file from an IDE or text editor,
> > rather than reading it manually.
> 
> ctags can partially solve the problem. The declaration and definition
> of a struct are all in the header. So finding the definition is OK.
> But it can not return function declaration. Is there a way to show
> function declaration as well? (For example, ./socket/sys/socket.h for
> setsockopt().)
> 
> $ grep '^icmphdr\>' tags
> icmphdr sysdeps/gnu/netinet/ip_icmp.h /^struct icmphdr$/;" s
> $ grep ^setsockopt tags
> setsockopt sysdeps/unix/sysv/linux/setsockopt.c /^setsockopt (int fd,
> int level, int optname, const void *optval, socklen_t len)$/;" f
> 
> ./socket/sys/socket.h
> 215:extern int setsockopt (int __fd, int __level, int __optname,

But, in the grep output you show you have both the declaration and the definition of setsockopt, no? In particular, you have the declaration from `socket.h` file.

I think you're confused because your editor/IDE navigates by default to the first match in the tags file, which probably happened to be the definition. I know vim does that in particular. What to do here is depends on the editor/IDE you're using. For vim specifically there is some command to go to the next tag (I think it's `:tn`).

In Emacs it is implemented in a better way: it shows you a list of all matches when there's more than one, and you get to chose which one you want. Idk, perhaps there's a plugin that implements similar behavior for vim…? Anyway, point being is that the answer depends on what you're using with the tags file.

On a side note, you do need to figure it out for your editor/IDE, because, for example, with struct fields you might get tons of matches, depending on circumstances. Remember, tags does not include context. So if project has multiple structs with a field `int foo;`, all of them will match when you'll try navigate to `foo` by using tags.

Re: How to look up where a structure is defined?

[Peng Yu]

On 3/3/21, Konstantin Kharlamov <hi-angel@yandex.ru> wrote:
> On Wed, 2021-03-03 at 16:24 -0600, Peng Yu wrote:
>> On Wed, Mar 3, 2021 at 2:23 PM Konstantin Kharlamov <hi-angel@yandex.ru>
>> wrote:
>> >
>> > On Wed, 2021-03-03 at 23:09 +0300, Konstantin Kharlamov wrote:
>> > > On Wed, 2021-03-03 at 13:51 -0600, Peng Yu wrote:
>> > > > This seems to be a complicated solution. I just want to get a
>> > > > database
>> > > > (a TSV file should be fine) of types and the header they appear. I
>> > > > don't want to build the project just to get this info.
>> > >
>> > > I see, well, the Universal Ctags I mentioned should work for you. It
>> > > doesn't
>> > > require building the project: you just run `ctags -R` or `ctags -Re`
>> > > (first
>> > > for
>> > > vim-style tags file, second one for emacs-style) over the repository,
>> > > and
>> > > you
>> > > get a `tags` or `TAGS` file with a list of definitions.
>> > >
>> > > Possible drawbacks on ctags I mentioned in the other email. Basically
>> > > it's
>> > > that
>> > > it doesn't take context into consideration.
>> > >
>> > > Regarding usage: the tags file it generates, although can be read for
>> > > human,
>> > > supposed to be read by text editors/IDEs. Since you mention a CSV
>> > > file, I
>> > > assume
>> > > you might want something human-readable. Please see option --output-
>> > > format= in
>> > > `man ctags` for details: I think you might want the `xref` format. (I
>> > > never
>> > > tried it myself, just reading the man it seems like it what you're
>> > > after).
>> >
>> > Although, I wouldn't hold my breath that reading a resulting xref file
>> > would
>> > be easy The reason being is that I expect a tags file created from
>> > glibc
>> > repo to be some hundreds of megabytes. For reference, a TAGS file I
>> > generated long ago for libreoffice project is sized at 183M.
>> >
>> > So yeah, you will probably want to use the file from an IDE or text
>> > editor,
>> > rather than reading it manually.
>>
>> ctags can partially solve the problem. The declaration and definition
>> of a struct are all in the header. So finding the definition is OK.
>> But it can not return function declaration. Is there a way to show
>> function declaration as well? (For example, ./socket/sys/socket.h for
>> setsockopt().)
>>
>> $ grep '^icmphdr\>' tags
>> icmphdr sysdeps/gnu/netinet/ip_icmp.h /^struct icmphdr$/;" s
>> $ grep ^setsockopt tags
>> setsockopt sysdeps/unix/sysv/linux/setsockopt.c /^setsockopt (int fd,
>> int level, int optname, const void *optval, socklen_t len)$/;" f

ctags only shows the above. The following is what I show where the
declaration is.

>> ./socket/sys/socket.h
>> 215:extern int setsockopt (int __fd, int __level, int __optname,
>
> But, in the grep output you show you have both the declaration and the
> definition of setsockopt, no? In particular, you have the declaration from
> `socket.h` file.
>
> I think you're confused because your editor/IDE navigates by default to the
> first match in the tags file, which probably happened to be the definition.
> I know vim does that in particular. What to do here is depends on the
> editor/IDE you're using. For vim specifically there is some command to go to
> the next tag (I think it's `:tn`).
>
> In Emacs it is implemented in a better way: it shows you a list of all
> matches when there's more than one, and you get to chose which one you want.
> Idk, perhaps there's a plugin that implements similar behavior for vim…?
> Anyway, point being is that the answer depends on what you're using with the
> tags file.
>
> On a side note, you do need to figure it out for your editor/IDE, because,
> for example, with struct fields you might get tons of matches, depending on
> circumstances. Remember, tags does not include context. So if project has
> multiple structs with a field `int foo;`, all of them will match when you'll
> try navigate to `foo` by using tags.
>
>


-- 
Regards,
Peng

Re: How to look up where a structure is defined?

[Konstantin Kharlamov]

On Wed, 2021-03-03 at 16:46 -0600, Peng Yu wrote:
> On 3/3/21, Konstantin Kharlamov <hi-angel@yandex.ru> wrote:
> > On Wed, 2021-03-03 at 16:24 -0600, Peng Yu wrote:
> > > On Wed, Mar 3, 2021 at 2:23 PM Konstantin Kharlamov <hi-angel@yandex.ru>
> > > wrote:
> > > > 
> > > > On Wed, 2021-03-03 at 23:09 +0300, Konstantin Kharlamov wrote:
> > > > > On Wed, 2021-03-03 at 13:51 -0600, Peng Yu wrote:
> > > > > > This seems to be a complicated solution. I just want to get a
> > > > > > database
> > > > > > (a TSV file should be fine) of types and the header they appear. I
> > > > > > don't want to build the project just to get this info.
> > > > > 
> > > > > I see, well, the Universal Ctags I mentioned should work for you. It
> > > > > doesn't
> > > > > require building the project: you just run `ctags -R` or `ctags -Re`
> > > > > (first
> > > > > for
> > > > > vim-style tags file, second one for emacs-style) over the repository,
> > > > > and
> > > > > you
> > > > > get a `tags` or `TAGS` file with a list of definitions.
> > > > > 
> > > > > Possible drawbacks on ctags I mentioned in the other email. Basically
> > > > > it's
> > > > > that
> > > > > it doesn't take context into consideration.
> > > > > 
> > > > > Regarding usage: the tags file it generates, although can be read for
> > > > > human,
> > > > > supposed to be read by text editors/IDEs. Since you mention a CSV
> > > > > file, I
> > > > > assume
> > > > > you might want something human-readable. Please see option --output-
> > > > > format= in
> > > > > `man ctags` for details: I think you might want the `xref` format. (I
> > > > > never
> > > > > tried it myself, just reading the man it seems like it what you're
> > > > > after).
> > > > 
> > > > Although, I wouldn't hold my breath that reading a resulting xref file
> > > > would
> > > > be easy The reason being is that I expect a tags file created from
> > > > glibc
> > > > repo to be some hundreds of megabytes. For reference, a TAGS file I
> > > > generated long ago for libreoffice project is sized at 183M.
> > > > 
> > > > So yeah, you will probably want to use the file from an IDE or text
> > > > editor,
> > > > rather than reading it manually.
> > > 
> > > ctags can partially solve the problem. The declaration and definition
> > > of a struct are all in the header. So finding the definition is OK.
> > > But it can not return function declaration. Is there a way to show
> > > function declaration as well? (For example, ./socket/sys/socket.h for
> > > setsockopt().)
> > > 
> > > $ grep '^icmphdr\>' tags
> > > icmphdr sysdeps/gnu/netinet/ip_icmp.h /^struct icmphdr$/;" s
> > > $ grep ^setsockopt tags
> > > setsockopt sysdeps/unix/sysv/linux/setsockopt.c /^setsockopt (int fd,
> > > int level, int optname, const void *optval, socklen_t len)$/;" f
> 
> ctags only shows the above. The following is what I show where the
> declaration is.

Ah, I see. Okay, so, I think including declarations by default is disabled, you probably need to enable it. There're various options on what to include in the tags file, see `ctags --list-kinds-full`, and then using an option kinda like `--kinds-C=+px` (p and x are letters from the --list-kinds-full) should work.

Re: How to look up where a structure is defined?

[Peng Yu]

On Wed, Mar 3, 2021 at 5:10 PM Konstantin Kharlamov <hi-angel@yandex.ru> wrote:
>
> On Wed, 2021-03-03 at 17:04 -0600, Peng Yu wrote:
> > $ ctags --c-kinds=+p -R -o - | grep '^setsockopt\>'
> > setsockopt      sysdeps/unix/sysv/linux/setsockopt.c    /^setsockopt (int fd,
> > int level, int optname, const void *optval, socklen_t len)$/;"  f
> >
> > ctags can not recognize the following line because there is a __THROW
> > at the end. Is there any other walkaround?
> >
> > https://github.com/bminor/glibc/blob/21c3f4b5368686ade28d90d8c7d79c4c95c72c1b/socket/sys/socket.h#L215
>
> So, when I run ctags over that file locally, I do get the definition, see:
>
>      λ ctags --c-kinds=+p -o - socket.h | grep '^setsockopt\>'
>     setsockopt      socket.h        /^extern int setsockopt (int __fd, int __level, int __optname,$/;"      p       typeref:typename:int
>
> Do you perhaps happen to have some old version of ctags? Mine is:
>
>  λ ctags --version
> Universal Ctags 5.9.0(p5.9.20210110.0), Copyright (C) 2015 Universal Ctags Team
> Universal Ctags is derived from Exuberant Ctags.
> Exuberant Ctags 5.8, Copyright (C) 1996-2009 Darren Hiebert
>   Compiled: Jan 15 2021, 00:04:58
>   URL: https://ctags.io/
>   Optional compiled features: +wildcards, +regex, +iconv, +option-directory, +xpath, +json, +interactive, +sandbox, +yaml, +packcc

OK. I installed the same version of ctags. Now, it can find it. Thanks.

$ ctags --version
Universal Ctags 5.9.0(0071282), Copyright (C) 2015 Universal Ctags Team
Universal Ctags is derived from Exuberant Ctags.
Exuberant Ctags 5.8, Copyright (C) 1996-2009 Darren Hiebert
  Compiled: Mar  3 2021, 17:15:45
  URL: https://ctags.io/
  Optional compiled features: +wildcards, +regex, +iconv,
+option-directory, +xpath, +json, +interactive, +yaml,
+case-insensitive-filenames, +packcc
$ ctags --c-kinds=+p -o - socket.h | grep '^setsockopt\>'
setsockopt    socket.h    /^extern int setsockopt (int __fd, int
__level, int __optname,$/;"    p    typeref:typename:int

-- 
Regards,
Peng