abidw for non-exported structures

abidw for non-exported structures

[Ketan Patil]

Hi,

I was experimenting with abidw and abidiff tools to check ABI changes in linux kernel.
I had performed few experiments:

  1.  Changing exported structures:
     *   I changed few structures (added some dummy members in between) which were exported and build the kernel and created abi xml file using abidw tool.
     *   Then I compared this xml with the baseline xml (kernel without any changes) using abidiff tool and I found that it catches this ABI change.
  2.  Changing non-exported structures:
     *   I changed few non-exported structures e.g. struct dm_ioctl In https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/include/uapi/linux/dm-ioctl.h?h=v5.10.18
     *   This struct is being used by a number of ioctls. So I think, changing this struct should be flagged as an ABI change.
     *   Then I build the kernel and created the new abi xml file using abidw tool and compared with baseline xml using abidiff tool.
     *   abidiff tool is not able to catch this. Also, I don't see any entry for dm_ioctl in the xml generated using abidw.


Does it mean that, abidw tool does not work for the non exported symbols? If yes, are you planning to add this feature?
Can you please let me know or redirect?

Thanks,
Ketan Patil

Re: abidw for non-exported structures

[Matthias Maennich]

Hi!

On Thu, Feb 25, 2021 at 11:25:53AM +0000, Ketan Patil via Libabigail wrote:
>Hi,
>
>I was experimenting with abidw and abidiff tools to check ABI changes in linux kernel.
>I had performed few experiments:
>
>  1.  Changing exported structures:
>     *   I changed few structures (added some dummy members in between) which were exported and build the kernel and created abi xml file using abidw tool.
>     *   Then I compared this xml with the baseline xml (kernel without any changes) using abidiff tool and I found that it catches this ABI change.
>  2.  Changing non-exported structures:
>     *   I changed few non-exported structures e.g. struct dm_ioctl In https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/include/uapi/linux/dm-ioctl.h?h=v5.10.18
>     *   This struct is being used by a number of ioctls. So I think, changing this struct should be flagged as an ABI change.
>     *   Then I build the kernel and created the new abi xml file using abidw tool and compared with baseline xml using abidiff tool.
>     *   abidiff tool is not able to catch this. Also, I don't see any entry for dm_ioctl in the xml generated using abidw.
>
>
>Does it mean that, abidw tool does not work for the non exported symbols? If yes, are you planning to add this feature?
>Can you please let me know or redirect?

libabigail works roughly like this (in its default mode): It reads
symbol information from the ELF symbol table and matches symbols with
corresponding DWARF information to gather type information associated
with it. The type information is recursively resolved to capture nested
types. So, fundamentally, to capture and eventually report a structure
in the ABI, it needs to be contributing to the definition of an exported
symbol. E.g. consider this snippet.

   | enum my_enum {
   |   VALUE, ANOTHER_VALUE
   | };
   |
   | struct my_struct {
   |   // some interesting bits
   | };
   |
   | struct another_struct;
   |
   | // consider publicly exported function
   | int my_function(int my_enum, struct my_struct s1, struct another_struct* s2) {
   |   if (my_enum == VALUE) {
   |     // something
   |   }
   | }

The enum technically participates in the ABI, but this is invisible for
libabigail as the type information is not connected. As opposed to
my_struct. This is part of the type information of my_function and
changes therefore would be discoverable. For the forward declared
another_struct, we have again the case that changes to this data
structure can't be captured as the type is passed by reference and the
details are hidden from the ABI. Not even we know anything further
about this struct. If my_function is not exported at all (i.e. not part
of the observable ABI), any analysis will ignore it, entirely and
rightfully.

For the Linux kernel there is an additional fact to consider. In the
default way (without using whitelists), symbols are considered part of
the ABI (or in this case Kernel Module Interface (KMI)), if they are
exported via EXPORT_SYMBOL* macros and therefore part of the ksymtab.

If a data structure is not participating in the type information of an
exported symbol, changes to it will not be discoverable. In order to
discover changes to those, you need to make them discoverable, e.g. by
exporting a function that takes 'struct dm_ioctl' as an argument type.

Cheers,
Matthias

>
>Thanks,
>Ketan Patil
>

RE: abidw for non-exported structures

[Ketan Patil]

Thanks a lot Matthias for the detailed answer.


-----Original Message-----
From: Matthias Maennich <maennich@google.com> 
Sent: Thursday, February 25, 2021 7:11 PM
To: Ketan Patil <ketanp@nvidia.com>
Cc: libabigail@sourceware.org; Sachin Nikam <Snikam@nvidia.com>; Nicolin Chen <nicolinc@nvidia.com>
Subject: Re: abidw for non-exported structures

External email: Use caution opening links or attachments


Hi!

On Thu, Feb 25, 2021 at 11:25:53AM +0000, Ketan Patil via Libabigail wrote:
>Hi,
>
>I was experimenting with abidw and abidiff tools to check ABI changes in linux kernel.
>I had performed few experiments:
>
>  1.  Changing exported structures:
>     *   I changed few structures (added some dummy members in between) which were exported and build the kernel and created abi xml file using abidw tool.
>     *   Then I compared this xml with the baseline xml (kernel without any changes) using abidiff tool and I found that it catches this ABI change.
>  2.  Changing non-exported structures:
>     *   I changed few non-exported structures e.g. struct dm_ioctl In https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/include/uapi/linux/dm-ioctl.h?h=v5.10.18
>     *   This struct is being used by a number of ioctls. So I think, changing this struct should be flagged as an ABI change.
>     *   Then I build the kernel and created the new abi xml file using abidw tool and compared with baseline xml using abidiff tool.
>     *   abidiff tool is not able to catch this. Also, I don't see any entry for dm_ioctl in the xml generated using abidw.
>
>
>Does it mean that, abidw tool does not work for the non exported symbols? If yes, are you planning to add this feature?
>Can you please let me know or redirect?

libabigail works roughly like this (in its default mode): It reads symbol information from the ELF symbol table and matches symbols with corresponding DWARF information to gather type information associated with it. The type information is recursively resolved to capture nested types. So, fundamentally, to capture and eventually report a structure in the ABI, it needs to be contributing to the definition of an exported symbol. E.g. consider this snippet.

   | enum my_enum {
   |   VALUE, ANOTHER_VALUE
   | };
   |
   | struct my_struct {
   |   // some interesting bits
   | };
   |
   | struct another_struct;
   |
   | // consider publicly exported function
   | int my_function(int my_enum, struct my_struct s1, struct another_struct* s2) {
   |   if (my_enum == VALUE) {
   |     // something
   |   }
   | }

The enum technically participates in the ABI, but this is invisible for libabigail as the type information is not connected. As opposed to my_struct. This is part of the type information of my_function and changes therefore would be discoverable. For the forward declared another_struct, we have again the case that changes to this data structure can't be captured as the type is passed by reference and the details are hidden from the ABI. Not even we know anything further about this struct. If my_function is not exported at all (i.e. not part of the observable ABI), any analysis will ignore it, entirely and rightfully.

For the Linux kernel there is an additional fact to consider. In the default way (without using whitelists), symbols are considered part of the ABI (or in this case Kernel Module Interface (KMI)), if they are exported via EXPORT_SYMBOL* macros and therefore part of the ksymtab.

If a data structure is not participating in the type information of an exported symbol, changes to it will not be discoverable. In order to discover changes to those, you need to make them discoverable, e.g. by exporting a function that takes 'struct dm_ioctl' as an argument type.

Cheers,
Matthias

>
>Thanks,
>Ketan Patil
>

RE: abidw for non-exported structures

[Sachin Nikam]

Hi,

I have another query.

Android libabigail tool  => https://android.googlesource.com/kernel/build/+/refs/heads/master/abi/README.md

I want to know how much android libabigail tool is diverged from https://www.sourceware.org/libabigail/wiki 

What are the enhancements or bug fixes done in android libabigail tool from sourceware tool?

Looping in some folks.

Best Regards,
Sachin Nikam.

-----Original Message-----
From: Ketan Patil <ketanp@nvidia.com> 
Sent: Wednesday, March 3, 2021 12:45 PM
To: Matthias Maennich <maennich@google.com>
Cc: libabigail@sourceware.org; Sachin Nikam <Snikam@nvidia.com>; Nicolin Chen <nicolinc@nvidia.com>
Subject: RE: abidw for non-exported structures

Thanks a lot Matthias for the detailed answer.


-----Original Message-----
From: Matthias Maennich <maennich@google.com> 
Sent: Thursday, February 25, 2021 7:11 PM
To: Ketan Patil <ketanp@nvidia.com>
Cc: libabigail@sourceware.org; Sachin Nikam <Snikam@nvidia.com>; Nicolin Chen <nicolinc@nvidia.com>
Subject: Re: abidw for non-exported structures

External email: Use caution opening links or attachments


Hi!

On Thu, Feb 25, 2021 at 11:25:53AM +0000, Ketan Patil via Libabigail wrote:
>Hi,
>
>I was experimenting with abidw and abidiff tools to check ABI changes in linux kernel.
>I had performed few experiments:
>
>  1.  Changing exported structures:
>     *   I changed few structures (added some dummy members in between) which were exported and build the kernel and created abi xml file using abidw tool.
>     *   Then I compared this xml with the baseline xml (kernel without any changes) using abidiff tool and I found that it catches this ABI change.
>  2.  Changing non-exported structures:
>     *   I changed few non-exported structures e.g. struct dm_ioctl In https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/include/uapi/linux/dm-ioctl.h?h=v5.10.18
>     *   This struct is being used by a number of ioctls. So I think, changing this struct should be flagged as an ABI change.
>     *   Then I build the kernel and created the new abi xml file using abidw tool and compared with baseline xml using abidiff tool.
>     *   abidiff tool is not able to catch this. Also, I don't see any entry for dm_ioctl in the xml generated using abidw.
>
>
>Does it mean that, abidw tool does not work for the non exported symbols? If yes, are you planning to add this feature?
>Can you please let me know or redirect?

libabigail works roughly like this (in its default mode): It reads symbol information from the ELF symbol table and matches symbols with corresponding DWARF information to gather type information associated with it. The type information is recursively resolved to capture nested types. So, fundamentally, to capture and eventually report a structure in the ABI, it needs to be contributing to the definition of an exported symbol. E.g. consider this snippet.

   | enum my_enum {
   |   VALUE, ANOTHER_VALUE
   | };
   |
   | struct my_struct {
   |   // some interesting bits
   | };
   |
   | struct another_struct;
   |
   | // consider publicly exported function
   | int my_function(int my_enum, struct my_struct s1, struct another_struct* s2) {
   |   if (my_enum == VALUE) {
   |     // something
   |   }
   | }

The enum technically participates in the ABI, but this is invisible for libabigail as the type information is not connected. As opposed to my_struct. This is part of the type information of my_function and changes therefore would be discoverable. For the forward declared another_struct, we have again the case that changes to this data structure can't be captured as the type is passed by reference and the details are hidden from the ABI. Not even we know anything further about this struct. If my_function is not exported at all (i.e. not part of the observable ABI), any analysis will ignore it, entirely and rightfully.

For the Linux kernel there is an additional fact to consider. In the default way (without using whitelists), symbols are considered part of the ABI (or in this case Kernel Module Interface (KMI)), if they are exported via EXPORT_SYMBOL* macros and therefore part of the ksymtab.

If a data structure is not participating in the type information of an exported symbol, changes to it will not be discoverable. In order to discover changes to those, you need to make them discoverable, e.g. by exporting a function that takes 'struct dm_ioctl' as an argument type.

Cheers,
Matthias

>
>Thanks,
>Ketan Patil
>