How to debug while using LTO?

How to debug while using LTO?

[Stefan Schulze Frielinghaus]

Hi everyone,

Currently I'm looking into a wrong-code bug and would like to understand
a certain optimization done by combine during local transformation.
Without LTO I would simply debug cc1 and step through combine.  However,
with LTO enabled AFAIK I have to debug lto1 instead.  In order to get
the lto1 command line of interest according to
https://gcc.gnu.org/legacy-ml/gcc/2009-11/msg00047.html
I have to pass -Wl,-debug to gcc in order to get the command for
collect2 to which itself I have to pass -plugin-opt=-debug in order to
get the command for lto-wrapper.  According to the aforementioned mail I
should add option -debug to lto-wrapper, however, it appears to me that
option -debug was removed.  I gave options -v and -### a chance without
luck, i.e., those only print the usual environment variables and
afterwards a list of object files like

/tmp/ccPEIV35.ltrans0.ltrans.o
/tmp/ccNmpKfS.debug.temp.o
/tmp/cceiCIFg.debug.temp.o
/tmp/ccZ4Qc7E.debug.temp.o
...

but no lto1 command.  Thus, how do you retrieve the lto1 command?

While desperate I retrieved it manually via strace.  However, the lto1
command refers to temporary files which have been erased meanwhile.  I
actually didn't expect that because I added -save-temps to all the
intermediate commands which is also reflected in the environment
variable COLLECT_GCC_OPTIONS.  Thus, how do you keep temporary files?

Cheers,
Stefan

Re: How to debug while using LTO?

[Richard Biener]

> Am 24.11.2022 um 17:28 schrieb Stefan Schulze Frielinghaus via Gcc <gcc@gcc.gnu.org>:
> 
> Hi everyone,
> 
> Currently I'm looking into a wrong-code bug and would like to understand
> a certain optimization done by combine during local transformation.
> Without LTO I would simply debug cc1 and step through combine.  However,
> with LTO enabled AFAIK I have to debug lto1 instead.  In order to get
> the lto1 command line of interest according to
> https://gcc.gnu.org/legacy-ml/gcc/2009-11/msg00047.html
> I have to pass -Wl,-debug to gcc in order to get the command for
> collect2 to which itself I have to pass -plugin-opt=-debug in order to
> get the command for lto-wrapper.  According to the aforementioned mail I
> should add option -debug to lto-wrapper, however, it appears to me that
> option -debug was removed.  I gave options -v and -### a chance without
> luck, i.e., those only print the usual environment variables and
> afterwards a list of object files like
> 
> /tmp/ccPEIV35.ltrans0.ltrans.o
> /tmp/ccNmpKfS.debug.temp.o
> /tmp/cceiCIFg.debug.temp.o
> /tmp/ccZ4Qc7E.debug.temp.o
> ...
> 
> but no lto1 command.  Thus, how do you retrieve the lto1 command?
> 
> While desperate I retrieved it manually via strace.  However, the lto1
> command refers to temporary files which have been erased meanwhile.  I
> actually didn't expect that because I added -save-temps to all the
> intermediate commands which is also reflected in the environment
> variable COLLECT_GCC_OPTIONS.  Thus, how do you keep temporary files?

Adding -v -save-temps and then running gdb on the lto1 command works and is what I usually do.

Richard 

> Cheers,
> Stefan

Re: How to debug while using LTO?

[Stefan Schulze Frielinghaus]

On Thu, Nov 24, 2022 at 05:53:53PM +0100, Richard Biener wrote:
> 
> 
> > Am 24.11.2022 um 17:28 schrieb Stefan Schulze Frielinghaus via Gcc <gcc@gcc.gnu.org>:
> > 
> > Hi everyone,
> > 
> > Currently I'm looking into a wrong-code bug and would like to understand
> > a certain optimization done by combine during local transformation.
> > Without LTO I would simply debug cc1 and step through combine.  However,
> > with LTO enabled AFAIK I have to debug lto1 instead.  In order to get
> > the lto1 command line of interest according to
> > https://gcc.gnu.org/legacy-ml/gcc/2009-11/msg00047.html
> > I have to pass -Wl,-debug to gcc in order to get the command for
> > collect2 to which itself I have to pass -plugin-opt=-debug in order to
> > get the command for lto-wrapper.  According to the aforementioned mail I
> > should add option -debug to lto-wrapper, however, it appears to me that
> > option -debug was removed.  I gave options -v and -### a chance without
> > luck, i.e., those only print the usual environment variables and
> > afterwards a list of object files like
> > 
> > /tmp/ccPEIV35.ltrans0.ltrans.o
> > /tmp/ccNmpKfS.debug.temp.o
> > /tmp/cceiCIFg.debug.temp.o
> > /tmp/ccZ4Qc7E.debug.temp.o
> > ...
> > 
> > but no lto1 command.  Thus, how do you retrieve the lto1 command?
> > 
> > While desperate I retrieved it manually via strace.  However, the lto1
> > command refers to temporary files which have been erased meanwhile.  I
> > actually didn't expect that because I added -save-temps to all the
> > intermediate commands which is also reflected in the environment
> > variable COLLECT_GCC_OPTIONS.  Thus, how do you keep temporary files?
> 
> Adding -v -save-temps and then running gdb on the lto1 command works and is what I usually do.

Strangely this is not the case for the current project I'm looking at.  The
link step is done via

/usr/local/bin/s390x-linux-gnu-gcc -save-temps \
  -O3 -DNDEBUG -flto=auto -fno-fat-lto-objects \
  -march=z15 -static -nostdlib \
  -Wl,--fatal-warnings -Wl,--no-warn-rwx-segments \
  -Werror -Wall -Wundef -T/devel/foo.ld \
  obj1.o ... objN.o

where -save-temps is given and was also given during compile time of each
object obj1.o ... objN.o file.  Anyhow, adding -Wl,-debug to the link command
above I see that -save-temps is also included in the environment variable
COLLECT_GCC_OPTIONS="... -save-temps ..." and I get the linker command

/usr/local/lib/gcc/s390x-linux-gnu/13.0.0/../../../../s390x-linux-gnu/bin/ld \
  -plugin /usr/local/libexec/gcc/s390x-linux-gnu/13.0.0/liblto_plugin.so \
  -plugin-opt=/usr/local/libexec/gcc/s390x-linux-gnu/13.0.0/lto-wrapper \
  -plugin-opt=-fresolution=foo.res \
  -m elf64_s390 -static -o foo \
  -L/usr/local/lib/gcc/s390x-linux-gnu/13.0.0 \
  -L/usr/local/lib/gcc/s390x-linux-gnu/13.0.0/../../../../s390x-linux-gnu/lib \
  --fatal-warnings --no-warn-rwx-segments \
  obj1.o ... objN.o -lgcc -T /devel/foo.ld

to which I add -plugin-opt=-debug in order to get the lto-wrapper command

/usr/local/libexec/gcc/s390x-linux-gnu/13.0.0/lto-wrapper \
  -fresolution=foo.res -flinker-output=exec \
  obj1.o ... objN.o

to which I add -v -save-temps in order to actually get the lto1 command.  From
the output printed to stderr I see one lto1 command:

/usr/local/libexec/gcc/s390x-linux-gnu/13.0.0/lto1 -quiet -dumpbase ./a.wpa \
  -march=z15 -m64 -mzarch -g -O3 -version -fno-openmp -fno-openacc -fno-pie \
  -fcf-protection=none -fltrans-output-list=/tmp/ccsuw4Nj.ltrans.out \
  -fwpa=32 -fresolution=foo.res -flinker-output=exec @./a.wpa.args.0

However, this lto1 invocation does no local transformation.  Having a look at
the strace output I see a second lto1 invocation:

/usr/local/libexec/gcc/s390x-linux-gnu/13.0.0/lto1 -quiet \
  -dumpbase ./a.ltrans0.ltrans -march=z15 -m64 -mzarch -g -O3 \
  -fno-openmp -fno-openacc -fno-pie -fcf-protection=none \
  -fltrans /tmp/ccsuw4Nj.ltrans0.o -o /tmp/cc69wMp3.s

This lto1 invocation seems to originate from Makefile /tmp/cc5piuXT.mk
generated by lto-wrapper.  To my surprise the Makefile as well as the temporary
file /tmp/ccsuw4Nj.ltrans0.o are removed although -save-temps was given all the
time.  Stepping through lto-wrapper with gdb I see that

  save_temps = 1;

is never executed while walking over decoded_options.

For the time being I hard coded save_temps=1 into lto-wrapper.cc and I am now
able to debug the lto1 invocation of interest.  Not sure whether there
is a more elegant way?

All this is based upon a rather huge CMake based project and so far I didn't
find the time to come up with a small reproducer.  Still hoping this might be
useful for someone else.

Cheers,
Stefan

Re: How to debug while using LTO?

[Richard Biener]

On Wed, Nov 30, 2022 at 11:49 AM Stefan Schulze Frielinghaus
<stefansf@linux.ibm.com> wrote:
>
> On Thu, Nov 24, 2022 at 05:53:53PM +0100, Richard Biener wrote:
> >
> >
> > > Am 24.11.2022 um 17:28 schrieb Stefan Schulze Frielinghaus via Gcc <gcc@gcc.gnu.org>:
> > >
> > > Hi everyone,
> > >
> > > Currently I'm looking into a wrong-code bug and would like to understand
> > > a certain optimization done by combine during local transformation.
> > > Without LTO I would simply debug cc1 and step through combine.  However,
> > > with LTO enabled AFAIK I have to debug lto1 instead.  In order to get
> > > the lto1 command line of interest according to
> > > https://gcc.gnu.org/legacy-ml/gcc/2009-11/msg00047.html
> > > I have to pass -Wl,-debug to gcc in order to get the command for
> > > collect2 to which itself I have to pass -plugin-opt=-debug in order to
> > > get the command for lto-wrapper.  According to the aforementioned mail I
> > > should add option -debug to lto-wrapper, however, it appears to me that
> > > option -debug was removed.  I gave options -v and -### a chance without
> > > luck, i.e., those only print the usual environment variables and
> > > afterwards a list of object files like
> > >
> > > /tmp/ccPEIV35.ltrans0.ltrans.o
> > > /tmp/ccNmpKfS.debug.temp.o
> > > /tmp/cceiCIFg.debug.temp.o
> > > /tmp/ccZ4Qc7E.debug.temp.o
> > > ...
> > >
> > > but no lto1 command.  Thus, how do you retrieve the lto1 command?
> > >
> > > While desperate I retrieved it manually via strace.  However, the lto1
> > > command refers to temporary files which have been erased meanwhile.  I
> > > actually didn't expect that because I added -save-temps to all the
> > > intermediate commands which is also reflected in the environment
> > > variable COLLECT_GCC_OPTIONS.  Thus, how do you keep temporary files?
> >
> > Adding -v -save-temps and then running gdb on the lto1 command works and is what I usually do.
>
> Strangely this is not the case for the current project I'm looking at.  The
> link step is done via
>
> /usr/local/bin/s390x-linux-gnu-gcc -save-temps \
>   -O3 -DNDEBUG -flto=auto -fno-fat-lto-objects \
>   -march=z15 -static -nostdlib \
>   -Wl,--fatal-warnings -Wl,--no-warn-rwx-segments \
>   -Werror -Wall -Wundef -T/devel/foo.ld \
>   obj1.o ... objN.o
>
> where -save-temps is given and was also given during compile time of each
> object obj1.o ... objN.o file.  Anyhow, adding -Wl,-debug to the link command
> above I see that -save-temps is also included in the environment variable
> COLLECT_GCC_OPTIONS="... -save-temps ..." and I get the linker command
>
> /usr/local/lib/gcc/s390x-linux-gnu/13.0.0/../../../../s390x-linux-gnu/bin/ld \
>   -plugin /usr/local/libexec/gcc/s390x-linux-gnu/13.0.0/liblto_plugin.so \
>   -plugin-opt=/usr/local/libexec/gcc/s390x-linux-gnu/13.0.0/lto-wrapper \
>   -plugin-opt=-fresolution=foo.res \
>   -m elf64_s390 -static -o foo \
>   -L/usr/local/lib/gcc/s390x-linux-gnu/13.0.0 \
>   -L/usr/local/lib/gcc/s390x-linux-gnu/13.0.0/../../../../s390x-linux-gnu/lib \
>   --fatal-warnings --no-warn-rwx-segments \
>   obj1.o ... objN.o -lgcc -T /devel/foo.ld
>
> to which I add -plugin-opt=-debug in order to get the lto-wrapper command
>
> /usr/local/libexec/gcc/s390x-linux-gnu/13.0.0/lto-wrapper \
>   -fresolution=foo.res -flinker-output=exec \
>   obj1.o ... objN.o
>
> to which I add -v -save-temps in order to actually get the lto1 command.  From
> the output printed to stderr I see one lto1 command:
>
> /usr/local/libexec/gcc/s390x-linux-gnu/13.0.0/lto1 -quiet -dumpbase ./a.wpa \
>   -march=z15 -m64 -mzarch -g -O3 -version -fno-openmp -fno-openacc -fno-pie \
>   -fcf-protection=none -fltrans-output-list=/tmp/ccsuw4Nj.ltrans.out \
>   -fwpa=32 -fresolution=foo.res -flinker-output=exec @./a.wpa.args.0
>
> However, this lto1 invocation does no local transformation.  Having a look at
> the strace output I see a second lto1 invocation:
>
> /usr/local/libexec/gcc/s390x-linux-gnu/13.0.0/lto1 -quiet \
>   -dumpbase ./a.ltrans0.ltrans -march=z15 -m64 -mzarch -g -O3 \
>   -fno-openmp -fno-openacc -fno-pie -fcf-protection=none \
>   -fltrans /tmp/ccsuw4Nj.ltrans0.o -o /tmp/cc69wMp3.s
>
> This lto1 invocation seems to originate from Makefile /tmp/cc5piuXT.mk
> generated by lto-wrapper.  To my surprise the Makefile as well as the temporary
> file /tmp/ccsuw4Nj.ltrans0.o are removed although -save-temps was given all the
> time.  Stepping through lto-wrapper with gdb I see that
>
>   save_temps = 1;
>
> is never executed while walking over decoded_options.
>
> For the time being I hard coded save_temps=1 into lto-wrapper.cc and I am now
> able to debug the lto1 invocation of interest.  Not sure whether there
> is a more elegant way?
>
> All this is based upon a rather huge CMake based project and so far I didn't
> find the time to come up with a small reproducer.  Still hoping this might be
> useful for someone else.

At some point passing -Wl,-debug -Wl,-v to the gcc driver was necessary
but I think we've meanwhile fixed it so that -save-temps -v works just fine,
preserving everything necessary to even invoke the LTRANS optimization
stages, so I'm not sure what kind of issue you are running into here.

>
> Cheers,
> Stefan