[GSoC]Bypass assembler when generating LTO object files

[GSoC]Bypass assembler when generating LTO object files

[Ankur Saini]

Hi,

I was browsing the list of submitted GSoC projects this year and the
project regarding bypassing assembler when generating LTO object files
caught my eye.

I already have a gcc built from source (sync-ed with trunk/master) and
launched the test-suite on it.

I am currently in process of understanding the primilary patch
(https://gcc.gnu.org/legacy-ml/gcc/2014-09/msg00340.html), and
experimenting with it.

are there any other things I should be aware of (useful Doc/blog or a
bug tracking the project) before proceeding further ?

I am Ankur Saini, a B.Tech CSE 3rd year student at USICT, GGSIPU india
and a former GSoC contributor at gcc ( worked on expanding gcc static
analyzer's C++ support in GSoC 2021
[https://gist.github.com/Arsenic-ATG/8f4ac194f460dd9b2c78cf51af39afef])

Thanks
- Ankur

Re: [GSoC]Bypass assembler when generating LTO object files

[Jan Hubicka]

Ankur,
> I was browsing the list of submitted GSoC projects this year and the
> project regarding bypassing assembler when generating LTO object files
> caught my eye.
I apologize for late reply.  I would be very happy to mentor this
project.
> 
> I already have a gcc built from source (sync-ed with trunk/master) and
> launched the test-suite on it.
> 
> I am currently in process of understanding the primilary patch
> (https://gcc.gnu.org/legacy-ml/gcc/2014-09/msg00340.html), and
> experimenting with it.
> 
> are there any other things I should be aware of (useful Doc/blog or a
> bug tracking the project) before proceeding further ?

I think it is pretty much all that exists.  Basically we will need to
implement everything that is necessary to stream out valid object file
directly from GCC rather than going through gas.  The experimental
prototype sort of worked but it was lacking few things.  First is the
production of proper object file hearder (it encodes things like
architeture ELF is produced for), production of symbol table that is
necessary to mark the LTO object file and also we now need a way to
stream debug info (DWARF) directly to the object from dwarf2out.

So I think first step would be to produce object files w/o debug info
which can be consumed by unmodified linkers and then look into DWARF
bytecode streaming.

Honza
> 
> I am Ankur Saini, a B.Tech CSE 3rd year student at USICT, GGSIPU india
> and a former GSoC contributor at gcc ( worked on expanding gcc static
> analyzer's C++ support in GSoC 2021
> [https://gist.github.com/Arsenic-ATG/8f4ac194f460dd9b2c78cf51af39afef])
> 
> Thanks
> - Ankur

Re: [GSoC]Bypass assembler when generating LTO object files

[Ankur Saini]

> On 08-Apr-2022, at 6:32 PM, Jan Hubicka <hubicka@kam.mff.cuni.cz> wrote:
> 
> Ankur,
>> I was browsing the list of submitted GSoC projects this year and the
>> project regarding bypassing assembler when generating LTO object files
>> caught my eye.
> I apologize for late reply.  I would be very happy to mentor this
> project.

Thanks for the reply, but unfortunately, due to some reasons, I would not
be able to take part in GSoC this year.  
But the project seems interesting and would be amazing opportunity to
learn a lot more things for me, so would it be okay if I try to give it a
go outside GSoC if no-one else picks it as their GSoC project this year ?

>> 
>> I already have a gcc built from source (sync-ed with trunk/master) and
>> launched the test-suite on it.
>> 
>> I am currently in process of understanding the primilary patch
>> (https://gcc.gnu.org/legacy-ml/gcc/2014-09/msg00340.html), and
>> experimenting with it.
>> 
>> are there any other things I should be aware of (useful Doc/blog or a
>> bug tracking the project) before proceeding further ?
> 
> I think it is pretty much all that exists.  Basically we will need to
> implement everything that is necessary to stream out valid object file
> directly from GCC rather than going through gas.  The experimental
> prototype sort of worked but it was lacking few things.

When I try to apply that patch on my local branch ( branched from trunk ),
it seem to be incompatible with the current working tree. Is there a
specific branch that I have to apply it to ? or is it due to the recent
file rename patch ( changing extensions from .c to .cc ) ? 

```
$ git apply --check bypass_asm_patch

error: patch failed: Makefile.in:1300
error: Makefile.in: patch does not apply
error: common.opt: No such file or directory
error: langhooks.c: No such file or directory
error: lto/Make-lang.in: No such file or directory
error: lto/lto-object.c: No such file or directory
error: lto/lto.c: No such file or directory
error: lto/lto.h: No such file or directory
error: lto-streamer.h: No such file or directory
error: toplev.c: No such file or directory
```

Thanks 
- Ankur 

Re: [GSoC]Bypass assembler when generating LTO object files

[Jan Hubicka]

Hi,
> 
> 
> > On 08-Apr-2022, at 6:32 PM, Jan Hubicka <hubicka@kam.mff.cuni.cz> wrote:
> > 
> > Ankur,
> >> I was browsing the list of submitted GSoC projects this year and the
> >> project regarding bypassing assembler when generating LTO object files
> >> caught my eye.
> > I apologize for late reply.  I would be very happy to mentor this
> > project.
> 
> Thanks for the reply, but unfortunately, due to some reasons, I would not
> be able to take part in GSoC this year.  
> But the project seems interesting and would be amazing opportunity to
> learn a lot more things for me, so would it be okay if I try to give it a
> go outside GSoC if no-one else picks it as their GSoC project this year ?

I would be still very happy to help with that! However it would be also
pity to not take part of GSoC, so if there is something I can help with
on that let me know.
> 
> >> 
> >> I already have a gcc built from source (sync-ed with trunk/master) and
> >> launched the test-suite on it.
> >> 
> >> I am currently in process of understanding the primilary patch
> >> (https://gcc.gnu.org/legacy-ml/gcc/2014-09/msg00340.html), and
> >> experimenting with it.
> >> 
> >> are there any other things I should be aware of (useful Doc/blog or a
> >> bug tracking the project) before proceeding further ?
> > 
> > I think it is pretty much all that exists.  Basically we will need to
> > implement everything that is necessary to stream out valid object file
> > directly from GCC rather than going through gas.  The experimental
> > prototype sort of worked but it was lacking few things.
> 
> When I try to apply that patch on my local branch ( branched from trunk ),
> it seem to be incompatible with the current working tree. Is there a
> specific branch that I have to apply it to ? or is it due to the recent
> file rename patch ( changing extensions from .c to .cc ) ? 
> 
> ```
> $ git apply --check bypass_asm_patch
> 
> error: patch failed: Makefile.in:1300
> error: Makefile.in: patch does not apply
> error: common.opt: No such file or directory
> error: langhooks.c: No such file or directory
> error: lto/Make-lang.in: No such file or directory
> error: lto/lto-object.c: No such file or directory
> error: lto/lto.c: No such file or directory
> error: lto/lto.h: No such file or directory
> error: lto-streamer.h: No such file or directory
> error: toplev.c: No such file or directory
> ```

I can try to update the patch, or it probably should apply to trunk
checked out around the date I sent the patch.  Indeed we need to change
c to cc but there are likely more changes since then - most importnatly
the early debug info.
At I will see how easy/hard is to make the patch build with current
trunk.
Honza
> 
> Thanks 
> - Ankur 
> 

Re: [GSoC]Bypass assembler when generating LTO object files

[Richard Biener]

On Tue, Apr 12, 2022 at 11:20 AM Jan Hubicka via Gcc <gcc@gcc.gnu.org> wrote:
>
> Hi,
> >
> >
> > > On 08-Apr-2022, at 6:32 PM, Jan Hubicka <hubicka@kam.mff.cuni.cz> wrote:
> > >
> > > Ankur,
> > >> I was browsing the list of submitted GSoC projects this year and the
> > >> project regarding bypassing assembler when generating LTO object files
> > >> caught my eye.
> > > I apologize for late reply.  I would be very happy to mentor this
> > > project.
> >
> > Thanks for the reply, but unfortunately, due to some reasons, I would not
> > be able to take part in GSoC this year.
> > But the project seems interesting and would be amazing opportunity to
> > learn a lot more things for me, so would it be okay if I try to give it a
> > go outside GSoC if no-one else picks it as their GSoC project this year ?
>
> I would be still very happy to help with that! However it would be also
> pity to not take part of GSoC, so if there is something I can help with
> on that let me know.
> >
> > >>
> > >> I already have a gcc built from source (sync-ed with trunk/master) and
> > >> launched the test-suite on it.
> > >>
> > >> I am currently in process of understanding the primilary patch
> > >> (https://gcc.gnu.org/legacy-ml/gcc/2014-09/msg00340.html), and
> > >> experimenting with it.
> > >>
> > >> are there any other things I should be aware of (useful Doc/blog or a
> > >> bug tracking the project) before proceeding further ?
> > >
> > > I think it is pretty much all that exists.  Basically we will need to
> > > implement everything that is necessary to stream out valid object file
> > > directly from GCC rather than going through gas.  The experimental
> > > prototype sort of worked but it was lacking few things.
> >
> > When I try to apply that patch on my local branch ( branched from trunk ),
> > it seem to be incompatible with the current working tree. Is there a
> > specific branch that I have to apply it to ? or is it due to the recent
> > file rename patch ( changing extensions from .c to .cc ) ?
> >
> > ```
> > $ git apply --check bypass_asm_patch
> >
> > error: patch failed: Makefile.in:1300
> > error: Makefile.in: patch does not apply
> > error: common.opt: No such file or directory
> > error: langhooks.c: No such file or directory
> > error: lto/Make-lang.in: No such file or directory
> > error: lto/lto-object.c: No such file or directory
> > error: lto/lto.c: No such file or directory
> > error: lto/lto.h: No such file or directory
> > error: lto-streamer.h: No such file or directory
> > error: toplev.c: No such file or directory
> > ```
>
> I can try to update the patch, or it probably should apply to trunk
> checked out around the date I sent the patch.  Indeed we need to change
> c to cc but there are likely more changes since then - most importnatly
> the early debug info.
> At I will see how easy/hard is to make the patch build with current
> trunk.

We do have ideas for the early debug with the asm-out abstraction to
also solve a different issue (missing simple-object support for mingw/darwin).
Namely assemble the early debug in a different file and include the resulting
native object in binary form in the compile output - not needing to write
assembly .data for that would be a good way to make this more viable.

You might want to talk to Martin Liska for this who I think had some
prototype on this?

Richard.

> Honza
> >
> > Thanks
> > - Ankur
> >

Re: [GSoC]Bypass assembler when generating LTO object files

[Martin Liška]

On 4/12/22 11:58, Richard Biener wrote:
> On Tue, Apr 12, 2022 at 11:20 AM Jan Hubicka via Gcc <gcc@gcc.gnu.org> wrote:
>>
>> Hi,
>>>
>>>
>>>> On 08-Apr-2022, at 6:32 PM, Jan Hubicka <hubicka@kam.mff.cuni.cz> wrote:
>>>>
>>>> Ankur,
>>>>> I was browsing the list of submitted GSoC projects this year and the
>>>>> project regarding bypassing assembler when generating LTO object files
>>>>> caught my eye.
>>>> I apologize for late reply.  I would be very happy to mentor this
>>>> project.
>>>
>>> Thanks for the reply, but unfortunately, due to some reasons, I would not
>>> be able to take part in GSoC this year.
>>> But the project seems interesting and would be amazing opportunity to
>>> learn a lot more things for me, so would it be okay if I try to give it a
>>> go outside GSoC if no-one else picks it as their GSoC project this year ?
>>
>> I would be still very happy to help with that! However it would be also
>> pity to not take part of GSoC, so if there is something I can help with
>> on that let me know.
>>>
>>>>>
>>>>> I already have a gcc built from source (sync-ed with trunk/master) and
>>>>> launched the test-suite on it.
>>>>>
>>>>> I am currently in process of understanding the primilary patch
>>>>> (https://gcc.gnu.org/legacy-ml/gcc/2014-09/msg00340.html), and
>>>>> experimenting with it.
>>>>>
>>>>> are there any other things I should be aware of (useful Doc/blog or a
>>>>> bug tracking the project) before proceeding further ?
>>>>
>>>> I think it is pretty much all that exists.  Basically we will need to
>>>> implement everything that is necessary to stream out valid object file
>>>> directly from GCC rather than going through gas.  The experimental
>>>> prototype sort of worked but it was lacking few things.
>>>
>>> When I try to apply that patch on my local branch ( branched from trunk ),
>>> it seem to be incompatible with the current working tree. Is there a
>>> specific branch that I have to apply it to ? or is it due to the recent
>>> file rename patch ( changing extensions from .c to .cc ) ?
>>>
>>> ```
>>> $ git apply --check bypass_asm_patch
>>>
>>> error: patch failed: Makefile.in:1300
>>> error: Makefile.in: patch does not apply
>>> error: common.opt: No such file or directory
>>> error: langhooks.c: No such file or directory
>>> error: lto/Make-lang.in: No such file or directory
>>> error: lto/lto-object.c: No such file or directory
>>> error: lto/lto.c: No such file or directory
>>> error: lto/lto.h: No such file or directory
>>> error: lto-streamer.h: No such file or directory
>>> error: toplev.c: No such file or directory
>>> ```
>>
>> I can try to update the patch, or it probably should apply to trunk
>> checked out around the date I sent the patch.  Indeed we need to change
>> c to cc but there are likely more changes since then - most importnatly
>> the early debug info.
>> At I will see how easy/hard is to make the patch build with current
>> trunk.
> 
> We do have ideas for the early debug with the asm-out abstraction to
> also solve a different issue (missing simple-object support for mingw/darwin).

Note the debug info will be stored to a different .s file, then the file
will be converted .o by GAS and then the bytecode will be stored to an ELF
section of a compiled object. I've got also binutils patch when we'll
be able to directly use the embedded section with compile.o@offset.

> Namely assemble the early debug in a different file and include the resulting
> native object in binary form in the compile output - not needing to write
> assembly .data for that would be a good way to make this more viable.

Btw. do you have any estimation how slow is GAS when we speak about debug info?
How much time can we save since the latest speed-up achieved by GAS?

> 
> You might want to talk to Martin Liska for this who I think had some
> prototype on this?

I can provide a prototype if needed.

Cheers,
Martin

> 
> Richard.
> 
>> Honza
>>>
>>> Thanks
>>> - Ankur
>>>

Re: [GSoC]Bypass assembler when generating LTO object files

[Iain Sandoe]

> On 12 Apr 2022, at 13:31, Martin Liška <mliska@suse.cz> wrote:
> 
> On 4/12/22 11:58, Richard Biener wrote:
>> On Tue, Apr 12, 2022 at 11:20 AM Jan Hubicka via Gcc <gcc@gcc.gnu.org> wrote:
>>> 
>>> Hi,
>>>> 
>>>> 
>>>>> On 08-Apr-2022, at 6:32 PM, Jan Hubicka <hubicka@kam.mff.cuni.cz> wrote:
>>>>> 
>>>>> Ankur,
>>>>>> I was browsing the list of submitted GSoC projects this year and the
>>>>>> project regarding bypassing assembler when generating LTO object files
>>>>>> caught my eye.
>>>>> I apologize for late reply.  I would be very happy to mentor this
>>>>> project.
>>>> 
>>>> Thanks for the reply, but unfortunately, due to some reasons, I would not
>>>> be able to take part in GSoC this year.
>>>> But the project seems interesting and would be amazing opportunity to
>>>> learn a lot more things for me, so would it be okay if I try to give it a
>>>> go outside GSoC if no-one else picks it as their GSoC project this year ?
>>> 
>>> I would be still very happy to help with that! However it would be also
>>> pity to not take part of GSoC, so if there is something I can help with
>>> on that let me know.
>>>> 
>>>>>> 
>>>>>> I already have a gcc built from source (sync-ed with trunk/master) and
>>>>>> launched the test-suite on it.
>>>>>> 
>>>>>> I am currently in process of understanding the primilary patch
>>>>>> (https://gcc.gnu.org/legacy-ml/gcc/2014-09/msg00340.html), and
>>>>>> experimenting with it.
>>>>>> 
>>>>>> are there any other things I should be aware of (useful Doc/blog or a
>>>>>> bug tracking the project) before proceeding further ?
>>>>> 
>>>>> I think it is pretty much all that exists.  Basically we will need to
>>>>> implement everything that is necessary to stream out valid object file
>>>>> directly from GCC rather than going through gas.  The experimental
>>>>> prototype sort of worked but it was lacking few things.
>>>> 
>>>> When I try to apply that patch on my local branch ( branched from trunk ),
>>>> it seem to be incompatible with the current working tree. Is there a
>>>> specific branch that I have to apply it to ? or is it due to the recent
>>>> file rename patch ( changing extensions from .c to .cc ) ?
>>>> 
>>>> ```
>>>> $ git apply --check bypass_asm_patch
>>>> 
>>>> error: patch failed: Makefile.in:1300
>>>> error: Makefile.in: patch does not apply
>>>> error: common.opt: No such file or directory
>>>> error: langhooks.c: No such file or directory
>>>> error: lto/Make-lang.in: No such file or directory
>>>> error: lto/lto-object.c: No such file or directory
>>>> error: lto/lto.c: No such file or directory
>>>> error: lto/lto.h: No such file or directory
>>>> error: lto-streamer.h: No such file or directory
>>>> error: toplev.c: No such file or directory
>>>> ```
>>> 
>>> I can try to update the patch, or it probably should apply to trunk
>>> checked out around the date I sent the patch.  Indeed we need to change
>>> c to cc but there are likely more changes since then - most importnatly
>>> the early debug info.
>>> At I will see how easy/hard is to make the patch build with current
>>> trunk.
>> We do have ideas for the early debug with the asm-out abstraction to
>> also solve a different issue (missing simple-object support for mingw/darwin).
> 
> Note the debug info will be stored to a different .s file, then the file
> will be converted .o by GAS and then the bytecode will be stored to an ELF
> section of a compiled object. I've got also binutils patch when we'll
> be able to directly use the embedded section with compile.o@offset.

Which will not work, as written, for Darwin since that is neither ELF nor does it
use GAS - but hopefully, we can find some equivalent mechanism (there is already
some support in the Darwin backend for switching asm context for LTO output).

>> Namely assemble the early debug in a different file and include the resulting
>> native object in binary form in the compile output - not needing to write
>> assembly .data for that would be a good way to make this more viable.
> 
> Btw. do you have any estimation how slow is GAS when we speak about debug info?
> How much time can we save since the latest speed-up achieved by GAS?
> 
>> You might want to talk to Martin Liska for this who I think had some
>> prototype on this?
> 
> I can provide a prototype if needed.

I’d like to be in to loop from the Darwin PoV..
thanks
Iain

> 
> Cheers,
> Martin
> 
>> Richard.
>>> Honza
>>>> 
>>>> Thanks
>>>> - Ankur

Re: [GSoC]Bypass assembler when generating LTO object files

[Richard Biener]

On Tue, Apr 12, 2022 at 2:53 PM Iain Sandoe <idsandoe@googlemail.com> wrote:
>
>
>
> > On 12 Apr 2022, at 13:31, Martin Liška <mliska@suse.cz> wrote:
> >
> > On 4/12/22 11:58, Richard Biener wrote:
> >> On Tue, Apr 12, 2022 at 11:20 AM Jan Hubicka via Gcc <gcc@gcc.gnu.org> wrote:
> >>>
> >>> Hi,
> >>>>
> >>>>
> >>>>> On 08-Apr-2022, at 6:32 PM, Jan Hubicka <hubicka@kam.mff.cuni.cz> wrote:
> >>>>>
> >>>>> Ankur,
> >>>>>> I was browsing the list of submitted GSoC projects this year and the
> >>>>>> project regarding bypassing assembler when generating LTO object files
> >>>>>> caught my eye.
> >>>>> I apologize for late reply.  I would be very happy to mentor this
> >>>>> project.
> >>>>
> >>>> Thanks for the reply, but unfortunately, due to some reasons, I would not
> >>>> be able to take part in GSoC this year.
> >>>> But the project seems interesting and would be amazing opportunity to
> >>>> learn a lot more things for me, so would it be okay if I try to give it a
> >>>> go outside GSoC if no-one else picks it as their GSoC project this year ?
> >>>
> >>> I would be still very happy to help with that! However it would be also
> >>> pity to not take part of GSoC, so if there is something I can help with
> >>> on that let me know.
> >>>>
> >>>>>>
> >>>>>> I already have a gcc built from source (sync-ed with trunk/master) and
> >>>>>> launched the test-suite on it.
> >>>>>>
> >>>>>> I am currently in process of understanding the primilary patch
> >>>>>> (https://gcc.gnu.org/legacy-ml/gcc/2014-09/msg00340.html), and
> >>>>>> experimenting with it.
> >>>>>>
> >>>>>> are there any other things I should be aware of (useful Doc/blog or a
> >>>>>> bug tracking the project) before proceeding further ?
> >>>>>
> >>>>> I think it is pretty much all that exists.  Basically we will need to
> >>>>> implement everything that is necessary to stream out valid object file
> >>>>> directly from GCC rather than going through gas.  The experimental
> >>>>> prototype sort of worked but it was lacking few things.
> >>>>
> >>>> When I try to apply that patch on my local branch ( branched from trunk ),
> >>>> it seem to be incompatible with the current working tree. Is there a
> >>>> specific branch that I have to apply it to ? or is it due to the recent
> >>>> file rename patch ( changing extensions from .c to .cc ) ?
> >>>>
> >>>> ```
> >>>> $ git apply --check bypass_asm_patch
> >>>>
> >>>> error: patch failed: Makefile.in:1300
> >>>> error: Makefile.in: patch does not apply
> >>>> error: common.opt: No such file or directory
> >>>> error: langhooks.c: No such file or directory
> >>>> error: lto/Make-lang.in: No such file or directory
> >>>> error: lto/lto-object.c: No such file or directory
> >>>> error: lto/lto.c: No such file or directory
> >>>> error: lto/lto.h: No such file or directory
> >>>> error: lto-streamer.h: No such file or directory
> >>>> error: toplev.c: No such file or directory
> >>>> ```
> >>>
> >>> I can try to update the patch, or it probably should apply to trunk
> >>> checked out around the date I sent the patch.  Indeed we need to change
> >>> c to cc but there are likely more changes since then - most importnatly
> >>> the early debug info.
> >>> At I will see how easy/hard is to make the patch build with current
> >>> trunk.
> >> We do have ideas for the early debug with the asm-out abstraction to
> >> also solve a different issue (missing simple-object support for mingw/darwin).
> >
> > Note the debug info will be stored to a different .s file, then the file
> > will be converted .o by GAS and then the bytecode will be stored to an ELF
> > section of a compiled object. I've got also binutils patch when we'll
> > be able to directly use the embedded section with compile.o@offset.
>
> Which will not work, as written, for Darwin since that is neither ELF nor does it
> use GAS - but hopefully, we can find some equivalent mechanism (there is already
> some support in the Darwin backend for switching asm context for LTO output).

I think using compile.o@offset will be optional and the fallback is to
extract the
"file" back to a regular object file just containing debug info like
we do currently
but with the difference that we do not need to understand the object
format since
it is created "correctly" by the assembler during compile-time (and we just use
the compile-object as a container to not confuse build systems).

> >> Namely assemble the early debug in a different file and include the resulting
> >> native object in binary form in the compile output - not needing to write
> >> assembly .data for that would be a good way to make this more viable.
> >
> > Btw. do you have any estimation how slow is GAS when we speak about debug info?
> > How much time can we save since the latest speed-up achieved by GAS?
> >
> >> You might want to talk to Martin Liska for this who I think had some
> >> prototype on this?
> >
> > I can provide a prototype if needed.
>
> I’d like to be in to loop from the Darwin PoV..
> thanks
> Iain
>
> >
> > Cheers,
> > Martin
> >
> >> Richard.
> >>> Honza
> >>>>
> >>>> Thanks
> >>>> - Ankur
>