[Request for Comments] Using Rust libraries in the Rust frontend

[Request for Comments] Using Rust libraries in the Rust frontend

[Arthur Cohen]

Hi everyone,

In order to increase the development speed of Rust features, we are 
seeking feedback on reusing some Rust components directly within our 
front-end. As mentioned in other conferences, the most important 
component we would like to integrate into the front-end is the polonius 
borrow-checker. Another component we've become interested in is the 
`rustc_format_parser` library, responsible for handling parsing and 
handling of format arguments as described in the documentation for 
`std::fmt` [1].

However, since these libraries are written in Rust, GCC in itself is not 
yet able to compile them. They all depend on the Rust standard library, 
which we cannot yet compile and link properly. This obviously raises a 
question - how to actually compile, integrate and distribute these 
components?

We do have the option to rewrite them from scratch, but we feel that 
spending time on these existing, correct, battle-tested and easily 
integrable components instead of focusing on other aspects of the 
language would be a mistake. Spending this time instead on Rust features 
that we are missing for compiling these components would, in our 
opinion, yield more results, as it would also help in compiling other 
Rust programs.

We could either distribute these components as compiled libraries, or 
look at integrating the official Rust compiler to our build system as a 
temporary measure. I am aware that this would mean restricting the Rust 
GCC front-end to platforms where the official Rust compiler is also 
available, which is less than ideal. However, this would only be 
temporary - as soon as the Rust front-end is able to compile these 
components, we would simply reuse them and compile them with gccrs as 
part of our bootstrapping process.

The documentation for `std::fmt` [1] describes all of the features 
available in Rust format strings. It also contains a grammar for the 
format-string parser, which we would need to re-implement on top of 
supporting all the formatting features. As a prototype, I wrote an FFI 
interface to the `rustc_format_parser`  library and integrated it to our 
macro expansion system, which took me less than a couple hours. In less 
than an afternoon, we had bindings for all of the exported types and 
functions in the library and had access to a compliant and performant 
Rust format string parser. But re-implementing a correct 
`format_args!()` parser - with the same performance as the Rust one, and 
the same amount of features - would probably take days, if not weeks. 
And we are talking about one of the simplest components we aim to reuse. 
Something like a fully-compliant trait solver for the Rust programming 
language would take months if not years to implement properly from scratch.

I would like to stress once again that relying on distributing compiled 
libraries or using `rustc` in our build system would be temporary, and 
that these measures would be removed as soon as gccrs is able to compile 
these components from source.

I am looking for comments on this decision as well as the best practices 
we could adopt. Have there been any similar decisions for other 
self-hosted front-ends? Any previous email threads/commits that you 
could point me to would be greatly appreciated.

Thanks,

Arthur

[1]: https://doc.rust-lang.org/std/fmt/

Re: [Request for Comments] Using Rust libraries in the Rust frontend

[Richard Biener]

On Mon, Jan 22, 2024 at 7:51 PM Arthur Cohen <arthur.cohen@embecosm.com> wrote:
>
> Hi everyone,
>
> In order to increase the development speed of Rust features, we are
> seeking feedback on reusing some Rust components directly within our
> front-end. As mentioned in other conferences, the most important
> component we would like to integrate into the front-end is the polonius
> borrow-checker. Another component we've become interested in is the
> `rustc_format_parser` library, responsible for handling parsing and
> handling of format arguments as described in the documentation for
> `std::fmt` [1].
>
> However, since these libraries are written in Rust, GCC in itself is not
> yet able to compile them. They all depend on the Rust standard library,
> which we cannot yet compile and link properly. This obviously raises a
> question - how to actually compile, integrate and distribute these
> components?
>
> We do have the option to rewrite them from scratch, but we feel that
> spending time on these existing, correct, battle-tested and easily
> integrable components instead of focusing on other aspects of the
> language would be a mistake. Spending this time instead on Rust features
> that we are missing for compiling these components would, in our
> opinion, yield more results, as it would also help in compiling other
> Rust programs.
>
> We could either distribute these components as compiled libraries, or
> look at integrating the official Rust compiler to our build system as a
> temporary measure. I am aware that this would mean restricting the Rust
> GCC front-end to platforms where the official Rust compiler is also
> available, which is less than ideal.

But that's only for the host part - you can still cross compile to another
target and possibly, once the GCC frontend can compile these libraries
itself, also use them to bootstrap a hosted version on that target -
speaking of ..

> However, this would only be
> temporary - as soon as the Rust front-end is able to compile these
> components, we would simply reuse them and compile them with gccrs as
> part of our bootstrapping process.

.. gccrs would then need to be able to build itself without those modules,
at least during stage1 so that the stage1 compiler can then be used to
build them.  Or you go like m2 and build a "mini-rust" that's just capable
of building the modules.

I think re-using parts already available is very sensible at this point.  Note
that while we might temporarily require a host rust compiler to boostrap
gccrs I'd rather not have the build system download something from the
internet - so at least the sources of those dependences need to be in the
GCC repository, possibly in a new toplevel directory.

> The documentation for `std::fmt` [1] describes all of the features
> available in Rust format strings. It also contains a grammar for the
> format-string parser, which we would need to re-implement on top of
> supporting all the formatting features. As a prototype, I wrote an FFI
> interface to the `rustc_format_parser`  library and integrated it to our
> macro expansion system, which took me less than a couple hours. In less
> than an afternoon, we had bindings for all of the exported types and
> functions in the library and had access to a compliant and performant
> Rust format string parser. But re-implementing a correct
> `format_args!()` parser - with the same performance as the Rust one, and
> the same amount of features - would probably take days, if not weeks.
> And we are talking about one of the simplest components we aim to reuse.
> Something like a fully-compliant trait solver for the Rust programming
> language would take months if not years to implement properly from scratch.
>
> I would like to stress once again that relying on distributing compiled
> libraries or using `rustc` in our build system would be temporary, and
> that these measures would be removed as soon as gccrs is able to compile
> these components from source.
>
> I am looking for comments on this decision as well as the best practices
> we could adopt. Have there been any similar decisions for other
> self-hosted front-ends? Any previous email threads/commits that you
> could point me to would be greatly appreciated.

Not in this very same way but the D frontend is a shim around the official
DMD frontend and the Go frontend imports the golang standard library
(but the frontend itself is written in C++ and doesn't use any part of it).

The C++ frontend uses part of the C++ standard library (but it can of
course build that itself - but it requires a host C++ compiler with library).

Richard.

> Thanks,
>
> Arthur
>
> [1]: https://doc.rust-lang.org/std/fmt/

Re: [Request for Comments] Using Rust libraries in the Rust frontend

[Arthur Cohen]

Hi Richard,

On 1/23/24 08:23, Richard Biener wrote:
> On Mon, Jan 22, 2024 at 7:51 PM Arthur Cohen <arthur.cohen@embecosm.com> wrote:
>>
>> Hi everyone,
>>
>> In order to increase the development speed of Rust features, we are
>> seeking feedback on reusing some Rust components directly within our
>> front-end. As mentioned in other conferences, the most important
>> component we would like to integrate into the front-end is the polonius
>> borrow-checker. Another component we've become interested in is the
>> `rustc_format_parser` library, responsible for handling parsing and
>> handling of format arguments as described in the documentation for
>> `std::fmt` [1].
>>
>> However, since these libraries are written in Rust, GCC in itself is not
>> yet able to compile them. They all depend on the Rust standard library,
>> which we cannot yet compile and link properly. This obviously raises a
>> question - how to actually compile, integrate and distribute these
>> components?
>>
>> We do have the option to rewrite them from scratch, but we feel that
>> spending time on these existing, correct, battle-tested and easily
>> integrable components instead of focusing on other aspects of the
>> language would be a mistake. Spending this time instead on Rust features
>> that we are missing for compiling these components would, in our
>> opinion, yield more results, as it would also help in compiling other
>> Rust programs.
>>
>> We could either distribute these components as compiled libraries, or
>> look at integrating the official Rust compiler to our build system as a
>> temporary measure. I am aware that this would mean restricting the Rust
>> GCC front-end to platforms where the official Rust compiler is also
>> available, which is less than ideal.
> 
> But that's only for the host part - you can still cross compile to another
> target and possibly, once the GCC frontend can compile these libraries
> itself, also use them to bootstrap a hosted version on that target -
> speaking of ..
> 
>> However, this would only be
>> temporary - as soon as the Rust front-end is able to compile these
>> components, we would simply reuse them and compile them with gccrs as
>> part of our bootstrapping process.
> 
> .. gccrs would then need to be able to build itself without those modules,
> at least during stage1 so that the stage1 compiler can then be used to
> build them.  Or you go like m2 and build a "mini-rust" that's just capable
> of building the modules.

Right, that makes a lot of sense. We should definitely be able to build 
the format string parser without a format string parser, as it does not 
use format strings for error handling or anything. And even if it did, 
it would be pretty easy to remove that and do the formatting by hand.

Similarly, the borrow checker is not "needed" for compilation and we do 
plan on building stage1 without it, while making it mandatory for 
stage2/3 builds.

> I think re-using parts already available is very sensible at this point.  Note
> that while we might temporarily require a host rust compiler to boostrap
> gccrs I'd rather not have the build system download something from the
> internet - so at least the sources of those dependences need to be in the
> GCC repository, possibly in a new toplevel directory.

Okay, that makes a lot of sense. I was thinking of adding a basic check 
for the Rust compiler to be present when compiling these components - 
and error out if that isn't the case. Are you suggesting we embed a full 
copy of rustc in GCC and build it from source when compiling the Rust 
frontend? Or am I misunderstanding?

>> The documentation for `std::fmt` [1] describes all of the features
>> available in Rust format strings. It also contains a grammar for the
>> format-string parser, which we would need to re-implement on top of
>> supporting all the formatting features. As a prototype, I wrote an FFI
>> interface to the `rustc_format_parser`  library and integrated it to our
>> macro expansion system, which took me less than a couple hours. In less
>> than an afternoon, we had bindings for all of the exported types and
>> functions in the library and had access to a compliant and performant
>> Rust format string parser. But re-implementing a correct
>> `format_args!()` parser - with the same performance as the Rust one, and
>> the same amount of features - would probably take days, if not weeks.
>> And we are talking about one of the simplest components we aim to reuse.
>> Something like a fully-compliant trait solver for the Rust programming
>> language would take months if not years to implement properly from scratch.
>>
>> I would like to stress once again that relying on distributing compiled
>> libraries or using `rustc` in our build system would be temporary, and
>> that these measures would be removed as soon as gccrs is able to compile
>> these components from source.
>>
>> I am looking for comments on this decision as well as the best practices
>> we could adopt. Have there been any similar decisions for other
>> self-hosted front-ends? Any previous email threads/commits that you
>> could point me to would be greatly appreciated.
> 
> Not in this very same way but the D frontend is a shim around the official
> DMD frontend and the Go frontend imports the golang standard library
> (but the frontend itself is written in C++ and doesn't use any part of it).
> 
> The C++ frontend uses part of the C++ standard library (but it can of
> course build that itself - but it requires a host C++ compiler with library).

Thanks for the pointers. I was wondering if this is something that the 
Ada frontend had faced at the beginning, but I've been told it does not 
have a lot of dependencies anyway so this might not be helpful.

> 
> Richard.
> 
>> Thanks,
>>
>> Arthur
>>
>> [1]: https://doc.rust-lang.org/std/fmt/

Thanks a lot for taking the time, I really appreciate it.

Best,

Arthur

Re: [Request for Comments] Using Rust libraries in the Rust frontend

[connor horman]

[-- Attachment #1: Type: text/plain, Size: 6650 bytes --]

Note that it may not make sense to include a source copy of rustc, as that
will itself require an (earlier) stage of rustc to build. mrustc can offer
a bootstrap for 1.54, but depending on the versions required, you may need
upwards of 10 additional rustc sources.

On Thu, 25 Jan 2024 at 10:04, Arthur Cohen <arthur.cohen@embecosm.com>
wrote:

> Hi Richard,
>
> On 1/23/24 08:23, Richard Biener wrote:
> > On Mon, Jan 22, 2024 at 7:51 PM Arthur Cohen <arthur.cohen@embecosm.com>
> wrote:
> >>
> >> Hi everyone,
> >>
> >> In order to increase the development speed of Rust features, we are
> >> seeking feedback on reusing some Rust components directly within our
> >> front-end. As mentioned in other conferences, the most important
> >> component we would like to integrate into the front-end is the polonius
> >> borrow-checker. Another component we've become interested in is the
> >> `rustc_format_parser` library, responsible for handling parsing and
> >> handling of format arguments as described in the documentation for
> >> `std::fmt` [1].
> >>
> >> However, since these libraries are written in Rust, GCC in itself is not
> >> yet able to compile them. They all depend on the Rust standard library,
> >> which we cannot yet compile and link properly. This obviously raises a
> >> question - how to actually compile, integrate and distribute these
> >> components?
> >>
> >> We do have the option to rewrite them from scratch, but we feel that
> >> spending time on these existing, correct, battle-tested and easily
> >> integrable components instead of focusing on other aspects of the
> >> language would be a mistake. Spending this time instead on Rust features
> >> that we are missing for compiling these components would, in our
> >> opinion, yield more results, as it would also help in compiling other
> >> Rust programs.
> >>
> >> We could either distribute these components as compiled libraries, or
> >> look at integrating the official Rust compiler to our build system as a
> >> temporary measure. I am aware that this would mean restricting the Rust
> >> GCC front-end to platforms where the official Rust compiler is also
> >> available, which is less than ideal.
> >
> > But that's only for the host part - you can still cross compile to
> another
> > target and possibly, once the GCC frontend can compile these libraries
> > itself, also use them to bootstrap a hosted version on that target -
> > speaking of ..
> >
> >> However, this would only be
> >> temporary - as soon as the Rust front-end is able to compile these
> >> components, we would simply reuse them and compile them with gccrs as
> >> part of our bootstrapping process.
> >
> > .. gccrs would then need to be able to build itself without those
> modules,
> > at least during stage1 so that the stage1 compiler can then be used to
> > build them.  Or you go like m2 and build a "mini-rust" that's just
> capable
> > of building the modules.
>
> Right, that makes a lot of sense. We should definitely be able to build
> the format string parser without a format string parser, as it does not
> use format strings for error handling or anything. And even if it did,
> it would be pretty easy to remove that and do the formatting by hand.
>
> Similarly, the borrow checker is not "needed" for compilation and we do
> plan on building stage1 without it, while making it mandatory for
> stage2/3 builds.
>
> > I think re-using parts already available is very sensible at this
> point.  Note
> > that while we might temporarily require a host rust compiler to boostrap
> > gccrs I'd rather not have the build system download something from the
> > internet - so at least the sources of those dependences need to be in the
> > GCC repository, possibly in a new toplevel directory.
>
> Okay, that makes a lot of sense. I was thinking of adding a basic check
> for the Rust compiler to be present when compiling these components -
> and error out if that isn't the case. Are you suggesting we embed a full
> copy of rustc in GCC and build it from source when compiling the Rust
> frontend? Or am I misunderstanding?
>
> >> The documentation for `std::fmt` [1] describes all of the features
> >> available in Rust format strings. It also contains a grammar for the
> >> format-string parser, which we would need to re-implement on top of
> >> supporting all the formatting features. As a prototype, I wrote an FFI
> >> interface to the `rustc_format_parser`  library and integrated it to our
> >> macro expansion system, which took me less than a couple hours. In less
> >> than an afternoon, we had bindings for all of the exported types and
> >> functions in the library and had access to a compliant and performant
> >> Rust format string parser. But re-implementing a correct
> >> `format_args!()` parser - with the same performance as the Rust one, and
> >> the same amount of features - would probably take days, if not weeks.
> >> And we are talking about one of the simplest components we aim to reuse.
> >> Something like a fully-compliant trait solver for the Rust programming
> >> language would take months if not years to implement properly from
> scratch.
> >>
> >> I would like to stress once again that relying on distributing compiled
> >> libraries or using `rustc` in our build system would be temporary, and
> >> that these measures would be removed as soon as gccrs is able to compile
> >> these components from source.
> >>
> >> I am looking for comments on this decision as well as the best practices
> >> we could adopt. Have there been any similar decisions for other
> >> self-hosted front-ends? Any previous email threads/commits that you
> >> could point me to would be greatly appreciated.
> >
> > Not in this very same way but the D frontend is a shim around the
> official
> > DMD frontend and the Go frontend imports the golang standard library
> > (but the frontend itself is written in C++ and doesn't use any part of
> it).
> >
> > The C++ frontend uses part of the C++ standard library (but it can of
> > course build that itself - but it requires a host C++ compiler with
> library).
>
> Thanks for the pointers. I was wondering if this is something that the
> Ada frontend had faced at the beginning, but I've been told it does not
> have a lot of dependencies anyway so this might not be helpful.
>
> >
> > Richard.
> >
> >> Thanks,
> >>
> >> Arthur
> >>
> >> [1]: https://doc.rust-lang.org/std/fmt/
>
> Thanks a lot for taking the time, I really appreciate it.
>
> Best,
>
> Arthur
>

Re: [Request for Comments] Using Rust libraries in the Rust frontend

[Iain Sandoe]

Hi Arthur,

> On 25 Jan 2024, at 15:03, Arthur Cohen <arthur.cohen@embecosm.com> wrote:

> On 1/23/24 08:23, Richard Biener wrote:
>> On Mon, Jan 22, 2024 at 7:51 PM Arthur Cohen <arthur.cohen@embecosm.com> wrote:
>>> 
>>> Hi everyone,
>>> 
>>> In order to increase the development speed of Rust features, we are
>>> seeking feedback on reusing some Rust components directly within our
>>> front-end. As mentioned in other conferences, the most important
>>> component we would like to integrate into the front-end is the polonius
>>> borrow-checker. Another component we've become interested in is the
>>> `rustc_format_parser` library, responsible for handling parsing and
>>> handling of format arguments as described in the documentation for
>>> `std::fmt` [1].
>>> 
>>> However, since these libraries are written in Rust, GCC in itself is not
>>> yet able to compile them. They all depend on the Rust standard library,
>>> which we cannot yet compile and link properly. This obviously raises a
>>> question - how to actually compile, integrate and distribute these
>>> components?
>>> 
>>> We do have the option to rewrite them from scratch, but we feel that
>>> spending time on these existing, correct, battle-tested and easily
>>> integrable components instead of focusing on other aspects of the
>>> language would be a mistake. Spending this time instead on Rust features
>>> that we are missing for compiling these components would, in our
>>> opinion, yield more results, as it would also help in compiling other
>>> Rust programs.
>>> 
>>> We could either distribute these components as compiled libraries, or
>>> look at integrating the official Rust compiler to our build system as a
>>> temporary measure. I am aware that this would mean restricting the Rust
>>> GCC front-end to platforms where the official Rust compiler is also
>>> available, which is less than ideal.
>> But that's only for the host part - you can still cross compile to another
>> target and possibly, once the GCC frontend can compile these libraries
>> itself, also use them to bootstrap a hosted version on that target -
>> speaking of ..
>>> However, this would only be
>>> temporary - as soon as the Rust front-end is able to compile these
>>> components, we would simply reuse them and compile them with gccrs as
>>> part of our bootstrapping process.
>> .. gccrs would then need to be able to build itself without those modules,
>> at least during stage1 so that the stage1 compiler can then be used to
>> build them.  Or you go like m2 and build a "mini-rust" that's just capable
>> of building the modules.
> 
> Right, that makes a lot of sense. We should definitely be able to build the format string parser without a format string parser, as it does not use format strings for error handling or anything. And even if it did, it would be pretty easy to remove that and do the formatting by hand.
> 
> Similarly, the borrow checker is not "needed" for compilation and we do plan on building stage1 without it, while making it mandatory for stage2/3 builds.

As long as you leave a route available for porting it to either new (or old) systems which do not have rustc (or any other rust compiler available).

E.g. with Ada it is possible to port to a new platform by first building a cross-compiler and then to use that cross-compiler to build a “native cross” (build != host == target) to provide an initial compiler on the target platform.

Those cross compilers are not bootstrapped (it is a single stage) - but it is reasonable to make it a condition that the $build compiler is the same (even exact) version as the sources you are trying to port to the new platform.

thanks
Iain

>> I think re-using parts already available is very sensible at this point.  Note
>> that while we might temporarily require a host rust compiler to boostrap
>> gccrs I'd rather not have the build system download something from the
>> internet - so at least the sources of those dependences need to be in the
>> GCC repository, possibly in a new toplevel directory.
> 
> Okay, that makes a lot of sense. I was thinking of adding a basic check for the Rust compiler to be present when compiling these components - and error out if that isn't the case. Are you suggesting we embed a full copy of rustc in GCC and build it from source when compiling the Rust frontend? Or am I misunderstanding?
> 
>>> The documentation for `std::fmt` [1] describes all of the features
>>> available in Rust format strings. It also contains a grammar for the
>>> format-string parser, which we would need to re-implement on top of
>>> supporting all the formatting features. As a prototype, I wrote an FFI
>>> interface to the `rustc_format_parser`  library and integrated it to our
>>> macro expansion system, which took me less than a couple hours. In less
>>> than an afternoon, we had bindings for all of the exported types and
>>> functions in the library and had access to a compliant and performant
>>> Rust format string parser. But re-implementing a correct
>>> `format_args!()` parser - with the same performance as the Rust one, and
>>> the same amount of features - would probably take days, if not weeks.
>>> And we are talking about one of the simplest components we aim to reuse.
>>> Something like a fully-compliant trait solver for the Rust programming
>>> language would take months if not years to implement properly from scratch.
>>> 
>>> I would like to stress once again that relying on distributing compiled
>>> libraries or using `rustc` in our build system would be temporary, and
>>> that these measures would be removed as soon as gccrs is able to compile
>>> these components from source.
>>> 
>>> I am looking for comments on this decision as well as the best practices
>>> we could adopt. Have there been any similar decisions for other
>>> self-hosted front-ends? Any previous email threads/commits that you
>>> could point me to would be greatly appreciated.
>> Not in this very same way but the D frontend is a shim around the official
>> DMD frontend and the Go frontend imports the golang standard library
>> (but the frontend itself is written in C++ and doesn't use any part of it).
>> The C++ frontend uses part of the C++ standard library (but it can of
>> course build that itself - but it requires a host C++ compiler with library).
> 
> Thanks for the pointers. I was wondering if this is something that the Ada frontend had faced at the beginning, but I've been told it does not have a lot of dependencies anyway so this might not be helpful.
> 
>> Richard.
>>> Thanks,
>>> 
>>> Arthur
>>> 
>>> [1]: https://doc.rust-lang.org/std/fmt/
> 
> Thanks a lot for taking the time, I really appreciate it.
> 
> Best,
> 
> Arthur

Re: [Request for Comments] Using Rust libraries in the Rust frontend

[Richard Biener]

> Am 25.01.2024 um 16:03 schrieb Arthur Cohen <arthur.cohen@embecosm.com>:
> 
> Hi Richard,
> 
>> On 1/23/24 08:23, Richard Biener wrote:
>>> On Mon, Jan 22, 2024 at 7:51 PM Arthur Cohen <arthur.cohen@embecosm.com> wrote:
>>> 
>>> Hi everyone,
>>> 
>>> In order to increase the development speed of Rust features, we are
>>> seeking feedback on reusing some Rust components directly within our
>>> front-end. As mentioned in other conferences, the most important
>>> component we would like to integrate into the front-end is the polonius
>>> borrow-checker. Another component we've become interested in is the
>>> `rustc_format_parser` library, responsible for handling parsing and
>>> handling of format arguments as described in the documentation for
>>> `std::fmt` [1].
>>> 
>>> However, since these libraries are written in Rust, GCC in itself is not
>>> yet able to compile them. They all depend on the Rust standard library,
>>> which we cannot yet compile and link properly. This obviously raises a
>>> question - how to actually compile, integrate and distribute these
>>> components?
>>> 
>>> We do have the option to rewrite them from scratch, but we feel that
>>> spending time on these existing, correct, battle-tested and easily
>>> integrable components instead of focusing on other aspects of the
>>> language would be a mistake. Spending this time instead on Rust features
>>> that we are missing for compiling these components would, in our
>>> opinion, yield more results, as it would also help in compiling other
>>> Rust programs.
>>> 
>>> We could either distribute these components as compiled libraries, or
>>> look at integrating the official Rust compiler to our build system as a
>>> temporary measure. I am aware that this would mean restricting the Rust
>>> GCC front-end to platforms where the official Rust compiler is also
>>> available, which is less than ideal.
>> But that's only for the host part - you can still cross compile to another
>> target and possibly, once the GCC frontend can compile these libraries
>> itself, also use them to bootstrap a hosted version on that target -
>> speaking of ..
>>> However, this would only be
>>> temporary - as soon as the Rust front-end is able to compile these
>>> components, we would simply reuse them and compile them with gccrs as
>>> part of our bootstrapping process.
>> .. gccrs would then need to be able to build itself without those modules,
>> at least during stage1 so that the stage1 compiler can then be used to
>> build them.  Or you go like m2 and build a "mini-rust" that's just capable
>> of building the modules.
> 
> Right, that makes a lot of sense. We should definitely be able to build the format string parser without a format string parser, as it does not use format strings for error handling or anything. And even if it did, it would be pretty easy to remove that and do the formatting by hand.
> 
> Similarly, the borrow checker is not "needed" for compilation and we do plan on building stage1 without it, while making it mandatory for stage2/3 builds.
> 
>> I think re-using parts already available is very sensible at this point.  Note
>> that while we might temporarily require a host rust compiler to boostrap
>> gccrs I'd rather not have the build system download something from the
>> internet - so at least the sources of those dependences need to be in the
>> GCC repository, possibly in a new toplevel directory.
> 
> Okay, that makes a lot of sense. I was thinking of adding a basic check for the Rust compiler to be present when compiling these components - and error out if that isn't the case. Are you suggesting we embed a full copy of rustc in GCC and build it from source when compiling the Rust frontend? Or am I misunderstanding?

No, you’d have the format parser and all it’s dependencies in the source tree but rely on an installed rustc to build that.  Maybe that doesn’t make sense - I’m not too familiar with the rust compilation model.

Richard 

>>> The documentation for `std::fmt` [1] describes all of the features
>>> available in Rust format strings. It also contains a grammar for the
>>> format-string parser, which we would need to re-implement on top of
>>> supporting all the formatting features. As a prototype, I wrote an FFI
>>> interface to the `rustc_format_parser`  library and integrated it to our
>>> macro expansion system, which took me less than a couple hours. In less
>>> than an afternoon, we had bindings for all of the exported types and
>>> functions in the library and had access to a compliant and performant
>>> Rust format string parser. But re-implementing a correct
>>> `format_args!()` parser - with the same performance as the Rust one, and
>>> the same amount of features - would probably take days, if not weeks.
>>> And we are talking about one of the simplest components we aim to reuse.
>>> Something like a fully-compliant trait solver for the Rust programming
>>> language would take months if not years to implement properly from scratch.
>>> 
>>> I would like to stress once again that relying on distributing compiled
>>> libraries or using `rustc` in our build system would be temporary, and
>>> that these measures would be removed as soon as gccrs is able to compile
>>> these components from source.
>>> 
>>> I am looking for comments on this decision as well as the best practices
>>> we could adopt. Have there been any similar decisions for other
>>> self-hosted front-ends? Any previous email threads/commits that you
>>> could point me to would be greatly appreciated.
>> Not in this very same way but the D frontend is a shim around the official
>> DMD frontend and the Go frontend imports the golang standard library
>> (but the frontend itself is written in C++ and doesn't use any part of it).
>> The C++ frontend uses part of the C++ standard library (but it can of
>> course build that itself - but it requires a host C++ compiler with library).
> 
> Thanks for the pointers. I was wondering if this is something that the Ada frontend had faced at the beginning, but I've been told it does not have a lot of dependencies anyway so this might not be helpful.
> 
>> Richard.
>>> Thanks,
>>> 
>>> Arthur
>>> 
>>> [1]: https://doc.rust-lang.org/std/fmt/
> 
> Thanks a lot for taking the time, I really appreciate it.
> 
> Best,
> 
> Arthur

Re: [Request for Comments] Using Rust libraries in the Rust frontend

[Arthur Cohen]

On 1/25/24 19:19, Richard Biener wrote:
> 
> 
>> Am 25.01.2024 um 16:03 schrieb Arthur Cohen <arthur.cohen@embecosm.com>:
>>
>> Hi Richard,
>>
>>> On 1/23/24 08:23, Richard Biener wrote:
>>>> On Mon, Jan 22, 2024 at 7:51 PM Arthur Cohen <arthur.cohen@embecosm.com> wrote:
>>>>
>>>> Hi everyone,
>>>>
>>>> In order to increase the development speed of Rust features, we are
>>>> seeking feedback on reusing some Rust components directly within our
>>>> front-end. As mentioned in other conferences, the most important
>>>> component we would like to integrate into the front-end is the polonius
>>>> borrow-checker. Another component we've become interested in is the
>>>> `rustc_format_parser` library, responsible for handling parsing and
>>>> handling of format arguments as described in the documentation for
>>>> `std::fmt` [1].
>>>>
>>>> However, since these libraries are written in Rust, GCC in itself is not
>>>> yet able to compile them. They all depend on the Rust standard library,
>>>> which we cannot yet compile and link properly. This obviously raises a
>>>> question - how to actually compile, integrate and distribute these
>>>> components?
>>>>
>>>> We do have the option to rewrite them from scratch, but we feel that
>>>> spending time on these existing, correct, battle-tested and easily
>>>> integrable components instead of focusing on other aspects of the
>>>> language would be a mistake. Spending this time instead on Rust features
>>>> that we are missing for compiling these components would, in our
>>>> opinion, yield more results, as it would also help in compiling other
>>>> Rust programs.
>>>>
>>>> We could either distribute these components as compiled libraries, or
>>>> look at integrating the official Rust compiler to our build system as a
>>>> temporary measure. I am aware that this would mean restricting the Rust
>>>> GCC front-end to platforms where the official Rust compiler is also
>>>> available, which is less than ideal.
>>> But that's only for the host part - you can still cross compile to another
>>> target and possibly, once the GCC frontend can compile these libraries
>>> itself, also use them to bootstrap a hosted version on that target -
>>> speaking of ..
>>>> However, this would only be
>>>> temporary - as soon as the Rust front-end is able to compile these
>>>> components, we would simply reuse them and compile them with gccrs as
>>>> part of our bootstrapping process.
>>> .. gccrs would then need to be able to build itself without those modules,
>>> at least during stage1 so that the stage1 compiler can then be used to
>>> build them.  Or you go like m2 and build a "mini-rust" that's just capable
>>> of building the modules.
>>
>> Right, that makes a lot of sense. We should definitely be able to build the format string parser without a format string parser, as it does not use format strings for error handling or anything. And even if it did, it would be pretty easy to remove that and do the formatting by hand.
>>
>> Similarly, the borrow checker is not "needed" for compilation and we do plan on building stage1 without it, while making it mandatory for stage2/3 builds.
>>
>>> I think re-using parts already available is very sensible at this point.  Note
>>> that while we might temporarily require a host rust compiler to boostrap
>>> gccrs I'd rather not have the build system download something from the
>>> internet - so at least the sources of those dependences need to be in the
>>> GCC repository, possibly in a new toplevel directory.
>>
>> Okay, that makes a lot of sense. I was thinking of adding a basic check for the Rust compiler to be present when compiling these components - and error out if that isn't the case. Are you suggesting we embed a full copy of rustc in GCC and build it from source when compiling the Rust frontend? Or am I misunderstanding?
> 
> No, you’d have the format parser and all it’s dependencies in the source tree but rely on an installed rustc to build that.  Maybe that doesn’t make sense - I’m not too familiar with the rust compilation model.

Ah, no worries that makes total sense! This is what I was planning on 
doing anyways. If compiling directly with rustc, you can specify paths 
where the compiler should look for already compiled crates. If you 
instead use cargo, the Rust build system, you can specify a path for 
specific dependencies.

Thanks,

Arthur

> 
> Richard
> 
>>>> The documentation for `std::fmt` [1] describes all of the features
>>>> available in Rust format strings. It also contains a grammar for the
>>>> format-string parser, which we would need to re-implement on top of
>>>> supporting all the formatting features. As a prototype, I wrote an FFI
>>>> interface to the `rustc_format_parser`  library and integrated it to our
>>>> macro expansion system, which took me less than a couple hours. In less
>>>> than an afternoon, we had bindings for all of the exported types and
>>>> functions in the library and had access to a compliant and performant
>>>> Rust format string parser. But re-implementing a correct
>>>> `format_args!()` parser - with the same performance as the Rust one, and
>>>> the same amount of features - would probably take days, if not weeks.
>>>> And we are talking about one of the simplest components we aim to reuse.
>>>> Something like a fully-compliant trait solver for the Rust programming
>>>> language would take months if not years to implement properly from scratch.
>>>>
>>>> I would like to stress once again that relying on distributing compiled
>>>> libraries or using `rustc` in our build system would be temporary, and
>>>> that these measures would be removed as soon as gccrs is able to compile
>>>> these components from source.
>>>>
>>>> I am looking for comments on this decision as well as the best practices
>>>> we could adopt. Have there been any similar decisions for other
>>>> self-hosted front-ends? Any previous email threads/commits that you
>>>> could point me to would be greatly appreciated.
>>> Not in this very same way but the D frontend is a shim around the official
>>> DMD frontend and the Go frontend imports the golang standard library
>>> (but the frontend itself is written in C++ and doesn't use any part of it).
>>> The C++ frontend uses part of the C++ standard library (but it can of
>>> course build that itself - but it requires a host C++ compiler with library).
>>
>> Thanks for the pointers. I was wondering if this is something that the Ada frontend had faced at the beginning, but I've been told it does not have a lot of dependencies anyway so this might not be helpful.
>>
>>> Richard.
>>>> Thanks,
>>>>
>>>> Arthur
>>>>
>>>> [1]: https://doc.rust-lang.org/std/fmt/
>>
>> Thanks a lot for taking the time, I really appreciate it.
>>
>> Best,
>>
>> Arthur

Re: [Request for Comments] Using Rust libraries in the Rust frontend

[Arthur Cohen]

Hi Iain,

On 1/25/24 17:26, Iain Sandoe wrote:
> Hi Arthur,
> 
>> On 25 Jan 2024, at 15:03, Arthur Cohen <arthur.cohen@embecosm.com> wrote:
> 
>> On 1/23/24 08:23, Richard Biener wrote:
>>> On Mon, Jan 22, 2024 at 7:51 PM Arthur Cohen <arthur.cohen@embecosm.com> wrote:
>>>>
>>>> Hi everyone,
>>>>
>>>> In order to increase the development speed of Rust features, we are
>>>> seeking feedback on reusing some Rust components directly within our
>>>> front-end. As mentioned in other conferences, the most important
>>>> component we would like to integrate into the front-end is the polonius
>>>> borrow-checker. Another component we've become interested in is the
>>>> `rustc_format_parser` library, responsible for handling parsing and
>>>> handling of format arguments as described in the documentation for
>>>> `std::fmt` [1].
>>>>
>>>> However, since these libraries are written in Rust, GCC in itself is not
>>>> yet able to compile them. They all depend on the Rust standard library,
>>>> which we cannot yet compile and link properly. This obviously raises a
>>>> question - how to actually compile, integrate and distribute these
>>>> components?
>>>>
>>>> We do have the option to rewrite them from scratch, but we feel that
>>>> spending time on these existing, correct, battle-tested and easily
>>>> integrable components instead of focusing on other aspects of the
>>>> language would be a mistake. Spending this time instead on Rust features
>>>> that we are missing for compiling these components would, in our
>>>> opinion, yield more results, as it would also help in compiling other
>>>> Rust programs.
>>>>
>>>> We could either distribute these components as compiled libraries, or
>>>> look at integrating the official Rust compiler to our build system as a
>>>> temporary measure. I am aware that this would mean restricting the Rust
>>>> GCC front-end to platforms where the official Rust compiler is also
>>>> available, which is less than ideal.
>>> But that's only for the host part - you can still cross compile to another
>>> target and possibly, once the GCC frontend can compile these libraries
>>> itself, also use them to bootstrap a hosted version on that target -
>>> speaking of ..
>>>> However, this would only be
>>>> temporary - as soon as the Rust front-end is able to compile these
>>>> components, we would simply reuse them and compile them with gccrs as
>>>> part of our bootstrapping process.
>>> .. gccrs would then need to be able to build itself without those modules,
>>> at least during stage1 so that the stage1 compiler can then be used to
>>> build them.  Or you go like m2 and build a "mini-rust" that's just capable
>>> of building the modules.
>>
>> Right, that makes a lot of sense. We should definitely be able to build the format string parser without a format string parser, as it does not use format strings for error handling or anything. And even if it did, it would be pretty easy to remove that and do the formatting by hand.
>>
>> Similarly, the borrow checker is not "needed" for compilation and we do plan on building stage1 without it, while making it mandatory for stage2/3 builds.
> 
> As long as you leave a route available for porting it to either new (or old) systems which do not have rustc (or any other rust compiler available).
> 
> E.g. with Ada it is possible to port to a new platform by first building a cross-compiler and then to use that cross-compiler to build a “native cross” (build != host == target) to provide an initial compiler on the target platform.

Thanks, this is really helpful. I'll have to figure out how to make that 
happen in the build system but this is definitely something we'd be 
interested in doing.

Thanks,

Arthur

> 
> Those cross compilers are not bootstrapped (it is a single stage) - but it is reasonable to make it a condition that the $build compiler is the same (even exact) version as the sources you are trying to port to the new platform.
> 
> thanks
> Iain
> 
>>> I think re-using parts already available is very sensible at this point.  Note
>>> that while we might temporarily require a host rust compiler to boostrap
>>> gccrs I'd rather not have the build system download something from the
>>> internet - so at least the sources of those dependences need to be in the
>>> GCC repository, possibly in a new toplevel directory.
>>
>> Okay, that makes a lot of sense. I was thinking of adding a basic check for the Rust compiler to be present when compiling these components - and error out if that isn't the case. Are you suggesting we embed a full copy of rustc in GCC and build it from source when compiling the Rust frontend? Or am I misunderstanding?
>>
>>>> The documentation for `std::fmt` [1] describes all of the features
>>>> available in Rust format strings. It also contains a grammar for the
>>>> format-string parser, which we would need to re-implement on top of
>>>> supporting all the formatting features. As a prototype, I wrote an FFI
>>>> interface to the `rustc_format_parser`  library and integrated it to our
>>>> macro expansion system, which took me less than a couple hours. In less
>>>> than an afternoon, we had bindings for all of the exported types and
>>>> functions in the library and had access to a compliant and performant
>>>> Rust format string parser. But re-implementing a correct
>>>> `format_args!()` parser - with the same performance as the Rust one, and
>>>> the same amount of features - would probably take days, if not weeks.
>>>> And we are talking about one of the simplest components we aim to reuse.
>>>> Something like a fully-compliant trait solver for the Rust programming
>>>> language would take months if not years to implement properly from scratch.
>>>>
>>>> I would like to stress once again that relying on distributing compiled
>>>> libraries or using `rustc` in our build system would be temporary, and
>>>> that these measures would be removed as soon as gccrs is able to compile
>>>> these components from source.
>>>>
>>>> I am looking for comments on this decision as well as the best practices
>>>> we could adopt. Have there been any similar decisions for other
>>>> self-hosted front-ends? Any previous email threads/commits that you
>>>> could point me to would be greatly appreciated.
>>> Not in this very same way but the D frontend is a shim around the official
>>> DMD frontend and the Go frontend imports the golang standard library
>>> (but the frontend itself is written in C++ and doesn't use any part of it).
>>> The C++ frontend uses part of the C++ standard library (but it can of
>>> course build that itself - but it requires a host C++ compiler with library).
>>
>> Thanks for the pointers. I was wondering if this is something that the Ada frontend had faced at the beginning, but I've been told it does not have a lot of dependencies anyway so this might not be helpful.
>>
>>> Richard.
>>>> Thanks,
>>>>
>>>> Arthur
>>>>
>>>> [1]: https://doc.rust-lang.org/std/fmt/
>>
>> Thanks a lot for taking the time, I really appreciate it.
>>
>> Best,
>>
>> Arthur
> 

Re: [Request for Comments] Using Rust libraries in the Rust frontend

[Arthur Cohen]

Hi Connor,

On 1/25/24 16:55, connor horman wrote:
> Note that it may not make sense to include a source copy of rustc, as 
> that will itself require an (earlier) stage of rustc to build. mrustc 
> can offer a bootstrap for 1.54, but depending on the versions required, 
> you may need upwards of 10 additional rustc sources.

Yes, I was a bit worried about the idea but it seems this is just me not 
knowing how to read so all is good. Using mrustc for this is a great 
idea however.

Thanks!

Arthur

> 
> On Thu, 25 Jan 2024 at 10:04, Arthur Cohen <arthur.cohen@embecosm.com 
> <mailto:arthur.cohen@embecosm.com>> wrote:
> 
>     Hi Richard,
> 
>     On 1/23/24 08:23, Richard Biener wrote:
>      > On Mon, Jan 22, 2024 at 7:51 PM Arthur Cohen
>     <arthur.cohen@embecosm.com <mailto:arthur.cohen@embecosm.com>> wrote:
>      >>
>      >> Hi everyone,
>      >>
>      >> In order to increase the development speed of Rust features, we are
>      >> seeking feedback on reusing some Rust components directly within our
>      >> front-end. As mentioned in other conferences, the most important
>      >> component we would like to integrate into the front-end is the
>     polonius
>      >> borrow-checker. Another component we've become interested in is the
>      >> `rustc_format_parser` library, responsible for handling parsing and
>      >> handling of format arguments as described in the documentation for
>      >> `std::fmt` [1].
>      >>
>      >> However, since these libraries are written in Rust, GCC in
>     itself is not
>      >> yet able to compile them. They all depend on the Rust standard
>     library,
>      >> which we cannot yet compile and link properly. This obviously
>     raises a
>      >> question - how to actually compile, integrate and distribute these
>      >> components?
>      >>
>      >> We do have the option to rewrite them from scratch, but we feel that
>      >> spending time on these existing, correct, battle-tested and easily
>      >> integrable components instead of focusing on other aspects of the
>      >> language would be a mistake. Spending this time instead on Rust
>     features
>      >> that we are missing for compiling these components would, in our
>      >> opinion, yield more results, as it would also help in compiling
>     other
>      >> Rust programs.
>      >>
>      >> We could either distribute these components as compiled
>     libraries, or
>      >> look at integrating the official Rust compiler to our build
>     system as a
>      >> temporary measure. I am aware that this would mean restricting
>     the Rust
>      >> GCC front-end to platforms where the official Rust compiler is also
>      >> available, which is less than ideal.
>      >
>      > But that's only for the host part - you can still cross compile
>     to another
>      > target and possibly, once the GCC frontend can compile these
>     libraries
>      > itself, also use them to bootstrap a hosted version on that target -
>      > speaking of ..
>      >
>      >> However, this would only be
>      >> temporary - as soon as the Rust front-end is able to compile these
>      >> components, we would simply reuse them and compile them with
>     gccrs as
>      >> part of our bootstrapping process.
>      >
>      > .. gccrs would then need to be able to build itself without those
>     modules,
>      > at least during stage1 so that the stage1 compiler can then be
>     used to
>      > build them.  Or you go like m2 and build a "mini-rust" that's
>     just capable
>      > of building the modules.
> 
>     Right, that makes a lot of sense. We should definitely be able to build
>     the format string parser without a format string parser, as it does not
>     use format strings for error handling or anything. And even if it did,
>     it would be pretty easy to remove that and do the formatting by hand.
> 
>     Similarly, the borrow checker is not "needed" for compilation and we do
>     plan on building stage1 without it, while making it mandatory for
>     stage2/3 builds.
> 
>      > I think re-using parts already available is very sensible at this
>     point.  Note
>      > that while we might temporarily require a host rust compiler to
>     boostrap
>      > gccrs I'd rather not have the build system download something
>     from the
>      > internet - so at least the sources of those dependences need to
>     be in the
>      > GCC repository, possibly in a new toplevel directory.
> 
>     Okay, that makes a lot of sense. I was thinking of adding a basic check
>     for the Rust compiler to be present when compiling these components -
>     and error out if that isn't the case. Are you suggesting we embed a
>     full
>     copy of rustc in GCC and build it from source when compiling the Rust
>     frontend? Or am I misunderstanding?
> 
>      >> The documentation for `std::fmt` [1] describes all of the features
>      >> available in Rust format strings. It also contains a grammar for the
>      >> format-string parser, which we would need to re-implement on top of
>      >> supporting all the formatting features. As a prototype, I wrote
>     an FFI
>      >> interface to the `rustc_format_parser`  library and integrated
>     it to our
>      >> macro expansion system, which took me less than a couple hours.
>     In less
>      >> than an afternoon, we had bindings for all of the exported types and
>      >> functions in the library and had access to a compliant and
>     performant
>      >> Rust format string parser. But re-implementing a correct
>      >> `format_args!()` parser - with the same performance as the Rust
>     one, and
>      >> the same amount of features - would probably take days, if not
>     weeks.
>      >> And we are talking about one of the simplest components we aim
>     to reuse.
>      >> Something like a fully-compliant trait solver for the Rust
>     programming
>      >> language would take months if not years to implement properly
>     from scratch.
>      >>
>      >> I would like to stress once again that relying on distributing
>     compiled
>      >> libraries or using `rustc` in our build system would be
>     temporary, and
>      >> that these measures would be removed as soon as gccrs is able to
>     compile
>      >> these components from source.
>      >>
>      >> I am looking for comments on this decision as well as the best
>     practices
>      >> we could adopt. Have there been any similar decisions for other
>      >> self-hosted front-ends? Any previous email threads/commits that you
>      >> could point me to would be greatly appreciated.
>      >
>      > Not in this very same way but the D frontend is a shim around the
>     official
>      > DMD frontend and the Go frontend imports the golang standard library
>      > (but the frontend itself is written in C++ and doesn't use any
>     part of it).
>      >
>      > The C++ frontend uses part of the C++ standard library (but it can of
>      > course build that itself - but it requires a host C++ compiler
>     with library).
> 
>     Thanks for the pointers. I was wondering if this is something that the
>     Ada frontend had faced at the beginning, but I've been told it does not
>     have a lot of dependencies anyway so this might not be helpful.
> 
>      >
>      > Richard.
>      >
>      >> Thanks,
>      >>
>      >> Arthur
>      >>
>      >> [1]: https://doc.rust-lang.org/std/fmt/
>     <https://doc.rust-lang.org/std/fmt/>
> 
>     Thanks a lot for taking the time, I really appreciate it.
> 
>     Best,
> 
>     Arthur
> 

Re: [Request for Comments] Using Rust libraries in the Rust frontend

[NightStrike]

[-- Attachment #1: Type: text/plain, Size: 402 bytes --]

On Thu, Jan 25, 2024, 11:27 Iain Sandoe <iain@sandoe.co.uk> wrote:

> E.g. with Ada it is possible to port to a new platform by first building a
> cross-compiler and then to use that cross-compiler to build a “native
> cross” (build != host == target) to provide an initial compiler on the
> target platform.
>

And that Ada solution is awful for nontraditional systems. Ditto for D.

>

Re: [Request for Comments] Using Rust libraries in the Rust frontend

[NightStrike]

[-- Attachment #1: Type: text/plain, Size: 524 bytes --]

On Mon, Jan 22, 2024, 12:31 Arthur Cohen <arthur.cohen@embecosm.com> wrote:

> I am aware that this would mean restricting the Rust
> GCC front-end to platforms where the official Rust compiler is also
> available, which is less than ideal. However, this would only be
> temporary - as soon as the Rust front-end is able to compile these
> components, we would simply reuse them and compile them with gccrs as
> part of our bootstrapping process.
>

The easiest way to make a permanent change is to make a temporary one.

>

Re: [Request for Comments] Using Rust libraries in the Rust frontend

[David Starner]

On Fri, Jan 26, 2024 at 3:40 PM NightStrike via Gcc <gcc@gcc.gnu.org> wrote:
>
> On Thu, Jan 25, 2024, 11:27 Iain Sandoe <iain@sandoe.co.uk> wrote:
>
> > E.g. with Ada it is possible to port to a new platform by first building a
> > cross-compiler and then to use that cross-compiler to build a “native
> > cross” (build != host == target) to provide an initial compiler on the
> > target platform.
> >
>
> And that Ada solution is awful for nontraditional systems. Ditto for D.

Why? And what solution would you suggest for compilers written in part
in their own languages? How do you get the first C compiler on the
platform without building a cross-compiler?

-- 
The standard is written in English . If you have trouble understanding
a particular section, read it again and again and again . . . Sit up
straight. Eat your vegetables. Do not mumble. -- _Pascal_, ISO 7185
(1991)