Re: [Request for Comments] Using Rust libraries in the Rust frontend

[Arthur Cohen <arthur.cohen@embecosm.com>]

Hi Connor,

On 1/25/24 16:55, connor horman wrote:
> Note that it may not make sense to include a source copy of rustc, as 
> that will itself require an (earlier) stage of rustc to build. mrustc 
> can offer a bootstrap for 1.54, but depending on the versions required, 
> you may need upwards of 10 additional rustc sources.

Yes, I was a bit worried about the idea but it seems this is just me not 
knowing how to read so all is good. Using mrustc for this is a great 
idea however.

Thanks!

Arthur

> 
> On Thu, 25 Jan 2024 at 10:04, Arthur Cohen <arthur.cohen@embecosm.com 
> <mailto:arthur.cohen@embecosm.com>> wrote:
> 
>     Hi Richard,
> 
>     On 1/23/24 08:23, Richard Biener wrote:
>      > On Mon, Jan 22, 2024 at 7:51 PM Arthur Cohen
>     <arthur.cohen@embecosm.com <mailto:arthur.cohen@embecosm.com>> wrote:
>      >>
>      >> Hi everyone,
>      >>
>      >> In order to increase the development speed of Rust features, we are
>      >> seeking feedback on reusing some Rust components directly within our
>      >> front-end. As mentioned in other conferences, the most important
>      >> component we would like to integrate into the front-end is the
>     polonius
>      >> borrow-checker. Another component we've become interested in is the
>      >> `rustc_format_parser` library, responsible for handling parsing and
>      >> handling of format arguments as described in the documentation for
>      >> `std::fmt` [1].
>      >>
>      >> However, since these libraries are written in Rust, GCC in
>     itself is not
>      >> yet able to compile them. They all depend on the Rust standard
>     library,
>      >> which we cannot yet compile and link properly. This obviously
>     raises a
>      >> question - how to actually compile, integrate and distribute these
>      >> components?
>      >>
>      >> We do have the option to rewrite them from scratch, but we feel that
>      >> spending time on these existing, correct, battle-tested and easily
>      >> integrable components instead of focusing on other aspects of the
>      >> language would be a mistake. Spending this time instead on Rust
>     features
>      >> that we are missing for compiling these components would, in our
>      >> opinion, yield more results, as it would also help in compiling
>     other
>      >> Rust programs.
>      >>
>      >> We could either distribute these components as compiled
>     libraries, or
>      >> look at integrating the official Rust compiler to our build
>     system as a
>      >> temporary measure. I am aware that this would mean restricting
>     the Rust
>      >> GCC front-end to platforms where the official Rust compiler is also
>      >> available, which is less than ideal.
>      >
>      > But that's only for the host part - you can still cross compile
>     to another
>      > target and possibly, once the GCC frontend can compile these
>     libraries
>      > itself, also use them to bootstrap a hosted version on that target -
>      > speaking of ..
>      >
>      >> However, this would only be
>      >> temporary - as soon as the Rust front-end is able to compile these
>      >> components, we would simply reuse them and compile them with
>     gccrs as
>      >> part of our bootstrapping process.
>      >
>      > .. gccrs would then need to be able to build itself without those
>     modules,
>      > at least during stage1 so that the stage1 compiler can then be
>     used to
>      > build them.  Or you go like m2 and build a "mini-rust" that's
>     just capable
>      > of building the modules.
> 
>     Right, that makes a lot of sense. We should definitely be able to build
>     the format string parser without a format string parser, as it does not
>     use format strings for error handling or anything. And even if it did,
>     it would be pretty easy to remove that and do the formatting by hand.
> 
>     Similarly, the borrow checker is not "needed" for compilation and we do
>     plan on building stage1 without it, while making it mandatory for
>     stage2/3 builds.
> 
>      > I think re-using parts already available is very sensible at this
>     point.  Note
>      > that while we might temporarily require a host rust compiler to
>     boostrap
>      > gccrs I'd rather not have the build system download something
>     from the
>      > internet - so at least the sources of those dependences need to
>     be in the
>      > GCC repository, possibly in a new toplevel directory.
> 
>     Okay, that makes a lot of sense. I was thinking of adding a basic check
>     for the Rust compiler to be present when compiling these components -
>     and error out if that isn't the case. Are you suggesting we embed a
>     full
>     copy of rustc in GCC and build it from source when compiling the Rust
>     frontend? Or am I misunderstanding?
> 
>      >> The documentation for `std::fmt` [1] describes all of the features
>      >> available in Rust format strings. It also contains a grammar for the
>      >> format-string parser, which we would need to re-implement on top of
>      >> supporting all the formatting features. As a prototype, I wrote
>     an FFI
>      >> interface to the `rustc_format_parser`  library and integrated
>     it to our
>      >> macro expansion system, which took me less than a couple hours.
>     In less
>      >> than an afternoon, we had bindings for all of the exported types and
>      >> functions in the library and had access to a compliant and
>     performant
>      >> Rust format string parser. But re-implementing a correct
>      >> `format_args!()` parser - with the same performance as the Rust
>     one, and
>      >> the same amount of features - would probably take days, if not
>     weeks.
>      >> And we are talking about one of the simplest components we aim
>     to reuse.
>      >> Something like a fully-compliant trait solver for the Rust
>     programming
>      >> language would take months if not years to implement properly
>     from scratch.
>      >>
>      >> I would like to stress once again that relying on distributing
>     compiled
>      >> libraries or using `rustc` in our build system would be
>     temporary, and
>      >> that these measures would be removed as soon as gccrs is able to
>     compile
>      >> these components from source.
>      >>
>      >> I am looking for comments on this decision as well as the best
>     practices
>      >> we could adopt. Have there been any similar decisions for other
>      >> self-hosted front-ends? Any previous email threads/commits that you
>      >> could point me to would be greatly appreciated.
>      >
>      > Not in this very same way but the D frontend is a shim around the
>     official
>      > DMD frontend and the Go frontend imports the golang standard library
>      > (but the frontend itself is written in C++ and doesn't use any
>     part of it).
>      >
>      > The C++ frontend uses part of the C++ standard library (but it can of
>      > course build that itself - but it requires a host C++ compiler
>     with library).
> 
>     Thanks for the pointers. I was wondering if this is something that the
>     Ada frontend had faced at the beginning, but I've been told it does not
>     have a lot of dependencies anyway so this might not be helpful.
> 
>      >
>      > Richard.
>      >
>      >> Thanks,
>      >>
>      >> Arthur
>      >>
>      >> [1]: https://doc.rust-lang.org/std/fmt/
>     <https://doc.rust-lang.org/std/fmt/>
> 
>     Thanks a lot for taking the time, I really appreciate it.
> 
>     Best,
> 
>     Arthur
>