RFC: Creating a live, all-encompassing architectural document for GCC

RFC: Creating a live, all-encompassing architectural document for GCC

[Diego Novillo]

This idea is still very raw in my mind, so apologies in advance for
being deliberately vague.  For the last few weeks I have been thinking
on ways to address the sorry state of our internal documentation.

We all agree that none of us has global knowledge of all aspects of
the compiler.  It's just impossible.  So, we have the collective
knowledge distributed all over the place but it is pretty hard for
someone to navigate the compiler without talking to N different
people.

So, I think the problem goes a bit beyond mere documentation of how a
module works at a high level.  I would like to have a navigable
document that also describes the flow of things, interfaces and
helpers.  Starting at main.c:main() and ending at toplev.c:finalize().

The key properties that I'm after are:

- Navigable.

From high-level details to low-level implementation of a single file.
This would be a set of browseable hierarchies that offer different
views of the documentation.  For instance, one view would be a
decomposition of modules, each of which would offer a view of
submodules and such.
Another view could focus on the compilation flow.

- Live and easy to modify.

It should be easy for an individual maintainer (or even user) to go in
and modify parts of the document that are incomplete/missing/wrong.
This and navigability suggest a wikipedia-like approach.  We even have
the beginnings of some of this in the wiki, so I would like to build
on that.


- Close correspondence to mainline.

This is where it gets hard.  We need to have a way of enforcing code
updates that change internal or external API properties to be
reflected in the document.  With this I don't mean that every single
patch should be accompanied with a documentation change.  However, if
a patch refactors a module and its internal interfaces are changed,
then the patch should be accompanied with a change to the
documentation.


- Transfer from document to code

The documentation for individual modules and files should be linked to
the actual source code.  Perhaps this could be automatically generated
with tools like javadoc or doxygen.

The high-level description of algorithms, strategies, heuristics,
interfaces, usage, etc live on the document and we provide links to
the automatically generated documentation or SVN live sources.  This
would provide more low-level documentation in the form of individual
function documents and the usual things we request in each file.


So, I think my inclination is to provide this document as a wiki.  The
current wiki should provide all the support we need.  We can create
different views and organization incrementally, it's easy for people
to edit it, and we could put guards around if it started being abused.

My ultimate goal is to facilitate the transfer of knowledge.  When
someone new wants to start working on GCC, we should be able to point
them to this document and say "here, drink from this hose".


Thanks.

Re: RFC: Creating a live, all-encompassing architectural document for GCC

[Basile STARYNKEVITCH]

Diego Novillo wrote:
> This idea is still very raw in my mind, so apologies in advance for
> being deliberately vague.  For the last few weeks I have been thinking
> on ways to address the sorry state of our internal documentation.
> 
> We all agree that none of us has global knowledge of all aspects of
> the compiler.  It's just impossible.  So, we have the collective
> knowledge distributed all over the place but it is pretty hard for
> someone to navigate the compiler without talking to N different
> people.
> 
> So, I think the problem goes a bit beyond mere documentation of how a
> module works at a high level.  I would like to have a navigable
> document that also describes the flow of things, interfaces and
> helpers.  Starting at main.c:main() and ending at toplev.c:finalize().


Maybe a possible approach would be to use literate programming 
techniques; By previous experience (still limited), I would believe that 
it would be more worthwhile on the interface files (ie the *.h files, 
some *.opt, etc...) than on the implementation files (*.c)

Still, it is a lot of work, and it would mean to change the coding 
guidelines & coding rules expected by GCC contributors.

I am more thinking loud than actually believing that it would be a good 
idea to switch to literate programming; I have mixed feelings towards 
this approach, which has been extensively used in the C-- compiler 
http://cminusminus.org/ . My personal (biased) view is that 
sophisticated compiler technology should be coded in higher level 
languages than C, C++ or Java

Maybe a better discipline would be to expect every coder to document his 
stuff on the GCC wiki.

All this is more a social issue than anything else. We developers don't 
like documenting our work (and this is sadly true for me too!)


(Diego, I am busy preparing the basilys branch, see my GCC summit paper).


-- 
Basile STARYNKEVITCH         http://starynkevitch.net/Basile/
email: basile<at>starynkevitch<dot>net mobile: +33 6 8501 2359
8, rue de la Faiencerie, 92340 Bourg La Reine, France
*** opinions {are only mines, sont seulement les miennes} ***

Re: RFC: Creating a live, all-encompassing architectural document for GCC

[skaller]

On Fri, 2007-10-26 at 19:34 +0200, Basile STARYNKEVITCH wrote:
> Diego Novillo wrote:

> I am more thinking loud than actually believing that it would be a good 
> idea to switch to literate programming; I have mixed feelings towards 
> this approach, which has been extensively used in the C-- compiler 
> http://cminusminus.org/ . My personal (biased) view is that 
> sophisticated compiler technology should be coded in higher level 
> languages than C, C++ or Java

Felix uses literate programming and a high level language (Ocaml).
Switching gcc to high level language will have more benefits and
be easier than using LP.

I like LP, but it is monolithic and invasive, and needs its
own tools (syntax colouring .. all gone :)

-- 
John Skaller <skaller at users dot sf dot net>
Felix, successor to C++: http://felix.sf.net

Re: RFC: Creating a live, all-encompassing architectural document for GCC

[Diego Novillo]

On 10/26/07, Basile STARYNKEVITCH <basile@starynkevitch.net> wrote:

> Maybe a possible approach would be to use literate programming
> techniques; By previous experience (still limited), I would believe that
> it would be more worthwhile on the interface files (ie the *.h files,
> some *.opt, etc...) than on the implementation files (*.c)

Pragmatically, we need to do this without forcing such structural
changes to the implementation.  Moving GCC to another language is
another problem that I'd like to keep separate from this.


> All this is more a social issue than anything else. We developers don't
> like documenting our work (and this is sadly true for me too!)

Agreed.  The documentation mechanism ultimately needs to be easy to
improve.  It will always be incomplete and imperfect.  I simply want
to make it less so.

> (Diego, I am busy preparing the basilys branch, see my GCC summit paper).

Good to hear.  Thanks.

RE: Creating a live, all-encompassing architectural document for GCC

[Eric Weddington]

 

> -----Original Message-----
> From: Diego Novillo [mailto:dnovillo@google.com] 
> Sent: Friday, October 26, 2007 11:10 AM
> To: gcc@gcc.gnu.org
> Subject: RFC: Creating a live, all-encompassing architectural 
> document for GCC
> 
> 
> It should be easy for an individual maintainer (or even user) to go in
> and modify parts of the document that are incomplete/missing/wrong.
> This and navigability suggest a wikipedia-like approach.  We even have
> the beginnings of some of this in the wiki, so I would like to build
> on that.
...
> However, if
> a patch refactors a module and its internal interfaces are changed,
> then the patch should be accompanied with a change to the
> documentation.
... 
> The documentation for individual modules and files should be linked to
> the actual source code.  Perhaps this could be automatically generated
> with tools like javadoc or doxygen.
...
> So, I think my inclination is to provide this document as a wiki.

I like the goals. But what I see above seems mutually exclusive.

It's reasonable to include doxygen, and change the code with the
documentation simultaneously. Here's the avr-libc user manual online as an
example of the output:
<http://www.nongnu.org/avr-libc/user-manual/>
It certainly meets the navigable requirement. I'm sure all of you have seen
other examples as well.

But you also want the user to be able to change an internals document via a
wiki? How does this work with a patch system? How do you propose to resolve
conflicts between a user edit, and maintainer's patch? Maybe I'm ignorant to
the capabilities of a wiki, but this is where it sounds like incompatible
systems.

Eric Weddington

Re: Creating a live, all-encompassing architectural document for GCC

[Diego Novillo]

On 10/26/07, Eric Weddington <eweddington@cso.atmel.com> wrote:


> <http://www.nongnu.org/avr-libc/user-manual/>
> It certainly meets the navigable requirement. I'm sure all of you have seen
> other examples as well.

Thanks.  I'll take a look.

>
> But you also want the user to be able to change an internals document via a
> wiki? How does this work with a patch system? How do you propose to resolve
> conflicts between a user edit, and maintainer's patch? Maybe I'm ignorant to
> the capabilities of a wiki, but this is where it sounds like incompatible
> systems.

That's just one option.  The impedance between wiki and the source
tree is that the wiki is not patch-based, which makes it more
accessible but it also increases chances that the code and the
documentation will diverge.

I maintain that divergence is inevitable, regardless of whether the
code and the documentation live together or not (without getting into
Literate Programming arguments).  So, we need to make it easy to keep
them both in sync.

Moving to a pure document extraction system like doxygen or javadoc
may be useful, but the problem there is that we miss all the
whole-system documentation, API interfaces, behaviours, etc.

Re: RFC: Creating a live, all-encompassing architectural document for GCC

[Gerald Pfeifer]

On Fri, 26 Oct 2007, Diego Novillo wrote:
> So, I think the problem goes a bit beyond mere documentation of how a
> module works at a high level.  I would like to have a navigable
> document that also describes the flow of things, interfaces and
> helpers.  Starting at main.c:main() and ending at toplev.c:finalize().

Something like this is a key element for documentation, but it's hard
to do the way we have been documenting things indeed.  That sounds like
a very good idea.

> - Navigable.

That seems to ask for, at least on the module level and below, for
something similar to literary programming or we'll run out of sync
quickly.

> - Close correspondence to mainline.
> 
> This is where it gets hard.  We need to have a way of enforcing code
> updates that change internal or external API properties to be
> reflected in the document.  With this I don't mean that every single
> patch should be accompanied with a documentation change.  However, if
> a patch refactors a module and its internal interfaces are changed,
> then the patch should be accompanied with a change to the
> documentation.

I guess that's my main concern as well: how can we keep the various
bits of documentation -- comments in the code, texinfo, and your
proposed one -- in sync and do so without adding much effort for
individual contributors?

Another concern is the question of copyright assignments.  We are
requiring these for texinfo changes, but do not have anything in
place for the Wiki.  If we now get significantly improved documentation
on the latter, we may not be able to move that into the regular manuals.
Is this an issue?

Gerald