[Ada] convert GNAT doc to sphinx

[Ada] convert GNAT doc to sphinx

[Arnaud Charlet]

As discussed last year, we've converted the GNAT main documentation
(gnat_rm.texi and gnat_ugn.texi) to reST/sphinx, so the master doc
can now be found under gcc/ada/doc.

We're keeping automatically generated .texi files for now under gcc/ada
so that people only having texinfo tools can still generate the documentation
in various formats.

Once GCC has more generally switched to reST/sphinx, we can finalize the
transition and make the .rst files first class citizens/built by default.

Currently you need to manually call the Makefile in the doc subdirectory
to generate documentation using sphinx, e.g:

make -C doc
will yield a help on the various available build targets:
<<
Please use `make <target>' where <target> is one of
  DOC_NAME.html       to make standalone HTML files
  DOC_NAME.pdf        to make LaTeX files and run them through pdflatex
  DOC_NAME.txt        to make text files
  DOC_NAME.texinfo    to make Texinfo files
  DOC_NAME.info       to make info files
  DOC_NAME.all        to build DOC_NAME for all previous formats
  all                 to build all documentations in all formats
  html-all            same as previous rule but only for HTML format
  pdf-all             same as previous rule but only for PDF format
  txt-all             same as previous rule but only for text format
  texinfo-all         same as previous rule but only for texinfo format
  info-all            same as previous rule but only for info format

DOC_NAME should be a documentation name in the following list:
  gnat_rm gnat_ugn

source and location can be overriden using SOURCEDIR and BUILDDIR variables
>>

For example:

$ make -C doc all
$ make -C doc gnat_ugn.pdf

The SOURCEDIR and BUILDDIR variables have been introduced in
doc/Makefile in order to ease further integration with the GCC Makefiles.

Not including the diff which is very large and not very interesting.

Tested on x86_64-pc-linux-gnu, committed on trunk.

2015-02-20  Arnaud Charlet  <charlet@adacore.com>                      

        * gnat_rm.texi, gnat_ugn.texi: Now automatically generated from
        sphinx in the doc directory.
        * doc: New directory containing sphinx versions of gnat_rm and gnat_ugn

Re: [Ada] convert GNAT doc to sphinx

[Tom de Vries]

On 20-02-15 10:17, Arnaud Charlet wrote:
>
> Tested on x86_64-pc-linux-gnu, committed on trunk.
>
> 2015-02-20  Arnaud Charlet  <charlet@adacore.com>
>
>          * gnat_rm.texi, gnat_ugn.texi: Now automatically generated from
>          sphinx in the doc directory.
>          * doc: New directory containing sphinx versions of gnat_rm and gnat_ugn
>

Hi,

I'm observing:
...
src/gcc/ada/gnat_rm.texi:13: warning: @dircategory missing argument
src/gcc/ada/gnat_ugn.texi:13: warning: @dircategory missing argument
src/gcc/ada/gnat_ugn.texi:19556: warning: could not find @image file 
`project-manager-figure.txt' nor alternate text
src/gcc/ada/gnat_ugn.texi:29593: warning: could not find @image file 
`rtlibrary-structure.txt' nor alternate text
...

Thanks,
- Tom

Re: [Ada] convert GNAT doc to sphinx

[Arnaud Charlet]

> >2015-02-20  Arnaud Charlet  <charlet@adacore.com>
> >
> >         * gnat_rm.texi, gnat_ugn.texi: Now automatically generated
> >         from
> >         sphinx in the doc directory.
> >         * doc: New directory containing sphinx versions of gnat_rm and
> >         gnat_ugn
> 
> Hi,
> 
> I'm observing:
> ...
> src/gcc/ada/gnat_rm.texi:13: warning: @dircategory missing argument
> src/gcc/ada/gnat_ugn.texi:13: warning: @dircategory missing argument
> src/gcc/ada/gnat_ugn.texi:19556: warning: could not find @image file
> `project-manager-figure.txt' nor alternate text
> src/gcc/ada/gnat_ugn.texi:29593: warning: could not find @image file
> `rtlibrary-structure.txt' nor alternate text

Two .png files were missing, now added:

2015-02-22  Arnaud Charlet  <charlet@adacore.com>
                      
	* doc/gnat_ugn/project-manager-figure.png,
	doc/gnat_ugn/rtlibrary-structure.png: New.     

As for the @dircategory I do not know, I couldn't find a proper documentation
for this node other than:
http://www.gnu.org/software/texinfo/manual/texinfo/html_node/Installing-Dir-Entries.html#index-dircategory
which is incomplete.

Arno

Re: [Ada] convert GNAT doc to sphinx

[Tom de Vries]

On 22-02-15 19:04, Arnaud Charlet wrote:
>>> 2015-02-20  Arnaud Charlet  <charlet@adacore.com>
>>>
>>>          * gnat_rm.texi, gnat_ugn.texi: Now automatically generated
>>>          from
>>>          sphinx in the doc directory.
>>>          * doc: New directory containing sphinx versions of gnat_rm and
>>>          gnat_ugn
>>
>> Hi,
>>
>> I'm observing:
>> ...
>> src/gcc/ada/gnat_rm.texi:13: warning: @dircategory missing argument
>> src/gcc/ada/gnat_ugn.texi:13: warning: @dircategory missing argument
>> src/gcc/ada/gnat_ugn.texi:19556: warning: could not find @image file
>> `project-manager-figure.txt' nor alternate text
>> src/gcc/ada/gnat_ugn.texi:29593: warning: could not find @image file
>> `rtlibrary-structure.txt' nor alternate text
>
> Two .png files were missing, now added:
>
> 2015-02-22  Arnaud Charlet  <charlet@adacore.com>
>
> 	* doc/gnat_ugn/project-manager-figure.png,
> 	doc/gnat_ugn/rtlibrary-structure.png: New.
>
> As for the @dircategory I do not know, I couldn't find a proper documentation
> for this node other than:
> http://www.gnu.org/software/texinfo/manual/texinfo/html_node/Installing-Dir-Entries.html#index-dircategory
> which is incomplete.
>

Your patch removes these arguments to dircategory:
...
$ git show bf5dffd3a47fe12ace71fe48e87cfb1b9ada1344 | grep dircategory
+@dircategory
-@dircategory GNU Ada tools
-@dircategory GNU Ada tools
+@dircategory
...

Thanks,
- Tom

Re: [Ada] convert GNAT doc to sphinx

[Arnaud Charlet]

> Your patch removes these arguments to dircategory:
> ...
> $ git show bf5dffd3a47fe12ace71fe48e87cfb1b9ada1344 | grep dircategory
> +@dircategory
> -@dircategory GNU Ada tools
> -@dircategory GNU Ada tools
> +@dircategory
> ...

Well OK but these are automatically generated now, and this doesn't really
answer my question about the documentation of @dircategory.

I'll put a kludge for now to work around this. In the long term, if we could
transition all docs to sphinx and get rid of texinfo that would be great.

Arno

Re: [Ada] convert GNAT doc to sphinx

[Tom de Vries]

On 22-02-15 20:15, Arnaud Charlet wrote:
> Well OK but these are automatically generated now, and this doesn't really
> answer my question about the documentation of @dircategory.

I didn't see a question here:
...
As for the @dircategory I do not know, I couldn't find a proper documentation
for this node other than: 
http://www.gnu.org/software/texinfo/manual/texinfo/html_node/Installing-Dir-Entries.html#index-dircategory 
which is incomplete.
...

Could you indicate what you found to be missing?

Thanks,
- Tom

Re: [Ada] convert GNAT doc to sphinx

[Arnaud Charlet]

> I didn't see a question here:
> ...
> As for the @dircategory I do not know, I couldn't find a proper documentation
> for this node other than:
> http://www.gnu.org/software/texinfo/manual/texinfo/html_node/Installing-Dir-Entries.html#index-dircategory
> which is incomplete.
> ...
> 
> Could you indicate what you found to be missing?

Well, what's the syntax for this command to start with, what are its
parameters, and are the parameter(s) mandatory? Is there e.g. a list of
all commands with their syntax and semantic documented?

Arno

Re: [Ada] convert GNAT doc to sphinx

[Tom de Vries]

On 22-02-15 21:48, Arnaud Charlet wrote:
>> I didn't see a question here:
>> ...
>> As for the @dircategory I do not know, I couldn't find a proper documentation
>> for this node other than:
>> http://www.gnu.org/software/texinfo/manual/texinfo/html_node/Installing-Dir-Entries.html#index-dircategory
>> which is incomplete.
>> ...
>>
>> Could you indicate what you found to be missing?
>
> Well, what's the syntax for this command to start with, what are its
> parameters, and are the parameter(s) mandatory?

I see what you mean.

 From reading the doc, I'd say the syntax is
   @dircategory <category name>
where <category name> specifies the category under which subsequent @direntry-s 
will be listed in 'dir' ( 
http://www.gnu.org/software/texinfo/manual/texinfo/html_node/Directory-File.html#Directory-File 
).

> Is there e.g. a list of
> all commands with their syntax and semantic documented?
>

I only found the index ( 
http://www.gnu.org/software/texinfo/manual/texinfo/html_node/Command-and-Variable-Index.html#Command-and-Variable-Index 
), which lists all commands, and gives links to where they are explained.

Thanks,
- Tom

Re: [Ada] convert GNAT doc to sphinx

[Joseph Myers]

On Sun, 22 Feb 2015, Arnaud Charlet wrote:

> Two .png files were missing, now added:
> 
> 2015-02-22  Arnaud Charlet  <charlet@adacore.com>
>                       
> 	* doc/gnat_ugn/project-manager-figure.png,
> 	doc/gnat_ugn/rtlibrary-structure.png: New.     

The maintainer-scripts/update_web_docs_svn script is still producing 
errors relating to those files: 
https://gcc.gnu.org/ml/gccadmin/2015-q1/msg00129.html

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: [Ada] convert GNAT doc to sphinx

[Arnaud Charlet]

> > Two .png files were missing, now added:
> > 
> > 2015-02-22  Arnaud Charlet  <charlet@adacore.com>
> >                       
> > 	* doc/gnat_ugn/project-manager-figure.png,
> > 	doc/gnat_ugn/rtlibrary-structure.png: New.     
> 
> The maintainer-scripts/update_web_docs_svn script is still producing
> errors relating to those files: 
> https://gcc.gnu.org/ml/gccadmin/2015-q1/msg00129.html

Indeed. I'll need some help here since I do not know where these .png files
are looked for or how to change that (I tried copying the .png file under
gcc/ada but this didn't help. I also tried adding -I$(srcdir)/ada/doc/gnat_ugn
to the doc/gnat_ugn.dvi rule in gcc/ada/gcc-interface/Make-lang.in rule, with
no success).

Basically the images can be found under gcc/ada/doc/gnat_ugn so for someone
familiar with texinfo this should be an easy change.

Arno

Re: [Ada] convert GNAT doc to sphinx

[Joseph Myers]

On Wed, 25 Feb 2015, Arnaud Charlet wrote:

> > > Two .png files were missing, now added:
> > > 
> > > 2015-02-22  Arnaud Charlet  <charlet@adacore.com>
> > >                       
> > > 	* doc/gnat_ugn/project-manager-figure.png,
> > > 	doc/gnat_ugn/rtlibrary-structure.png: New.     
> > 
> > The maintainer-scripts/update_web_docs_svn script is still producing
> > errors relating to those files: 
> > https://gcc.gnu.org/ml/gccadmin/2015-q1/msg00129.html
> 
> Indeed. I'll need some help here since I do not know where these .png files
> are looked for or how to change that (I tried copying the .png file under
> gcc/ada but this didn't help. I also tried adding -I$(srcdir)/ada/doc/gnat_ugn
> to the doc/gnat_ugn.dvi rule in gcc/ada/gcc-interface/Make-lang.in rule, with
> no success).
> 
> Basically the images can be found under gcc/ada/doc/gnat_ugn so for someone
> familiar with texinfo this should be an easy change.

update_web_docs_svn does not use the makefiles.  So you need to update the 
find command therein not to remove anything that's part of the sources for 
this documentation, and possibly update -I options for building manuals as 
well.

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: [Ada] convert GNAT doc to sphinx

[Arnaud Charlet]

> So you need to update the
> find command therein not to remove anything that's part of the sources for
> this documentation, and possibly update -I options for building manuals as
> well.

I've added a -I gcc/gcc/ada/doc/gnat_ugn there, that's as far as my
knowledge goes for this script so I hope this is enough.

I can help in transitioning the script to sphinx though, that would seem
more interesting/productive at this stage.

Arno

Re: [Ada] convert GNAT doc to sphinx

[Richard Biener]

On Sun, Feb 22, 2015 at 8:15 PM, Arnaud Charlet <charlet@adacore.com> wrote:
>> Your patch removes these arguments to dircategory:
>> ...
>> $ git show bf5dffd3a47fe12ace71fe48e87cfb1b9ada1344 | grep dircategory
>> +@dircategory
>> -@dircategory GNU Ada tools
>> -@dircategory GNU Ada tools
>> +@dircategory
>> ...
>
> Well OK but these are automatically generated now, and this doesn't really
> answer my question about the documentation of @dircategory.
>
> I'll put a kludge for now to work around this. In the long term, if we could
> transition all docs to sphinx and get rid of texinfo that would be great.

I also see nonsensical @direntry - just compare to what is there on the
4.9 branch.

Richard.

> Arno

Re: [Ada] convert GNAT doc to sphinx

[Joseph Myers]

On Wed, 25 Feb 2015, Arnaud Charlet wrote:

> > So you need to update the
> > find command therein not to remove anything that's part of the sources for
> > this documentation, and possibly update -I options for building manuals as
> > well.
> 
> I've added a -I gcc/gcc/ada/doc/gnat_ugn there, that's as far as my
> knowledge goes for this script so I hope this is enough.

Well, since by default the find command deletes all files except those 
known to be documentation sources, you need at least to change it not to 
delete those particular files (and the actual RST sources of these 
manuals).

> I can help in transitioning the script to sphinx though, that would seem
> more interesting/productive at this stage.

See the existing code to handle Sphinx documentation for the JIT.

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: [Ada] convert GNAT doc to sphinx

[Arnaud Charlet]

> > I've added a -I gcc/gcc/ada/doc/gnat_ugn there, that's as far as my
> > knowledge goes for this script so I hope this is enough.
> 
> Well, since by default the find command deletes all files except those
> known to be documentation sources, you need at least to change it not to
> delete those particular files (and the actual RST sources of these
> manuals).

OK, I've applied the patch below, hopefully it should do the job.

> > I can help in transitioning the script to sphinx though, that would seem
> > more interesting/productive at this stage.
> 
> See the existing code to handle Sphinx documentation for the JIT.

That's a good reference. We'll need a more recent version of sphinx than
1.0 though (at least 1.2.2, or even better, 1.3b2 which is the version we use
at AdaCore).

Arno
--
--- update_web_docs_svn (revision 220961)
+++ update_web_docs_svn (working copy)
@@ -107,10 +107,8 @@
   svn -q export $SVNROOT/tags/$RELEASE gcc
 fi
 
-# Remove all unwanted files.  This is needed (a) to build the Ada
-# generator programs with the installed library, not the new one and
-# (b) to avoid packaging all the sources instead of only documentation
-# sources.
+# Remove all unwanted files.  This is needed to avoid packaging all the
+# sources instead of only documentation sources.
 # Note that we have to preserve gcc/jit/docs since the jit docs are
 # not .texi files (Makefile, .rst and .png), and the jit docs use
 # include directives to pull in content from jit/jit-common.h and
@@ -120,6 +118,7 @@
   -o -path gcc/gcc/doc/include/texinfo.tex \
   -o -path gcc/gcc/BASE-VER \
   -o -path gcc/gcc/DEV-PHASE \
+  -o -path "gcc/gcc/ada/doc/gnat_ugn/*.png" \
   -o -path "gcc/gcc/jit/docs/*" \
   -o -path "gcc/gcc/jit/jit-common.h" \
   -o -path "gcc/gcc/jit/notes.txt" \

Re: [Ada] convert GNAT doc to sphinx

[Joseph Myers]

On Wed, 25 Feb 2015, Arnaud Charlet wrote:

> > See the existing code to handle Sphinx documentation for the JIT.
> 
> That's a good reference. We'll need a more recent version of sphinx than
> 1.0 though (at least 1.2.2, or even better, 1.3b2 which is the version we use
> at AdaCore).

1.0 is apparently the most recent version readily available for RHEL 6 
which gcc.gnu.org runs.

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: [Ada] convert GNAT doc to sphinx

[David Malcolm]

On Wed, 2015-02-25 at 16:47 +0100, Arnaud Charlet wrote:
> > > I've added a -I gcc/gcc/ada/doc/gnat_ugn there, that's as far as my
> > > knowledge goes for this script so I hope this is enough.
> > 
> > Well, since by default the find command deletes all files except those
> > known to be documentation sources, you need at least to change it not to
> > delete those particular files (and the actual RST sources of these
> > manuals).
> 
> OK, I've applied the patch below, hopefully it should do the job.

Thanks for working on this.  I'm a big fan of sphinx (both for its ease
of use, and the quality of the generated HTML), and I hope that more GNU
documentation transitions to it.  (I actually *enjoy* writing docs using
sphinx, which I can't say for other toolchains).  As noted below, I'm
using it for libgccjit.

> > > I can help in transitioning the script to sphinx though, that would seem
> > > more interesting/productive at this stage.
> > 
> > See the existing code to handle Sphinx documentation for the JIT.
> 
> That's a good reference. 

You may also want to look at the bug I used to track the jit work:
  https://gcc.gnu.org/bugzilla/show_bug.cgi?id=64257
(aka "JIT documentation is not yet on the GCC website"), though
hopefully the pertinent information is now all in comments in the
update_web_docs_svn script.

> We'll need a more recent version of sphinx than
> 1.0 though (at least 1.2.2, or even better, 1.3b2 which is the version we use
> at AdaCore).

The specific version of sphinx in use for gcc.gnu.org is 1.0.8, from
EPEL6, which installs sphinx's executable to
  /usr/bin/sphinx-1.0-build
(the default RHEL6 /usr/bin/sphinx-build is for sphinx 0.6.6).

FWIW, I've filed:
  https://bugzilla.redhat.com/show_bug.cgi?id=1189218
to request a more recent version of sphinx in EPEL6 (i.e. beyond 1.0.8).
The jit just needs 1.0, but I'd prefer the "pyramid" theme (added in
1.1) to the one currently in use ("sphinxdoc").  We could even have a
gcc or gnu-specific theme (llvm has one, fwiw), but let's walk before we
run :)

gcc/doc/install.texi currently has:

  @item Sphinx version 1.0 (or later)

  Necessary to regenerate @file{jit/docs/_build/texinfo} from the @file{.rst}
  files in the directories below @file{jit/docs}.

so that too would need updating if you want to bump the minimum
requirement.

> --- update_web_docs_svn (revision 220961)
> +++ update_web_docs_svn (working copy)
> @@ -107,10 +107,8 @@
>    svn -q export $SVNROOT/tags/$RELEASE gcc
>  fi
>  
> -# Remove all unwanted files.  This is needed (a) to build the Ada
> -# generator programs with the installed library, not the new one and
> -# (b) to avoid packaging all the sources instead of only documentation
> -# sources.
> +# Remove all unwanted files.  This is needed to avoid packaging all the
> +# sources instead of only documentation sources.
>  # Note that we have to preserve gcc/jit/docs since the jit docs are
>  # not .texi files (Makefile, .rst and .png), and the jit docs use
>  # include directives to pull in content from jit/jit-common.h and
> @@ -120,6 +118,7 @@
>    -o -path gcc/gcc/doc/include/texinfo.tex \
>    -o -path gcc/gcc/BASE-VER \
>    -o -path gcc/gcc/DEV-PHASE \
> +  -o -path "gcc/gcc/ada/doc/gnat_ugn/*.png" \
>    -o -path "gcc/gcc/jit/docs/*" \
>    -o -path "gcc/gcc/jit/jit-common.h" \
>    -o -path "gcc/gcc/jit/notes.txt" \

How are you testing this?  When getting the jit docs to work I had to
hack up the script somewhat to be able to test it on a development box
(e.g. to cut back "MANUALS" to just libcpp to speed up testing).

Don't you also need to preserve the Makefile at this stage, and indeed
the .rst files?

To get sphinx-generated HTML for the jit onto the gcc website, I needed
to patch the script in three places:
  * the part above,
  * to add an invocation of the Makefile, and
  * to implement the copying up from the build location to the install
location (since the jit docs have nested subdirectories, plus .js
and .css files provided by sphinx).

Hope this is helpful
Dave

Re: [Ada] convert GNAT doc to sphinx

[Arnaud Charlet]

Thanks for your feedback, very useful!

> How are you testing this?  When getting the jit docs to work I had to
> hack up the script somewhat to be able to test it on a development box
> (e.g. to cut back "MANUALS" to just libcpp to speed up testing).

That's a good question. Truth is that so far I'm not since I do not know
how to test it :-) That's why I wanted to minimize my changes and was asking
for help, since testing these changes is not very practical/easy.

> Don't you also need to preserve the Makefile at this stage, and indeed
> the .rst files?

When we'll switch the script to using sphinx for GNAT docs, yes.

For now the temporary state is that we're still building docs from the
.texi files (now generated by sphinx). Certainly not ideal but it more or
less does the job.

> To get sphinx-generated HTML for the jit onto the gcc website, I needed
> to patch the script in three places:
>   * the part above,
>   * to add an invocation of the Makefile, and
>   * to implement the copying up from the build location to the install
> location (since the jit docs have nested subdirectories, plus .js
> and .css files provided by sphinx).

Makes sense.

> Hope this is helpful

It definitely is!

Arno

Re: [Ada] convert GNAT doc to sphinx

[Mike Stump]

On Feb 26, 2015, at 9:07 AM, David Malcolm <dmalcolm@redhat.com> wrote:
>> We'll need a more recent version of sphinx than
>> 1.0 though (at least 1.2.2, or even better, 1.3b2 which is the version we use
>> at AdaCore).
> 
> The specific version of sphinx in use for gcc.gnu.org is 1.0.8

So, if people check in the generated files, then I think we should just bump the version required up to one that works well.  The people that want to regen will need to ensure they update, but that seems reasonable.

> gcc/doc/install.texi currently has:
> 
>  @item Sphinx version 1.0 (or later)
> 
>  Necessary to regenerate @file{jit/docs/_build/texinfo} from the @file{.rst}
>  files in the directories below @file{jit/docs}.
> 
> so that too would need updating if you want to bump the minimum
> requirement.

Re: [Ada] convert GNAT doc to sphinx

[Matthias Klose]

On 02/20/2015 10:17 AM, Arnaud Charlet wrote:
> As discussed last year, we've converted the GNAT main documentation
> (gnat_rm.texi and gnat_ugn.texi) to reST/sphinx, so the master doc
> can now be found under gcc/ada/doc.

there is an empty directory left in SVN:

  gcc/ada/doc/share/_static

ok to remove?

  Matthias

Re: [Ada] convert GNAT doc to sphinx

[Arnaud Charlet]

>> As discussed last year, we've converted the GNAT main documentation
>> (gnat_rm.texi and gnat_ugn.texi) to reST/sphinx, so the master doc
>> can now be found under gcc/ada/doc.
> there is an empty directory left in SVN:
>
>    gcc/ada/doc/share/_static
>
> ok to remove?

No, this (empty currently) directory is needed to run sphinx.
Feel free to e.g. add a .cvsignore or .gitignore file if it can help.

Arno