hacking the Tcl parser

hacking the Tcl parser

[Lori Flynn]

Hi everyone,

I also (like Emmanuel in her Aug.8th 2001 post) am interested in hacking
the Tcl parser so it can parse procedures declared slightly differently.
A colleague has done a lot of work towards this, the details are described
below.  (I'm wanting 'instproc' to be recognized just as Tcl's 'proc' as a
keyword which is followed by a function name... OTcl declares methods this
way. )  Right now we can compile snavigator and it will now recognize
variables declared within an instproc as variables (so they turn up when I
use snavigator's filter and ask it to show me all variables). However, the
instproc function/method doesn't show up in the filter.  

I imagine the fact that lines with instproc have a word (the class) before
the keyword 'instproc' is part of the problem, with the parsing.  

Any ideas, I don't have time to create a whole new parser for OTcl and am
hoping for some clue about how to go about completing this hack...

Thanks in advance for clues!!
Lori

*************************************************

These are the changes so far:
/projects/ccrg/mmosko/code/SN/source/Makefile 
	Changed top line to lower one:
	#all-sid: all-tcl all-tk all-libremote
	all-sid: all-tcl all-tk
Makefile.ct     
Makefile.ct: (reverse lines 25 & 26)
     24 Tree.h Tree.c:          tcl.ast
     25     cg -cdimRDIO    tcl.ast;
     26 #   ast -cd    tcl.ast;
Trafo.c    change variable 'To_names' to  'to_names'
tcl.puma   change variable 'To_names' to  'to_names'
itcl.puma Added {} around all c code, plus line 12 included sn.h (not
	the file paf.h)
tcl.ast   Changed includes for path (no 'include' directory and 
		also name of files is different: Idents.h,StringM.h,
		and Position.h),line 45 ' instproc       = .', and
		iinstproc added line 14 and 25 to list of keywords.
Scanner.c: Add the typecasts to (int)
   1051 yyEol ((int)TokenPtr + (int)TokenLength - (int)p - 1);
   Lines 16 and 23 include "Position.h" (not Positions.h)
   Lines 
Parser.c:  Where have "yySnyAttribute.fragment.'*'", changed 
	it to "'*'"
	Commented out function  "\\ReleaseTreeModule	();"
	Wherever had phrase like yySynAttribute.fragment.'$',
		changed to just '$'  (same with $, }, { , ] )
	Line 40  #	include "Position.h" (not "Positions.h")
	
Parser.h:  
	Added '#include <string.h>'
	Added:
	# if defined __STDC__ | defined __cplusplus
	# define ARGS(parameters)	parameters
	# else
	# define ARGS(parameters)	()
	# endif
Parser.lalr:
tcl.pars:  Comment out ReleaseTreeModule if the function is still there.
Tree.h: Line 85 add iinstproc to list of keywords
Tree.c: 
Lines 12 and 22, change StringMem.h to StringM.h
Lines 15 and 25, change Positions.h to Position.h
Comment out lines 231 - 235
    226 tTree Tree_Alloc ()
    227 {
    228  register yytBlockPtr yyBlockPtr = yyBlockList;
    229  register int i;
    230 
    231  //if (yyMaxSize == 0)
    232   //for (i = 1; i <= 50; i ++) {
    233    //Tree_NodeSize [i] = (Tree_NodeSize [i] + yyMaxAlign - 1) &
yyAlignMasks [yyMaxAlign];
    234    //yyMaxSize = Max (Tree_NodeSize [i], yyMaxSize);
    235   //}
    236 
    237  yyBlockList = (yytBlockPtr) Alloc (sizeof (yytBlock));









From Emmanuelle's mail August 9 2001:

> I want to modify the Tcl parser
> that it can parse procedures declared that way but I need a 
> little help
> to do that; the source code of SN is very large and I don't know from
> where to start.

The TCL parser can be found in:

/source/snavigator/parser/tcl

There is documentation on how to add parsers:

/source/snvavigator/doc/html/enterprise/progref

But the TCL parser does not follow use this API. Do as we say, not as we
do.
It inserts symbols with put_symbol. The TCL parsers uses a
parser-generator
/ compiler-compiler, but it doesn't use yacc/bison. Puma or something. If
you wish to modify the code, you will need this tool.

Happy hacking,
Craig Keogh

Re: hacking the Tcl parser

[Ian Roxborough]

Hi Lori,

\rI was hoping that the public domain version of cocktail would
be a big step forward, we've never had access to a version before
so all our fixes had to be done to the generated C code.

Anyway, I've had no luck with cocktail, I can get it to build,
but it can't even deal with the example files provided with
the sources. (This is under RH6.2.)

I may try again and build cocktail under solaris or maybe a 
more up-to-date version of linux.  BTW, if you try to build
it a run into "ar" error messages you need to edit the Makefile
and add an "r" flag.  e.g. "ar cu ..." -> "ar cur ...".
(Atleast that worked for me.)

If anybody has gotten cocktail to work properly (i.e. atleast
run the tests that come with it) then please tell what platform
and how. Thanks.

It's pretty cool that you've gotten this far already, editting
generated files is normally a nightmare.  If it's possible to
revive the correct generation of the Tcl parser then this would
be a big boost for the project and we should be able to fix
parser problems without soooo much pain.

Ian.

On Fri, 11 Jan 2002 18:12:39 -0800 (PST) Lori Flynn  <lori@cse.ucsc.edu> wrote:
>
> Hi everyone,
> 
> I also (like Emmanuel in her Aug.8th 2001 post) am interested in hacking
> the Tcl parser so it can parse procedures declared slightly differently.
> A colleague has done a lot of work towards this, the details are described
> below.  (I'm wanting 'instproc' to be recognized just as Tcl's 'proc' as a
> keyword which is followed by a function name... OTcl declares methods this
> way. )  Right now we can compile snavigator and it will now recognize
> variables declared within an instproc as variables (so they turn up when I
> use snavigator's filter and ask it to show me all variables). However, the
> instproc function/method doesn't show up in the filter.  
> 
> I imagine the fact that lines with instproc have a word (the class) before
> the keyword 'instproc' is part of the problem, with the parsing.  
> 
> Any ideas, I don't have time to create a whole new parser for OTcl and am
> hoping for some clue about how to go about completing this hack...
> 
> Thanks in advance for clues!!
> Lori
> 
> *************************************************
> 
> These are the changes so far:
> /projects/ccrg/mmosko/code/SN/source/Makefile 
> 	Changed top line to lower one:
> 	#all-sid: all-tcl all-tk all-libremote
> 	all-sid: all-tcl all-tk
> Makefile.ct     
> Makefile.ct: (reverse lines 25 & 26)
>      24 Tree.h Tree.c:          tcl.ast
>      25     cg -cdimRDIO    tcl.ast;
>      26 #   ast -cd    tcl.ast;
> Trafo.c    change variable 'To_names' to  'to_names'
> tcl.puma   change variable 'To_names' to  'to_names'
> itcl.puma Added {} around all c code, plus line 12 included sn.h (not
> 	the file paf.h)
> tcl.ast   Changed includes for path (no 'include' directory and 
> 		also name of files is different: Idents.h,StringM.h,
> 		and Position.h),line 45 ' instproc       = .', and
> 		iinstproc added line 14 and 25 to list of keywords.
> Scanner.c: Add the typecasts to (int)
>    1051 yyEol ((int)TokenPtr + (int)TokenLength - (int)p - 1);
>    Lines 16 and 23 include "Position.h" (not Positions.h)
>    Lines 
> Parser.c:  Where have "yySnyAttribute.fragment.'*'", changed 
> 	it to "'*'"
> 	Commented out function  "\\ReleaseTreeModule	();"
> 	Wherever had phrase like yySynAttribute.fragment.'$',
> 		changed to just '$'  (same with $, }, { , ] )
> 	Line 40  #	include "Position.h" (not "Positions.h")
> 	
> Parser.h:  
> 	Added '#include <string.h>'
> 	Added:
> 	# if defined __STDC__ | defined __cplusplus
> 	# define ARGS(parameters)	parameters
> 	# else
> 	# define ARGS(parameters)	()
> 	# endif
> Parser.lalr:
> tcl.pars:  Comment out ReleaseTreeModule if the function is still there.
> Tree.h: Line 85 add iinstproc to list of keywords
> Tree.c: 
> Lines 12 and 22, change StringMem.h to StringM.h
> Lines 15 and 25, change Positions.h to Position.h
> Comment out lines 231 - 235
>     226 tTree Tree_Alloc ()
>     227 {
>     228  register yytBlockPtr yyBlockPtr = yyBlockList;
>     229  register int i;
>     230 
>     231  //if (yyMaxSize == 0)
>     232   //for (i = 1; i <= 50; i ++) {
>     233    //Tree_NodeSize [i] = (Tree_NodeSize [i] + yyMaxAlign - 1) &
> yyAlignMasks [yyMaxAlign];
>     234    //yyMaxSize = Max (Tree_NodeSize [i], yyMaxSize);
>     235   //}
>     236 
>     237  yyBlockList = (yytBlockPtr) Alloc (sizeof (yytBlock));
> 
> 
> 
> 
> 
> 
> 
> 
> 
> From Emmanuelle's mail August 9 2001:
> 
> > I want to modify the Tcl parser
> > that it can parse procedures declared that way but I need a 
> > little help
> > to do that; the source code of SN is very large and I don't know from
> > where to start.
> 
> The TCL parser can be found in:
> 
> /source/snavigator/parser/tcl
> 
> There is documentation on how to add parsers:
> 
> /source/snvavigator/doc/html/enterprise/progref
> 
> But the TCL parser does not follow use this API. Do as we say, not as we
> do.
> It inserts symbols with put_symbol. The TCL parsers uses a
> parser-generator
> / compiler-compiler, but it doesn't use yacc/bison. Puma or something. If
> you wish to modify the code, you will need this tool.
> 
> Happy hacking,
> Craig Keogh
> 
> 
>