[Bug translator/10025] New: Problem with conditional preprocessing

[Bug translator/10025] New: Problem with conditional preprocessing

[eteo at redhat dot com]

I tried to do this in my script:

 13 %( $# != 2 %?
 14 
 15 probe begin {
 16         printf("usage:\n\tstap errsnoop.stp sec limit\n")
 17         exit()
 18 }
 19 
 20 %:
 21 
 22 global error, trace
 23 
 24 probe syscall.* {
 25         trace[tid()] = argstr
 26 }
[...]

and it gave me the following error:

$ stap ~/errsnoop.stp 1 25
parse error: expected identifier or '*'
	saw: operator '*' at /home/test/errsnoop.stp:24:15
     source: probe syscall.* {
                           ^
[...]

It looks like having wildcards in the false-token would trigger a parse error
even though it is expecting either a wildcard (or an identifier).

$ stap -V
SystemTap translator/driver (version 0.9.5/0.140 commit dcfd7fed + changes)
Copyright (C) 2005-2009 Red Hat, Inc. and others
This is free software; see the source for copying conditions.

-- 
           Summary: Problem with conditional preprocessing
           Product: systemtap
           Version: unspecified
            Status: NEW
          Severity: normal
          Priority: P2
         Component: translator
        AssignedTo: systemtap at sources dot redhat dot com
        ReportedBy: eteo at redhat dot com


http://sourceware.org/bugzilla/show_bug.cgi?id=10025

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug translator/10025] Problem with conditional preprocessing

[eteo at redhat dot com]

------- Additional Comments From eteo at redhat dot com  2009-04-02 15:39 -------
This is related to bug 5376.

-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=10025

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug translator/10025] wildcards lose when embedded in conditional %( %) expressions

[fche at redhat dot com]

------- Additional Comments From fche at redhat dot com  2009-04-02 16:12 -------
The reason is that the fix for bug #5376 introduced context-sensitive
*tokenization* (by adding the "wildcard" flag to lexer/parser::scan* and
friends), but this context is not available when scanning the preprocessor
THEN/ELSE token streams.


-- 
           What    |Removed                     |Added
----------------------------------------------------------------------------
            Summary|Problem with conditional    |wildcards lose when embedded
                   |preprocessing               |in conditional %( %)
                   |                            |expressions


http://sourceware.org/bugzilla/show_bug.cgi?id=10025

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug translator/10025] wildcards lose when embedded in conditional %( %) expressions

[fche at redhat dot com]

------- Additional Comments From fche at redhat dot com  2010-05-13 15:48 -------
*** Bug 11596 has been marked as a duplicate of this bug. ***

-- 
           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |dsmith at redhat dot com


http://sourceware.org/bugzilla/show_bug.cgi?id=10025

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug translator/10025] wildcards lose when embedded in conditional %( %) expressions

[dsmith at redhat dot com]

------- Additional Comments From dsmith at redhat dot com  2010-05-13 17:16 -------
Here's a "band-aid" patch that seems to work, but doesn't really fix the
underlying issues:

diff --git a/parse.cxx b/parse.cxx
index 37e0762..93a7d2e 100644
--- a/parse.cxx
+++ b/parse.cxx
@@ -1526,7 +1526,9 @@ parser::parse_probe_point ()
       const token* t = next (true); // wildcard scanning here
       if (! (t->type == tok_identifier
 	     // we must allow ".return" and ".function", which are keywords
-	     || t->type == tok_keyword))
+	     || t->type == tok_keyword
+	     // sometimes "*" is treated like an operator
+	     || (t->type == tok_operator && t->content == "*")))
         throw parse_error ("expected identifier or '*'");
 


-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=10025

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug translator/10025] wildcards lose when embedded in conditional %( %) expressions

[jistone at redhat dot com]

------- Additional Comments From jistone at redhat dot com  2010-05-13 18:02 -------
(In reply to comment #4)
> Here's a "band-aid" patch that seems to work, but doesn't really fix the
> underlying issues:

That only helps for the simplest wildcard, "*".  For anything more complex, say
"foo*", you'll end up with tok_identifier("foo") followed by tok_operator("*"),
whereas outside of the preprocessor it's a single tok_identifier("foo*").

I thought about peeking ahead to the next token and merging them, but you also
need to make sure that we don't merge them when they truly are two tokens, like
"foo *" (which in a probe point should be a syntax error).

Anyway if we figure this out, I suggest making that kludge permanent, and remove
the equally kludgy wildcard flag from the tokenizer.

-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=10025

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug translator/10025] wildcards lose when embedded in conditional %( %) expressions

[fche at redhat dot com]

------- Additional Comments From fche at redhat dot com  2010-05-18 04:43 -------
> I thought about peeking ahead to the next token and merging them, but you also
> need to make sure that we don't merge them when they truly are two tokens, like
> "foo *" (which in a probe point should be a syntax error).

Maybe we can relax that 'should be a syntax error' thing and accept it anyway.
(This could be a stap version-sensitive parser change if desired).  So the
idea would be to do token-pasting within parse_probe_point only, gluing
together tok_identifier / tok_keyword / selected-tok_operator into a single
synthetic tok_identifier for the probe_point component.  If we want to be
selective to the presence of whitespace, one not-too-gross way may be to
have a new flag within tokens, set by the lexer, to indicate that some
whitespace was skipped since the last token.  We'd stop the gluing upon
such a token.


-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=10025

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug translator/10025] wildcards lose when embedded in conditional %( %) expressions

[jistone at redhat dot com]

------- Additional Comments From jistone at redhat dot com  2010-05-20 01:13 -------
Created an attachment (id=4803)
 --> (http://sourceware.org/bugzilla/attachment.cgi?id=4803&action=view)
Don't read ahead tokens within preprocessor blocks

PR10025: Don't read ahead tokens within preprocessor blocks

Since some aspects of the lexer are contextual, like wildcards in
probepoint identifiers, we need to wait for the right context in the
preprocessor rather than reading a bunch of tokens at once.


-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=10025

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug translator/10025] wildcards lose when embedded in conditional %( %) expressions

[jistone at redhat dot com]

------- Additional Comments From jistone at redhat dot com  2010-05-20 01:21 -------
(In reply to comment #6)
> If we want to be selective to the presence of whitespace, one
> not-too-gross way may be to have a new flag within tokens, set by the
> lexer, to indicate that some whitespace was skipped since the last
> token.  We'd stop the gluing upon such a token.

The patch I just attached solves the problem without changing the wildcard flag.
 However, if you still want to lessen the wildcard impact, I think we could add
a just-ate-whitespace flag to the lexer instead of carrying it around in every
token.  My patch makes it so we never read more than one token from the lexer at
a time, so the flag would still be current.

-- 
           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |jistone at redhat dot com


http://sourceware.org/bugzilla/show_bug.cgi?id=10025

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug translator/10025] wildcards lose when embedded in conditional %( %) expressions

[fche at redhat dot com]

------- Additional Comments From fche at redhat dot com  2010-05-20 14:54 -------
(In reply to comment #7)
>[...]

I don't entirely grok it, but if it passes test cases including
nested %( %) tests with/without wildcards, great.


-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=10025

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug translator/10025] wildcards lose when embedded in conditional %( %) expressions

[fche at redhat dot com]

------- Additional Comments From fche at redhat dot com  2010-05-21 18:15 -------
Does your code make this test case work?

stap -e 'probe begin { log ( %( 1==1 %? "hello " @1 %) ) }' world
                                        ^^^^^^^^^^^

Note token-gluing within the %( %)


-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=10025

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug translator/10025] wildcards lose when embedded in conditional %( %) expressions

[jistone at redhat dot com]

------- Additional Comments From jistone at redhat dot com  2010-05-21 18:37 -------
(In reply to comment #10)
> Does your code make this test case work?
> 
> stap -e 'probe begin { log ( %( 1==1 %? "hello " @1 %) ) }' world
>                                         ^^^^^^^^^^^
> 
> Note token-gluing within the %( %)

It doesn't work, but neither does this:

  stap -e 'probe begin { log ( "hello " @1 ) }' world

So this isn't really an issue with the preprocessor.  I think the way 11208 was
implemented prevents the @1 token replacement from being noticed when looking
ahead to merge strings.  In fact, even the ("/usr" @1 "/bin") example given in
NEWS doesn't parse for the same reason.  I'll open a separate bugzilla on this.

-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=10025

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug translator/10025] wildcards lose when embedded in conditional %( %) expressions

[jistone at redhat dot com]

------- Additional Comments From jistone at redhat dot com  2010-05-21 21:39 -------
e92f256 Don't read ahead tokens within preprocessor blocks
2e3f94c Test '*' operator vs. identifier while preprocessing


-- 
           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|                            |FIXED


http://sourceware.org/bugzilla/show_bug.cgi?id=10025

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.