[Bug translator/10690] New: need way to produce bigger procfs output

[Bug translator/10690] New: need way to produce bigger procfs output

[fche at redhat dot com]

There exist uses for procfs("file").write and especially .read,
for which it is desirable for stap scripts to produce much more than
MAXSTRINGLEN bytes of output.

The probe point syntax could be extended to specify a .maxsize(NNN)
option for the probes.  Then, extra syntax of some sort would be
needed to write/read the $value variable.  Perhaps something like
this for the output side:

probe procfs("file").read.maxbuf(8192) {
  $value .= sprint("hello\n")
  $value .= sprint("more and more test\n")
}

(i.e., we'd have to special-case/optimize the concatenate-assign operator.)

I don't have a simple idea about new syntax to parse extra-long incoming
$value strings.  Maybe a special substr that takes string-references instead
of values.?

-- 
           Summary: need way to produce bigger procfs output
           Product: systemtap
           Version: unspecified
            Status: NEW
          Severity: normal
          Priority: P2
         Component: translator
        AssignedTo: systemtap at sources dot redhat dot com
        ReportedBy: fche at redhat dot com


http://sourceware.org/bugzilla/show_bug.cgi?id=10690

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug translator/10690] need way to produce bigger procfs output

[przemyslaw at pawelczyk dot it]

------- Additional Comments From przemyslaw at pawelczyk dot it  2009-09-23 14:42 -------
This is an invaluable feature for getting various statistics (e.g. @hist_log) on 
demand.

Maybe initially it would be easier to introduce new function (procfs_print?) 
appending provided text to output buffer?

Increasing default buffer size can be over the top, but for users who really need 
it (instead of writing .maxbuf in almost all procfs probes), we can also introduce 
some define (PROCFS_BUFSIZE?) that could be altered via -D.

-- 
           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |przemyslaw at pawelczyk dot
                   |                            |it


http://sourceware.org/bugzilla/show_bug.cgi?id=10690

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug translator/10690] need way to produce bigger procfs output

[fche at redhat dot com]

-- 
           What    |Removed                     |Added
----------------------------------------------------------------------------
OtherBugsDependingO|                            |11179
              nThis|                            |


http://sourceware.org/bugzilla/show_bug.cgi?id=10690

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug translator/10690] need way to produce bigger procfs output

[dsmith at redhat dot com]

------- Additional Comments From dsmith at redhat dot com  2010-01-19 22:31 -------
Some additional info:

One complication in producing bigger procfs input/output is that the maximum
buffer size the kernel will input/output in one call to the procfs
read_proc()/write_proc() is PAGE_SIZE bytes long.  (This is why read_proc() gets
an 'offset' parameter.)

Some kernel procfs file handling code now uses the 'seq' file interface to help
simplify large procfs file handling.  Systemtap may need to consider this.

-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=10690

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug translator/10690] need way to produce bigger procfs output

[dsmith at redhat dot com]

------- Additional Comments From dsmith at redhat dot com  2010-01-22 20:58 -------
(In reply to comment #0)
> There exist uses for procfs("file").write and especially .read,
> for which it is desirable for stap scripts to produce much more than
> MAXSTRINGLEN bytes of output.

I can see the value for .read to produce more than MAXSTRINGLEN bytes of output,
but I'm unsure of the value of .write to be able to read more than MAXSTRINGLEN
bytes of input.

Assuming we could read more than MAXSTRINGLEN bytes of input, what could we then
do with that data?  We can't assign (all of it) to another string variable.  We
could print it (which might fail if the the number of bytes is greater than
STP_BUFFER_SIZE).  I'm unsure of anything else we could do with it.

-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=10690

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug translator/10690] need way to produce bigger procfs output

[dsmith at redhat dot com]

------- Additional Comments From dsmith at redhat dot com  2010-01-22 21:24 -------
I've done some testing on bigger procfs input/output with the current code,
trying to see what happens when MAXSTRINGLEN is changed.  For the .read tests, I
output a string MAXSTRINGLEN bytes long. For the .write tests, I fed a test file
with > 8k worth of data into the procfs file.

Here's what I found.

MAXSTRINGLEN = 512 (the default)
.read = OK
.write = bug - probe gets called multiple times, in MAXSTRINGLEN chunks, but the
last chunk is a mix of old and new data

MAXSTRINGLEN = 4096
.read = bug, only seeing ~3000 chars?
.write = bug, probe gets called multiple times in MAXSTRINGLEN chunks, but the
last chunk is a mix of old and new data

MAXSTRINGLEN = 8192
.read = bug, only seeing 1st 4k of data
.write = bug, probe gets called multiple times in MAXSTRINGLEN chunks, but the
last chunk is a mix of old and new data


-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=10690

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug translator/10690] need way to produce bigger procfs output

[dsmith at redhat dot com]

------- Additional Comments From dsmith at redhat dot com  2010-01-26 19:46 -------
Bug #11223 was added to cover the procfs write problem mentioned in comment 4.

-- 
           What    |Removed                     |Added
----------------------------------------------------------------------------
Bug 10690 depends on bug 11223, which changed state.

Bug 11223 Summary: procfs write probe problem
http://sourceware.org/bugzilla/show_bug.cgi?id=11223

           What    |Old Value                   |New Value
----------------------------------------------------------------------------
             Status|NEW                         |ASSIGNED
             Status|ASSIGNED                    |RESOLVED
         Resolution|                            |FIXEDBug 10690 depends on bug 11223, which changed state.

Bug 11223 Summary: procfs write probe problem
http://sourceware.org/bugzilla/show_bug.cgi?id=11223

           What    |Old Value                   |New Value
----------------------------------------------------------------------------
             Status|NEW                         |ASSIGNED
             Status|ASSIGNED                    |RESOLVED
         Resolution|                            |FIXED

http://sourceware.org/bugzilla/show_bug.cgi?id=10690

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug translator/10690] need way to produce bigger procfs output

[dsmith at redhat dot com]

------- Additional Comments From dsmith at redhat dot com  2010-02-04 19:21 -------
(In reply to comment #4)
> I've done some testing on bigger procfs input/output with the current code,
> trying to see what happens when MAXSTRINGLEN is changed.  For the .read tests, I
> output a string MAXSTRINGLEN bytes long. For the .write tests, I fed a test file
> with > 8k worth of data into the procfs file.
> 
> Here's what I found...

With the fixes to bug #11223 and bug #11078, all the problems mentioned in
comment #4 are fixed.  Input/output of MAXSTRINGLEN sizes of 512, 4096, and 8192
work correctly.

-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=10690

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug translator/10690] need way to produce bigger procfs output

[dsmith at redhat dot com]

------- Additional Comments From dsmith at redhat dot com  2010-02-04 19:31 -------
To build up bigger procfs output, I had assumed we'd use the string
concatenation operator '.='.  Assume the following script.

====
global large_string = "abcdefg"
probe procfs("large_string").read {                                             
      $value = large_string                                                     
      $value .= large_string                                                    
}                                                                               
====

Unfortunately, this doesn't work:

# stap -v ../procfs_read3.stp
Pass 1: parsed user script and 65 library script(s) using
83492virt/20220res/2260shr kb, in 120usr/40sys/164real ms.
semantic error: Operator-assign expressions on target variables not implemented:
operator '.=' at ../procfs_read3.stp:4:14 while resolving probe point
procfs("large_string").read
        source:       $value .= large_string
                             ^
Pass 2: analyzed script: 0 probe(s), 1 function(s), 0 embed(s), 1 global(s)
using 83888virt/20864res/2488shr kb, in 0usr/0sys/6real ms.
Pass 2: analysis failed.  Try again with another '--vp 01' option.

From looking at var_expanding_visitor::visit_assignment() in tapsets.cxx, this
is explicitly denied, with the following comment:

      // FIXME: for the time being, we only support plan $foo = bar,            
      // not += or any other op= variant. This is fixable, but a bit            
      // ugly.                                                                  

It might be time to fix this problem.

-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=10690

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug translator/10690] need way to produce bigger procfs output

[jistone at redhat dot com]

------- Additional Comments From jistone at redhat dot com  2010-02-09 00:05 -------
(In reply to comment #7)
> To build up bigger procfs output, I had assumed we'd use the string
> concatenation operator '.='.
> [...]
> From looking at var_expanding_visitor::visit_assignment() in tapsets.cxx, this
> is explicitly denied, with the following comment:
> 
>   // FIXME: for the time being, we only support plan $foo = bar,            
>   // not += or any other op= variant. This is fixable, but a bit            
>   // ugly.                                                                  
> 
> It might be time to fix this problem.

Well, the most general way I would fix that is to transform into an explicit
read-modify-write, which doesn't help if these procfs strings are oversized.

But in this specific case, I think we can just specialize visit_assignment() in
procfs_var_expanding_visitor, mapping ".=" to a new _procfs_value_append()
instead of _procfs_value_set().

-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=10690

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug translator/10690] need way to produce bigger procfs output

[fche at redhat dot com]

------- Additional Comments From fche at redhat dot com  2010-02-09 15:24 -------
(In reply to comment #8)
> But in this specific case, I think we can just specialize visit_assignment() in
> procfs_var_expanding_visitor, mapping ".=" to a new _procfs_value_append()
> instead of _procfs_value_set().

Exactly right.

-- 


http://sourceware.org/bugzilla/show_bug.cgi?id=10690

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug translator/10690] need way to produce bigger procfs output

[dsmith at redhat dot com]

------- Additional Comments From dsmith at redhat dot com  2010-02-24 17:05 -------
I've implemented the '.=' operator for procfs read probes in commit a50de93. 
Being able to increase the buffer size is still to come.

-- 
           What    |Removed                     |Added
----------------------------------------------------------------------------
         AssignedTo|systemtap at sources dot    |dsmith at redhat dot com
                   |redhat dot com              |
             Status|NEW                         |ASSIGNED


http://sourceware.org/bugzilla/show_bug.cgi?id=10690

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.