* MAXSTRINGLEN applied to printf()?
@ 2012-08-09 4:02 halcyonic
2012-08-09 16:56 ` Josh Stone
0 siblings, 1 reply; 6+ messages in thread
From: halcyonic @ 2012-08-09 4:02 UTC (permalink / raw)
To: systemtap
If I try to printf() more than MAXSTRINGLEN characters without outputting a newline, does that mean I'm exceeding the string length limit? I.e., is just a normal string serving as the printf buffer, and all the normal rules about strings apply to it? (I was trying to get around the string length limits by issuing multiple printf()'s, but that seems to be backfiring...)
Thanks,
Nick
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: MAXSTRINGLEN applied to printf()?
2012-08-09 4:02 MAXSTRINGLEN applied to printf()? halcyonic
@ 2012-08-09 16:56 ` Josh Stone
2012-08-09 19:51 ` halcyonic
0 siblings, 1 reply; 6+ messages in thread
From: Josh Stone @ 2012-08-09 16:56 UTC (permalink / raw)
To: halcyonic; +Cc: systemtap
On 08/08/2012 09:01 PM, halcyonic@gmail.com wrote:
> If I try to printf() more than MAXSTRINGLEN characters without
> outputting a newline, does that mean I'm exceeding the string length
> limit? I.e., is just a normal string serving as the printf buffer,
> and all the normal rules about strings apply to it? (I was trying to
> get around the string length limits by issuing multiple printf()'s,
> but that seems to be backfiring...)
As long as you're not using intermediate strings, MAXSTRINGLEN should
not limit you. There is a different limit STP_BUFFER_SIZE which is 8192
bytes, but I believe even that is for each individual printf call.
As before, examples of what you're trying and the result would be helpful.
Josh
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: MAXSTRINGLEN applied to printf()?
2012-08-09 16:56 ` Josh Stone
@ 2012-08-09 19:51 ` halcyonic
2012-08-09 20:31 ` Josh Stone
0 siblings, 1 reply; 6+ messages in thread
From: halcyonic @ 2012-08-09 19:51 UTC (permalink / raw)
To: Josh Stone; +Cc: systemtap
Sorry...didn't want to spam a lot of code and output.
So, for example, I get a long output that contains this fragment:
... ,"localhost.localdomain"localhost.localdomain_2012-8-8-23-47-30.bz2", ...
...generated by the following probe (I've set string lengths to be 2048). The thing to notice is that any string concatenation I do should always involve the addition of a pair of quotes, so I should never be able to get an output that involves an odd number of quotes (as per above). So either I'm somehow getting intermingled output (I don't think I am...the rest of the output looks perfectly fine), or somehow a string is getting silently truncated somewhere (fyi, all the filenames it's listing are of the form localhost.localdomain_2012-8-8-23-47-30.bz2).
Any theories appreciated.
Thanks,
Nick
-----
probe syscall.getdents.return {
if ((execname() != "stap") && !is_fd_blacklisted(pid(), $fd)) {
printf("{ \"arglist\":")
arglist = "[ "
if ($return > 0) {
total_entries = 0
dirent = $dirent
total_bytes = 0
current_bytes = 0
while (total_bytes < $return) {
if (dirent == 0) {
break
}
nextarg = clean_string(user_string_warn(@cast(dirent, "struct linux_dirent")->d_name))
len = @cast(dirent, "struct linux_dirent")->d_reclen
formatlen = strlen(nextarg) + 2
dirent += len
total_bytes += len
total_entries += 1
if (total_entries < 256) {
if (current_bytes + formatlen > 2048) {
printf("%s", arglist)
arglist = ""
current_bytes = 0
}
arglist .= "\"".nextarg."\""
current_bytes += formatlen
if (total_bytes < $return) {
arglist .= ","
}
}
}
}
printf("%s],", arglist)
# arglist = substr(arglist, 0, strlen(arglist)-1)."]"
# outstr = "{ "
# outstr .= "\"arglist\": "
# outstr .= arglist.","
outstr .= "\"count\": "
outstr .= sprintf("%u", $count).","
outstr .= "\"execname\": \""
outstr .= clean_string(execname())."\","
outstr .= "\"fd\": "
outstr .= sprintf("%d", $fd).","
outstr .= "\"op\": \""
outstr .= clean_string("GETDENTS")."\","
outstr .= "\"pid\": "
outstr .= sprintf("%d", pid()).","
outstr .= "\"ppid\": "
outstr .= sprintf("%d", ppid()).","
outstr .= "\"return\": "
outstr .= sprintf("%d", $return).","
outstr .= "\"timestamp\": "
outstr .= sprintf("%d", gettimeofday_ms()).","
outstr .= "\"total_bytes\": "
outstr .= sprintf("%u", total_bytes).","
outstr .= "\"total_entries\": "
outstr .= sprintf("%u", total_entries).","
outstr .= "\"uid\": "
outstr .= sprintf("%d", uid())."}\n"
printf("%s", outstr)
}
}
-----
On Aug 9, 2012, at 12:56 PM, Josh Stone <jistone@redhat.com> wrote:
> On 08/08/2012 09:01 PM, halcyonic@gmail.com wrote:
>> If I try to printf() more than MAXSTRINGLEN characters without
>> outputting a newline, does that mean I'm exceeding the string length
>> limit? I.e., is just a normal string serving as the printf buffer,
>> and all the normal rules about strings apply to it? (I was trying to
>> get around the string length limits by issuing multiple printf()'s,
>> but that seems to be backfiring...)
>
> As long as you're not using intermediate strings, MAXSTRINGLEN should
> not limit you. There is a different limit STP_BUFFER_SIZE which is 8192
> bytes, but I believe even that is for each individual printf call.
>
> As before, examples of what you're trying and the result would be helpful.
>
>
> Josh
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: MAXSTRINGLEN applied to printf()?
2012-08-09 19:51 ` halcyonic
@ 2012-08-09 20:31 ` Josh Stone
2012-08-09 22:25 ` halcyonic
0 siblings, 1 reply; 6+ messages in thread
From: Josh Stone @ 2012-08-09 20:31 UTC (permalink / raw)
To: halcyonic; +Cc: systemtap
On 08/09/2012 12:51 PM, halcyonic@gmail.com wrote:
> Sorry...didn't want to spam a lot of code and output.
Thanks - what you have here is not nearly large enough that I'd call it
spam, and it's much better than us trying to guess what's going on.
> So, for example, I get a long output that contains this fragment:
>
> ... ,"localhost.localdomain"localhost.localdomain_2012-8-8-23-47-30.bz2", ...
>
> ...generated by the following probe (I've set string lengths to be
> 2048). The thing to notice is that any string concatenation I do
> should always involve the addition of a pair of quotes, so I should
> never be able to get an output that involves an odd number of quotes
> (as per above).
But if your concatenation goes beyond MAXSTRINGLEN, then one of the pair
of quotes may be silently truncated.
> So either I'm somehow getting intermingled output (I
> don't think I am...the rest of the output looks perfectly fine), or
> somehow a string is getting silently truncated somewhere (fyi, all
> the filenames it's listing are of the form
> localhost.localdomain_2012-8-8-23-47-30.bz2).
>
> Any theories appreciated.
>
> Thanks, Nick
>
> -----
> probe syscall.getdents.return {
> if ((execname() != "stap") && !is_fd_blacklisted(pid(), $fd)) {
> printf("{ \"arglist\":")
> arglist = "[ "
> if ($return > 0) {
> total_entries = 0
> dirent = $dirent
> total_bytes = 0
> current_bytes = 0
> while (total_bytes < $return) {
> if (dirent == 0) {
> break
> }
> nextarg = clean_string(user_string_warn(@cast(dirent, "struct linux_dirent")->d_name))
What is clean_string(), something to sanitize to printable characters?
You might like user_string_quoted(), or "text_strn(str, 0, 1)" to quote
it after the fact.
> len = @cast(dirent, "struct linux_dirent")->d_reclen
> formatlen = strlen(nextarg) + 2
> dirent += len
> total_bytes += len
> total_entries += 1
>
> if (total_entries < 256) {
> if (current_bytes + formatlen > 2048) {
> printf("%s", arglist)
> arglist = ""
> current_bytes = 0
> }
>
> arglist .= "\"".nextarg."\""
> current_bytes += formatlen
> if (total_bytes < $return) {
> arglist .= ","
> }
> }
Ok, I see you're trying to avoid MAXSTRINGLEN here. I think your bug
may be simply when you add "," to arglist without also incrementing
current_bytes, so arglist is longer than you think when you check if a
printf is due.
Also, don't forget that a \0 terminator has to be present within
MAXSTRINGLEN too, so you really only have 2047 bytes to play with.
> }
> }
> printf("%s],", arglist)
> # arglist = substr(arglist, 0, strlen(arglist)-1)."]"
> # outstr = "{ "
> # outstr .= "\"arglist\": "
> # outstr .= arglist.","
> outstr .= "\"count\": "
> outstr .= sprintf("%u", $count).","
> outstr .= "\"execname\": \""
> outstr .= clean_string(execname())."\","
> outstr .= "\"fd\": "
> outstr .= sprintf("%d", $fd).","
> outstr .= "\"op\": \""
> outstr .= clean_string("GETDENTS")."\","
> outstr .= "\"pid\": "
> outstr .= sprintf("%d", pid()).","
> outstr .= "\"ppid\": "
> outstr .= sprintf("%d", ppid()).","
> outstr .= "\"return\": "
> outstr .= sprintf("%d", $return).","
> outstr .= "\"timestamp\": "
> outstr .= sprintf("%d", gettimeofday_ms()).","
> outstr .= "\"total_bytes\": "
> outstr .= sprintf("%u", total_bytes).","
> outstr .= "\"total_entries\": "
> outstr .= sprintf("%u", total_entries).","
> outstr .= "\"uid\": "
> outstr .= sprintf("%d", uid())."}\n"
> printf("%s", outstr)
> }
> }
> -----
With your MAXTRINGLEN=2048, you're probably not hitting limits here at
the end, but every single one of these ".", ".=", and "sprintf" create
string temporaries. The code we generate for this will be pretty
inefficient, with lots of string copies. So I'd recommend building as
much as you can directly into that final output format, e.g.
printf("\"count\":%u,\"execname\":%s, ...\n",
$count, clean_string(execname()), ...)
Josh
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: MAXSTRINGLEN applied to printf()?
2012-08-09 20:31 ` Josh Stone
@ 2012-08-09 22:25 ` halcyonic
2012-08-09 23:43 ` Josh Stone
0 siblings, 1 reply; 6+ messages in thread
From: halcyonic @ 2012-08-09 22:25 UTC (permalink / raw)
To: Josh Stone; +Cc: systemtap
Sigh...that was the other reason I was afraid of posting code: it would turn out to be a stupid bug. I think you may be right that it was simply not accounting for the commas. Sorry for the distraction. :-/
Thanks,
Nick
On Aug 9, 2012, at 4:31 PM, Josh Stone <jistone@redhat.com> wrote:
> On 08/09/2012 12:51 PM, halcyonic@gmail.com wrote:
>> Sorry...didn't want to spam a lot of code and output.
>
> Thanks - what you have here is not nearly large enough that I'd call it
> spam, and it's much better than us trying to guess what's going on.
>
>> So, for example, I get a long output that contains this fragment:
>>
>> ... ,"localhost.localdomain"localhost.localdomain_2012-8-8-23-47-30.bz2", ...
>>
>> ...generated by the following probe (I've set string lengths to be
>> 2048). The thing to notice is that any string concatenation I do
>> should always involve the addition of a pair of quotes, so I should
>> never be able to get an output that involves an odd number of quotes
>> (as per above).
>
> But if your concatenation goes beyond MAXSTRINGLEN, then one of the pair
> of quotes may be silently truncated.
>
>> So either I'm somehow getting intermingled output (I
>> don't think I am...the rest of the output looks perfectly fine), or
>> somehow a string is getting silently truncated somewhere (fyi, all
>> the filenames it's listing are of the form
>> localhost.localdomain_2012-8-8-23-47-30.bz2).
>>
>> Any theories appreciated.
>>
>> Thanks, Nick
>>
>> -----
>> probe syscall.getdents.return {
>> if ((execname() != "stap") && !is_fd_blacklisted(pid(), $fd)) {
>> printf("{ \"arglist\":")
>> arglist = "[ "
>> if ($return > 0) {
>> total_entries = 0
>> dirent = $dirent
>> total_bytes = 0
>> current_bytes = 0
>> while (total_bytes < $return) {
>> if (dirent == 0) {
>> break
>> }
>> nextarg = clean_string(user_string_warn(@cast(dirent, "struct linux_dirent")->d_name))
>
> What is clean_string(), something to sanitize to printable characters?
> You might like user_string_quoted(), or "text_strn(str, 0, 1)" to quote
> it after the fact.
>
>> len = @cast(dirent, "struct linux_dirent")->d_reclen
>> formatlen = strlen(nextarg) + 2
>> dirent += len
>> total_bytes += len
>> total_entries += 1
>>
>> if (total_entries < 256) {
>> if (current_bytes + formatlen > 2048) {
>> printf("%s", arglist)
>> arglist = ""
>> current_bytes = 0
>> }
>>
>> arglist .= "\"".nextarg."\""
>> current_bytes += formatlen
>> if (total_bytes < $return) {
>> arglist .= ","
>> }
>> }
>
> Ok, I see you're trying to avoid MAXSTRINGLEN here. I think your bug
> may be simply when you add "," to arglist without also incrementing
> current_bytes, so arglist is longer than you think when you check if a
> printf is due.
>
> Also, don't forget that a \0 terminator has to be present within
> MAXSTRINGLEN too, so you really only have 2047 bytes to play with.
>
>> }
>> }
>> printf("%s],", arglist)
>> # arglist = substr(arglist, 0, strlen(arglist)-1)."]"
>> # outstr = "{ "
>> # outstr .= "\"arglist\": "
>> # outstr .= arglist.","
>> outstr .= "\"count\": "
>> outstr .= sprintf("%u", $count).","
>> outstr .= "\"execname\": \""
>> outstr .= clean_string(execname())."\","
>> outstr .= "\"fd\": "
>> outstr .= sprintf("%d", $fd).","
>> outstr .= "\"op\": \""
>> outstr .= clean_string("GETDENTS")."\","
>> outstr .= "\"pid\": "
>> outstr .= sprintf("%d", pid()).","
>> outstr .= "\"ppid\": "
>> outstr .= sprintf("%d", ppid()).","
>> outstr .= "\"return\": "
>> outstr .= sprintf("%d", $return).","
>> outstr .= "\"timestamp\": "
>> outstr .= sprintf("%d", gettimeofday_ms()).","
>> outstr .= "\"total_bytes\": "
>> outstr .= sprintf("%u", total_bytes).","
>> outstr .= "\"total_entries\": "
>> outstr .= sprintf("%u", total_entries).","
>> outstr .= "\"uid\": "
>> outstr .= sprintf("%d", uid())."}\n"
>> printf("%s", outstr)
>> }
>> }
>> -----
>
> With your MAXTRINGLEN=2048, you're probably not hitting limits here at
> the end, but every single one of these ".", ".=", and "sprintf" create
> string temporaries. The code we generate for this will be pretty
> inefficient, with lots of string copies. So I'd recommend building as
> much as you can directly into that final output format, e.g.
>
> printf("\"count\":%u,\"execname\":%s, ...\n",
> $count, clean_string(execname()), ...)
>
>
> Josh
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: MAXSTRINGLEN applied to printf()?
2012-08-09 22:25 ` halcyonic
@ 2012-08-09 23:43 ` Josh Stone
0 siblings, 0 replies; 6+ messages in thread
From: Josh Stone @ 2012-08-09 23:43 UTC (permalink / raw)
To: halcyonic, SystemTap
On 08/09/2012 03:25 PM, halcyonic@gmail.com wrote:
> Sigh...that was the other reason I was afraid of posting code: it
> would turn out to be a stupid bug. I think you may be right that it
> was simply not accounting for the commas. Sorry for the distraction.
> :-/
Well, sharing code lets us track down bugs more quickly, whether the
problem lies in your own code or elsewhere, so I don't think you need to
feel bad. Everyone makes mistakes, and this one was fairly subtle for
me to identify. :) Do let us know if you still find issues.
Josh
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2012-08-09 23:43 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2012-08-09 4:02 MAXSTRINGLEN applied to printf()? halcyonic
2012-08-09 16:56 ` Josh Stone
2012-08-09 19:51 ` halcyonic
2012-08-09 20:31 ` Josh Stone
2012-08-09 22:25 ` halcyonic
2012-08-09 23:43 ` Josh Stone
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).