[RFC] Crash extension for SystemTap

[RFC] Crash extension for SystemTap

[Satoru MORIYA]

Hi,

Here is an extension(shared object) of the crash to retrieve the trace
data of systemtap scripts.

I'd like to analyze what caused the kernel panic by using the systemtap.
However, currently the systemtap's trace data can't be retrieved from a
dumped image easily. So, I developed a crash's extension which retrieves
the data recorded by systemtap from the dumped image.
Here is a brief document of this extension. This extension supports the new
utt-based buffer as well as the bulk-mode buffer of old systemtap module.

I have tested this extention on the following system.
  * FC6, i386, kernel-2.6.21, systemtap-0.5.14, crash-4.0-1.1
  * FC6, i386, kernel-2.6.20, systemtap-0.5.13/14, crash-4.0-1.1
  * RHEL5, i386, kernel-2.6.18-8.el5, systemtap-0.5.12, crash-4.0-3.14


Preparation
==============
(A) Build the shared-object(stplog.so).

1. Put Makefile and stplog.c into a directory ($DIR)
    $ cd $DIR

2. Make the symbolic link to the crash source code directory
    $ ln -s $WHERE_CRASH_PLACED crash

3. Build
    $ make

(B) Make the crash dump which includes SystemTap trace data.
    (*)If you analyze the live system memory, ignore this section.

1. Install kdump
     If you use FC6, see following URL.
     http://fedoraproject.org/wiki/FC6KdumpKexecHowTo?highlight=%28kdump%29

2. Use SystemTap
    $ stap foo.stp

3. Panic
    $ echo c > /proc/sysrq-trigger

How to use
==============
1. start crash
    $ crash vmlinux vmcore
    (*) If you analyze the live system memory, you don't need "vmcore".
         $ crash vmlinux

2. load the shared-object
    crash> extend $(WHERE_OBJ_PLACED)/stplog.so

3. retrieve the data
    crash> stplog -m <mod_name>
    (*) <mod_name> is the name of trace module from which you retrieve data.

4. You can get output files under the directory whose name is <mod_name>.

Output
==============
stplog command makes a file per channel buffer of relayfs(equivalent to per cpu).
And it also removes padding bytes.


I believe this command is very useful for system administrators
if they monitor their systems with SystemTap.

Best Regards,

---
Satoru MORIYA
Linux Technology Center
Hitachi, Ltd., Systems Development Laboratory
E-mail: satoru.moriya.br@hitachi.com

Re: [Crash-utility] [RFC] Crash extension for SystemTap

[Dave Anderson]

Satoru MORIYA wrote:
> Hi,
> 
> Here is an extension(shared object) of the crash to retrieve the trace
> data of systemtap scripts.
> 

Hi Satoru,

I think you also meant to attach the stplog.c (and its own Makefile?)
to your post?

Is the format of the trace data used by systemtap always the same?
I.e., is it always a kernel buffer filled with ASCII data?

You mention that the command makes a file in a subdirectory of the
running crash session.  Wouldn't it be more flexible to dump the
output to the terminal?  And then if you want to save it to a file,
just do something like this:

  crash> stplog -m mod_name > outputfile

Anyway, I think it's time that I create a repository for extensions
on my people site...

Thanks,
   Dave


> I'd like to analyze what caused the kernel panic by using the systemtap.
> However, currently the systemtap's trace data can't be retrieved from a
> dumped image easily. So, I developed a crash's extension which retrieves
> the data recorded by systemtap from the dumped image.
> Here is a brief document of this extension. This extension supports the new
> utt-based buffer as well as the bulk-mode buffer of old systemtap module.
> 
> I have tested this extention on the following system.
>   * FC6, i386, kernel-2.6.21, systemtap-0.5.14, crash-4.0-1.1
>   * FC6, i386, kernel-2.6.20, systemtap-0.5.13/14, crash-4.0-1.1
>   * RHEL5, i386, kernel-2.6.18-8.el5, systemtap-0.5.12, crash-4.0-3.14
> 
> 
> Preparation
> ==============
> (A) Build the shared-object(stplog.so).
> 
> 1. Put Makefile and stplog.c into a directory ($DIR)
>     $ cd $DIR
> 
> 2. Make the symbolic link to the crash source code directory
>     $ ln -s $WHERE_CRASH_PLACED crash
> 
> 3. Build
>     $ make
> 
> (B) Make the crash dump which includes SystemTap trace data.
>     (*)If you analyze the live system memory, ignore this section.
> 
> 1. Install kdump
>      If you use FC6, see following URL.
>      http://fedoraproject.org/wiki/FC6KdumpKexecHowTo?highlight=%28kdump%29
> 
> 2. Use SystemTap
>     $ stap foo.stp
> 
> 3. Panic
>     $ echo c > /proc/sysrq-trigger
> 
> How to use
> ==============
> 1. start crash
>     $ crash vmlinux vmcore
>     (*) If you analyze the live system memory, you don't need "vmcore".
>          $ crash vmlinux
> 
> 2. load the shared-object
>     crash> extend $(WHERE_OBJ_PLACED)/stplog.so
> 
> 3. retrieve the data
>     crash> stplog -m <mod_name>
>     (*) <mod_name> is the name of trace module from which you retrieve data.
> 
> 4. You can get output files under the directory whose name is <mod_name>.
> 
> Output
> ==============
> stplog command makes a file per channel buffer of relayfs(equivalent to per cpu).
> And it also removes padding bytes.
> 
> 
> I believe this command is very useful for system administrators
> if they monitor their systems with SystemTap.
> 
> Best Regards,
> 
> ---
> Satoru MORIYA
> Linux Technology Center
> Hitachi, Ltd., Systems Development Laboratory
> E-mail: satoru.moriya.br@hitachi.com
> 
> --
> Crash-utility mailing list
> Crash-utility@redhat.com
> https://www.redhat.com/mailman/listinfo/crash-utility

Re: [RFC] Crash extension for SystemTap

[Frank Eigler]

Satoru MORIYA <satoru.moriya.br@hitachi.com> writes:

> [...]
> (A) Build the shared-object(stplog.so).
> 1. Put Makefile and stplog.c into a directory ($DIR)
>     $ cd $DIR
> 2. Make the symbolic link to the crash source code directory
>     $ ln -s $WHERE_CRASH_PLACED crash

It would be nice if crash(8) had a -devel package against which such
an extension could be built.  I submitted RH bug #241045 to try to fix
this.  If that is done, we can include the stplog sources in systemtap,
and build a .so as a part of our build.

> [...]  I believe this command is very useful for system
> administrators if they monitor their systems with SystemTap.

Indeed!  Thank you.


- FChE

Re: [RFC] Crash extension for SystemTap

[Satoru MORIYA]

[-- Attachment #1: Type: text/plain, Size: 2608 bytes --]

Hi,

Sorry, I forgot to attach my source code.
Here is the source code.

Satoru MORIYA wrote:

> Hi,
> 
> Here is an extension(shared object) of the crash to retrieve the trace
> data of systemtap scripts.
> 
> I'd like to analyze what caused the kernel panic by using the systemtap.
> However, currently the systemtap's trace data can't be retrieved from a
> dumped image easily. So, I developed a crash's extension which retrieves
> the data recorded by systemtap from the dumped image.
> Here is a brief document of this extension. This extension supports the new
> utt-based buffer as well as the bulk-mode buffer of old systemtap module.
> 
> I have tested this extention on the following system.
>   * FC6, i386, kernel-2.6.21, systemtap-0.5.14, crash-4.0-1.1
>   * FC6, i386, kernel-2.6.20, systemtap-0.5.13/14, crash-4.0-1.1
>   * RHEL5, i386, kernel-2.6.18-8.el5, systemtap-0.5.12, crash-4.0-3.14
> 
> 
> Preparation
> ==============
> (A) Build the shared-object(stplog.so).
> 
> 1. Put Makefile and stplog.c into a directory ($DIR)
>     $ cd $DIR
> 
> 2. Make the symbolic link to the crash source code directory
>     $ ln -s $WHERE_CRASH_PLACED crash
> 
> 3. Build
>     $ make
> 
> (B) Make the crash dump which includes SystemTap trace data.
>     (*)If you analyze the live system memory, ignore this section.
> 
> 1. Install kdump
>      If you use FC6, see following URL.
>      http://fedoraproject.org/wiki/FC6KdumpKexecHowTo?highlight=%28kdump%29
> 
> 2. Use SystemTap
>     $ stap foo.stp
> 
> 3. Panic
>     $ echo c > /proc/sysrq-trigger
> 
> How to use
> ==============
> 1. start crash
>     $ crash vmlinux vmcore
>     (*) If you analyze the live system memory, you don't need "vmcore".
>          $ crash vmlinux
> 
> 2. load the shared-object
>     crash> extend $(WHERE_OBJ_PLACED)/stplog.so
> 
> 3. retrieve the data
>     crash> stplog -m <mod_name>
>     (*) <mod_name> is the name of trace module from which you retrieve data.
> 
> 4. You can get output files under the directory whose name is <mod_name>.
> 
> Output
> ==============
> stplog command makes a file per channel buffer of relayfs(equivalent to per cpu).
> And it also removes padding bytes.
> 
> 
> I believe this command is very useful for system administrators
> if they monitor their systems with SystemTap.
> 
> Best Regards,
> 
> ---
> Satoru MORIYA
> Linux Technology Center
> Hitachi, Ltd., Systems Development Laboratory
> E-mail: satoru.moriya.br@hitachi.com
> 
> 
> 

-- 
---
Satoru MORIYA
Linux Technology Center
Hitachi, Ltd., Systems Development Laboratory
E-mail: satoru.moriya.br@hitachi.com

[-- Attachment #2: Makefile --]
[-- Type: text/plain, Size: 458 bytes --]

TARGET=stplog.so
CFILE=stplog.c

CFLAGS= -shared -rdynamic -DX86 
CFLAGS+= -I./crash -Wall

PRJNAME=libcrash_for_systemtap
VERSION=`date +%Y%m%d`

$(TARGET):$(CFILE)
	gcc $(CFLAGS) -o $@ $(CFILE)

clean:
	rm -f -r $(TARGET) *~

dist:distclean
	mkdir $(PRJNAME)-$(VERSION)
	cp $(CFILE) Makefile README $(PRJNAME)-$(VERSION)
	tar cvjf $(PRJNAME)-$(VERSION).tar.bz2 $(PRJNAME)-$(VERSION)
	rm -f -r $(PRJNAME)-$(VERSION)

distclean:
	rm -f -r $(TARGET) *~ crash

[-- Attachment #3: stplog.c --]
[-- Type: text/plain, Size: 9402 bytes --]

/*
 crash shared object for retrieving systemtap buffer
 Copyright (c) 2007 Hitachi,Ltd.,
 Created by Satoru Moriya <satoru.moriya.br@hitachi.com>
 
 This program is free software; you can redistribute it and/or modify
 it under the terms of the GNU General Public License as published by
 the Free Software Foundation; either version 2 of the License, or
 (at your option) any later version.

 This program is distributed in the hope that it will be useful,
 but WITHOUT ANY WARRANTY; without even the implied warranty of
 MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 GNU General Public License for more details.

 You should have received a copy of the GNU General Public License
 along with this program; if not, write to the Free Software
 Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
*/

#include "defs.h"

#define STPLOG_NO_MOD  -1
#define STPLOG_NO_SYM  -2

struct rchan_offsets {
	long	subbuf_size;
	long	n_subbufs;
	long	buf;
	long	buf_start;
	long	buf_offset;
	long	buf_subbufs_produced;
	long	buf_padding;
};

struct fake_rchan_buf {
	void	*start;
	size_t	offset;
	size_t	subbufs_produced;
	size_t	*padding;
};

struct fake_rchan {
	size_t	subbuf_size;
	size_t	n_subbufs;
};

struct per_cpu_data {
	struct fake_rchan_buf	buf;
};

static struct rchan_offsets rchan_offsets;
static struct fake_rchan chan;
static struct per_cpu_data per_cpu[NR_CPUS];
static FILE *outfp;
static char *subbuf;
static jmp_buf saved_env;
static int is_global;
static int old_format;

void cmd_systemtaplog(void);
char *help_systemtaplog[];

static struct command_table_entry command_table[] = {
	{"stplog", cmd_systemtaplog, help_systemtaplog, 0},
	{NULL},
};

static void cleanup(void)
{
	if (outfp) {
		fclose(outfp);
		outfp = NULL;
	}
	if (subbuf) {
		free(subbuf);
		subbuf = NULL;
	}
}

static int get_rchan_offsets(void)
{
	rchan_offsets.subbuf_size = MEMBER_OFFSET("rchan", "subbuf_size");
	if (rchan_offsets.subbuf_size < 0)
		goto ERR;
	rchan_offsets.n_subbufs = MEMBER_OFFSET("rchan", "n_subbufs");
	if (rchan_offsets.n_subbufs < 0)
		goto ERR;
	rchan_offsets.buf = MEMBER_OFFSET("rchan", "buf");
	if (rchan_offsets.buf < 0)
		goto ERR;
	rchan_offsets.buf_start = MEMBER_OFFSET("rchan_buf", "start");
	if (rchan_offsets.buf_start < 0)
		goto ERR;
	rchan_offsets.buf_offset = MEMBER_OFFSET("rchan_buf", "offset");
	if (rchan_offsets.buf_offset < 0)
		goto ERR;
	rchan_offsets.buf_subbufs_produced
		= MEMBER_OFFSET("rchan_buf", "subbufs_produced");
	if (rchan_offsets.buf_subbufs_produced < 0)
		goto ERR;
	rchan_offsets.buf_padding = MEMBER_OFFSET("rchan_buf", "padding");
	if (rchan_offsets.buf_padding < 0)
		goto ERR;
	return 0;
ERR:
	error(WARNING, "cannot get rchan offset\n");
	return -1;
}

static ulong get_rchan(ulong chan_addr) 
{
	ulong rchan;

	readmem(chan_addr, KVADDR, &rchan, sizeof(void*),
		"stp_channel", FAULT_ON_ERROR);
	readmem(rchan + rchan_offsets.subbuf_size,
		KVADDR, &chan.subbuf_size, sizeof(size_t),
		"stp_channel.subbuf_size", FAULT_ON_ERROR);
	readmem(rchan + rchan_offsets.n_subbufs,
		KVADDR, &chan.n_subbufs, sizeof(size_t),
		"stp_channel.n_subbufs", FAULT_ON_ERROR);

	return rchan;
}

static void get_rchan_buf(int cpu, ulong rchan) 
{
	ulong rchan_buf;
	struct per_cpu_data *pcd;

	pcd = &per_cpu[cpu];
	readmem(rchan + rchan_offsets.buf + sizeof(void*) * cpu,
		KVADDR, &rchan_buf, sizeof(void*),
		"stp_channel.buf", FAULT_ON_ERROR);
	readmem(rchan_buf + rchan_offsets.buf_start,
		KVADDR, &pcd->buf.start, sizeof(void*),
		"stp_channel.buf.start", FAULT_ON_ERROR);
	readmem(rchan_buf + rchan_offsets.buf_offset,
		KVADDR, &pcd->buf.offset, sizeof(size_t),
		"stp_channel.buf.offset", FAULT_ON_ERROR);
	readmem(rchan_buf + rchan_offsets.buf_subbufs_produced,
		KVADDR, &pcd->buf.subbufs_produced, sizeof(size_t),
		"stp_channel.buf.subbufs_produced", FAULT_ON_ERROR);
	readmem(rchan_buf + rchan_offsets.buf_padding,
		KVADDR, &pcd->buf.padding, sizeof(size_t*),
		"stp_channel.buf.padding", FAULT_ON_ERROR);
}

static ulong get_symbol_addr(char *module, char *symbol)
{
	int i;
	struct syment *sym, *sym_end;
	struct load_module *lm;

        for (i = 0; i < kt->mods_installed; i++) {
                lm = &st->load_modules[i];
                if (!STREQ(module, lm->mod_name))
                        continue;
                sym = lm->mod_symtable;
                sym_end = lm->mod_symend;		
                for ( ; sym <= sym_end; sym++) {
			if(STREQ(sym->name, symbol))
				return ((ulong)sym->value);
                }
		error(WARNING, "'%s' doesn't have the symbol named '%s'.\n", 
		      module, symbol);
		return STPLOG_NO_SYM;
        }
	error(WARNING, "'%s' is not loaded.\n", module);
	return STPLOG_NO_MOD;
}

static ulong get_rchan_addr(ulong stp_utt_addr)
{
	ulong stp_utt;

	readmem(stp_utt_addr, KVADDR, &stp_utt, sizeof(void*),
		"stp_utt", FAULT_ON_ERROR);
	return (stp_utt + sizeof(int));
}

static int check_global_buffer(ulong rchan)
{
	int cpu;
	ulong rchan_buf[2];
	
	for (cpu = 0; cpu < 2; cpu++) {
		readmem(rchan + rchan_offsets.buf + sizeof(void*) * cpu,
			KVADDR, &rchan_buf[cpu], sizeof(void*),
			"stp_channel.buf", FAULT_ON_ERROR);
	}
	if (rchan_buf[0] == rchan_buf[1])
		return 1;
	return 0;
}

static int setup_global_data(char *module) 
{
	int i;
	ulong stp_utt_addr = 0;
	ulong stp_rchan_addr = 0;
	ulong rchan;

	stp_utt_addr = get_symbol_addr(module, "_stp_utt");
	if (stp_utt_addr == STPLOG_NO_MOD) {
		return -1;
	} else if (stp_utt_addr == STPLOG_NO_SYM) {
		stp_rchan_addr = get_symbol_addr(module, "_stp_chan");
		if (stp_rchan_addr == STPLOG_NO_SYM)
			return -1;
		old_format = 1;
	} else {
		stp_rchan_addr = get_rchan_addr(stp_utt_addr);
	}
	rchan = get_rchan(stp_rchan_addr);
	for (i = 0; i < kt->cpus; i++)
		get_rchan_buf(i, rchan);

	if (kt->cpus > 1) {
		is_global = check_global_buffer(rchan);
	}
	
	return 0;
}

static int output_cpu_logs(char *module)
{
	int i, max = 256;
	struct per_cpu_data *pcd;
	size_t n, idx, start, end, ready, len;
	unsigned padding;
	char fname[max + 1], *source;
	DIR *dir;

	/* check and create log directory */
	dir = opendir(module);
	if (dir) {
		closedir(dir);
	} else {
		if (mkdir(module, S_IRWXU) < 0) {
			error(WARNING, "cannot create log directory '%s\n'", module);
			return -1;
		}
	}

	/* allocate subbuf memory */
	subbuf = malloc(chan.subbuf_size);
	if (!subbuf) {
		error(WARNING, "cannot allocate memory\n");
		return -1;
	}

	fname[max] = '\0';
	for (i = 0; i < kt->cpus; i++) {
		pcd = &per_cpu[i];
		ready = pcd->buf.subbufs_produced + (pcd->buf.offset ? 1 : 0);
		if (ready > chan.n_subbufs) {
			start = ready % chan.n_subbufs;
			end = start + chan.n_subbufs;
		} else {
			start = 0;
			end = ready;
		}
		/* print information */
		fprintf(fp, "--- generating 'cpu%d' ---\n", i);
		fprintf(fp, "  subbufs ready on relayfs:%d\n", ready);
		fprintf(fp, "    n_subbufs:%d, read from:%d to:%d (offset:%d)\n\n",
		       chan.n_subbufs, start, end, pcd->buf.offset);

		/* create log file */
		snprintf(fname, max, "%s/cpu%d", module, i);
		outfp = fopen(fname, "w");
		if (!outfp) {
			error(WARNING, "cannot create log file '%s'\n", fname);
			return -1;
		}
		for (n = start; n < end; n++) {
			/* read relayfs subbufs and write to log file */
			idx = n % chan.n_subbufs;
			source = pcd->buf.start + idx * chan.subbuf_size;
			readmem((ulong)pcd->buf.padding + sizeof(padding) * idx,
				KVADDR, &padding, sizeof(padding),
				"padding", FAULT_ON_ERROR);
			if (n == end - 1 && pcd->buf.offset) {
				len = pcd->buf.offset;
			} else {
				len = chan.subbuf_size;
			}			
			if (old_format == 1) {
				source += sizeof(padding);
				len -= sizeof(padding) + padding;
			} else {
				len -= padding;
			}
			if (len) {
				readmem((ulong)source, KVADDR, subbuf, len,
					"subbuf", FAULT_ON_ERROR);
				if (fwrite(subbuf, len, 1, outfp) != 1) {
					error(WARNING, "cannot write log data\n");
					return -1;
				}
			}
		}
		fclose(outfp);
		outfp = NULL;
		if (is_global == 1)
			break;
	}
	return 0;
}

static void do_systemtaplog(char *module)
{
	if (setup_global_data(module) < 0)
		return;
	if (output_cpu_logs(module) < 0)
		return;
}

void cmd_systemtaplog(void)
{

	int c;
	char *module = NULL;

	while ((c = getopt(argcnt, args, "m:")) != EOF) {
		switch (c) {
		case 'm':
			module = optarg;
			break;
		default:
			argerrs++;
			break;
		}
	}

	if (!module || argerrs)
		cmd_usage(pc->curcmd, SYNOPSIS);

	saved_env[0] = pc->main_loop_env[0];
	if (setjmp(pc->main_loop_env))
		goto EXIT;

	do_systemtaplog(module);
EXIT:
	cleanup();
	pc->main_loop_env[0] = saved_env[0];
}

char *help_systemtaplog[] = {
	"systemtaplog",
	"Retrieve SystemTap log data",
	"-m module_name",
	"  Retrieve SystemTap's log data and write them to files.\n",
	"    -m module_name  All valid SystemTap log data made by the trace",
	"                    module which name is 'module_name' are written",
	"                    into log files in `module_name` directory. The",
	"                    name of each log file is cpu0, cpu1...cpuN. ",
	"                    They have same format data as channel buffer",
	"                    except padding(This command removes padding). ",
	NULL,
};

void __attribute__ ((constructor)) systemtaplog_init(void) 
{
	if (get_rchan_offsets() < 0)
		return;

	register_extension(command_table);
	return;
}

void __attribute__ ((destructor)) systemtaplog_fini(void)
{
	return;
}

Re: [Crash-utility] Re: [RFC] Crash extension for SystemTap

[Dave Anderson]

Satoru MORIYA wrote:
> Hi,
> 
> Sorry, I forgot to attach my source code.
> Here is the source code.
> 

Hi Satoru,

This looks really good for the most part.  I do have a few suggestions.

The systemtaplog_init() function must be renamed "_init()" in order
for it to be called when the dlopen() call is made.  Otherwise,
the extend command fails because no commands get registered:

   crash> extend stplog.so
   extend: ./stplog.so: no commands registered: shared object unloaded
   crash>

The only other thing that makes me a little nervous is the temporary
relocation of the setjmp buffer.  That was never meant to be used
by individual commands -- although nothing prevents it.  I see that it
is used to force your cleanup() routine to be called.  And your
cleanup() routine exists to free() the subbuf buffer and fclose()
the outfp.  A couple suggestions:

(1) if you use GETBUF() instead of malloc(), then the buffer will get freed
     automatically by restore_sanity() prior to the next command prompt -- even
     if you don't do an accompanying FREEBUF().
(2) with respect to the outfp file pointer, you could recognize whether it
     was left open when the command runs again, and fclose() it then.  Worse
     case, a single file pointer would be left open for the remainder of the
     crash session.

I guess that I never expected an extension file would ever do any file
opening/closing, and so I presumed that the current crash-state restoration
mechanism would suffice.  I now see that it would make sense to allow the
registration of some kind of cleanup() function for each extension module.
But I also don't want to break any backwards-compatibility with older extension
modules.  So perhaps I could simply allow modules to register an additional
command with a new "CLEANUP" flag, which would not show up on the help menu,
but would be called during restore_sanity() prior to each command prompt.

Also, I really like your get_symbol_addr() function, which restricts
the symbol search to a specific module.  Sometimes there are symbol
name collisions between the base kernel symbols and the various
module symbols, and the symbol_search() function returns the first
one found, typically base kernel version.  You could use get_syment_array()
to get an array of symbols with the same name, and then parse the list
for the one belonging to your module, but having a function that targets
a specific module makes more sense in your case.  So I'm going to add
a new function in symbols.c that does the same kind of thing as your
get_symbol_addr() function.

Anyway, aside from the requirement to rename systemtaplog_init(), you can do
whatever you want -- it's your module!

Thanks,
   Dave
















> Satoru MORIYA wrote:
> 
>> Hi,
>>
>> Here is an extension(shared object) of the crash to retrieve the trace
>> data of systemtap scripts.
>>
>> I'd like to analyze what caused the kernel panic by using the systemtap.
>> However, currently the systemtap's trace data can't be retrieved from a
>> dumped image easily. So, I developed a crash's extension which retrieves
>> the data recorded by systemtap from the dumped image.
>> Here is a brief document of this extension. This extension supports 
>> the new
>> utt-based buffer as well as the bulk-mode buffer of old systemtap module.
>>
>> I have tested this extention on the following system.
>>   * FC6, i386, kernel-2.6.21, systemtap-0.5.14, crash-4.0-1.1
>>   * FC6, i386, kernel-2.6.20, systemtap-0.5.13/14, crash-4.0-1.1
>>   * RHEL5, i386, kernel-2.6.18-8.el5, systemtap-0.5.12, crash-4.0-3.14
>>
>>
>> Preparation
>> ==============
>> (A) Build the shared-object(stplog.so).
>>
>> 1. Put Makefile and stplog.c into a directory ($DIR)
>>     $ cd $DIR
>>
>> 2. Make the symbolic link to the crash source code directory
>>     $ ln -s $WHERE_CRASH_PLACED crash
>>
>> 3. Build
>>     $ make
>>
>> (B) Make the crash dump which includes SystemTap trace data.
>>     (*)If you analyze the live system memory, ignore this section.
>>
>> 1. Install kdump
>>      If you use FC6, see following URL.
>>      
>> http://fedoraproject.org/wiki/FC6KdumpKexecHowTo?highlight=%28kdump%29
>>
>> 2. Use SystemTap
>>     $ stap foo.stp
>>
>> 3. Panic
>>     $ echo c > /proc/sysrq-trigger
>>
>> How to use
>> ==============
>> 1. start crash
>>     $ crash vmlinux vmcore
>>     (*) If you analyze the live system memory, you don't need "vmcore".
>>          $ crash vmlinux
>>
>> 2. load the shared-object
>>     crash> extend $(WHERE_OBJ_PLACED)/stplog.so
>>
>> 3. retrieve the data
>>     crash> stplog -m <mod_name>
>>     (*) <mod_name> is the name of trace module from which you retrieve 
>> data.
>>
>> 4. You can get output files under the directory whose name is <mod_name>.
>>
>> Output
>> ==============
>> stplog command makes a file per channel buffer of relayfs(equivalent 
>> to per cpu).
>> And it also removes padding bytes.
>>
>>
>> I believe this command is very useful for system administrators
>> if they monitor their systems with SystemTap.
>>
>> Best Regards,
>>
>> ---
>> Satoru MORIYA
>> Linux Technology Center
>> Hitachi, Ltd., Systems Development Laboratory
>> E-mail: satoru.moriya.br@hitachi.com
>>
>>
>>
> 
> 
> ------------------------------------------------------------------------
> 
> TARGET=stplog.so
> CFILE=stplog.c
> 
> CFLAGS= -shared -rdynamic -DX86 
> CFLAGS+= -I./crash -Wall
> 
> PRJNAME=libcrash_for_systemtap
> VERSION=`date +%Y%m%d`
> 
> $(TARGET):$(CFILE)
> 	gcc $(CFLAGS) -o $@ $(CFILE)
> 
> clean:
> 	rm -f -r $(TARGET) *~
> 
> dist:distclean
> 	mkdir $(PRJNAME)-$(VERSION)
> 	cp $(CFILE) Makefile README $(PRJNAME)-$(VERSION)
> 	tar cvjf $(PRJNAME)-$(VERSION).tar.bz2 $(PRJNAME)-$(VERSION)
> 	rm -f -r $(PRJNAME)-$(VERSION)
> 
> distclean:
> 	rm -f -r $(TARGET) *~ crash
> 
> 
> ------------------------------------------------------------------------
> 
> /*
>  crash shared object for retrieving systemtap buffer
>  Copyright (c) 2007 Hitachi,Ltd.,
>  Created by Satoru Moriya <satoru.moriya.br@hitachi.com>
>  
>  This program is free software; you can redistribute it and/or modify
>  it under the terms of the GNU General Public License as published by
>  the Free Software Foundation; either version 2 of the License, or
>  (at your option) any later version.
> 
>  This program is distributed in the hope that it will be useful,
>  but WITHOUT ANY WARRANTY; without even the implied warranty of
>  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
>  GNU General Public License for more details.
> 
>  You should have received a copy of the GNU General Public License
>  along with this program; if not, write to the Free Software
>  Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
> */
> 
> #include "defs.h"
> 
> #define STPLOG_NO_MOD  -1
> #define STPLOG_NO_SYM  -2
> 
> struct rchan_offsets {
> 	long	subbuf_size;
> 	long	n_subbufs;
> 	long	buf;
> 	long	buf_start;
> 	long	buf_offset;
> 	long	buf_subbufs_produced;
> 	long	buf_padding;
> };
> 
> struct fake_rchan_buf {
> 	void	*start;
> 	size_t	offset;
> 	size_t	subbufs_produced;
> 	size_t	*padding;
> };
> 
> struct fake_rchan {
> 	size_t	subbuf_size;
> 	size_t	n_subbufs;
> };
> 
> struct per_cpu_data {
> 	struct fake_rchan_buf	buf;
> };
> 
> static struct rchan_offsets rchan_offsets;
> static struct fake_rchan chan;
> static struct per_cpu_data per_cpu[NR_CPUS];
> static FILE *outfp;
> static char *subbuf;
> static jmp_buf saved_env;
> static int is_global;
> static int old_format;
> 
> void cmd_systemtaplog(void);
> char *help_systemtaplog[];
> 
> static struct command_table_entry command_table[] = {
> 	{"stplog", cmd_systemtaplog, help_systemtaplog, 0},
> 	{NULL},
> };
> 
> static void cleanup(void)
> {
> 	if (outfp) {
> 		fclose(outfp);
> 		outfp = NULL;
> 	}
> 	if (subbuf) {
> 		free(subbuf);
> 		subbuf = NULL;
> 	}
> }
> 
> static int get_rchan_offsets(void)
> {
> 	rchan_offsets.subbuf_size = MEMBER_OFFSET("rchan", "subbuf_size");
> 	if (rchan_offsets.subbuf_size < 0)
> 		goto ERR;
> 	rchan_offsets.n_subbufs = MEMBER_OFFSET("rchan", "n_subbufs");
> 	if (rchan_offsets.n_subbufs < 0)
> 		goto ERR;
> 	rchan_offsets.buf = MEMBER_OFFSET("rchan", "buf");
> 	if (rchan_offsets.buf < 0)
> 		goto ERR;
> 	rchan_offsets.buf_start = MEMBER_OFFSET("rchan_buf", "start");
> 	if (rchan_offsets.buf_start < 0)
> 		goto ERR;
> 	rchan_offsets.buf_offset = MEMBER_OFFSET("rchan_buf", "offset");
> 	if (rchan_offsets.buf_offset < 0)
> 		goto ERR;
> 	rchan_offsets.buf_subbufs_produced
> 		= MEMBER_OFFSET("rchan_buf", "subbufs_produced");
> 	if (rchan_offsets.buf_subbufs_produced < 0)
> 		goto ERR;
> 	rchan_offsets.buf_padding = MEMBER_OFFSET("rchan_buf", "padding");
> 	if (rchan_offsets.buf_padding < 0)
> 		goto ERR;
> 	return 0;
> ERR:
> 	error(WARNING, "cannot get rchan offset\n");
> 	return -1;
> }
> 
> static ulong get_rchan(ulong chan_addr) 
> {
> 	ulong rchan;
> 
> 	readmem(chan_addr, KVADDR, &rchan, sizeof(void*),
> 		"stp_channel", FAULT_ON_ERROR);
> 	readmem(rchan + rchan_offsets.subbuf_size,
> 		KVADDR, &chan.subbuf_size, sizeof(size_t),
> 		"stp_channel.subbuf_size", FAULT_ON_ERROR);
> 	readmem(rchan + rchan_offsets.n_subbufs,
> 		KVADDR, &chan.n_subbufs, sizeof(size_t),
> 		"stp_channel.n_subbufs", FAULT_ON_ERROR);
> 
> 	return rchan;
> }
> 
> static void get_rchan_buf(int cpu, ulong rchan) 
> {
> 	ulong rchan_buf;
> 	struct per_cpu_data *pcd;
> 
> 	pcd = &per_cpu[cpu];
> 	readmem(rchan + rchan_offsets.buf + sizeof(void*) * cpu,
> 		KVADDR, &rchan_buf, sizeof(void*),
> 		"stp_channel.buf", FAULT_ON_ERROR);
> 	readmem(rchan_buf + rchan_offsets.buf_start,
> 		KVADDR, &pcd->buf.start, sizeof(void*),
> 		"stp_channel.buf.start", FAULT_ON_ERROR);
> 	readmem(rchan_buf + rchan_offsets.buf_offset,
> 		KVADDR, &pcd->buf.offset, sizeof(size_t),
> 		"stp_channel.buf.offset", FAULT_ON_ERROR);
> 	readmem(rchan_buf + rchan_offsets.buf_subbufs_produced,
> 		KVADDR, &pcd->buf.subbufs_produced, sizeof(size_t),
> 		"stp_channel.buf.subbufs_produced", FAULT_ON_ERROR);
> 	readmem(rchan_buf + rchan_offsets.buf_padding,
> 		KVADDR, &pcd->buf.padding, sizeof(size_t*),
> 		"stp_channel.buf.padding", FAULT_ON_ERROR);
> }
> 
> static ulong get_symbol_addr(char *module, char *symbol)
> {
> 	int i;
> 	struct syment *sym, *sym_end;
> 	struct load_module *lm;
> 
>         for (i = 0; i < kt->mods_installed; i++) {
>                 lm = &st->load_modules[i];
>                 if (!STREQ(module, lm->mod_name))
>                         continue;
>                 sym = lm->mod_symtable;
>                 sym_end = lm->mod_symend;		
>                 for ( ; sym <= sym_end; sym++) {
> 			if(STREQ(sym->name, symbol))
> 				return ((ulong)sym->value);
>                 }
> 		error(WARNING, "'%s' doesn't have the symbol named '%s'.\n", 
> 		      module, symbol);
> 		return STPLOG_NO_SYM;
>         }
> 	error(WARNING, "'%s' is not loaded.\n", module);
> 	return STPLOG_NO_MOD;
> }
> 
> static ulong get_rchan_addr(ulong stp_utt_addr)
> {
> 	ulong stp_utt;
> 
> 	readmem(stp_utt_addr, KVADDR, &stp_utt, sizeof(void*),
> 		"stp_utt", FAULT_ON_ERROR);
> 	return (stp_utt + sizeof(int));
> }
> 
> static int check_global_buffer(ulong rchan)
> {
> 	int cpu;
> 	ulong rchan_buf[2];
> 	
> 	for (cpu = 0; cpu < 2; cpu++) {
> 		readmem(rchan + rchan_offsets.buf + sizeof(void*) * cpu,
> 			KVADDR, &rchan_buf[cpu], sizeof(void*),
> 			"stp_channel.buf", FAULT_ON_ERROR);
> 	}
> 	if (rchan_buf[0] == rchan_buf[1])
> 		return 1;
> 	return 0;
> }
> 
> static int setup_global_data(char *module) 
> {
> 	int i;
> 	ulong stp_utt_addr = 0;
> 	ulong stp_rchan_addr = 0;
> 	ulong rchan;
> 
> 	stp_utt_addr = get_symbol_addr(module, "_stp_utt");
> 	if (stp_utt_addr == STPLOG_NO_MOD) {
> 		return -1;
> 	} else if (stp_utt_addr == STPLOG_NO_SYM) {
> 		stp_rchan_addr = get_symbol_addr(module, "_stp_chan");
> 		if (stp_rchan_addr == STPLOG_NO_SYM)
> 			return -1;
> 		old_format = 1;
> 	} else {
> 		stp_rchan_addr = get_rchan_addr(stp_utt_addr);
> 	}
> 	rchan = get_rchan(stp_rchan_addr);
> 	for (i = 0; i < kt->cpus; i++)
> 		get_rchan_buf(i, rchan);
> 
> 	if (kt->cpus > 1) {
> 		is_global = check_global_buffer(rchan);
> 	}
> 	
> 	return 0;
> }
> 
> static int output_cpu_logs(char *module)
> {
> 	int i, max = 256;
> 	struct per_cpu_data *pcd;
> 	size_t n, idx, start, end, ready, len;
> 	unsigned padding;
> 	char fname[max + 1], *source;
> 	DIR *dir;
> 
> 	/* check and create log directory */
> 	dir = opendir(module);
> 	if (dir) {
> 		closedir(dir);
> 	} else {
> 		if (mkdir(module, S_IRWXU) < 0) {
> 			error(WARNING, "cannot create log directory '%s\n'", module);
> 			return -1;
> 		}
> 	}
> 
> 	/* allocate subbuf memory */
> 	subbuf = malloc(chan.subbuf_size);
> 	if (!subbuf) {
> 		error(WARNING, "cannot allocate memory\n");
> 		return -1;
> 	}
> 
> 	fname[max] = '\0';
> 	for (i = 0; i < kt->cpus; i++) {
> 		pcd = &per_cpu[i];
> 		ready = pcd->buf.subbufs_produced + (pcd->buf.offset ? 1 : 0);
> 		if (ready > chan.n_subbufs) {
> 			start = ready % chan.n_subbufs;
> 			end = start + chan.n_subbufs;
> 		} else {
> 			start = 0;
> 			end = ready;
> 		}
> 		/* print information */
> 		fprintf(fp, "--- generating 'cpu%d' ---\n", i);
> 		fprintf(fp, "  subbufs ready on relayfs:%d\n", ready);
> 		fprintf(fp, "    n_subbufs:%d, read from:%d to:%d (offset:%d)\n\n",
> 		       chan.n_subbufs, start, end, pcd->buf.offset);
> 
> 		/* create log file */
> 		snprintf(fname, max, "%s/cpu%d", module, i);
> 		outfp = fopen(fname, "w");
> 		if (!outfp) {
> 			error(WARNING, "cannot create log file '%s'\n", fname);
> 			return -1;
> 		}
> 		for (n = start; n < end; n++) {
> 			/* read relayfs subbufs and write to log file */
> 			idx = n % chan.n_subbufs;
> 			source = pcd->buf.start + idx * chan.subbuf_size;
> 			readmem((ulong)pcd->buf.padding + sizeof(padding) * idx,
> 				KVADDR, &padding, sizeof(padding),
> 				"padding", FAULT_ON_ERROR);
> 			if (n == end - 1 && pcd->buf.offset) {
> 				len = pcd->buf.offset;
> 			} else {
> 				len = chan.subbuf_size;
> 			}			
> 			if (old_format == 1) {
> 				source += sizeof(padding);
> 				len -= sizeof(padding) + padding;
> 			} else {
> 				len -= padding;
> 			}
> 			if (len) {
> 				readmem((ulong)source, KVADDR, subbuf, len,
> 					"subbuf", FAULT_ON_ERROR);
> 				if (fwrite(subbuf, len, 1, outfp) != 1) {
> 					error(WARNING, "cannot write log data\n");
> 					return -1;
> 				}
> 			}
> 		}
> 		fclose(outfp);
> 		outfp = NULL;
> 		if (is_global == 1)
> 			break;
> 	}
> 	return 0;
> }
> 
> static void do_systemtaplog(char *module)
> {
> 	if (setup_global_data(module) < 0)
> 		return;
> 	if (output_cpu_logs(module) < 0)
> 		return;
> }
> 
> void cmd_systemtaplog(void)
> {
> 
> 	int c;
> 	char *module = NULL;
> 
> 	while ((c = getopt(argcnt, args, "m:")) != EOF) {
> 		switch (c) {
> 		case 'm':
> 			module = optarg;
> 			break;
> 		default:
> 			argerrs++;
> 			break;
> 		}
> 	}
> 
> 	if (!module || argerrs)
> 		cmd_usage(pc->curcmd, SYNOPSIS);
> 
> 	saved_env[0] = pc->main_loop_env[0];
> 	if (setjmp(pc->main_loop_env))
> 		goto EXIT;
> 
> 	do_systemtaplog(module);
> EXIT:
> 	cleanup();
> 	pc->main_loop_env[0] = saved_env[0];
> }
> 
> char *help_systemtaplog[] = {
> 	"systemtaplog",
> 	"Retrieve SystemTap log data",
> 	"-m module_name",
> 	"  Retrieve SystemTap's log data and write them to files.\n",
> 	"    -m module_name  All valid SystemTap log data made by the trace",
> 	"                    module which name is 'module_name' are written",
> 	"                    into log files in `module_name` directory. The",
> 	"                    name of each log file is cpu0, cpu1...cpuN. ",
> 	"                    They have same format data as channel buffer",
> 	"                    except padding(This command removes padding). ",
> 	NULL,
> };
> 
> void __attribute__ ((constructor)) systemtaplog_init(void) 
> {
> 	if (get_rchan_offsets() < 0)
> 		return;
> 
> 	register_extension(command_table);
> 	return;
> }
> 
> void __attribute__ ((destructor)) systemtaplog_fini(void)
> {
> 	return;
> }
> 
> 
> ------------------------------------------------------------------------
> 
> --
> Crash-utility mailing list
> Crash-utility@redhat.com
> https://www.redhat.com/mailman/listinfo/crash-utility

Re: [Crash-utility] Re: [RFC] Crash extension for SystemTap

[Satoru MORIYA]

[-- Attachment #1: Type: text/plain, Size: 3252 bytes --]

Hi Dave,

Thank you so much for your review and crash-4.0-4.2.
And I am sorry for delaying my reply for a long time.

I updated my extension for crash-4.0-4.2.


Dave Anderson wrote:
 > The systemtaplog_init() function must be renamed "_init()" in order
 > for it to be called when the dlopen() call is made.  Otherwise,
 > the extend command fails because no commands get registered:

OK, I changed systemtaplog_init() to _init().

 > The only other thing that makes me a little nervous is the temporary
 > relocation of the setjmp buffer.  That was never meant to be used
 > by individual commands -- although nothing prevents it.  I see that it
 > is used to force your cleanup() routine to be called.  And your
 > cleanup() routine exists to free() the subbuf buffer and fclose()
 > the outfp.  A couple suggestions:
 >
 > (1) if you use GETBUF() instead of malloc(), then the buffer will get freed
 >     automatically by restore_sanity() prior to the next command prompt
 > -- even
 >     if you don't do an accompanying FREEBUF().
 > (2) with respect to the outfp file pointer, you could recognize whether it
 >     was left open when the command runs again, and fclose() it then.  Worse
 >     case, a single file pointer would be left open for the remainder of the
 >     crash session.
 >

That's right, and I agreed with you. I'll stop relocation and implement this
part based on your suggestions.

 > [...]So perhaps I could simply allow modules to register an additional
 > command with a new "CLEANUP" flag, which would not show up on the help
 > menu, but would be called during restore_sanity() prior to each
 > command prompt.

Thanks to the new function of crash-4.0-4.2, I can register an cleanup
routine of my extension. And I changed setup_global_data() to use
symbol_value_module() which was introduced in crash-4.0-4.2 instead of
my original function.

Additionally, I changed command option. I removed -m option and added
-o option. I can specify the output directory using -o option.

You can use this extension like this:

Preparation
==============
(A) Build the shared-object(stplog.so).

1. Install crash-devel package.
     $ rpm -ivh crash-devel

2.  $ cd libcrash_for_systemtap

3. Build
     $ make

(B) Make the crash dump which includes SystemTap trace data.
     (*)If you analyze the live system memory, ignore this section.

1. Install kdump
      If you use FC6, see following URL.
      http://fedoraproject.org/wiki/FC6KdumpKexecHowTo?highlight=%28kdump%29

2. Use SystemTap on bulkmode
     $ stap -b foo.stp

3. Panic
     $ echo c > /proc/sysrq-trigger

How to use
==============
1. start crash
     $ crash vmlinux vmcore
     (*) If you analyze the live system memory, you don't have to enter vmcore.

2. load the shared-object
     crash> extend $(WHERE_OBJ_PLACED)/stplog.so

3. retrieve the data
     crash> stplog -o output_dir mod_name

4. You can get output files under output_dir directory.


Output
==============
stplog command makes a file per cpu buffer of relayfs. Of cause, it also
removes the padding bytes in the subbufs of relayfs automatically.

Best Regards,
-- 
---
Satoru MORIYA
Linux Technology Center
Hitachi, Ltd., Systems Development Laboratory
E-mail: satoru.moriya.br@hitachi.com

[-- Attachment #2: stplog.c --]
[-- Type: text/plain, Size: 9390 bytes --]

 /*
 crash shared object for retrieving systemtap buffer
 Copyright (c) 2007 Hitachi,Ltd.,
 Created by Satoru Moriya <satoru.moriya.br@hitachi.com>
 
 This program is free software; you can redistribute it and/or modify
 it under the terms of the GNU General Public License as published by
 the Free Software Foundation; either version 2 of the License, or
 (at your option) any later version.

 This program is distributed in the hope that it will be useful,
 but WITHOUT ANY WARRANTY; without even the implied warranty of
 MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 GNU General Public License for more details.

 You should have received a copy of the GNU General Public License
 along with this program; if not, write to the Free Software
 Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
*/

#include <crash/defs.h>

#define STPLOG_NO_MOD  -1
#define STPLOG_NO_SYM  -2

struct rchan_offsets {
	long	subbuf_size;
	long	n_subbufs;
	long	buf;
	long	buf_start;
	long	buf_offset;
	long	buf_subbufs_produced;
	long	buf_padding;
};

struct fake_rchan_buf {
	void	*start;
	size_t	offset;
	size_t	subbufs_produced;
	size_t	*padding;
};

struct fake_rchan {
	size_t	subbuf_size;
	size_t	n_subbufs;
};

struct per_cpu_data {
	struct fake_rchan_buf	buf;
};

static struct rchan_offsets rchan_offsets;
static struct fake_rchan chan;
static struct per_cpu_data per_cpu[NR_CPUS];
static FILE *outfp;
static char *subbuf;
static int is_global;
static int old_format;

void cmd_stplog(void);
void cmd_stplog_cleanup(void);
char *help_stplog[];
char *help_stplog_cleanup[];

static struct command_table_entry command_table[] = {
	{"stplog", cmd_stplog, help_stplog, 0},
	{"stplog_cleanup", cmd_stplog_cleanup, help_stplog_cleanup, CLEANUP},
	{NULL, NULL, NULL, 0},
};

static void get_rchan_offsets(void)
{
	rchan_offsets.subbuf_size = MEMBER_OFFSET("rchan", "subbuf_size");
	if (rchan_offsets.subbuf_size < 0)
		goto ERR;
	rchan_offsets.n_subbufs = MEMBER_OFFSET("rchan", "n_subbufs");
	if (rchan_offsets.n_subbufs < 0)
		goto ERR;
	rchan_offsets.buf = MEMBER_OFFSET("rchan", "buf");
	if (rchan_offsets.buf < 0)
		goto ERR;
	rchan_offsets.buf_start = MEMBER_OFFSET("rchan_buf", "start");
	if (rchan_offsets.buf_start < 0)
		goto ERR;
	rchan_offsets.buf_offset = MEMBER_OFFSET("rchan_buf", "offset");
	if (rchan_offsets.buf_offset < 0)
		goto ERR;
	rchan_offsets.buf_subbufs_produced
		= MEMBER_OFFSET("rchan_buf", "subbufs_produced");
	if (rchan_offsets.buf_subbufs_produced < 0)
		goto ERR;
	rchan_offsets.buf_padding = MEMBER_OFFSET("rchan_buf", "padding");
	if (rchan_offsets.buf_padding < 0)
		goto ERR;
	return;
ERR:
	error(FATAL, "cannot get rchan offset\n");
}

static ulong get_rchan(ulong chan_addr) 
{
	ulong rchan;

	readmem(chan_addr, KVADDR, &rchan, sizeof(void*),
		"stp_channel", FAULT_ON_ERROR);
	readmem(rchan + rchan_offsets.subbuf_size,
		KVADDR, &chan.subbuf_size, sizeof(size_t),
		"stp_channel.subbuf_size", FAULT_ON_ERROR);
	readmem(rchan + rchan_offsets.n_subbufs,
		KVADDR, &chan.n_subbufs, sizeof(size_t),
		"stp_channel.n_subbufs", FAULT_ON_ERROR);
	return rchan;
}

static void get_rchan_buf(int cpu, ulong rchan) 
{
	ulong rchan_buf;
	struct per_cpu_data *pcd;

	pcd = &per_cpu[cpu];
	readmem(rchan + rchan_offsets.buf + sizeof(void*) * cpu,
		KVADDR, &rchan_buf, sizeof(void*),
		"stp_channel.buf", FAULT_ON_ERROR);
	readmem(rchan_buf + rchan_offsets.buf_start,
		KVADDR, &pcd->buf.start, sizeof(void*),
		"stp_channel.buf.start", FAULT_ON_ERROR);
	readmem(rchan_buf + rchan_offsets.buf_offset,
		KVADDR, &pcd->buf.offset, sizeof(size_t),
		"stp_channel.buf.offset", FAULT_ON_ERROR);
	readmem(rchan_buf + rchan_offsets.buf_subbufs_produced,
		KVADDR, &pcd->buf.subbufs_produced, sizeof(size_t),
		"stp_channel.buf.subbufs_produced", FAULT_ON_ERROR);
	readmem(rchan_buf + rchan_offsets.buf_padding,
		KVADDR, &pcd->buf.padding, sizeof(size_t*),
		"stp_channel.buf.padding", FAULT_ON_ERROR);
	return;
}

static ulong get_rchan_addr(ulong stp_utt_addr)
{
	ulong stp_utt;

	readmem(stp_utt_addr, KVADDR, &stp_utt, sizeof(void*),
		"stp_utt", FAULT_ON_ERROR);
	return (stp_utt + sizeof(int));
}

static int check_global_buffer(ulong rchan)
{
	int cpu;
	ulong rchan_buf[2];
	
	for (cpu = 0; cpu < 2; cpu++) {
		readmem(rchan + rchan_offsets.buf + sizeof(void*) * cpu,
			KVADDR, &rchan_buf[cpu], sizeof(void*),
			"stp_channel.buf", FAULT_ON_ERROR);
	}
	if (rchan_buf[0] == rchan_buf[1])
		return 1;
	return 0;
}

static void setup_global_data(char *module) 
{
	int i;
	ulong stp_utt_addr = 0;
	ulong stp_rchan_addr = 0;
 	ulong rchan;

	stp_utt_addr = symbol_value_module("_stp_utt", module);
	if (stp_utt_addr == 0) {
		stp_rchan_addr = symbol_value_module("_stp_chan", module);
		if (stp_rchan_addr == 0) {
			error(FATAL, "Failed to find _stp_utt/_stp_chan.\n",
			      module);
		}
		old_format = 1;
	} else {
		stp_rchan_addr = get_rchan_addr(stp_utt_addr);
		if (stp_rchan_addr == 0) {
			error(FATAL, "Failed to find _stp_utt/_stp_chan.\n",
			      module);
		}
	}
	rchan = get_rchan(stp_rchan_addr);
	for (i = 0; i < kt->cpus; i++)
		get_rchan_buf(i, rchan);

	if (kt->cpus > 1) {
		is_global = check_global_buffer(rchan);
	}
	return;
}

static void output_cpu_logs(char *filename)
{
	int i, max = 256;
	struct per_cpu_data *pcd;
	size_t n, idx, start, end, ready, len;
	unsigned padding;
	char fname[max + 1], *source;
	DIR *dir;

	/* check and create log directory */
	dir = opendir(filename);
	if (dir) {
		closedir(dir);
	} else {
		if (mkdir(filename, S_IRWXU) < 0) {
			error(FATAL, "cannot create log directory '%s\n'", filename);
		}
	}

	/* allocate subbuf memory */
	subbuf = GETBUF(chan.subbuf_size);
	if (!subbuf) {
		error(FATAL, "cannot allocate memory\n");
	}

	fname[max] = '\0';
	for (i = 0; i < kt->cpus; i++) {
		int adjust = 0;
		pcd = &per_cpu[i];

		if (pcd->buf.offset == 0 || 
		    pcd->buf.offset == chan.subbuf_size + 1) {
			adjust = 0;
		} else {
			adjust = 1;
		}
		ready = pcd->buf.subbufs_produced + adjust;

		if (ready > chan.n_subbufs) {
			start = ready;
			end = start + chan.n_subbufs;
		} else {
			start = 0;
			end = ready;
		}
		/* print information */
		fprintf(fp, "--- generating 'cpu%d' ---\n", i);
		fprintf(fp, "  subbufs ready on relayfs:%ld\n", (long)ready);
		fprintf(fp, "    n_subbufs:%ld, read from:%ld to:%ld (offset:%ld)\n\n",
			(long)chan.n_subbufs, (long)start, (long)end, (long)pcd->buf.offset);

		/* create log file */
		snprintf(fname, max, "%s/cpu%d", filename, i);
		outfp = fopen(fname, "w");
		if (!outfp) {
			error(FATAL, "cannot create log file '%s'\n", fname);
		}
		for (n = start; n < end; n++) {
			/* read relayfs subbufs and write to log file */
			idx = n % chan.n_subbufs;
			source = pcd->buf.start + idx * chan.subbuf_size;
			readmem((ulong)pcd->buf.padding + sizeof(padding) * idx,
				KVADDR, &padding, sizeof(padding),
				"padding", FAULT_ON_ERROR);
			if (n == end - 1 &&  0 < pcd->buf.offset &&
			    pcd->buf.offset < chan.subbuf_size) {
				len = pcd->buf.offset;
			} else {
				len = chan.subbuf_size;
			}			
			if (old_format == 1) {
				source += sizeof(padding);
				len -= sizeof(padding) + padding;
			} else {
				len -= padding;
			}
			if (len) {
				readmem((ulong)source, KVADDR, subbuf, len,
					"subbuf", FAULT_ON_ERROR);
				if (fwrite(subbuf, len, 1, outfp) != 1) {
					error(FATAL, "cannot write log data\n");
				}
			}
		}
		fclose(outfp);
		outfp = NULL;
		if (is_global == 1)
			break;
	}
	if (subbuf) {
		FREEBUF(subbuf);
		subbuf = NULL;
	}
	return;
}

static void do_stplog(char *module, char *filename)
{
	setup_global_data(module);
	output_cpu_logs(filename);
	return;
}

void cmd_stplog(void)
{

	int c;
	char *module = NULL;
	char *filename = NULL;

	while ((c = getopt(argcnt, args, "o:")) != EOF) {
		switch (c) {
		case 'o':
			filename = optarg;
			break;
		default:
			argerrs++;
			break;
		}
	}
	module = args[optind];

	if (!module || argerrs)
		cmd_usage(pc->curcmd, SYNOPSIS);

	if (filename == NULL && module != NULL)
		filename = module;
	do_stplog(module, filename);
	return;
}

void cmd_stplog_cleanup(void)
{
	if (outfp) {
		fclose(outfp);
		outfp = NULL;
	}
	return;
}

char *help_stplog[] = {
	"systemtaplog",
	"Retrieve SystemTap log data",
	"[-o dir_name] module_name",
	"  Retrieve SystemTap's log data and write them to files.\n",
	"    module_name     All valid SystemTap log data made by the trace",
	"                    module which name is 'module_name' are written",
	"                    into log files. If you don't use -o option, the",
	"                    log files are created in `module_name` directory.", 
	"                    The name of each log file is cpu0, cpu1...cpuN. ",
	"                    They have same format data as channel buffer",
	"                    except padding(This command removes padding). ",
	"",
	"    -o file_name    Specify the output directory.",
	NULL,
};

char *help_stplog_cleanup[] = {
	"systemtaplog cleanup (hidden)",
	"Cleanup command for stplog",
	"",
	"  This command is called during restore_sanity() prior to each ",
	"  command prompt to close the files which was opened and failed to",
	"  close by stplog command.",
	NULL,
};

static void __attribute__ ((constructor)) _init(void) 
{
	get_rchan_offsets();
	register_extension(command_table);
	return;
}


static void __attribute__ ((destructor)) _fini(void)
{
	return;
}

[-- Attachment #3: Makefile --]
[-- Type: text/plain, Size: 652 bytes --]

TARGET=stplog.so
CFILE=stplog.c

CFLAGS= -shared -rdynamic
CFLAGS+= -I./crash -Wall

PRJNAME=libcrash_for_systemtap
VERSION=`date +%Y%m%d`

ARCH=$(shell uname -i)

ifeq ($(ARCH), i386)
 CFLAGS += -DX86
else
 ifeq ($(ARCH), ia64)
  CFLAGS += -DIA64
 else
  ifeq ($(ARCH), x86_64)
   CFLAGS += -DX86_64
  endif 
 endif
endif


$(TARGET):$(CFILE)
	echo $(CFLAGS)
	gcc $(CFLAGS) -o $@ $(CFILE)

clean:
	rm -f -r $(TARGET) *~

dist:distclean
	mkdir $(PRJNAME)-$(VERSION)
	cp $(CFILE) Makefile README $(PRJNAME)-$(VERSION)
	tar cvjf $(PRJNAME)-$(VERSION).tar.bz2 $(PRJNAME)-$(VERSION)
	rm -f -r $(PRJNAME)-$(VERSION)

distclean:
	rm -f -r $(TARGET) *~ crash

Re: [RFC] Crash extension for SystemTap

[Frank Ch. Eigler]

Satoru MORIYA <satoru.moriya.br@hitachi.com> writes:

> Thank you so much for your review and crash-4.0-4.2.
> And I am sorry for delaying my reply for a long time.
> I updated my extension for crash-4.0-4.2.
> [...]

Thank you.  Do you think it is ready to be integrated into the
systemtap source tree and build system?


- FChE

Re: [RFC] Crash extension for SystemTap

[Frank Ch. Eigler]

Satoru MORIYA <satoru.moriya.br@hitachi.com> writes:

> [...] I updated my extension for crash-4.0-4.2. [...]

This module is now built as a part of systemtap.  The "staplog.so"
file is installed into $prefix/lib/systemtap, for loading using
crash's "extend" command.

As a part of the import, I took the liberty of renaming it "staplog".
I could not make it print data from a crash session running against
the live kernel [1], so it still needs more work.  It would be nice if
it also learned to operate against the plain, non-"-b", tracing
backend.

- FChE


[1]
% stap -b something.stp &
% sudo crash

crash 4.0-4.3
[...]
      KERNEL: /usr/lib/debug/lib/modules/2.6.21-1.3194.fc7debug/vmlinux
    DUMPFILE: /dev/crash
        CPUS: 4
        DATE: Mon Jul  2 15:34:21 2007
      UPTIME: 20 days, 00:56:36
LOAD AVERAGE: 2.97, 1.62, 1.25
       TASKS: 268
    NODENAME: super.elastic.org
     RELEASE: 2.6.21-1.3194.fc7debug
     VERSION: #1 SMP Wed May 23 22:12:03 EDT 2007
     MACHINE: x86_64  (2659 Mhz)
      MEMORY: 4 GB
[...]
crash> extend ..../staplog.so
..../staplog.so: shared object loaded
crash> staplog stap_96acf0ec9a0b8e844b920b552e0b992a_232
staplog: Failed to find _stp_utt/_stp_chan.
crash> 
crash> quit

Re: [Crash-utility] Re: [RFC] Crash extension for SystemTap

[Satoru MORIYA]

[-- Attachment #1: Type: text/plain, Size: 1142 bytes --]

Hi Frank,

Thank you for including the crash extension in systemtap tree.
And I'm sorry for delaying my reply for a long time.

 > [...]I could not make it print data from a crash session running
 > against the live kernel [...]

I'm Sorry, I didn't test my crash extension on x86_64.
Now I tested it on x86_64 and fixed bugs, so I think you can
use it on x86_64 architecture.

Changes:
  - fix the way of calculating the offset of rchan in struct utt_trace
  - fix the type of padding value in output_cpu_logs()
  - fix the console messages
  - don't make a directory and files if there's no data in relay buffer

I have tested this extention on the following systems:
* FC7,i386,kernel-2.6.22,systemtap-0.5.15,crash-4.0-4.5
* FC7,x86_64,kernel-2.6.21-1.3194.fc7,systemtap-0.5.15,crash-4.0-4.5
* FC7,x86_64,kernel-2.6.21-1.3194.fc7debug,systemtap-0.5.15,crash-4.0-4.5
* RHEL5,i386,kernel-2.6.18-8.el5,systemtap-0.5.15,crash-4.0-4.5
* RHEL5,ia64,kernel-2.6.18-8.el5,systemtap-0.5.15,crash-4.0-4.5

Best regards,

-- 
---
Satoru MORIYA
Linux Technology Center
Hitachi, Ltd., Systems Development Laboratory
E-mail: satoru.moriya.br@hitachi.com

[-- Attachment #2: staplog_fix.patch --]
[-- Type: text/plain, Size: 4814 bytes --]

Index: systemtap/staplog.c
===================================================================
--- systemtap.orig/staplog.c
+++ systemtap/staplog.c
@@ -48,9 +48,6 @@
 
 #include <crash/defs.h>
 
-#define STPLOG_NO_MOD  -1
-#define STPLOG_NO_SYM  -2
-
 struct rchan_offsets {
 	long	subbuf_size;
 	long	n_subbufs;
@@ -167,10 +164,35 @@ static void get_rchan_buf(int cpu, ulong
 static ulong get_rchan_addr(ulong stp_utt_addr)
 {
 	ulong stp_utt;
+	long offset;
 
 	readmem(stp_utt_addr, KVADDR, &stp_utt, sizeof(void*),
 		"stp_utt", FAULT_ON_ERROR);
-	return (stp_utt + sizeof(int));
+
+	/*
+	 * If we couldn't get the member offset of struct utt_trace.rchan,
+	 * i.e. the debuginfo of the trace module isn't available, we use
+	 * sizeof(long) as the offset instead. Currently struct utt_trace
+	 * is defined as below:
+	 *
+	 *     struct utt_trace {
+	 *             int trace_state;
+	 *             struct rchan *rchan;
+	 *             ...
+	 *     }
+	 *
+	 * Although the type of the preceding member is int, sizeof(long)
+	 * is OK, because rchan is aligned with long size on both 32-bit
+	 * and 64-bit environment. When the definision of struct utt_trace
+	 * changed, we must check if this code is correct.
+	 */
+	if ((offset = MEMBER_OFFSET("utt_trace", "rchan")) < 0) {
+		error(WARNING, "The debuginfo of the trace module hasn't been loaded. "
+		      "You may not be able to retrieve the correct trace data.\n");
+		offset = sizeof(long);
+	}
+
+	return (stp_utt + (ulong)offset);
 }
 
 static int check_global_buffer(ulong rchan)
@@ -220,25 +242,15 @@ static void setup_global_data(char *modu
 	return;
 }
 
-static void output_cpu_logs(char *filename)
+static void output_cpu_logs(char *dirname)
 {
 	int i, max = 256;
 	struct per_cpu_data *pcd;
 	size_t n, idx, start, end, ready, len;
-	unsigned padding;
+	size_t padding;
 	char fname[max + 1], *source;
 	DIR *dir;
 
-	/* check and create log directory */
-	dir = opendir(filename);
-	if (dir) {
-		closedir(dir);
-	} else {
-		if (mkdir(filename, S_IRWXU) < 0) {
-			error(FATAL, "cannot create log directory '%s\n'", filename);
-		}
-	}
-
 	/* allocate subbuf memory */
 	subbuf = GETBUF(chan.subbuf_size);
 	if (!subbuf) {
@@ -257,6 +269,15 @@ static void output_cpu_logs(char *filena
 			adjust = 1;
 		}
 		ready = pcd->buf.subbufs_produced + adjust;
+		if (ready == 0) {
+			if (is_global == 1) {
+				error(WARNING, "There is no data in the relay buffer.\n");
+				break;
+			} else {
+				error(WARNING, "[cpu:%d]There is no data in the relay buffer.\n", i);
+				continue;
+			}
+		}
 
 		if (ready > chan.n_subbufs) {
 			start = ready;
@@ -266,13 +287,30 @@ static void output_cpu_logs(char *filena
 			end = ready;
 		}
 		/* print information */
-		fprintf(fp, "--- generating 'cpu%d' ---\n", i);
+		if (is_global == 1) {
+			fprintf(fp, "--- generating 'global' ---\n");
+		} else {
+			fprintf(fp, "--- generating 'cpu%d' ---\n", i);
+		}
 		fprintf(fp, "  subbufs ready on relayfs:%ld\n", (long)ready);
-		fprintf(fp, "    n_subbufs:%ld, read from:%ld to:%ld (offset:%ld)\n\n",
-			(long)chan.n_subbufs, (long)start, (long)end, (long)pcd->buf.offset);
-
-		/* create log file */
-		snprintf(fname, max, "%s/cpu%d", filename, i);
+		fprintf(fp, "  n_subbufs:%ld, read from:%ld to:%ld (offset:%ld)\n\n",
+			(long)chan.n_subbufs, (long)(start ? start - chan.n_subbufs : start),
+			(long)(start ? end - 1 - chan.n_subbufs : end - 1), (long)pcd->buf.offset);
+
+		/* create log dir and file */
+		dir = opendir(dirname);
+		if (dir) {
+			closedir(dir);
+		} else {
+			if (mkdir(dirname, S_IRWXU) < 0) {
+				error(FATAL, "cannot create log directory '%s\n'", dirname);
+			}
+		}
+		if (is_global == 1) {
+			snprintf(fname, max, "%s/global", dirname, i);
+		} else {
+			snprintf(fname, max, "%s/cpu%d", dirname, i);
+		}
 		outfp = fopen(fname, "w");
 		if (!outfp) {
 			error(FATAL, "cannot create log file '%s'\n", fname);
@@ -316,10 +354,10 @@ static void output_cpu_logs(char *filena
 	return;
 }
 
-static void do_staplog(char *module, char *filename)
+static void do_staplog(char *module, char *dirname)
 {
 	setup_global_data(module);
-	output_cpu_logs(filename);
+	output_cpu_logs(dirname);
 	return;
 }
 
@@ -328,12 +366,12 @@ void cmd_staplog(void)
 
 	int c;
 	char *module = NULL;
-	char *filename = NULL;
+	char *dirname = NULL;
 
 	while ((c = getopt(argcnt, args, "o:")) != EOF) {
 		switch (c) {
 		case 'o':
-			filename = optarg;
+			dirname = optarg;
 			break;
 		default:
 			argerrs++;
@@ -345,9 +383,9 @@ void cmd_staplog(void)
 	if (!module || argerrs)
 		cmd_usage(pc->curcmd, SYNOPSIS);
 
-	if (filename == NULL && module != NULL)
-		filename = module;
-	do_staplog(module, filename);
+	if (dirname == NULL && module != NULL)
+		dirname = module;
+	do_staplog(module, dirname);
 	return;
 }
 

Re: [RFC] Crash extension for SystemTap

[Frank Ch. Eigler]

Satoru MORIYA <satoru.moriya.br@hitachi.com> writes:

>[...]
> I'm Sorry, I didn't test my crash extension on x86_64.
> Now I tested it on x86_64 and fixed bugs, so I think you can
> use it on x86_64 architecture.

Thanks, I committed your changes.  Please feel free to directly
maintain this source file in the cvs tree yourself, including
ChangeLog entries for them.  (If needed, first apply for CVS-write
access at the systemtap "get involved" web pages.)

- FChE