* [RFC] Crash extension for SystemTap @ 2007-05-23 11:46 Satoru MORIYA 2007-05-23 13:22 ` [Crash-utility] " Dave Anderson ` (2 more replies) 0 siblings, 3 replies; 10+ messages in thread From: Satoru MORIYA @ 2007-05-23 11:46 UTC (permalink / raw) To: systemtap, crash-utility Cc: yumiko.sugita.yf, masami.hiramatsu.pt, soshima, haoki Hi, Here is an extension(shared object) of the crash to retrieve the trace data of systemtap scripts. I'd like to analyze what caused the kernel panic by using the systemtap. However, currently the systemtap's trace data can't be retrieved from a dumped image easily. So, I developed a crash's extension which retrieves the data recorded by systemtap from the dumped image. Here is a brief document of this extension. This extension supports the new utt-based buffer as well as the bulk-mode buffer of old systemtap module. I have tested this extention on the following system. * FC6, i386, kernel-2.6.21, systemtap-0.5.14, crash-4.0-1.1 * FC6, i386, kernel-2.6.20, systemtap-0.5.13/14, crash-4.0-1.1 * RHEL5, i386, kernel-2.6.18-8.el5, systemtap-0.5.12, crash-4.0-3.14 Preparation ============== (A) Build the shared-object(stplog.so). 1. Put Makefile and stplog.c into a directory ($DIR) $ cd $DIR 2. Make the symbolic link to the crash source code directory $ ln -s $WHERE_CRASH_PLACED crash 3. Build $ make (B) Make the crash dump which includes SystemTap trace data. (*)If you analyze the live system memory, ignore this section. 1. Install kdump If you use FC6, see following URL. http://fedoraproject.org/wiki/FC6KdumpKexecHowTo?highlight=%28kdump%29 2. Use SystemTap $ stap foo.stp 3. Panic $ echo c > /proc/sysrq-trigger How to use ============== 1. start crash $ crash vmlinux vmcore (*) If you analyze the live system memory, you don't need "vmcore". $ crash vmlinux 2. load the shared-object crash> extend $(WHERE_OBJ_PLACED)/stplog.so 3. retrieve the data crash> stplog -m <mod_name> (*) <mod_name> is the name of trace module from which you retrieve data. 4. You can get output files under the directory whose name is <mod_name>. Output ============== stplog command makes a file per channel buffer of relayfs(equivalent to per cpu). And it also removes padding bytes. I believe this command is very useful for system administrators if they monitor their systems with SystemTap. Best Regards, --- Satoru MORIYA Linux Technology Center Hitachi, Ltd., Systems Development Laboratory E-mail: satoru.moriya.br@hitachi.com ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [Crash-utility] [RFC] Crash extension for SystemTap 2007-05-23 11:46 [RFC] Crash extension for SystemTap Satoru MORIYA @ 2007-05-23 13:22 ` Dave Anderson 2007-05-23 19:01 ` Frank Eigler 2007-05-24 0:54 ` Satoru MORIYA 2 siblings, 0 replies; 10+ messages in thread From: Dave Anderson @ 2007-05-23 13:22 UTC (permalink / raw) To: Discussion list for crash utility usage, maintenance and development Cc: systemtap, yumiko.sugita.yf Satoru MORIYA wrote: > Hi, > > Here is an extension(shared object) of the crash to retrieve the trace > data of systemtap scripts. > Hi Satoru, I think you also meant to attach the stplog.c (and its own Makefile?) to your post? Is the format of the trace data used by systemtap always the same? I.e., is it always a kernel buffer filled with ASCII data? You mention that the command makes a file in a subdirectory of the running crash session. Wouldn't it be more flexible to dump the output to the terminal? And then if you want to save it to a file, just do something like this: crash> stplog -m mod_name > outputfile Anyway, I think it's time that I create a repository for extensions on my people site... Thanks, Dave > I'd like to analyze what caused the kernel panic by using the systemtap. > However, currently the systemtap's trace data can't be retrieved from a > dumped image easily. So, I developed a crash's extension which retrieves > the data recorded by systemtap from the dumped image. > Here is a brief document of this extension. This extension supports the new > utt-based buffer as well as the bulk-mode buffer of old systemtap module. > > I have tested this extention on the following system. > * FC6, i386, kernel-2.6.21, systemtap-0.5.14, crash-4.0-1.1 > * FC6, i386, kernel-2.6.20, systemtap-0.5.13/14, crash-4.0-1.1 > * RHEL5, i386, kernel-2.6.18-8.el5, systemtap-0.5.12, crash-4.0-3.14 > > > Preparation > ============== > (A) Build the shared-object(stplog.so). > > 1. Put Makefile and stplog.c into a directory ($DIR) > $ cd $DIR > > 2. Make the symbolic link to the crash source code directory > $ ln -s $WHERE_CRASH_PLACED crash > > 3. Build > $ make > > (B) Make the crash dump which includes SystemTap trace data. > (*)If you analyze the live system memory, ignore this section. > > 1. Install kdump > If you use FC6, see following URL. > http://fedoraproject.org/wiki/FC6KdumpKexecHowTo?highlight=%28kdump%29 > > 2. Use SystemTap > $ stap foo.stp > > 3. Panic > $ echo c > /proc/sysrq-trigger > > How to use > ============== > 1. start crash > $ crash vmlinux vmcore > (*) If you analyze the live system memory, you don't need "vmcore". > $ crash vmlinux > > 2. load the shared-object > crash> extend $(WHERE_OBJ_PLACED)/stplog.so > > 3. retrieve the data > crash> stplog -m <mod_name> > (*) <mod_name> is the name of trace module from which you retrieve data. > > 4. You can get output files under the directory whose name is <mod_name>. > > Output > ============== > stplog command makes a file per channel buffer of relayfs(equivalent to per cpu). > And it also removes padding bytes. > > > I believe this command is very useful for system administrators > if they monitor their systems with SystemTap. > > Best Regards, > > --- > Satoru MORIYA > Linux Technology Center > Hitachi, Ltd., Systems Development Laboratory > E-mail: satoru.moriya.br@hitachi.com > > -- > Crash-utility mailing list > Crash-utility@redhat.com > https://www.redhat.com/mailman/listinfo/crash-utility ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [RFC] Crash extension for SystemTap 2007-05-23 11:46 [RFC] Crash extension for SystemTap Satoru MORIYA 2007-05-23 13:22 ` [Crash-utility] " Dave Anderson @ 2007-05-23 19:01 ` Frank Eigler 2007-05-24 0:54 ` Satoru MORIYA 2 siblings, 0 replies; 10+ messages in thread From: Frank Eigler @ 2007-05-23 19:01 UTC (permalink / raw) To: Discussion list for crash utility usage, maintenance and development Cc: systemtap, yumiko.sugita.yf Satoru MORIYA <satoru.moriya.br@hitachi.com> writes: > [...] > (A) Build the shared-object(stplog.so). > 1. Put Makefile and stplog.c into a directory ($DIR) > $ cd $DIR > 2. Make the symbolic link to the crash source code directory > $ ln -s $WHERE_CRASH_PLACED crash It would be nice if crash(8) had a -devel package against which such an extension could be built. I submitted RH bug #241045 to try to fix this. If that is done, we can include the stplog sources in systemtap, and build a .so as a part of our build. > [...] I believe this command is very useful for system > administrators if they monitor their systems with SystemTap. Indeed! Thank you. - FChE ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [RFC] Crash extension for SystemTap 2007-05-23 11:46 [RFC] Crash extension for SystemTap Satoru MORIYA 2007-05-23 13:22 ` [Crash-utility] " Dave Anderson 2007-05-23 19:01 ` Frank Eigler @ 2007-05-24 0:54 ` Satoru MORIYA 2007-05-24 14:44 ` [Crash-utility] " Dave Anderson 2 siblings, 1 reply; 10+ messages in thread From: Satoru MORIYA @ 2007-05-24 0:54 UTC (permalink / raw) To: Satoru MORIYA Cc: systemtap, crash-utility, yumiko.sugita.yf, masami.hiramatsu.pt, soshima, haoki [-- Attachment #1: Type: text/plain, Size: 2608 bytes --] Hi, Sorry, I forgot to attach my source code. Here is the source code. Satoru MORIYA wrote: > Hi, > > Here is an extension(shared object) of the crash to retrieve the trace > data of systemtap scripts. > > I'd like to analyze what caused the kernel panic by using the systemtap. > However, currently the systemtap's trace data can't be retrieved from a > dumped image easily. So, I developed a crash's extension which retrieves > the data recorded by systemtap from the dumped image. > Here is a brief document of this extension. This extension supports the new > utt-based buffer as well as the bulk-mode buffer of old systemtap module. > > I have tested this extention on the following system. > * FC6, i386, kernel-2.6.21, systemtap-0.5.14, crash-4.0-1.1 > * FC6, i386, kernel-2.6.20, systemtap-0.5.13/14, crash-4.0-1.1 > * RHEL5, i386, kernel-2.6.18-8.el5, systemtap-0.5.12, crash-4.0-3.14 > > > Preparation > ============== > (A) Build the shared-object(stplog.so). > > 1. Put Makefile and stplog.c into a directory ($DIR) > $ cd $DIR > > 2. Make the symbolic link to the crash source code directory > $ ln -s $WHERE_CRASH_PLACED crash > > 3. Build > $ make > > (B) Make the crash dump which includes SystemTap trace data. > (*)If you analyze the live system memory, ignore this section. > > 1. Install kdump > If you use FC6, see following URL. > http://fedoraproject.org/wiki/FC6KdumpKexecHowTo?highlight=%28kdump%29 > > 2. Use SystemTap > $ stap foo.stp > > 3. Panic > $ echo c > /proc/sysrq-trigger > > How to use > ============== > 1. start crash > $ crash vmlinux vmcore > (*) If you analyze the live system memory, you don't need "vmcore". > $ crash vmlinux > > 2. load the shared-object > crash> extend $(WHERE_OBJ_PLACED)/stplog.so > > 3. retrieve the data > crash> stplog -m <mod_name> > (*) <mod_name> is the name of trace module from which you retrieve data. > > 4. You can get output files under the directory whose name is <mod_name>. > > Output > ============== > stplog command makes a file per channel buffer of relayfs(equivalent to per cpu). > And it also removes padding bytes. > > > I believe this command is very useful for system administrators > if they monitor their systems with SystemTap. > > Best Regards, > > --- > Satoru MORIYA > Linux Technology Center > Hitachi, Ltd., Systems Development Laboratory > E-mail: satoru.moriya.br@hitachi.com > > > -- --- Satoru MORIYA Linux Technology Center Hitachi, Ltd., Systems Development Laboratory E-mail: satoru.moriya.br@hitachi.com [-- Attachment #2: Makefile --] [-- Type: text/plain, Size: 458 bytes --] TARGET=stplog.so CFILE=stplog.c CFLAGS= -shared -rdynamic -DX86 CFLAGS+= -I./crash -Wall PRJNAME=libcrash_for_systemtap VERSION=`date +%Y%m%d` $(TARGET):$(CFILE) gcc $(CFLAGS) -o $@ $(CFILE) clean: rm -f -r $(TARGET) *~ dist:distclean mkdir $(PRJNAME)-$(VERSION) cp $(CFILE) Makefile README $(PRJNAME)-$(VERSION) tar cvjf $(PRJNAME)-$(VERSION).tar.bz2 $(PRJNAME)-$(VERSION) rm -f -r $(PRJNAME)-$(VERSION) distclean: rm -f -r $(TARGET) *~ crash [-- Attachment #3: stplog.c --] [-- Type: text/plain, Size: 9402 bytes --] /* crash shared object for retrieving systemtap buffer Copyright (c) 2007 Hitachi,Ltd., Created by Satoru Moriya <satoru.moriya.br@hitachi.com> This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA */ #include "defs.h" #define STPLOG_NO_MOD -1 #define STPLOG_NO_SYM -2 struct rchan_offsets { long subbuf_size; long n_subbufs; long buf; long buf_start; long buf_offset; long buf_subbufs_produced; long buf_padding; }; struct fake_rchan_buf { void *start; size_t offset; size_t subbufs_produced; size_t *padding; }; struct fake_rchan { size_t subbuf_size; size_t n_subbufs; }; struct per_cpu_data { struct fake_rchan_buf buf; }; static struct rchan_offsets rchan_offsets; static struct fake_rchan chan; static struct per_cpu_data per_cpu[NR_CPUS]; static FILE *outfp; static char *subbuf; static jmp_buf saved_env; static int is_global; static int old_format; void cmd_systemtaplog(void); char *help_systemtaplog[]; static struct command_table_entry command_table[] = { {"stplog", cmd_systemtaplog, help_systemtaplog, 0}, {NULL}, }; static void cleanup(void) { if (outfp) { fclose(outfp); outfp = NULL; } if (subbuf) { free(subbuf); subbuf = NULL; } } static int get_rchan_offsets(void) { rchan_offsets.subbuf_size = MEMBER_OFFSET("rchan", "subbuf_size"); if (rchan_offsets.subbuf_size < 0) goto ERR; rchan_offsets.n_subbufs = MEMBER_OFFSET("rchan", "n_subbufs"); if (rchan_offsets.n_subbufs < 0) goto ERR; rchan_offsets.buf = MEMBER_OFFSET("rchan", "buf"); if (rchan_offsets.buf < 0) goto ERR; rchan_offsets.buf_start = MEMBER_OFFSET("rchan_buf", "start"); if (rchan_offsets.buf_start < 0) goto ERR; rchan_offsets.buf_offset = MEMBER_OFFSET("rchan_buf", "offset"); if (rchan_offsets.buf_offset < 0) goto ERR; rchan_offsets.buf_subbufs_produced = MEMBER_OFFSET("rchan_buf", "subbufs_produced"); if (rchan_offsets.buf_subbufs_produced < 0) goto ERR; rchan_offsets.buf_padding = MEMBER_OFFSET("rchan_buf", "padding"); if (rchan_offsets.buf_padding < 0) goto ERR; return 0; ERR: error(WARNING, "cannot get rchan offset\n"); return -1; } static ulong get_rchan(ulong chan_addr) { ulong rchan; readmem(chan_addr, KVADDR, &rchan, sizeof(void*), "stp_channel", FAULT_ON_ERROR); readmem(rchan + rchan_offsets.subbuf_size, KVADDR, &chan.subbuf_size, sizeof(size_t), "stp_channel.subbuf_size", FAULT_ON_ERROR); readmem(rchan + rchan_offsets.n_subbufs, KVADDR, &chan.n_subbufs, sizeof(size_t), "stp_channel.n_subbufs", FAULT_ON_ERROR); return rchan; } static void get_rchan_buf(int cpu, ulong rchan) { ulong rchan_buf; struct per_cpu_data *pcd; pcd = &per_cpu[cpu]; readmem(rchan + rchan_offsets.buf + sizeof(void*) * cpu, KVADDR, &rchan_buf, sizeof(void*), "stp_channel.buf", FAULT_ON_ERROR); readmem(rchan_buf + rchan_offsets.buf_start, KVADDR, &pcd->buf.start, sizeof(void*), "stp_channel.buf.start", FAULT_ON_ERROR); readmem(rchan_buf + rchan_offsets.buf_offset, KVADDR, &pcd->buf.offset, sizeof(size_t), "stp_channel.buf.offset", FAULT_ON_ERROR); readmem(rchan_buf + rchan_offsets.buf_subbufs_produced, KVADDR, &pcd->buf.subbufs_produced, sizeof(size_t), "stp_channel.buf.subbufs_produced", FAULT_ON_ERROR); readmem(rchan_buf + rchan_offsets.buf_padding, KVADDR, &pcd->buf.padding, sizeof(size_t*), "stp_channel.buf.padding", FAULT_ON_ERROR); } static ulong get_symbol_addr(char *module, char *symbol) { int i; struct syment *sym, *sym_end; struct load_module *lm; for (i = 0; i < kt->mods_installed; i++) { lm = &st->load_modules[i]; if (!STREQ(module, lm->mod_name)) continue; sym = lm->mod_symtable; sym_end = lm->mod_symend; for ( ; sym <= sym_end; sym++) { if(STREQ(sym->name, symbol)) return ((ulong)sym->value); } error(WARNING, "'%s' doesn't have the symbol named '%s'.\n", module, symbol); return STPLOG_NO_SYM; } error(WARNING, "'%s' is not loaded.\n", module); return STPLOG_NO_MOD; } static ulong get_rchan_addr(ulong stp_utt_addr) { ulong stp_utt; readmem(stp_utt_addr, KVADDR, &stp_utt, sizeof(void*), "stp_utt", FAULT_ON_ERROR); return (stp_utt + sizeof(int)); } static int check_global_buffer(ulong rchan) { int cpu; ulong rchan_buf[2]; for (cpu = 0; cpu < 2; cpu++) { readmem(rchan + rchan_offsets.buf + sizeof(void*) * cpu, KVADDR, &rchan_buf[cpu], sizeof(void*), "stp_channel.buf", FAULT_ON_ERROR); } if (rchan_buf[0] == rchan_buf[1]) return 1; return 0; } static int setup_global_data(char *module) { int i; ulong stp_utt_addr = 0; ulong stp_rchan_addr = 0; ulong rchan; stp_utt_addr = get_symbol_addr(module, "_stp_utt"); if (stp_utt_addr == STPLOG_NO_MOD) { return -1; } else if (stp_utt_addr == STPLOG_NO_SYM) { stp_rchan_addr = get_symbol_addr(module, "_stp_chan"); if (stp_rchan_addr == STPLOG_NO_SYM) return -1; old_format = 1; } else { stp_rchan_addr = get_rchan_addr(stp_utt_addr); } rchan = get_rchan(stp_rchan_addr); for (i = 0; i < kt->cpus; i++) get_rchan_buf(i, rchan); if (kt->cpus > 1) { is_global = check_global_buffer(rchan); } return 0; } static int output_cpu_logs(char *module) { int i, max = 256; struct per_cpu_data *pcd; size_t n, idx, start, end, ready, len; unsigned padding; char fname[max + 1], *source; DIR *dir; /* check and create log directory */ dir = opendir(module); if (dir) { closedir(dir); } else { if (mkdir(module, S_IRWXU) < 0) { error(WARNING, "cannot create log directory '%s\n'", module); return -1; } } /* allocate subbuf memory */ subbuf = malloc(chan.subbuf_size); if (!subbuf) { error(WARNING, "cannot allocate memory\n"); return -1; } fname[max] = '\0'; for (i = 0; i < kt->cpus; i++) { pcd = &per_cpu[i]; ready = pcd->buf.subbufs_produced + (pcd->buf.offset ? 1 : 0); if (ready > chan.n_subbufs) { start = ready % chan.n_subbufs; end = start + chan.n_subbufs; } else { start = 0; end = ready; } /* print information */ fprintf(fp, "--- generating 'cpu%d' ---\n", i); fprintf(fp, " subbufs ready on relayfs:%d\n", ready); fprintf(fp, " n_subbufs:%d, read from:%d to:%d (offset:%d)\n\n", chan.n_subbufs, start, end, pcd->buf.offset); /* create log file */ snprintf(fname, max, "%s/cpu%d", module, i); outfp = fopen(fname, "w"); if (!outfp) { error(WARNING, "cannot create log file '%s'\n", fname); return -1; } for (n = start; n < end; n++) { /* read relayfs subbufs and write to log file */ idx = n % chan.n_subbufs; source = pcd->buf.start + idx * chan.subbuf_size; readmem((ulong)pcd->buf.padding + sizeof(padding) * idx, KVADDR, &padding, sizeof(padding), "padding", FAULT_ON_ERROR); if (n == end - 1 && pcd->buf.offset) { len = pcd->buf.offset; } else { len = chan.subbuf_size; } if (old_format == 1) { source += sizeof(padding); len -= sizeof(padding) + padding; } else { len -= padding; } if (len) { readmem((ulong)source, KVADDR, subbuf, len, "subbuf", FAULT_ON_ERROR); if (fwrite(subbuf, len, 1, outfp) != 1) { error(WARNING, "cannot write log data\n"); return -1; } } } fclose(outfp); outfp = NULL; if (is_global == 1) break; } return 0; } static void do_systemtaplog(char *module) { if (setup_global_data(module) < 0) return; if (output_cpu_logs(module) < 0) return; } void cmd_systemtaplog(void) { int c; char *module = NULL; while ((c = getopt(argcnt, args, "m:")) != EOF) { switch (c) { case 'm': module = optarg; break; default: argerrs++; break; } } if (!module || argerrs) cmd_usage(pc->curcmd, SYNOPSIS); saved_env[0] = pc->main_loop_env[0]; if (setjmp(pc->main_loop_env)) goto EXIT; do_systemtaplog(module); EXIT: cleanup(); pc->main_loop_env[0] = saved_env[0]; } char *help_systemtaplog[] = { "systemtaplog", "Retrieve SystemTap log data", "-m module_name", " Retrieve SystemTap's log data and write them to files.\n", " -m module_name All valid SystemTap log data made by the trace", " module which name is 'module_name' are written", " into log files in `module_name` directory. The", " name of each log file is cpu0, cpu1...cpuN. ", " They have same format data as channel buffer", " except padding(This command removes padding). ", NULL, }; void __attribute__ ((constructor)) systemtaplog_init(void) { if (get_rchan_offsets() < 0) return; register_extension(command_table); return; } void __attribute__ ((destructor)) systemtaplog_fini(void) { return; } ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [Crash-utility] Re: [RFC] Crash extension for SystemTap 2007-05-24 0:54 ` Satoru MORIYA @ 2007-05-24 14:44 ` Dave Anderson 2007-06-22 13:43 ` Satoru MORIYA 0 siblings, 1 reply; 10+ messages in thread From: Dave Anderson @ 2007-05-24 14:44 UTC (permalink / raw) To: Discussion list for crash utility usage, maintenance and development Cc: Satoru MORIYA, systemtap, yumiko.sugita.yf Satoru MORIYA wrote: > Hi, > > Sorry, I forgot to attach my source code. > Here is the source code. > Hi Satoru, This looks really good for the most part. I do have a few suggestions. The systemtaplog_init() function must be renamed "_init()" in order for it to be called when the dlopen() call is made. Otherwise, the extend command fails because no commands get registered: crash> extend stplog.so extend: ./stplog.so: no commands registered: shared object unloaded crash> The only other thing that makes me a little nervous is the temporary relocation of the setjmp buffer. That was never meant to be used by individual commands -- although nothing prevents it. I see that it is used to force your cleanup() routine to be called. And your cleanup() routine exists to free() the subbuf buffer and fclose() the outfp. A couple suggestions: (1) if you use GETBUF() instead of malloc(), then the buffer will get freed automatically by restore_sanity() prior to the next command prompt -- even if you don't do an accompanying FREEBUF(). (2) with respect to the outfp file pointer, you could recognize whether it was left open when the command runs again, and fclose() it then. Worse case, a single file pointer would be left open for the remainder of the crash session. I guess that I never expected an extension file would ever do any file opening/closing, and so I presumed that the current crash-state restoration mechanism would suffice. I now see that it would make sense to allow the registration of some kind of cleanup() function for each extension module. But I also don't want to break any backwards-compatibility with older extension modules. So perhaps I could simply allow modules to register an additional command with a new "CLEANUP" flag, which would not show up on the help menu, but would be called during restore_sanity() prior to each command prompt. Also, I really like your get_symbol_addr() function, which restricts the symbol search to a specific module. Sometimes there are symbol name collisions between the base kernel symbols and the various module symbols, and the symbol_search() function returns the first one found, typically base kernel version. You could use get_syment_array() to get an array of symbols with the same name, and then parse the list for the one belonging to your module, but having a function that targets a specific module makes more sense in your case. So I'm going to add a new function in symbols.c that does the same kind of thing as your get_symbol_addr() function. Anyway, aside from the requirement to rename systemtaplog_init(), you can do whatever you want -- it's your module! Thanks, Dave > Satoru MORIYA wrote: > >> Hi, >> >> Here is an extension(shared object) of the crash to retrieve the trace >> data of systemtap scripts. >> >> I'd like to analyze what caused the kernel panic by using the systemtap. >> However, currently the systemtap's trace data can't be retrieved from a >> dumped image easily. So, I developed a crash's extension which retrieves >> the data recorded by systemtap from the dumped image. >> Here is a brief document of this extension. This extension supports >> the new >> utt-based buffer as well as the bulk-mode buffer of old systemtap module. >> >> I have tested this extention on the following system. >> * FC6, i386, kernel-2.6.21, systemtap-0.5.14, crash-4.0-1.1 >> * FC6, i386, kernel-2.6.20, systemtap-0.5.13/14, crash-4.0-1.1 >> * RHEL5, i386, kernel-2.6.18-8.el5, systemtap-0.5.12, crash-4.0-3.14 >> >> >> Preparation >> ============== >> (A) Build the shared-object(stplog.so). >> >> 1. Put Makefile and stplog.c into a directory ($DIR) >> $ cd $DIR >> >> 2. Make the symbolic link to the crash source code directory >> $ ln -s $WHERE_CRASH_PLACED crash >> >> 3. Build >> $ make >> >> (B) Make the crash dump which includes SystemTap trace data. >> (*)If you analyze the live system memory, ignore this section. >> >> 1. Install kdump >> If you use FC6, see following URL. >> >> http://fedoraproject.org/wiki/FC6KdumpKexecHowTo?highlight=%28kdump%29 >> >> 2. Use SystemTap >> $ stap foo.stp >> >> 3. Panic >> $ echo c > /proc/sysrq-trigger >> >> How to use >> ============== >> 1. start crash >> $ crash vmlinux vmcore >> (*) If you analyze the live system memory, you don't need "vmcore". >> $ crash vmlinux >> >> 2. load the shared-object >> crash> extend $(WHERE_OBJ_PLACED)/stplog.so >> >> 3. retrieve the data >> crash> stplog -m <mod_name> >> (*) <mod_name> is the name of trace module from which you retrieve >> data. >> >> 4. You can get output files under the directory whose name is <mod_name>. >> >> Output >> ============== >> stplog command makes a file per channel buffer of relayfs(equivalent >> to per cpu). >> And it also removes padding bytes. >> >> >> I believe this command is very useful for system administrators >> if they monitor their systems with SystemTap. >> >> Best Regards, >> >> --- >> Satoru MORIYA >> Linux Technology Center >> Hitachi, Ltd., Systems Development Laboratory >> E-mail: satoru.moriya.br@hitachi.com >> >> >> > > > ------------------------------------------------------------------------ > > TARGET=stplog.so > CFILE=stplog.c > > CFLAGS= -shared -rdynamic -DX86 > CFLAGS+= -I./crash -Wall > > PRJNAME=libcrash_for_systemtap > VERSION=`date +%Y%m%d` > > $(TARGET):$(CFILE) > gcc $(CFLAGS) -o $@ $(CFILE) > > clean: > rm -f -r $(TARGET) *~ > > dist:distclean > mkdir $(PRJNAME)-$(VERSION) > cp $(CFILE) Makefile README $(PRJNAME)-$(VERSION) > tar cvjf $(PRJNAME)-$(VERSION).tar.bz2 $(PRJNAME)-$(VERSION) > rm -f -r $(PRJNAME)-$(VERSION) > > distclean: > rm -f -r $(TARGET) *~ crash > > > ------------------------------------------------------------------------ > > /* > crash shared object for retrieving systemtap buffer > Copyright (c) 2007 Hitachi,Ltd., > Created by Satoru Moriya <satoru.moriya.br@hitachi.com> > > This program is free software; you can redistribute it and/or modify > it under the terms of the GNU General Public License as published by > the Free Software Foundation; either version 2 of the License, or > (at your option) any later version. > > This program is distributed in the hope that it will be useful, > but WITHOUT ANY WARRANTY; without even the implied warranty of > MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > GNU General Public License for more details. > > You should have received a copy of the GNU General Public License > along with this program; if not, write to the Free Software > Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA > */ > > #include "defs.h" > > #define STPLOG_NO_MOD -1 > #define STPLOG_NO_SYM -2 > > struct rchan_offsets { > long subbuf_size; > long n_subbufs; > long buf; > long buf_start; > long buf_offset; > long buf_subbufs_produced; > long buf_padding; > }; > > struct fake_rchan_buf { > void *start; > size_t offset; > size_t subbufs_produced; > size_t *padding; > }; > > struct fake_rchan { > size_t subbuf_size; > size_t n_subbufs; > }; > > struct per_cpu_data { > struct fake_rchan_buf buf; > }; > > static struct rchan_offsets rchan_offsets; > static struct fake_rchan chan; > static struct per_cpu_data per_cpu[NR_CPUS]; > static FILE *outfp; > static char *subbuf; > static jmp_buf saved_env; > static int is_global; > static int old_format; > > void cmd_systemtaplog(void); > char *help_systemtaplog[]; > > static struct command_table_entry command_table[] = { > {"stplog", cmd_systemtaplog, help_systemtaplog, 0}, > {NULL}, > }; > > static void cleanup(void) > { > if (outfp) { > fclose(outfp); > outfp = NULL; > } > if (subbuf) { > free(subbuf); > subbuf = NULL; > } > } > > static int get_rchan_offsets(void) > { > rchan_offsets.subbuf_size = MEMBER_OFFSET("rchan", "subbuf_size"); > if (rchan_offsets.subbuf_size < 0) > goto ERR; > rchan_offsets.n_subbufs = MEMBER_OFFSET("rchan", "n_subbufs"); > if (rchan_offsets.n_subbufs < 0) > goto ERR; > rchan_offsets.buf = MEMBER_OFFSET("rchan", "buf"); > if (rchan_offsets.buf < 0) > goto ERR; > rchan_offsets.buf_start = MEMBER_OFFSET("rchan_buf", "start"); > if (rchan_offsets.buf_start < 0) > goto ERR; > rchan_offsets.buf_offset = MEMBER_OFFSET("rchan_buf", "offset"); > if (rchan_offsets.buf_offset < 0) > goto ERR; > rchan_offsets.buf_subbufs_produced > = MEMBER_OFFSET("rchan_buf", "subbufs_produced"); > if (rchan_offsets.buf_subbufs_produced < 0) > goto ERR; > rchan_offsets.buf_padding = MEMBER_OFFSET("rchan_buf", "padding"); > if (rchan_offsets.buf_padding < 0) > goto ERR; > return 0; > ERR: > error(WARNING, "cannot get rchan offset\n"); > return -1; > } > > static ulong get_rchan(ulong chan_addr) > { > ulong rchan; > > readmem(chan_addr, KVADDR, &rchan, sizeof(void*), > "stp_channel", FAULT_ON_ERROR); > readmem(rchan + rchan_offsets.subbuf_size, > KVADDR, &chan.subbuf_size, sizeof(size_t), > "stp_channel.subbuf_size", FAULT_ON_ERROR); > readmem(rchan + rchan_offsets.n_subbufs, > KVADDR, &chan.n_subbufs, sizeof(size_t), > "stp_channel.n_subbufs", FAULT_ON_ERROR); > > return rchan; > } > > static void get_rchan_buf(int cpu, ulong rchan) > { > ulong rchan_buf; > struct per_cpu_data *pcd; > > pcd = &per_cpu[cpu]; > readmem(rchan + rchan_offsets.buf + sizeof(void*) * cpu, > KVADDR, &rchan_buf, sizeof(void*), > "stp_channel.buf", FAULT_ON_ERROR); > readmem(rchan_buf + rchan_offsets.buf_start, > KVADDR, &pcd->buf.start, sizeof(void*), > "stp_channel.buf.start", FAULT_ON_ERROR); > readmem(rchan_buf + rchan_offsets.buf_offset, > KVADDR, &pcd->buf.offset, sizeof(size_t), > "stp_channel.buf.offset", FAULT_ON_ERROR); > readmem(rchan_buf + rchan_offsets.buf_subbufs_produced, > KVADDR, &pcd->buf.subbufs_produced, sizeof(size_t), > "stp_channel.buf.subbufs_produced", FAULT_ON_ERROR); > readmem(rchan_buf + rchan_offsets.buf_padding, > KVADDR, &pcd->buf.padding, sizeof(size_t*), > "stp_channel.buf.padding", FAULT_ON_ERROR); > } > > static ulong get_symbol_addr(char *module, char *symbol) > { > int i; > struct syment *sym, *sym_end; > struct load_module *lm; > > for (i = 0; i < kt->mods_installed; i++) { > lm = &st->load_modules[i]; > if (!STREQ(module, lm->mod_name)) > continue; > sym = lm->mod_symtable; > sym_end = lm->mod_symend; > for ( ; sym <= sym_end; sym++) { > if(STREQ(sym->name, symbol)) > return ((ulong)sym->value); > } > error(WARNING, "'%s' doesn't have the symbol named '%s'.\n", > module, symbol); > return STPLOG_NO_SYM; > } > error(WARNING, "'%s' is not loaded.\n", module); > return STPLOG_NO_MOD; > } > > static ulong get_rchan_addr(ulong stp_utt_addr) > { > ulong stp_utt; > > readmem(stp_utt_addr, KVADDR, &stp_utt, sizeof(void*), > "stp_utt", FAULT_ON_ERROR); > return (stp_utt + sizeof(int)); > } > > static int check_global_buffer(ulong rchan) > { > int cpu; > ulong rchan_buf[2]; > > for (cpu = 0; cpu < 2; cpu++) { > readmem(rchan + rchan_offsets.buf + sizeof(void*) * cpu, > KVADDR, &rchan_buf[cpu], sizeof(void*), > "stp_channel.buf", FAULT_ON_ERROR); > } > if (rchan_buf[0] == rchan_buf[1]) > return 1; > return 0; > } > > static int setup_global_data(char *module) > { > int i; > ulong stp_utt_addr = 0; > ulong stp_rchan_addr = 0; > ulong rchan; > > stp_utt_addr = get_symbol_addr(module, "_stp_utt"); > if (stp_utt_addr == STPLOG_NO_MOD) { > return -1; > } else if (stp_utt_addr == STPLOG_NO_SYM) { > stp_rchan_addr = get_symbol_addr(module, "_stp_chan"); > if (stp_rchan_addr == STPLOG_NO_SYM) > return -1; > old_format = 1; > } else { > stp_rchan_addr = get_rchan_addr(stp_utt_addr); > } > rchan = get_rchan(stp_rchan_addr); > for (i = 0; i < kt->cpus; i++) > get_rchan_buf(i, rchan); > > if (kt->cpus > 1) { > is_global = check_global_buffer(rchan); > } > > return 0; > } > > static int output_cpu_logs(char *module) > { > int i, max = 256; > struct per_cpu_data *pcd; > size_t n, idx, start, end, ready, len; > unsigned padding; > char fname[max + 1], *source; > DIR *dir; > > /* check and create log directory */ > dir = opendir(module); > if (dir) { > closedir(dir); > } else { > if (mkdir(module, S_IRWXU) < 0) { > error(WARNING, "cannot create log directory '%s\n'", module); > return -1; > } > } > > /* allocate subbuf memory */ > subbuf = malloc(chan.subbuf_size); > if (!subbuf) { > error(WARNING, "cannot allocate memory\n"); > return -1; > } > > fname[max] = '\0'; > for (i = 0; i < kt->cpus; i++) { > pcd = &per_cpu[i]; > ready = pcd->buf.subbufs_produced + (pcd->buf.offset ? 1 : 0); > if (ready > chan.n_subbufs) { > start = ready % chan.n_subbufs; > end = start + chan.n_subbufs; > } else { > start = 0; > end = ready; > } > /* print information */ > fprintf(fp, "--- generating 'cpu%d' ---\n", i); > fprintf(fp, " subbufs ready on relayfs:%d\n", ready); > fprintf(fp, " n_subbufs:%d, read from:%d to:%d (offset:%d)\n\n", > chan.n_subbufs, start, end, pcd->buf.offset); > > /* create log file */ > snprintf(fname, max, "%s/cpu%d", module, i); > outfp = fopen(fname, "w"); > if (!outfp) { > error(WARNING, "cannot create log file '%s'\n", fname); > return -1; > } > for (n = start; n < end; n++) { > /* read relayfs subbufs and write to log file */ > idx = n % chan.n_subbufs; > source = pcd->buf.start + idx * chan.subbuf_size; > readmem((ulong)pcd->buf.padding + sizeof(padding) * idx, > KVADDR, &padding, sizeof(padding), > "padding", FAULT_ON_ERROR); > if (n == end - 1 && pcd->buf.offset) { > len = pcd->buf.offset; > } else { > len = chan.subbuf_size; > } > if (old_format == 1) { > source += sizeof(padding); > len -= sizeof(padding) + padding; > } else { > len -= padding; > } > if (len) { > readmem((ulong)source, KVADDR, subbuf, len, > "subbuf", FAULT_ON_ERROR); > if (fwrite(subbuf, len, 1, outfp) != 1) { > error(WARNING, "cannot write log data\n"); > return -1; > } > } > } > fclose(outfp); > outfp = NULL; > if (is_global == 1) > break; > } > return 0; > } > > static void do_systemtaplog(char *module) > { > if (setup_global_data(module) < 0) > return; > if (output_cpu_logs(module) < 0) > return; > } > > void cmd_systemtaplog(void) > { > > int c; > char *module = NULL; > > while ((c = getopt(argcnt, args, "m:")) != EOF) { > switch (c) { > case 'm': > module = optarg; > break; > default: > argerrs++; > break; > } > } > > if (!module || argerrs) > cmd_usage(pc->curcmd, SYNOPSIS); > > saved_env[0] = pc->main_loop_env[0]; > if (setjmp(pc->main_loop_env)) > goto EXIT; > > do_systemtaplog(module); > EXIT: > cleanup(); > pc->main_loop_env[0] = saved_env[0]; > } > > char *help_systemtaplog[] = { > "systemtaplog", > "Retrieve SystemTap log data", > "-m module_name", > " Retrieve SystemTap's log data and write them to files.\n", > " -m module_name All valid SystemTap log data made by the trace", > " module which name is 'module_name' are written", > " into log files in `module_name` directory. The", > " name of each log file is cpu0, cpu1...cpuN. ", > " They have same format data as channel buffer", > " except padding(This command removes padding). ", > NULL, > }; > > void __attribute__ ((constructor)) systemtaplog_init(void) > { > if (get_rchan_offsets() < 0) > return; > > register_extension(command_table); > return; > } > > void __attribute__ ((destructor)) systemtaplog_fini(void) > { > return; > } > > > ------------------------------------------------------------------------ > > -- > Crash-utility mailing list > Crash-utility@redhat.com > https://www.redhat.com/mailman/listinfo/crash-utility ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [Crash-utility] Re: [RFC] Crash extension for SystemTap 2007-05-24 14:44 ` [Crash-utility] " Dave Anderson @ 2007-06-22 13:43 ` Satoru MORIYA 2007-06-25 16:55 ` Frank Ch. Eigler 2007-07-02 19:48 ` Frank Ch. Eigler 0 siblings, 2 replies; 10+ messages in thread From: Satoru MORIYA @ 2007-06-22 13:43 UTC (permalink / raw) To: Dave Anderson Cc: Discussion list for crash utility usage, maintenance and development, systemtap, yumiko.sugita.yf, soshima, haoki, masami.hiramatsu.pt [-- Attachment #1: Type: text/plain, Size: 3252 bytes --] Hi Dave, Thank you so much for your review and crash-4.0-4.2. And I am sorry for delaying my reply for a long time. I updated my extension for crash-4.0-4.2. Dave Anderson wrote: > The systemtaplog_init() function must be renamed "_init()" in order > for it to be called when the dlopen() call is made. Otherwise, > the extend command fails because no commands get registered: OK, I changed systemtaplog_init() to _init(). > The only other thing that makes me a little nervous is the temporary > relocation of the setjmp buffer. That was never meant to be used > by individual commands -- although nothing prevents it. I see that it > is used to force your cleanup() routine to be called. And your > cleanup() routine exists to free() the subbuf buffer and fclose() > the outfp. A couple suggestions: > > (1) if you use GETBUF() instead of malloc(), then the buffer will get freed > automatically by restore_sanity() prior to the next command prompt > -- even > if you don't do an accompanying FREEBUF(). > (2) with respect to the outfp file pointer, you could recognize whether it > was left open when the command runs again, and fclose() it then. Worse > case, a single file pointer would be left open for the remainder of the > crash session. > That's right, and I agreed with you. I'll stop relocation and implement this part based on your suggestions. > [...]So perhaps I could simply allow modules to register an additional > command with a new "CLEANUP" flag, which would not show up on the help > menu, but would be called during restore_sanity() prior to each > command prompt. Thanks to the new function of crash-4.0-4.2, I can register an cleanup routine of my extension. And I changed setup_global_data() to use symbol_value_module() which was introduced in crash-4.0-4.2 instead of my original function. Additionally, I changed command option. I removed -m option and added -o option. I can specify the output directory using -o option. You can use this extension like this: Preparation ============== (A) Build the shared-object(stplog.so). 1. Install crash-devel package. $ rpm -ivh crash-devel 2. $ cd libcrash_for_systemtap 3. Build $ make (B) Make the crash dump which includes SystemTap trace data. (*)If you analyze the live system memory, ignore this section. 1. Install kdump If you use FC6, see following URL. http://fedoraproject.org/wiki/FC6KdumpKexecHowTo?highlight=%28kdump%29 2. Use SystemTap on bulkmode $ stap -b foo.stp 3. Panic $ echo c > /proc/sysrq-trigger How to use ============== 1. start crash $ crash vmlinux vmcore (*) If you analyze the live system memory, you don't have to enter vmcore. 2. load the shared-object crash> extend $(WHERE_OBJ_PLACED)/stplog.so 3. retrieve the data crash> stplog -o output_dir mod_name 4. You can get output files under output_dir directory. Output ============== stplog command makes a file per cpu buffer of relayfs. Of cause, it also removes the padding bytes in the subbufs of relayfs automatically. Best Regards, -- --- Satoru MORIYA Linux Technology Center Hitachi, Ltd., Systems Development Laboratory E-mail: satoru.moriya.br@hitachi.com [-- Attachment #2: stplog.c --] [-- Type: text/plain, Size: 9390 bytes --] /* crash shared object for retrieving systemtap buffer Copyright (c) 2007 Hitachi,Ltd., Created by Satoru Moriya <satoru.moriya.br@hitachi.com> This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA */ #include <crash/defs.h> #define STPLOG_NO_MOD -1 #define STPLOG_NO_SYM -2 struct rchan_offsets { long subbuf_size; long n_subbufs; long buf; long buf_start; long buf_offset; long buf_subbufs_produced; long buf_padding; }; struct fake_rchan_buf { void *start; size_t offset; size_t subbufs_produced; size_t *padding; }; struct fake_rchan { size_t subbuf_size; size_t n_subbufs; }; struct per_cpu_data { struct fake_rchan_buf buf; }; static struct rchan_offsets rchan_offsets; static struct fake_rchan chan; static struct per_cpu_data per_cpu[NR_CPUS]; static FILE *outfp; static char *subbuf; static int is_global; static int old_format; void cmd_stplog(void); void cmd_stplog_cleanup(void); char *help_stplog[]; char *help_stplog_cleanup[]; static struct command_table_entry command_table[] = { {"stplog", cmd_stplog, help_stplog, 0}, {"stplog_cleanup", cmd_stplog_cleanup, help_stplog_cleanup, CLEANUP}, {NULL, NULL, NULL, 0}, }; static void get_rchan_offsets(void) { rchan_offsets.subbuf_size = MEMBER_OFFSET("rchan", "subbuf_size"); if (rchan_offsets.subbuf_size < 0) goto ERR; rchan_offsets.n_subbufs = MEMBER_OFFSET("rchan", "n_subbufs"); if (rchan_offsets.n_subbufs < 0) goto ERR; rchan_offsets.buf = MEMBER_OFFSET("rchan", "buf"); if (rchan_offsets.buf < 0) goto ERR; rchan_offsets.buf_start = MEMBER_OFFSET("rchan_buf", "start"); if (rchan_offsets.buf_start < 0) goto ERR; rchan_offsets.buf_offset = MEMBER_OFFSET("rchan_buf", "offset"); if (rchan_offsets.buf_offset < 0) goto ERR; rchan_offsets.buf_subbufs_produced = MEMBER_OFFSET("rchan_buf", "subbufs_produced"); if (rchan_offsets.buf_subbufs_produced < 0) goto ERR; rchan_offsets.buf_padding = MEMBER_OFFSET("rchan_buf", "padding"); if (rchan_offsets.buf_padding < 0) goto ERR; return; ERR: error(FATAL, "cannot get rchan offset\n"); } static ulong get_rchan(ulong chan_addr) { ulong rchan; readmem(chan_addr, KVADDR, &rchan, sizeof(void*), "stp_channel", FAULT_ON_ERROR); readmem(rchan + rchan_offsets.subbuf_size, KVADDR, &chan.subbuf_size, sizeof(size_t), "stp_channel.subbuf_size", FAULT_ON_ERROR); readmem(rchan + rchan_offsets.n_subbufs, KVADDR, &chan.n_subbufs, sizeof(size_t), "stp_channel.n_subbufs", FAULT_ON_ERROR); return rchan; } static void get_rchan_buf(int cpu, ulong rchan) { ulong rchan_buf; struct per_cpu_data *pcd; pcd = &per_cpu[cpu]; readmem(rchan + rchan_offsets.buf + sizeof(void*) * cpu, KVADDR, &rchan_buf, sizeof(void*), "stp_channel.buf", FAULT_ON_ERROR); readmem(rchan_buf + rchan_offsets.buf_start, KVADDR, &pcd->buf.start, sizeof(void*), "stp_channel.buf.start", FAULT_ON_ERROR); readmem(rchan_buf + rchan_offsets.buf_offset, KVADDR, &pcd->buf.offset, sizeof(size_t), "stp_channel.buf.offset", FAULT_ON_ERROR); readmem(rchan_buf + rchan_offsets.buf_subbufs_produced, KVADDR, &pcd->buf.subbufs_produced, sizeof(size_t), "stp_channel.buf.subbufs_produced", FAULT_ON_ERROR); readmem(rchan_buf + rchan_offsets.buf_padding, KVADDR, &pcd->buf.padding, sizeof(size_t*), "stp_channel.buf.padding", FAULT_ON_ERROR); return; } static ulong get_rchan_addr(ulong stp_utt_addr) { ulong stp_utt; readmem(stp_utt_addr, KVADDR, &stp_utt, sizeof(void*), "stp_utt", FAULT_ON_ERROR); return (stp_utt + sizeof(int)); } static int check_global_buffer(ulong rchan) { int cpu; ulong rchan_buf[2]; for (cpu = 0; cpu < 2; cpu++) { readmem(rchan + rchan_offsets.buf + sizeof(void*) * cpu, KVADDR, &rchan_buf[cpu], sizeof(void*), "stp_channel.buf", FAULT_ON_ERROR); } if (rchan_buf[0] == rchan_buf[1]) return 1; return 0; } static void setup_global_data(char *module) { int i; ulong stp_utt_addr = 0; ulong stp_rchan_addr = 0; ulong rchan; stp_utt_addr = symbol_value_module("_stp_utt", module); if (stp_utt_addr == 0) { stp_rchan_addr = symbol_value_module("_stp_chan", module); if (stp_rchan_addr == 0) { error(FATAL, "Failed to find _stp_utt/_stp_chan.\n", module); } old_format = 1; } else { stp_rchan_addr = get_rchan_addr(stp_utt_addr); if (stp_rchan_addr == 0) { error(FATAL, "Failed to find _stp_utt/_stp_chan.\n", module); } } rchan = get_rchan(stp_rchan_addr); for (i = 0; i < kt->cpus; i++) get_rchan_buf(i, rchan); if (kt->cpus > 1) { is_global = check_global_buffer(rchan); } return; } static void output_cpu_logs(char *filename) { int i, max = 256; struct per_cpu_data *pcd; size_t n, idx, start, end, ready, len; unsigned padding; char fname[max + 1], *source; DIR *dir; /* check and create log directory */ dir = opendir(filename); if (dir) { closedir(dir); } else { if (mkdir(filename, S_IRWXU) < 0) { error(FATAL, "cannot create log directory '%s\n'", filename); } } /* allocate subbuf memory */ subbuf = GETBUF(chan.subbuf_size); if (!subbuf) { error(FATAL, "cannot allocate memory\n"); } fname[max] = '\0'; for (i = 0; i < kt->cpus; i++) { int adjust = 0; pcd = &per_cpu[i]; if (pcd->buf.offset == 0 || pcd->buf.offset == chan.subbuf_size + 1) { adjust = 0; } else { adjust = 1; } ready = pcd->buf.subbufs_produced + adjust; if (ready > chan.n_subbufs) { start = ready; end = start + chan.n_subbufs; } else { start = 0; end = ready; } /* print information */ fprintf(fp, "--- generating 'cpu%d' ---\n", i); fprintf(fp, " subbufs ready on relayfs:%ld\n", (long)ready); fprintf(fp, " n_subbufs:%ld, read from:%ld to:%ld (offset:%ld)\n\n", (long)chan.n_subbufs, (long)start, (long)end, (long)pcd->buf.offset); /* create log file */ snprintf(fname, max, "%s/cpu%d", filename, i); outfp = fopen(fname, "w"); if (!outfp) { error(FATAL, "cannot create log file '%s'\n", fname); } for (n = start; n < end; n++) { /* read relayfs subbufs and write to log file */ idx = n % chan.n_subbufs; source = pcd->buf.start + idx * chan.subbuf_size; readmem((ulong)pcd->buf.padding + sizeof(padding) * idx, KVADDR, &padding, sizeof(padding), "padding", FAULT_ON_ERROR); if (n == end - 1 && 0 < pcd->buf.offset && pcd->buf.offset < chan.subbuf_size) { len = pcd->buf.offset; } else { len = chan.subbuf_size; } if (old_format == 1) { source += sizeof(padding); len -= sizeof(padding) + padding; } else { len -= padding; } if (len) { readmem((ulong)source, KVADDR, subbuf, len, "subbuf", FAULT_ON_ERROR); if (fwrite(subbuf, len, 1, outfp) != 1) { error(FATAL, "cannot write log data\n"); } } } fclose(outfp); outfp = NULL; if (is_global == 1) break; } if (subbuf) { FREEBUF(subbuf); subbuf = NULL; } return; } static void do_stplog(char *module, char *filename) { setup_global_data(module); output_cpu_logs(filename); return; } void cmd_stplog(void) { int c; char *module = NULL; char *filename = NULL; while ((c = getopt(argcnt, args, "o:")) != EOF) { switch (c) { case 'o': filename = optarg; break; default: argerrs++; break; } } module = args[optind]; if (!module || argerrs) cmd_usage(pc->curcmd, SYNOPSIS); if (filename == NULL && module != NULL) filename = module; do_stplog(module, filename); return; } void cmd_stplog_cleanup(void) { if (outfp) { fclose(outfp); outfp = NULL; } return; } char *help_stplog[] = { "systemtaplog", "Retrieve SystemTap log data", "[-o dir_name] module_name", " Retrieve SystemTap's log data and write them to files.\n", " module_name All valid SystemTap log data made by the trace", " module which name is 'module_name' are written", " into log files. If you don't use -o option, the", " log files are created in `module_name` directory.", " The name of each log file is cpu0, cpu1...cpuN. ", " They have same format data as channel buffer", " except padding(This command removes padding). ", "", " -o file_name Specify the output directory.", NULL, }; char *help_stplog_cleanup[] = { "systemtaplog cleanup (hidden)", "Cleanup command for stplog", "", " This command is called during restore_sanity() prior to each ", " command prompt to close the files which was opened and failed to", " close by stplog command.", NULL, }; static void __attribute__ ((constructor)) _init(void) { get_rchan_offsets(); register_extension(command_table); return; } static void __attribute__ ((destructor)) _fini(void) { return; } [-- Attachment #3: Makefile --] [-- Type: text/plain, Size: 652 bytes --] TARGET=stplog.so CFILE=stplog.c CFLAGS= -shared -rdynamic CFLAGS+= -I./crash -Wall PRJNAME=libcrash_for_systemtap VERSION=`date +%Y%m%d` ARCH=$(shell uname -i) ifeq ($(ARCH), i386) CFLAGS += -DX86 else ifeq ($(ARCH), ia64) CFLAGS += -DIA64 else ifeq ($(ARCH), x86_64) CFLAGS += -DX86_64 endif endif endif $(TARGET):$(CFILE) echo $(CFLAGS) gcc $(CFLAGS) -o $@ $(CFILE) clean: rm -f -r $(TARGET) *~ dist:distclean mkdir $(PRJNAME)-$(VERSION) cp $(CFILE) Makefile README $(PRJNAME)-$(VERSION) tar cvjf $(PRJNAME)-$(VERSION).tar.bz2 $(PRJNAME)-$(VERSION) rm -f -r $(PRJNAME)-$(VERSION) distclean: rm -f -r $(TARGET) *~ crash ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [RFC] Crash extension for SystemTap 2007-06-22 13:43 ` Satoru MORIYA @ 2007-06-25 16:55 ` Frank Ch. Eigler 2007-07-02 19:48 ` Frank Ch. Eigler 1 sibling, 0 replies; 10+ messages in thread From: Frank Ch. Eigler @ 2007-06-25 16:55 UTC (permalink / raw) To: Discussion list for crash utility usage, maintenance and development Cc: Dave Anderson, systemtap, yumiko.sugita.yf Satoru MORIYA <satoru.moriya.br@hitachi.com> writes: > Thank you so much for your review and crash-4.0-4.2. > And I am sorry for delaying my reply for a long time. > I updated my extension for crash-4.0-4.2. > [...] Thank you. Do you think it is ready to be integrated into the systemtap source tree and build system? - FChE ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [RFC] Crash extension for SystemTap 2007-06-22 13:43 ` Satoru MORIYA 2007-06-25 16:55 ` Frank Ch. Eigler @ 2007-07-02 19:48 ` Frank Ch. Eigler 2007-08-20 13:42 ` [Crash-utility] " Satoru MORIYA 1 sibling, 1 reply; 10+ messages in thread From: Frank Ch. Eigler @ 2007-07-02 19:48 UTC (permalink / raw) To: Discussion list for crash utility usage, maintenance and development Cc: Dave Anderson, systemtap, yumiko.sugita.yf Satoru MORIYA <satoru.moriya.br@hitachi.com> writes: > [...] I updated my extension for crash-4.0-4.2. [...] This module is now built as a part of systemtap. The "staplog.so" file is installed into $prefix/lib/systemtap, for loading using crash's "extend" command. As a part of the import, I took the liberty of renaming it "staplog". I could not make it print data from a crash session running against the live kernel [1], so it still needs more work. It would be nice if it also learned to operate against the plain, non-"-b", tracing backend. - FChE [1] % stap -b something.stp & % sudo crash crash 4.0-4.3 [...] KERNEL: /usr/lib/debug/lib/modules/2.6.21-1.3194.fc7debug/vmlinux DUMPFILE: /dev/crash CPUS: 4 DATE: Mon Jul 2 15:34:21 2007 UPTIME: 20 days, 00:56:36 LOAD AVERAGE: 2.97, 1.62, 1.25 TASKS: 268 NODENAME: super.elastic.org RELEASE: 2.6.21-1.3194.fc7debug VERSION: #1 SMP Wed May 23 22:12:03 EDT 2007 MACHINE: x86_64 (2659 Mhz) MEMORY: 4 GB [...] crash> extend ..../staplog.so ..../staplog.so: shared object loaded crash> staplog stap_96acf0ec9a0b8e844b920b552e0b992a_232 staplog: Failed to find _stp_utt/_stp_chan. crash> crash> quit ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [Crash-utility] Re: [RFC] Crash extension for SystemTap 2007-07-02 19:48 ` Frank Ch. Eigler @ 2007-08-20 13:42 ` Satoru MORIYA 2007-08-20 17:50 ` Frank Ch. Eigler 0 siblings, 1 reply; 10+ messages in thread From: Satoru MORIYA @ 2007-08-20 13:42 UTC (permalink / raw) To: Discussion list for crash utility usage, maintenance and development Cc: yumiko.sugita.yf, systemtap [-- Attachment #1: Type: text/plain, Size: 1142 bytes --] Hi Frank, Thank you for including the crash extension in systemtap tree. And I'm sorry for delaying my reply for a long time. > [...]I could not make it print data from a crash session running > against the live kernel [...] I'm Sorry, I didn't test my crash extension on x86_64. Now I tested it on x86_64 and fixed bugs, so I think you can use it on x86_64 architecture. Changes: - fix the way of calculating the offset of rchan in struct utt_trace - fix the type of padding value in output_cpu_logs() - fix the console messages - don't make a directory and files if there's no data in relay buffer I have tested this extention on the following systems: * FC7,i386,kernel-2.6.22,systemtap-0.5.15,crash-4.0-4.5 * FC7,x86_64,kernel-2.6.21-1.3194.fc7,systemtap-0.5.15,crash-4.0-4.5 * FC7,x86_64,kernel-2.6.21-1.3194.fc7debug,systemtap-0.5.15,crash-4.0-4.5 * RHEL5,i386,kernel-2.6.18-8.el5,systemtap-0.5.15,crash-4.0-4.5 * RHEL5,ia64,kernel-2.6.18-8.el5,systemtap-0.5.15,crash-4.0-4.5 Best regards, -- --- Satoru MORIYA Linux Technology Center Hitachi, Ltd., Systems Development Laboratory E-mail: satoru.moriya.br@hitachi.com [-- Attachment #2: staplog_fix.patch --] [-- Type: text/plain, Size: 4814 bytes --] Index: systemtap/staplog.c =================================================================== --- systemtap.orig/staplog.c +++ systemtap/staplog.c @@ -48,9 +48,6 @@ #include <crash/defs.h> -#define STPLOG_NO_MOD -1 -#define STPLOG_NO_SYM -2 - struct rchan_offsets { long subbuf_size; long n_subbufs; @@ -167,10 +164,35 @@ static void get_rchan_buf(int cpu, ulong static ulong get_rchan_addr(ulong stp_utt_addr) { ulong stp_utt; + long offset; readmem(stp_utt_addr, KVADDR, &stp_utt, sizeof(void*), "stp_utt", FAULT_ON_ERROR); - return (stp_utt + sizeof(int)); + + /* + * If we couldn't get the member offset of struct utt_trace.rchan, + * i.e. the debuginfo of the trace module isn't available, we use + * sizeof(long) as the offset instead. Currently struct utt_trace + * is defined as below: + * + * struct utt_trace { + * int trace_state; + * struct rchan *rchan; + * ... + * } + * + * Although the type of the preceding member is int, sizeof(long) + * is OK, because rchan is aligned with long size on both 32-bit + * and 64-bit environment. When the definision of struct utt_trace + * changed, we must check if this code is correct. + */ + if ((offset = MEMBER_OFFSET("utt_trace", "rchan")) < 0) { + error(WARNING, "The debuginfo of the trace module hasn't been loaded. " + "You may not be able to retrieve the correct trace data.\n"); + offset = sizeof(long); + } + + return (stp_utt + (ulong)offset); } static int check_global_buffer(ulong rchan) @@ -220,25 +242,15 @@ static void setup_global_data(char *modu return; } -static void output_cpu_logs(char *filename) +static void output_cpu_logs(char *dirname) { int i, max = 256; struct per_cpu_data *pcd; size_t n, idx, start, end, ready, len; - unsigned padding; + size_t padding; char fname[max + 1], *source; DIR *dir; - /* check and create log directory */ - dir = opendir(filename); - if (dir) { - closedir(dir); - } else { - if (mkdir(filename, S_IRWXU) < 0) { - error(FATAL, "cannot create log directory '%s\n'", filename); - } - } - /* allocate subbuf memory */ subbuf = GETBUF(chan.subbuf_size); if (!subbuf) { @@ -257,6 +269,15 @@ static void output_cpu_logs(char *filena adjust = 1; } ready = pcd->buf.subbufs_produced + adjust; + if (ready == 0) { + if (is_global == 1) { + error(WARNING, "There is no data in the relay buffer.\n"); + break; + } else { + error(WARNING, "[cpu:%d]There is no data in the relay buffer.\n", i); + continue; + } + } if (ready > chan.n_subbufs) { start = ready; @@ -266,13 +287,30 @@ static void output_cpu_logs(char *filena end = ready; } /* print information */ - fprintf(fp, "--- generating 'cpu%d' ---\n", i); + if (is_global == 1) { + fprintf(fp, "--- generating 'global' ---\n"); + } else { + fprintf(fp, "--- generating 'cpu%d' ---\n", i); + } fprintf(fp, " subbufs ready on relayfs:%ld\n", (long)ready); - fprintf(fp, " n_subbufs:%ld, read from:%ld to:%ld (offset:%ld)\n\n", - (long)chan.n_subbufs, (long)start, (long)end, (long)pcd->buf.offset); - - /* create log file */ - snprintf(fname, max, "%s/cpu%d", filename, i); + fprintf(fp, " n_subbufs:%ld, read from:%ld to:%ld (offset:%ld)\n\n", + (long)chan.n_subbufs, (long)(start ? start - chan.n_subbufs : start), + (long)(start ? end - 1 - chan.n_subbufs : end - 1), (long)pcd->buf.offset); + + /* create log dir and file */ + dir = opendir(dirname); + if (dir) { + closedir(dir); + } else { + if (mkdir(dirname, S_IRWXU) < 0) { + error(FATAL, "cannot create log directory '%s\n'", dirname); + } + } + if (is_global == 1) { + snprintf(fname, max, "%s/global", dirname, i); + } else { + snprintf(fname, max, "%s/cpu%d", dirname, i); + } outfp = fopen(fname, "w"); if (!outfp) { error(FATAL, "cannot create log file '%s'\n", fname); @@ -316,10 +354,10 @@ static void output_cpu_logs(char *filena return; } -static void do_staplog(char *module, char *filename) +static void do_staplog(char *module, char *dirname) { setup_global_data(module); - output_cpu_logs(filename); + output_cpu_logs(dirname); return; } @@ -328,12 +366,12 @@ void cmd_staplog(void) int c; char *module = NULL; - char *filename = NULL; + char *dirname = NULL; while ((c = getopt(argcnt, args, "o:")) != EOF) { switch (c) { case 'o': - filename = optarg; + dirname = optarg; break; default: argerrs++; @@ -345,9 +383,9 @@ void cmd_staplog(void) if (!module || argerrs) cmd_usage(pc->curcmd, SYNOPSIS); - if (filename == NULL && module != NULL) - filename = module; - do_staplog(module, filename); + if (dirname == NULL && module != NULL) + dirname = module; + do_staplog(module, dirname); return; } ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [RFC] Crash extension for SystemTap 2007-08-20 13:42 ` [Crash-utility] " Satoru MORIYA @ 2007-08-20 17:50 ` Frank Ch. Eigler 0 siblings, 0 replies; 10+ messages in thread From: Frank Ch. Eigler @ 2007-08-20 17:50 UTC (permalink / raw) To: Discussion list for crash utility usage, maintenance and development Cc: systemtap, yumiko.sugita.yf Satoru MORIYA <satoru.moriya.br@hitachi.com> writes: >[...] > I'm Sorry, I didn't test my crash extension on x86_64. > Now I tested it on x86_64 and fixed bugs, so I think you can > use it on x86_64 architecture. Thanks, I committed your changes. Please feel free to directly maintain this source file in the cvs tree yourself, including ChangeLog entries for them. (If needed, first apply for CVS-write access at the systemtap "get involved" web pages.) - FChE ^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2007-08-20 16:20 UTC | newest] Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2007-05-23 11:46 [RFC] Crash extension for SystemTap Satoru MORIYA 2007-05-23 13:22 ` [Crash-utility] " Dave Anderson 2007-05-23 19:01 ` Frank Eigler 2007-05-24 0:54 ` Satoru MORIYA 2007-05-24 14:44 ` [Crash-utility] " Dave Anderson 2007-06-22 13:43 ` Satoru MORIYA 2007-06-25 16:55 ` Frank Ch. Eigler 2007-07-02 19:48 ` Frank Ch. Eigler 2007-08-20 13:42 ` [Crash-utility] " Satoru MORIYA 2007-08-20 17:50 ` Frank Ch. Eigler
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).