From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 2284 invoked by alias); 13 Apr 2010 17:17:09 -0000 Received: (qmail 2269 invoked by uid 22791); 13 Apr 2010 17:17:06 -0000 X-SWARE-Spam-Status: No, hits=1.1 required=5.0 tests=BAYES_50,TW_CZ,TW_DP,TW_XJ,TW_XZ,T_RP_MATCHES_RCVD X-Spam-Check-By: sourceware.org Received: from e28smtp05.in.ibm.com (HELO e28smtp05.in.ibm.com) (122.248.162.5) by sourceware.org (qpsmtpd/0.43rc1) with ESMTP; Tue, 13 Apr 2010 17:16:58 +0000 Received: from d28relay03.in.ibm.com (d28relay03.in.ibm.com [9.184.220.60]) by e28smtp05.in.ibm.com (8.14.3/8.13.1) with ESMTP id o3DHGpWH003140 for ; Tue, 13 Apr 2010 22:46:51 +0530 Received: from d28av03.in.ibm.com (d28av03.in.ibm.com [9.184.220.65]) by d28relay03.in.ibm.com (8.13.8/8.13.8/NCO v10.0) with ESMTP id o3DHGpGt3334314 for ; Tue, 13 Apr 2010 22:46:51 +0530 Received: from d28av03.in.ibm.com (loopback [127.0.0.1]) by d28av03.in.ibm.com (8.14.3/8.13.1/NCO v10.0 AVout) with ESMTP id o3DHGpwA011577 for ; Wed, 14 Apr 2010 03:16:51 +1000 Received: from apj.in.ibm.com ([9.124.218.111]) by d28av03.in.ibm.com (8.14.3/8.13.1/NCO v10.0 AVin) with ESMTP id o3DHGoVl011558; Wed, 14 Apr 2010 03:16:50 +1000 Message-ID: <4BC4A701.30801@linux.vnet.ibm.com> Date: Tue, 13 Apr 2010 17:17:00 -0000 From: Anithra P Janakiraman User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.4pre) Gecko/20090922 Fedora/3.0-3.9.b4.fc12 Thunderbird/3.0b4 MIME-Version: 1.0 To: "Frank Ch. Eigler" CC: "systemtap@sourceware.org" Subject: Re: [RFC] Framework for easy distribution of SystemTap scripts (V3) References: <4B1FE14D.5070307@linux.vnet.ibm.com> <4B685611.5090402@linux.vnet.ibm.com> <4BA7D773.5050106@linux.vnet.ibm.com> <4BBC4EA1.9040608@linux.vnet.ibm.com> <4BBF536D.60908@linux.vnet.ibm.com> <20100409163949.GE28054@redhat.com> In-Reply-To: <20100409163949.GE28054@redhat.com> Content-Type: multipart/mixed; boundary="------------020602020901050401080306" X-IsSubscribed: yes Mailing-List: contact systemtap-help@sourceware.org; run by ezmlm Precedence: bulk List-Id: List-Subscribe: List-Post: List-Help: , Sender: systemtap-owner@sourceware.org X-SW-Source: 2010-q2/txt/msg00113.txt.bz2 This is a multi-part message in MIME format. --------------020602020901050401080306 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Content-length: 4718 Hi Frank, Thanks again, attaching a V3 Regards, Anithra. On 04/09/2010 10:09 PM, Frank Ch. Eigler wrote: >> >> We have placed the script in the scripts directory, We dont want to >> install the script, so didn't see the use for a man page. > > Why *not* install it? How are people supposed to get a hold of your > script if it doesn't get installed? This script will be used by a very select set of users - say admins who wish to debug a customer machine, dont think it would be right to install it as part of the systemtap/systemtap-runtime rpms as it is not going to be used by the 'regular' stap user. It can be obtained from the source. >>>> +A post processing script can be provided using the -p option. >>> >>> You should specify what exactly this means. A '/bin/sh -c' command >>> line to pipe stdout through? >> >> The post processing script can do anything the administrator creating >> the package wishes to do, maybe even fwd the output to a support team, >> or provide suggestions based on the output. Have added an explanation. > > The key part is to say that for -p SCRIPT, SCRIPT is actually a file name, > whose contents will be copied into the new package, and which will be run > via a pipe to consume the stdout of stdrun. Done. Changed! > >>>> + *) STAP_OPTIONS=$@ >>>> + echo $STAP_OPTIONS >>> >>> You'll need to watch the quoting in these $foo variables. >>> They could contain spaces etc. >> >> Didn't understand the comment. stap-buildpkg script.stp -n "package >> name" -v 1 would work. > > Yes, but you need to check your quotation throughout. In some cases, > there is "$foo", and in other cases, only $foo. The difference can > matter in some contexts. So I'm just saying to check and/or test with > all possible spacey arguments. Done. > > >>> FWIW, you could include this template within the main buildpkg script, as >>> in >>> >>> echo> $PKG<< 'END' >>> function run() >>> .... >>> END >>> >> >> True. This is for simplicity of design. It is intuitively a template and >> a separate file, so we decided to keep it like that esp since we had no >> intentions of installing the build-pkg script. > > I guess simplicity is in the eye of the beholder. You are having to > edit the thing with sed and concatenate things before& after: this > too seems clumsy. I agree that the sed part may not look very neat, but it is intuitive - we replace the place-holders in the template with their corresponding values. I think it makes the package easier to understand. We can still change if you think it is absolutely necessary :) > > From just eyeballing the new version of the scripts, it's not clear > how this template file is located. (If it were installed, you could > autoconf and @prefix@-inform it.) > Assuming that the template file will be located in the same directory as the stap-buildpkg script, along with the README. Currently placed in the scripts dir. > >>>> +ps -ef | grep TEMPLATE_PKG_NAME | grep stapio | awk '{print $2}' | xargs >>>> kill -SIGINT >>> >>> You wouldn't need this is you stored $! after the "staprun ...&" a few >>> lines ago. >> >> Yes. We had done it this way keeping the older versions of SystemTap in >> mind, the $! returns pid of 'staprun' in older versions. I've changed >> the template > > (There still appears some ps -ef | grepping in the new version; probably > all of that is unnecessary, if you save child pids properly.) With the -start option, staprun executes in the background while the script exits. To stop users need to run the package with -stop, it would not be possible to store the child pids in this scenario. > > >>> Why does this template need to exist? The framework maker script >>> could transcribe all of the options right into the output script. >> >> We want to provide a set of 'default' options. [...] > > But one can provide defaults in lots of ways. If you believe time-limited > staprun execution is a common and basic option, you can make that an option > provided first class within the stap-buildpkg script, and override it with > a first-class command line options. > > To have a separate configuration file for this only, you'd need to > argue why some options have to be treated differently from others. The problem here is that we are not aware of the options or the no of options before hand. This would vary from script to script. The only option that we know would be always needed is 'time'. The config file is not used by the framework at all, it is needed only by the resulting package. Moreover each config option would be associated with a help text that can be appended to the resulting package' help text. Regards, Anithra. --------------020602020901050401080306 Content-Type: text/plain; name="distributionframework_v3.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="distributionframework_v3.patch" Content-length: 12597 diff --git a/scripts/build_pkg/README.stap-buildpkg b/scripts/build_pkg/README.stap-buildpkg new file mode 100644 index 0000000..443f841 --- /dev/null +++ b/scripts/build_pkg/README.stap-buildpkg @@ -0,0 +1,70 @@ +SystemTap distribution framework + +This is free software. +See the COPYING file for redistribution/modification terms. + +Introduction: + +The SystemTap distribution framework aims to: + +1. packages a set of systemtap scripts in a format that circumvents as much as +possible, external dependencies like kernel-debuginfo. +2. generate a package that will self-extract, run the script, and post +process the output with minimal intervention from users. + +Usage: + +stap-buildpkg scriptfile [options] [stap-options] +Options: + -n pkg_name : specify a package name + -v pkg_version : specify a package version + -p script : post processing script + -c pkg_config : provide config parameters through a file + -h help_config : help text for config parameters + All additonal options will be passed to the stap command + +pkg_name and pkg_version will be used to set the name and version of the +resulting executable. For the -p script option, script is a file name, whose +contents will be copied into the new package. It will be executed post +completion of the staprun command. The post processing script can be used to +process the output any which way the user choses. This could include filtering +the output,offering suggestions based on the output data, or even forwarding the +output to a support team. +An optional config file that contains post_processing script specific config +options and corresponding help entries can also be specified using the -c and +-h options respectively.For eg: If a script expects the pid and ip address as +options the pkg_config file could contain the following: +pid=5632 +ip=127.0.0.1 +Note: These options are the 'default' options and can be overridden through +the command line when the resulting package is executed. These options will be +passed to post_processing script to filter the output and not to the +stap/staprun command. The help_config contains help entries for any +additional parameters. Eg: +ipaddress=[ip] ip is a standard IPv4 address. +pid=[pid] pid of the process to be monitored. +The help text would be appended to the usage/help instructions of the +resulting package. +The resulting package is a self-extracting binary. When invoked it would untar +itself, execute the script(by invoking staprun) and post process. + +The usage for the the resulting package: + + [options] [parameters] + +Options: + * --run -r Runs the scripts + for x minutes where x can be passed as a parameter + in the form time=[x]. The default value is 10 minutes. + Post processing is performed after the script completes. + * --start -s Invokes the script as a background process. + * --stop -x Stops the script and performs post processing. + * --all Same as --run + * --help Displays this usage text. + +Parameters: + + * time=[x] x is in minutes. Runs the script for x minutes. valid + for --run(-r) o --start(-s) or --all(-a) options only + Any other parameter specified in the help_config. + diff --git a/scripts/build_pkg/stap-buildpkg b/scripts/build_pkg/stap-buildpkg new file mode 100755 index 0000000..2eb70f3 --- /dev/null +++ b/scripts/build_pkg/stap-buildpkg @@ -0,0 +1,191 @@ +##!/bin/bash +# +# Script to auto-generate executable packages from stap scripts +# + +prog=stap-buildpkg + +echo_usage () { + echo $"Usage: $prog scriptfile [options] [stap-options]" + echo $"Options:" + echo $" -n pkg_name : specify a package name" + echo $" -v pkg_version : specify a package version" + echo $" -c pkg_config : provide config parameters through a file" + echo $" -p post_process script : post processing script" + echo $" -h help_config : help text for config parameters" + echo $" All additonal options will be passed to the stap command" +} + +parse_args() { +while [ -n "$1" ]; do + case $1 in + -n) PKG_NAME=$2 + shift 2 + echo $PKG_NAME + ;; + -v) PKG_VERSION=$2 + shift 2 + echo $PKG_VERSION + ;; + -c) PKG_INPUT_CONFIG=$2 + echo $PKG_INPUT_CONFIG + shift 2 + ;; + -p) PKG_POST_SCRIPT=$2 + echo $PKG_POST_SCRIPT + shift 2 + ;; + -h) PKG_HELP_CONFIG=$2 + echo $PKG_HELP_CONFIG + shift 2 + ;; + *) STAP_OPTIONS=$@ + echo $STAP_OPTIONS + shift $# +esac +done +} + +validate_args() { + if test -z $PKG_NAME; then + PKG_NAME=`basename $script .stp` + fi + if test -z $PKG_VERSION; then + PKG_VERSION=1.0 + fi + if test -z $PKG_RELEASE; then + PKG_RELEASE="0" + fi + PKG_ARCH=`uname -i` +} + +module_gen() { +stap -p4 -m $PKG_NAME.ko $script $STAP_OPTIONS +if [ -f $PKG_NAME.ko ] +then + echo " Module $PKG_NAME.ko generated successfully "; +else + echo " Module $PKG_NAME.ko generation failed. Exiting now... " + rm -rf $TEMPDIR + exit; +fi +} + +create_wrappers() { +echo "Creating install wrapper from template.."; +echo "s/TEMPLATE_PKG_NAME/$PKG_NAME/g" > $TEMPDIR/sd-installer-patterns +echo "s/TEMPLATE_PKG_VERSION/$PKG_VERSION/g" >> $TEMPDIR/sd-installer-patterns +echo "s/TEMPLATE_PKG_RELEASE/$PKG_RELEASE/" >> $TEMPDIR/sd-installer-patterns +echo "s/TEMPLATE_ARCH/$PKG_ARCH/g" >> $TEMPDIR/sd-installer-patterns + +sed -f $TEMPDIR/sd-installer-patterns template.buildpkg > $TEMPDIR/wrapper +csplit -s $TEMPDIR/wrapper '/extractor template/' +mv xx00 ${PKG_NAME}_install +mv xx01 ${PKG_NAME}_binextractor + +chmod a+x ${PKG_NAME}_install +rm $TEMPDIR/sd-installer-patterns +rm $TEMPDIR/wrapper +} + +copy_files_to_temp() { + +runtime=systemtap-runtime +if test -z $(rpm -qa $runtime); then + runtime=systemtap +fi + +echo "using systemtap runtime executables from $runtime rpm" + +cp `rpm -ql $runtime | grep staprun | grep -v staprun.8.gz` $TEMPDIR/. +cp `rpm -ql $runtime | grep stapio` $TEMPDIR/. + +echo "Copying Systemtap module.." +mv $PKG_NAME.ko $TEMPDIR/ +echo "Copying config file.." +echo "time=2m" > $TEMPDIR/$PKG_NAME.config +if [ -f $PKG_INPUT_CONFIG ] && [ "$PKG_INPUT_CONFIG" != "" ] +then + cat $PKG_INPUT_CONFIG >> $TEMPDIR/$PKG_NAME.config +fi +# Copying help file, or adding an empty file if none is provided. +if [ -f $PKG_HELP_CONFIG ] && [ "$PKG_HELP_CONFIG" != "" ] +then + cp $PKG_HELP_CONFIG $TEMPDIR/$PKG_NAME.help +else + echo " " > $TEMPDIR/$PKG_NAME.help +fi +echo "Copying the post processing script" +# copying the post processing script, or adding an empty file if none is provided. +if [ -f $PKG_POST_SCRIPT ] && [ "$PKG_POST_SCRIPT" != "" ] +then + cp $PKG_POST_SCRIPT $TEMPDIR/$PKG_NAME.post +else + echo " " > $TEMPDIR/$PKG_NAME.post +fi +chmod +x $TEMPDIR/$PKG_NAME.post +mv ${PKG_NAME}_install $TEMPDIR/ +} + +#------------------------------------------------------------------ +# Mainline script +#------------------------------------------------------------------ +script=$1 +echo $script +shift 1 + +if test -z $script; then + echo "Please provide a script name" + echo_usage + exit +fi + +if [ ! -f "$script" ]; then + echo "Script file does not exist" + echo "Please provide a valid script name" + echo_usage + exit +fi + + +#OPTS=`getopt -s bash -u -o 'n:v:R:c:p:h:' -- $@` + +#parse arguments +parse_args $@ +validate_args + +# generate a tmp directory to store all components:remember to clean it up ! +TEMPDIR=/tmp/$PKG_NAME-$PKG_VERSION; +mkdir $TEMPDIR + +# Generate Stap module +echo "Generating Stap module.."; +#module_gen + +#generate installer(wrapper) scripts +#create_wrappers + +#copy systemtap-runtime files +#copy_files_to_temp + +MY_PWD=`pwd` +cd $TEMPDIR ; tar -cjf $PKG_NAME-$PKG_VERSION.tar.bz2 * +cd $MY_PWD; + +echo "Building the self-extracting binary.." +mkdir $TEMPDIR/$PKG_NAME +mv $TEMPDIR/${PKG_NAME}_install $TEMPDIR/$PKG_NAME/ +tar -xjf $TEMPDIR/$PKG_NAME-$PKG_VERSION.tar.bz2 -C $TEMPDIR/$PKG_NAME/ +mv ${PKG_NAME}_binextractor $TEMPDIR/${PKG_NAME}_binextractor 2>/dev/null + +MY_PWD=`pwd` +cd $TEMPDIR; +tar -czf - $PKG_NAME/ >> ${PKG_NAME}_binextractor +cd $MY_PWD; +mv $TEMPDIR/${PKG_NAME}_binextractor ./$PKG_NAME-$PKG_VERSION +echo "The installer is at `pwd`/$PKG_NAME-$PKG_VERSION" +chmod +x $PKG_NAME-$PKG_VERSION + +#Cleanup the mess in temp dir +rm -rf $TEMPDIR +exit diff --git a/scripts/build_pkg/template.buildpkg b/scripts/build_pkg/template.buildpkg new file mode 100644 index 0000000..07d7926 --- /dev/null +++ b/scripts/build_pkg/template.buildpkg @@ -0,0 +1,142 @@ +#!/bin/bash +RUN_DIR=`pwd` + +function run() +{ +echo "Gathering data... Please wait" +# Log file appears as systemtap_tcpipstat.out +TS=`date "+%Y%m%d-%H:%M:%S"` +export SYSTEMTAP_STAPIO=$RUN_DIR/stapio +$RUN_DIR/staprun $RUN_DIR/TEMPLATE_PKG_NAME.ko > $RUN_DIR/TEMPLATE_PKG_NAME.out 2>&1 & +stappid=$! +sleep $time +echo "Completed" +mv $RUN_DIR/TEMPLATE_PKG_NAME.out $RUN_DIR/TEMPLATE_PKG_NAME_${TS}.out +echo "Output file TEMPLATE_PKG_NAME_${TS}.out is in $RUN_DIR directory" +kill -SIGINT $stappid +echo "Postprocessing..." +$RUN_DIR/TEMPLATE_PKG_NAME.post $RUN_DIR/TEMPLATE_PKG_NAME_${TS}.out +rm $RUN_DIR/staprun +rm $RUN_DIR/stapio +} + +function start() +{ + # check to see if tapset is installed +echo "Gathering data..." +export SYSTEMTAP_STAPIO=$RUN_DIR/stapio +$RUN_DIR/staprun $RUN_DIR/TEMPLATE_PKG_NAME.ko > $RUN_DIR/TEMPLATE_PKG_NAME_tmp.out 2>&1 & +echo "Run TEMPLATE_PKG_NAME --stop to stop the script and process output" +} + +function stop() +{ +TS=`date "+%Y%m%d-%H:%M:%S"` +echo "Stopping systemtap script" +ps -ef | grep TEMPLATE_PKG_NAME | grep stapio | awk '{print $2}' | xargs kill -SIGINT +echo "Done" +mv $RUN_DIR/TEMPLATE_PKG_NAME_tmp.out $RUN_DIR/TEMPLATE_PKG_NAME_${TS}.out +echo "Output file TEMPLATE_PKG_NAME_${TS}.out is in $RUN_DIR directory" +echo "Postprocessing..." +$RUN_DIR/TEMPLATE_PKG_NAME.post $RUN_DIR/TEMPLATE_PKG_NAME_${TS}.out +rm $RUN_DIR/staprun +rm $RUN_DIR/stapio +} + +function helptext() +{ + echo " " + echo "USAGE: TEMPLATE_PKG_NAME [options] [parameters]" + echo " " + echo "Options: " + echo " --run -r Runs the scripts for x minutes where x can be passed as a parameter using time=[x]. " + echo " The default value is 10 minutes. Post processing is performed after the script completes." + echo " --start -s Runs the script as a background process" + echo " --stop -x Stops the script and performs post processing" + echo " --all Runs the script and processes the output" + echo " --help Displays this usage text" + echo " " + echo "Parameters: " + echo " " + echo " time=[x] x is in minutes. Runs the script for x minutes. valid for --run(-r)" + echo " --start(-s) or --all(-a) options only" + cat $RUN_DIR/TEMPLATE_PKG_NAME.help + echo "" +} + +function parseparameters() +{ +for i in $@ +do + tmpa=${1%%=*} + tmpb=${1#$tmpa=} + export $tmpa=$tmpb + shift +done +} + + +option=$1 +shift + + + FILE=$RUN_DIR/TEMPLATE_PKG_NAME.config + # make sure file exist and readable + if [ ! -f $FILE ]; then + echo "$FILE : does not exist" + elif [ ! -r $FILE ]; then + echo "$FILE: cannot read" + fi + + + + exec 3<&0 + exec 0<$FILE + while read line + do + tmpa=${line%%=*} + tmpb=${line#$tmpa=} + declare $tmpa=$tmpb + done + exec 0<&3 + + for i in $@ + do + tmpa=${1%%=*} + tmpb=${1#$tmpa=} + export $tmpa=$tmpb + shift + done + + +case $option in + -r | --run ) + run + exit + ;; + -s | --start ) + start + exit + ;; + -h | --help ) helptext + exit + ;; + -x | --stop ) stop + exit + ;; + -a | --all ) run + exit + ;; + * ) helptext + exit 1 +esac + +## extractor template +#!/bin/sh -e +sed -e '1,/^exit$/d' "$0" | tar -xzf - +cd TEMPLATE_PKG_NAME + ./TEMPLATE_PKG_NAME_install $@ +cd .. +rm -f TEMPLATE_PKG_NAME/TEMPLATE_PKG_NAME.* +rm -f TEMPLATE_PKG_NAME/TEMPLATE_PKG_NAME_install +exit --------------020602020901050401080306--