Re: [RFC] Framework for easy distribution of SystemTap scripts (V3)

[Anithra P Janakiraman <anithra@linux.vnet.ibm.com>]

[-- Attachment #1: Type: text/plain, Size: 4718 bytes --]

Hi Frank,

Thanks again, attaching a V3

Regards,
Anithra.


On 04/09/2010 10:09 PM, Frank Ch. Eigler wrote:


>>
>> We have placed the script in the scripts directory, We dont want to
>> install the script, so didn't see the use for a man page.
>
> Why *not* install it?  How are people supposed to get a hold of your
> script if it doesn't get installed?

This script will be used by a very select set of users - say admins who 
wish to debug a customer machine, dont think it would be right to 
install it as part of the systemtap/systemtap-runtime rpms as it is not 
going to be used by the 'regular' stap user. It can be obtained from the 
source.


>>>> +A post processing script can be provided using the -p option.
>>>
>>> You should specify what exactly this means.  A '/bin/sh -c' command
>>> line to pipe stdout through?
>>
>> The post processing script can do anything the administrator creating
>> the package wishes to do, maybe even fwd the output to a support team,
>> or provide suggestions based on the output. Have added an explanation.
>
> The key part is to say that for -p SCRIPT, SCRIPT is actually a file name,
> whose contents will be copied into the new package, and which will be run
> via a pipe to consume the stdout of stdrun.

Done. Changed!

>
>>>> +        *)                     STAP_OPTIONS=$@
>>>> +				echo $STAP_OPTIONS
>>>
>>> You'll need to watch the quoting in these $foo variables.
>>> They could contain spaces etc.
>>
>> Didn't understand the comment. stap-buildpkg script.stp -n "package
>> name" -v 1  would work.
>
> Yes, but you need to check your quotation throughout.  In some cases,
> there is "$foo", and in other cases, only $foo.  The difference can
> matter in some contexts.  So I'm just saying to check and/or test with
> all possible spacey arguments.

Done.

>
>
>>> FWIW, you could include this template within the main buildpkg script, as
>>> in
>>>
>>> echo>   $PKG<<   'END'
>>> function run()
>>> ....
>>> END
>>>
>>
>> True. This is for simplicity of design. It is intuitively a template and
>> a separate file, so we decided to keep it like that esp since we had no
>> intentions of installing the build-pkg script.
>
> I guess simplicity is in the eye of the beholder.  You are having to
> edit the thing with sed and concatenate things before&  after: this
> too seems clumsy.

I agree that the sed part may not look very neat, but it is intuitive - 
we replace the place-holders in the template with their corresponding
values. I think it makes the package easier to understand. We can still 
change if you think it is absolutely necessary :)

>
>  From just eyeballing the new version of the scripts, it's not clear
> how this template file is located.  (If it were installed, you could
> autoconf and @prefix@-inform it.)
>

Assuming that the template file will be located in the same directory as 
the stap-buildpkg script, along with the README. Currently placed in the 
scripts dir.

>
>>>> +ps -ef | grep TEMPLATE_PKG_NAME | grep stapio | awk '{print $2}' | xargs
>>>> kill -SIGINT
>>>
>>> You wouldn't need this is you stored $! after the "staprun ...&" a few
>>> lines ago.
>>
>> Yes. We had done it this way keeping the older versions of SystemTap in
>> mind, the $! returns pid of 'staprun' in older versions. I've changed
>> the template
>
> (There still appears some ps -ef | grepping in the new version; probably
> all of that is unnecessary, if you save child pids properly.)

With the -start option, staprun executes in the background while the 
script exits. To stop users need to run the package with -stop, it would 
not be possible to store the child pids in this scenario.

>
>
>>> Why does this template need to exist?  The framework maker script
>>> could transcribe all of the options right into the output script.
>>
>> We want to provide a set of 'default' options.  [...]
>
> But one can provide defaults in lots of ways.  If you believe time-limited
> staprun execution is a common and basic option, you can make that an option
> provided first class within the stap-buildpkg script, and override it with
> a first-class command line options.
>
> To have a separate configuration file for this only, you'd need to
> argue why some options have to be treated differently from others.

The problem here is that we are not aware of the options or the no of 
options before hand. This would vary from script to script. The only 
option that we know would be always needed is 'time'. The config file is 
not used by the framework at all, it is needed only by the resulting 
package. Moreover each config option would be associated with a help 
text that can be appended to the resulting package' help text.

Regards,
Anithra.


[-- Attachment #2: distributionframework_v3.patch --]
[-- Type: text/plain, Size: 12597 bytes --]

diff --git a/scripts/build_pkg/README.stap-buildpkg b/scripts/build_pkg/README.stap-buildpkg
new file mode 100644
index 0000000..443f841
--- /dev/null
+++ b/scripts/build_pkg/README.stap-buildpkg
@@ -0,0 +1,70 @@
+SystemTap distribution framework
+
+This is free software.
+See the COPYING file for redistribution/modification terms.
+
+Introduction:
+
+The SystemTap distribution framework aims to:
+
+1. packages a set of systemtap scripts in a format that circumvents as much as
+possible, external dependencies like kernel-debuginfo.  
+2.  generate a package that will self-extract,  run the script, and post
+process the output with minimal intervention from  users.
+
+Usage: 
+
+stap-buildpkg scriptfile [options] [stap-options]
+Options:
+       -n pkg_name              : specify a package name
+       -v pkg_version           : specify a package version
+       -p script 		: post processing script
+       -c pkg_config            : provide config parameters through a file
+       -h help_config           : help text for config parameters
+       All additonal options will be passed to the stap command
+
+pkg_name and pkg_version will be used to set the name and version of the
+resulting executable. For the -p script option, script is  a file name, whose
+contents will be copied into the new package. It will be executed post
+completion of the staprun command.  The post processing script can be used to
+process the output any which way the user choses. This could include filtering
+the output,offering suggestions based on the output data, or even forwarding the
+output to a support team.  
+An optional config file that contains post_processing script specific config
+options and corresponding help entries can also be specified using the -c and
+-h options respectively.For eg: If a script expects the pid and ip address as
+options the pkg_config file could contain the following: 
+pid=5632
+ip=127.0.0.1 
+Note: These options are the 'default' options and can be overridden through
+the command line when the resulting package is executed. These options will be
+passed to post_processing script to filter the output and not to the
+stap/staprun command.  The help_config contains help entries for any
+additional parameters. Eg: 
+ipaddress=[ip] ip is a standard IPv4 address.
+pid=[pid] pid of the process to be monitored.  
+The help text would be appended to the usage/help instructions of the
+resulting package.  
+The resulting package is a self-extracting binary. When invoked it would untar
+itself, execute the script(by invoking staprun) and post process.
+
+The usage for the the resulting package:
+
+<package_name> [options] [parameters]
+
+Options:
+   * --run -r   Runs the scripts
+                for x minutes where x can be passed as a parameter 	
+		in the form time=[x]. The default value is 10 minutes. 
+		Post processing is performed after the script completes.
+   * --start -s Invokes the script as a background process.
+   * --stop -x  Stops the script and performs post processing.
+   * --all      Same as --run
+   * --help     Displays this usage text.
+
+Parameters:
+
+   * time=[x]   x is in minutes. Runs the script for x minutes. valid
+                for --run(-r) o --start(-s) or --all(-a) options only
+   Any other parameter specified in the help_config.
+
diff --git a/scripts/build_pkg/stap-buildpkg b/scripts/build_pkg/stap-buildpkg
new file mode 100755
index 0000000..2eb70f3
--- /dev/null
+++ b/scripts/build_pkg/stap-buildpkg
@@ -0,0 +1,191 @@
+##!/bin/bash
+# 
+#  Script to auto-generate executable packages from stap scripts
+#
+
+prog=stap-buildpkg
+
+echo_usage () {
+  echo $"Usage: $prog scriptfile [options] [stap-options]"
+  echo $"Options:"
+  echo $"       -n pkg_name   		: specify a package name"
+  echo $"       -v pkg_version		: specify a package version"
+  echo $"       -c pkg_config  		: provide config parameters through a file"
+  echo $"       -p post_process script   : post processing script"
+  echo $"       -h help_config		: help text for config parameters"
+  echo $"       All additonal options will be passed to the stap command"
+}
+
+parse_args() {
+while [ -n "$1" ]; do
+     case $1 in
+        -n)                    PKG_NAME=$2
+				shift 2
+				echo $PKG_NAME
+                                ;;
+        -v)			PKG_VERSION=$2
+                                shift 2
+				echo $PKG_VERSION
+                                ;;
+        -c) 		        PKG_INPUT_CONFIG=$2
+				echo $PKG_INPUT_CONFIG
+				shift 2
+                                ;;
+        -p)			PKG_POST_SCRIPT=$2
+				echo $PKG_POST_SCRIPT
+				shift 2
+                                ;;
+        -h) 		        PKG_HELP_CONFIG=$2
+				echo $PKG_HELP_CONFIG
+                                shift 2
+                                ;;
+        *)                     STAP_OPTIONS=$@ 
+				echo $STAP_OPTIONS
+                                shift $#
+esac
+done
+}
+
+validate_args() {
+ if test -z $PKG_NAME; then
+   PKG_NAME=`basename $script .stp`
+ fi
+ if test -z $PKG_VERSION; then
+   PKG_VERSION=1.0
+ fi
+ if test -z $PKG_RELEASE; then
+   PKG_RELEASE="0"
+ fi
+ PKG_ARCH=`uname -i`
+}
+
+module_gen() {
+stap -p4 -m $PKG_NAME.ko $script $STAP_OPTIONS
+if [ -f $PKG_NAME.ko ]
+then
+        echo " Module $PKG_NAME.ko generated successfully ";
+else
+        echo " Module $PKG_NAME.ko generation failed. Exiting now... "
+        rm -rf $TEMPDIR
+        exit;
+fi
+}
+
+create_wrappers() {
+echo "Creating install wrapper from template..";
+echo "s/TEMPLATE_PKG_NAME/$PKG_NAME/g" > $TEMPDIR/sd-installer-patterns
+echo "s/TEMPLATE_PKG_VERSION/$PKG_VERSION/g" >> $TEMPDIR/sd-installer-patterns
+echo "s/TEMPLATE_PKG_RELEASE/$PKG_RELEASE/" >> $TEMPDIR/sd-installer-patterns
+echo "s/TEMPLATE_ARCH/$PKG_ARCH/g" >> $TEMPDIR/sd-installer-patterns
+
+sed -f $TEMPDIR/sd-installer-patterns template.buildpkg > $TEMPDIR/wrapper
+csplit -s $TEMPDIR/wrapper '/extractor template/'
+mv xx00 ${PKG_NAME}_install
+mv xx01 ${PKG_NAME}_binextractor
+
+chmod a+x ${PKG_NAME}_install
+rm $TEMPDIR/sd-installer-patterns
+rm $TEMPDIR/wrapper
+}
+
+copy_files_to_temp() {
+
+runtime=systemtap-runtime
+if test -z $(rpm -qa $runtime); then
+ runtime=systemtap 
+fi
+
+echo "using systemtap runtime executables from $runtime rpm"
+
+cp `rpm -ql $runtime | grep staprun | grep -v staprun.8.gz` $TEMPDIR/.
+cp `rpm -ql $runtime | grep stapio` $TEMPDIR/.
+
+echo "Copying Systemtap module.."
+mv $PKG_NAME.ko $TEMPDIR/ 
+echo "Copying config file.."
+echo "time=2m" > $TEMPDIR/$PKG_NAME.config
+if [ -f $PKG_INPUT_CONFIG ] && [ "$PKG_INPUT_CONFIG" != "" ]
+then 
+   cat $PKG_INPUT_CONFIG >> $TEMPDIR/$PKG_NAME.config
+fi
+# Copying help file, or adding an empty file if none is provided.
+if [ -f $PKG_HELP_CONFIG ] && [ "$PKG_HELP_CONFIG" != "" ]
+then
+        cp $PKG_HELP_CONFIG $TEMPDIR/$PKG_NAME.help
+else
+        echo " " > $TEMPDIR/$PKG_NAME.help
+fi
+echo "Copying the post processing script"
+# copying the post processing script, or adding an empty file if none is provided.
+if [ -f $PKG_POST_SCRIPT ] && [ "$PKG_POST_SCRIPT" != "" ]
+then
+    cp $PKG_POST_SCRIPT  $TEMPDIR/$PKG_NAME.post
+else
+   echo " " > $TEMPDIR/$PKG_NAME.post
+fi
+chmod +x $TEMPDIR/$PKG_NAME.post
+mv ${PKG_NAME}_install $TEMPDIR/
+}
+
+#------------------------------------------------------------------
+# Mainline script
+#------------------------------------------------------------------
+script=$1
+echo $script
+shift 1
+
+if test -z $script; then
+ echo "Please provide a script name"
+ echo_usage
+ exit
+fi
+
+if [ ! -f "$script" ]; then
+ echo "Script file does not exist"
+ echo "Please provide a valid script name"
+ echo_usage
+ exit
+fi
+
+ 
+#OPTS=`getopt -s bash -u -o 'n:v:R:c:p:h:' -- $@`
+
+#parse arguments
+parse_args $@
+validate_args
+
+# generate a tmp directory to store all components:remember to clean it up !
+TEMPDIR=/tmp/$PKG_NAME-$PKG_VERSION;
+mkdir  $TEMPDIR
+
+# Generate Stap module
+echo "Generating Stap module..";
+#module_gen
+
+#generate installer(wrapper) scripts
+#create_wrappers
+
+#copy systemtap-runtime files
+#copy_files_to_temp
+
+MY_PWD=`pwd`
+cd $TEMPDIR ; tar -cjf $PKG_NAME-$PKG_VERSION.tar.bz2 *
+cd $MY_PWD;
+
+echo "Building the self-extracting binary.."
+mkdir $TEMPDIR/$PKG_NAME
+mv $TEMPDIR/${PKG_NAME}_install $TEMPDIR/$PKG_NAME/
+tar -xjf $TEMPDIR/$PKG_NAME-$PKG_VERSION.tar.bz2 -C $TEMPDIR/$PKG_NAME/
+mv  ${PKG_NAME}_binextractor $TEMPDIR/${PKG_NAME}_binextractor 2>/dev/null
+
+MY_PWD=`pwd`
+cd $TEMPDIR; 
+tar -czf - $PKG_NAME/ >> ${PKG_NAME}_binextractor
+cd $MY_PWD;
+mv $TEMPDIR/${PKG_NAME}_binextractor ./$PKG_NAME-$PKG_VERSION
+echo "The installer is at `pwd`/$PKG_NAME-$PKG_VERSION"
+chmod +x $PKG_NAME-$PKG_VERSION
+
+#Cleanup the mess in temp dir
+rm -rf $TEMPDIR
+exit
diff --git a/scripts/build_pkg/template.buildpkg b/scripts/build_pkg/template.buildpkg
new file mode 100644
index 0000000..07d7926
--- /dev/null
+++ b/scripts/build_pkg/template.buildpkg
@@ -0,0 +1,142 @@
+#!/bin/bash
+RUN_DIR=`pwd`
+
+function run()
+{
+echo "Gathering data... Please wait"
+# Log file appears as systemtap_tcpipstat.out
+TS=`date "+%Y%m%d-%H:%M:%S"`
+export SYSTEMTAP_STAPIO=$RUN_DIR/stapio
+$RUN_DIR/staprun $RUN_DIR/TEMPLATE_PKG_NAME.ko > $RUN_DIR/TEMPLATE_PKG_NAME.out 2>&1 &
+stappid=$!
+sleep $time
+echo "Completed"
+mv $RUN_DIR/TEMPLATE_PKG_NAME.out $RUN_DIR/TEMPLATE_PKG_NAME_${TS}.out
+echo "Output file TEMPLATE_PKG_NAME_${TS}.out is in $RUN_DIR directory"
+kill -SIGINT $stappid
+echo "Postprocessing..."
+$RUN_DIR/TEMPLATE_PKG_NAME.post $RUN_DIR/TEMPLATE_PKG_NAME_${TS}.out
+rm $RUN_DIR/staprun
+rm $RUN_DIR/stapio
+}
+
+function start()
+{
+ # check to see if tapset is installed
+echo "Gathering data..."
+export SYSTEMTAP_STAPIO=$RUN_DIR/stapio
+$RUN_DIR/staprun $RUN_DIR/TEMPLATE_PKG_NAME.ko > $RUN_DIR/TEMPLATE_PKG_NAME_tmp.out 2>&1 &
+echo "Run TEMPLATE_PKG_NAME --stop to stop the script and process output"
+}
+
+function stop()
+{
+TS=`date "+%Y%m%d-%H:%M:%S"`
+echo "Stopping systemtap script"
+ps -ef | grep TEMPLATE_PKG_NAME | grep stapio | awk '{print $2}' | xargs kill -SIGINT
+echo "Done"
+mv $RUN_DIR/TEMPLATE_PKG_NAME_tmp.out $RUN_DIR/TEMPLATE_PKG_NAME_${TS}.out
+echo "Output file TEMPLATE_PKG_NAME_${TS}.out is in $RUN_DIR directory"
+echo "Postprocessing..."
+$RUN_DIR/TEMPLATE_PKG_NAME.post $RUN_DIR/TEMPLATE_PKG_NAME_${TS}.out
+rm $RUN_DIR/staprun
+rm $RUN_DIR/stapio
+}
+
+function helptext()
+{
+ echo " "
+ echo "USAGE: TEMPLATE_PKG_NAME [options] [parameters]"
+ echo " "
+ echo "Options: "
+ echo " --run -r	Runs the scripts for x minutes where x can be passed as a parameter using time=[x]. "
+ echo "		The default value is 10 minutes. Post processing is performed after the script completes."
+ echo " --start -s	Runs the script as a background process"
+ echo " --stop -x	Stops the script and performs post processing"
+ echo " --all		Runs the script and processes the output"
+ echo " --help		Displays this usage text"
+ echo " "
+ echo "Parameters: "
+ echo " "
+ echo " time=[x]	x is in minutes. Runs the script for x minutes. valid for --run(-r)"
+ echo "		--start(-s) or --all(-a) options only"
+ 	cat $RUN_DIR/TEMPLATE_PKG_NAME.help
+ echo ""
+}
+
+function parseparameters()
+{
+for i in $@
+do
+        tmpa=${1%%=*}
+        tmpb=${1#$tmpa=}
+        export $tmpa=$tmpb
+        shift
+done
+}
+
+
+option=$1
+shift
+
+
+  FILE=$RUN_DIR/TEMPLATE_PKG_NAME.config
+   # make sure file exist and readable
+   if [ ! -f $FILE ]; then
+  	echo "$FILE : does not exist"
+   elif [ ! -r $FILE ]; then
+  	echo "$FILE: cannot read"
+   fi
+
+
+
+   exec 3<&0
+   exec 0<$FILE
+   while read line
+	do
+        tmpa=${line%%=*}
+        tmpb=${line#$tmpa=}
+        declare $tmpa=$tmpb
+	done
+   exec 0<&3
+
+   for i in $@
+	do
+	        tmpa=${1%%=*}
+        	tmpb=${1#$tmpa=}
+	        export $tmpa=$tmpb
+        	shift
+	done
+
+
+case $option in
+	-r | --run ) 	 			
+				run
+				exit
+				;;
+        -s | --start )         
+				start
+				exit
+                                ;;
+        -h | --help )           helptext
+                                exit
+                                ;;
+        -x | --stop )		stop
+				exit
+				;;
+	-a | --all )		run
+				exit
+				;;
+        * )                     helptext
+                                exit 1
+esac
+
+## extractor template
+#!/bin/sh -e
+sed -e '1,/^exit$/d' "$0" | tar -xzf - 
+cd TEMPLATE_PKG_NAME 
+ ./TEMPLATE_PKG_NAME_install $@
+cd ..
+rm -f TEMPLATE_PKG_NAME/TEMPLATE_PKG_NAME.*
+rm -f TEMPLATE_PKG_NAME/TEMPLATE_PKG_NAME_install
+exit