tgolem - automated execution framework

tgolem is a framework for automated execution of tasks on remote hosts via ssh.

tgolem is free software released under the GPL. See the file COPYING for details.

The big picture of using tgolem looks like this:

1) tgolem executes scripts on some hosts and writes a "result file".

2) report generators read the result file and create reports
   from it.

Steps 1 and 2 have a n-to-m relation. You can combine any number of result files and create any number of reports from them. Step 1 is called the "online" part, step 2 is the "offline" part of running tgolem.

The following sections will give some details on these steps and the tgolem architecture.

tgolem itself needs no installation, it is pure Perl and can be run out of its source directory. See the following section for required Perl modules.

tgolem needs the following Perl modules that can be obtained from CPAN:

Config::Scoped
Error
HTML::Entities
HTML::Template::Expr
Log::Agent
Log::Agent::Driver::File
Test::More
YAML

The report generation scripts in the "cacao" directory also need:

Statistics::Descriptive

All other required modules should be contained in a recent Perl installation. You can use make uses to get a list of all modules used by tgolem. Note that some may be optional.

The most comfortable way to install Perl modules is using packages for your OS distribution. The next best way is to use the CPAN installer shell like:

perl -MCPAN -e shell

You can run make test to run tgolem's regression tests. All of them should pass. Please report any problems (see end of this file).

If you edit test.conf to fit your system, you can do make runtest to run the job test.pl. Look for <<< CONFIG markers in test.conf. Note that errors and warnings are part of this test. The results will be listed in log/index.html.

(online)

Somehow tgolem has to know which scripts to execute on which hosts in which sequence. This is controlled by a "job". A "job" is just a Perl script using some special tgolem commands. This makes tgolem extremely flexible because you can use the full Perl language to control execution. There are special commands (in the module Tgolem::Code) for dealing with scripts, parallel execution and errors in a simple way.

It is important to understand that while it controls the scripts executed on remote hosts, the tgolem job itself runs locally. tgolem does not require perl on the remote hosts.

(online [and offline])

The config file contains info on the hosts and the scripts to execute. It is written in the syntax accepted by the Config::Scoped module.

(online)

A tgolem script is a sequence of commands (usually shell commands) together with some additional info.

Scripts do not support any fancy control flow statements, that's what jobs are for. The commands of a script are just executed one after the other. If one command fails, the whole script is considered to have failed.

(online)

A scripter is a tgolem module that knows how to execute a certain kind of script. For example Tgolem::Scripter::Shell executes shell scripts. A scripter can do arbitrary pre-processing of the script's commands and arbitrary post-processing of the script's output. The scripter also logs all input to and output from the script.

(Technical note: The logging is actually done by something called the "wrapper", not the scripter itself. This is not visible for the user, however.)

(the interface from online to offline)

The ultimate purpose of a scripter is to produce results from the script's output. The smallest unit of tgolem results is the "message". A message may be something like:

*) an info, error or warning message produced by a command.

*) a key-value pair produced by a gathering scripter.

*) success or failure of a script.

Each message happens within a certain "context", that specifies on which host, by which script, in which configuration, etc. the message was produced.

tgolem collects the result messages and stores them in the "result file". The Perl module Tgolem::Results can then be used by report generators to read this file.

Once you have written the config file and a tgolem job, you can run tgolem like this:

./tgolem -f configfile -D log.dir=yourlogdir -D log.print=1 -D log.color=1

tgolem will put all logs and the result file into "yourlogdir". The last two options make tgolem print nicely colored messages to the terminal. They do not influence the results in any way.

Once you have a result file, you can generate a basic HTML report from it:

./html_report -f configfile -r yourlogdir/results -D log.dir=yourlogdir

The report will be written to yourlogdir/index.html.

Report generation will depend very much on the application. You can look at the "cacao" directory for examples of some ugly ad-hoc report generators.

I'm always happy to get feedback. Please contact me at <edwin.steiner@gmx.net>. Please include tgolem in the subject line.