Let’s assume you have written a program that performs some computations and you want to time and to evaluate the execution time. In the example below you find a simple shell script that performs a number of calls to a python script that computes the faculty of a given integer parameter.

Test program (fac.sh). 

#! /bin/bash

for t in $(seq 1 3);
do
    for i in $(seq 0 1000 50000);
    do
        echo "n=$i"
        time python fac.py $i;
    done
done

You could capture the output in a file called fac.res (for result data).

$./fac.sh > fac.res 2>&1

The content of fac.res looks like this:

n=0

real    0m0.014s
user    0m0.016s
sys     0m0.000s
n=1000

real    0m0.014s
user    0m0.008s
sys     0m0.000s
n=2000

real    0m0.018s
user    0m0.012s
sys     0m0.012s
n=3000

real    0m0.023s
user    0m0.020s
sys     0m0.004s
n=4000

real    0m0.032s
user    0m0.016s
sys     0m0.004s
...

When we look at the output above we see that the value of the parameter n is printed at start. After that we see the output of the time command that consists of real, user, and system time.

To parse this results file we have to create a Plotmeister template file.

We have two options:

The simple template format should be enough in most cases. To create a simple template we simply copy the output of one experimental run into an editor, e.g., from

n=0

real    0m0.014s
user    0m0.016s
sys     0m0.000s

Then we remove the empty lines and start to substitute the values that should be retrieved by variables. We start with n=0. We are interested in the value of n. Thus, we write n=@{i,n}. That means that we want to read the value of n which is an integer and this value should be named n in the output. Hence, @{} denotes the variable. We proceed the same way with the real, user and sys time. But in these cases we need to tell Plotmeister that it should look for floating point values. Thus, we write for example @{f,real}. The character # has a special meaning in the simple template and means that "multiple whitespaces are allowed".

At last, we have to tell Plotmeister how often one of these lines does occur in the output. To do so, we first have to enumerate the lines 1:,2:,…

Then we tell in the states line how often a line occurs in the sequence of the given lines. In our example each line is printed only once before the next line starts. Thus, we say states: 1 2 3 4.

The code below shows the resulting simple template for our parsing problem:

Template fac.tpl (simple). 

1:n=@{i,n}
2:real#0m@{f,real}s
3:user#0m@{f,user}s
4:sys#0m@{f,sys}s
states: 1 2 3 4

Internally, the simple template is transformed into a sequence of regular expressions. In case that the users want to have direct control over the parsing process or they need to extract values that are not provided in the simple template (e.g., date formats, etc.) then you can write an extended template.

The following extended template shows the corresponding template for our example. Here, we again declare the lines that should be parsed. In contrast to the simple template we specify the regular expression directly. After all lines have been defined we have to identify the variables by naming them. For example 1,1:n means that the first variable in line 1 is called n.

Template fac.tpl (ext). 

1:n=(\d+)
2:real\s+0m([\d\.]+)s
3:user\s+0m([\d\.]+)s
4:sys\s+0m([\d\.]+)s
1,1:n
2,1:real
3,1:user
4,1:sys
states: 1 2 3 4

Now we can finally start parsing the data.

To do so we print our results file to stdout and pipe it to the Plotmeister parser.

In case you defined a simple template you would call:

$ cat fac.res | pm_parse_simple.py -t fac.tpl

or for an extended template you would execute:

$ cat fac.res | pm_parse_ext.py -t fac.tpl

The parser output will look in both cases like this:

#################
# cols=5
# rpt_cnt  n    real    sys     user
#################
0       0       0.038   0.008   0.012
1       1000    0.016   0.004   0.012
2       2000    0.025   0.008   0.012
3       3000    0.023   0.008   0.016
4       4000    0.027   0.008   0.020
5       5000    0.028   0.000   0.028
6       6000    0.040   0.000   0.036
7       7000    0.046   0.004   0.040
8       8000    0.056   0.004   0.052
9       9000    0.086   0.008   0.064
10      10000   0.084   0.004   0.080
11      11000   0.102   0.004   0.096
...

The parser outputs data in PMD file format. The first column contains a unique line number. The other columns are labeled as specified in the template, e.g., n, real, user.

In order to save the parser output we redirect stdout into a file called fac.pmd.

$ cat fac.res | pm_parse_simple.py -t fac.tpl > fac.pmd