RRD-BEGINNERS(1)             rrdtool             RRD-BEGINNERS(1)



NNAAMMEE
       rrd-beginners - RRDtool Beginners' Guide

SSYYNNOOPPSSIISS
       Helping new RRDtool users to understand the basics of RRD-
       tool

DDEESSCCRRIIPPTTIIOONN
       This manual is an attempt to assist beginners in under-
       standing the concepts of RRDtool. It sheds a light on dif-
       ferences between RRDtool and other databases. With help of
       an example, it explains the structure of RRDtool database.
       This is followed by an overview of the "graph" feature of
       RRDtool.  At the end, it has sample scripts that illus-
       trate the usage/wrapping of RRDtool within Shell or Perl
       scripts.

       WWhhaatt mmaakkeess RRRRDDttooooll ssoo ssppeecciiaall??

       RRDtool is GNU licensed software developed by Tobias
       Oetiker, a system manager at the Swiss Federal Institute
       of Technology. Though it is a database, there are distinct
       differences between RRDtool databases and other databases
       as listed below:

       +o   RRDtool stores data; that makes it a back-end tool.
           The RRDtool command set allows the creation of graphs;
           that makes it a front-end tool as well. Other
           databases just store data and can not create graphs.

       +o   In case of linear databases, new data gets appended at
           the bottom of the database table. Thus its size keeps
           on increasing, whereas the size of an RRDtool database
           is determined at creation time. Imagine an RRDtool
           database as the perimeter of a circle. Data is added
           along the perimeter. When new data reaches the start-
           ing point, it overwrites existing data. This way, the
           size of an RRDtool database always remains constant.
           The name "Round Robin" stems from this behavior.

       +o   Other databases store the values as supplied. RRDtool
           can be configured to calculate the rate of change from
           the previous to the current value and store this
           information instead.

       +o   Other databases get updated when values are supplied.
           The RRDtool database is structured in such a way that
           it needs data at predefined time intervals. If it does
           not get a new value during the interval, it stores an
           UNKNOWN value for that interval. So, when using the
           RRDtool database, it is imperative to use scripts that
           run at regular intervals to ensure a constant data
           flow to update the RRDtool database.

       RRDtool is designed to store time series of data. With
       every data update, an assosiated time stamp is stored.
       Time is always expressed in seconds passed since epoch
       (01-01-1970). RRDtool can be installed on Unix as well as
       Windows. It comes with a command set to carry out various
       operations on RRD databases. This command set can be
       accessed from the command line, as well as from Shell or
       Perl scripts. The scripts act as wrappers for accessing
       data stored in RRDtool databases.

       UUnnddeerrssttaannddiinngg bbyy aann eexxaammppllee

       The structure of an RRD database is different than other
       linear databases.  Other databases define tables with
       columns, and many other parameters. These definitions
       sometimes are very complex, especially in large databases.
       RRDtool databases are primarily used for monitoring pur-
       poses and hence are very simple in structure. The parame-
       ters that need to be defined are variables that hold val-
       ues and archives of those values. Being time sensitive, a
       couple of time related parameters are also defined.
       Because of its structure, the definition of an RRDtool
       database also includes a provision to specify specific
       actions to take in the absence of update values. Data
       Source (DS), heartbeat, Date Source Type (DST), Round
       Robin Archive (RRA), and Consolidation Function (CF) are
       some of the terminologies related to RRDtool databases.

       The structure of a database and the terminology associated
       with it can be best explained with an example.

        rrdtool create target.rrd \
                --start 1023654125 \
                --step 300 \
                DS:mem:GAUGE:600:0:671744 \
                RRA:AVERAGE:0.5:12:24 \
                RRA:AVERAGE:0.5:288:31

       This example creates a database named _t_a_r_g_e_t_._r_r_d. Start
       time (1'023'654'125) is specified in total number of sec-
       onds since epoch (time in seconds since 01-01-1970). While
       updating the database, the update time is also specified.
       This update time MUST be large (later) then start time and
       MUST be in seconds since epoch.

       The step of 300 seconds indicates that database expects
       new values every 300 seconds. The wrapper script should be
       scheduled to run every sstteepp seconds so that it updates the
       database every sstteepp seconds.

       DS (Data Source) is the actual variable which relates to
       the parameter on the device that is monitored. Its syntax
       is

        DS:variable_name:DST:heartbeat:min:max

       DDSS is a key word. "variable_name" is a name under which
       the parameter is saved in the database. There can be as
       many DSs in a database as needed. After every step inter-
       val, a new value of DS is supplied to update the database.
       This value is also called Primary Data Point ((PPDDPP)). In our
       example mentioned above, a new PDP is generated every 300
       seconds.

       Note, that if you do NOT supply new datapoints exactly
       every 300 seconds, this is not a problem, RRDtool will
       interpolate the data accordingly.

       DDSSTT (Data Source Type) defines the type of the DS. It can
       be COUNTER, DERIVE, ABSOLUTE, GAUGE. A DS declared as
       COUNTER will save the rate of change of the value over a
       step period. This assumes that the value is always
       increasing (the difference between the current and the
       previous value is greater than 0). Traffic counters on a
       router are an ideal candidate for using COUNTER as DST.
       DERIVE is the same as COUNTER, but it allows negative val-
       ues as well. If you want to see the rate of _c_h_a_n_g_e in free
       diskspace on your server, then you might want to use the
       DERIVE data type. ABSOLUTE also saves the rate of change,
       but it assumes that the previous value is set to 0. The
       difference between the current and the previous value is
       always equal to the current value. Thus it just stores the
       current value divided by the step interval (300 seconds in
       our example). GAUGE does not save the rate of change. It
       saves the actual value itself. There are no divisions or
       calculations. Memory consumption in a server is a typical
       example of gauge. The difference between the different
       types DSTs can be explained better with the following
       example:

        Values       = 300, 600, 900, 1200
        Step         = 300 seconds
        COUNTER DS   =    1,  1,   1,    1
        DERIVE DS    =    1,  1,   1,    1
        ABSOLUTE DS  =    1,  2,   3,    4
        GAUGE DS     = 300, 600, 900, 1200

       The next parameter is hheeaarrttbbeeaatt. In our example, heartbeat
       is 600 seconds. If the database does not get a new PDP
       within 300 seconds, it will wait for another 300 seconds
       (total 600 seconds).  If it doesn't receive any PDP within
       600 seconds, it will save an UNKNOWN value into the
       database. This UNKNOWN value is a special feature of RRD-
       tool - it is much better than to assume a missing value
       was 0 (zero) or any other number which might also be a
       valid data value.  For example, the traffic flow counter
       on a router keeps increasing. Lets say, a value is missed
       for an interval and 0 is stored instead of UNKNOWN. Now
       when the next value becomes available, it will calculate
       the difference between the current value and the previous
       value (0) which is not correct. So, inserting the value
       UNKNOWN makes much more sense here.

       The next two parameters are the minimum and maximum value,
       respectively. If the variable to be stored has predictable
       maximum and minimum values, this should be specified here.
       Any update value falling out of this range will be stored
       as UNKNOWN.

       The next line declares a round robin archive (RRA). The
       syntax for declaring an RRA is

        RRA:CF:xff:step:rows

       RRA is the keyword to declare RRAs. The consolidation
       function (CF) can be AVERAGE, MINIMUM, MAXIMUM, and LAST.
       The concept of the consolidated data point (CDP) comes
       into the picture here. A CDP is CFed (averaged, maxi-
       mum/minimum value or last value) from _s_t_e_p number of PDPs.
       This RRA will hold _r_o_w_s CDPs.

       Lets have a look at the example above. For the first RRA,
       12 (steps) PDPs (DS variables) are AVERAGEed (CF) to form
       one CDP. 24 (rows) of theses CDPs are archived. Each PDP
       occurs at 300 seconds. 12 PDPs represent 12 times 300 sec-
       onds which is 1 hour. It means 1 CDP (which is equal to 12
       PDPs) represents data worth 1 hour. 24 such CDPs represent
       1 day (1 hour times 24 CDPs). This means, this RRA is an
       archive for one day. After 24 CDPs, CDP number 25 will
       replace the 1st CDP. The second RRA saves 31 CDPs; each
       CPD represents an AVERAGE value for a day (288 PDPs, each
       covering 300 seconds = 24 hours). Therefore this RRA is an
       archive for one month. A single database can have many
       RRAs. If there are multiple DSs, each individual RRA will
       save data for all the DSs in the database. For example, if
       a database has 3 DSs and daily, weekly, monthly, and
       yearly RRAs are declared, then each RRA will hold data
       from all 3 data sources.

       GGrraapphhiiccaall MMaaggiicc

       Another important feature of RRDtool is its ability to
       create graphs. The "graph" command uses the "fetch" com-
       mand internally to retrieve values from the database. With
       the retrieved values it draws graphs as defined by the
       parameters supplied on the command line. A single graph
       can show different DS (Data Sources) from a database. It
       is also possible to show the values from more than one
       database in a single graph. Often, it is necessary to per-
       form some math on the values retrieved from the database
       before plotting them. For example, in SNMP replies, memory
       consumption values are usually specified in KBytes and
       traffic flow on interfaces is specified in Bytes. Graphs
       for these values will be more meaningful if values are
       represented in MBytes and mbps. The RRDtool graph command
       allows to define such conversions. Apart from mathematical
       calculations, it is also possible to perform logical oper-
       ations such as greater than, less than, and if/then/else.
       If a database contains more than one RRA archive, then a
       question may arise - how does RRDtool decide which RRA
       archive to use for retrieving the values? RRDtool looks at
       several things when making its choice. First it makes sure
       that the RRA covers as much of the graphing time frame as
       possible. Second it looks at the resolution of the RRA
       compared to the resolution of the graph. It tries to find
       one which has the same or higher better resolution. With
       the "-r" option you can force RRDtool to assume a differ-
       ent resolution than the one calculated from the pixel
       width of the graph.

       Values of different variables can be presented in 5 dif-
       ferent shapes in a graph - AREA, LINE1, LINE2, LINE3, and
       STACK. AREA is represented by a solid colored area with
       values as the boundary of this area. LINE1/2/3 (increasing
       width) are just plain lines representing the values. STACK
       is also an area but it is "stack"ed on top AREA or
       LINE1/2/3. Another important thing to note is that vari-
       ables are plotted in the order they are defined in the
       graph command. Therefore care must be taken to define
       STACK only after defining AREA/LINE. It is also possible
       to put formatted comments within the graph.  Detailed
       instructions can be found in the graph manual.

       WWrraappppiinngg RRRRDDttooooll wwiitthhiinn SShheellll//PPeerrll ssccrriipptt

       After understanding RRDtool it is now a time to actually
       use RRDtool in scripts. Tasks involved in network manage-
       ment are data collection, data storage, and data
       retrieval. In the following example, the previously cre-
       ated target.rrd database is used. Data collection and data
       storage is done using Shell scripts. Data retrieval and
       report generation is done using Perl scripts. These
       scripts are shown below:

       _S_h_e_l_l _s_c_r_i_p_t _(_c_o_l_l_e_c_t_s _d_a_t_a_, _u_p_d_a_t_e_s _d_a_t_a_b_a_s_e_)

        #!/bin/sh
        a=0
        while [ "$a" == 0 ]; do
        snmpwalk -c public 192.168.1.250 hrSWRunPerfMem > snmp_reply
            total_mem=`awk 'BEGIN {tot_mem=0}
                                  { if ($NF == "KBytes")
                                    {tot_mem=tot_mem+$(NF-1)}
                                  }
                            END {print tot_mem}' snmp_reply`
            # I can use N as a replacement for the current time
            rrdtool update target.rrd N:$total_mem
            # sleep until the next 300 seconds are full
            perl -e 'sleep 300 - time % 300'
        done # end of while loop

       _P_e_r_l _s_c_r_i_p_t _(_r_e_t_r_i_e_v_e_s _d_a_t_a _f_r_o_m _d_a_t_a_b_a_s_e _a_n_d _g_e_n_e_r_a_t_e_s
       _g_r_a_p_h_s _a_n_d _s_t_a_t_i_s_t_i_c_s_)

        #!/usr/bin/perl -w
        # This script fetches data from target.rrd, creates a graph of memory
        # consumption on the target (Dual P3 Processor 1 GHz, 656 MB RAM)

        # call the RRD perl module
        use lib qw( /usr/local/rrdtool-1.0.41/lib/perl ../lib/perl );
        use RRDs;
        my $cur_time = time();                # set current time
        my $end_time = $cur_time - 86400;     # set end time to 24 hours ago
        my $start_time = $end_time - 2592000; # set start 30 days in the past






































        # fetch average values from the RRD database between start and end time
        my ($start,$step,$ds_names,$data) =
            RRDs::fetch("target.rrd", "AVERAGE",
                        "-r", "600", "-s", "$start_time", "-e", "$end_time");
        # save fetched values in a 2-dimensional array
        my $rows = 0;
        my $columns = 0;
        my $time_variable = $start;
        foreach $line (@$data) {
          $vals[$rows][$columns] = $time_variable;
          $time_variable = $time_variable + $step;
          foreach $val (@$line) {
                  $vals[$rows][++$columns] = $val;}
          $rows++;
          $columns = 0;
        }
        my $tot_time = 0;
        my $count = 0;
        # save the values from the 2-dimensional into a 1-dimensional array
        for $i ( 0 .. $#vals ) {
            $tot_mem[$count] = $vals[$i][1];
            $count++;
        }
        my $tot_mem_sum = 0;
        # calculate the total of all values
        for $i ( 0 .. ($count-1) ) {
            $tot_mem_sum = $tot_mem_sum + $tot_mem[$i];
        }
        # calculate the average of the array
        my $tot_mem_ave = $tot_mem_sum/($count);
        # create the graph
        RRDs::graph ("/images/mem_$count.png",   \
                    "--title= Memory Usage",    \
                    "--vertical-label=Memory Consumption (MB)", \
                    "--start=$start_time",      \
                    "--end=$end_time",          \
                    "--color=BACK#CCCCCC",      \
                    "--color=CANVAS#CCFFFF",    \
                    "--color=SHADEB#9999CC",    \
                    "--height=125",             \
                    "--upper-limit=656",        \
                    "--lower-limit=0",          \
                    "--rigid",                  \
                    "--base=1024",              \
                    "DEF:tot_mem=target.rrd:mem:AVERAGE", \
                    "CDEF:tot_mem_cor=tot_mem,0,671744,LIMIT,UN,0,tot_mem,IF,1024,/",\
                    "CDEF:machine_mem=tot_mem,656,+,tot_mem,-",\
                    "COMMENT:Memory Consumption between $start_time",\
                    "COMMENT:    and $end_time                     ",\
                    "HRULE:656#000000:Maximum Available Memory - 656 MB",\
                    "AREA:machine_mem#CCFFFF:Memory Unused",   \
                    "AREA:tot_mem_cor#6699CC:Total memory consumed in MB");
        my $err=RRDs::error;
        if ($err) {print "problem generating the graph: $err\n";}
        # print the output
        print "Average memory consumption is ";
        printf "%5.2f",$tot_mem_ave/1024;
        print " MB. Graphical representation can be found at /images/mem_$count.png.";

AAUUTTHHOORR
       Ketan Patel <k2pattu@yahoo.com>



1.2.15                      2006-07-14           RRD-BEGINNERS(1)
