Datasinks

Documentation / Standard Edition / User Guide / Agent / Datasinks

Datasinks

The following section describes all available datasinks. The MyARM agent implementation or daemons are using datasinks to export measured ARM data to a configured datasink destination.

Null datasink

The null datasink is a really no operation datasink. Say you have setup a complete environment with ARM instrumentation enabled, but in some circumstances you want no transactions to be measured. Just configure the use of this null datasink and no data will be recorded at all:

# agent uses a datasink named noop
agent.sink.name = noop
# noop defines a null datasink type
noop.type = null

SQLite datasink

The sqlite3 datasink uses the SQLite database as described in SQLite database.

MySQL datasink

The mysql datasink uses the MySQL database as described in section MySQL database.

File datasink

Using the file datasink all ARM data are stored in a flat file. This file can be read by the myarmfiled daemon to forward the ARM data to another destination like a TCP connection or a real database. The main purpose of this datasink in conjunction with the myarmfiled daemon is to decouple writing ARM data to the database from the instrumented application using simple file IO.

In addition to the standard datasink properties the file datasink uses the following properties:

<name>.workfile
specifies the complete filename (including directory names) for the work file. The process ID of the running process will be appended to make the file unique. The ARM data is written to this file and when its closed it is moved to the configured directory.

Default is /tmp/myarmfile.data.

<name>.rolling.seconds
specifies the number of seconds after which a new workfile will be used. The old workfile will be moved to the configured directory.

Default is 60 seconds.

<name>.rolling.size
specifies the maximal size in bytes of the workfile. If the workfile gets bigger a new workfile is opened and the old workfile is moved to the configured directory.

Default is 128 KB.

<name>.diskusage.max_used (New since "1.3.x.0")
specifies the maximal size in bytes of all ARM data files in the myarmfiled directory. If this limit is reached any new ARM data files will be dropped and an error will be reported.

Default is 100 MB.

<name>.diskusage.min_free (New since "1.3.x.0")
specifies the minimal free size in bytes of the file system the myarmfiled directory resides. If this limit is reached any new ARM data files will be dropped and an error will be reported.

Default is 200 MB.

basic.filestorage.reader.directory (New since "2.1.x.0")
specifies the directory to move closed files to. See basic file storage configuration section.

A sample file datasink configuration can look like:

# specify datasink name for the agent
agent.sink.name = sink_file
# datasink file type
sink_file.type = file
# set up work file for file datasink
sink_file.workfile = /opt/myarm/var/myarmfile.data
# time interval in seconds to use a new work file and move
# the old to the myarmfiled directory.
sink_file.rolling.seconds = 60
# number of maximal bytes for the work file. If this limit is
# reached the current work file is closed, moved to the
# myarmfiled directory and a new work file is created. (128KB)
sink_file.rolling.size = 128KB
# only use up to 100 MB of ARM data files, if this limit 
# is reached new ARM data files are dropped.
sink_file.diskusage.max_used = 100MB
# at least 200 MB of disk space should be left free
sink_file.diskusage.min_free = 200MB

TCP datasink

The tcp datasink is used in conjunction with the myarmtcpd tool. It connects to the myarmtcpd program and hands all ARM data to the myarmtcpd process through a TCP socket connection. Note that the tcp datasink retries to connect to the myarmtcpd daemon periodically as long as ARM data available to send to the myarmtcpd daemon. If there is no ARM data available to send for at least <name>.connection.idle time the connection is closed.

In addition to the standard datasink properties the tcp datasink uses the following properties:

<name>.host
name of the host to connect to the myarmtcpd process.

Default is localhost.

<name>.port
port number the myarmtcpd process is listening and accepts connections.

Default is 5557.

<name>.connection.idle (New since "1.3.x.0")
number of seconds to wait before closing the connection to the myarmtcpd if no ARM data needed to be send.

Default is 300s. (New since "1.3.x.7") is the minimum of 30s and the maximum of 3600s for the connection idle time.

<name>.connection.keepalive (New since "1.2.x.5")
number of seconds to send a keep alive message to the myarmtcpd if no ARM data are send.

Default is 60s. (New since "1.3.x.7") is the minimum of 30s and the maximum of the half of the <name>.connection.idle property for the keepalive time.

<name>.connection.reconnect (New since "1.4.x.0")
defines an interval value in seconds to wait before MyARM retries to connect to the myarmtcpd process again. Default is 30 second, the minimum is 10s and the maximum is 300s.
<name>.readwrite.timeout
timeout value in milliseconds to wait for reads or writes of ARM data through the connection.

Default is 500ms. (New since "1.4.x.2") is the minimum of 250ms and the maximum of 2000ms.

<name>.reply.timeout (New since "1.3.x.7")
timeout value in milliseconds to wait for a reply from the myarmtcpd daemon.

Default is 2000ms, the minimum is 500ms and the maximum is 4000ms.

Configuration example

# agent uses a datasink named sink_tcp
agent.sink.name = sink_tcp
# set up sink type
sink_tcp.type = tcp
# set up host myarmtcpd is running on
sink_tcp.host  = localhost
# set up port myarmtcpd is listening for incoming connections
sink_tcp.port = 5557
# read/write time out interval in milliseconds
sink_tcp.readwrite.timeout = 1000
# the interval for reconnecting to the daemon in seconds
sink_tcp.connection.reconnect = 60
# number of seconds to send a keep alive message if no data
# was sent to the myarmtcpd
sink_tcp.connection.keepalive = 60
# number of seconds to close the connection to myarmtcpd if no
# ARM data was sent to the server
sink_tcp.connection.idle = 300

Route datasink

The route datasink can be used to route measured data to more than one datasink. This can be useful in many cases. For example you want to store all measured ARM data in a database, but also to be passed through the trigger datasink to trigger some action based on measurements. The following properties can be used for the route datasink:

<name>.sinks
specifies a list of datasink names to route data to. For example:
# agent uses a datasink named sink_route
agent.sink.name = sink_route
# sink_route defines a route datasink type
sink_route.type = route
# the following sink names are configured for the route sink.
sink_route.sinks = tcp1 trigger1
# route all transaction of the application ImportantApp to the
# trigger sink
trigger1.filter.tran = filter tran=ImportantApp:*
# a TCP datasink sending all ARM data to a running myarmtcpd
tcp1.type = tcp
tcp1.host = myhost
tcp1.port = 10815
# a Trigger datasink logging all failed transactions
trigger1.type = trigger
trigger1.tran.list = failed
trigger1.failed.filter=match tran=*:* {tran.status==tran:failed}
trigger1.failed.execute= log: %a.%t failed
<sinkname>.filter.app
defines a filter application and transaction instances for the sink named sinkname .
<sinkname>.filter.tran
defines a filter transaction instances for the sink named sinkname.

Routing filter

For both application and/or transaction filter the following EBNF grammar describes the syntax of the filter where the following terminals are

APPNAME
application definition name of the current transaction or an asterix for matching any application definition name.
TRANNAME
transaction definiiton name of the current transaction or an asterix for matching any transaction definition name.
input ::= "FILTER" not expression next_expression
not ::= empty
| "NOT"
next_expression ::= empty
| "OR" expression
| "AND" expression
expression ::= "TRAN" "=" APPNAME ":" TRANNAME
| "APP "=" APPNAME
empty ::=

Trigger datasink

The trigger datasink can be used to evaluate various expressions online and to execute some action based on the result of the evaluation. The following properties can be used for the trigger datasink:

<name>.tran.list
specifies a list of trigger names to evaluate for each measured transaction.
<name>.<triggername>.filter
defines the filter for the trigger named <triggername>.
<name>.<triggername>.execute
defines the action to execute if the corresponding filter evaluates to true.
<name>.<triggername>.stop
if set to true, stop processing of defined triggers.

Transaction trigger filter

A transaction trigger filter defines the set of transaction instances to evaluate trigger expressions. Filters are defined by transaction and/or application definition names. A transaction filter is defined as follows:

match tran'='<appName>':'<tranName> '{' <expression> '}' 
      [or|and '{' <expression> '}']

where

match
is the literal string starting a trigger filter.
tran
is the literal string indicating a transaction filter.
<appName>
is the application definition name or the asterisk character ('*') matching any application definition name.
<tranName>
is the transaction definition name or the the asterisk character ('*') matching any transaction definition name.
<expression>
is the expression to evaluate for each transaction instance matching the application and/or transaction definition names specified before.
or
logical or-operator between two expressions.
and
logical and-operator between two expressions.
Transaction expression

A transaction expression defined in a transaction filter is evaluated for each transaction instance. The expression can use various values of the current transaction instance for its evaluation. For example it is very easy to execute some action for any failed transaction:

match tran=*:* { tran.status == tran:failed }

The field references the status of the current transaction and is compared to the status value.

Currently the following transaction values are supported in an expression:

tran.duration
duration (response time) of the current transaction.
tran.status
execution status of the current transaction.

The following section describes the complete filter in EBNF (Extended Backus-Naur Form).

Transaction filter EBNF

The following EBNF (Extended Backus-Naur Form) grammar describes the syntax of the complete transaction filter where the following terminals are

APPNAME
application definition name of the current transaction.
TRANNAME
transaction definition name of the current transaction.
METNAME
metric definition name of an associated metric of the current transaction.
NUMBER
a numeric constant.
input ::= "MATCH" "TRAN" "=" match_tran match_expression
match_tran ::= APPNAME ":" TRANNAME
match_expression ::= "{" expression "}" match_logical
match_logical ::= empty
| "OR" match_expression
| "AND" match_expression
expression ::= "TRAN" tran_expr
| NUMBER
| expression "+" expression
| expression "-" expression
| expression "*" expression
| expression "/" expression
| expression "<" "=" expression
| expression ">" "=" expression
| expression "=" "=" expression
| expression "!" "=" expression
| expression "<" expression
| expression ">" expression
| "!" expression
| "(" expression ")"
tran_expr ::= "." tran_field
| ":" tran_const
tran_field ::= "STATUS" | "DURATION" | "METRIC" "." METNAME
tran_const ::= "GOOD" | "FAILED" | "ABORTED" | "UNKNOWN"
empty ::=

Trigger action

The trigger action defines which action should be executed if the trigger expression evaluated to true. The following actions are currently supported:

log:
logs the following string to the configured log facility. For example:
log: info: Transaction \%a:\%t for \$(tran.uri} failed.
command:
executes the following string as a shell command.

Note this operation can be much expensive regarding performance thus it should not be used inside instrumented applications. You can use this trigger action in one of the MyARM daemons like myarmfiled or myarmtcpd. For example:

command: /usr/local/bin/sendemail.sh \
         -s "Transaction %a:%t for $(tran.uri} failed." \
         fatal@yourdomain.com
Trigger action text substitution

Within the trigger action string the following text substitution are done by the trigger sink:

%a
is substituted by the application definition name of the current transaction.
%t
is substituted by the transaction definition name of the current transaction.
$(tran.value)
is substituted by the appropriate transaction value. where value is one of the following transaction data field names:
metric.name
the contents of the metric named name.
uri
the URI field of the current transaction.
status
the stop status of the current transaction.
duration
the duration of the current transaction (response time).