Skip to content

Metrics

Metrics are used to measure and compare the performance of the system during the stresstest. They are divided into task metrics, worker metrics, and query metrics.

Task metrics are calculated for every query execution across the whole task. Worker metrics are calculated for every query execution of one worker. Query metrics are calculated for every execution of one query across one worker and across every worker.

For a detailed description of how results for tasks, workers and queries are reported in the RDF result file, please refer to the section RDF results.

Configuration

The metrics are configured in the metrics section of the configuration file. To enable a metric, add an entry to the metrics list with the type of the metric. Some metrics (PQPS, PAvgQPS) require the configuration of a penalty value, which is the time in milliseconds that a failed query will be penalized with.

metrics:
  - type: "QPS"
  - type: "AvgQPS"
  - type: "PQPS"
    penalty: 180000 # in milliseconds

If the metrics section is not present in the configuration file, the following default configuration is used:

metrics:
  - type: "AES"
  - type: "EachQuery"
  - type: "QPS"
  - type: "AvgQPS"
  - type: "NoQ"
  - type: "NoQPH"
  - type: "QMPH"

Available metrics

Name Configuration type Additional parameters Scope Description
Queries per second QPS query The number of successfully executed queries per second. It is calculated by dividing the number of successfully executed queries
Average queries per second AvgQPS task, worker The average number of queries successfully executed per second. It is calculated by dividing the sum of the QPS values of every query the task or worker has by the number of queries.
Number of queries NoQ task, worker The number of successfully executed queries. This metric is calculated for each worker and for the whole task.
Number of queries per hour NoQPH task, worker The number of successfully executed queries per hour. It is calculated by dividing the number of successfully executed queries by their sum of time (in hours) it took to execute them. The metric value for the task is the sum of the metric for each worker.
Query mixes per hour QMPH task, worker The number of query mixes executed per hour. A query mix is the set of queries executed by a worker, or the whole task. This metric is calculated for each worker and for the whole task. It is calculated by dividing the number of successfully executed queries by the number of queries inside the query mix and by their sum of time (in hours) it took to execute them.
Penalized queries per second PQPS penalty (in milliseconds) query The number of queries executed per second, penalized by the number of failed queries. It is calculated by dividing the number of successful and failed query executions by their sum of time (in seconds) it took to execute them. If a query fails, the time it took to execute it is set to the given penalty value.
Penalized average queries per second PAvgQPS penalty (in milliseconds) task, worker The average number of queries executed per second, penalized by the number of failed queries. It is calculated by dividing the sum of the PQPS of each query the task or worker has executed by the number of queries.
Aggregated execution statistics AES task, worker see below
Each execution statistic EachQuery query see below

Other metrics

Aggregated Execution Statistics (AES)

This metric collects for each query that belongs to a worker or a task a number of statistics that are aggregated for each execution.

Name Description
succeeded The number of successful executions.
failed The number of failed executions.
resultSize The size of the HTTP response. (only stores the last result)
timeOuts The number of executions that resulted with a timeout.
wrongCodes The number of HTTP status codes received that were not 200.
unknownExceptions The number of unknown exceptions during execution.
totalTime The total time it took to execute the queries.

The resultSize is the size of the HTTP response in bytes and is an exception to the aggregation.

Each Execution Statistic (EachQuery)

This metric collects statistics for each execution of a query.

Name Description
run The number of the execution.
startTime The time stamp where the execution started.
time The time it took to execute the query.
success If the execution was successful.
code Numerical value of the end state of the execution. (success=0, timeout=110, http_error=111, exception=1)
resultSize The size of the HTTP response.
exception The exception that occurred during execution. (if any occurred)
httpCode The HTTP status code received. (if any was received)
responseBody The hash of the HTTP response body. (only if parseResults inside the stresstest has been set to true)