Metrics
Metrics are used to measure and compare the performance of the system during the stresstest. They are divided into task metrics, worker metrics, and query metrics.
Task metrics are calculated for every query execution across the whole task. Worker metrics are calculated for every query execution of one worker. Query metrics are calculated for every execution of one query across one worker and across every worker.
For a detailed description of how results for tasks, workers and queries are reported in the RDF result file, please refer to the section RDF results.
Configuration
The metrics are configured in the metrics
section of the configuration file.
To enable a metric, add an entry to the metrics
list with the type
of the metric.
Some metrics (PQPS
, PAvgQPS
) require the configuration of a penalty
value,
which is the time in milliseconds that a failed query will be penalized with.
metrics:
- type: "QPS"
- type: "AvgQPS"
- type: "PQPS"
penalty: 180000 # in milliseconds
If the metrics
section is not present in the configuration file, the following default configuration is used:
metrics:
- type: "AES"
- type: "EachQuery"
- type: "QPS"
- type: "AvgQPS"
- type: "NoQ"
- type: "NoQPH"
- type: "QMPH"
Available metrics
Name | Configuration type | Additional parameters | Scope | Description |
---|---|---|---|---|
Queries per second | QPS |
query | The number of successfully executed queries per second. It is calculated by dividing the number of successfully executed queries | |
Average queries per second | AvgQPS |
task, worker | The average number of queries successfully executed per second. It is calculated by dividing the sum of the QPS values of every query the task or worker has by the number of queries. | |
Number of queries | NoQ |
task, worker | The number of successfully executed queries. This metric is calculated for each worker and for the whole task. | |
Number of queries per hour | NoQPH |
task, worker | The number of successfully executed queries per hour. It is calculated by dividing the number of successfully executed queries by their sum of time (in hours) it took to execute them. The metric value for the task is the sum of the metric for each worker. | |
Query mixes per hour | QMPH |
task, worker | The number of query mixes executed per hour. A query mix is the set of queries executed by a worker, or the whole task. This metric is calculated for each worker and for the whole task. It is calculated by dividing the number of successfully executed queries by the number of queries inside the query mix and by their sum of time (in hours) it took to execute them. | |
Penalized queries per second | PQPS |
penalty (in milliseconds) |
query | The number of queries executed per second, penalized by the number of failed queries. It is calculated by dividing the number of successful and failed query executions by their sum of time (in seconds) it took to execute them. If a query fails, the time it took to execute it is set to the given penalty value. |
Penalized average queries per second | PAvgQPS |
penalty (in milliseconds) |
task, worker | The average number of queries executed per second, penalized by the number of failed queries. It is calculated by dividing the sum of the PQPS of each query the task or worker has executed by the number of queries. |
Aggregated execution statistics | AES |
task, worker | see below | |
Each execution statistic | EachQuery |
query | see below |
Other metrics
Aggregated Execution Statistics (AES)
This metric collects for each query that belongs to a worker or a task a number of statistics that are aggregated for each execution.
Name | Description |
---|---|
succeeded |
The number of successful executions. |
failed |
The number of failed executions. |
resultSize |
The size of the HTTP response. (only stores the last result) |
timeOuts |
The number of executions that resulted with a timeout. |
wrongCodes |
The number of HTTP status codes received that were not 200. |
unknownExceptions |
The number of unknown exceptions during execution. |
totalTime |
The total time it took to execute the queries. |
The resultSize
is the size of the HTTP response in bytes and is an exception to the aggregation.
Each Execution Statistic (EachQuery)
This metric collects statistics for each execution of a query.
Name | Description |
---|---|
run |
The number of the execution. |
startTime |
The time stamp where the execution started. |
time |
The time it took to execute the query. |
success |
If the execution was successful. |
code |
Numerical value of the end state of the execution. (success=0, timeout=110, http_error=111, exception=1) |
resultSize |
The size of the HTTP response. |
exception |
The exception that occurred during execution. (if any occurred) |
httpCode |
The HTTP status code received. (if any was received) |
responseBody |
The hash of the HTTP response body. (only if parseResults inside the stresstest has been set to true ) |