Performance Benchmarks

On this page, we collect performance benchmarks of Kieker monitoring and other instrumentation frameworks, such as OpenTelemetry and inspectIT, for comparison utilizing MooBench.

Details of the comparison of OpenTelemetry, inspectIT and Kieker can be found in https://oceanrep.geomar.de/id/eprint/55367/1/SSP2021overhead.pdf

Kieker Observability Framework – Java

These are the performance benchmark results for Kieker 4 Java probes utilizing different writer back end.

  • No instrumentation serves as a baseline value indicating how much execution time can be contributed to the benchmark program itself.
  • Deactivated probe as above, but with woven in probes that do not produce any output and do not aggregate information.
  • No logging are activated probes, but the collected data is not send utilizing a null writer.
  • Text File utilizes the text file logging. This is usually slow, as data is written in text files and getting more irrelevant in the performance benchmarks. The diagram below, therefore, does not show the results.
  • Binary File utilizes the binary file logging without compression.
  • Binary TCP utilizes a TCP connection to send the monitoring data to a logging service.
setup run mean ci sd 1.quartile median 3.quartile max min
No instrumentation current 0.105 0.000 0.032 0.105 0.105 0.105 9.091 0.103
No instrumentation past 0.111 0.000 0.030 0.110 0.111 0.111 10.309 0.105
Deactivated probe current 0.143 0.000 0.039 0.142 0.143 0.143 24.608 0.139
Deactivated probe past 0.705 0.007 3.530 0.673 0.689 0.707 2549.518 0.642
No logging current 1.896 0.006 3.259 1.847 1.858 1.878 441.046 1.826
No logging past 2.683 0.008 3.967 2.640 2.666 2.695 2494.489 2.435
Binary file current 3.952 0.023 11.551 2.562 3.663 5.121 11083.390 2.212
Binary file past 5.073 0.029 14.677 3.510 4.475 5.990 13375.797 3.201
Binary TCP current 4.421 0.049 25.171 4.311 4.604 4.821 24668.610 2.069
Binary TCP past 6.871 0.024 12.055 6.554 6.907 7.488 9812.935 3.102

Kieker Observability Framework – Python

Kieker 4 Python is the reimplementation of the probe and monitoring side of Kieker addressing programs written in Python. Here we also want to know how well certain parts perform.

  • No instrumentation is the baseline execution time of the benchmark program.
  • Deactivated probe is the execution time with integrated probes, but these are deactivated.
  • No logging is with active probes, but with a null writer.
  • Text File is with an activated probe logging to a text file.
  • Binary TCP is with an activated probe logging to a remote logging server.

Each of these setups, except of no instrumentation, are tested utilizing two different probe introduction methods (here labeled A and B).

setup run mean ci sd 1.quartile median 3.quartile max min
No instrumentation current 1.534 0.001 0.660 1.505 1.521 1.543 588.608 1.441
No instrumentation past 1.524 0.001 0.395 1.494 1.507 1.527 215.072 1.444
Deactivated probe - A current 1.527 0.001 0.662 1.493 1.508 1.530 614.929 1.440
Deactivated probe - A past 1.549 0.001 0.533 1.516 1.528 1.555 390.824 1.464
Deactivated probe - B current 1.508 0.000 0.229 1.483 1.497 1.515 63.715 1.435
Deactivated probe - B past 1.600 0.001 0.385 1.570 1.584 1.616 202.988 1.490
No logging - A current 1.546 0.001 0.271 1.513 1.530 1.551 61.385 1.460
No logging - A past 1.520 0.001 0.627 1.489 1.505 1.525 514.950 1.431
No logging - B current 1.542 0.001 0.605 1.513 1.527 1.548 538.293 1.466
No logging - B past 2.200 0.001 0.637 2.160 2.175 2.201 521.686 2.102
Text file - A current 1.550 0.002 0.935 1.514 1.529 1.550 655.727 1.468
Text file - A past 1.533 0.002 0.867 1.494 1.511 1.548 662.328 1.439
Text file - B current 1.534 0.000 0.235 1.509 1.522 1.540 75.356 1.440
Text file - B past 22.404 0.019 9.820 21.509 21.887 23.153 3508.747 21.044
Binary TCP - A current 1.537 0.001 0.406 1.509 1.524 1.543 276.986 1.456
Binary TCP - A past 1.553 0.001 0.413 1.525 1.539 1.557 234.678 1.468
Binary TCP - B current 1.560 0.001 0.689 1.527 1.543 1.560 634.239 1.453
Binary TCP - B past 9.151 0.002 1.236 8.948 9.012 9.150 566.963 5.142

OpenTelemetry

OpenTelemetry (https://opentelemetry.io/) is a widely used monitoring tool, compatible with many programming languages. MooBench measures the overhead of OpenTelemetry using the following configurations:

  • No instrumentation is the baseline execution time of the benchmark program.
  • No Logging measures the execution time of the benchmark program with OpenTelemetry instrumentation, but without measuring or
    logging anything.
  • Logging measures the execution time of the benchmark program with OpenTelemetry instrumentation and writing every output to the standard output. Text file logging is slow and getting more irrelevant in the performance benchmarks. The diagram below, therefore, does not show the results.
  • Zipkin measures the execution time of the benchmark program with obtaining every method invocation and sending them to a local Zipkin instance.
  • Prometheus measures the execution time of the benchmark program with obtaining only aggregated execution times, which are stored by a local Prometheus instance.
setup run mean ci sd 1.quartile median 3.quartile max min
No instrumentation current 0.108 0.000 0.133 0.106 0.107 0.108 89.919 0.101
No instrumentation past 0.108 0.000 0.070 0.106 0.107 0.108 37.668 0.101
No logging current 7.378 0.021 10.652 7.153 7.215 7.286 3746.864 6.902
No logging past 7.378 0.019 9.889 7.173 7.231 7.299 3213.571 6.945
Zipkin current 12.869 0.031 15.632 11.207 13.138 13.292 3780.664 10.615
Zipkin past 13.131 0.036 18.415 12.607 12.699 12.815 6044.233 12.092
Prometheus current 14.120 0.027 13.521 13.703 13.903 14.051 5034.665 13.172
Prometheus past 13.232 0.023 11.884 12.921 13.019 13.115 3724.907 12.538

InspectIT

InspectIT (https://www.inspectit.rocks/) is a widely used monitoring tool, which is allows to trace the execution of Java programs. MooBench
measures the overhead of InspectIT using the following configurations:

  • No instrumentation is the baseline execution time of the benchmark program.
  • Deactivated processing measures the execution time of the benchmark program with inspectIT instrumentation, but without measuring or logging
    anything.
  • No Logging measures the execution time of the benchmark program with inspectIT instrumentation and just discarding the measurement results.
  • Zipkin measures the execution time of the benchmark program with obtaining every method invocation and sending them to a local Zipkin instance.
  • Prometheus measures the execution time of the benchmark program with obtaining only aggregated execution times, which are stored by a local Prometheus instance.
setup run mean ci sd 1.quartile median 3.quartile max min
No instrumentation current 0.108 0.000 0.056 0.107 0.108 0.109 23.715 0.100
No instrumentation past 0.107 0.000 0.097 0.106 0.107 0.108 84.836 0.101
Deactivated probe current 3.400 0.044 22.320 3.227 3.271 3.320 10527.630 3.096
Deactivated probe past 4.172 0.060 30.385 3.997 4.042 4.093 19703.667 3.825
No logging current 26.245 0.027 13.778 25.597 25.789 26.021 4132.888 24.615
No logging past 25.636 0.034 17.418 24.859 25.038 25.264 6378.142 23.973
Zipkin current 30.074 0.044 22.700 29.248 29.474 29.732 4541.090 28.142
Zipkin past 29.336 0.051 26.187 27.306 29.744 30.084 11971.969 24.487
Prometheus current 8.952 0.056 28.369 8.528 8.620 8.728 13197.560 8.267
Prometheus past 9.808 0.036 18.396 9.520 9.645 9.801 10505.098 9.219