Performance Benchmarks

On this page, we collect performance benchmarks of Kieker monitoring and other instrumentation frameworks, such as OpenTelemetry and inspectIT, for comparison utilizing MooBench.

Details of the comparison of OpenTelemetry, inspectIT and Kieker can be found in https://oceanrep.geomar.de/id/eprint/55367/1/SSP2021overhead.pdf

Kieker Observability Framework – Java

These are the performance benchmark results for Kieker 4 Java probes utilizing different writer back end.

  • No instrumentation serves as a baseline value indicating how much execution time can be contributed to the benchmark program itself.
  • Deactivated probe as above, but with woven in probes that do not produce any output and do not aggregate information.
  • No logging are activated probes, but the collected data is not send utilizing a null writer.
  • Text File utilizes the text file logging. This is usually slow, as data is written in text files and getting more irrelevant in the performance benchmarks. The diagram below, therefore, does not show the results.
  • Binary File utilizes the binary file logging without compression.
  • Binary TCP utilizes a TCP connection to send the monitoring data to a logging service.
setup run mean ci sd 1.quartile median 3.quartile max min
No instrumentation current 0.105 0.000 0.032 0.105 0.105 0.105 9.091 0.103
No instrumentation past 0.111 0.000 0.030 0.110 0.111 0.111 10.309 0.105
Deactivated probe current 0.143 0.000 0.039 0.142 0.143 0.143 24.608 0.139
Deactivated probe past 0.705 0.007 3.530 0.673 0.689 0.707 2549.518 0.642
No logging current 1.896 0.006 3.259 1.847 1.858 1.878 441.046 1.826
No logging past 2.683 0.008 3.967 2.640 2.666 2.695 2494.489 2.435
Binary file current 3.952 0.023 11.551 2.562 3.663 5.121 11083.390 2.212
Binary file past 5.073 0.029 14.677 3.510 4.475 5.990 13375.797 3.201
Binary TCP current 4.421 0.049 25.171 4.311 4.604 4.821 24668.610 2.069
Binary TCP past 6.871 0.024 12.055 6.554 6.907 7.488 9812.935 3.102

Kieker Observability Framework – Python

Kieker 4 Python is the reimplementation of the probe and monitoring side of Kieker addressing programs written in Python. Here we also want to know how well certain parts perform.

  • No instrumentation is the baseline execution time of the benchmark program.
  • Deactivated probe is the execution time with integrated probes, but these are deactivated.
  • No logging is with active probes, but with a null writer.
  • Text File is with an activated probe logging to a text file.
  • Binary TCP is with an activated probe logging to a remote logging server.

Each of these setups, except of no instrumentation, are tested utilizing two different probe introduction methods (here labeled A and B).

setup run mean ci sd 1.quartile median 3.quartile max min
No instrumentation current 1.488 0.000 0.139 1.475 1.482 1.489 21.021 1.431
No instrumentation past 1.514 0.000 0.152 1.478 1.504 1.536 23.518 1.409
Deactivated probe - A current 1.526 0.000 0.166 1.471 1.482 1.619 23.832 1.429
Deactivated probe - A past 1.518 0.000 0.176 1.470 1.485 1.558 32.625 1.405
Deactivated probe - B current 1.574 0.000 0.181 1.510 1.526 1.660 32.299 1.438
Deactivated probe - B past 2.764 0.001 0.275 2.643 2.668 2.912 37.256 2.570
No logging - A current 1.529 0.000 0.179 1.483 1.493 1.513 29.488 1.422
No logging - A past 1.526 0.000 0.176 1.463 1.523 1.575 27.936 1.406
No logging - B current 1.509 0.000 0.175 1.478 1.488 1.503 29.067 1.433
No logging - B past 22.927 0.002 0.819 22.651 22.729 22.883 87.112 22.248
Text file - A current 1.532 0.000 0.155 1.512 1.522 1.534 33.996 1.459
Text file - A past 1.514 0.000 0.168 1.473 1.502 1.526 37.343 1.409
Text file - B current 1.503 0.000 0.153 1.475 1.484 1.492 24.708 1.424
Text file - B past 669.087 0.591 301.428 641.212 653.898 692.720 113565.481 628.204
Binary TCP - A current 1.517 0.000 0.173 1.470 1.478 1.498 26.994 1.431
Binary TCP - A past 1.507 0.000 0.152 1.472 1.509 1.519 25.165 1.408
Binary TCP - B current 1.549 0.000 0.165 1.491 1.507 1.637 19.408 1.437
Binary TCP - B past 245.355 0.035 17.777 240.018 241.503 244.709 562.622 119.275

OpenTelemetry

OpenTelemetry (https://opentelemetry.io/) is a widely used monitoring tool, compatible with many programming languages. MooBench measures the overhead of OpenTelemetry using the following configurations:

  • No instrumentation is the baseline execution time of the benchmark program.
  • No Logging measures the execution time of the benchmark program with OpenTelemetry instrumentation, but without measuring or
    logging anything.
  • Logging measures the execution time of the benchmark program with OpenTelemetry instrumentation and writing every output to the standard output. Text file logging is slow and getting more irrelevant in the performance benchmarks. The diagram below, therefore, does not show the results.
  • Zipkin measures the execution time of the benchmark program with obtaining every method invocation and sending them to a local Zipkin instance.
  • Prometheus measures the execution time of the benchmark program with obtaining only aggregated execution times, which are stored by a local Prometheus instance.
setup run mean ci sd 1.quartile median 3.quartile max min
No instrumentation current 0.105 0.000 0.029 0.105 0.105 0.105 8.821 0.101
No instrumentation past 0.108 0.000 0.030 0.108 0.108 0.109 12.855 0.104
No logging current 6.815 0.014 7.056 6.676 6.718 6.764 1987.063 6.488
No logging past 7.006 0.015 7.508 6.856 6.895 6.942 2170.751 6.710
Zipkin current 11.545 0.024 12.368 10.525 10.589 10.745 3127.990 9.354
Zipkin past 11.593 0.032 16.540 10.870 10.947 11.068 6835.528 10.191
Prometheus current 11.103 0.018 9.196 10.794 11.027 11.150 2384.793 10.273

InspectIT

InspectIT (https://www.inspectit.rocks/) is a widely used monitoring tool, which is allows to trace the execution of Java programs. MooBench
measures the overhead of InspectIT using the following configurations:

  • No instrumentation is the baseline execution time of the benchmark program.
  • Deactivated processing measures the execution time of the benchmark program with inspectIT instrumentation, but without measuring or logging
    anything.
  • No Logging measures the execution time of the benchmark program with inspectIT instrumentation and just discarding the measurement results.
  • Zipkin measures the execution time of the benchmark program with obtaining every method invocation and sending them to a local Zipkin instance.
  • Prometheus measures the execution time of the benchmark program with obtaining only aggregated execution times, which are stored by a local Prometheus instance.
setup run mean ci sd 1.quartile median 3.quartile max min
No instrumentation current 0.105 0.000 0.108 0.105 0.105 0.105 105.411 0.103
No instrumentation past 0.105 0.000 0.042 0.104 0.105 0.105 22.831 0.102
Deactivated probe current 3.996 0.309 157.700 3.695 3.719 3.750 156404.500 3.590
Deactivated probe past 14.842 0.021 10.864 14.553 14.647 14.753 4193.511 14.032
No logging current 25.345 0.024 12.054 24.723 24.876 25.095 3221.619 24.147
No logging past 40.087 0.061 30.875 39.029 39.276 39.690 23970.749 36.460
Zipkin current 24.604 0.041 21.058 23.538 23.676 23.831 12213.670 20.771
Zipkin past 42.876 0.053 27.107 40.676 40.959 41.380 9789.640 35.859
Prometheus current 10.935 0.036 18.401 10.661 10.766 10.961 10065.170 10.115
Prometheus past 25.877 0.027 13.979 25.367 25.526 25.718 6098.196 24.721