Performance Benchmarks

On this page, we collect performance benchmarks of Kieker monitoring and other instrumentation frameworks, such as OpenTelemetry and inspectIT, for comparison utilizing MooBench.

Details of the comparison of OpenTelemetry, inspectIT and Kieker can be found in https://oceanrep.geomar.de/id/eprint/55367/1/SSP2021overhead.pdf

Kieker Observability Framework – Java

These are the performance benchmark results for Kieker 4 Java probes utilizing different writer back end.

  • No instrumentation serves as a baseline value indicating how much execution time can be contributed to the benchmark program itself.
  • Deactivated probe as above, but with woven in probes that do not produce any output and do not aggregate information.
  • No logging are activated probes, but the collected data is not send utilizing a null writer.
  • Text File utilizes the text file logging. This is usually slow, as data is written in text files and getting more irrelevant in the performance benchmarks. The diagram below, therefore, does not show the results.
  • Binary File utilizes the binary file logging without compression.
  • Binary TCP utilizes a TCP connection to send the monitoring data to a logging service.
setup run mean ci sd 1.quartile median 3.quartile max min
No instrumentation current 0.105 0.000 0.032 0.105 0.105 0.105 9.091 0.103
No instrumentation past 0.111 0.000 0.030 0.110 0.111 0.111 10.309 0.105
Deactivated probe current 0.143 0.000 0.039 0.142 0.143 0.143 24.608 0.139
Deactivated probe past 0.705 0.007 3.530 0.673 0.689 0.707 2549.518 0.642
No logging current 1.896 0.006 3.259 1.847 1.858 1.878 441.046 1.826
No logging past 2.683 0.008 3.967 2.640 2.666 2.695 2494.489 2.435
Binary file current 3.952 0.023 11.551 2.562 3.663 5.121 11083.390 2.212
Binary file past 5.073 0.029 14.677 3.510 4.475 5.990 13375.797 3.201
Binary TCP current 4.421 0.049 25.171 4.311 4.604 4.821 24668.610 2.069
Binary TCP past 6.871 0.024 12.055 6.554 6.907 7.488 9812.935 3.102

Kieker Observability Framework – Python

Kieker 4 Python is the reimplementation of the probe and monitoring side of Kieker addressing programs written in Python. Here we also want to know how well certain parts perform.

  • No instrumentation is the baseline execution time of the benchmark program.
  • Deactivated probe is the execution time with integrated probes, but these are deactivated.
  • No logging is with active probes, but with a null writer.
  • Text File is with an activated probe logging to a text file.
  • Binary TCP is with an activated probe logging to a remote logging server.

Each of these setups, except of no instrumentation, are tested utilizing two different probe introduction methods (here labeled A and B).

setup run mean ci sd 1.quartile median 3.quartile max min
No instrumentation current 1.491 0.001 0.699 1.445 1.454 1.475 185.637 1.390
No instrumentation past 1.517 0.000 0.172 1.491 1.509 1.530 63.927 1.431
Deactivated probe - A current 1.472 0.001 0.578 1.447 1.454 1.460 196.878 1.404
Deactivated probe - A past 1.583 0.000 0.175 1.558 1.570 1.613 29.270 1.421
Deactivated probe - B current 1.615 0.001 0.645 1.605 1.617 1.626 183.605 1.407
Deactivated probe - B past 2.134 0.000 0.221 2.066 2.082 2.207 40.341 2.006
No logging - A current 1.474 0.001 0.665 1.450 1.459 1.468 187.781 1.412
No logging - A past 1.527 0.000 0.174 1.474 1.510 1.543 26.388 1.423
No logging - B current 1.479 0.001 0.522 1.452 1.460 1.469 191.849 1.410
No logging - B past 12.232 0.001 0.501 12.062 12.109 12.254 64.798 11.838
Text file - A current 1.521 0.005 2.403 1.452 1.474 1.597 2314.730 1.396
Text file - A past 1.536 0.000 0.169 1.485 1.506 1.593 29.646 1.428
Text file - B current 1.506 0.001 0.661 1.457 1.466 1.486 199.347 1.402
Text file - B past 335.251 0.300 152.891 321.301 327.584 347.121 59867.499 314.797
Binary TCP - A current 1.628 0.001 0.610 1.612 1.620 1.627 173.479 1.560
Binary TCP - A past 1.491 0.000 0.160 1.463 1.486 1.497 28.024 1.409
Binary TCP - B current 1.543 0.007 3.332 1.460 1.505 1.616 3258.399 1.407
Binary TCP - B past 123.475 0.018 8.936 120.794 121.539 123.148 293.340 60.361

OpenTelemetry

OpenTelemetry (https://opentelemetry.io/) is a widely used monitoring tool, compatible with many programming languages. MooBench measures the overhead of OpenTelemetry using the following configurations:

  • No instrumentation is the baseline execution time of the benchmark program.
  • No Logging measures the execution time of the benchmark program with OpenTelemetry instrumentation, but without measuring or
    logging anything.
  • Logging measures the execution time of the benchmark program with OpenTelemetry instrumentation and writing every output to the standard output. Text file logging is slow and getting more irrelevant in the performance benchmarks. The diagram below, therefore, does not show the results.
  • Zipkin measures the execution time of the benchmark program with obtaining every method invocation and sending them to a local Zipkin instance.
  • Prometheus measures the execution time of the benchmark program with obtaining only aggregated execution times, which are stored by a local Prometheus instance.
setup run mean ci sd 1.quartile median 3.quartile max min
No instrumentation current 0.101 0.000 0.165 0.100 0.101 0.102 157.527 0.098
No instrumentation past 0.106 0.000 0.036 0.105 0.105 0.106 19.186 0.103
No logging current 7.883 0.018 9.017 7.681 7.721 7.772 2396.843 7.535
No logging past 7.127 0.015 7.424 6.968 7.019 7.081 2198.083 6.730
Zipkin current 11.796 0.031 15.689 11.008 11.065 11.143 5409.936 10.760
Zipkin past 11.962 0.029 14.728 11.193 11.265 11.378 5176.523 10.185
Prometheus current 12.967 0.021 10.850 12.720 12.770 12.831 2737.855 12.505

InspectIT

InspectIT (https://www.inspectit.rocks/) is a widely used monitoring tool, which is allows to trace the execution of Java programs. MooBench
measures the overhead of InspectIT using the following configurations:

  • No instrumentation is the baseline execution time of the benchmark program.
  • Deactivated processing measures the execution time of the benchmark program with inspectIT instrumentation, but without measuring or logging
    anything.
  • No Logging measures the execution time of the benchmark program with inspectIT instrumentation and just discarding the measurement results.
  • Zipkin measures the execution time of the benchmark program with obtaining every method invocation and sending them to a local Zipkin instance.
  • Prometheus measures the execution time of the benchmark program with obtaining only aggregated execution times, which are stored by a local Prometheus instance.
setup run mean ci sd 1.quartile median 3.quartile max min
No instrumentation current 0.102 0.000 0.165 0.101 0.102 0.102 157.986 0.098
No instrumentation past 0.109 0.000 0.036 0.108 0.108 0.109 16.606 0.105
Deactivated probe current 4.073 0.043 21.759 3.932 3.960 3.993 11150.270 3.819
Deactivated probe past 14.401 0.021 10.462 14.107 14.195 14.301 3949.525 13.704
No logging current 24.561 0.040 20.599 23.538 23.733 23.987 15511.970 22.767
No logging past 38.937 0.044 22.651 37.977 38.197 38.541 13551.478 36.186
Zipkin current 25.342 0.037 18.969 24.064 24.196 24.359 4464.420 21.216
Zipkin past 43.800 0.149 76.060 41.281 41.555 41.977 65323.615 36.406
Prometheus current 8.258 0.046 23.507 8.010 8.061 8.181 11598.890 7.838
Prometheus past 24.394 0.026 13.079 23.932 24.059 24.215 4437.843 23.415