Performance Benchmarks

On this page, we collect performance benchmarks of Kieker monitoring and other instrumentation frameworks, such as OpenTelemetry and inspectIT, for comparison utilizing MooBench.

Details of the comparison of OpenTelemetry, inspectIT and Kieker can be found in https://oceanrep.geomar.de/id/eprint/55367/1/SSP2021overhead.pdf

Kieker Observability Framework – Java

These are the performance benchmark results for Kieker 4 Java probes utilizing different writer back end.

  • No instrumentation serves as a baseline value indicating how much execution time can be contributed to the benchmark program itself.
  • Deactivated probe as above, but with woven in probes that do not produce any output and do not aggregate information.
  • No logging are activated probes, but the collected data is not send utilizing a null writer.
  • Text File utilizes the text file logging. This is usually slow, as data is written in text files and getting more irrelevant in the performance benchmarks. The diagram below, therefore, does not show the results.
  • Binary File utilizes the binary file logging without compression.
  • Binary TCP utilizes a TCP connection to send the monitoring data to a logging service.
setup run mean ci sd 1.quartile median 3.quartile max min
No instrumentation current 0.116 0.000 0.028 0.115 0.116 0.118 7.699 0.108
No instrumentation past 0.107 0.000 0.045 0.106 0.107 0.107 23.605 0.103
Deactivated probe current 0.718 0.008 3.925 0.683 0.701 0.720 2774.132 0.658
Deactivated probe past 0.717 0.007 3.600 0.686 0.701 0.719 2680.724 0.652
No logging current 2.669 0.011 5.375 2.616 2.636 2.658 2817.344 2.532
No logging past 2.751 0.010 5.151 2.698 2.719 2.745 2788.698 2.634
Binary file current 4.813 0.013 6.651 3.399 4.354 4.680 3734.424 2.977
Binary file past 5.134 0.021 10.704 3.526 4.499 6.225 8615.828 3.191
Binary TCP current 6.553 0.011 5.771 6.626 6.790 7.173 2689.553 3.208
Binary TCP past 7.063 0.011 5.594 6.697 6.973 7.536 2621.835 3.009

Kieker Observability Framework – Python

Kieker 4 Python is the reimplementation of the probe and monitoring side of Kieker addressing programs written in Python. Here we also want to know how well certain parts perform.

  • No instrumentation is the baseline execution time of the benchmark program.
  • Deactivated probe is the execution time with integrated probes, but these are deactivated.
  • No logging is with active probes, but with a null writer.
  • Text File is with an activated probe logging to a text file.
  • Binary TCP is with an activated probe logging to a remote logging server.

Each of these setups, except of no instrumentation, are tested utilizing two different probe introduction methods (here labeled A and B).

setup run mean ci sd 1.quartile median 3.quartile max min
No instrumentation current 1.536 0.000 0.163 1.483 1.493 1.628 19.445 1.435
No instrumentation past 1.589 0.000 0.165 1.576 1.605 1.613 24.621 1.408
Deactivated probe - A current 1.519 0.000 0.168 1.490 1.505 1.517 27.005 1.389
Deactivated probe - A past 1.493 0.000 0.171 1.457 1.470 1.493 60.337 1.403
Deactivated probe - B current 4.022 0.001 0.341 3.810 3.849 4.225 95.351 3.726
Deactivated probe - B past 3.946 0.001 0.283 3.839 3.868 4.064 40.977 3.750
No logging - A current 1.500 0.000 0.194 1.446 1.455 1.596 96.466 1.388
No logging - A past 1.572 0.000 0.176 1.496 1.577 1.603 28.724 1.424
No logging - B current 43.919 0.003 1.534 43.418 43.574 43.764 138.194 42.591
No logging - B past 45.214 0.005 2.456 44.300 44.483 44.788 1215.639 43.527
Text file - A current 1.586 0.000 0.186 1.495 1.573 1.655 38.356 1.426
Text file - A past 1.572 0.000 0.172 1.546 1.555 1.575 35.187 1.424
Text file - B current 1329.479 0.963 491.519 1277.713 1293.586 1378.756 133532.400 1252.142
Text file - B past 1332.875 1.680 857.130 1276.498 1301.497 1380.078 603750.373 1249.869
Binary TCP - A current 1.664 0.000 0.174 1.666 1.682 1.691 28.586 1.455
Binary TCP - A past 1.517 0.000 0.168 1.484 1.493 1.530 27.216 1.418
Binary TCP - B current 497.886 0.068 34.465 487.682 490.306 496.081 1387.352 236.203
Binary TCP - B past 484.084 0.065 33.137 474.732 477.537 482.787 1134.446 235.409

OpenTelemetry

OpenTelemetry (https://opentelemetry.io/) is a widely used monitoring tool, compatible with many programming languages. MooBench measures the overhead of OpenTelemetry using the following configurations:

  • No instrumentation is the baseline execution time of the benchmark program.
  • No Logging measures the execution time of the benchmark program with OpenTelemetry instrumentation, but without measuring or
    logging anything.
  • Logging measures the execution time of the benchmark program with OpenTelemetry instrumentation and writing every output to the standard output. Text file logging is slow and getting more irrelevant in the performance benchmarks. The diagram below, therefore, does not show the results.
  • Zipkin measures the execution time of the benchmark program with obtaining every method invocation and sending them to a local Zipkin instance.
  • Prometheus measures the execution time of the benchmark program with obtaining only aggregated execution times, which are stored by a local Prometheus instance.
setup run mean ci sd 1.quartile median 3.quartile max min
No instrumentation current 0.108 0.000 0.030 0.107 0.108 0.109 9.221 0.103
No instrumentation past 0.106 0.000 0.098 0.106 0.106 0.106 88.412 0.102
No logging current 7.067 0.015 7.580 6.919 6.951 6.992 2305.677 6.787
No logging past 6.655 0.019 9.599 6.508 6.544 6.592 4264.288 6.354
Zipkin current 11.924 0.027 13.754 11.203 11.424 11.590 3353.580 9.616
Zipkin past 11.097 0.025 12.994 10.441 10.512 10.634 3678.223 10.030

InspectIT

InspectIT (https://www.inspectit.rocks/) is a widely used monitoring tool, which is allows to trace the execution of Java programs. MooBench
measures the overhead of InspectIT using the following configurations:

  • No instrumentation is the baseline execution time of the benchmark program.
  • Deactivated processing measures the execution time of the benchmark program with inspectIT instrumentation, but without measuring or logging
    anything.
  • No Logging measures the execution time of the benchmark program with inspectIT instrumentation and just discarding the measurement results.
  • Zipkin measures the execution time of the benchmark program with obtaining every method invocation and sending them to a local Zipkin instance.
  • Prometheus measures the execution time of the benchmark program with obtaining only aggregated execution times, which are stored by a local Prometheus instance.
setup run mean ci sd 1.quartile median 3.quartile max min
No instrumentation current 0.109 0.000 0.049 0.108 0.108 0.109 26.652 0.103
No instrumentation past 0.112 0.000 0.033 0.110 0.111 0.112 13.552 0.105
Deactivated probe current 14.344 0.021 10.830 14.071 14.152 14.249 3957.442 13.595
Deactivated probe past 13.788 0.021 10.846 13.482 13.576 13.688 3179.313 13.053
No logging current 39.792 0.031 15.844 38.781 39.006 39.302 2996.074 37.629
No logging past 41.204 0.041 21.049 40.270 40.530 40.874 9548.529 35.899
Zipkin current 43.649 0.057 29.299 40.956 41.306 41.772 14845.380 35.963
Zipkin past 43.372 0.092 46.817 41.195 41.510 41.900 32717.984 36.388
Prometheus current 25.728 0.025 12.510 25.295 25.410 25.546 3072.799 24.710
Prometheus past 24.690 0.026 13.218 24.259 24.395 24.574 4557.257 22.604