Performance Benchmarks

On this page, we collect performance benchmarks of Kieker monitoring and other instrumentation frameworks, such as OpenTelemetry and inspectIT, for comparison utilizing MooBench.

Details of the comparison of OpenTelemetry, inspectIT and Kieker can be found in https://oceanrep.geomar.de/id/eprint/55367/1/SSP2021overhead.pdf

Kieker Observability Framework – Java

These are the performance benchmark results for Kieker 4 Java probes utilizing different writer back end.

  • No instrumentation serves as a baseline value indicating how much execution time can be contributed to the benchmark program itself.
  • Deactivated probe as above, but with woven in probes that do not produce any output and do not aggregate information.
  • No logging are activated probes, but the collected data is not send utilizing a null writer.
  • Text File utilizes the text file logging. This is usually slow, as data is written in text files and getting more irrelevant in the performance benchmarks. The diagram below, therefore, does not show the results.
  • Binary File utilizes the binary file logging without compression.
  • Binary TCP utilizes a TCP connection to send the monitoring data to a logging service.
setup run mean ci sd 1.quartile median 3.quartile max min
No instrumentation current 0.120 0.000 0.113 0.118 0.119 0.120 88.005 0.111
No instrumentation past 0.113 0.000 0.050 0.112 0.113 0.114 25.514 0.106
Deactivated probe current 0.151 0.002 0.818 0.148 0.150 0.151 814.711 0.143
Deactivated probe past 0.145 0.000 0.094 0.143 0.144 0.145 55.856 0.136
No collection current 1.735 0.007 3.611 1.693 1.700 1.715 590.244 1.665
No collection past 1.716 0.012 5.976 1.660 1.668 1.685 3097.746 1.630
Binary file current 3.477 0.011 5.679 2.259 2.927 4.915 2399.223 2.098
Binary file past 3.526 0.011 5.569 2.234 3.011 4.915 2973.535 2.063
Binary TCP current 5.725 0.010 5.165 4.936 5.903 6.567 897.404 2.006
Binary TCP past 4.996 0.021 10.936 3.496 5.339 5.902 7880.567 2.376

Kieker Observability Framework – Python

Kieker 4 Python is the reimplementation of the probe and monitoring side of Kieker addressing programs written in Python. Here we also want to know how well certain parts perform.

  • No instrumentation is the baseline execution time of the benchmark program.
  • Deactivated probe is the execution time with integrated probes, but these are deactivated.
  • No logging is with active probes, but with a null writer.
  • Text File is with an activated probe logging to a text file.
  • Binary TCP is with an activated probe logging to a remote logging server.

Each of these setups, except of no instrumentation, are tested utilizing two different probe introduction methods (here labeled A and B).

setup run mean ci sd 1.quartile median 3.quartile max min
No instrumentation current 1.425 0.001 0.296 1.406 1.414 1.428 112.765 1.370
No instrumentation past 1.452 0.001 0.482 1.425 1.437 1.455 354.362 1.384
Deactivated probe - A current 1.449 0.000 0.224 1.428 1.437 1.453 56.751 1.393
Deactivated probe - A past 1.449 0.004 1.816 1.424 1.434 1.453 1717.744 1.385
Deactivated probe - B current 1.443 0.001 0.287 1.423 1.430 1.443 77.352 1.389
Deactivated probe - B past 1.432 0.001 0.256 1.412 1.422 1.437 72.736 1.374
No logging - A current 1.460 0.001 0.296 1.425 1.435 1.462 96.314 1.388
No logging - A past 1.446 0.001 0.265 1.422 1.432 1.454 80.639 1.381
No logging - B current 1.450 0.001 0.284 1.423 1.434 1.450 77.709 1.380
No logging - B past 1.444 0.001 0.285 1.415 1.427 1.449 91.093 1.369
Text file - A current 1.471 0.001 0.296 1.444 1.454 1.477 70.061 1.405
Text file - A past 1.448 0.001 0.334 1.424 1.435 1.452 146.489 1.382
Text file - B current 1.498 0.000 0.253 1.470 1.490 1.512 68.252 1.406
Text file - B past 1.446 0.001 0.277 1.421 1.430 1.447 90.505 1.381
Binary TCP - A current 1.549 0.001 0.281 1.532 1.540 1.550 77.546 1.495
Binary TCP - A past 1.438 0.001 0.314 1.409 1.419 1.437 106.305 1.370
Binary TCP - B current 1.433 0.001 0.280 1.411 1.422 1.443 64.321 1.377
Binary TCP - B past 1.483 0.001 0.292 1.465 1.474 1.487 115.593 1.408

OpenTelemetry

OpenTelemetry (https://opentelemetry.io/) is a widely used monitoring tool, compatible with many programming languages. MooBench measures the overhead of OpenTelemetry using the following configurations:

  • No instrumentation is the baseline execution time of the benchmark program.
  • No Logging measures the execution time of the benchmark program with OpenTelemetry instrumentation, but without measuring or
    logging anything.
  • Logging measures the execution time of the benchmark program with OpenTelemetry instrumentation and writing every output to the standard output. Text file logging is slow and getting more irrelevant in the performance benchmarks. The diagram below, therefore, does not show the results.
  • Zipkin measures the execution time of the benchmark program with obtaining every method invocation and sending them to a local Zipkin instance.
  • Prometheus measures the execution time of the benchmark program with obtaining only aggregated execution times, which are stored by a local Prometheus instance.
setup run mean ci sd 1.quartile median 3.quartile max min
No instrumentation current 0.119 0.000 0.068 0.117 0.119 0.120 39.636 0.113
No instrumentation past 0.115 0.002 0.899 0.112 0.113 0.114 866.596 0.107
No logging current 7.287 0.022 11.297 7.073 7.134 7.202 5454.922 6.824
No logging past 6.994 0.022 11.335 6.769 6.828 6.896 4171.810 6.536
Zipkin current 11.573 0.032 16.115 10.994 11.100 11.253 4389.802 10.578
Zipkin past 13.413 0.035 17.983 12.745 12.879 13.774 4715.906 12.199
Prometheus current 10.033 0.024 12.452 9.758 9.847 9.957 4006.781 9.429
Prometheus past 10.237 0.025 12.621 9.949 10.032 10.127 4310.069 9.588

InspectIT

InspectIT (https://www.inspectit.rocks/) is a widely used monitoring tool, which is allows to trace the execution of Java programs. MooBench
measures the overhead of InspectIT using the following configurations:

  • No instrumentation is the baseline execution time of the benchmark program.
  • Deactivated processing measures the execution time of the benchmark program with inspectIT instrumentation, but without measuring or logging
    anything.
  • No Logging measures the execution time of the benchmark program with inspectIT instrumentation and just discarding the measurement results.
  • Zipkin measures the execution time of the benchmark program with obtaining every method invocation and sending them to a local Zipkin instance.
  • Prometheus measures the execution time of the benchmark program with obtaining only aggregated execution times, which are stored by a local Prometheus instance.
setup run mean ci sd 1.quartile median 3.quartile max min
No instrumentation current 0.122 0.000 0.043 0.120 0.121 0.123 27.479 0.114
No instrumentation past 0.113 0.000 0.070 0.112 0.113 0.114 43.436 0.106
Deactivated probe current 3.341 0.050 25.641 3.175 3.216 3.267 13962.480 3.050
Deactivated probe past 3.673 0.047 24.162 3.482 3.520 3.568 12030.538 3.347
No logging current 25.282 0.027 13.880 24.663 24.829 25.032 4072.937 23.796
No logging past 24.849 0.029 15.015 24.210 24.390 24.611 4038.894 23.308
Zipkin current 26.399 0.047 24.043 25.235 25.450 25.759 7523.748 24.340
Zipkin past 27.010 0.046 23.631 25.979 26.300 26.570 8736.641 23.882
Prometheus current 8.194 0.047 23.812 7.963 8.021 8.103 10979.200 7.773
Prometheus past 8.825 0.038 19.320 8.540 8.693 8.794 10470.380 8.223