Performance Benchmarks

On this page, we collect performance benchmarks of Kieker monitoring and other instrumentation frameworks, such as OpenTelemetry and inspectIT, for comparison utilizing MooBench.

Details of the comparison of OpenTelemetry, inspectIT and Kieker can be found in https://oceanrep.geomar.de/id/eprint/55367/1/SSP2021overhead.pdf

Kieker Observability Framework – Java

These are the performance benchmark results for Kieker 4 Java probes utilizing different writer back end.

  • No instrumentation serves as a baseline value indicating how much execution time can be contributed to the benchmark program itself.
  • Deactivated probe as above, but with woven in probes that do not produce any output and do not aggregate information.
  • No logging are activated probes, but the collected data is not send utilizing a null writer.
  • Text File utilizes the text file logging. This is usually slow, as data is written in text files and getting more irrelevant in the performance benchmarks. The diagram below, therefore, does not show the results.
  • Binary File utilizes the binary file logging without compression.
  • Binary TCP utilizes a TCP connection to send the monitoring data to a logging service.
setup run mean ci sd 1.quartile median 3.quartile max min
No instrumentation current 0.073 0.000 0.278 0.072 0.072 0.073 605.524 0.068
No instrumentation past 0.073 0.000 0.116 0.072 0.072 0.073 256.975 0.068
Deactivated probe current 0.097 0.000 0.085 0.095 0.096 0.098 128.510 0.085
Deactivated probe past 0.096 0.000 0.073 0.094 0.096 0.097 137.357 0.085
No collection current 1.127 0.004 7.229 1.081 1.088 1.097 9482.828 1.050
No collection past 1.135 0.007 8.067 1.092 1.098 1.107 9596.351 1.060
Binary file current 3.802 0.003 4.605 1.882 3.212 4.489 3755.336 1.378
Binary file past 3.877 0.004 4.615 1.983 3.499 4.615 3843.547 1.389
Binary TCP current 3.021 0.005 8.424 2.093 2.568 3.452 6789.625 1.303
Binary TCP past 3.903 0.007 8.775 2.383 3.461 4.631 8547.157 1.300

Kieker Observability Framework – Python

Kieker 4 Python is the reimplementation of the probe and monitoring side of Kieker addressing programs written in Python. Here we also want to know how well certain parts perform.

  • No instrumentation is the baseline execution time of the benchmark program.
  • Deactivated probe is the execution time with integrated probes, but these are deactivated.
  • No logging is with active probes, but with a null writer.
  • Text File is with an activated probe logging to a text file.
  • Binary TCP is with an activated probe logging to a remote logging server.

Each of these setups, except of no instrumentation, are tested utilizing two different probe introduction methods (here labeled A and B).

setup run mean ci sd 1.quartile median 3.quartile max min
No instrumentation current 0.897 0.000 0.173 0.883 0.893 0.904 184.012 0.844
No instrumentation past 0.904 0.000 0.431 0.889 0.898 0.911 810.148 0.849
Deactivated probe - A current 2.153 0.000 0.715 2.129 2.142 2.156 1409.483 2.065
Deactivated probe - A past 2.147 0.000 0.591 2.126 2.138 2.154 1327.758 2.058
Deactivated probe - B current 2.200 0.001 0.847 2.166 2.188 2.212 1996.501 2.088
Deactivated probe - B past 2.173 0.001 0.753 2.147 2.164 2.185 1357.420 2.072
No logging - A current 63.459 0.002 3.818 62.432 62.965 63.817 2070.173 60.683
No logging - A past 61.782 0.003 3.439 61.162 61.602 62.072 2065.771 59.129
No logging - B current 17.602 0.001 1.982 17.403 17.519 17.682 2031.019 16.906
No logging - B past 17.441 0.001 1.800 17.268 17.376 17.523 2025.385 16.765
Text file - A current 411.200 0.011 18.183 406.001 409.264 413.409 21941.640 393.708
Text file - A past 409.539 0.037 47.057 398.809 402.573 410.222 45625.747 385.628
Text file - B current 321.209 0.014 22.850 316.620 319.286 322.723 33904.530 307.309
Text file - B past 322.207 0.038 50.250 308.219 311.932 336.982 40844.003 299.489
Binary TCP - A current 216.646 0.004 7.067 212.630 216.500 219.847 1638.114 137.346
Binary TCP - A past 208.904 0.007 8.654 202.646 208.303 213.579 2320.055 133.827
Binary TCP - B current 146.151 0.007 11.122 138.072 145.118 155.444 2294.889 80.626
Binary TCP - B past 144.685 0.009 11.923 135.673 143.471 153.861 1620.321 79.930

OpenTelemetry

OpenTelemetry (https://opentelemetry.io/) is a widely used monitoring tool, compatible with many programming languages. MooBench measures the overhead of OpenTelemetry using the following configurations:

  • No instrumentation is the baseline execution time of the benchmark program.
  • No Logging measures the execution time of the benchmark program with OpenTelemetry instrumentation, but without measuring or
    logging anything.
  • Logging measures the execution time of the benchmark program with OpenTelemetry instrumentation and writing every output to the standard output. Text file logging is slow and getting more irrelevant in the performance benchmarks. The diagram below, therefore, does not show the results.
  • Zipkin measures the execution time of the benchmark program with obtaining every method invocation and sending them to a local Zipkin instance.
  • Prometheus measures the execution time of the benchmark program with obtaining only aggregated execution times, which are stored by a local Prometheus instance.
setup run mean ci sd 1.quartile median 3.quartile max min
No instrumentation current 0.073 0.000 0.066 0.072 0.073 0.073 130.184 0.068
No instrumentation past 0.073 0.000 0.156 0.072 0.072 0.073 329.645 0.068
No logging current 4.295 0.007 11.187 4.108 4.170 4.287 7279.509 3.893
No logging past 4.319 0.008 10.973 4.126 4.206 4.307 6639.910 3.933
Zipkin current 6.984 0.010 15.629 6.526 6.646 6.996 6868.047 6.333
Zipkin past 7.003 0.013 15.884 6.595 6.675 6.982 5905.410 6.362
Prometheus current 6.263 0.007 11.561 6.062 6.126 6.235 5987.473 5.827
Prometheus past 6.256 0.010 12.599 6.057 6.129 6.215 8969.217 5.836

InspectIT

InspectIT (https://www.inspectit.rocks/) is a widely used monitoring tool, which is allows to trace the execution of Java programs. MooBench
measures the overhead of InspectIT using the following configurations:

  • No instrumentation is the baseline execution time of the benchmark program.
  • Deactivated processing measures the execution time of the benchmark program with inspectIT instrumentation, but without measuring or logging
    anything.
  • No Logging measures the execution time of the benchmark program with inspectIT instrumentation and just discarding the measurement results.
  • Zipkin measures the execution time of the benchmark program with obtaining every method invocation and sending them to a local Zipkin instance.
  • Prometheus measures the execution time of the benchmark program with obtaining only aggregated execution times, which are stored by a local Prometheus instance.
setup run mean ci sd 1.quartile median 3.quartile max min
No instrumentation current 0.073 0.000 0.206 0.072 0.072 0.073 624.309 0.068
No instrumentation past 0.073 0.000 0.452 0.072 0.072 0.073 1299.562 0.068
Deactivated probe current 1.799 0.013 20.834 0.073 2.427 2.497 14118.760 0.068
Deactivated probe past 2.382 0.019 23.991 2.350 2.475 2.518 12737.798 0.623
No logging current 12.318 0.009 14.691 11.596 12.094 12.364 6918.689 11.222
No logging past 9.389 0.011 13.680 5.760 11.605 12.012 6219.207 2.760
Zipkin current 12.057 0.014 22.458 11.383 11.636 12.179 11045.300 11.020
Zipkin past 10.290 0.015 19.811 11.207 11.393 11.844 8200.646 1.412
Prometheus current 3.788 0.013 20.396 0.073 5.041 5.251 14098.430 0.068
Prometheus past 4.523 0.016 20.025 2.657 5.311 5.534 15309.345 0.719