Performance Benchmarks

On this page, we collect performance benchmarks of Kieker monitoring and other instrumentation frameworks, such as OpenTelemetry and inspectIT, for comparison utilizing MooBench.

Details of the comparison of OpenTelemetry, inspectIT and Kieker can be found in https://oceanrep.geomar.de/id/eprint/55367/1/SSP2021overhead.pdf

Kieker Observability Framework – Java

These are the performance benchmark results for Kieker 4 Java probes utilizing different writer back end.

  • No instrumentation serves as a baseline value indicating how much execution time can be contributed to the benchmark program itself.
  • Deactivated probe as above, but with woven in probes that do not produce any output and do not aggregate information.
  • No logging are activated probes, but the collected data is not send utilizing a null writer.
  • Text File utilizes the text file logging. This is usually slow, as data is written in text files and getting more irrelevant in the performance benchmarks. The diagram below, therefore, does not show the results.
  • Binary File utilizes the binary file logging without compression.
  • Binary TCP utilizes a TCP connection to send the monitoring data to a logging service.
setup run mean ci sd 1.quartile median 3.quartile max min
No instrumentation current 0.105 0.000 0.032 0.105 0.105 0.105 9.091 0.103
No instrumentation past 0.111 0.000 0.030 0.110 0.111 0.111 10.309 0.105
Deactivated probe current 0.143 0.000 0.039 0.142 0.143 0.143 24.608 0.139
Deactivated probe past 0.705 0.007 3.530 0.673 0.689 0.707 2549.518 0.642
No logging current 1.896 0.006 3.259 1.847 1.858 1.878 441.046 1.826
No logging past 2.683 0.008 3.967 2.640 2.666 2.695 2494.489 2.435
Binary file current 3.952 0.023 11.551 2.562 3.663 5.121 11083.390 2.212
Binary file past 5.073 0.029 14.677 3.510 4.475 5.990 13375.797 3.201
Binary TCP current 4.421 0.049 25.171 4.311 4.604 4.821 24668.610 2.069
Binary TCP past 6.871 0.024 12.055 6.554 6.907 7.488 9812.935 3.102

Kieker Observability Framework – Python

Kieker 4 Python is the reimplementation of the probe and monitoring side of Kieker addressing programs written in Python. Here we also want to know how well certain parts perform.

  • No instrumentation is the baseline execution time of the benchmark program.
  • Deactivated probe is the execution time with integrated probes, but these are deactivated.
  • No logging is with active probes, but with a null writer.
  • Text File is with an activated probe logging to a text file.
  • Binary TCP is with an activated probe logging to a remote logging server.

Each of these setups, except of no instrumentation, are tested utilizing two different probe introduction methods (here labeled A and B).

setup run mean ci sd 1.quartile median 3.quartile max min
No instrumentation current 1.507 0.001 0.661 1.480 1.495 1.514 600.564 1.430
No instrumentation past 1.502 0.000 0.156 1.483 1.495 1.509 42.481 1.431
Deactivated probe - A current 1.549 0.000 0.246 1.516 1.531 1.552 60.601 1.467
Deactivated probe - A past 1.555 0.000 0.170 1.515 1.527 1.617 26.238 1.425
Deactivated probe - B current 1.552 0.001 0.635 1.526 1.539 1.558 583.056 1.475
Deactivated probe - B past 1.854 0.000 0.202 1.788 1.805 1.935 36.563 1.722
No logging - A current 1.559 0.001 0.750 1.524 1.542 1.564 707.533 1.463
No logging - A past 1.528 0.000 0.177 1.478 1.501 1.528 27.931 1.423
No logging - B current 1.534 0.001 0.612 1.504 1.521 1.542 556.071 1.449
No logging - B past 6.867 0.001 0.332 6.768 6.796 6.876 38.654 6.634
Text file - A current 1.527 0.001 0.257 1.493 1.508 1.529 113.731 1.444
Text file - A past 1.533 0.000 0.162 1.498 1.513 1.563 31.799 1.443
Text file - B current 1.522 0.001 0.260 1.493 1.507 1.527 55.790 1.427
Text file - B past 168.346 0.146 74.334 161.357 164.505 174.274 26794.209 158.074
Binary TCP - A current 1.547 0.001 0.545 1.514 1.530 1.552 331.045 1.459
Binary TCP - A past 1.504 0.000 0.167 1.466 1.483 1.498 27.510 1.420
Binary TCP - B current 1.522 0.001 0.274 1.497 1.511 1.528 72.880 1.453
Binary TCP - B past 62.563 0.009 4.566 61.189 61.570 62.442 156.618 30.923

OpenTelemetry

OpenTelemetry (https://opentelemetry.io/) is a widely used monitoring tool, compatible with many programming languages. MooBench measures the overhead of OpenTelemetry using the following configurations:

  • No instrumentation is the baseline execution time of the benchmark program.
  • No Logging measures the execution time of the benchmark program with OpenTelemetry instrumentation, but without measuring or
    logging anything.
  • Logging measures the execution time of the benchmark program with OpenTelemetry instrumentation and writing every output to the standard output. Text file logging is slow and getting more irrelevant in the performance benchmarks. The diagram below, therefore, does not show the results.
  • Zipkin measures the execution time of the benchmark program with obtaining every method invocation and sending them to a local Zipkin instance.
  • Prometheus measures the execution time of the benchmark program with obtaining only aggregated execution times, which are stored by a local Prometheus instance.
setup run mean ci sd 1.quartile median 3.quartile max min
No instrumentation current 0.110 0.000 0.068 0.108 0.109 0.110 30.306 0.103
No instrumentation past 0.106 0.000 0.032 0.105 0.105 0.106 13.458 0.102
No logging current 7.503 0.018 9.315 7.328 7.384 7.450 2483.429 7.104
No logging past 6.975 0.014 7.240 6.826 6.873 6.927 2092.719 6.612
Zipkin current 14.551 0.039 19.724 14.154 14.254 14.368 8591.219 13.718
Zipkin past 11.749 0.027 13.662 10.856 10.923 11.058 4253.031 9.766
Prometheus current 13.414 0.023 11.538 13.116 13.205 13.306 3933.300 12.717
Prometheus past 11.103 0.018 9.196 10.794 11.027 11.150 2384.793 10.273

InspectIT

InspectIT (https://www.inspectit.rocks/) is a widely used monitoring tool, which is allows to trace the execution of Java programs. MooBench
measures the overhead of InspectIT using the following configurations:

  • No instrumentation is the baseline execution time of the benchmark program.
  • Deactivated processing measures the execution time of the benchmark program with inspectIT instrumentation, but without measuring or logging
    anything.
  • No Logging measures the execution time of the benchmark program with inspectIT instrumentation and just discarding the measurement results.
  • Zipkin measures the execution time of the benchmark program with obtaining every method invocation and sending them to a local Zipkin instance.
  • Prometheus measures the execution time of the benchmark program with obtaining only aggregated execution times, which are stored by a local Prometheus instance.
setup run mean ci sd 1.quartile median 3.quartile max min
No instrumentation current 0.109 0.000 0.086 0.108 0.109 0.110 69.851 0.103
No instrumentation past 0.107 0.000 0.072 0.106 0.107 0.107 61.005 0.104
Deactivated probe current 3.320 0.052 26.279 3.140 3.181 3.234 12551.880 2.984
Deactivated probe past 9.196 0.165 84.131 8.899 8.955 9.023 80247.415 8.645
No logging current 24.535 0.030 15.263 23.895 24.071 24.268 5230.042 22.947
No logging past 32.136 0.034 17.355 31.346 31.532 31.813 8386.824 30.163
Zipkin current 29.928 0.042 21.645 29.674 29.914 30.150 5909.618 23.938
Zipkin past 34.202 0.095 48.436 32.409 32.614 32.903 38620.381 28.587
Prometheus current 8.694 0.033 16.758 8.429 8.603 8.695 10360.250 8.113
Prometheus past 17.679 0.031 15.817 17.310 17.426 17.601 7344.764 16.784