Performance Benchmarks

On this page, we collect performance benchmarks of Kieker monitoring and other instrumentation frameworks, such as OpenTelemetry and inspectIT, for comparison utilizing MooBench.

Details of the comparison of OpenTelemetry, inspectIT and Kieker can be found in https://oceanrep.geomar.de/id/eprint/55367/1/SSP2021overhead.pdf

Kieker Observability Framework – Java

These are the performance benchmark results for Kieker 4 Java probes utilizing different writer back end.

  • No instrumentation serves as a baseline value indicating how much execution time can be contributed to the benchmark program itself.
  • Deactivated probe as above, but with woven in probes that do not produce any output and do not aggregate information.
  • No logging are activated probes, but the collected data is not send utilizing a null writer.
  • Text File utilizes the text file logging. This is usually slow, as data is written in text files and getting more irrelevant in the performance benchmarks. The diagram below, therefore, does not show the results.
  • Binary File utilizes the binary file logging without compression.
  • Binary TCP utilizes a TCP connection to send the monitoring data to a logging service.
setup run mean ci sd 1.quartile median 3.quartile max min
No instrumentation current 0.108 0.000 0.066 0.106 0.107 0.108 43.890 0.101
No instrumentation past 0.108 0.000 0.063 0.106 0.107 0.108 38.786 0.102
Deactivated probe current 0.140 0.000 0.129 0.138 0.139 0.140 80.789 0.131
Deactivated probe past 0.142 0.000 0.103 0.139 0.140 0.141 71.030 0.133
No collection current 1.694 0.020 10.437 1.633 1.643 1.673 9763.777 1.600
No collection past 1.643 0.010 4.950 1.589 1.598 1.621 1808.038 1.554
Binary file current 3.460 0.008 4.305 2.177 2.933 4.894 3238.525 2.022
Binary file past 3.365 0.010 5.267 2.169 2.844 4.799 2688.190 1.989
Binary TCP current 4.720 0.011 5.561 3.252 5.152 5.768 1011.228 2.314
Binary TCP past 4.638 0.017 8.629 3.245 4.393 5.822 6524.094 2.294

Kieker Observability Framework – Python

Kieker 4 Python is the reimplementation of the probe and monitoring side of Kieker addressing programs written in Python. Here we also want to know how well certain parts perform.

  • No instrumentation is the baseline execution time of the benchmark program.
  • Deactivated probe is the execution time with integrated probes, but these are deactivated.
  • No logging is with active probes, but with a null writer.
  • Text File is with an activated probe logging to a text file.
  • Binary TCP is with an activated probe logging to a remote logging server.

Each of these setups, except of no instrumentation, are tested utilizing two different probe introduction methods (here labeled A and B).

setup run mean ci sd 1.quartile median 3.quartile max min
No instrumentation current 1.410 0.001 0.265 1.392 1.400 1.414 74.379 1.360
No instrumentation past 1.541 0.001 0.281 1.518 1.530 1.546 122.958 1.465
Deactivated probe - A current 1.426 0.001 0.280 1.406 1.415 1.431 76.258 1.370
Deactivated probe - A past 1.559 0.001 0.509 1.519 1.541 1.568 356.095 1.457
Deactivated probe - B current 1.402 0.000 0.240 1.383 1.393 1.407 64.718 1.347
Deactivated probe - B past 1.531 0.001 0.301 1.503 1.521 1.544 81.932 1.451
No logging - A current 1.430 0.001 0.271 1.403 1.413 1.444 83.791 1.365
No logging - A past 1.551 0.001 0.323 1.519 1.535 1.555 148.579 1.463
No logging - B current 1.407 0.000 0.234 1.387 1.397 1.413 73.978 1.349
No logging - B past 1.546 0.001 0.361 1.513 1.531 1.551 218.719 1.458
Text file - A current 1.419 0.000 0.244 1.397 1.408 1.426 62.094 1.356
Text file - A past 1.574 0.001 0.335 1.540 1.562 1.586 151.437 1.479
Text file - B current 1.425 0.001 0.265 1.406 1.415 1.427 79.853 1.364
Text file - B past 1.528 0.001 0.321 1.506 1.516 1.530 115.095 1.456
Binary TCP - A current 1.431 0.001 0.296 1.398 1.410 1.428 100.542 1.360
Binary TCP - A past 1.541 0.001 0.372 1.516 1.530 1.548 213.429 1.452
Binary TCP - B current 1.428 0.001 0.290 1.410 1.418 1.430 132.047 1.368
Binary TCP - B past 1.538 0.001 0.382 1.513 1.526 1.543 196.287 1.446

OpenTelemetry

OpenTelemetry (https://opentelemetry.io/) is a widely used monitoring tool, compatible with many programming languages. MooBench measures the overhead of OpenTelemetry using the following configurations:

  • No instrumentation is the baseline execution time of the benchmark program.
  • No Logging measures the execution time of the benchmark program with OpenTelemetry instrumentation, but without measuring or
    logging anything.
  • Logging measures the execution time of the benchmark program with OpenTelemetry instrumentation and writing every output to the standard output. Text file logging is slow and getting more irrelevant in the performance benchmarks. The diagram below, therefore, does not show the results.
  • Zipkin measures the execution time of the benchmark program with obtaining every method invocation and sending them to a local Zipkin instance.
  • Prometheus measures the execution time of the benchmark program with obtaining only aggregated execution times, which are stored by a local Prometheus instance.
setup run mean ci sd 1.quartile median 3.quartile max min
No instrumentation current 0.111 0.006 3.253 0.106 0.107 0.109 3249.927 0.100
No instrumentation past 0.109 0.000 0.055 0.107 0.108 0.109 37.598 0.102
No logging current 6.758 0.024 12.024 6.521 6.583 6.655 4619.383 6.278
No logging past 6.996 0.022 11.297 6.762 6.822 6.892 4503.085 6.520
Zipkin current 13.043 0.036 18.543 11.955 12.113 14.160 4947.788 11.354
Zipkin past 13.388 0.034 17.116 12.837 12.960 13.558 4574.826 12.330
Prometheus current 10.117 0.022 11.268 9.840 9.925 10.024 3963.120 9.498
Prometheus past 10.452 0.024 12.484 10.163 10.245 10.339 4281.820 9.781

InspectIT

InspectIT (https://www.inspectit.rocks/) is a widely used monitoring tool, which is allows to trace the execution of Java programs. MooBench
measures the overhead of InspectIT using the following configurations:

  • No instrumentation is the baseline execution time of the benchmark program.
  • Deactivated processing measures the execution time of the benchmark program with inspectIT instrumentation, but without measuring or logging
    anything.
  • No Logging measures the execution time of the benchmark program with inspectIT instrumentation and just discarding the measurement results.
  • Zipkin measures the execution time of the benchmark program with obtaining every method invocation and sending them to a local Zipkin instance.
  • Prometheus measures the execution time of the benchmark program with obtaining only aggregated execution times, which are stored by a local Prometheus instance.
setup run mean ci sd 1.quartile median 3.quartile max min
No instrumentation current 0.107 0.000 0.079 0.106 0.107 0.108 63.929 0.100
No instrumentation past 0.108 0.000 0.108 0.107 0.108 0.109 73.295 0.102
Deactivated probe current 3.752 0.045 22.795 3.559 3.600 3.650 10677.370 3.403
Deactivated probe past 3.522 0.048 24.363 3.266 3.305 3.356 11667.280 3.111
No logging current 24.850 0.028 14.516 24.246 24.429 24.652 4088.946 23.357
No logging past 24.685 0.031 15.692 24.027 24.194 24.398 3903.724 23.190
Zipkin current 25.302 0.041 20.908 24.285 24.505 24.779 4870.008 23.290
Zipkin past 30.995 0.048 24.705 29.736 30.830 31.129 7690.691 24.736
Prometheus current 8.844 0.050 25.310 8.617 8.686 8.760 12973.000 8.327
Prometheus past 8.743 0.041 21.083 8.430 8.596 8.696 12651.765 8.089