Performance Benchmarks

On this page, we collect performance benchmarks of Kieker monitoring and other instrumentation frameworks, such as OpenTelemetry and inspectIT, for comparison utilizing MooBench.

Details of the comparison of OpenTelemetry, inspectIT and Kieker can be found in https://oceanrep.geomar.de/id/eprint/55367/1/SSP2021overhead.pdf

Kieker Observability Framework – Java

These are the performance benchmark results for Kieker 4 Java probes utilizing different writer back end.

  • No instrumentation serves as a baseline value indicating how much execution time can be contributed to the benchmark program itself.
  • Deactivated probe as above, but with woven in probes that do not produce any output and do not aggregate information.
  • No logging are activated probes, but the collected data is not send utilizing a null writer.
  • Text File utilizes the text file logging. This is usually slow, as data is written in text files and getting more irrelevant in the performance benchmarks. The diagram below, therefore, does not show the results.
  • Binary File utilizes the binary file logging without compression.
  • Binary TCP utilizes a TCP connection to send the monitoring data to a logging service.
setup run mean ci sd 1.quartile median 3.quartile max min
No instrumentation current 0.108 0.000 0.056 0.106 0.107 0.108 33.575 0.101
No instrumentation past 0.108 0.000 0.059 0.106 0.108 0.108 39.961 0.102
Deactivated probe current 0.145 0.000 0.157 0.143 0.143 0.144 89.934 0.134
Deactivated probe past 0.144 0.000 0.057 0.143 0.143 0.145 30.495 0.133
No collection current 1.713 0.010 5.107 1.648 1.655 1.667 615.351 1.618
No collection past 1.675 0.009 4.510 1.622 1.630 1.649 1190.457 1.591
Binary file current 3.405 0.012 6.243 2.160 2.799 4.841 2587.610 2.008
Binary file past 4.457 0.011 5.647 2.337 4.246 5.793 2981.489 1.987
Binary TCP current 4.666 0.020 10.217 3.274 4.864 5.814 7984.876 2.310
Binary TCP past 4.856 0.021 10.517 3.440 4.946 5.887 8192.977 2.300

Kieker Observability Framework – Python

Kieker 4 Python is the reimplementation of the probe and monitoring side of Kieker addressing programs written in Python. Here we also want to know how well certain parts perform.

  • No instrumentation is the baseline execution time of the benchmark program.
  • Deactivated probe is the execution time with integrated probes, but these are deactivated.
  • No logging is with active probes, but with a null writer.
  • Text File is with an activated probe logging to a text file.
  • Binary TCP is with an activated probe logging to a remote logging server.

Each of these setups, except of no instrumentation, are tested utilizing two different probe introduction methods (here labeled A and B).

setup run mean ci sd 1.quartile median 3.quartile max min
No instrumentation current 1.500 0.001 0.279 1.473 1.485 1.497 75.791 1.444
No instrumentation past 1.505 0.001 0.574 1.478 1.489 1.506 476.424 1.444
Deactivated probe - A current 1.541 0.006 3.202 1.504 1.530 1.553 3192.449 1.467
Deactivated probe - A past 1.503 0.001 0.364 1.473 1.489 1.509 226.307 1.437
Deactivated probe - B current 1.479 0.001 0.279 1.448 1.469 1.496 64.215 1.415
Deactivated probe - B past 1.490 0.001 0.364 1.466 1.475 1.492 216.362 1.429
No logging - A current 1.455 0.001 0.273 1.430 1.438 1.452 62.697 1.398
No logging - A past 1.529 0.001 0.375 1.500 1.511 1.525 220.517 1.464
No logging - B current 1.498 0.001 0.332 1.467 1.481 1.501 218.729 1.434
No logging - B past 1.498 0.001 0.379 1.467 1.486 1.503 237.889 1.427
Text file - A current 1.537 0.001 0.688 1.503 1.525 1.550 618.598 1.472
Text file - A past 1.557 0.001 0.442 1.503 1.518 1.541 207.389 1.461
Text file - B current 1.521 0.001 0.284 1.497 1.510 1.525 71.016 1.461
Text file - B past 1.524 0.001 0.392 1.488 1.506 1.529 233.560 1.432
Binary TCP - A current 1.495 0.001 0.301 1.473 1.487 1.501 88.617 1.426
Binary TCP - A past 1.511 0.001 0.336 1.484 1.497 1.514 170.013 1.444
Binary TCP - B current 1.472 0.000 0.252 1.451 1.461 1.480 65.733 1.418
Binary TCP - B past 1.473 0.001 0.349 1.449 1.461 1.480 185.987 1.413

OpenTelemetry

OpenTelemetry (https://opentelemetry.io/) is a widely used monitoring tool, compatible with many programming languages. MooBench measures the overhead of OpenTelemetry using the following configurations:

  • No instrumentation is the baseline execution time of the benchmark program.
  • No Logging measures the execution time of the benchmark program with OpenTelemetry instrumentation, but without measuring or
    logging anything.
  • Logging measures the execution time of the benchmark program with OpenTelemetry instrumentation and writing every output to the standard output. Text file logging is slow and getting more irrelevant in the performance benchmarks. The diagram below, therefore, does not show the results.
  • Zipkin measures the execution time of the benchmark program with obtaining every method invocation and sending them to a local Zipkin instance.
  • Prometheus measures the execution time of the benchmark program with obtaining only aggregated execution times, which are stored by a local Prometheus instance.
setup run mean ci sd 1.quartile median 3.quartile max min
No instrumentation current 0.108 0.000 0.035 0.106 0.107 0.108 15.485 0.101
No instrumentation past 0.109 0.000 0.105 0.106 0.107 0.109 79.342 0.102
No logging current 7.091 0.023 11.509 6.845 6.905 6.976 4209.424 6.587
No logging past 7.204 0.020 10.111 6.988 7.045 7.113 3161.640 6.743
Zipkin current 13.543 0.034 17.504 13.158 13.268 13.398 5083.536 12.642
Zipkin past 13.356 0.038 19.394 12.916 13.151 13.288 7223.200 11.957
Prometheus current 10.228 0.027 13.534 9.932 10.013 10.104 4883.598 9.545
Prometheus past 11.341 0.024 12.226 11.044 11.142 11.248 4397.754 10.624

InspectIT

InspectIT (https://www.inspectit.rocks/) is a widely used monitoring tool, which is allows to trace the execution of Java programs. MooBench
measures the overhead of InspectIT using the following configurations:

  • No instrumentation is the baseline execution time of the benchmark program.
  • Deactivated processing measures the execution time of the benchmark program with inspectIT instrumentation, but without measuring or logging
    anything.
  • No Logging measures the execution time of the benchmark program with inspectIT instrumentation and just discarding the measurement results.
  • Zipkin measures the execution time of the benchmark program with obtaining every method invocation and sending them to a local Zipkin instance.
  • Prometheus measures the execution time of the benchmark program with obtaining only aggregated execution times, which are stored by a local Prometheus instance.
setup run mean ci sd 1.quartile median 3.quartile max min
No instrumentation current 0.112 0.000 0.037 0.110 0.111 0.113 15.690 0.101
No instrumentation past 0.109 0.001 0.355 0.107 0.108 0.109 346.343 0.101
Deactivated probe current 3.824 0.043 21.805 3.676 3.722 3.770 10473.960 3.497
Deactivated probe past 3.290 0.047 23.977 3.130 3.165 3.209 11707.912 2.979
No logging current 24.905 0.033 16.621 24.245 24.423 24.649 4347.698 23.450
No logging past 24.588 0.032 16.129 23.895 24.082 24.311 5774.934 22.960
Zipkin current 30.132 0.051 26.211 26.024 30.523 30.866 7900.938 24.440
Zipkin past 28.750 0.045 23.064 28.070 28.471 28.970 5889.270 24.689
Prometheus current 8.850 0.047 24.078 8.593 8.653 8.744 11996.470 8.369
Prometheus past 8.715 0.046 23.224 8.430 8.548 8.646 12583.584 8.135