Performance Benchmarks

On this page, we collect performance benchmarks of Kieker monitoring and other instrumentation frameworks, such as OpenTelemetry and inspectIT, for comparison utilizing MooBench.

Details of the comparison of OpenTelemetry, inspectIT and Kieker can be found in https://oceanrep.geomar.de/id/eprint/55367/1/SSP2021overhead.pdf

Kieker Observability Framework – Java

These are the performance benchmark results for Kieker 4 Java probes utilizing different writer back end.

  • No instrumentation serves as a baseline value indicating how much execution time can be contributed to the benchmark program itself.
  • Deactivated probe as above, but with woven in probes that do not produce any output and do not aggregate information.
  • No logging are activated probes, but the collected data is not send utilizing a null writer.
  • Text File utilizes the text file logging. This is usually slow, as data is written in text files and getting more irrelevant in the performance benchmarks. The diagram below, therefore, does not show the results.
  • Binary File utilizes the binary file logging without compression.
  • Binary TCP utilizes a TCP connection to send the monitoring data to a logging service.
setup run mean ci sd 1.quartile median 3.quartile max min
No instrumentation current 0.123 0.000 0.090 0.121 0.122 0.123 60.689 0.116
No instrumentation past 0.120 0.000 0.083 0.117 0.119 0.121 49.603 0.112
Deactivated probe current 0.152 0.000 0.070 0.150 0.152 0.154 41.853 0.142
Deactivated probe past 0.152 0.001 0.265 0.149 0.150 0.152 246.661 0.142
No collection current 1.793 0.010 5.150 1.728 1.734 1.747 983.962 1.696
No collection past 1.740 0.008 4.234 1.688 1.697 1.709 778.064 1.663
Binary file current 3.949 0.013 6.867 2.302 3.855 5.154 2512.458 2.094
Binary file past 4.520 0.013 6.400 2.546 4.481 5.776 3737.410 2.090
Binary TCP current 4.779 0.016 8.241 4.672 4.873 5.113 6691.996 2.076
Binary TCP past 5.124 0.019 9.525 3.656 5.544 6.006 6505.411 2.386

Kieker Observability Framework – Python

Kieker 4 Python is the reimplementation of the probe and monitoring side of Kieker addressing programs written in Python. Here we also want to know how well certain parts perform.

  • No instrumentation is the baseline execution time of the benchmark program.
  • Deactivated probe is the execution time with integrated probes, but these are deactivated.
  • No logging is with active probes, but with a null writer.
  • Text File is with an activated probe logging to a text file.
  • Binary TCP is with an activated probe logging to a remote logging server.

Each of these setups, except of no instrumentation, are tested utilizing two different probe introduction methods (here labeled A and B).

setup run mean ci sd 1.quartile median 3.quartile max min
No instrumentation current 1.440 0.001 0.276 1.413 1.421 1.439 73.558 1.380
No instrumentation past 1.440 0.001 0.304 1.418 1.427 1.442 103.535 1.380
Deactivated probe - A current 1.522 0.006 3.313 1.492 1.509 1.530 3246.617 1.433
Deactivated probe - A past 1.445 0.001 0.359 1.421 1.429 1.446 181.051 1.381
Deactivated probe - B current 1.446 0.000 0.234 1.425 1.434 1.452 61.700 1.386
Deactivated probe - B past 1.445 0.001 0.405 1.421 1.432 1.448 236.255 1.382
No logging - A current 1.484 0.001 0.678 1.458 1.468 1.489 629.420 1.420
No logging - A past 1.477 0.001 0.293 1.447 1.458 1.480 77.820 1.404
No logging - B current 1.441 0.000 0.245 1.420 1.430 1.445 68.465 1.384
No logging - B past 1.447 0.001 0.277 1.422 1.433 1.453 77.520 1.377
Text file - A current 1.447 0.000 0.228 1.422 1.431 1.450 59.132 1.383
Text file - A past 1.445 0.001 0.294 1.422 1.431 1.448 105.928 1.384
Text file - B current 1.474 0.000 0.243 1.449 1.462 1.484 63.863 1.403
Text file - B past 1.451 0.001 0.381 1.427 1.438 1.453 215.304 1.387
Binary TCP - A current 1.498 0.001 0.302 1.450 1.468 1.493 92.309 1.400
Binary TCP - A past 1.448 0.001 0.272 1.425 1.433 1.447 79.276 1.390
Binary TCP - B current 1.442 0.007 3.323 1.409 1.418 1.437 3309.745 1.373
Binary TCP - B past 1.448 0.001 0.482 1.421 1.435 1.457 351.898 1.374

OpenTelemetry

OpenTelemetry (https://opentelemetry.io/) is a widely used monitoring tool, compatible with many programming languages. MooBench measures the overhead of OpenTelemetry using the following configurations:

  • No instrumentation is the baseline execution time of the benchmark program.
  • No Logging measures the execution time of the benchmark program with OpenTelemetry instrumentation, but without measuring or
    logging anything.
  • Logging measures the execution time of the benchmark program with OpenTelemetry instrumentation and writing every output to the standard output. Text file logging is slow and getting more irrelevant in the performance benchmarks. The diagram below, therefore, does not show the results.
  • Zipkin measures the execution time of the benchmark program with obtaining every method invocation and sending them to a local Zipkin instance.
  • Prometheus measures the execution time of the benchmark program with obtaining only aggregated execution times, which are stored by a local Prometheus instance.
setup run mean ci sd 1.quartile median 3.quartile max min
No instrumentation current 0.121 0.000 0.065 0.120 0.120 0.121 49.767 0.114
No instrumentation past 0.121 0.000 0.124 0.118 0.120 0.121 106.803 0.113
No logging current 6.889 0.018 9.405 6.694 6.742 6.799 3676.093 6.498
No logging past 7.139 0.020 10.218 6.937 6.988 7.048 4593.690 6.721
Zipkin current 13.497 0.037 18.680 13.090 13.203 13.334 4557.080 12.648
Zipkin past 12.827 0.037 18.690 11.669 12.795 12.992 5312.662 10.566
Prometheus current 10.819 0.032 16.225 10.482 10.561 10.653 7986.670 10.106
Prometheus past 10.201 0.025 12.593 9.912 10.000 10.100 5022.246 9.552

InspectIT

InspectIT (https://www.inspectit.rocks/) is a widely used monitoring tool, which is allows to trace the execution of Java programs. MooBench
measures the overhead of InspectIT using the following configurations:

  • No instrumentation is the baseline execution time of the benchmark program.
  • Deactivated processing measures the execution time of the benchmark program with inspectIT instrumentation, but without measuring or logging
    anything.
  • No Logging measures the execution time of the benchmark program with inspectIT instrumentation and just discarding the measurement results.
  • Zipkin measures the execution time of the benchmark program with obtaining every method invocation and sending them to a local Zipkin instance.
  • Prometheus measures the execution time of the benchmark program with obtaining only aggregated execution times, which are stored by a local Prometheus instance.
setup run mean ci sd 1.quartile median 3.quartile max min
No instrumentation current 0.122 0.000 0.053 0.120 0.122 0.123 42.476 0.113
No instrumentation past 0.119 0.000 0.090 0.117 0.118 0.120 64.329 0.111
Deactivated probe current 3.680 0.045 23.130 3.515 3.554 3.602 11509.160 3.369
Deactivated probe past 3.430 0.043 22.070 3.269 3.309 3.362 11565.201 3.146
No logging current 24.942 0.027 13.592 24.228 24.455 24.734 3011.937 23.257
No logging past 25.592 0.028 14.493 24.950 25.134 25.357 3925.366 24.028
Zipkin current 25.881 0.048 24.493 23.646 24.018 28.270 9683.436 22.459
Zipkin past 28.586 0.045 22.750 26.614 29.002 29.344 8286.624 24.362
Prometheus current 8.919 0.034 17.231 8.587 8.852 8.975 10473.530 8.263
Prometheus past 8.788 0.036 18.427 8.507 8.687 8.780 10097.189 8.230