Performance Benchmarks

On this page, we collect performance benchmarks of Kieker monitoring and other instrumentation frameworks, such as OpenTelemetry and inspectIT, for comparison utilizing MooBench.

Details of the comparison of OpenTelemetry, inspectIT and Kieker can be found in https://oceanrep.geomar.de/id/eprint/55367/1/SSP2021overhead.pdf

Kieker Observability Framework – Java

These are the performance benchmark results for Kieker 4 Java probes utilizing different writer back end.

  • No instrumentation serves as a baseline value indicating how much execution time can be contributed to the benchmark program itself.
  • Deactivated probe as above, but with woven in probes that do not produce any output and do not aggregate information.
  • No logging are activated probes, but the collected data is not send utilizing a null writer.
  • Text File utilizes the text file logging. This is usually slow, as data is written in text files and getting more irrelevant in the performance benchmarks. The diagram below, therefore, does not show the results.
  • Binary File utilizes the binary file logging without compression.
  • Binary TCP utilizes a TCP connection to send the monitoring data to a logging service.
setup run mean ci sd 1.quartile median 3.quartile max min
No instrumentation current 0.107 0.000 0.037 0.106 0.107 0.107 18.338 0.103
No instrumentation past 0.106 0.000 0.041 0.105 0.105 0.106 18.300 0.103
Deactivated probe current 0.719 0.007 3.667 0.686 0.704 0.723 2632.995 0.662
Deactivated probe past 0.711 0.014 7.344 0.676 0.690 0.707 6574.008 0.647
No logging current 2.704 0.009 4.600 2.650 2.678 2.709 2689.634 2.571
No logging past 2.672 0.010 5.070 2.619 2.640 2.667 2672.071 2.557
Binary file current 5.194 0.033 16.665 3.493 4.446 6.752 12503.070 3.320
Binary file past 5.116 0.025 12.501 3.498 4.424 6.052 10592.934 3.119
Binary TCP current 7.045 0.012 6.094 6.788 6.959 7.386 2654.065 3.323
Binary TCP past 6.845 0.012 6.198 6.643 6.946 7.393 3460.576 3.048

Kieker Observability Framework – Python

Kieker 4 Python is the reimplementation of the probe and monitoring side of Kieker addressing programs written in Python. Here we also want to know how well certain parts perform.

  • No instrumentation is the baseline execution time of the benchmark program.
  • Deactivated probe is the execution time with integrated probes, but these are deactivated.
  • No logging is with active probes, but with a null writer.
  • Text File is with an activated probe logging to a text file.
  • Binary TCP is with an activated probe logging to a remote logging server.

Each of these setups, except of no instrumentation, are tested utilizing two different probe introduction methods (here labeled A and B).

setup run mean ci sd 1.quartile median 3.quartile max min
No instrumentation current 1.625 0.000 0.187 1.610 1.618 1.627 60.361 1.568
No instrumentation past 1.579 0.000 0.202 1.502 1.603 1.618 101.180 1.444
Deactivated probe - A current 1.594 0.000 0.164 1.484 1.631 1.643 20.915 1.422
Deactivated probe - A past 1.552 0.000 0.169 1.509 1.529 1.626 25.488 1.417
Deactivated probe - B current 3.897 0.001 0.346 3.731 3.751 4.137 154.395 3.668
Deactivated probe - B past 4.039 0.001 0.332 3.849 3.924 4.269 71.405 3.759
No logging - A current 1.596 0.000 0.179 1.496 1.644 1.663 23.383 1.428
No logging - A past 1.542 0.000 0.207 1.475 1.486 1.629 32.388 1.422
No logging - B current 45.295 0.004 1.957 44.433 44.763 45.203 232.776 43.116
No logging - B past 44.329 0.003 1.767 43.614 43.868 44.200 229.500 42.664
Text file - A current 1.496 0.000 0.144 1.481 1.489 1.498 22.437 1.432
Text file - A past 1.544 0.000 0.178 1.491 1.506 1.604 42.019 1.426
Text file - B current 1303.967 0.872 444.850 1252.295 1268.780 1353.263 123691.300 1225.412
Text file - B past 1317.143 1.361 694.501 1258.629 1290.317 1361.947 277588.948 1231.250
Binary TCP - A current 1.623 0.000 0.198 1.624 1.640 1.653 95.799 1.422
Binary TCP - A past 1.613 0.000 0.160 1.596 1.606 1.616 28.039 1.529
Binary TCP - B current 479.327 0.071 36.476 469.511 472.145 477.045 1250.649 236.278
Binary TCP - B past 488.592 0.118 60.304 479.949 482.744 487.795 32470.853 234.408

OpenTelemetry

OpenTelemetry (https://opentelemetry.io/) is a widely used monitoring tool, compatible with many programming languages. MooBench measures the overhead of OpenTelemetry using the following configurations:

  • No instrumentation is the baseline execution time of the benchmark program.
  • No Logging measures the execution time of the benchmark program with OpenTelemetry instrumentation, but without measuring or
    logging anything.
  • Logging measures the execution time of the benchmark program with OpenTelemetry instrumentation and writing every output to the standard output. Text file logging is slow and getting more irrelevant in the performance benchmarks. The diagram below, therefore, does not show the results.
  • Zipkin measures the execution time of the benchmark program with obtaining every method invocation and sending them to a local Zipkin instance.
  • Prometheus measures the execution time of the benchmark program with obtaining only aggregated execution times, which are stored by a local Prometheus instance.
setup run mean ci sd 1.quartile median 3.quartile max min
No instrumentation current 0.112 0.000 0.038 0.110 0.112 0.112 17.742 0.106
No instrumentation past 0.107 0.000 0.057 0.106 0.107 0.108 39.498 0.103
No logging current 6.179 0.023 11.535 6.019 6.050 6.090 6641.389 5.865
No logging past 6.215 0.028 14.349 6.048 6.095 6.169 7974.036 5.851
Zipkin current 9.176 0.024 12.496 8.577 8.631 8.709 3412.705 8.366
Zipkin past 10.255 0.044 22.234 9.484 9.548 9.633 14524.329 9.201

InspectIT

InspectIT (https://www.inspectit.rocks/) is a widely used monitoring tool, which is allows to trace the execution of Java programs. MooBench
measures the overhead of InspectIT using the following configurations:

  • No instrumentation is the baseline execution time of the benchmark program.
  • Deactivated processing measures the execution time of the benchmark program with inspectIT instrumentation, but without measuring or logging
    anything.
  • No Logging measures the execution time of the benchmark program with inspectIT instrumentation and just discarding the measurement results.
  • Zipkin measures the execution time of the benchmark program with obtaining every method invocation and sending them to a local Zipkin instance.
  • Prometheus measures the execution time of the benchmark program with obtaining only aggregated execution times, which are stored by a local Prometheus instance.
setup run mean ci sd 1.quartile median 3.quartile max min
No instrumentation current 0.105 0.000 0.037 0.105 0.105 0.105 16.470 0.103
No instrumentation past 0.108 0.000 0.038 0.108 0.108 0.109 16.457 0.103
Deactivated probe current 14.283 0.021 10.959 14.015 14.095 14.197 3130.755 13.659
Deactivated probe past 13.711 0.023 11.883 13.435 13.510 13.600 5194.710 12.874
No logging current 40.993 0.051 26.209 39.839 40.145 40.651 20754.910 35.247
No logging past 39.155 0.060 30.703 38.129 38.337 38.625 23155.482 36.696
Zipkin current 45.233 0.057 29.097 42.700 42.895 43.193 14689.030 38.018
Zipkin past 43.161 0.094 47.898 40.633 40.876 41.226 35469.788 35.948
Prometheus current 26.251 0.025 12.687 25.806 25.946 26.103 3583.350 25.178
Prometheus past 25.106 0.039 20.112 24.624 24.749 24.899 13613.036 24.115