Performance Benchmarks

On this page, we collect performance benchmarks of Kieker monitoring and other instrumentation frameworks, such as OpenTelemetry and inspectIT, for comparison utilizing MooBench.

Details of the comparison of OpenTelemetry, inspectIT and Kieker can be found in https://oceanrep.geomar.de/id/eprint/55367/1/SSP2021overhead.pdf

Kieker Observability Framework – Java

These are the performance benchmark results for Kieker 4 Java probes utilizing different writer back end.

  • No instrumentation serves as a baseline value indicating how much execution time can be contributed to the benchmark program itself.
  • Deactivated probe as above, but with woven in probes that do not produce any output and do not aggregate information.
  • No logging are activated probes, but the collected data is not send utilizing a null writer.
  • Text File utilizes the text file logging. This is usually slow, as data is written in text files and getting more irrelevant in the performance benchmarks. The diagram below, therefore, does not show the results.
  • Binary File utilizes the binary file logging without compression.
  • Binary TCP utilizes a TCP connection to send the monitoring data to a logging service.
setup run mean ci sd 1.quartile median 3.quartile max min
No instrumentation current 0.109 0.000 0.074 0.107 0.108 0.109 21.645 0.103
No instrumentation past 0.109 0.000 0.058 0.106 0.108 0.108 39.455 0.101
Deactivated probe current 0.140 0.000 0.156 0.137 0.138 0.140 113.723 0.131
Deactivated probe past 0.145 0.000 0.110 0.143 0.144 0.145 61.190 0.134
No collection current 1.647 0.010 5.074 1.578 1.586 1.613 633.406 1.547
No collection past 1.701 0.010 4.948 1.639 1.646 1.659 749.011 1.608
Binary file current 3.337 0.008 4.262 2.258 2.837 4.752 3336.838 1.981
Binary file past 3.663 0.011 5.700 2.207 3.167 5.110 2948.688 2.006
Binary TCP current 4.597 0.019 9.599 3.194 4.678 5.828 7947.764 2.294
Binary TCP past 4.799 0.020 10.431 3.340 5.006 5.871 8213.294 2.318

Kieker Observability Framework – Python

Kieker 4 Python is the reimplementation of the probe and monitoring side of Kieker addressing programs written in Python. Here we also want to know how well certain parts perform.

  • No instrumentation is the baseline execution time of the benchmark program.
  • Deactivated probe is the execution time with integrated probes, but these are deactivated.
  • No logging is with active probes, but with a null writer.
  • Text File is with an activated probe logging to a text file.
  • Binary TCP is with an activated probe logging to a remote logging server.

Each of these setups, except of no instrumentation, are tested utilizing two different probe introduction methods (here labeled A and B).

setup run mean ci sd 1.quartile median 3.quartile max min
No instrumentation current 1.512 0.001 0.326 1.485 1.498 1.514 200.553 1.450
No instrumentation past 1.494 0.001 0.338 1.469 1.480 1.493 173.531 1.438
Deactivated probe - A current 1.531 0.001 0.308 1.475 1.507 1.555 162.937 1.438
Deactivated probe - A past 1.528 0.003 1.766 1.493 1.513 1.534 1671.324 1.454
Deactivated probe - B current 1.490 0.001 0.272 1.462 1.480 1.505 81.246 1.430
Deactivated probe - B past 1.480 0.001 0.299 1.453 1.468 1.491 101.607 1.420
No logging - A current 1.474 0.001 0.289 1.454 1.466 1.480 115.849 1.422
No logging - A past 1.489 0.001 0.294 1.463 1.473 1.488 101.883 1.429
No logging - B current 1.523 0.001 0.267 1.497 1.503 1.525 65.750 1.470
No logging - B past 1.509 0.001 0.337 1.477 1.496 1.514 189.931 1.443
Text file - A current 1.526 0.001 0.271 1.489 1.512 1.535 82.202 1.453
Text file - A past 1.531 0.001 0.528 1.492 1.512 1.536 401.987 1.458
Text file - B current 1.499 0.001 0.274 1.479 1.488 1.500 70.508 1.444
Text file - B past 1.521 0.001 0.412 1.494 1.508 1.525 252.958 1.454
Binary TCP - A current 1.465 0.001 0.277 1.444 1.455 1.472 63.256 1.410
Binary TCP - A past 1.502 0.001 0.296 1.479 1.493 1.507 102.191 1.436
Binary TCP - B current 1.556 0.001 0.272 1.535 1.546 1.558 56.205 1.489
Binary TCP - B past 1.498 0.002 1.039 1.470 1.481 1.498 885.015 1.436

OpenTelemetry

OpenTelemetry (https://opentelemetry.io/) is a widely used monitoring tool, compatible with many programming languages. MooBench measures the overhead of OpenTelemetry using the following configurations:

  • No instrumentation is the baseline execution time of the benchmark program.
  • No Logging measures the execution time of the benchmark program with OpenTelemetry instrumentation, but without measuring or
    logging anything.
  • Logging measures the execution time of the benchmark program with OpenTelemetry instrumentation and writing every output to the standard output. Text file logging is slow and getting more irrelevant in the performance benchmarks. The diagram below, therefore, does not show the results.
  • Zipkin measures the execution time of the benchmark program with obtaining every method invocation and sending them to a local Zipkin instance.
  • Prometheus measures the execution time of the benchmark program with obtaining only aggregated execution times, which are stored by a local Prometheus instance.
setup run mean ci sd 1.quartile median 3.quartile max min
No instrumentation current 0.112 0.000 0.056 0.110 0.110 0.112 31.156 0.103
No instrumentation past 0.108 0.000 0.062 0.106 0.107 0.109 39.753 0.101
No logging current 7.059 0.024 12.484 6.807 6.863 6.930 6008.446 6.566
No logging past 7.064 0.022 11.172 6.826 6.884 6.952 3947.081 6.582
Zipkin current 11.998 0.031 15.932 10.955 11.116 13.116 4097.325 10.459
Zipkin past 13.711 0.035 17.642 13.301 13.444 13.577 5550.238 12.679
Prometheus current 10.320 0.023 11.545 10.047 10.131 10.226 3221.292 9.644
Prometheus past 10.485 0.025 12.774 10.182 10.270 10.369 4503.228 9.782

InspectIT

InspectIT (https://www.inspectit.rocks/) is a widely used monitoring tool, which is allows to trace the execution of Java programs. MooBench
measures the overhead of InspectIT using the following configurations:

  • No instrumentation is the baseline execution time of the benchmark program.
  • Deactivated processing measures the execution time of the benchmark program with inspectIT instrumentation, but without measuring or logging
    anything.
  • No Logging measures the execution time of the benchmark program with inspectIT instrumentation and just discarding the measurement results.
  • Zipkin measures the execution time of the benchmark program with obtaining every method invocation and sending them to a local Zipkin instance.
  • Prometheus measures the execution time of the benchmark program with obtaining only aggregated execution times, which are stored by a local Prometheus instance.
setup run mean ci sd 1.quartile median 3.quartile max min
No instrumentation current 0.110 0.000 0.090 0.108 0.109 0.110 61.930 0.103
No instrumentation past 0.111 0.000 0.134 0.108 0.109 0.111 112.479 0.101
Deactivated probe current 3.660 0.035 18.045 3.484 3.529 3.586 8503.840 3.310
Deactivated probe past 3.581 0.044 22.364 3.425 3.467 3.515 10666.948 3.257
No logging current 25.317 0.031 15.798 24.673 24.841 25.057 3708.559 23.874
No logging past 24.758 0.031 16.025 24.066 24.255 24.489 4458.920 23.162
Zipkin current 30.499 0.041 20.812 29.626 29.854 30.118 5768.192 23.912
Zipkin past 30.143 0.050 25.553 27.804 30.240 30.602 7707.530 24.656
Prometheus current 8.652 0.033 16.670 8.397 8.566 8.665 9591.477 8.030
Prometheus past 8.780 0.048 24.283 8.509 8.595 8.698 12122.360 8.260