Performance Benchmarks

On this page, we collect performance benchmarks of Kieker monitoring and other instrumentation frameworks, such as OpenTelemetry and inspectIT, for comparison utilizing MooBench.

Details of the comparison of OpenTelemetry, inspectIT and Kieker can be found in https://oceanrep.geomar.de/id/eprint/55367/1/SSP2021overhead.pdf

Kieker Observability Framework – Java

These are the performance benchmark results for Kieker 4 Java probes utilizing different writer back end.

  • No instrumentation serves as a baseline value indicating how much execution time can be contributed to the benchmark program itself.
  • Deactivated probe as above, but with woven in probes that do not produce any output and do not aggregate information.
  • No logging are activated probes, but the collected data is not send utilizing a null writer.
  • Text File utilizes the text file logging. This is usually slow, as data is written in text files and getting more irrelevant in the performance benchmarks. The diagram below, therefore, does not show the results.
  • Binary File utilizes the binary file logging without compression.
  • Binary TCP utilizes a TCP connection to send the monitoring data to a logging service.
setup run mean ci sd 1.quartile median 3.quartile max min
No instrumentation current 0.107 0.000 0.028 0.106 0.107 0.108 6.991 0.103
No instrumentation past 0.106 0.000 0.038 0.105 0.106 0.106 19.911 0.103
Deactivated probe current 0.722 0.007 3.741 0.688 0.705 0.724 2831.426 0.662
Deactivated probe past 0.716 0.009 4.569 0.683 0.699 0.717 3613.669 0.654
No logging current 2.636 0.009 4.424 2.585 2.608 2.637 2755.728 2.523
No logging past 2.687 0.010 4.900 2.633 2.659 2.688 2726.085 2.559
Binary file current 5.145 0.032 16.395 3.440 4.324 6.297 15315.000 2.966
Binary file past 5.148 0.026 13.277 3.490 4.435 6.269 9927.778 3.267
Binary TCP current 6.717 0.010 4.992 6.618 6.838 7.374 2569.973 3.241
Binary TCP past 6.996 0.012 5.962 6.731 6.962 7.491 2882.818 3.240

Kieker Observability Framework – Python

Kieker 4 Python is the reimplementation of the probe and monitoring side of Kieker addressing programs written in Python. Here we also want to know how well certain parts perform.

  • No instrumentation is the baseline execution time of the benchmark program.
  • Deactivated probe is the execution time with integrated probes, but these are deactivated.
  • No logging is with active probes, but with a null writer.
  • Text File is with an activated probe logging to a text file.
  • Binary TCP is with an activated probe logging to a remote logging server.

Each of these setups, except of no instrumentation, are tested utilizing two different probe introduction methods (here labeled A and B).

setup run mean ci sd 1.quartile median 3.quartile max min
No instrumentation current 1.452 0.000 0.150 1.433 1.442 1.451 24.218 1.375
No instrumentation past 1.583 0.000 0.181 1.550 1.582 1.592 61.654 1.502
Deactivated probe - A current 1.459 0.000 0.150 1.441 1.448 1.455 29.359 1.387
Deactivated probe - A past 1.579 0.000 0.169 1.497 1.579 1.643 26.967 1.430
Deactivated probe - B current 4.071 0.001 0.361 3.876 3.943 4.276 32.572 3.769
Deactivated probe - B past 3.951 0.001 0.333 3.778 3.812 4.191 100.372 3.707
No logging - A current 1.535 0.000 0.165 1.464 1.501 1.584 31.129 1.404
No logging - A past 1.574 0.000 0.184 1.491 1.570 1.652 27.493 1.431
No logging - B current 45.899 0.004 2.276 44.639 44.875 45.277 138.827 43.814
No logging - B past 44.947 0.004 1.858 44.167 44.438 44.796 231.323 43.050
Text file - A current 1.472 0.000 0.152 1.458 1.464 1.470 28.508 1.411
Text file - A past 1.507 0.000 0.156 1.477 1.487 1.519 31.241 1.425
Text file - B current 1333.168 0.992 506.189 1280.763 1298.012 1383.216 208697.100 1252.349
Text file - B past 1306.579 0.958 488.995 1253.269 1273.662 1354.810 157856.955 1226.279
Binary TCP - A current 1.487 0.000 0.159 1.439 1.450 1.508 20.212 1.370
Binary TCP - A past 1.619 0.000 0.179 1.617 1.631 1.643 61.305 1.450
Binary TCP - B current 474.788 0.068 34.556 464.686 468.344 474.914 1234.384 232.374
Binary TCP - B past 478.081 0.081 41.512 468.759 471.453 476.334 9016.700 235.971

OpenTelemetry

OpenTelemetry (https://opentelemetry.io/) is a widely used monitoring tool, compatible with many programming languages. MooBench measures the overhead of OpenTelemetry using the following configurations:

  • No instrumentation is the baseline execution time of the benchmark program.
  • No Logging measures the execution time of the benchmark program with OpenTelemetry instrumentation, but without measuring or
    logging anything.
  • Logging measures the execution time of the benchmark program with OpenTelemetry instrumentation and writing every output to the standard output. Text file logging is slow and getting more irrelevant in the performance benchmarks. The diagram below, therefore, does not show the results.
  • Zipkin measures the execution time of the benchmark program with obtaining every method invocation and sending them to a local Zipkin instance.
  • Prometheus measures the execution time of the benchmark program with obtaining only aggregated execution times, which are stored by a local Prometheus instance.
setup run mean ci sd 1.quartile median 3.quartile max min
No instrumentation current 0.108 0.000 0.034 0.107 0.108 0.109 18.176 0.103
No instrumentation past 0.110 0.000 0.041 0.108 0.109 0.110 21.412 0.104
No logging current 6.221 0.023 11.643 6.088 6.119 6.154 6831.619 5.955
No logging past 6.262 0.024 12.240 6.110 6.144 6.190 6927.165 5.941
Zipkin current 10.029 0.024 12.479 9.681 9.735 9.804 3409.771 8.591
Zipkin past 9.543 0.029 14.767 8.886 8.941 9.018 6207.305 8.461

InspectIT

InspectIT (https://www.inspectit.rocks/) is a widely used monitoring tool, which is allows to trace the execution of Java programs. MooBench
measures the overhead of InspectIT using the following configurations:

  • No instrumentation is the baseline execution time of the benchmark program.
  • Deactivated processing measures the execution time of the benchmark program with inspectIT instrumentation, but without measuring or logging
    anything.
  • No Logging measures the execution time of the benchmark program with inspectIT instrumentation and just discarding the measurement results.
  • Zipkin measures the execution time of the benchmark program with obtaining every method invocation and sending them to a local Zipkin instance.
  • Prometheus measures the execution time of the benchmark program with obtaining only aggregated execution times, which are stored by a local Prometheus instance.
setup run mean ci sd 1.quartile median 3.quartile max min
No instrumentation current 0.117 0.000 0.031 0.117 0.118 0.118 9.940 0.108
No instrumentation past 0.106 0.000 0.035 0.106 0.106 0.106 14.578 0.103
Deactivated probe current 14.853 0.022 11.075 14.560 14.654 14.767 3817.834 13.978
Deactivated probe past 14.202 0.022 11.054 13.907 13.991 14.095 3651.417 13.486
No logging current 41.202 0.035 17.818 40.038 40.558 40.975 6476.630 35.070
No logging past 41.445 0.049 24.797 40.384 40.657 41.075 16917.555 36.493
Zipkin current 44.167 0.083 42.397 41.892 42.153 42.495 21172.690 37.014
Zipkin past 43.501 0.070 35.709 41.026 41.244 41.571 21207.247 36.407
Prometheus current 25.905 0.025 12.759 25.365 25.563 25.783 3028.702 24.333
Prometheus past 25.517 0.028 14.513 25.044 25.171 25.319 6056.179 24.476