Performance Benchmarks

On this page, we collect performance benchmarks of Kieker monitoring and other instrumentation frameworks, such as OpenTelemetry and inspectIT, for comparison utilizing MooBench.

Details of the comparison of OpenTelemetry, inspectIT and Kieker can be found in https://oceanrep.geomar.de/id/eprint/55367/1/SSP2021overhead.pdf

Kieker Observability Framework – Java

These are the performance benchmark results for Kieker 4 Java probes utilizing different writer back end.

  • No instrumentation serves as a baseline value indicating how much execution time can be contributed to the benchmark program itself.
  • Deactivated probe as above, but with woven in probes that do not produce any output and do not aggregate information.
  • No logging are activated probes, but the collected data is not send utilizing a null writer.
  • Text File utilizes the text file logging. This is usually slow, as data is written in text files and getting more irrelevant in the performance benchmarks. The diagram below, therefore, does not show the results.
  • Binary File utilizes the binary file logging without compression.
  • Binary TCP utilizes a TCP connection to send the monitoring data to a logging service.
setup run mean ci sd 1.quartile median 3.quartile max min
No instrumentation current 0.106 0.000 0.036 0.105 0.105 0.107 9.711 0.103
No instrumentation past 0.107 0.000 0.063 0.105 0.106 0.107 42.960 0.103
Deactivated probe current 0.719 0.022 11.030 0.681 0.694 0.709 10460.790 0.650
Deactivated probe past 0.707 0.007 3.619 0.675 0.691 0.709 2593.116 0.648
No logging current 2.656 0.010 5.017 2.603 2.623 2.648 2576.551 2.545
No logging past 2.719 0.010 5.248 2.666 2.689 2.719 2856.578 2.585
Binary file current 5.024 0.028 14.289 3.460 4.380 5.570 12971.910 2.977
Binary file past 5.198 0.023 11.789 3.518 4.459 6.640 8935.049 3.152
Binary TCP current 6.989 0.010 5.184 6.752 6.999 7.561 2525.177 3.008
Binary TCP past 6.794 0.019 9.449 6.675 6.906 7.322 6321.136 3.210

Kieker Observability Framework – Python

Kieker 4 Python is the reimplementation of the probe and monitoring side of Kieker addressing programs written in Python. Here we also want to know how well certain parts perform.

  • No instrumentation is the baseline execution time of the benchmark program.
  • Deactivated probe is the execution time with integrated probes, but these are deactivated.
  • No logging is with active probes, but with a null writer.
  • Text File is with an activated probe logging to a text file.
  • Binary TCP is with an activated probe logging to a remote logging server.

Each of these setups, except of no instrumentation, are tested utilizing two different probe introduction methods (here labeled A and B).

setup run mean ci sd 1.quartile median 3.quartile max min
No instrumentation current 1.620 0.000 0.240 1.512 1.664 1.680 152.749 1.449
No instrumentation past 1.578 0.000 0.182 1.502 1.593 1.611 71.946 1.438
Deactivated probe - A current 1.537 0.000 0.165 1.479 1.489 1.636 22.679 1.432
Deactivated probe - A past 1.521 0.000 0.172 1.460 1.503 1.586 33.187 1.402
Deactivated probe - B current 4.033 0.001 0.313 3.859 3.889 4.281 29.266 3.770
Deactivated probe - B past 4.090 0.001 0.333 3.854 4.068 4.275 46.484 3.756
No logging - A current 1.538 0.000 0.231 1.469 1.480 1.631 29.893 1.411
No logging - A past 1.531 0.000 0.181 1.478 1.487 1.604 41.297 1.431
No logging - B current 43.780 0.003 1.588 43.264 43.424 43.627 226.574 42.495
No logging - B past 44.576 0.003 1.718 43.816 44.099 44.508 235.460 42.834
Text file - A current 1.567 0.000 0.172 1.499 1.517 1.658 34.470 1.421
Text file - A past 1.540 0.000 0.180 1.482 1.497 1.597 28.689 1.425
Text file - B current 1310.357 1.517 773.893 1248.739 1288.702 1352.007 268250.000 1220.766
Text file - B past 1324.273 1.179 601.346 1266.403 1297.217 1370.665 222062.591 1238.238
Binary TCP - A current 1.612 0.000 0.159 1.595 1.603 1.610 20.835 1.555
Binary TCP - A past 1.599 0.000 0.173 1.580 1.593 1.614 49.127 1.443
Binary TCP - B current 497.173 0.056 28.354 488.843 491.642 496.727 910.627 236.340
Binary TCP - B past 480.314 0.298 152.207 471.373 474.219 479.351 126635.037 233.150

OpenTelemetry

OpenTelemetry (https://opentelemetry.io/) is a widely used monitoring tool, compatible with many programming languages. MooBench measures the overhead of OpenTelemetry using the following configurations:

  • No instrumentation is the baseline execution time of the benchmark program.
  • No Logging measures the execution time of the benchmark program with OpenTelemetry instrumentation, but without measuring or
    logging anything.
  • Logging measures the execution time of the benchmark program with OpenTelemetry instrumentation and writing every output to the standard output. Text file logging is slow and getting more irrelevant in the performance benchmarks. The diagram below, therefore, does not show the results.
  • Zipkin measures the execution time of the benchmark program with obtaining every method invocation and sending them to a local Zipkin instance.
  • Prometheus measures the execution time of the benchmark program with obtaining only aggregated execution times, which are stored by a local Prometheus instance.
setup run mean ci sd 1.quartile median 3.quartile max min
No instrumentation current 0.108 0.000 0.030 0.107 0.108 0.109 9.424 0.103
No instrumentation past 0.106 0.000 0.140 0.104 0.106 0.106 131.173 0.102
No logging current 5.829 0.023 11.605 5.660 5.711 5.795 6734.181 5.445
No logging past 6.941 0.044 22.561 6.749 6.797 6.886 11838.318 6.585
Zipkin current 10.223 0.063 32.285 9.342 9.398 9.476 25105.210 9.137
Zipkin past 10.300 0.025 13.010 9.756 9.845 9.958 4469.802 9.238

InspectIT

InspectIT (https://www.inspectit.rocks/) is a widely used monitoring tool, which is allows to trace the execution of Java programs. MooBench
measures the overhead of InspectIT using the following configurations:

  • No instrumentation is the baseline execution time of the benchmark program.
  • Deactivated processing measures the execution time of the benchmark program with inspectIT instrumentation, but without measuring or logging
    anything.
  • No Logging measures the execution time of the benchmark program with inspectIT instrumentation and just discarding the measurement results.
  • Zipkin measures the execution time of the benchmark program with obtaining every method invocation and sending them to a local Zipkin instance.
  • Prometheus measures the execution time of the benchmark program with obtaining only aggregated execution times, which are stored by a local Prometheus instance.
setup run mean ci sd 1.quartile median 3.quartile max min
No instrumentation current 0.108 0.000 0.031 0.107 0.107 0.108 7.603 0.103
No instrumentation past 0.108 0.000 0.060 0.107 0.108 0.108 42.635 0.104
Deactivated probe current 13.706 0.022 11.139 13.440 13.511 13.592 4318.438 12.669
Deactivated probe past 13.332 0.029 14.899 13.053 13.133 13.230 8961.494 12.689
No logging current 39.084 0.086 43.651 38.003 38.218 38.527 40712.820 37.177
No logging past 39.431 0.038 19.466 38.463 38.674 38.954 8042.703 35.319
Zipkin current 43.526 0.055 28.310 40.934 41.171 41.531 13114.530 36.275
Zipkin past 43.606 0.059 29.848 41.298 41.527 41.853 14687.909 36.513
Prometheus current 24.440 0.043 21.805 24.025 24.153 24.302 17995.760 23.543
Prometheus past 24.842 0.047 24.191 24.287 24.420 24.581 15211.870 23.710