Performance Benchmarks

On this page, we collect performance benchmarks of Kieker monitoring and other instrumentation frameworks, such as OpenTelemetry and inspectIT, for comparison utilizing MooBench.

Details of the comparison of OpenTelemetry, inspectIT and Kieker can be found in https://oceanrep.geomar.de/id/eprint/55367/1/SSP2021overhead.pdf


Kieker Observability Framework – Java

These are the performance benchmark results for Kieker 4 Java probes utilizing different writer back end.

  • No instrumentation serves as a baseline value indicating how much execution time can be contributed to the benchmark program itself.
  • Deactivated probe as above, but with woven in probes that do not produce any output and do not aggregate information.
  • No logging are activated probes, but the collected data is not send utilizing a null writer.
  • Text File utilizes the text file logging. This is usually slow, as data is written in text files and getting more irrelevant in the performance benchmarks. The diagram below, therefore, does not show the results.
  • Binary File utilizes the binary file logging without compression.
  • Binary TCP utilizes a TCP connection to send the monitoring data to a logging service.
setup run mean ci sd 1.quartile median 3.quartile max min
No instrumentation current 0.105 0.000 0.166 0.105 0.105 0.105 163.590 0.102
No instrumentation past 0.105 0.000 0.039 0.104 0.105 0.106 21.430 0.103
Deactivated probe current 0.713 0.007 3.532 0.682 0.695 0.715 2597.790 0.647
Deactivated probe past 0.712 0.011 5.466 0.680 0.693 0.711 4594.639 0.645
No logging current 2.753 0.009 4.389 2.702 2.725 2.753 2689.260 2.622
No logging past 2.670 0.010 5.202 2.616 2.639 2.667 2762.614 2.549
Binary file current 5.198 0.024 12.224 3.553 4.528 6.701 11000.430 3.036
Binary file past 5.106 0.019 9.844 3.487 4.427 5.801 7311.650 3.217
Binary TCP current 6.584 0.010 4.960 6.415 6.811 7.206 2532.722 2.995
Binary TCP past 6.942 0.011 5.770 6.674 6.963 7.594 3042.470 3.158

Kieker Observability Framework – Python

Kieker 4 Python is the reimplementation of the probe and monitoring side of Kieker addressing programs written in Python. Here we also want to know how well certain parts perform.

  • No instrumentation is the baseline execution time of the benchmark program.
  • Deactivated probe is the execution time with integrated probes, but these are deactivated.
  • No logging is with active probes, but with a null writer.
  • Text File is with an activated probe logging to a text file.
  • Binary TCP is with an activated probe logging to a remote logging server.

Each of these setups, except of no instrumentation, are tested utilizing two different probe introduction methods (here labeled A and B).

setup run mean ci sd 1.quartile median 3.quartile max min
No instrumentation current 1.558 0.000 0.159 1.463 1.605 1.623 26.449 1.415
No instrumentation past 1.541 0.000 0.173 1.490 1.546 1.558 61.779 1.437
Deactivated probe - A current 1.471 0.000 0.149 1.455 1.464 1.473 26.728 1.409
Deactivated probe - A past 1.565 0.000 0.173 1.511 1.529 1.645 33.017 1.438
Deactivated probe - B current 4.003 0.001 0.310 3.832 3.857 4.258 32.699 3.742
Deactivated probe - B past 4.004 0.001 0.319 3.825 3.872 4.245 46.810 3.745
No logging - A current 1.509 0.000 0.155 1.470 1.481 1.494 24.553 1.414
No logging - A past 1.551 0.000 0.190 1.486 1.495 1.641 32.217 1.434
No logging - B current 45.962 0.005 2.442 44.622 44.902 45.503 761.567 43.528
No logging - B past 44.609 0.003 1.762 43.906 44.119 44.397 229.830 42.991
Text file - A current 1.489 0.000 0.210 1.438 1.449 1.567 88.035 1.379
Text file - A past 1.517 0.000 0.167 1.473 1.485 1.538 40.152 1.419
Text file - B current 1333.186 1.130 576.366 1274.658 1308.859 1378.831 200411.600 1247.484
Text file - B past 1309.262 1.046 533.655 1254.210 1279.046 1356.333 192068.605 1227.112
Binary TCP - A current 1.539 0.000 0.185 1.510 1.519 1.529 25.716 1.421
Binary TCP - A past 1.616 0.000 0.161 1.609 1.623 1.633 26.868 1.479
Binary TCP - B current 460.782 0.066 33.496 451.416 453.893 458.814 915.261 233.276
Binary TCP - B past 476.830 0.091 46.575 468.001 470.754 475.616 16802.655 235.691

OpenTelemetry

OpenTelemetry (https://opentelemetry.io/) is a widely used monitoring tool, compatible with many programming languages. MooBench measures the overhead of OpenTelemetry using the following configurations:

  • No instrumentation is the baseline execution time of the benchmark program.
  • No Logging measures the execution time of the benchmark program with OpenTelemetry instrumentation, but without measuring or
    logging anything.
  • Logging measures the execution time of the benchmark program with OpenTelemetry instrumentation and writing every output to the standard output. Text file logging is slow and getting more irrelevant in the performance benchmarks. The diagram below, therefore, does not show the results.
  • Zipkin measures the execution time of the benchmark program with obtaining every method invocation and sending them to a local Zipkin instance.
  • Prometheus measures the execution time of the benchmark program with obtaining only aggregated execution times, which are stored by a local Prometheus instance.
setup run mean ci sd 1.quartile median 3.quartile max min
No instrumentation current 0.109 0.000 0.037 0.108 0.108 0.109 15.740 0.103
No instrumentation past 0.107 0.000 0.043 0.106 0.107 0.108 24.040 0.103
No logging current 6.959 0.024 12.164 6.814 6.869 6.923 6658.850 6.474
No logging past 6.346 0.026 13.199 6.200 6.237 6.291 7526.263 6.016
Zipkin current 10.227 0.023 11.748 9.846 9.902 9.967 3465.966 8.726
Zipkin past 9.902 0.033 17.033 9.188 9.245 9.319 9000.528 8.548

InspectIT

InspectIT (https://www.inspectit.rocks/) is a widely used monitoring tool, which is allows to trace the execution of Java programs. MooBench
measures the overhead of InspectIT using the following configurations:

  • No instrumentation is the baseline execution time of the benchmark program.
  • Deactivated processing measures the execution time of the benchmark program with inspectIT instrumentation, but without measuring or logging
    anything.
  • No Logging measures the execution time of the benchmark program with inspectIT instrumentation and just discarding the measurement results.
  • Zipkin measures the execution time of the benchmark program with obtaining every method invocation and sending them to a local Zipkin instance.
  • Prometheus measures the execution time of the benchmark program with obtaining only aggregated execution times, which are stored by a local Prometheus instance.
setup run mean ci sd 1.quartile median 3.quartile max min
No instrumentation current 0.109 0.000 0.038 0.107 0.108 0.109 18.194 0.103
No instrumentation past 0.107 0.000 0.032 0.106 0.107 0.107 11.610 0.103
Deactivated probe current 13.558 0.023 11.724 13.311 13.430 13.537 4687.333 12.526
Deactivated probe past 14.122 0.022 11.149 13.801 13.887 13.993 4171.516 13.314
No logging current 38.581 0.170 86.893 37.510 37.745 38.094 85299.720 36.581
No logging past 41.877 0.046 23.440 40.909 41.150 41.480 13185.056 37.720
Zipkin current 44.834 0.046 23.474 42.882 43.154 43.523 4349.490 37.862
Zipkin past 41.741 0.084 42.667 39.326 39.565 39.922 28009.389 34.772
Prometheus current 24.515 0.045 22.989 23.990 24.107 24.251 19158.000 23.412
Prometheus past 24.784 0.032 16.259 24.281 24.396 24.535 8402.395 23.772