Performance Benchmarks

On this page, we collect performance benchmarks of Kieker monitoring and other instrumentation frameworks, such as OpenTelemetry and inspectIT, for comparison utilizing MooBench.

Details of the comparison of OpenTelemetry, inspectIT and Kieker can be found in https://oceanrep.geomar.de/id/eprint/55367/1/SSP2021overhead.pdf


Kieker Observability Framework – Java

These are the performance benchmark results for Kieker 4 Java probes utilizing different writer back end.

  • No instrumentation serves as a baseline value indicating how much execution time can be contributed to the benchmark program itself.
  • Deactivated probe as above, but with woven in probes that do not produce any output and do not aggregate information.
  • No logging are activated probes, but the collected data is not send utilizing a null writer.
  • Text File utilizes the text file logging. This is usually slow, as data is written in text files and getting more irrelevant in the performance benchmarks. The diagram below, therefore, does not show the results.
  • Binary File utilizes the binary file logging without compression.
  • Binary TCP utilizes a TCP connection to send the monitoring data to a logging service.
setup run mean ci sd 1.quartile median 3.quartile max min
No instrumentation current 0.107 0.000 0.112 0.105 0.105 0.107 107.757 0.102
No instrumentation past 0.107 0.000 0.037 0.107 0.107 0.107 18.154 0.102
Deactivated probe current 0.700 0.008 3.939 0.667 0.684 0.701 2785.998 0.638
Deactivated probe past 0.709 0.007 3.806 0.677 0.693 0.710 2767.151 0.646
No logging current 2.785 0.011 5.466 2.735 2.754 2.780 2917.991 2.663
No logging past 2.778 0.018 8.947 2.716 2.740 2.769 6838.754 2.643
Binary file current 4.912 0.041 20.996 3.465 4.425 4.753 14549.840 3.276
Binary file past 4.933 0.038 19.314 3.451 4.336 5.323 18138.952 3.029
Binary TCP current 7.475 0.012 6.207 6.794 7.051 7.805 2604.604 3.026
Binary TCP past 7.304 0.015 7.469 6.688 7.055 7.810 4599.124 3.062

Kieker Observability Framework – Python

Kieker 4 Python is the reimplementation of the probe and monitoring side of Kieker addressing programs written in Python. Here we also want to know how well certain parts perform.

  • No instrumentation is the baseline execution time of the benchmark program.
  • Deactivated probe is the execution time with integrated probes, but these are deactivated.
  • No logging is with active probes, but with a null writer.
  • Text File is with an activated probe logging to a text file.
  • Binary TCP is with an activated probe logging to a remote logging server.

Each of these setups, except of no instrumentation, are tested utilizing two different probe introduction methods (here labeled A and B).

setup run mean ci sd 1.quartile median 3.quartile max min
No instrumentation current 1.553 0.000 0.148 1.543 1.551 1.557 32.374 1.466
No instrumentation past 1.638 0.000 0.170 1.580 1.625 1.712 25.647 1.472
Deactivated probe - A current 1.544 0.000 0.241 1.530 1.538 1.546 185.860 1.470
Deactivated probe - A past 1.684 0.000 0.186 1.646 1.665 1.731 58.116 1.496
Deactivated probe - B current 3.837 0.001 0.300 3.806 3.824 3.844 179.221 3.704
Deactivated probe - B past 3.934 0.001 0.293 3.816 3.862 4.046 57.429 3.702
No logging - A current 1.551 0.000 0.153 1.539 1.549 1.557 24.037 1.476
No logging - A past 2.960 0.000 0.228 2.849 2.943 3.022 38.925 2.742
No logging - B current 45.010 0.004 1.827 44.194 44.467 44.838 231.526 43.199
No logging - B past 45.881 0.004 2.134 44.914 45.150 45.552 479.667 43.885
Text file - A current 1.554 0.000 0.164 1.509 1.518 1.534 24.582 1.438
Text file - A past 12.789 0.010 5.222 12.255 12.523 13.231 1438.907 11.936
Text file - B current 1324.650 1.083 552.708 1262.506 1309.014 1367.245 208107.800 1232.760
Text file - B past 1314.638 1.227 625.843 1258.432 1287.586 1360.628 227982.995 1229.429
Binary TCP - A current 1.636 0.000 0.177 1.525 1.683 1.692 25.941 1.473
Binary TCP - A past 6.802 0.001 0.529 6.643 6.749 6.831 118.737 4.497
Binary TCP - B current 481.077 0.061 30.971 472.512 475.316 480.049 1359.387 232.355
Binary TCP - B past 482.065 0.057 29.322 473.374 476.245 481.465 1364.527 238.453

OpenTelemetry

OpenTelemetry (https://opentelemetry.io/) is a widely used monitoring tool, compatible with many programming languages. MooBench measures the overhead of OpenTelemetry using the following configurations:

  • No instrumentation is the baseline execution time of the benchmark program.
  • No Logging measures the execution time of the benchmark program with OpenTelemetry instrumentation, but without measuring or
    logging anything.
  • Logging measures the execution time of the benchmark program with OpenTelemetry instrumentation and writing every output to the standard output. Text file logging is slow and getting more irrelevant in the performance benchmarks. The diagram below, therefore, does not show the results.
  • Zipkin measures the execution time of the benchmark program with obtaining every method invocation and sending them to a local Zipkin instance.
  • Prometheus measures the execution time of the benchmark program with obtaining only aggregated execution times, which are stored by a local Prometheus instance.
setup run mean ci sd 1.quartile median 3.quartile max min
No instrumentation current 0.104 0.000 0.034 0.103 0.104 0.105 17.846 0.103
No instrumentation past 0.110 0.000 0.051 0.109 0.111 0.111 34.153 0.106
No logging current 6.770 0.019 9.659 6.625 6.664 6.718 5559.332 6.415
No logging past 6.722 0.041 20.952 6.551 6.589 6.645 14368.074 6.395
Zipkin current 9.218 0.026 13.234 8.715 8.761 8.819 3491.802 8.506
Zipkin past 10.696 0.086 43.951 10.001 10.171 10.443 37711.450 9.564

InspectIT

InspectIT (https://www.inspectit.rocks/) is a widely used monitoring tool, which is allows to trace the execution of Java programs. MooBench
measures the overhead of InspectIT using the following configurations:

  • No instrumentation is the baseline execution time of the benchmark program.
  • Deactivated processing measures the execution time of the benchmark program with inspectIT instrumentation, but without measuring or logging
    anything.
  • No Logging measures the execution time of the benchmark program with inspectIT instrumentation and just discarding the measurement results.
  • Zipkin measures the execution time of the benchmark program with obtaining every method invocation and sending them to a local Zipkin instance.
  • Prometheus measures the execution time of the benchmark program with obtaining only aggregated execution times, which are stored by a local Prometheus instance.
setup run mean ci sd 1.quartile median 3.quartile max min
No instrumentation current 0.104 0.000 0.030 0.103 0.104 0.105 8.014 0.103
No instrumentation past 0.106 0.000 0.028 0.105 0.106 0.106 11.188 0.102
Deactivated probe current 13.329 0.021 10.847 12.997 13.076 13.171 4258.459 12.454
Deactivated probe past 13.799 0.032 16.090 13.467 13.570 13.694 9318.532 12.893
No logging current 39.474 0.131 66.986 38.249 38.561 38.960 65053.980 37.220
No logging past 42.475 0.055 27.950 41.342 41.557 41.865 20336.162 40.255
Zipkin current 42.120 0.049 24.934 39.724 39.935 40.233 4283.695 35.068
Zipkin past 43.762 0.062 31.663 41.123 41.402 41.798 14576.749 36.255
Prometheus current 26.006 0.026 13.351 25.477 25.685 25.864 2933.227 24.453
Prometheus past 23.779 0.030 15.330 23.352 23.469 23.601 5817.934 22.716