Performance Benchmarks

On this page, we collect performance benchmarks of Kieker monitoring and other instrumentation frameworks, such as OpenTelemetry and inspectIT, for comparison utilizing MooBench.

Details of the comparison of OpenTelemetry, inspectIT and Kieker can be found in https://oceanrep.geomar.de/id/eprint/55367/1/SSP2021overhead.pdf

Kieker Observability Framework – Java

These are the performance benchmark results for Kieker 4 Java probes utilizing different writer back end.

  • No instrumentation serves as a baseline value indicating how much execution time can be contributed to the benchmark program itself.
  • Deactivated probe as above, but with woven in probes that do not produce any output and do not aggregate information.
  • No logging are activated probes, but the collected data is not send utilizing a null writer.
  • Text File utilizes the text file logging. This is usually slow, as data is written in text files and getting more irrelevant in the performance benchmarks. The diagram below, therefore, does not show the results.
  • Binary File utilizes the binary file logging without compression.
  • Binary TCP utilizes a TCP connection to send the monitoring data to a logging service.
setup run mean ci sd 1.quartile median 3.quartile max min
No instrumentation current 0.073 0.000 0.057 0.072 0.072 0.073 51.615 0.068
No instrumentation past 0.085 0.000 0.041 0.084 0.084 0.085 27.243 0.079
Deactivated probe current 0.096 0.000 0.119 0.095 0.096 0.097 108.277 0.085
Deactivated probe past 0.110 0.000 0.077 0.108 0.109 0.111 51.860 0.099
No collection current 1.163 0.019 9.819 1.112 1.120 1.129 9131.629 1.078
No collection past 1.290 0.016 7.922 1.244 1.249 1.257 6346.290 1.219
Binary file current 2.605 0.005 2.581 1.740 2.806 3.192 2250.930 1.390
Binary file past 4.142 0.012 6.107 2.496 4.178 4.975 4338.373 1.557
Binary TCP current 5.628 0.020 10.034 4.195 4.662 7.343 6485.906 1.311
Binary TCP past 5.004 0.017 8.727 3.788 4.799 6.195 6914.907 1.580

Kieker Observability Framework – Python

Kieker 4 Python is the reimplementation of the probe and monitoring side of Kieker addressing programs written in Python. Here we also want to know how well certain parts perform.

  • No instrumentation is the baseline execution time of the benchmark program.
  • Deactivated probe is the execution time with integrated probes, but these are deactivated.
  • No logging is with active probes, but with a null writer.
  • Text File is with an activated probe logging to a text file.
  • Binary TCP is with an activated probe logging to a remote logging server.

Each of these setups, except of no instrumentation, are tested utilizing two different probe introduction methods (here labeled A and B).

setup run mean ci sd 1.quartile median 3.quartile max min
No instrumentation current 0.892 0.003 1.435 0.879 0.886 0.897 1419.176 0.850
No instrumentation past 1.030 0.000 0.196 1.015 1.024 1.036 114.555 0.980
Deactivated probe - A current 2.171 0.000 0.241 2.154 2.165 2.177 164.761 2.095
Deactivated probe - A past 1.674 0.001 0.314 1.656 1.667 1.681 236.443 1.607
Deactivated probe - B current 2.210 0.003 1.436 2.187 2.202 2.220 1415.293 2.127
Deactivated probe - B past 1.686 0.002 0.848 1.665 1.677 1.693 776.815 1.612
No logging - A current 62.076 0.007 3.814 61.563 61.930 62.310 2067.866 60.268
No logging - A past 32.359 0.004 2.020 31.984 32.152 32.359 1081.910 31.235
No logging - B current 17.583 0.004 2.231 17.483 17.541 17.604 2018.586 17.171
No logging - B past 9.483 0.002 0.793 9.423 9.450 9.483 460.534 9.282
Text file - A current 450.475 0.150 76.542 446.645 448.960 451.865 43293.000 437.017
Text file - A past 202.934 0.025 12.869 200.361 201.923 204.232 11451.445 194.722
Text file - B current 316.708 0.110 56.241 311.886 314.190 317.512 33085.490 304.562
Text file - B past 158.648 0.045 23.059 156.541 157.647 159.283 19588.743 152.977
Binary TCP - A current 209.395 0.017 8.550 203.169 206.899 214.686 441.472 137.519
Binary TCP - A past 110.970 0.021 10.706 109.173 109.885 111.012 9395.506 106.083
Binary TCP - B current 148.613 0.024 12.195 139.223 146.594 157.855 262.523 82.883
Binary TCP - B past 60.818 0.021 10.514 58.759 60.709 61.624 9645.449 56.650

OpenTelemetry

OpenTelemetry (https://opentelemetry.io/) is a widely used monitoring tool, compatible with many programming languages. MooBench measures the overhead of OpenTelemetry using the following configurations:

  • No instrumentation is the baseline execution time of the benchmark program.
  • No Logging measures the execution time of the benchmark program with OpenTelemetry instrumentation, but without measuring or
    logging anything.
  • Logging measures the execution time of the benchmark program with OpenTelemetry instrumentation and writing every output to the standard output. Text file logging is slow and getting more irrelevant in the performance benchmarks. The diagram below, therefore, does not show the results.
  • Zipkin measures the execution time of the benchmark program with obtaining every method invocation and sending them to a local Zipkin instance.
  • Prometheus measures the execution time of the benchmark program with obtaining only aggregated execution times, which are stored by a local Prometheus instance.
setup run mean ci sd 1.quartile median 3.quartile max min
No instrumentation current 0.073 0.000 0.131 0.072 0.072 0.073 120.353 0.068
No instrumentation past 0.085 0.000 0.068 0.083 0.084 0.085 57.615 0.079
No logging current 4.043 0.019 9.458 3.903 3.927 3.989 4217.975 3.831
No logging past 4.826 0.017 8.898 4.664 4.705 4.785 3124.562 4.549
Zipkin current 6.674 0.040 20.498 6.379 6.421 6.581 14075.360 6.215
Zipkin past 8.504 0.030 15.544 7.957 8.151 8.421 5358.435 7.454
Prometheus current 6.049 0.029 14.967 5.885 5.923 5.990 6916.420 5.730
Prometheus past 7.135 0.028 14.258 6.907 6.960 7.058 7300.894 6.684

InspectIT

InspectIT (https://www.inspectit.rocks/) is a widely used monitoring tool, which is allows to trace the execution of Java programs. MooBench
measures the overhead of InspectIT using the following configurations:

  • No instrumentation is the baseline execution time of the benchmark program.
  • Deactivated processing measures the execution time of the benchmark program with inspectIT instrumentation, but without measuring or logging
    anything.
  • No Logging measures the execution time of the benchmark program with inspectIT instrumentation and just discarding the measurement results.
  • Zipkin measures the execution time of the benchmark program with obtaining every method invocation and sending them to a local Zipkin instance.
  • Prometheus measures the execution time of the benchmark program with obtaining only aggregated execution times, which are stored by a local Prometheus instance.
setup run mean ci sd 1.quartile median 3.quartile max min
No instrumentation current 0.072 0.000 0.021 0.071 0.072 0.073 15.634 0.068
No instrumentation past 0.085 0.000 0.038 0.084 0.084 0.085 22.028 0.080
Deactivated probe current 2.644 0.051 25.849 2.498 2.521 2.577 13207.500 2.412
Deactivated probe past 2.763 0.047 24.118 2.581 2.604 2.648 11892.569 2.507
No logging current 12.391 0.029 14.861 12.071 12.122 12.253 3667.554 11.215
No logging past 9.260 0.014 7.361 8.978 9.047 9.159 2110.683 8.694
Zipkin current 11.984 0.039 19.936 11.203 11.594 11.973 5299.136 11.036
Zipkin past 10.307 0.021 10.934 9.919 10.031 10.257 3410.516 9.547
Prometheus current 5.404 0.040 20.264 5.152 5.236 5.343 11244.180 4.914
Prometheus past 6.302 0.042 21.571 6.075 6.167 6.280 12136.237 5.908