Performance Benchmarks

On this page, we collect performance benchmarks of Kieker monitoring and other instrumentation frameworks, such as OpenTelemetry and inspectIT, for comparison utilizing MooBench.

Details of the comparison of OpenTelemetry, inspectIT and Kieker can be
found in https://oceanrep.geomar.de/id/eprint/55367/1/SSP2021overhead.pdf

Kieker 4 Java – Monitoring

These are the performance benchmark results for Kieker 4 Java probes utilizing different writer back end.

  • No instrumentation serves as a baseline value indicating how much execution time can be contributed to the benchmark program itself.
  • Deactivated probe as above, but with woven in probes that do not produce any output and do not aggregate information.
  • No logging are activated probes, but the collected data is not send utilizing a null writer.
  • Text File utilizes the text file logging. This is usually slow, as data is written in text files.
  • Binary File utilizes the binary file logging without compression.
  • Binary TCP utilizes a TCP connection to send the monitoring data to a logging service.
setup run mean ci sd 1.quartile median 3.quartile min max
No instrumentation current 0.119 0.000 0.037 0.117 0.120 0.122 15.646 0.110
No instrumentation past 0.105 0.000 0.035 0.104 0.105 0.106 11.468 0.101
Deactivated probe current 0.596 0.007 3.462 0.566 0.580 0.593 2555.127 0.545
Deactivated probe past 0.640 0.031 16.059 0.596 0.607 0.622 15602.097 0.569
No logging current 3.090 0.009 4.724 3.028 3.052 3.078 2518.366 2.959
No logging past 2.897 0.008 4.096 2.847 2.864 2.883 2507.475 2.789
Binary file current 6.898 1.515 773.033 3.662 4.707 5.440 576828.400 3.164
Binary file past 7.245 1.371 699.666 3.546 4.239 4.684 588512.651 3.143
Binary TCP current 6.572 0.009 4.845 6.497 6.788 7.049 2291.365 3.136
Binary TCP past 6.754 0.024 12.396 6.707 6.989 7.249 10435.819 3.089
Text file past 205.104 2.994 1527.378 177.831 191.699 195.023 606650.300 3.319

Kieker 4 Python – Monitoring

Kieker 4 Python is the reimplementation of the probe and monitoring side of Kieker addressing programs written in Python. Here we also want to know how well certain parts perform.

  • No instrumentation is the baseline execution time of the benchmark program.
  • Deactivated probe is the execution time with integrated probes, but these are deactivated.
  • No logging is with active probes, but with a null writer.
  • Text File is with an activated probe logging to a text file.
  • Binary TCP is with an activated probe logging to a remote logging server.

Each of these setups, except of no instrumentation, are tested utilizing two different probe introduction methods (here labeled A and B).

setup run mean ci sd 1.quartile median 3.quartile min max
No instrumentation current 4.291 0.001 0.388 4.098 4.125 4.546 38.927 4.012
No instrumentation past 4.337 0.001 0.405 4.169 4.196 4.606 55.547 4.094
Deactivated probe - A current 9.474 0.001 0.711 8.974 9.083 9.969 188.559 8.836
Deactivated probe - A past 9.687 0.490 250.251 9.059 9.098 10.006 249887.328 8.939
Deactivated probe - B current 9.498 0.001 0.711 9.055 9.104 10.058 46.923 8.932
Deactivated probe - B past 9.507 0.001 0.687 9.121 9.157 10.013 112.978 9.002
No logging - A current 354.405 0.017 8.648 351.410 352.057 352.940 3420.691 348.380
No logging - A past 357.374 0.021 10.492 353.109 353.955 355.142 1232.267 348.770
No logging - B current 119.020 0.006 3.159 118.020 118.243 118.515 309.177 116.951
No logging - B past 118.536 0.008 4.020 117.483 117.713 117.996 1552.564 116.308
Text file - A current 2242.988 1.620 826.744 2131.541 2292.324 2316.316 142099.100 2094.907
Text file - A past 2255.638 1.705 869.877 2146.780 2196.973 2333.991 211284.558 2114.266
Text file - B current 1911.626 1.260 643.102 1822.744 1862.927 1981.224 215506.100 1792.523
Text file - B past 1922.366 1.389 708.682 1831.216 1876.795 1989.504 205940.853 1800.145
Binary TCP - A current 1169.070 0.078 39.896 1155.487 1159.236 1178.218 3914.534 820.815
Binary TCP - A past 1152.892 0.105 53.805 1135.752 1139.073 1159.395 4326.078 813.847
Binary TCP - B current 835.833 0.068 34.530 825.334 828.324 841.498 3901.386 527.293
Binary TCP - B past 838.071 0.088 44.979 825.089 827.773 843.400 4115.523 533.532

OpenTelemetry

OpenTelemetry (https://opentelemetry.io/) is a widely used monitoring tool, compatible with many programming languages. MooBench measures the overhead of OpenTelemetry using the following configurations:

  • No instrumentation is the baseline execution time of the benchmark program.
  • No Logging measures the execution time of the benchmark program with OpenTelemetry instrumentation, but without measuring or
    logging anything.
  • Logging measures the execution time of the benchmark program with OpenTelemetry instrumentation and writing every output to the standard output.
  • Zipkin measures the execution time of the benchmark program with obtaining every method invocation and sending them to a local Zipkin instance.
  • Prometheus measures the execution time of the benchmark program with obtaining only aggregated execution times, which are stored by a local Prometheus instance.
setup run mean ci sd 1.quartile median 3.quartile min max
No instrumentation current 0.107 0.000 0.030 0.106 0.107 0.107 8.655 0.105
No instrumentation past 0.106 0.000 0.036 0.105 0.106 0.107 12.803 0.101
No logging current 6.736 0.016 7.961 6.576 6.620 6.672 2486.728 6.403
No logging past 6.563 0.021 10.816 6.392 6.442 6.498 5269.657 6.196
Zipkin current 10.237 0.024 12.488 9.452 9.517 9.592 5296.669 8.309
Zipkin past 10.315 0.027 13.680 9.545 9.696 9.872 6265.108 8.283
Logging past 171.173 4.412 2251.166 157.627 158.919 161.858 659016.000 140.407
Prometheus past 8.911 0.063 32.345 8.660 8.715 8.789 28723.475 8.441

InspectIT

InspectIT (https://www.inspectit.rocks/) is a widely used monitoring tool, which is allows to trace the execution of Java programs. MooBench
measures the overhead of InspectIT using the following configurations:

  • No instrumentation is the baseline execution time of the benchmark program.
  • Deactivated processing measures the execution time of the benchmark program with inspectIT instrumentation, but without measuring or logging
    anything.
  • No Logging measures the execution time of the benchmark program with inspectIT instrumentation and just discarding the measurement results.
  • Zipkin measures the execution time of the benchmark program with obtaining every method invocation and sending them to a local Zipkin instance.
  • Prometheus measures the execution time of the benchmark program with obtaining only aggregated execution times, which are stored by a local Prometheus instance.
setup run mean ci sd 1.quartile median 3.quartile min max
No instrumentation current 37.902 0.014 7.379 35.233 38.575 38.942 294.504 33.964
No instrumentation past 37.034 0.014 7.398 34.402 37.644 38.082 387.112 33.083
Deactivated probe current 46.207 0.063 32.339 45.203 45.440 45.718 26948.750 40.170
Deactivated probe past 45.511 0.046 23.424 44.659 44.928 45.216 16769.268 39.607
No logging current 74.593 0.046 23.248 72.501 72.826 73.277 3526.965 70.666
No logging past 73.640 0.056 28.721 71.578 71.920 72.376 9483.956 65.480
Zipkin current 75.911 0.224 114.252 66.502 71.880 72.952 80807.010 64.210
Zipkin past 74.906 0.806 411.248 65.986 69.817 71.474 336480.280 63.076
Prometheus current 53.856 0.023 11.795 52.841 53.122 53.449 5384.540 46.878
Prometheus past 53.068 0.040 20.593 51.950 52.222 52.545 10802.277 47.251