Performance Benchmarks

On this page, we collect performance benchmarks of Kieker monitoring and other instrumentation frameworks, such as OpenTelemetry and inspectIT, for comparison utilizing MooBench.

Details of the comparison of OpenTelemetry, inspectIT and Kieker can be
found in https://oceanrep.geomar.de/id/eprint/55367/1/SSP2021overhead.pdf

Kieker 4 Java – Monitoring

These are the performance benchmark results for Kieker 4 Java probes utilizing different writer back end.

  • No instrumentation serves as a baseline value indicating how much execution time can be contributed to the benchmark program itself.
  • Deactivated probe as above, but with woven in probes that do not produce any output and do not aggregate information.
  • No logging are activated probes, but the collected data is not send utilizing a null writer.
  • Text File utilizes the text file logging. This is usually slow, as data is written in text files.
  • Binary File utilizes the binary file logging without compression.
  • Binary TCP utilizes a TCP connection to send the monitoring data to a logging service.
setup run mean ci sd 1.quartile median 3.quartile min max
No instrumentation current 0.112 0.000 0.039 0.111 0.112 0.112 9.271 0.105
No instrumentation past 0.112 0.000 0.046 0.109 0.109 0.110 9.545 0.104
Deactivated probe current 1.260 0.007 1.141 0.573 0.595 2.557 17.528 0.543
Deactivated probe past 1.259 0.007 1.125 0.576 0.602 2.537 19.815 0.544
No logging current 19.274 0.016 2.543 17.561 19.057 20.661 45.809 3.786
No logging past 20.381 0.017 2.703 18.628 20.293 21.991 93.916 4.890
Text file current 239.509 12.406 2001.634 217.280 231.452 237.370 633047.600 3.646
Text file past 240.550 15.758 2542.472 218.231 223.779 228.156 755687.935 3.916
Binary file current 8.935 0.785 126.630 5.772 7.434 11.651 40034.250 3.033
Binary file past 9.364 0.125 20.189 5.935 8.290 12.812 6155.748 3.386
Binary TCP current 14.691 0.028 4.464 13.060 14.860 17.367 121.002 3.397
Binary TCP past 12.415 0.023 3.765 10.438 13.066 14.258 104.358 3.496
Chart by Visualizer

Kieker 4 Python – Monitoring

Kieker 4 Python is the reimplementation of the probe and monitoring side of Kieker addressing programs written in Python. Here we also want to know how well certain parts perform.

  • No instrumentation is the baseline execution time of the benchmark program.
  • Deactivated probe is the execution time with integrated probes, but these are deactivated.
  • No logging is with active probes, but with a null writer.
  • Text File is with an activated probe logging to a text file.
  • Binary TCP is with an activated probe logging to a remote logging server.

Each of these setups, except of no instrumentation, are tested utilizing two different probe introduction methods (here labeled A and B).

setup run mean ci sd 1.quartile median 3.quartile min max
No instrumentation current 4.567 0.002 0.276 4.529 4.548 4.567 22.489 4.468
No instrumentation past 4.648 0.002 0.348 4.601 4.617 4.635 24.179 4.506
Deactivated probe - A current 10.257 0.004 0.573 10.186 10.211 10.239 98.391 10.066
Deactivated probe - A past 9.963 0.004 0.720 9.446 10.081 10.390 44.492 9.009
Deactivated probe - B current 10.541 0.004 0.570 10.468 10.496 10.527 52.111 10.337
Deactivated probe - B past 9.654 0.006 1.028 9.145 9.548 10.083 188.510 9.008
No logging - A current 358.996 0.057 9.201 355.536 356.191 357.082 552.396 352.040
No logging - A past 449.306 0.435 70.172 359.903 479.229 500.913 722.823 356.014
No logging - B current 122.831 0.031 5.079 120.387 120.645 121.078 308.498 119.331
No logging - B past 119.721 0.030 4.828 117.349 118.190 118.582 298.525 116.347
Text file - A current 2218.756 6.333 1021.760 2102.023 2270.604 2287.972 288312.800 2075.391
Text file - A past 2500.608 3.887 627.131 2176.650 2252.910 2881.993 70447.531 2066.657
Text file - B current 1871.358 3.307 533.528 1782.482 1828.220 1938.245 104983.000 1754.401
Text file - B past 2319.427 5.253 847.592 1878.407 2341.820 2709.386 128298.515 1764.704
Binary TCP - A current 1130.763 0.156 25.198 1117.794 1120.736 1141.519 1835.200 808.696
Binary TCP - A past 1119.039 0.322 52.027 1092.738 1099.917 1144.478 2594.154 863.729
Binary TCP - B current 804.755 0.296 47.715 789.330 792.030 808.399 1751.592 523.591
Binary TCP - B past 815.542 0.181 29.235 801.766 804.321 822.032 1704.844 584.738
Chart by Visualizer

OpenTelemetry

OpenTelemetry (https://opentelemetry.io/) is a widely used monitoring tool, compatible with many programming languages. MooBench measures the overhead of OpenTelemetry using the following configurations:

  • No instrumentation is the baseline execution time of the benchmark program.
  • No Logging measures the execution time of the benchmark program with OpenTelemetry instrumentation, but without measuring or
    logging anything.
  • Logging measures the execution time of the benchmark program with OpenTelemetry instrumentation and writing every output to the standard output.
  • Zipkin measures the execution time of the benchmark program with obtaining every method invocation and sending them to a local Zipkin instance.
  • Prometheus measures the execution time of the benchmark program with obtaining only aggregated execution times, which are stored by a local Prometheus instance.
setup run mean ci sd 1.quartile median 3.quartile min max
No instrumentation current 0.109 0.000 0.040 0.105 0.105 0.107 9.376 0.103
No instrumentation past 0.131 0.000 0.065 0.131 0.131 0.134 12.259 0.105
No logging current 6.420 0.072 11.577 6.176 6.218 6.274 3656.509 6.019
No logging past 6.385 0.082 13.245 6.099 6.309 6.372 4187.614 5.923
Logging current 245.240 67.160 10835.810 147.628 149.473 161.600 1899318.000 142.813
Logging past 234.404 80.104 12924.327 152.110 153.742 171.636 3030748.330 145.883
Zipkin current 10.363 0.115 18.558 8.471 8.566 12.814 4228.631 8.251
Zipkin past 10.236 0.121 19.455 9.162 9.277 9.457 4482.920 8.467
Prometheus current 9.314 0.119 19.127 8.966 9.036 9.192 4372.340 8.764
Prometheus past 9.850 0.116 18.748 9.053 9.141 10.605 4262.398 8.752
Chart by Visualizer

InspectIT

InspectIT (https://www.inspectit.rocks/) is a widely used monitoring tool, which is allows to trace the execution of Java programs. MooBench
measures the overhead of InspectIT using the following configurations:

  • No instrumentation is the baseline execution time of the benchmark program.
  • Deactivated processing measures the execution time of the benchmark program with inspectIT instrumentation, but without measuring or logging
    anything.
  • No Logging measures the execution time of the benchmark program with inspectIT instrumentation and just discarding the measurement results.
  • Zipkin measures the execution time of the benchmark program with obtaining every method invocation and sending them to a local Zipkin instance.
  • Prometheus measures the execution time of the benchmark program with obtaining only aggregated execution times, which are stored by a local Prometheus instance.
setup run mean ci sd 1.quartile median 3.quartile min max
No instrumentation current 32.031 0.040 6.420 30.920 31.067 31.315 243.247 30.401
No instrumentation past 40.085 0.056 9.048 33.746 42.341 42.742 379.025 31.292
Deactivated probe current 41.482 0.049 7.911 40.158 40.408 40.673 389.929 35.874
Deactivated probe past 43.712 0.159 25.590 40.021 40.874 43.897 7554.275 38.755
No logging current 68.698 0.344 55.549 64.912 65.286 65.845 8925.844 63.506
No logging past 74.616 0.346 55.778 69.735 70.113 70.839 8552.580 68.323
Zipkin current 73.099 0.504 81.316 66.443 66.866 67.891 13022.170 59.766
Zipkin past 77.031 0.551 88.888 68.497 69.474 72.406 12943.444 61.931
Prometheus current 49.627 0.073 11.784 47.108 47.345 47.697 2596.750 46.193
Prometheus past 48.620 0.069 11.194 46.537 47.019 50.131 2073.437 42.702
Chart by Visualizer