Performance Benchmarks

On this page, we collect performance benchmarks of Kieker monitoring and other instrumentation frameworks, such as OpenTelemetry and inspectIT, for comparison utilizing MooBench.

Details of the comparison of OpenTelemetry, inspectIT and Kieker can be
found in https://oceanrep.geomar.de/id/eprint/55367/1/SSP2021overhead.pdf

Kieker 4 Java – Monitoring

These are the performance benchmark results for Kieker 4 Java probes utilizing different writer back end.

  • No instrumentation serves as a baseline value indicating how much execution time can be contributed to the benchmark program itself.
  • Deactivated probe as above, but with woven in probes that do not produce any output and do not aggregate information.
  • No logging are activated probes, but the collected data is not send utilizing a null writer.
  • Text File utilizes the text file logging. This is usually slow, as data is written in text files.
  • Binary File utilizes the binary file logging without compression.
  • Binary TCP utilizes a TCP connection to send the monitoring data to a logging service.
setup run mean ci sd 1.quartile median 3.quartile min max
No instrumentation current 0.158 0.000 0.167 0.157 0.158 0.158 159.440 0.103
No instrumentation past 0.151 0.000 0.046 0.149 0.150 0.153 18.176 0.108
Deactivated probe current 0.594 0.006 3.102 0.564 0.577 0.593 2287.069 0.537
Deactivated probe past 0.727 0.007 3.293 0.671 0.686 0.768 2420.194 0.576
No logging current 18.552 0.010 5.149 17.526 18.514 19.407 2604.983 3.741
No logging past 19.261 0.011 5.234 17.681 19.228 20.742 2556.738 3.636
Text file current 233.764 4.645 2370.052 201.646 217.767 221.357 2043597.000 3.316
Text file past 234.279 7.862 3721.324 210.406 217.526 220.695 2285053.177 3.243
Binary file current 8.052 4.346 2217.238 3.512 4.445 4.838 2057018.000 3.258
Binary file past 17.981 13.279 6774.761 3.709 4.671 6.773 4376490.139 3.235
Binary TCP current 9.924 0.069 35.211 8.316 10.536 12.047 25275.840 3.198
Binary TCP past 9.665 0.011 5.497 8.440 10.308 11.578 2665.790 3.175
Chart by Visualizer

Kieker 4 Python – Monitoring

Kieker 4 Python is the reimplementation of the probe and monitoring side of Kieker addressing programs written in Python. Here we also want to know how well certain parts perform.

  • No instrumentation is the baseline execution time of the benchmark program.
  • Deactivated probe is the execution time with integrated probes, but these are deactivated.
  • No logging is with active probes, but with a null writer.
  • Text File is with an activated probe logging to a text file.
  • Binary TCP is with an activated probe logging to a remote logging server.

Each of these setups, except of no instrumentation, are tested utilizing two different probe introduction methods (here labeled A and B).

setup run mean ci sd 1.quartile median 3.quartile min max
No instrumentation current 4.254 0.001 0.366 4.087 4.116 4.523 35.476 3.998
No instrumentation past 4.331 0.001 0.475 4.171 4.234 4.416 269.717 4.077
Deactivated probe - A current 9.549 0.001 0.691 9.314 9.342 9.383 189.528 9.189
Deactivated probe - A past 9.382 0.002 0.688 9.149 9.181 9.318 293.491 8.996
Deactivated probe - B current 9.064 0.001 0.509 8.941 8.971 9.008 189.632 8.764
Deactivated probe - B past 9.437 0.003 1.503 9.096 9.140 9.891 1220.060 8.950
No logging - A current 357.593 0.019 9.786 353.794 354.551 355.599 974.062 350.371
No logging - A past 404.203 0.169 85.466 354.668 355.856 452.798 6594.564 351.030
No logging - B current 120.042 0.006 3.238 119.020 119.226 119.481 1065.177 117.914
No logging - B past 128.662 0.034 17.381 119.154 123.548 129.982 381.606 117.007
Text file - A current 2215.957 1.228 626.493 2105.651 2145.591 2289.835 165750.700 2075.579
Text file - A past 2347.611 1.468 703.769 2101.106 2261.813 2412.633 148964.648 2065.155
Text file - B current 2167.505 1.299 662.782 1806.814 1951.092 2753.693 140334.500 1765.531
Text file - B past 1975.387 1.132 548.527 1817.190 1898.270 2062.030 151700.269 1768.465
Binary TCP - A current 1149.019 0.145 74.034 1115.662 1120.581 1142.472 3453.386 815.791
Binary TCP - A past 1134.057 0.144 67.697 1102.989 1107.242 1130.848 3401.103 799.445
Binary TCP - B current 824.403 0.076 38.584 809.955 812.854 830.328 3145.507 542.263
Binary TCP - B past 813.005 0.157 75.522 797.434 800.180 817.581 34268.398 516.104
Chart by Visualizer

OpenTelemetry

OpenTelemetry (https://opentelemetry.io/) is a widely used monitoring tool, compatible with many programming languages. MooBench measures the overhead of OpenTelemetry using the following configurations:

  • No instrumentation is the baseline execution time of the benchmark program.
  • No Logging measures the execution time of the benchmark program with OpenTelemetry instrumentation, but without measuring or
    logging anything.
  • Logging measures the execution time of the benchmark program with OpenTelemetry instrumentation and writing every output to the standard output.
  • Zipkin measures the execution time of the benchmark program with obtaining every method invocation and sending them to a local Zipkin instance.
  • Prometheus measures the execution time of the benchmark program with obtaining only aggregated execution times, which are stored by a local Prometheus instance.
setup run mean ci sd 1.quartile median 3.quartile min max
No instrumentation current 0.106 0.000 0.028 0.105 0.105 0.106 6.528 0.101
No instrumentation past 0.108 0.000 0.034 0.108 0.108 0.109 17.980 0.103
No logging current 6.193 0.016 8.007 5.236 5.309 7.805 2475.215 5.041
No logging past 5.942 0.022 10.574 5.749 5.800 5.959 5239.459 5.571
Logging current 189.861 10.916 5569.518 149.158 150.681 159.561 2571526.000 143.469
Logging past 182.879 14.969 6974.329 150.608 151.934 156.710 3041169.279 134.278
Zipkin current 10.091 0.042 21.538 9.357 9.414 9.499 18543.070 8.278
Zipkin past 11.359 0.032 14.866 10.203 10.305 11.912 6376.025 9.307
Prometheus current 9.245 0.014 7.318 9.052 9.106 9.169 2053.945 8.826
Prometheus past 9.604 0.020 8.955 9.321 9.375 9.470 2896.351 9.077
Chart by Visualizer

InspectIT

InspectIT (https://www.inspectit.rocks/) is a widely used monitoring tool, which is allows to trace the execution of Java programs. MooBench
measures the overhead of InspectIT using the following configurations:

  • No instrumentation is the baseline execution time of the benchmark program.
  • Deactivated processing measures the execution time of the benchmark program with inspectIT instrumentation, but without measuring or logging
    anything.
  • No Logging measures the execution time of the benchmark program with inspectIT instrumentation and just discarding the measurement results.
  • Zipkin measures the execution time of the benchmark program with obtaining every method invocation and sending them to a local Zipkin instance.
  • Prometheus measures the execution time of the benchmark program with obtaining only aggregated execution times, which are stored by a local Prometheus instance.
setup run mean ci sd 1.quartile median 3.quartile min max
No instrumentation current 31.827 0.012 6.344 31.024 31.180 31.375 1290.470 30.254
No instrumentation past 34.389 0.016 7.951 31.510 32.515 34.436 1685.824 30.610
Deactivated probe current 42.574 0.040 20.622 40.220 40.489 40.822 7959.011 35.634
Deactivated probe past 43.285 0.037 17.079 40.572 40.823 41.213 5669.909 36.291
No logging current 73.158 0.048 24.384 67.903 68.259 68.958 3477.380 66.438
No logging past 71.725 0.062 28.155 67.782 68.128 68.698 4711.280 64.526
Zipkin current 73.127 0.130 66.141 67.438 67.814 68.521 28251.180 60.340
Zipkin past 73.087 0.150 69.659 67.586 68.280 69.160 29585.727 60.851
Prometheus current 49.593 0.028 14.263 47.542 47.848 48.192 6624.008 42.325
Prometheus past 50.813 0.041 20.233 47.693 47.977 48.397 14473.344 42.521
Chart by Visualizer