Performance Benchmarks

On this page, we collect performance benchmarks of Kieker monitoring and other instrumentation frameworks, such as OpenTelemetry and inspectIT, for comparison utilizing MooBench.

Details of the comparison of OpenTelemetry, inspectIT and Kieker can be
found in https://oceanrep.geomar.de/id/eprint/55367/1/SSP2021overhead.pdf

Kieker 4 Java – Monitoring

These are the performance benchmark results for Kieker 4 Java probes utilizing different writer back end.

  • No instrumentation serves as a baseline value indicating how much execution time can be contributed to the benchmark program itself.
  • Deactivated probe as above, but with woven in probes that do not produce any output and do not aggregate information.
  • No logging are activated probes, but the collected data is not send utilizing a null writer.
  • Text File utilizes the text file logging. This is usually slow, as data is written in text files.
  • Binary File utilizes the binary file logging without compression.
  • Binary TCP utilizes a TCP connection to send the monitoring data to a logging service.
setup run mean ci sd 1.quartile median 3.quartile min max
No instrumentation current 0.104 0.000 0.036 0.103 0.103 0.105 18.261 0.101
No instrumentation past 0.108 0.000 0.029 0.106 0.107 0.109 9.568 0.104
Deactivated probe current 0.589 0.006 3.265 0.557 0.574 0.590 2345.813 0.536
Deactivated probe past 0.594 0.007 3.406 0.567 0.575 0.592 2490.503 0.542
No logging current 2.952 0.009 4.445 2.893 2.911 2.933 2827.223 2.830
No logging past 8.495 0.015 7.618 7.803 8.344 8.978 5565.403 3.215
Text file current 200.646 2.825 1441.124 178.058 190.009 196.843 584592.900 3.232
Text file past 199.712 2.651 1352.417 174.652 188.684 192.020 656978.143 3.210
Binary file current 7.709 1.737 885.995 3.345 4.195 4.347 670365.500 3.189
Binary file past 7.361 1.793 914.732 3.654 4.571 5.280 630877.738 3.211
Binary TCP current 6.807 0.025 12.739 6.672 7.037 7.429 9389.723 3.155
Binary TCP past 7.242 0.049 25.084 6.711 7.201 7.636 21058.512 3.133

Kieker 4 Python – Monitoring

Kieker 4 Python is the reimplementation of the probe and monitoring side of Kieker addressing programs written in Python. Here we also want to know how well certain parts perform.

  • No instrumentation is the baseline execution time of the benchmark program.
  • Deactivated probe is the execution time with integrated probes, but these are deactivated.
  • No logging is with active probes, but with a null writer.
  • Text File is with an activated probe logging to a text file.
  • Binary TCP is with an activated probe logging to a remote logging server.

Each of these setups, except of no instrumentation, are tested utilizing two different probe introduction methods (here labeled A and B).

setup run mean ci sd 1.quartile median 3.quartile min max
No instrumentation current 4.357 0.001 0.433 4.091 4.507 4.546 51.294 4.008
No instrumentation past 4.306 0.001 0.413 4.110 4.153 4.572 59.843 4.029
Deactivated probe - A current 9.663 0.001 0.717 9.115 10.048 10.132 56.057 8.985
Deactivated probe - A past 9.553 0.002 0.944 9.131 9.182 10.096 436.452 8.959
Deactivated probe - B current 9.898 0.001 0.713 10.014 10.074 10.111 158.641 8.951
Deactivated probe - B past 9.449 0.001 0.616 9.245 9.276 9.746 187.030 9.070
No logging - A current 357.101 0.020 10.213 352.656 353.814 355.922 1242.214 348.391
No logging - A past 359.896 0.020 10.075 356.054 356.789 357.820 2229.216 350.907
No logging - B current 120.944 0.010 5.188 118.371 118.645 119.143 303.907 117.275
No logging - B past 121.450 0.008 4.261 119.720 119.970 120.318 318.682 118.025
Text file - A current 2268.855 1.711 872.778 2158.320 2203.681 2345.419 159063.900 2128.050
Text file - A past 2256.306 1.939 989.123 2141.389 2193.368 2333.805 356278.559 2107.280
Text file - B current 1920.418 1.547 789.171 1827.300 1872.102 1985.211 184227.800 1791.310
Text file - B past 1937.183 1.515 773.139 1841.551 1960.236 1999.208 188705.805 1801.363
Binary TCP - A current 1168.811 0.088 45.144 1153.377 1156.486 1177.280 4248.198 832.278
Binary TCP - A past 1147.513 0.607 309.907 1132.100 1135.498 1155.165 284660.023 810.983
Binary TCP - B current 841.126 0.061 31.188 830.335 833.723 849.399 3913.742 543.322
Binary TCP - B past 843.960 0.101 51.684 830.267 833.474 848.942 11642.493 537.298

OpenTelemetry

OpenTelemetry (https://opentelemetry.io/) is a widely used monitoring tool, compatible with many programming languages. MooBench measures the overhead of OpenTelemetry using the following configurations:

  • No instrumentation is the baseline execution time of the benchmark program.
  • No Logging measures the execution time of the benchmark program with OpenTelemetry instrumentation, but without measuring or
    logging anything.
  • Logging measures the execution time of the benchmark program with OpenTelemetry instrumentation and writing every output to the standard output.
  • Zipkin measures the execution time of the benchmark program with obtaining every method invocation and sending them to a local Zipkin instance.
  • Prometheus measures the execution time of the benchmark program with obtaining only aggregated execution times, which are stored by a local Prometheus instance.
setup run mean ci sd 1.quartile median 3.quartile min max
No instrumentation current 0.108 0.000 0.033 0.107 0.107 0.108 8.374 0.105
No instrumentation past 0.112 0.000 0.036 0.110 0.112 0.113 10.278 0.106
No logging current 6.165 0.021 10.531 5.966 6.069 6.178 4156.508 5.596
No logging past 6.413 0.020 10.366 6.233 6.298 6.362 4339.917 6.031
Logging current 159.407 4.548 2320.310 147.944 149.026 151.797 764858.600 131.444
Logging past 162.138 3.919 1999.311 151.434 153.257 155.959 684830.202 134.025
Zipkin current 9.588 0.023 11.829 9.083 9.156 9.259 5359.557 7.951
Zipkin past 10.347 0.030 15.539 9.660 9.726 9.858 9591.788 8.822
Prometheus current 9.477 0.018 9.046 9.265 9.309 9.368 2505.684 9.073
Prometheus past 9.048 0.027 13.690 8.826 8.911 8.976 8229.641 8.619

InspectIT

InspectIT (https://www.inspectit.rocks/) is a widely used monitoring tool, which is allows to trace the execution of Java programs. MooBench
measures the overhead of InspectIT using the following configurations:

  • No instrumentation is the baseline execution time of the benchmark program.
  • Deactivated processing measures the execution time of the benchmark program with inspectIT instrumentation, but without measuring or logging
    anything.
  • No Logging measures the execution time of the benchmark program with inspectIT instrumentation and just discarding the measurement results.
  • Zipkin measures the execution time of the benchmark program with obtaining every method invocation and sending them to a local Zipkin instance.
  • Prometheus measures the execution time of the benchmark program with obtaining only aggregated execution times, which are stored by a local Prometheus instance.
setup run mean ci sd 1.quartile median 3.quartile min max
No instrumentation current 37.709 0.015 7.399 35.346 37.944 38.209 301.542 33.274
No instrumentation past 37.029 0.015 7.625 34.542 36.853 38.179 1297.717 33.309
Deactivated probe current 44.808 0.062 31.700 43.813 44.071 44.373 30362.280 38.826
Deactivated probe past 44.990 0.052 26.475 44.194 44.604 44.888 15518.462 39.064
No logging current 73.013 0.047 24.070 71.094 71.416 71.848 3683.990 69.414
No logging past 74.690 0.092 46.897 72.168 72.713 73.154 30033.287 70.380
Zipkin current 73.169 0.123 62.505 64.409 69.722 70.671 24261.160 62.369
Zipkin past 74.220 0.193 98.621 65.121 69.604 71.496 45830.169 62.953
Prometheus current 53.068 0.179 91.249 51.779 52.068 52.401 60258.680 46.216
Prometheus past 53.313 0.076 38.895 51.871 52.326 52.660 24959.505 46.045