Performance Benchmarks

On this page, we collect performance benchmarks of Kieker monitoring and other instrumentation frameworks, such as OpenTelemetry and inspectIT, for comparison utilizing MooBench.

Details of the comparison of OpenTelemetry, inspectIT and Kieker can be
found in https://oceanrep.geomar.de/id/eprint/55367/1/SSP2021overhead.pdf

Kieker 4 Java – Monitoring

These are the performance benchmark results for Kieker 4 Java probes utilizing different writer back end.

  • No instrumentation serves as a baseline value indicating how much execution time can be contributed to the benchmark program itself.
  • Deactivated probe as above, but with woven in probes that do not produce any output and do not aggregate information.
  • No logging are activated probes, but the collected data is not send utilizing a null writer.
  • Text File utilizes the text file logging. This is usually slow, as data is written in text files.
  • Binary File utilizes the binary file logging without compression.
  • Binary TCP utilizes a TCP connection to send the monitoring data to a logging service.
setup run mean ci sd 1.quartile median 3.quartile min max
No instrumentation current 0.110 0.000 0.034 0.107 0.107 0.116 10.303 0.106
No instrumentation past 0.111 0.000 0.038 0.108 0.110 0.112 11.274 0.106
Deactivated probe current 0.721 0.007 3.374 0.690 0.705 0.725 2419.057 0.657
Deactivated probe past 0.729 0.007 3.745 0.693 0.710 0.731 2561.950 0.653
No logging current 2.771 0.009 4.819 2.716 2.741 2.772 2842.527 2.643
No logging past 2.702 0.008 4.200 2.651 2.674 2.700 2750.920 2.583
Binary file current 4.881 0.046 23.480 3.522 4.278 4.739 22766.870 3.042
Binary file past 5.156 0.043 22.066 3.592 4.385 5.509 20865.684 3.153
Binary TCP current 6.851 0.010 5.213 6.718 6.889 7.071 2544.941 3.441
Binary TCP past 6.942 0.018 9.113 6.488 6.913 7.358 5555.395 3.031

Kieker 4 Python – Monitoring

Kieker 4 Python is the reimplementation of the probe and monitoring side of Kieker addressing programs written in Python. Here we also want to know how well certain parts perform.

  • No instrumentation is the baseline execution time of the benchmark program.
  • Deactivated probe is the execution time with integrated probes, but these are deactivated.
  • No logging is with active probes, but with a null writer.
  • Text File is with an activated probe logging to a text file.
  • Binary TCP is with an activated probe logging to a remote logging server.

Each of these setups, except of no instrumentation, are tested utilizing two different probe introduction methods (here labeled A and B).

setup run mean ci sd 1.quartile median 3.quartile min max
No instrumentation current 4.235 0.001 0.388 4.078 4.093 4.518 42.729 4.017
No instrumentation past 4.414 0.001 0.439 4.181 4.415 4.637 109.810 4.094
Deactivated probe - A current 9.802 0.001 0.739 9.283 9.360 10.319 56.430 9.162
Deactivated probe - A past 9.557 0.063 31.908 9.157 9.213 10.024 31387.807 9.017
Deactivated probe - B current 9.720 0.002 0.773 9.318 9.360 10.341 188.890 9.185
Deactivated probe - B past 9.577 0.001 0.709 9.197 9.242 10.073 125.522 9.058
No logging - A current 359.894 0.023 11.666 354.985 355.699 356.813 761.336 351.408
No logging - A past 358.323 0.020 10.021 354.502 355.223 356.241 1587.188 351.057
No logging - B current 119.982 0.008 4.222 118.239 118.508 118.866 325.473 116.980
No logging - B past 119.935 0.007 3.684 118.634 118.865 119.176 472.215 117.508
Text file - A current 2257.229 1.703 868.934 2148.528 2192.423 2333.895 206143.200 2115.293
Text file - A past 2232.790 1.527 778.989 2124.837 2198.580 2310.435 160802.710 2091.704
Text file - B current 1928.421 1.336 681.650 1838.922 1878.157 1997.888 192529.300 1804.665
Text file - B past 1925.116 1.264 644.690 1835.118 1878.839 1993.458 178405.610 1804.250
Binary TCP - A current 1168.400 0.075 38.362 1155.077 1158.401 1177.697 3540.205 837.724
Binary TCP - A past 1146.896 0.093 47.598 1132.999 1136.308 1156.047 19384.088 810.390
Binary TCP - B current 850.393 0.081 41.183 835.244 837.934 858.125 3910.014 544.160
Binary TCP - B past 834.901 0.081 41.485 822.804 825.759 840.187 3920.542 524.739

OpenTelemetry

OpenTelemetry (https://opentelemetry.io/) is a widely used monitoring tool, compatible with many programming languages. MooBench measures the overhead of OpenTelemetry using the following configurations:

  • No instrumentation is the baseline execution time of the benchmark program.
  • No Logging measures the execution time of the benchmark program with OpenTelemetry instrumentation, but without measuring or
    logging anything.
  • Logging measures the execution time of the benchmark program with OpenTelemetry instrumentation and writing every output to the standard output.
  • Zipkin measures the execution time of the benchmark program with obtaining every method invocation and sending them to a local Zipkin instance.
  • Prometheus measures the execution time of the benchmark program with obtaining only aggregated execution times, which are stored by a local Prometheus instance.
setup run mean ci sd 1.quartile median 3.quartile min max
No instrumentation current 0.107 0.000 0.039 0.106 0.107 0.107 12.137 0.104
No instrumentation past 0.105 0.000 0.037 0.104 0.105 0.105 14.495 0.102
No logging current 6.296 0.044 22.701 6.135 6.166 6.203 19183.930 6.020
No logging past 6.388 0.029 14.545 6.212 6.272 6.331 8381.638 6.063
Zipkin current 10.280 0.041 21.066 9.482 9.529 9.598 17246.980 8.407
Zipkin past 9.740 0.027 13.714 9.112 9.207 9.312 5093.106 8.066

InspectIT

InspectIT (https://www.inspectit.rocks/) is a widely used monitoring tool, which is allows to trace the execution of Java programs. MooBench
measures the overhead of InspectIT using the following configurations:

  • No instrumentation is the baseline execution time of the benchmark program.
  • Deactivated processing measures the execution time of the benchmark program with inspectIT instrumentation, but without measuring or logging
    anything.
  • No Logging measures the execution time of the benchmark program with inspectIT instrumentation and just discarding the measurement results.
  • Zipkin measures the execution time of the benchmark program with obtaining every method invocation and sending them to a local Zipkin instance.
  • Prometheus measures the execution time of the benchmark program with obtaining only aggregated execution times, which are stored by a local Prometheus instance.
setup run mean ci sd 1.quartile median 3.quartile min max
No instrumentation current 38.902 0.015 7.421 36.268 39.680 40.204 295.167 34.822
No instrumentation past 39.335 0.015 7.743 37.453 37.921 40.583 316.647 35.365
Deactivated probe current 40.676 0.015 7.623 40.012 40.288 40.577 357.406 35.246
Deactivated probe past 40.793 0.015 7.500 40.070 40.346 40.617 406.721 37.034
No logging current 40.090 0.015 7.786 39.770 40.181 40.474 417.436 34.804
No logging past 40.776 0.015 7.485 40.077 40.359 40.627 355.959 35.271
Zipkin current 40.456 0.015 7.621 39.842 40.087 40.350 378.547 35.125
Zipkin past 40.790 0.015 7.808 39.908 40.592 40.881 469.592 35.357
Prometheus current 41.426 0.015 7.536 40.557 40.805 41.078 1334.888 35.937
Prometheus past 40.858 0.015 7.664 40.208 40.493 40.780 559.986 35.351