Performance Benchmarks

On this page, we collect performance benchmarks of Kieker monitoring and other instrumentation frameworks, such as OpenTelemetry and inspectIT, for comparison utilizing MooBench.

Details of the comparison of OpenTelemetry, inspectIT and Kieker can be
found in https://oceanrep.geomar.de/id/eprint/55367/1/SSP2021overhead.pdf

Kieker 4 Java – Monitoring

These are the performance benchmark results for Kieker 4 Java probes utilizing different writer back end.

  • No instrumentation serves as a baseline value indicating how much execution time can be contributed to the benchmark program itself.
  • Deactivated probe as above, but with woven in probes that do not produce any output and do not aggregate information.
  • No logging are activated probes, but the collected data is not send utilizing a null writer.
  • Text File utilizes the text file logging. This is usually slow, as data is written in text files.
  • Binary File utilizes the binary file logging without compression.
  • Binary TCP utilizes a TCP connection to send the monitoring data to a logging service.
setup run mean ci sd 1.quartile median 3.quartile min max
No instrumentation current 0.104 0.000 0.036 0.103 0.103 0.105 8.271 0.101
No instrumentation past 0.110 0.000 0.046 0.110 0.110 0.112 28.809 0.104
Deactivated probe current 0.611 0.006 3.303 0.583 0.595 0.610 2396.199 0.548
Deactivated probe past 0.592 0.007 3.319 0.564 0.573 0.591 2430.381 0.541
No logging current 2.909 0.007 3.675 2.862 2.876 2.896 2174.345 2.821
No logging past 2.988 0.011 5.412 2.929 2.951 2.977 3481.115 2.864
Binary file current 7.245 1.695 864.938 3.243 3.958 4.275 608339.300 3.129
Binary file past 7.349 1.625 828.930 3.442 4.001 4.502 594068.802 3.227
Binary TCP current 7.038 0.037 19.091 6.804 7.171 7.351 18419.890 3.248
Binary TCP past 6.945 0.055 28.149 6.738 7.002 7.298 26321.164 3.062
Text file past 202.164 2.798 1427.655 176.683 190.028 194.289 552153.731 3.246

Kieker 4 Python – Monitoring

Kieker 4 Python is the reimplementation of the probe and monitoring side of Kieker addressing programs written in Python. Here we also want to know how well certain parts perform.

  • No instrumentation is the baseline execution time of the benchmark program.
  • Deactivated probe is the execution time with integrated probes, but these are deactivated.
  • No logging is with active probes, but with a null writer.
  • Text File is with an activated probe logging to a text file.
  • Binary TCP is with an activated probe logging to a remote logging server.

Each of these setups, except of no instrumentation, are tested utilizing two different probe introduction methods (here labeled A and B).

setup run mean ci sd 1.quartile median 3.quartile min max
No instrumentation current 4.325 0.001 0.462 4.159 4.177 4.601 176.635 4.077
No instrumentation past 4.315 0.001 0.390 4.174 4.298 4.428 114.082 4.091
Deactivated probe - A current 9.382 0.003 1.308 9.011 9.045 9.993 1093.309 8.900
Deactivated probe - A past 9.320 0.001 0.671 9.071 9.109 9.373 218.733 8.951
Deactivated probe - B current 9.606 0.001 0.724 9.161 9.208 10.171 188.997 9.027
Deactivated probe - B past 9.354 0.001 0.657 9.116 9.144 9.296 194.346 8.993
No logging - A current 364.684 0.026 13.164 358.755 359.498 360.817 1992.641 355.193
No logging - A past 361.783 0.020 10.457 357.870 358.866 359.992 1387.308 352.378
No logging - B current 118.983 0.008 3.942 117.545 117.793 118.117 304.469 116.296
No logging - B past 119.191 0.008 4.286 117.833 118.045 118.327 1055.841 116.724
Text file - A current 2241.650 1.573 802.578 2136.319 2171.894 2323.512 142771.900 2104.141
Text file - A past 2260.164 1.735 885.124 2149.638 2212.363 2337.175 281269.632 2116.933
Text file - B current 1919.303 1.234 629.856 1831.191 1870.450 1987.960 136456.100 1799.805
Text file - B past 1935.910 1.440 734.860 1841.735 1914.344 2001.322 164833.883 1808.883
Binary TCP - A current 1162.555 0.060 30.655 1150.321 1153.477 1173.542 4259.069 828.887
Binary TCP - A past 1150.773 0.071 36.062 1137.770 1140.948 1160.641 4232.643 816.363
Binary TCP - B current 844.270 0.063 31.969 833.774 836.403 850.114 3904.180 538.483
Binary TCP - B past 843.170 0.076 38.826 831.307 834.074 848.992 3876.372 531.774

OpenTelemetry

OpenTelemetry (https://opentelemetry.io/) is a widely used monitoring tool, compatible with many programming languages. MooBench measures the overhead of OpenTelemetry using the following configurations:

  • No instrumentation is the baseline execution time of the benchmark program.
  • No Logging measures the execution time of the benchmark program with OpenTelemetry instrumentation, but without measuring or
    logging anything.
  • Logging measures the execution time of the benchmark program with OpenTelemetry instrumentation and writing every output to the standard output.
  • Zipkin measures the execution time of the benchmark program with obtaining every method invocation and sending them to a local Zipkin instance.
  • Prometheus measures the execution time of the benchmark program with obtaining only aggregated execution times, which are stored by a local Prometheus instance.
setup run mean ci sd 1.quartile median 3.quartile min max
No instrumentation current 0.106 0.000 0.043 0.103 0.104 0.110 17.960 0.101
No instrumentation past 0.113 0.000 0.046 0.111 0.113 0.114 23.874 0.109
No logging current 6.135 0.021 10.943 5.964 6.011 6.061 4663.394 5.789
No logging past 6.733 0.027 13.585 6.573 6.621 6.671 7693.293 6.364
Zipkin current 10.640 0.026 13.415 9.778 9.891 10.029 5587.737 8.448
Zipkin past 9.875 0.043 21.861 9.000 9.195 9.356 16788.903 8.449
Logging past 171.343 4.595 2344.379 157.490 158.862 161.676 675895.863 140.191
Prometheus past 8.968 0.055 28.304 8.726 8.783 8.857 24650.676 8.503

InspectIT

InspectIT (https://www.inspectit.rocks/) is a widely used monitoring tool, which is allows to trace the execution of Java programs. MooBench
measures the overhead of InspectIT using the following configurations:

  • No instrumentation is the baseline execution time of the benchmark program.
  • Deactivated processing measures the execution time of the benchmark program with inspectIT instrumentation, but without measuring or logging
    anything.
  • No Logging measures the execution time of the benchmark program with inspectIT instrumentation and just discarding the measurement results.
  • Zipkin measures the execution time of the benchmark program with obtaining every method invocation and sending them to a local Zipkin instance.
  • Prometheus measures the execution time of the benchmark program with obtaining only aggregated execution times, which are stored by a local Prometheus instance.
setup run mean ci sd 1.quartile median 3.quartile min max
No instrumentation current 37.339 0.015 7.435 34.531 37.896 38.212 350.581 33.250
No instrumentation past 37.120 0.015 7.476 34.772 36.778 38.398 821.342 33.629
Deactivated probe current 44.863 0.105 53.396 43.872 44.111 44.396 47718.340 38.963
Deactivated probe past 45.106 0.053 27.196 44.409 44.720 45.008 21014.290 39.553
No logging current 72.967 0.046 23.498 70.808 71.128 71.570 3536.829 69.140
No logging past 73.724 0.067 33.986 71.497 71.829 72.288 18870.115 67.880
Zipkin current 75.384 0.166 84.725 66.172 71.326 72.246 33574.590 63.707
Zipkin past 76.035 0.133 67.995 66.931 71.607 73.285 20157.011 64.761
Prometheus current 53.301 0.020 10.236 52.191 52.434 52.731 2717.576 46.687
Prometheus past 53.765 0.049 24.869 52.692 52.983 53.320 21597.856 46.793