Performance Benchmarks

On this page, we collect performance benchmarks of Kieker monitoring and other instrumentation frameworks, such as OpenTelemetry and inspectIT, for comparison utilizing MooBench.

Details of the comparison of OpenTelemetry, inspectIT and Kieker can be found in https://oceanrep.geomar.de/id/eprint/55367/1/SSP2021overhead.pdf

Kieker Observability Framework – Java

These are the performance benchmark results for Kieker 4 Java probes utilizing different writer back end.

  • No instrumentation serves as a baseline value indicating how much execution time can be contributed to the benchmark program itself.
  • Deactivated probe as above, but with woven in probes that do not produce any output and do not aggregate information.
  • No logging are activated probes, but the collected data is not send utilizing a null writer.
  • Text File utilizes the text file logging. This is usually slow, as data is written in text files and getting more irrelevant in the performance benchmarks. The diagram below, therefore, does not show the results.
  • Binary File utilizes the binary file logging without compression.
  • Binary TCP utilizes a TCP connection to send the monitoring data to a logging service.
setup run mean ci sd 1.quartile median 3.quartile max min
No instrumentation current 0.073 0.000 0.219 0.072 0.072 0.073 619.344 0.068
No instrumentation past 0.073 0.000 0.243 0.072 0.072 0.073 545.714 0.068
Deactivated probe current 0.096 0.000 0.088 0.094 0.095 0.097 169.602 0.084
Deactivated probe past 0.096 0.000 0.079 0.095 0.096 0.097 140.568 0.085
No collection current 1.126 0.004 6.036 1.084 1.091 1.100 9533.647 1.049
No collection past 1.127 0.005 7.432 1.084 1.091 1.099 9553.773 1.052
Binary file current 3.982 0.003 4.577 1.965 3.367 4.733 3740.320 1.380
Binary file past 3.818 0.003 4.560 1.927 3.341 4.532 3762.439 1.377
Binary TCP current 3.734 0.005 8.697 2.185 3.344 4.234 8003.040 1.301
Binary TCP past 3.270 0.006 8.395 2.172 2.880 3.772 7332.098 1.296

Kieker Observability Framework – Python

Kieker 4 Python is the reimplementation of the probe and monitoring side of Kieker addressing programs written in Python. Here we also want to know how well certain parts perform.

  • No instrumentation is the baseline execution time of the benchmark program.
  • Deactivated probe is the execution time with integrated probes, but these are deactivated.
  • No logging is with active probes, but with a null writer.
  • Text File is with an activated probe logging to a text file.
  • Binary TCP is with an activated probe logging to a remote logging server.

Each of these setups, except of no instrumentation, are tested utilizing two different probe introduction methods (here labeled A and B).

setup run mean ci sd 1.quartile median 3.quartile max min
No instrumentation current 0.901 0.001 1.016 0.886 0.896 0.909 2265.475 0.843
No instrumentation past 0.898 0.000 0.325 0.883 0.893 0.905 644.241 0.844
Deactivated probe - A current 2.157 0.000 0.679 2.134 2.147 2.163 1405.958 2.068
Deactivated probe - A past 2.159 0.000 0.763 2.135 2.149 2.164 1535.932 2.069
Deactivated probe - B current 2.202 0.000 0.742 2.167 2.194 2.221 2200.607 2.087
Deactivated probe - B past 2.193 0.001 0.786 2.162 2.183 2.206 1693.010 2.087
No logging - A current 63.390 0.002 3.900 62.453 63.004 63.667 2460.785 60.139
No logging - A past 63.235 0.002 3.710 62.360 62.851 63.547 2080.595 60.194
No logging - B current 17.720 0.001 1.625 17.426 17.623 17.858 2344.035 16.948
No logging - B past 17.639 0.001 1.833 17.446 17.559 17.724 2050.079 16.960
Text file - A current 417.664 0.017 27.226 407.820 411.575 417.041 36195.700 393.531
Text file - A past 413.435 0.020 29.411 405.737 409.103 414.421 33758.273 393.202
Text file - B current 325.265 0.017 27.321 318.983 321.357 324.840 30786.290 309.870
Text file - B past 323.821 0.022 32.056 315.816 318.761 327.860 36872.034 306.890
Binary TCP - A current 213.041 0.005 8.783 207.004 212.109 217.630 2566.241 134.741
Binary TCP - A past 214.491 0.005 7.935 209.598 213.844 218.063 2048.922 136.416
Binary TCP - B current 147.773 0.007 11.682 139.041 147.143 156.897 2650.855 81.792
Binary TCP - B past 146.754 0.008 11.465 138.372 145.653 155.938 2187.390 80.968

OpenTelemetry

OpenTelemetry (https://opentelemetry.io/) is a widely used monitoring tool, compatible with many programming languages. MooBench measures the overhead of OpenTelemetry using the following configurations:

  • No instrumentation is the baseline execution time of the benchmark program.
  • No Logging measures the execution time of the benchmark program with OpenTelemetry instrumentation, but without measuring or
    logging anything.
  • Logging measures the execution time of the benchmark program with OpenTelemetry instrumentation and writing every output to the standard output. Text file logging is slow and getting more irrelevant in the performance benchmarks. The diagram below, therefore, does not show the results.
  • Zipkin measures the execution time of the benchmark program with obtaining every method invocation and sending them to a local Zipkin instance.
  • Prometheus measures the execution time of the benchmark program with obtaining only aggregated execution times, which are stored by a local Prometheus instance.
setup run mean ci sd 1.quartile median 3.quartile max min
No instrumentation current 0.073 0.000 0.199 0.072 0.072 0.073 608.479 0.068
No instrumentation past 0.073 0.000 0.086 0.072 0.073 0.073 175.825 0.068
No logging current 4.310 0.007 10.923 4.128 4.181 4.272 5802.467 3.959
No logging past 4.315 0.007 10.830 4.131 4.199 4.303 6955.534 3.897
Zipkin current 7.172 0.010 16.754 6.681 6.775 7.141 16864.830 6.381
Zipkin past 6.967 0.011 16.141 6.539 6.641 6.973 8469.853 6.325
Prometheus current 6.247 0.007 11.686 6.041 6.103 6.200 5970.051 5.894
Prometheus past 6.268 0.008 12.025 6.069 6.133 6.228 6942.411 5.842

InspectIT

InspectIT (https://www.inspectit.rocks/) is a widely used monitoring tool, which is allows to trace the execution of Java programs. MooBench
measures the overhead of InspectIT using the following configurations:

  • No instrumentation is the baseline execution time of the benchmark program.
  • Deactivated processing measures the execution time of the benchmark program with inspectIT instrumentation, but without measuring or logging
    anything.
  • No Logging measures the execution time of the benchmark program with inspectIT instrumentation and just discarding the measurement results.
  • Zipkin measures the execution time of the benchmark program with obtaining every method invocation and sending them to a local Zipkin instance.
  • Prometheus measures the execution time of the benchmark program with obtaining only aggregated execution times, which are stored by a local Prometheus instance.
setup run mean ci sd 1.quartile median 3.quartile max min
No instrumentation current 0.073 0.000 0.091 0.072 0.072 0.073 167.524 0.068
No instrumentation past 0.073 0.000 0.226 0.072 0.072 0.073 646.381 0.068
Deactivated probe current 2.618 0.016 25.012 2.465 2.487 2.535 12918.890 2.380
Deactivated probe past 2.080 0.015 22.368 1.233 2.450 2.506 14005.557 0.205
No logging current 11.132 0.009 14.983 11.494 12.098 12.421 7259.826 0.068
No logging past 11.196 0.009 14.417 10.045 11.858 12.212 7157.439 6.251
Zipkin current 12.048 0.014 22.985 11.355 11.563 12.103 16276.620 10.981
Zipkin past 11.192 0.014 21.415 11.221 11.429 11.960 10370.458 5.804
Prometheus current 5.070 0.013 21.267 5.288 5.428 5.649 13579.070 0.068
Prometheus past 4.119 0.013 20.117 1.952 5.151 5.371 14300.526 0.224