Performance Benchmarks

On this page, we collect performance benchmarks of Kieker monitoring and other instrumentation frameworks, such as OpenTelemetry and inspectIT, for comparison utilizing MooBench.

Details of the comparison of OpenTelemetry, inspectIT and Kieker can be found in https://oceanrep.geomar.de/id/eprint/55367/1/SSP2021overhead.pdf

Kieker Observability Framework – Java

These are the performance benchmark results for Kieker 4 Java probes utilizing different writer back end.

  • No instrumentation serves as a baseline value indicating how much execution time can be contributed to the benchmark program itself.
  • Deactivated probe as above, but with woven in probes that do not produce any output and do not aggregate information.
  • No logging are activated probes, but the collected data is not send utilizing a null writer.
  • Text File utilizes the text file logging. This is usually slow, as data is written in text files and getting more irrelevant in the performance benchmarks. The diagram below, therefore, does not show the results.
  • Binary File utilizes the binary file logging without compression.
  • Binary TCP utilizes a TCP connection to send the monitoring data to a logging service.
setup run mean ci sd 1.quartile median 3.quartile max min
No instrumentation current 0.107 0.000 0.028 0.106 0.107 0.107 12.475 0.104
No instrumentation past 0.106 0.000 0.041 0.105 0.106 0.106 22.172 0.102
Deactivated probe current 0.719 0.007 3.728 0.691 0.702 0.720 2716.936 0.655
Deactivated probe past 0.708 0.008 3.848 0.677 0.690 0.709 2748.379 0.646
No logging current 2.739 0.012 6.070 2.683 2.705 2.734 3281.730 2.628
No logging past 2.694 0.010 5.273 2.638 2.660 2.688 3026.603 2.575
Binary file current 5.228 0.044 22.234 3.494 4.495 6.729 18803.230 3.306
Binary file past 5.108 0.029 14.762 3.503 4.482 6.089 11747.556 3.285
Binary TCP current 7.118 0.060 30.430 6.720 6.956 7.411 27037.440 2.963
Binary TCP past 6.915 0.011 5.819 6.643 6.927 7.471 2809.011 3.142

Kieker Observability Framework – Python

Kieker 4 Python is the reimplementation of the probe and monitoring side of Kieker addressing programs written in Python. Here we also want to know how well certain parts perform.

  • No instrumentation is the baseline execution time of the benchmark program.
  • Deactivated probe is the execution time with integrated probes, but these are deactivated.
  • No logging is with active probes, but with a null writer.
  • Text File is with an activated probe logging to a text file.
  • Binary TCP is with an activated probe logging to a remote logging server.

Each of these setups, except of no instrumentation, are tested utilizing two different probe introduction methods (here labeled A and B).

setup run mean ci sd 1.quartile median 3.quartile max min
No instrumentation current 1.590 0.000 0.186 1.498 1.623 1.665 28.068 1.428
No instrumentation past 1.570 0.000 0.162 1.531 1.543 1.618 27.317 1.468
Deactivated probe - A current 1.474 0.000 0.153 1.458 1.465 1.473 23.884 1.411
Deactivated probe - A past 1.594 0.000 0.180 1.522 1.599 1.649 45.574 1.456
Deactivated probe - B current 4.041 0.001 0.314 3.836 3.871 4.266 36.427 3.731
Deactivated probe - B past 3.971 0.001 0.340 3.803 3.840 4.163 115.317 3.706
No logging - A current 1.528 0.000 0.160 1.503 1.511 1.520 21.019 1.456
No logging - A past 1.552 0.000 0.202 1.512 1.527 1.577 102.314 1.449
No logging - B current 44.800 0.004 2.240 43.579 43.760 44.221 234.279 42.808
No logging - B past 45.142 0.004 2.145 44.074 44.286 44.880 244.348 43.115
Text file - A current 1.533 0.000 0.187 1.475 1.484 1.625 28.758 1.412
Text file - A past 1.569 0.000 0.183 1.506 1.560 1.576 43.397 1.441
Text file - B current 1314.070 1.176 600.227 1258.674 1280.300 1361.663 325697.700 1229.084
Text file - B past 1320.334 1.185 604.546 1258.258 1316.203 1362.312 248525.652 1228.901
Binary TCP - A current 1.638 0.000 0.207 1.640 1.662 1.673 101.515 1.453
Binary TCP - A past 1.581 0.000 0.184 1.502 1.570 1.655 56.432 1.446
Binary TCP - B current 478.005 1.961 1000.306 468.908 471.681 476.412 1000428.000 234.105
Binary TCP - B past 482.254 0.062 31.516 473.343 476.270 481.275 2691.430 235.249

OpenTelemetry

OpenTelemetry (https://opentelemetry.io/) is a widely used monitoring tool, compatible with many programming languages. MooBench measures the overhead of OpenTelemetry using the following configurations:

  • No instrumentation is the baseline execution time of the benchmark program.
  • No Logging measures the execution time of the benchmark program with OpenTelemetry instrumentation, but without measuring or
    logging anything.
  • Logging measures the execution time of the benchmark program with OpenTelemetry instrumentation and writing every output to the standard output. Text file logging is slow and getting more irrelevant in the performance benchmarks. The diagram below, therefore, does not show the results.
  • Zipkin measures the execution time of the benchmark program with obtaining every method invocation and sending them to a local Zipkin instance.
  • Prometheus measures the execution time of the benchmark program with obtaining only aggregated execution times, which are stored by a local Prometheus instance.
setup run mean ci sd 1.quartile median 3.quartile max min
No instrumentation current 0.108 0.000 0.045 0.106 0.107 0.108 23.643 0.103
No instrumentation past 0.105 0.000 0.036 0.105 0.105 0.105 19.332 0.103
No logging current 7.185 0.042 21.239 7.015 7.053 7.101 10720.580 6.865
No logging past 7.064 0.064 32.683 6.835 6.927 7.006 28672.885 6.635
Zipkin current 10.962 0.025 12.818 10.434 10.591 10.721 3392.149 10.157
Zipkin past 10.041 0.029 14.748 9.462 9.519 9.595 5629.390 9.012

InspectIT

InspectIT (https://www.inspectit.rocks/) is a widely used monitoring tool, which is allows to trace the execution of Java programs. MooBench
measures the overhead of InspectIT using the following configurations:

  • No instrumentation is the baseline execution time of the benchmark program.
  • Deactivated processing measures the execution time of the benchmark program with inspectIT instrumentation, but without measuring or logging
    anything.
  • No Logging measures the execution time of the benchmark program with inspectIT instrumentation and just discarding the measurement results.
  • Zipkin measures the execution time of the benchmark program with obtaining every method invocation and sending them to a local Zipkin instance.
  • Prometheus measures the execution time of the benchmark program with obtaining only aggregated execution times, which are stored by a local Prometheus instance.
setup run mean ci sd 1.quartile median 3.quartile max min
No instrumentation current 0.107 0.000 0.157 0.106 0.107 0.107 153.510 0.103
No instrumentation past 0.105 0.000 0.036 0.104 0.105 0.105 13.896 0.102
Deactivated probe current 13.907 0.022 11.069 13.592 13.682 13.799 3119.681 13.171
Deactivated probe past 13.544 0.032 16.090 13.241 13.319 13.413 9132.622 12.882
No logging current 40.236 0.070 35.808 39.257 39.456 39.743 31849.040 34.955
No logging past 40.485 0.055 28.284 39.415 39.674 40.027 21401.016 37.408
Zipkin current 41.783 0.059 30.206 39.177 39.458 39.885 18348.710 34.472
Zipkin past 42.151 0.118 60.068 39.839 40.098 40.457 41780.495 35.157
Prometheus current 22.397 0.025 12.811 21.853 22.047 22.332 3598.517 21.288
Prometheus past 24.383 0.029 14.841 23.868 24.074 24.260 5892.105 23.135