Performance Benchmarks

On this page, we collect performance benchmarks of Kieker monitoring and other instrumentation frameworks, such as OpenTelemetry and inspectIT, for comparison utilizing MooBench.

Details of the comparison of OpenTelemetry, inspectIT and Kieker can be found in https://oceanrep.geomar.de/id/eprint/55367/1/SSP2021overhead.pdf

Kieker Observability Framework – Java

These are the performance benchmark results for Kieker 4 Java probes utilizing different writer back end.

  • No instrumentation serves as a baseline value indicating how much execution time can be contributed to the benchmark program itself.
  • Deactivated probe as above, but with woven in probes that do not produce any output and do not aggregate information.
  • No logging are activated probes, but the collected data is not send utilizing a null writer.
  • Text File utilizes the text file logging. This is usually slow, as data is written in text files and getting more irrelevant in the performance benchmarks. The diagram below, therefore, does not show the results.
  • Binary File utilizes the binary file logging without compression.
  • Binary TCP utilizes a TCP connection to send the monitoring data to a logging service.
setup run mean ci sd 1.quartile median 3.quartile max min
No instrumentation current 0.107 0.000 0.055 0.105 0.106 0.107 31.092 0.103
No instrumentation past 0.106 0.000 0.102 0.105 0.106 0.106 91.734 0.102
Deactivated probe current 0.725 0.007 3.673 0.696 0.708 0.727 2664.491 0.660
Deactivated probe past 0.714 0.008 4.048 0.682 0.697 0.716 3105.840 0.650
No logging current 2.745 0.010 5.094 2.688 2.708 2.734 2660.429 2.623
No logging past 2.720 0.009 4.649 2.668 2.692 2.721 2709.916 2.592
Binary file current 4.936 0.011 5.719 3.485 4.400 4.863 2694.427 3.028
Binary file past 5.175 0.025 12.758 3.522 4.483 6.493 10472.592 3.153
Binary TCP current 7.321 0.012 6.100 6.685 7.068 7.985 2637.722 2.971
Binary TCP past 6.789 0.011 5.529 6.573 6.886 7.348 2768.185 3.117

Kieker Observability Framework – Python

Kieker 4 Python is the reimplementation of the probe and monitoring side of Kieker addressing programs written in Python. Here we also want to know how well certain parts perform.

  • No instrumentation is the baseline execution time of the benchmark program.
  • Deactivated probe is the execution time with integrated probes, but these are deactivated.
  • No logging is with active probes, but with a null writer.
  • Text File is with an activated probe logging to a text file.
  • Binary TCP is with an activated probe logging to a remote logging server.

Each of these setups, except of no instrumentation, are tested utilizing two different probe introduction methods (here labeled A and B).

setup run mean ci sd 1.quartile median 3.quartile max min
No instrumentation current 1.585 0.000 0.169 1.579 1.603 1.614 28.372 1.383
No instrumentation past 1.571 0.000 0.170 1.507 1.594 1.608 43.481 1.459
Deactivated probe - A current 1.469 0.000 0.146 1.455 1.462 1.469 15.206 1.399
Deactivated probe - A past 1.525 0.000 0.159 1.476 1.522 1.558 26.860 1.419
Deactivated probe - B current 3.939 0.001 0.320 3.768 3.801 4.180 38.236 3.680
Deactivated probe - B past 3.977 0.001 0.321 3.805 3.834 4.224 66.527 3.724
No logging - A current 1.503 0.000 0.150 1.487 1.497 1.505 29.374 1.426
No logging - A past 1.542 0.000 0.169 1.481 1.525 1.573 25.732 1.423
No logging - B current 44.397 0.002 1.141 44.104 44.225 44.367 97.648 43.473
No logging - B past 45.459 0.004 2.152 44.397 44.673 45.153 496.444 43.292
Text file - A current 1.507 0.000 0.156 1.477 1.486 1.497 29.904 1.421
Text file - A past 1.498 0.000 0.183 1.458 1.468 1.543 59.645 1.402
Text file - B current 1346.135 1.200 612.180 1286.496 1323.483 1391.757 215823.800 1255.848
Text file - B past 1319.893 1.045 533.087 1263.970 1291.273 1366.825 179851.790 1236.886
Binary TCP - A current 1.543 0.000 0.207 1.485 1.495 1.594 31.043 1.428
Binary TCP - A past 1.580 0.000 0.182 1.564 1.576 1.586 43.818 1.436
Binary TCP - B current 481.055 0.074 37.802 470.591 473.523 479.190 934.451 235.353
Binary TCP - B past 469.419 0.081 41.309 460.072 462.657 467.558 8870.373 234.632

OpenTelemetry

OpenTelemetry (https://opentelemetry.io/) is a widely used monitoring tool, compatible with many programming languages. MooBench measures the overhead of OpenTelemetry using the following configurations:

  • No instrumentation is the baseline execution time of the benchmark program.
  • No Logging measures the execution time of the benchmark program with OpenTelemetry instrumentation, but without measuring or
    logging anything.
  • Logging measures the execution time of the benchmark program with OpenTelemetry instrumentation and writing every output to the standard output. Text file logging is slow and getting more irrelevant in the performance benchmarks. The diagram below, therefore, does not show the results.
  • Zipkin measures the execution time of the benchmark program with obtaining every method invocation and sending them to a local Zipkin instance.
  • Prometheus measures the execution time of the benchmark program with obtaining only aggregated execution times, which are stored by a local Prometheus instance.
setup run mean ci sd 1.quartile median 3.quartile max min
No instrumentation current 0.105 0.000 0.037 0.105 0.105 0.105 22.218 0.102
No instrumentation past 0.109 0.000 0.039 0.108 0.109 0.110 18.054 0.104
No logging current 6.339 0.019 9.654 6.208 6.240 6.286 5357.575 6.037
No logging past 6.610 0.024 12.200 6.461 6.506 6.556 6793.559 6.207
Zipkin current 9.296 0.023 11.602 8.769 8.819 8.896 3427.817 8.606
Zipkin past 9.883 0.026 13.255 9.365 9.421 9.492 4835.613 8.592

InspectIT

InspectIT (https://www.inspectit.rocks/) is a widely used monitoring tool, which is allows to trace the execution of Java programs. MooBench
measures the overhead of InspectIT using the following configurations:

  • No instrumentation is the baseline execution time of the benchmark program.
  • Deactivated processing measures the execution time of the benchmark program with inspectIT instrumentation, but without measuring or logging
    anything.
  • No Logging measures the execution time of the benchmark program with inspectIT instrumentation and just discarding the measurement results.
  • Zipkin measures the execution time of the benchmark program with obtaining every method invocation and sending them to a local Zipkin instance.
  • Prometheus measures the execution time of the benchmark program with obtaining only aggregated execution times, which are stored by a local Prometheus instance.
setup run mean ci sd 1.quartile median 3.quartile max min
No instrumentation current 0.117 0.000 0.045 0.115 0.115 0.117 23.683 0.108
No instrumentation past 0.107 0.000 0.036 0.106 0.107 0.107 16.373 0.103
Deactivated probe current 13.018 0.021 10.779 12.745 12.834 12.944 3063.823 12.389
Deactivated probe past 13.882 0.022 11.391 13.611 13.712 13.818 4169.343 13.007
No logging current 41.235 0.030 15.422 40.502 40.677 40.903 3111.885 35.946
No logging past 40.010 0.110 55.920 38.945 39.199 39.582 51220.346 36.522
Zipkin current 42.805 0.046 23.310 40.959 41.234 41.571 4272.525 36.264
Zipkin past 44.154 0.058 29.582 41.941 42.185 42.534 12766.865 37.122
Prometheus current 26.847 0.025 12.734 26.331 26.494 26.706 3590.533 25.761
Prometheus past 25.013 0.037 18.710 24.513 24.635 24.781 12545.921 23.940