Performance Benchmarks

On this page, we collect performance benchmarks of Kieker monitoring and other instrumentation frameworks, such as OpenTelemetry and inspectIT, for comparison utilizing MooBench.

Details of the comparison of OpenTelemetry, inspectIT and Kieker can be found in https://oceanrep.geomar.de/id/eprint/55367/1/SSP2021overhead.pdf

Kieker Observability Framework – Java

These are the performance benchmark results for Kieker 4 Java probes utilizing different writer back end.

  • No instrumentation serves as a baseline value indicating how much execution time can be contributed to the benchmark program itself.
  • Deactivated probe as above, but with woven in probes that do not produce any output and do not aggregate information.
  • No logging are activated probes, but the collected data is not send utilizing a null writer.
  • Text File utilizes the text file logging. This is usually slow, as data is written in text files and getting more irrelevant in the performance benchmarks. The diagram below, therefore, does not show the results.
  • Binary File utilizes the binary file logging without compression.
  • Binary TCP utilizes a TCP connection to send the monitoring data to a logging service.
setup run mean ci sd 1.quartile median 3.quartile max min
No instrumentation current 0.105 0.000 0.032 0.105 0.105 0.105 9.091 0.103
No instrumentation past 0.111 0.000 0.030 0.110 0.111 0.111 10.309 0.105
Deactivated probe current 0.143 0.000 0.039 0.142 0.143 0.143 24.608 0.139
Deactivated probe past 0.705 0.007 3.530 0.673 0.689 0.707 2549.518 0.642
No logging current 1.896 0.006 3.259 1.847 1.858 1.878 441.046 1.826
No logging past 2.683 0.008 3.967 2.640 2.666 2.695 2494.489 2.435
Binary file current 3.952 0.023 11.551 2.562 3.663 5.121 11083.390 2.212
Binary file past 5.073 0.029 14.677 3.510 4.475 5.990 13375.797 3.201
Binary TCP current 4.421 0.049 25.171 4.311 4.604 4.821 24668.610 2.069
Binary TCP past 6.871 0.024 12.055 6.554 6.907 7.488 9812.935 3.102

Kieker Observability Framework – Python

Kieker 4 Python is the reimplementation of the probe and monitoring side of Kieker addressing programs written in Python. Here we also want to know how well certain parts perform.

  • No instrumentation is the baseline execution time of the benchmark program.
  • Deactivated probe is the execution time with integrated probes, but these are deactivated.
  • No logging is with active probes, but with a null writer.
  • Text File is with an activated probe logging to a text file.
  • Binary TCP is with an activated probe logging to a remote logging server.

Each of these setups, except of no instrumentation, are tested utilizing two different probe introduction methods (here labeled A and B).

setup run mean ci sd 1.quartile median 3.quartile max min
No instrumentation current 1.455 0.001 0.650 1.430 1.441 1.460 591.857 1.393
No instrumentation past 1.516 0.001 0.518 1.488 1.501 1.521 401.135 1.437
Deactivated probe - A current 1.508 0.001 0.702 1.477 1.490 1.506 639.673 1.434
Deactivated probe - A past 1.542 0.001 0.627 1.500 1.514 1.538 529.072 1.449
Deactivated probe - B current 1.471 0.001 0.598 1.447 1.457 1.471 543.632 1.408
Deactivated probe - B past 1.566 0.001 0.505 1.540 1.552 1.577 392.527 1.470
No logging - A current 1.494 0.001 0.660 1.463 1.476 1.494 613.687 1.419
No logging - A past 1.534 0.002 0.805 1.503 1.517 1.535 618.556 1.444
No logging - B current 1.487 0.001 0.594 1.454 1.464 1.482 518.172 1.408
No logging - B past 1.875 0.001 0.615 1.843 1.858 1.881 531.196 1.789
Text file - A current 1.466 0.002 0.874 1.437 1.453 1.468 631.413 1.398
Text file - A past 1.544 0.001 0.567 1.511 1.526 1.554 384.912 1.460
Text file - B current 1.470 0.001 0.761 1.438 1.448 1.468 712.799 1.385
Text file - B past 12.007 0.010 5.209 11.535 11.772 12.375 2425.703 11.280
Binary TCP - A current 1.490 0.001 0.573 1.465 1.475 1.492 523.859 1.428
Binary TCP - A past 1.553 0.001 0.329 1.523 1.539 1.557 148.559 1.468
Binary TCP - B current 1.498 0.001 0.526 1.471 1.484 1.508 449.693 1.427
Binary TCP - B past 5.348 0.002 0.787 5.230 5.270 5.351 338.664 3.285

OpenTelemetry

OpenTelemetry (https://opentelemetry.io/) is a widely used monitoring tool, compatible with many programming languages. MooBench measures the overhead of OpenTelemetry using the following configurations:

  • No instrumentation is the baseline execution time of the benchmark program.
  • No Logging measures the execution time of the benchmark program with OpenTelemetry instrumentation, but without measuring or
    logging anything.
  • Logging measures the execution time of the benchmark program with OpenTelemetry instrumentation and writing every output to the standard output. Text file logging is slow and getting more irrelevant in the performance benchmarks. The diagram below, therefore, does not show the results.
  • Zipkin measures the execution time of the benchmark program with obtaining every method invocation and sending them to a local Zipkin instance.
  • Prometheus measures the execution time of the benchmark program with obtaining only aggregated execution times, which are stored by a local Prometheus instance.
setup run mean ci sd 1.quartile median 3.quartile max min
No instrumentation current 0.108 0.000 0.048 0.106 0.107 0.108 23.296 0.101
No instrumentation past 0.108 0.000 0.072 0.106 0.107 0.109 52.022 0.101
No logging current 7.310 0.022 11.134 7.066 7.129 7.202 4113.536 6.822
No logging past 7.438 0.021 10.708 7.224 7.283 7.352 3727.643 6.982
Zipkin current 11.570 0.034 17.117 10.966 11.069 11.221 4927.664 10.583
Zipkin past 13.034 0.041 21.098 12.523 12.622 12.742 9733.431 12.027
Prometheus current 13.391 0.027 13.663 13.088 13.182 13.287 7721.984 12.694
Prometheus past 13.612 0.024 12.089 13.281 13.382 13.486 3858.851 12.856

InspectIT

InspectIT (https://www.inspectit.rocks/) is a widely used monitoring tool, which is allows to trace the execution of Java programs. MooBench
measures the overhead of InspectIT using the following configurations:

  • No instrumentation is the baseline execution time of the benchmark program.
  • Deactivated processing measures the execution time of the benchmark program with inspectIT instrumentation, but without measuring or logging
    anything.
  • No Logging measures the execution time of the benchmark program with inspectIT instrumentation and just discarding the measurement results.
  • Zipkin measures the execution time of the benchmark program with obtaining every method invocation and sending them to a local Zipkin instance.
  • Prometheus measures the execution time of the benchmark program with obtaining only aggregated execution times, which are stored by a local Prometheus instance.
setup run mean ci sd 1.quartile median 3.quartile max min
No instrumentation current 0.109 0.000 0.065 0.107 0.108 0.109 49.328 0.101
No instrumentation past 0.108 0.000 0.076 0.106 0.107 0.109 63.728 0.101
Deactivated probe current 3.212 0.046 23.391 3.048 3.086 3.135 13261.190 2.930
Deactivated probe past 3.723 0.053 26.951 3.534 3.578 3.634 15525.226 3.383
No logging current 24.569 0.033 16.813 23.834 24.013 24.234 6360.235 22.764
No logging past 25.637 0.031 15.606 24.937 25.119 25.343 5423.900 23.993
Zipkin current 28.522 0.043 21.833 27.637 27.878 28.158 6353.672 26.406
Zipkin past 29.682 0.049 24.897 28.256 29.587 29.888 9309.609 26.310
Prometheus current 8.873 0.033 17.059 8.626 8.764 8.855 10346.480 8.226
Prometheus past 9.279 0.033 17.015 8.989 9.151 9.274 9763.072 8.673