Performance Benchmarks

On this page, we collect performance benchmarks of Kieker monitoring and other instrumentation frameworks, such as OpenTelemetry and inspectIT, for comparison utilizing MooBench.

Details of the comparison of OpenTelemetry, inspectIT and Kieker can be found in https://oceanrep.geomar.de/id/eprint/55367/1/SSP2021overhead.pdf

Kieker Observability Framework – Java

These are the performance benchmark results for Kieker 4 Java probes utilizing different writer back end.

  • No instrumentation serves as a baseline value indicating how much execution time can be contributed to the benchmark program itself.
  • Deactivated probe as above, but with woven in probes that do not produce any output and do not aggregate information.
  • No logging are activated probes, but the collected data is not send utilizing a null writer.
  • Text File utilizes the text file logging. This is usually slow, as data is written in text files and getting more irrelevant in the performance benchmarks. The diagram below, therefore, does not show the results.
  • Binary File utilizes the binary file logging without compression.
  • Binary TCP utilizes a TCP connection to send the monitoring data to a logging service.
setup run mean ci sd 1.quartile median 3.quartile max min
No instrumentation current 0.104 0.000 0.027 0.103 0.103 0.104 8.356 0.101
No instrumentation past 0.108 0.000 0.035 0.108 0.108 0.109 15.808 0.104
Deactivated probe current 0.719 0.007 3.346 0.690 0.702 0.721 2401.473 0.648
Deactivated probe past 0.707 0.008 3.948 0.674 0.690 0.708 2583.494 0.645
No logging current 2.662 0.008 4.063 2.615 2.636 2.661 2360.562 2.560
No logging past 2.713 0.009 4.808 2.660 2.682 2.709 2529.696 2.586
Binary file current 4.861 0.017 8.716 3.453 4.382 4.738 6886.440 2.991
Binary file past 5.229 0.039 19.851 3.562 4.542 6.278 18323.842 3.131
Binary TCP current 7.114 0.065 32.937 6.640 7.078 7.964 32489.940 3.302
Binary TCP past 7.101 0.011 5.560 6.645 6.838 7.442 2524.646 3.029

Kieker Observability Framework – Python

Kieker 4 Python is the reimplementation of the probe and monitoring side of Kieker addressing programs written in Python. Here we also want to know how well certain parts perform.

  • No instrumentation is the baseline execution time of the benchmark program.
  • Deactivated probe is the execution time with integrated probes, but these are deactivated.
  • No logging is with active probes, but with a null writer.
  • Text File is with an activated probe logging to a text file.
  • Binary TCP is with an activated probe logging to a remote logging server.

Each of these setups, except of no instrumentation, are tested utilizing two different probe introduction methods (here labeled A and B).

setup run mean ci sd 1.quartile median 3.quartile max min
No instrumentation current 1.508 0.000 0.154 1.459 1.468 1.542 25.528 1.395
No instrumentation past 1.566 0.000 0.164 1.537 1.557 1.617 22.448 1.403
Deactivated probe - A current 1.495 0.000 0.175 1.463 1.477 1.511 39.459 1.402
Deactivated probe - A past 1.527 0.000 0.165 1.486 1.512 1.529 29.433 1.416
Deactivated probe - B current 4.045 0.001 0.348 3.848 3.876 4.273 30.556 3.767
Deactivated probe - B past 4.026 0.001 0.328 3.841 3.918 4.236 58.047 3.754
No logging - A current 1.520 0.000 0.161 1.488 1.495 1.510 24.307 1.432
No logging - A past 1.507 0.000 0.182 1.452 1.471 1.536 44.512 1.394
No logging - B current 43.813 0.002 1.141 43.512 43.646 43.805 98.497 42.867
No logging - B past 44.236 0.003 1.681 43.676 43.823 44.015 278.075 42.921
Text file - A current 1.539 0.000 0.170 1.494 1.504 1.533 36.496 1.412
Text file - A past 1.537 0.000 0.167 1.478 1.506 1.581 28.572 1.414
Text file - B current 1340.152 1.191 607.741 1283.664 1312.309 1386.862 267913.800 1257.969
Text file - B past 1331.584 1.137 580.200 1276.109 1301.577 1378.573 201163.577 1250.232
Binary TCP - A current 1.461 0.000 0.151 1.436 1.445 1.454 33.759 1.374
Binary TCP - A past 1.594 0.000 0.173 1.525 1.617 1.631 29.749 1.423
Binary TCP - B current 489.451 0.072 36.939 478.385 481.247 488.090 983.555 236.456
Binary TCP - B past 490.153 0.065 33.128 480.318 483.137 488.886 1290.135 237.281

OpenTelemetry

OpenTelemetry (https://opentelemetry.io/) is a widely used monitoring tool, compatible with many programming languages. MooBench measures the overhead of OpenTelemetry using the following configurations:

  • No instrumentation is the baseline execution time of the benchmark program.
  • No Logging measures the execution time of the benchmark program with OpenTelemetry instrumentation, but without measuring or
    logging anything.
  • Logging measures the execution time of the benchmark program with OpenTelemetry instrumentation and writing every output to the standard output. Text file logging is slow and getting more irrelevant in the performance benchmarks. The diagram below, therefore, does not show the results.
  • Zipkin measures the execution time of the benchmark program with obtaining every method invocation and sending them to a local Zipkin instance.
  • Prometheus measures the execution time of the benchmark program with obtaining only aggregated execution times, which are stored by a local Prometheus instance.
setup run mean ci sd 1.quartile median 3.quartile max min
No instrumentation current 0.106 0.000 0.030 0.105 0.105 0.108 9.886 0.103
No instrumentation past 0.108 0.000 0.045 0.106 0.108 0.108 25.129 0.103
No logging current 7.005 0.014 7.312 6.857 6.898 6.946 2180.263 6.715
No logging past 7.268 0.015 7.664 7.114 7.162 7.218 2525.313 6.933
Zipkin current 11.685 0.052 26.514 10.848 10.896 10.958 16368.360 9.637
Zipkin past 11.494 0.033 16.634 10.835 10.941 11.049 6536.085 10.214

InspectIT

InspectIT (https://www.inspectit.rocks/) is a widely used monitoring tool, which is allows to trace the execution of Java programs. MooBench
measures the overhead of InspectIT using the following configurations:

  • No instrumentation is the baseline execution time of the benchmark program.
  • Deactivated processing measures the execution time of the benchmark program with inspectIT instrumentation, but without measuring or logging
    anything.
  • No Logging measures the execution time of the benchmark program with inspectIT instrumentation and just discarding the measurement results.
  • Zipkin measures the execution time of the benchmark program with obtaining every method invocation and sending them to a local Zipkin instance.
  • Prometheus measures the execution time of the benchmark program with obtaining only aggregated execution times, which are stored by a local Prometheus instance.
setup run mean ci sd 1.quartile median 3.quartile max min
No instrumentation current 0.108 0.000 0.033 0.107 0.108 0.108 11.399 0.103
No instrumentation past 0.106 0.000 0.033 0.106 0.106 0.107 14.070 0.102
Deactivated probe current 14.989 0.022 11.345 14.709 14.804 14.897 4556.480 14.039
Deactivated probe past 14.645 0.023 11.766 14.327 14.415 14.528 5430.656 13.917
No logging current 40.494 0.029 14.604 39.581 39.780 40.060 2834.434 35.172
No logging past 38.852 0.051 26.079 37.895 38.112 38.416 18211.863 36.056
Zipkin current 44.920 0.046 23.592 42.297 42.603 43.033 3812.345 37.215
Zipkin past 44.813 0.116 59.376 42.198 42.565 43.245 50571.411 37.082
Prometheus current 26.273 0.036 18.479 25.741 25.884 26.052 14133.810 25.251
Prometheus past 25.408 0.029 14.744 24.953 25.073 25.215 6226.515 24.175