Performance Benchmarks

On this page, we collect performance benchmarks of Kieker monitoring and other instrumentation frameworks, such as OpenTelemetry and inspectIT, for comparison utilizing MooBench.

Details of the comparison of OpenTelemetry, inspectIT and Kieker can be found in https://oceanrep.geomar.de/id/eprint/55367/1/SSP2021overhead.pdf

Kieker Observability Framework – Java

These are the performance benchmark results for Kieker 4 Java probes utilizing different writer back end.

  • No instrumentation serves as a baseline value indicating how much execution time can be contributed to the benchmark program itself.
  • Deactivated probe as above, but with woven in probes that do not produce any output and do not aggregate information.
  • No logging are activated probes, but the collected data is not send utilizing a null writer.
  • Text File utilizes the text file logging. This is usually slow, as data is written in text files and getting more irrelevant in the performance benchmarks. The diagram below, therefore, does not show the results.
  • Binary File utilizes the binary file logging without compression.
  • Binary TCP utilizes a TCP connection to send the monitoring data to a logging service.
setup run mean ci sd 1.quartile median 3.quartile max min
No instrumentation current 0.107 0.000 0.147 0.106 0.107 0.107 106.919 0.103
No instrumentation past 0.106 0.000 0.041 0.105 0.105 0.106 19.472 0.102
Deactivated probe current 0.707 0.007 3.597 0.675 0.693 0.710 2587.857 0.650
Deactivated probe past 0.705 0.008 4.191 0.671 0.687 0.707 2718.394 0.642
No logging current 2.750 0.010 5.261 2.698 2.720 2.748 2838.294 2.632
No logging past 2.681 0.010 5.186 2.634 2.660 2.690 2998.269 2.443
Binary file current 5.326 0.025 12.935 3.516 4.470 6.886 11432.730 3.325
Binary file past 4.896 0.034 17.436 3.436 4.309 5.357 14151.524 3.093
Binary TCP current 6.501 0.011 5.662 6.606 6.838 7.156 2536.661 3.242
Binary TCP past 7.087 0.019 9.512 6.667 6.970 7.487 7252.186 3.138

Kieker Observability Framework – Python

Kieker 4 Python is the reimplementation of the probe and monitoring side of Kieker addressing programs written in Python. Here we also want to know how well certain parts perform.

  • No instrumentation is the baseline execution time of the benchmark program.
  • Deactivated probe is the execution time with integrated probes, but these are deactivated.
  • No logging is with active probes, but with a null writer.
  • Text File is with an activated probe logging to a text file.
  • Binary TCP is with an activated probe logging to a remote logging server.

Each of these setups, except of no instrumentation, are tested utilizing two different probe introduction methods (here labeled A and B).

setup run mean ci sd 1.quartile median 3.quartile max min
No instrumentation current 1.531 0.000 0.150 1.510 1.521 1.531 22.866 1.462
No instrumentation past 1.535 0.000 0.205 1.497 1.507 1.550 100.733 1.445
Deactivated probe - A current 1.580 0.000 0.183 1.482 1.610 1.654 34.364 1.404
Deactivated probe - A past 1.523 0.000 0.174 1.473 1.515 1.545 32.996 1.417
Deactivated probe - B current 4.018 0.001 0.328 3.828 3.861 4.251 41.120 3.739
Deactivated probe - B past 4.040 0.001 0.341 3.853 3.890 4.246 76.900 3.761
No logging - A current 1.515 0.000 0.163 1.460 1.470 1.603 25.270 1.406
No logging - A past 1.532 0.000 0.190 1.491 1.505 1.550 98.424 1.433
No logging - B current 43.930 0.003 1.627 43.367 43.559 43.826 246.802 42.524
No logging - B past 44.372 0.004 1.954 43.493 43.683 44.096 239.409 42.590
Text file - A current 1.491 0.000 0.165 1.453 1.463 1.476 20.519 1.397
Text file - A past 1.555 0.000 0.180 1.498 1.528 1.605 35.124 1.442
Text file - B current 1310.840 1.186 605.147 1254.861 1278.755 1357.998 170139.200 1226.204
Text file - B past 1315.892 1.081 551.643 1257.599 1296.660 1361.481 195745.209 1228.428
Binary TCP - A current 1.517 0.000 0.168 1.503 1.511 1.519 71.987 1.459
Binary TCP - A past 1.543 0.000 0.173 1.476 1.516 1.626 39.626 1.428
Binary TCP - B current 472.348 0.061 31.236 463.301 466.029 471.209 3442.607 229.496
Binary TCP - B past 481.623 0.057 28.931 473.129 476.049 480.981 1788.948 233.559

OpenTelemetry

OpenTelemetry (https://opentelemetry.io/) is a widely used monitoring tool, compatible with many programming languages. MooBench measures the overhead of OpenTelemetry using the following configurations:

  • No instrumentation is the baseline execution time of the benchmark program.
  • No Logging measures the execution time of the benchmark program with OpenTelemetry instrumentation, but without measuring or
    logging anything.
  • Logging measures the execution time of the benchmark program with OpenTelemetry instrumentation and writing every output to the standard output. Text file logging is slow and getting more irrelevant in the performance benchmarks. The diagram below, therefore, does not show the results.
  • Zipkin measures the execution time of the benchmark program with obtaining every method invocation and sending them to a local Zipkin instance.
  • Prometheus measures the execution time of the benchmark program with obtaining only aggregated execution times, which are stored by a local Prometheus instance.
setup run mean ci sd 1.quartile median 3.quartile max min
No instrumentation current 0.108 0.000 0.150 0.106 0.107 0.108 147.029 0.103
No instrumentation past 0.105 0.000 0.099 0.105 0.105 0.105 88.995 0.103
No logging current 7.508 0.037 18.846 7.326 7.389 7.450 10742.890 7.132
No logging past 7.222 0.053 27.162 7.018 7.087 7.152 19627.000 6.839
Zipkin current 11.208 0.028 14.489 10.540 10.672 10.778 7336.043 9.224
Zipkin past 10.795 0.028 14.168 10.102 10.156 10.235 4648.460 9.778

InspectIT

InspectIT (https://www.inspectit.rocks/) is a widely used monitoring tool, which is allows to trace the execution of Java programs. MooBench
measures the overhead of InspectIT using the following configurations:

  • No instrumentation is the baseline execution time of the benchmark program.
  • Deactivated processing measures the execution time of the benchmark program with inspectIT instrumentation, but without measuring or logging
    anything.
  • No Logging measures the execution time of the benchmark program with inspectIT instrumentation and just discarding the measurement results.
  • Zipkin measures the execution time of the benchmark program with obtaining every method invocation and sending them to a local Zipkin instance.
  • Prometheus measures the execution time of the benchmark program with obtaining only aggregated execution times, which are stored by a local Prometheus instance.
setup run mean ci sd 1.quartile median 3.quartile max min
No instrumentation current 0.112 0.000 0.038 0.111 0.112 0.112 17.739 0.106
No instrumentation past 0.105 0.000 0.099 0.105 0.105 0.105 85.376 0.101
Deactivated probe current 12.757 0.053 27.041 12.461 12.545 12.642 24727.940 12.037
Deactivated probe past 13.486 0.026 13.477 13.193 13.274 13.375 6244.534 12.833
No logging current 39.279 0.032 16.434 38.345 38.532 38.783 3413.132 37.397
No logging past 40.683 0.044 22.480 39.604 39.841 40.173 12387.247 38.132
Zipkin current 43.890 0.052 26.409 41.823 42.043 42.371 9697.215 36.971
Zipkin past 43.620 0.091 46.410 41.211 41.476 41.844 31635.067 36.398
Prometheus current 24.832 0.108 55.268 24.272 24.399 24.566 46957.290 23.640
Prometheus past 23.852 0.037 18.928 23.361 23.560 23.725 12427.835 22.755