Performance Benchmarks

On this page, we collect performance benchmarks of Kieker monitoring and other instrumentation frameworks, such as OpenTelemetry and inspectIT, for comparison utilizing MooBench.

Details of the comparison of OpenTelemetry, inspectIT and Kieker can be found in https://oceanrep.geomar.de/id/eprint/55367/1/SSP2021overhead.pdf

Kieker Observability Framework – Java

These are the performance benchmark results for Kieker 4 Java probes utilizing different writer back end.

  • No instrumentation serves as a baseline value indicating how much execution time can be contributed to the benchmark program itself.
  • Deactivated probe as above, but with woven in probes that do not produce any output and do not aggregate information.
  • No logging are activated probes, but the collected data is not send utilizing a null writer.
  • Text File utilizes the text file logging. This is usually slow, as data is written in text files and getting more irrelevant in the performance benchmarks. The diagram below, therefore, does not show the results.
  • Binary File utilizes the binary file logging without compression.
  • Binary TCP utilizes a TCP connection to send the monitoring data to a logging service.
setup run mean ci sd 1.quartile median 3.quartile max min
No instrumentation current 0.072 0.000 0.019 0.072 0.072 0.073 10.934 0.068
No instrumentation past 0.122 0.000 0.070 0.120 0.121 0.123 51.108 0.113
Deactivated probe current 0.097 0.000 0.069 0.094 0.096 0.098 43.094 0.085
Deactivated probe past 0.152 0.000 0.128 0.149 0.151 0.153 106.501 0.141
No collection current 1.133 0.020 10.181 1.090 1.095 1.101 9679.364 1.066
No collection past 1.763 0.016 8.017 1.699 1.706 1.720 5526.910 1.669
Binary file current 4.168 0.013 6.569 2.467 4.195 4.964 5426.226 1.379
Binary file past 4.033 0.013 6.463 2.279 4.064 5.157 3835.969 2.077
Binary TCP current 5.049 0.016 8.160 3.469 4.490 6.838 6930.516 1.309
Binary TCP past 5.248 0.023 11.548 4.753 5.731 5.917 8374.193 2.381

Kieker Observability Framework – Python

Kieker 4 Python is the reimplementation of the probe and monitoring side of Kieker addressing programs written in Python. Here we also want to know how well certain parts perform.

  • No instrumentation is the baseline execution time of the benchmark program.
  • Deactivated probe is the execution time with integrated probes, but these are deactivated.
  • No logging is with active probes, but with a null writer.
  • Text File is with an activated probe logging to a text file.
  • Binary TCP is with an activated probe logging to a remote logging server.

Each of these setups, except of no instrumentation, are tested utilizing two different probe introduction methods (here labeled A and B).

setup run mean ci sd 1.quartile median 3.quartile max min
No instrumentation current 0.885 0.000 0.147 0.874 0.880 0.890 107.284 0.848
No instrumentation past 1.446 0.001 0.273 1.419 1.432 1.450 90.546 1.378
Deactivated probe - A current 2.155 0.000 0.229 2.140 2.150 2.161 181.473 2.082
Deactivated probe - A past 1.471 0.001 0.628 1.443 1.458 1.483 469.009 1.394
Deactivated probe - B current 2.202 0.003 1.454 2.178 2.193 2.208 1436.124 2.117
Deactivated probe - B past 1.449 0.001 0.257 1.423 1.437 1.460 74.059 1.375
No logging - A current 63.537 0.007 3.819 62.806 63.132 63.531 2074.985 61.347
No logging - A past 1.468 0.001 0.312 1.441 1.456 1.478 139.573 1.400
No logging - B current 17.782 0.003 1.287 17.680 17.725 17.775 731.601 17.434
No logging - B past 1.457 0.001 0.479 1.432 1.444 1.465 358.619 1.391
Text file - A current 404.678 0.050 25.514 399.560 402.670 407.270 22813.810 388.317
Text file - A past 1.476 0.001 0.294 1.436 1.452 1.476 82.618 1.396
Text file - B current 316.105 0.090 45.819 311.916 314.113 317.369 39001.740 304.830
Text file - B past 1.477 0.001 0.382 1.443 1.463 1.486 220.211 1.390
Binary TCP - A current 220.760 0.042 21.204 217.187 218.598 220.836 18693.720 211.046
Binary TCP - A past 1.458 0.000 0.252 1.432 1.445 1.465 64.501 1.382
Binary TCP - B current 120.465 0.040 20.581 116.365 120.257 122.075 18969.670 112.179
Binary TCP - B past 1.445 0.001 0.657 1.420 1.431 1.448 491.892 1.381

OpenTelemetry

OpenTelemetry (https://opentelemetry.io/) is a widely used monitoring tool, compatible with many programming languages. MooBench measures the overhead of OpenTelemetry using the following configurations:

  • No instrumentation is the baseline execution time of the benchmark program.
  • No Logging measures the execution time of the benchmark program with OpenTelemetry instrumentation, but without measuring or
    logging anything.
  • Logging measures the execution time of the benchmark program with OpenTelemetry instrumentation and writing every output to the standard output. Text file logging is slow and getting more irrelevant in the performance benchmarks. The diagram below, therefore, does not show the results.
  • Zipkin measures the execution time of the benchmark program with obtaining every method invocation and sending them to a local Zipkin instance.
  • Prometheus measures the execution time of the benchmark program with obtaining only aggregated execution times, which are stored by a local Prometheus instance.
setup run mean ci sd 1.quartile median 3.quartile max min
No instrumentation current 0.073 0.000 0.033 0.072 0.072 0.073 27.850 0.068
No instrumentation past 0.120 0.000 0.084 0.118 0.120 0.120 64.278 0.112
No logging current 4.039 0.017 8.488 3.886 3.926 4.022 2767.493 3.814
No logging past 7.141 0.019 9.829 6.940 6.996 7.059 3856.920 6.706
Zipkin current 7.029 0.025 12.874 6.490 6.542 6.911 4160.827 6.358
Zipkin past 13.136 0.035 18.079 12.373 13.005 13.139 4725.142 10.838
Prometheus current 6.029 0.029 14.677 5.834 5.871 5.954 8532.591 5.667
Prometheus past 10.354 0.023 11.570 10.077 10.161 10.257 3825.873 9.721

InspectIT

InspectIT (https://www.inspectit.rocks/) is a widely used monitoring tool, which is allows to trace the execution of Java programs. MooBench
measures the overhead of InspectIT using the following configurations:

  • No instrumentation is the baseline execution time of the benchmark program.
  • Deactivated processing measures the execution time of the benchmark program with inspectIT instrumentation, but without measuring or logging
    anything.
  • No Logging measures the execution time of the benchmark program with inspectIT instrumentation and just discarding the measurement results.
  • Zipkin measures the execution time of the benchmark program with obtaining every method invocation and sending them to a local Zipkin instance.
  • Prometheus measures the execution time of the benchmark program with obtaining only aggregated execution times, which are stored by a local Prometheus instance.
setup run mean ci sd 1.quartile median 3.quartile max min
No instrumentation current 0.073 0.000 0.029 0.072 0.072 0.073 15.966 0.069
No instrumentation past 0.121 0.000 0.067 0.119 0.120 0.121 42.933 0.114
Deactivated probe current 2.432 0.045 23.009 2.259 2.276 2.322 11753.220 2.203
Deactivated probe past 3.517 0.047 23.963 3.352 3.390 3.437 12094.881 3.230
No logging current 0.073 0.000 0.026 0.072 0.072 0.073 15.914 0.068
No logging past 25.205 0.030 15.083 24.497 24.709 24.968 5066.379 23.536
Zipkin current 0.073 0.000 0.058 0.071 0.072 0.073 16.332 0.068
Zipkin past 29.349 0.040 20.637 28.257 28.656 29.441 5718.039 27.010
Prometheus current 5.331 0.040 20.571 5.136 5.215 5.316 11590.640 4.998
Prometheus past 8.857 0.038 19.580 8.555 8.729 8.833 10652.159 8.278