Performance Benchmarks

On this page, we collect performance benchmarks of Kieker monitoring and other instrumentation frameworks, such as OpenTelemetry and inspectIT, for comparison utilizing MooBench.

Details of the comparison of OpenTelemetry, inspectIT and Kieker can be found in https://oceanrep.geomar.de/id/eprint/55367/1/SSP2021overhead.pdf

Kieker Observability Framework – Java

These are the performance benchmark results for Kieker 4 Java probes utilizing different writer back end.

  • No instrumentation serves as a baseline value indicating how much execution time can be contributed to the benchmark program itself.
  • Deactivated probe as above, but with woven in probes that do not produce any output and do not aggregate information.
  • No logging are activated probes, but the collected data is not send utilizing a null writer.
  • Text File utilizes the text file logging. This is usually slow, as data is written in text files and getting more irrelevant in the performance benchmarks. The diagram below, therefore, does not show the results.
  • Binary File utilizes the binary file logging without compression.
  • Binary TCP utilizes a TCP connection to send the monitoring data to a logging service.
setup run mean ci sd 1.quartile median 3.quartile max min
No instrumentation current 0.073 0.000 0.075 0.072 0.072 0.073 167.181 0.068
No instrumentation past 0.073 0.000 0.282 0.072 0.072 0.073 797.185 0.068
Deactivated probe current 0.096 0.000 0.061 0.094 0.095 0.096 109.191 0.085
Deactivated probe past 0.096 0.000 0.085 0.094 0.095 0.097 163.013 0.084
No collection current 1.132 0.005 8.047 1.089 1.094 1.102 9997.704 1.051
No collection past 1.128 0.004 7.077 1.084 1.091 1.100 9562.559 1.051
Binary file current 3.504 0.002 3.541 1.802 3.230 4.079 3344.886 1.387
Binary file past 3.937 0.003 4.568 1.954 3.367 4.693 3739.461 1.378
Binary TCP current 3.881 0.005 8.359 2.249 3.549 4.540 7396.736 1.291
Binary TCP past 3.546 0.005 8.349 2.176 3.196 3.992 7976.331 1.300

Kieker Observability Framework – Python

Kieker 4 Python is the reimplementation of the probe and monitoring side of Kieker addressing programs written in Python. Here we also want to know how well certain parts perform.

  • No instrumentation is the baseline execution time of the benchmark program.
  • Deactivated probe is the execution time with integrated probes, but these are deactivated.
  • No logging is with active probes, but with a null writer.
  • Text File is with an activated probe logging to a text file.
  • Binary TCP is with an activated probe logging to a remote logging server.

Each of these setups, except of no instrumentation, are tested utilizing two different probe introduction methods (here labeled A and B).

setup run mean ci sd 1.quartile median 3.quartile max min
No instrumentation current 0.900 0.000 0.176 0.885 0.896 0.909 175.922 0.845
No instrumentation past 0.899 0.000 0.639 0.884 0.895 0.908 1343.375 0.844
Deactivated probe - A current 2.157 0.000 0.688 2.134 2.147 2.162 1408.417 2.067
Deactivated probe - A past 2.157 0.000 0.672 2.134 2.147 2.162 1441.068 2.068
Deactivated probe - B current 2.190 0.000 0.615 2.158 2.181 2.207 1416.469 2.086
Deactivated probe - B past 2.198 0.001 0.851 2.164 2.188 2.215 2090.389 2.086
No logging - A current 62.887 0.002 3.443 62.276 62.686 63.176 2305.668 60.169
No logging - A past 63.276 0.002 3.817 62.351 62.857 63.546 2365.622 59.937
No logging - B current 17.613 0.001 1.701 17.425 17.556 17.720 2114.641 16.877
No logging - B past 17.686 0.001 1.624 17.444 17.603 17.800 2244.705 16.956
Text file - A current 415.167 0.015 24.547 410.450 413.260 416.989 36193.860 398.367
Text file - A past 415.592 0.018 27.491 407.476 411.012 416.121 36555.409 393.679
Text file - B current 328.353 0.016 25.203 320.152 322.810 327.074 32705.800 310.923
Text file - B past 324.990 0.017 26.698 318.232 320.849 325.850 32383.676 309.118
Binary TCP - A current 217.090 0.006 9.378 210.852 215.383 222.207 2392.698 134.822
Binary TCP - A past 214.028 0.005 8.468 208.331 213.459 218.443 2168.505 135.278
Binary TCP - B current 151.330 0.008 13.064 141.315 149.689 161.286 1545.637 81.880
Binary TCP - B past 147.928 0.008 11.983 139.015 147.082 157.163 2571.563 81.587

OpenTelemetry

OpenTelemetry (https://opentelemetry.io/) is a widely used monitoring tool, compatible with many programming languages. MooBench measures the overhead of OpenTelemetry using the following configurations:

  • No instrumentation is the baseline execution time of the benchmark program.
  • No Logging measures the execution time of the benchmark program with OpenTelemetry instrumentation, but without measuring or
    logging anything.
  • Logging measures the execution time of the benchmark program with OpenTelemetry instrumentation and writing every output to the standard output. Text file logging is slow and getting more irrelevant in the performance benchmarks. The diagram below, therefore, does not show the results.
  • Zipkin measures the execution time of the benchmark program with obtaining every method invocation and sending them to a local Zipkin instance.
  • Prometheus measures the execution time of the benchmark program with obtaining only aggregated execution times, which are stored by a local Prometheus instance.
setup run mean ci sd 1.quartile median 3.quartile max min
No instrumentation current 0.073 0.000 0.050 0.072 0.072 0.073 120.134 0.068
No instrumentation past 0.073 0.000 0.171 0.072 0.072 0.073 494.619 0.068
No logging current 4.430 0.006 10.037 4.262 4.317 4.400 6592.031 4.040
No logging past 4.320 0.007 10.888 4.142 4.200 4.292 6292.272 3.930
Zipkin current 6.968 0.011 16.985 6.521 6.612 6.947 10917.210 6.305
Zipkin past 7.071 0.010 16.375 6.610 6.724 7.058 11847.885 6.341
Prometheus current 6.264 0.007 11.518 6.068 6.114 6.235 4390.117 5.884
Prometheus past 6.252 0.007 11.679 6.049 6.114 6.206 5768.995 5.874

InspectIT

InspectIT (https://www.inspectit.rocks/) is a widely used monitoring tool, which is allows to trace the execution of Java programs. MooBench
measures the overhead of InspectIT using the following configurations:

  • No instrumentation is the baseline execution time of the benchmark program.
  • Deactivated processing measures the execution time of the benchmark program with inspectIT instrumentation, but without measuring or logging
    anything.
  • No Logging measures the execution time of the benchmark program with inspectIT instrumentation and just discarding the measurement results.
  • Zipkin measures the execution time of the benchmark program with obtaining every method invocation and sending them to a local Zipkin instance.
  • Prometheus measures the execution time of the benchmark program with obtaining only aggregated execution times, which are stored by a local Prometheus instance.
setup run mean ci sd 1.quartile median 3.quartile max min
No instrumentation current 0.073 0.000 0.075 0.072 0.072 0.073 161.572 0.068
No instrumentation past 0.073 0.000 0.115 0.072 0.072 0.073 272.063 0.068
Deactivated probe current 2.336 0.014 23.333 2.313 2.481 2.542 16789.080 0.068
Deactivated probe past 2.286 0.015 23.546 1.560 2.473 2.523 13135.230 1.259
No logging current 11.038 0.009 14.475 11.530 11.772 12.353 8497.143 0.068
No logging past 10.879 0.009 14.680 11.128 12.026 12.372 6982.923 1.635
Zipkin current 10.708 0.013 20.500 11.266 11.460 11.947 10049.850 0.069
Zipkin past 11.822 0.014 22.245 11.329 11.526 12.085 12588.969 9.706
Prometheus current 3.915 0.011 18.193 0.073 5.295 5.514 14455.140 0.068
Prometheus past 4.553 0.013 20.485 3.150 5.340 5.555 14671.960 0.106