Performance Benchmarks

On this page, we collect performance benchmarks of Kieker monitoring and other instrumentation frameworks, such as OpenTelemetry and inspectIT, for comparison utilizing MooBench.

Details of the comparison of OpenTelemetry, inspectIT and Kieker can be found in https://oceanrep.geomar.de/id/eprint/55367/1/SSP2021overhead.pdf

Kieker Observability Framework – Java

These are the performance benchmark results for Kieker 4 Java probes utilizing different writer back end.

  • No instrumentation serves as a baseline value indicating how much execution time can be contributed to the benchmark program itself.
  • Deactivated probe as above, but with woven in probes that do not produce any output and do not aggregate information.
  • No logging are activated probes, but the collected data is not send utilizing a null writer.
  • Text File utilizes the text file logging. This is usually slow, as data is written in text files and getting more irrelevant in the performance benchmarks. The diagram below, therefore, does not show the results.
  • Binary File utilizes the binary file logging without compression.
  • Binary TCP utilizes a TCP connection to send the monitoring data to a logging service.
setup run mean ci sd 1.quartile median 3.quartile max min
No instrumentation current 0.120 0.000 0.069 0.117 0.119 0.121 25.212 0.111
No instrumentation past 0.118 0.000 0.041 0.116 0.117 0.118 18.862 0.109
Deactivated probe current 0.156 0.000 0.096 0.150 0.152 0.155 81.518 0.141
Deactivated probe past 0.149 0.001 0.511 0.147 0.148 0.149 487.059 0.139
No collection current 1.786 0.009 4.846 1.714 1.736 1.750 603.388 1.684
No collection past 1.726 0.010 5.211 1.671 1.679 1.692 2047.499 1.644
Binary file current 3.727 0.011 5.636 2.228 3.114 5.036 2400.387 2.069
Binary file past 4.503 0.010 5.322 2.414 4.456 5.762 2849.662 2.083
Binary TCP current 4.819 0.027 13.708 3.322 5.104 5.836 10570.730 2.412
Binary TCP past 4.960 0.018 9.208 3.427 5.286 5.929 6076.965 2.405

Kieker Observability Framework – Python

Kieker 4 Python is the reimplementation of the probe and monitoring side of Kieker addressing programs written in Python. Here we also want to know how well certain parts perform.

  • No instrumentation is the baseline execution time of the benchmark program.
  • Deactivated probe is the execution time with integrated probes, but these are deactivated.
  • No logging is with active probes, but with a null writer.
  • Text File is with an activated probe logging to a text file.
  • Binary TCP is with an activated probe logging to a remote logging server.

Each of these setups, except of no instrumentation, are tested utilizing two different probe introduction methods (here labeled A and B).

setup run mean ci sd 1.quartile median 3.quartile max min
No instrumentation current 1.444 0.001 0.306 1.414 1.424 1.443 109.630 1.373
No instrumentation past 1.441 0.001 0.364 1.417 1.427 1.446 212.097 1.376
Deactivated probe - A current 1.448 0.001 0.275 1.422 1.431 1.449 91.577 1.381
Deactivated probe - A past 1.451 0.002 1.049 1.427 1.436 1.455 921.910 1.386
Deactivated probe - B current 1.452 0.001 0.685 1.429 1.438 1.455 638.555 1.390
Deactivated probe - B past 1.438 0.001 0.266 1.416 1.426 1.441 73.934 1.381
No logging - A current 1.540 0.001 0.278 1.507 1.518 1.542 70.138 1.460
No logging - A past 1.450 0.000 0.243 1.424 1.439 1.461 78.016 1.377
No logging - B current 1.470 0.001 0.259 1.444 1.457 1.480 79.433 1.393
No logging - B past 1.448 0.001 0.271 1.424 1.434 1.453 79.189 1.381
Text file - A current 1.444 0.000 0.252 1.417 1.427 1.446 59.468 1.381
Text file - A past 1.442 0.001 0.385 1.419 1.429 1.445 237.783 1.381
Text file - B current 1.414 0.001 0.690 1.393 1.401 1.414 628.091 1.357
Text file - B past 1.441 0.001 0.272 1.410 1.420 1.439 76.980 1.373
Binary TCP - A current 1.444 0.000 0.249 1.424 1.433 1.445 60.362 1.391
Binary TCP - A past 1.461 0.001 0.298 1.434 1.445 1.462 85.911 1.391
Binary TCP - B current 1.416 0.001 0.257 1.397 1.405 1.419 60.540 1.363
Binary TCP - B past 1.452 0.001 0.271 1.432 1.442 1.456 83.262 1.387

OpenTelemetry

OpenTelemetry (https://opentelemetry.io/) is a widely used monitoring tool, compatible with many programming languages. MooBench measures the overhead of OpenTelemetry using the following configurations:

  • No instrumentation is the baseline execution time of the benchmark program.
  • No Logging measures the execution time of the benchmark program with OpenTelemetry instrumentation, but without measuring or
    logging anything.
  • Logging measures the execution time of the benchmark program with OpenTelemetry instrumentation and writing every output to the standard output. Text file logging is slow and getting more irrelevant in the performance benchmarks. The diagram below, therefore, does not show the results.
  • Zipkin measures the execution time of the benchmark program with obtaining every method invocation and sending them to a local Zipkin instance.
  • Prometheus measures the execution time of the benchmark program with obtaining only aggregated execution times, which are stored by a local Prometheus instance.
setup run mean ci sd 1.quartile median 3.quartile max min
No instrumentation current 0.120 0.000 0.099 0.117 0.119 0.120 83.830 0.113
No instrumentation past 0.131 0.001 0.490 0.128 0.129 0.131 464.916 0.118
No logging current 7.159 0.019 9.765 6.961 7.010 7.068 3843.304 6.782
No logging past 7.028 0.021 10.803 6.808 6.870 6.940 4515.671 6.569
Zipkin current 12.772 0.032 16.575 12.483 12.619 12.755 4998.224 10.083
Zipkin past 13.280 0.034 17.452 12.110 13.219 13.754 4980.691 11.524
Prometheus current 10.275 0.023 11.561 10.003 10.085 10.179 3277.453 9.661
Prometheus past 10.258 0.024 12.400 9.974 10.054 10.146 4252.205 9.617

InspectIT

InspectIT (https://www.inspectit.rocks/) is a widely used monitoring tool, which is allows to trace the execution of Java programs. MooBench
measures the overhead of InspectIT using the following configurations:

  • No instrumentation is the baseline execution time of the benchmark program.
  • Deactivated processing measures the execution time of the benchmark program with inspectIT instrumentation, but without measuring or logging
    anything.
  • No Logging measures the execution time of the benchmark program with inspectIT instrumentation and just discarding the measurement results.
  • Zipkin measures the execution time of the benchmark program with obtaining every method invocation and sending them to a local Zipkin instance.
  • Prometheus measures the execution time of the benchmark program with obtaining only aggregated execution times, which are stored by a local Prometheus instance.
setup run mean ci sd 1.quartile median 3.quartile max min
No instrumentation current 0.120 0.000 0.135 0.117 0.119 0.120 76.076 0.111
No instrumentation past 0.116 0.000 0.053 0.114 0.116 0.117 27.848 0.108
Deactivated probe current 3.536 0.035 17.701 3.390 3.434 3.495 9274.692 3.260
Deactivated probe past 3.710 0.048 24.433 3.489 3.529 3.581 12927.030 3.349
No logging current 25.714 0.029 14.698 25.097 25.272 25.482 3551.040 24.098
No logging past 24.667 0.027 13.908 23.993 24.215 24.477 3603.955 23.096
Zipkin current 30.850 0.046 23.297 30.433 30.687 30.945 7582.052 24.633
Zipkin past 29.020 0.049 25.070 28.074 28.352 28.605 10774.773 26.486
Prometheus current 8.781 0.036 18.113 8.540 8.677 8.768 9204.482 8.239
Prometheus past 9.014 0.044 22.410 8.744 8.854 8.946 11301.311 8.465