On this page, we collect performance benchmarks of Kieker monitoring and other instrumentation frameworks, such as OpenTelemetry and inspectIT, for comparison utilizing MooBench.
Details of the comparison of OpenTelemetry, inspectIT and Kieker can be found in https://oceanrep.geomar.de/id/eprint/55367/1/SSP2021overhead.pdf
Kieker Observability Framework – Java
These are the performance benchmark results for Kieker 4 Java probes utilizing different writer back end.
- No instrumentation serves as a baseline value indicating how much execution time can be contributed to the benchmark program itself.
- Deactivated probe as above, but with woven in probes that do not produce any output and do not aggregate information.
- No logging are activated probes, but the collected data is not send utilizing a null writer.
- Text File utilizes the text file logging. This is usually slow, as data is written in text files and getting more irrelevant in the performance benchmarks. The diagram below, therefore, does not show the results.
- Binary File utilizes the binary file logging without compression.
- Binary TCP utilizes a TCP connection to send the monitoring data to a logging service.
| setup | run | mean | ci | sd | 1.quartile | median | 3.quartile | max | min |
|---|---|---|---|---|---|---|---|---|---|
| No instrumentation | current | 0.106 | 0.000 | 0.028 | 0.103 | 0.104 | 0.108 | 11.703 | 0.103 |
| No instrumentation | past | 0.109 | 0.000 | 0.041 | 0.108 | 0.109 | 0.109 | 21.252 | 0.103 |
| Deactivated probe | current | 0.721 | 0.007 | 3.627 | 0.693 | 0.703 | 0.723 | 2741.081 | 0.653 |
| Deactivated probe | past | 0.711 | 0.007 | 3.704 | 0.679 | 0.694 | 0.713 | 2689.640 | 0.647 |
| No logging | current | 2.688 | 0.009 | 4.578 | 2.632 | 2.660 | 2.692 | 2773.794 | 2.569 |
| No logging | past | 2.749 | 0.025 | 12.930 | 2.684 | 2.710 | 2.738 | 11094.693 | 2.612 |
| Binary file | current | 5.320 | 0.027 | 13.813 | 3.551 | 4.509 | 6.766 | 12471.860 | 2.986 |
| Binary file | past | 5.085 | 0.043 | 21.867 | 3.507 | 4.345 | 6.018 | 20552.263 | 3.114 |
| Binary TCP | current | 6.753 | 0.012 | 6.028 | 6.551 | 6.789 | 7.423 | 2628.729 | 3.246 |
| Binary TCP | past | 7.160 | 0.018 | 9.032 | 6.589 | 7.002 | 7.748 | 6670.248 | 3.028 |
Kieker Observability Framework – Python
Kieker 4 Python is the reimplementation of the probe and monitoring side of Kieker addressing programs written in Python. Here we also want to know how well certain parts perform.
- No instrumentation is the baseline execution time of the benchmark program.
- Deactivated probe is the execution time with integrated probes, but these are deactivated.
- No logging is with active probes, but with a null writer.
- Text File is with an activated probe logging to a text file.
- Binary TCP is with an activated probe logging to a remote logging server.
Each of these setups, except of no instrumentation, are tested utilizing two different probe introduction methods (here labeled A and B).
| setup | run | mean | ci | sd | 1.quartile | median | 3.quartile | max | min |
|---|---|---|---|---|---|---|---|---|---|
| No instrumentation | current | 1.553 | 0.000 | 0.148 | 1.543 | 1.551 | 1.557 | 32.374 | 1.466 |
| No instrumentation | past | 1.638 | 0.000 | 0.170 | 1.580 | 1.625 | 1.712 | 25.647 | 1.472 |
| Deactivated probe - A | current | 1.544 | 0.000 | 0.241 | 1.530 | 1.538 | 1.546 | 185.860 | 1.470 |
| Deactivated probe - A | past | 1.684 | 0.000 | 0.186 | 1.646 | 1.665 | 1.731 | 58.116 | 1.496 |
| Deactivated probe - B | current | 3.837 | 0.001 | 0.300 | 3.806 | 3.824 | 3.844 | 179.221 | 3.704 |
| Deactivated probe - B | past | 3.934 | 0.001 | 0.293 | 3.816 | 3.862 | 4.046 | 57.429 | 3.702 |
| No logging - A | current | 1.551 | 0.000 | 0.153 | 1.539 | 1.549 | 1.557 | 24.037 | 1.476 |
| No logging - A | past | 2.960 | 0.000 | 0.228 | 2.849 | 2.943 | 3.022 | 38.925 | 2.742 |
| No logging - B | current | 45.010 | 0.004 | 1.827 | 44.194 | 44.467 | 44.838 | 231.526 | 43.199 |
| No logging - B | past | 45.881 | 0.004 | 2.134 | 44.914 | 45.150 | 45.552 | 479.667 | 43.885 |
| Text file - A | current | 1.554 | 0.000 | 0.164 | 1.509 | 1.518 | 1.534 | 24.582 | 1.438 |
| Text file - A | past | 12.789 | 0.010 | 5.222 | 12.255 | 12.523 | 13.231 | 1438.907 | 11.936 |
| Text file - B | current | 1324.650 | 1.083 | 552.708 | 1262.506 | 1309.014 | 1367.245 | 208107.800 | 1232.760 |
| Text file - B | past | 1314.638 | 1.227 | 625.843 | 1258.432 | 1287.586 | 1360.628 | 227982.995 | 1229.429 |
| Binary TCP - A | current | 1.636 | 0.000 | 0.177 | 1.525 | 1.683 | 1.692 | 25.941 | 1.473 |
| Binary TCP - A | past | 6.802 | 0.001 | 0.529 | 6.643 | 6.749 | 6.831 | 118.737 | 4.497 |
| Binary TCP - B | current | 481.077 | 0.061 | 30.971 | 472.512 | 475.316 | 480.049 | 1359.387 | 232.355 |
| Binary TCP - B | past | 482.065 | 0.057 | 29.322 | 473.374 | 476.245 | 481.465 | 1364.527 | 238.453 |
OpenTelemetry
OpenTelemetry (https://opentelemetry.io/) is a widely used monitoring tool, compatible with many programming languages. MooBench measures the overhead of OpenTelemetry using the following configurations:
- No instrumentation is the baseline execution time of the benchmark program.
- No Logging measures the execution time of the benchmark program with OpenTelemetry instrumentation, but without measuring or
logging anything. - Logging measures the execution time of the benchmark program with OpenTelemetry instrumentation and writing every output to the standard output. Text file logging is slow and getting more irrelevant in the performance benchmarks. The diagram below, therefore, does not show the results.
- Zipkin measures the execution time of the benchmark program with obtaining every method invocation and sending them to a local Zipkin instance.
- Prometheus measures the execution time of the benchmark program with obtaining only aggregated execution times, which are stored by a local Prometheus instance.
| setup | run | mean | ci | sd | 1.quartile | median | 3.quartile | max | min |
|---|---|---|---|---|---|---|---|---|---|
| No instrumentation | current | 0.105 | 0.000 | 0.038 | 0.105 | 0.105 | 0.105 | 14.148 | 0.102 |
| No instrumentation | past | 0.105 | 0.000 | 0.074 | 0.104 | 0.105 | 0.105 | 60.103 | 0.102 |
| No logging | current | 6.455 | 0.019 | 9.645 | 6.311 | 6.351 | 6.395 | 5462.770 | 6.106 |
| No logging | past | 6.889 | 0.042 | 21.548 | 6.704 | 6.747 | 6.821 | 10791.797 | 6.544 |
| Zipkin | current | 9.475 | 0.023 | 11.865 | 8.984 | 9.024 | 9.077 | 3264.397 | 8.807 |
| Zipkin | past | 10.447 | 0.050 | 25.366 | 9.854 | 9.911 | 9.984 | 19644.458 | 9.267 |
InspectIT
InspectIT (https://www.inspectit.rocks/) is a widely used monitoring tool, which is allows to trace the execution of Java programs. MooBench
measures the overhead of InspectIT using the following configurations:
- No instrumentation is the baseline execution time of the benchmark program.
- Deactivated processing measures the execution time of the benchmark program with inspectIT instrumentation, but without measuring or logging
anything. - No Logging measures the execution time of the benchmark program with inspectIT instrumentation and just discarding the measurement results.
- Zipkin measures the execution time of the benchmark program with obtaining every method invocation and sending them to a local Zipkin instance.
- Prometheus measures the execution time of the benchmark program with obtaining only aggregated execution times, which are stored by a local Prometheus instance.
| setup | run | mean | ci | sd | 1.quartile | median | 3.quartile | max | min |
|---|---|---|---|---|---|---|---|---|---|
| No instrumentation | current | 0.104 | 0.000 | 0.033 | 0.103 | 0.103 | 0.104 | 10.885 | 0.101 |
| No instrumentation | past | 0.107 | 0.000 | 0.032 | 0.106 | 0.106 | 0.107 | 12.644 | 0.103 |
| Deactivated probe | current | 13.864 | 0.097 | 49.369 | 13.499 | 13.584 | 13.684 | 30737.700 | 12.941 |
| Deactivated probe | past | 14.052 | 0.037 | 18.704 | 13.685 | 13.791 | 13.938 | 13069.672 | 13.169 |
| No logging | current | 42.139 | 0.033 | 17.022 | 41.204 | 41.419 | 41.729 | 3398.960 | 40.339 |
| No logging | past | 41.167 | 0.045 | 23.003 | 40.116 | 40.349 | 40.686 | 12201.946 | 38.986 |
| Zipkin | current | 43.642 | 0.140 | 71.580 | 41.330 | 41.549 | 41.872 | 53456.890 | 36.450 |
| Zipkin | past | 42.972 | 0.074 | 37.613 | 40.223 | 40.477 | 40.857 | 24827.684 | 35.554 |
| Prometheus | current | 25.457 | 0.026 | 13.508 | 24.942 | 25.057 | 25.197 | 5336.627 | 24.355 |
| Prometheus | past | 24.253 | 0.035 | 17.657 | 23.815 | 23.925 | 24.055 | 8481.151 | 23.260 |