Performance Benchmarks

On this page, we collect performance benchmarks of Kieker monitoring and other instrumentation frameworks, such as OpenTelemetry and inspectIT, for comparison utilizing MooBench.

Details of the comparison of OpenTelemetry, inspectIT and Kieker can be
found in https://oceanrep.geomar.de/id/eprint/55367/1/SSP2021overhead.pdf

Kieker 4 Java – Monitoring

These are the performance benchmark results for Kieker 4 Java probes utilizing different writer back end.

• No instrumentation serves as a baseline value indicating how much execution time can be contributed to the benchmark program itself.
• Deactivated probe as above, but with woven in probes that do not produce any output and do not aggregate information.
• No logging are activated probes, but the collected data is not send utilizing a null writer.
• Text File utilizes the text file logging. This is usually slow, as data is written in text files.
• Binary File utilizes the binary file logging without compression.
• Binary TCP utilizes a TCP connection to send the monitoring data to a logging service.

setup	run	mean	ci	sd	1.quartile	median	3.quartile	min	max
No instrumentation	current	0.112	0.000	0.039	0.111	0.112	0.112	9.271	0.105
No instrumentation	past	0.112	0.000	0.046	0.109	0.109	0.110	9.545	0.104
Deactivated probe	current	1.260	0.007	1.141	0.573	0.595	2.557	17.528	0.543
Deactivated probe	past	1.259	0.007	1.125	0.576	0.602	2.537	19.815	0.544
No logging	current	19.274	0.016	2.543	17.561	19.057	20.661	45.809	3.786
No logging	past	20.381	0.017	2.703	18.628	20.293	21.991	93.916	4.890
Text file	current	239.509	12.406	2001.634	217.280	231.452	237.370	633047.600	3.646
Text file	past	240.550	15.758	2542.472	218.231	223.779	228.156	755687.935	3.916
Binary file	current	8.935	0.785	126.630	5.772	7.434	11.651	40034.250	3.033
Binary file	past	9.364	0.125	20.189	5.935	8.290	12.812	6155.748	3.386
Binary TCP	current	14.691	0.028	4.464	13.060	14.860	17.367	121.002	3.397
Binary TCP	past	12.415	0.023	3.765	10.438	13.066	14.258	104.358	3.496

Kieker 4 Python – Monitoring

Kieker 4 Python is the reimplementation of the probe and monitoring side of Kieker addressing programs written in Python. Here we also want to know how well certain parts perform.

• No instrumentation is the baseline execution time of the benchmark program.
• Deactivated probe is the execution time with integrated probes, but these are deactivated.
• No logging is with active probes, but with a null writer.
• Text File is with an activated probe logging to a text file.
• Binary TCP is with an activated probe logging to a remote logging server.

Each of these setups, except of no instrumentation, are tested utilizing two different probe introduction methods (here labeled A and B).

setup	run	mean	ci	sd	1.quartile	median	3.quartile	min	max
No instrumentation	current	4.567	0.002	0.276	4.529	4.548	4.567	22.489	4.468
No instrumentation	past	4.648	0.002	0.348	4.601	4.617	4.635	24.179	4.506
Deactivated probe - A	current	10.257	0.004	0.573	10.186	10.211	10.239	98.391	10.066
Deactivated probe - A	past	9.963	0.004	0.720	9.446	10.081	10.390	44.492	9.009
Deactivated probe - B	current	10.541	0.004	0.570	10.468	10.496	10.527	52.111	10.337
Deactivated probe - B	past	9.654	0.006	1.028	9.145	9.548	10.083	188.510	9.008
No logging - A	current	358.996	0.057	9.201	355.536	356.191	357.082	552.396	352.040
No logging - A	past	449.306	0.435	70.172	359.903	479.229	500.913	722.823	356.014
No logging - B	current	122.831	0.031	5.079	120.387	120.645	121.078	308.498	119.331
No logging - B	past	119.721	0.030	4.828	117.349	118.190	118.582	298.525	116.347
Text file - A	current	2218.756	6.333	1021.760	2102.023	2270.604	2287.972	288312.800	2075.391
Text file - A	past	2500.608	3.887	627.131	2176.650	2252.910	2881.993	70447.531	2066.657
Text file - B	current	1871.358	3.307	533.528	1782.482	1828.220	1938.245	104983.000	1754.401
Text file - B	past	2319.427	5.253	847.592	1878.407	2341.820	2709.386	128298.515	1764.704
Binary TCP - A	current	1130.763	0.156	25.198	1117.794	1120.736	1141.519	1835.200	808.696
Binary TCP - A	past	1119.039	0.322	52.027	1092.738	1099.917	1144.478	2594.154	863.729
Binary TCP - B	current	804.755	0.296	47.715	789.330	792.030	808.399	1751.592	523.591
Binary TCP - B	past	815.542	0.181	29.235	801.766	804.321	822.032	1704.844	584.738

OpenTelemetry

OpenTelemetry (https://opentelemetry.io/) is a widely used monitoring tool, compatible with many programming languages. MooBench measures the overhead of OpenTelemetry using the following configurations:

• No instrumentation is the baseline execution time of the benchmark program.
• No Logging measures the execution time of the benchmark program with OpenTelemetry instrumentation, but without measuring or
  logging anything.
• Logging measures the execution time of the benchmark program with OpenTelemetry instrumentation and writing every output to the standard output.
• Zipkin measures the execution time of the benchmark program with obtaining every method invocation and sending them to a local Zipkin instance.
• Prometheus measures the execution time of the benchmark program with obtaining only aggregated execution times, which are stored by a local Prometheus instance.

setup	run	mean	ci	sd	1.quartile	median	3.quartile	min	max
No instrumentation	current	0.109	0.000	0.040	0.105	0.105	0.107	9.376	0.103
No instrumentation	past	0.131	0.000	0.065	0.131	0.131	0.134	12.259	0.105
No logging	current	6.420	0.072	11.577	6.176	6.218	6.274	3656.509	6.019
No logging	past	6.385	0.082	13.245	6.099	6.309	6.372	4187.614	5.923
Logging	current	245.240	67.160	10835.810	147.628	149.473	161.600	1899318.000	142.813
Logging	past	234.404	80.104	12924.327	152.110	153.742	171.636	3030748.330	145.883
Zipkin	current	10.363	0.115	18.558	8.471	8.566	12.814	4228.631	8.251
Zipkin	past	10.236	0.121	19.455	9.162	9.277	9.457	4482.920	8.467
Prometheus	current	9.314	0.119	19.127	8.966	9.036	9.192	4372.340	8.764
Prometheus	past	9.850	0.116	18.748	9.053	9.141	10.605	4262.398	8.752

InspectIT

InspectIT (https://www.inspectit.rocks/) is a widely used monitoring tool, which is allows to trace the execution of Java programs. MooBench
measures the overhead of InspectIT using the following configurations:

• No instrumentation is the baseline execution time of the benchmark program.
• Deactivated processing measures the execution time of the benchmark program with inspectIT instrumentation, but without measuring or logging
  anything.
• No Logging measures the execution time of the benchmark program with inspectIT instrumentation and just discarding the measurement results.
• Zipkin measures the execution time of the benchmark program with obtaining every method invocation and sending them to a local Zipkin instance.
• Prometheus measures the execution time of the benchmark program with obtaining only aggregated execution times, which are stored by a local Prometheus instance.

setup	run	mean	ci	sd	1.quartile	median	3.quartile	min	max
No instrumentation	current	32.031	0.040	6.420	30.920	31.067	31.315	243.247	30.401
No instrumentation	past	40.085	0.056	9.048	33.746	42.341	42.742	379.025	31.292
Deactivated probe	current	41.482	0.049	7.911	40.158	40.408	40.673	389.929	35.874
Deactivated probe	past	43.712	0.159	25.590	40.021	40.874	43.897	7554.275	38.755
No logging	current	68.698	0.344	55.549	64.912	65.286	65.845	8925.844	63.506
No logging	past	74.616	0.346	55.778	69.735	70.113	70.839	8552.580	68.323
Zipkin	current	73.099	0.504	81.316	66.443	66.866	67.891	13022.170	59.766
Zipkin	past	77.031	0.551	88.888	68.497	69.474	72.406	12943.444	61.931
Prometheus	current	49.627	0.073	11.784	47.108	47.345	47.697	2596.750	46.193
Prometheus	past	48.620	0.069	11.194	46.537	47.019	50.131	2073.437	42.702