An article by Nils Ehmke (nils AT rhocas.de)
I think the original idea from Kieker was quite simple: put some monitoring probes into your application, add a properties file, and start the monitored application. After some time (or maybe after the application was stopped), you would collect the data and analyze it. Although this might be enough for some applications, this is not flexible enough for an enterprise business application. Such applications often have a different lifecycle compared to desktop applications, as they can not simply be restarted at any given time. In such scenarios one should be able to enable/disable the monitoring and even change the configuration, without the need to start and stop the whole application. Unfortunately, it is not possible to change the configuration of a started monitoring controller in Kieker. And the monitoring controller follows the singleton pattern, so we can’t do anything … or can we?
Actually, it is indeed possible to change the configuration and even have multiple monitoring controllers running simultaneously. Depending on your probes, it might require some mental acrobatics. In this blog post I focus on the first part and target to change the configuration of the monitoring at runtime by exchanging the monitoring controller. I will not detail the second part with simultaneously running monitoring controllers, as this requires some additional steps.
I implemented the concept of this blog post in the earlier introduced Kieker Trace Diagnosis. Feel free to take a look at it.
The main issue which we encounter with multiple controllers is, that most of the probes, which are shipped with Kieker, are using the singleton instance of the controller. This cannot be changed, as this is hard-coded in those probes.
Fortunately, we can implement our own probes and interceptors (which, depending on your architecture, might be a good idea anyway). But I will explain this later. Let’s start with storing our current monitoring controller instance somewhere. Although in a CDI (Context und Dependency Injection) context a singleton-scoped bean might be a good place for this, I prefer using a simple holder class which follows itself the singleton pattern. This is necessary, as in some CDI frameworks, the interceptors or proxies cannot access the CDI context themselves. Even more importantly, you might encounter situations in which you want to add probes to classes outside of your CDI context.
Now that we can store our own monitoring controller (or controllers), let us take a look at changing the configuration of a running monitoring controller instance. Most of the sub controllers within the monitoring controller store the configuration in final fields. We have no chance to change this configuration. So instead of changing it, we simply terminate the old monitoring and start a new one. This will lead to a new monitoring log on the file system, but should be fine for most scenarios.
As you can see, we clear the old monitoring controller instance and terminate it in an ordered fashion. That means, that the probes might have no controller instance for some time and have to be able to handle this. While we are at it, we terminate the singleton instance of Kieker, if it has been started.
Now that the old monitoring has been terminated, we can simply create a new one by using the static create method of the monitoring controller class. In this case I perform this programmatically based on the given DTO (Data Transfer Object), as Kieker Trace Diagnosis has a GUI view in which the user can configure the monitoring.
Here is an important hint: Make sure that you never, ever set the USE_SHUTDOWN_HOOK property to true. This property makes sure that Kieker registers a shutdown hook to terminate the monitoring in an ordered fashion. Although this is usually a good idea, the shutdown hook will act as a GC root in the JVM, which means that the old monitoring controller cannot be removed by the GC. If the monitoring controller is not removed, the writer controller cannot be removed by the GC either. This leads to a (potentially) huge writer queue, that cannot be garbage-collected. You will have a significant memory leak in your application, after you restarted your monitoring multiple times.
Instead, you should implement something like a shutdown hook yourself and only terminate the current monitoring controller in your MonitoringControllerHolder.
Now how does the probe look like? It is actually straight forward, as it retrieves the current monitoring controller when being instantiated. It just has to check whether the controller is actually available. If the controller is already terminated and we still try to send something to it, the fired records are simply discarded.
As Kieker Trace Diagnosis uses Guice as CDI container, I simply implemented a method interceptor which would be wrapped around some of my beans. In your application this will likely be around service or controller methods.
The interceptor follows the simple (and usual) pattern for the probes. The actual method invocation is put around a try-catch-block. Before we start, we create the probe (which fires the BeforeOperationEvent). If the method invocation fails, we inform the probe about the failure. The stop method at the end fires, depending on whether an exception occurred or not, an AfterOperationEvent or an AfterOperationFailedEvent.
So what are the downsides of this approach?
- We still have a singleton monitoring instance at least being initialized. The access to the trace registry instantiates the singleton instance. We can reduce the impact of this instance by providing a minimalistic monitoring.properties file, which will make sure that the singleton instance does not require that much memory.
- Changing the controller of the probes can lead to incomplete or damaged traces. Make sure your analysis can handle such incomplete data (Kieker Trace Diagnosis can handle this, by the way).
- The probes hold a reference to the monitoring controller. This can lead to issues if you are having infinite loops in your application (for instance an asynchronous timer), as the controller instances cannot be garbage collected, even if they are terminated. We can simply counter this by using Java’s WeakReference class for the monitoring controller inside the probes. In this case the probes have to be able to handle the fact, that the controller instance can be null suddenly.
Summarized, this blog post showed how to change the configuration of Kieker at runtime by swapping the whole monitoring controller instance. As long as you use customized probes, this can be implemented with little effort. Note that this whole blog post demonstrated only how to change the configuration of the monitoring. Another step would be to change which probes are active at runtime, thus leading to a customized adaptive monitoring. Such a monitoring could even be narrowed down to a single client or user in your application. This, however, might be a topic for another blog post.