New Tools and Stages for Kieker

The Kieker monitoring framework and toolset comprises different monitoring probes utilizing different probe introduction mechanisms for Java and other languages, a wide variety of analysis stages to receive, read and analyze monitoring data, and tools to support handling data and creating analyses based on monitoring data. In recent years we started to use the new pipe-and-filter framework TeeTime for the analysis and tool part which reduces the amount of configuration and implementation code necessary to create your own combination of analyses.

In the iObserve project, we heavily relied on TeeTime and the new Kieker stages (filter) implemented and ported to work with TeeTime. In this process, we created new tools and ported older Kieker tools to TeeTime. Furthermore, we updated the build process to create ready to use packages for these tools. Therefore, users of Kieker tooling do no longer need to download the complete Kieker package and extract the tools they need.

In this article, we will introduce the new stages and tools which originated in iObserve or where first used by iObserve, as the key user for the addition. As these additions are numerous, we will present only a brief overview in this article and add new posts to provide more details to certain new additions.

In case you are in a rush, all the new features can be found in the current Kieker snapshot and will be part of Kieker 1.15.

New Tool Packages

In the past Kieker produced one large distribution package containing all tools, libraries, probes, and setups. This had the advantage to present especially students with an easy way to get all what they need in one download. However, nowadays Kieker is used in more research and commercial projects and people want to have packages for specific tasks. To support this need, we now produce specific packages for tools which can be found here. As our efforts will progress, more tools will be available separately. Documentation to the tools is available on our wiki.

Kieker Tools Library

The tools in Kieker are highly configurable. Especially when it comes to reading, receiving or writing monitoring logs or other information. This requires that tools can adapt their pipe-and-filter configuration at startup time. As this can be cumbersome to maintain this adaptive feature in an ever growing number of tools, we collected such functionality in the Kieker tool library, reducing the implementation cost. Therefore, most tools can receive data from any reader implemented in Kieker, e.g., binary TCP (compressed or uncompressed), via REST API and from files in different formats.

Ongoing Changes to the Build System

The Kieker build system was based on an growing ant file in the past. We switched to gradle in 2015. However, many special constructs from the ant file had to be ported to gradle and do not fit typical modern Java gradle project layouts and workflows. Therefore, we slowly migrate to a more canonical layout for the Kieker project. One effect of this migration are the aforementioned tool packages. Furthermore, these changes will enable us to create releases faster and provide better support for releases in the long run.

Documentation

Kieker comes with a tutorial and usage guide in PDF. While this was a practical approach in the past, it is not fitting for today. Therefore, we are moving the Kieker documentation to our wiki pages. In the process, we also upgrade the documentation to reflect the current state of Kieker. The Kieker wiki can be found here.

Further Plans

Our future plans aim to improve build quality, release cycle and delivery of packages. We will also collect new stages (filters) and tools from the iObserve project. Furthermore, we will expand and document existing features of Kieker to support C, Fortran, Python and Javascript with monitoring probes capable to communicate with the collector and all tools using the same adaptive reader stages.

We already support C and at least two ways to instrument C and C++ applications one utilizing Aspect C++ and one utilizing features of the GNU Compiler Collection which supports instrumentation for all gcc supported languages. We have also an old version for Perl and other platforms and languages, which will be made available eventually.

Switcheroo: Changing the Kieker Configuration at Runtime

An article by Nils Ehmke (nils AT rhocas.de)

I think the original idea from Kieker was quite simple: put some monitoring probes into your application, add a properties file, and start the monitored application. After some time (or maybe after the application was stopped), you would collect the data and analyze it. Although this might be enough for some applications, this is not flexible enough for an enterprise business application. Such applications often have a different lifecycle compared to desktop applications, as they can not simply be restarted at any given time. In such scenarios one should be able to enable/disable the monitoring and even change the configuration, without the need to start and stop the whole application. Unfortunately, it is not possible to change the configuration of a started monitoring controller in Kieker. And the monitoring controller follows the singleton pattern, so we can’t do anything … or can we?

Continue reading

Kieker Trace Diagnosis 3.0.0 Released

We are pleased to anounce, that we released Version 3.0.0 of our Kieker Trace Diagnosis tool on January 23, 2018. As usual, you can download the release from https://github.com/kieker-monitoring/kieker-trace-diagnosis-ui/releases.

Some distinct and selected features:

  • Kieker Trace Diagnosis uses and requires now Java 9.
  • The monitoring log import has been further accelerated.
  • The tool supports now the writing and reading of ASCII monitoring logs.

We also fixed some minor bugs. You can find the full change log in the release archive.

Kieker Trace Diagnosis UI Release

An article by Nils Ehmke

While the Kieker Monitoring Framework is very suitable for collecting huge amount of monitoring data of your application, the question arises, how these monitoring logs can be analyzed. The tools shipped with Kieker (like the trace analysis tool) can help you only partially, because most of them build upon graph representations which tend to become huge, complex, and incomprehensible for larger applications.

To support the performance analysis of your application, we developed the Kieker Trace Diagnosis tool including a graphical user interface, which we want to introduce in this blog post.

Continue reading

Kieker’s Development in Five Minutes – 1.10

Using the tool Gource, we created a nice visualization showing the development of Kieker in only five minutes until the release of 1.10. The video contains parts of our Git history and shows not only how the project has grown since 2008, but also how many different developers already contributed to the project.

Thanks to Fenil Conic (@fenilconic) for the music (Hollywood Reality 3).

Monitoring of Remote Procedure Calls

In this blog post, the participants of the master’s project on “Distributed Monitoring”, which has been conducted at Kiel University in winter 2013/2014, report about their results:

Our task has been the development and implementation of concepts for monitoring remote procedure calls in distributed software systems, employing selected Java EE technologies. PubFlow, a distributed software system for scientific workflow automation, has served for the evaluation of the developed approach. We have used and extended a high-throughput tuned version of Kieker. ExplorViz has been used for live trace processing and visualization.

Continue reading

Kieker’s Development in Five Minutes

Using the tool Gource (http://code.google.com/p/gource/), we created a nice visualization showing the development of Kieker in only five minutes. The video contains parts of our Git history and shows not only how the project has grown since 2008, but also how many different developers already contributed to the project.

Thanks to Fenil Conic (@fenilconic) for the music (Hollywood Reality 3).