We started the development of instrumenting Python last year and have developed monitoring probes for Python and two weaving approaches. They will be presented (hopefully) at the Symposium for Software Performance. However, the tooling is already available and can be found on GitHub. Currently, we are integrating new features, cleanup the code and write documentation for end users. All these artifacts will become available in the general Kieker documentation.
Kieker now provides monitoring probes for C, Fortran and any other language supported by the GNU Compiler Collection or the Intel compilers including ifort. The code is currently available on GitHub. In the near future, we will provide Debian/Ubuntu packages for the library and examples.
We developed in iObserve support to monitor user behavior, create user behavior models and cluster them. These features will now be integrated in the the Kieker main repository to ease their use and make them available for a larger audience.
The behavior model monitoring provides additional entry level probes that record calls from web browsers or other services interacting with the monitored service.
The Kieker monitoring framework and toolset comprises different monitoring probes utilizing different probe introduction mechanisms for Java and other languages, a wide variety of analysis stages to receive, read and analyze monitoring data, and tools to support handling data and creating analyses based on monitoring data. In recent years we started to use the new pipe-and-filter framework TeeTime for the analysis and tool part which reduces the amount of configuration and implementation code necessary to create your own combination of analyses.
In the iObserve project, we heavily relied on TeeTime and the new Kieker stages (filter) implemented and ported to work with TeeTime. In this process, we created new tools and ported older Kieker tools to TeeTime. Furthermore, we updated the build process to create ready to use packages for these tools. Therefore, users of Kieker tooling do no longer need to download the complete Kieker package and extract the tools they need.
In this article, we will introduce the new stages and tools which originated in iObserve or where first used by iObserve, as the key user for the addition. As these additions are numerous, we will present only a brief overview in this article and add new posts to provide more details to certain new additions.
In case you are in a rush, all the new features can be found in the current Kieker snapshot and will be part of Kieker 1.15.
New Tool Packages
In the past Kieker produced one large distribution package containing all tools, libraries, probes, and setups. This had the advantage to present especially students with an easy way to get all what they need in one download. However, nowadays Kieker is used in more research and commercial projects and people want to have packages for specific tasks. To support this need, we now produce specific packages for tools which can be found here. As our efforts will progress, more tools will be available separately. Documentation to the tools is available on our wiki.
- Trace Analysis Tool allows to analyze call traces through applications
- Convert Logging Timestamps converts timestamps into local time
- Log Replayer allows to replay logs to feed them into analysis services, like iObserve and ExplorViz.
- Collector – Kieker Data Bridge allows to receive monitoring data from different source including TCP/binary, HTTP/JSON, files etc. and stores them in a joint log file or repeats them to another node.
- Resource Monitor
- Trace Analysis — GUI a graphical UI to control the Trace Analysis Tool
- Dot-Pic File Converter Helper script to process output files from the Trace Analysis Tool
Kieker Tools Library
The tools in Kieker are highly configurable. Especially when it comes to reading, receiving or writing monitoring logs or other information. This requires that tools can adapt their pipe-and-filter configuration at startup time. As this can be cumbersome to maintain this adaptive feature in an ever growing number of tools, we collected such functionality in the Kieker tool library, reducing the implementation cost. Therefore, most tools can receive data from any reader implemented in Kieker, e.g., binary TCP (compressed or uncompressed), via REST API and from files in different formats.
Ongoing Changes to the Build System
The Kieker build system was based on an growing ant file in the past. We switched to gradle in 2015. However, many special constructs from the ant file had to be ported to gradle and do not fit typical modern Java gradle project layouts and workflows. Therefore, we slowly migrate to a more canonical layout for the Kieker project. One effect of this migration are the aforementioned tool packages. Furthermore, these changes will enable us to create releases faster and provide better support for releases in the long run.
Kieker comes with a tutorial and usage guide in PDF. While this was a practical approach in the past, it is not fitting for today. Therefore, we are moving the Kieker documentation to our wiki pages. In the process, we also upgrade the documentation to reflect the current state of Kieker. The Kieker wiki can be found here.
We already support C and at least two ways to instrument C and C++ applications one utilizing Aspect C++ and one utilizing features of the GNU Compiler Collection which supports instrumentation for all gcc supported languages. We have also an old version for Perl and other platforms and languages, which will be made available eventually.
In an effort to make performance measurements around Kieker more accessible, we provide now performance benchmarks for Kieker directly on our website. Currently we are evaluating plugins which suits our needs and can be integrated with our Jenkins build.