We are happy to announce the release of the second version of the HOBBIT Benchmarking Platform. During the last months, we improved the stability of the platform as well as the set of features it supports. The full list of the features implemented in version 2 are listed below.
- Improved the usability of the platform for normal users.
- Added filtering options for list of experiments and challenges.
- Added log file download for system and benchmark developers.
- Better visualization of the platform status including information about the maximum runtime of the currently running experiment.
- Enabled the deletion of experiments from the queue by the user who created them.
- Implemented the visualization of result sets as diagrams.
- Improved the support of challenges.
- Added support for repeatable challenges.
- Added leaderboards for challenges.
- Added a field to add a link to a homepages to a challenge.
- Improved the usability of the platform for local deployments.
- Added an optional configuration file which can be used to define the maximum runtime of experiments based on the benchmark used in the experiment and whether it belongs to a challenge.
- Deactivation of docker pull commands for local development.
- Eased the deployment process of the platform.
- Offered a Java SDK to ease the development of benchmarks and system adapters.
- Added support of the forwarding of parameters from the benchmark to the system.
- Added support for gathering the amount of resources used by the benchmarked system.
- Added support for streaming data removing the limitation of single tasks or results being bound to a maximum size of 1GB.
In this blog post, we highlight selected features from the new release.
Visualization of result sets. We have implemented the visualization of result sets, which is now available in the HOBBIT platform user interface. The diagrams are generated automatically as long as data cube triples are available in the result set of an experiment. An example of such a diagram is shown in Figure 1, where the triples per second returned by a triple store are plotted.
Resource usage gathering. In the HOBBIT platform version 2.0 it is now possible to request statistics about resources used by a benchmarked system. Such statistics enable the comparison of the benchmarked systems in terms of the resources (CPU, memory) they consume during an experiment and not only in terms of the execution time. The implementation of this feature required additional setup of the cluster since the statistics are spread across the Docker Swarm cluster and are not make available in one place by Docker. We have decided to utilize Prometheus with cAdvisor and node-exporter for the statistics gathering. At the time of writing, Docker Swarm does not provide native facilities for such statistics.
Platform Deployment. In order to deploy the new version of the platform, several changes were made to the deployment setup:
- Migration to the latest stable Docker Swarm 17.12.
- Changing the reverse proxy setup from jwilder nginx to traefik.
- Addition of the Prometheus server, cAdvisor and node-exporters to the setup for statistics gathering.
The migration to the latest Docker Swarm 17.12 provides a number of features out of the box such as rolling upgrades and load balancing. However, Docker Swarm 17.12 and the old Docker Swarm we were using are not compatible and thus the major challenge for us was to migrate the Docker container instantiation to the latest Docker version in which Docker Swarm handles stacks of services. We had also changed the reverse proxy setup to traefik due to the fact that nginx does not play well with Docker Swarm. Traefik on the other hand is easy to setup and has integration with letsencrypt, which provides SSL certificates without much effort.
During the next weeks, we will further ease the setup of the platform and improve its documentation. If you have any questions regarding usage of the platform or problems with the technical setup, feel free to open an issue in our issue tracker on the Github.