The HOBBIT evaluation platform is a distributed FAIR benchmarking platform for the Linked Data lifecycle. This means that the platform was designed to provide means to:
- benchmark any step of the linked data lifecycle, including generation and acquisition, analytics and processing, storage and curation as well as visualization and services,
- ensure that benchmarking results can be found, accessed, integrated and reused easily (FAIR principles),
- benchmark Big Data platforms by being the first distributed benchmarking platform for Linked data.
We offer an open source evaluation platform that can be downloaded and executed locally. Secondly, we offer an online instance of the platform for a) running public challenges and b) making sure that even people without the required infrastructure are able to run the benchmarks they are interested in.
The HOBBIT benchmarking platform ensures that:
- The benchmarks are easy to use.
- New benchmarks can be easily created and added to the platform by third parties.
- The evaluation can be scaled out to large datasets and on distributed architectures.
- The publishing and analysis of the results of different systems can be carried out in a uniform manner across the different benchmarks.
The first version of the platform has been released in February 2017. It offers the main features of an evaluation platform and has been further enhanced over time. The release of the second version is planned for February 2018. For this release we are mainly focussing on the usability of the platform as well as its support for additional features that can be used by the benchmark implementations (e.g., shared volumes or hardware statistics).
How to use the platform
The online instance of the HOBBIT benchmarking platform is accessible at master.project-hobbit.eu. More information about its features and how to use it can be found in our deliverable D2.2.1 and our platform wiki. The code of the platform and the benchmarks are accessible at https://github.com/hobbit-project. All benchmarks are also available on our CKAN: https://ckan.project-hobbit.eu/dataset, along with their source code and related publications.