Linked Data technologies are proving their efficiency in creating interoperable IT systems that integrate disparate data sources. However, it is still unclear which solutions scale up the needs of modern business applications. Benchmarks have been created to support decisions on choosing the best software solution. However, they commonly fail to reflect business requirements in terms of (1) providing meaningful performance indicators or (2) being based on data that reflects the reality. Moreover, the results of benchmarks are rarely ever comparable, given that they are performed on heterogeneous hardware. All this goes without mentioning that the majority of the current benchmarks focus mostly on storage and query evaluation, leaving the preceding steps in the Linked Data life cycle (see figure) out although they are of central importance in the value generation chain.
The goals of Hobbit are:
- The development of benchmarks on data that reflects real use case scenarios within realistic settings.
- The collection and provision of corresponding industry-relevant key performance indicators (KPIs).
- The computation of comparable results on standardized hardware.
- The institution of an independent and thus bias-free organization to conduct regular benchmarks and provide the European industry with up-to-date performance results.
To achieve this we aim to work very closely with industrial partners. This includes partners that aim to adopt, are curious about or have already adopted Linked Data technologies as well as partners that develop solutions for the different steps of the linked data lifecycle. In an initial participation stage, we will collect Key Performance Indicators (KPI) that are relevant for the industry, to evaluate industry solutions that address a particular processing step of the Linked Data lifecycle.
By this close collaboration we will establish the benchmarking community where all participants can benefit from the reported results. This is why it is important to join the Hobbit community: your industry will take part in the benchmark definition and evaluation and you will obtain meaningful benchmark reports that will help in the process of evaluating technologies.
Follow this link for registering to the Hobbit community!