HOBBIT provides benchmarks for storage and curation systems.
The Data Storage benchmark focuses on the typical challenges faced by the storage systems. It extends the Social Network Benchmark, developed in the context of the FP7 Linked Data Benchmarking Council (LDBC) by introducing important modifications in its synthetic data generator and dataset, and by modifying and transforming its SQL queries to SPARQL. This has been carried out while preserving the benchmark’s most relevant features:
- high insert rate with time-dependent and largely repetitive or cyclic data,
- exploitation of structure and physical organization adapted to the key dimensions of the data,
- bulk loading support,
- interactive complex read queries based on the well defined choke points,
- simple lookups,
- concurrency and
- high throughput.
The key performance indicators for the benchmarks include the time compression ratio (the ratio between real time and simulated time in which the system under test was able to answer queries in reasonable time), throughput (number of queries per second) and bulk loading time.
The Versioning Benchmark aims to test the ability of versioning systems to efficiently manage evolving Linked Data datasets and queries evaluated across multiple versions of these datasets. The benchmark is based on Linked Data Benchmark Council (LDBC) Semantic Publishing Benchmark (SPB). It leverages the scenario of the BBC media organization, which makes heavy use of Linked Data Technologies, such as RDF and SPARQL. The Benchmark is not tailored to any versioning strategy or system. It follows a choke point based design and extends the queries of SPB with features that stress the systems under test.