Benchmark I – Generation & Acquisition

HOBBIT provides benchmarks to measure the performance of SPARQL query processing systems when faced with streams of data from industrial machinery in terms of efficiency and completeness.

Sensor Streams Benchmark

The constant growth of the Linked Data Web in velocity and volume has increased the need for triple stores to ingest streams of data and perform queries on this data efficiently. The aim of this benchmark is to measure the performance of SPARQL query processing systems when faced with streams of data from industrial machinery in terms of efficiency and completeness. Our goal is to measure the performance of triple stores by storing and retrieving RDF data, considering two main choke points:

  • Scalability (Data volume): Number of triples per stream and number of streams.
  • Time complexity (Data velocity): Number of triples per second.

We aim to reflect real loads on triple stores used in real applications. We hence use the following datasets:

We increased the size and velocity of RDF data used in our benchmarks to evaluate how well a system can store streaming RDF data obtained from industry. The data is generated in parallel from one or multiple resources and inserted in triple stores by means of  SPARQL INSERT queries. This facet of triple stores has (to the best of our knowledge) never been benchmarked before. SPARQL SELECT queries are used to test the system’s ingestion performance and storage abilities. The benchmarks test for the scalability as well as the accuracy of systems. The Key Performance Indicators for the benchmarks include:

  • their precision, recall and F-measure (both micro and macro)
  • their average answer time by reporting the average delay between the moment at which the SELECT query has been executed and the point in time at which the results are received, and
  • their triples per second, as a fraction of the total number of triples that were inserted during a stream divided by the total time needed for those triples to be inserted

Unstructured Sensor Streams

HOBBIT also provides benchmarks to measure the performance of entity recognition and linking systems for unstructured streams of natural-language data. Hence, HOBBIT reuses some of the concepts developed within the Gerbil framework. These concepts were migrated and adapted to the HOBBIT architecture. Extraction tasks for the entity recognition and linking systems comprise the recognition of known and unknown entities inside the data as well as the linking to different knowledge bases. We designed our benchmarking solutions and our benchmark generator to allow for performance-based evaluations considering two main choke points:

  • Data volume: the volume of knowledge (number of triples) extracted from a document
  • Data velocity: the number of messages a system can deal with in a specific time unit

For the benchmarks the following datasets are integrated:

The benchmarks utilize of the following key performance indicators:

  • precision, recall and F-measure
  • the number of F1-measure points a system achieves per second for a given number of documents

Detailed information is given in the first version of the data extraction benchmark for unstructured data (Deliverable 3.2.1).