Benchmark II – Analysis & Processing

HOBBIT provides benchmarks for the linking and analysis of Linked data. The linking benchmark tests the performance of instance matching methods while the analytics benchmark tests the performance of machine learning methods (supervised and unsupervised) for data analytics.

  • Linking:  Linking spatial resources requires techniques that differ from the classical mostly string-based approaches. In particular, considering the topology of the spatial resources and the topological relations between them is of central importance to systems that manage spatial data. Due to the large amount of available geo-spatial datasets employed in Linked Data and in several domains, it is critical that benchmarks for geo-spatial link discovery are developed.  We propose two benchmarks that deal with link discovery for spatial data:
    1. The Linking Benchmark generator, is simple and can be used not only by instance matching tools, but also by SPARQL engines that deal with query answering over geospatial data such as STRABON [1]. The choke points for this benchmark are a subset of the ones that were used for the development of SPIMBENCH [2]. The ontologies used to represent trajectories are fairly simple, and do not consider complex RDF or OWL schema constructs. The test cases implemented in the benchmark focus on string-based transformations with different (a) levels (b) types of spatial object representations and (c) types of date representations. Furthermore, the benchmark supports addition and deletion of properties. The datasets that implement those test cases can be used by Instance Matching tools to identify matching entities. In a nutshell, the benchmark can be used to check whether two traces with their points annotated with place names designate the same trajectory.
    2. The Spatial Benchmark generator, is more complex and implements all  Dimensionally Extended nine-Intersection Model (DE-9IM) [3] topological relations between trajectories in the two dimensional space. To the best of our knowledge such a generic benchmark, that takes as input trajectories and checks the performance of linking systems for spatial data does not exist. For the design of this benchmark, we focused on (a) on the correct implementation of all the topological relations of the DE-9IM topological model and (b) on producing large enough datasets to stress the systems under test. The supported relations are: Equals, Disjoint, Touches, Contains/Within, Covers/CoveredBy, Intersects, Crosses, Overlaps. To the best of our knowledge, there exist few systems that implement all the topological relations of DE-9IM, hence the benchmark already addresses the first choke point set. Moreover, we produced large synthetic datasets using TomTom’s original data, and hence we are able to challenge the systems regarding scalability.
  • Both benchmarks test for the scalability as well as the accuracy of systems. The Key Performance Indicators for the benchmarks include:
    • precision, recall and F-measure
    • Time performance
  • The description of the first version of the benchmarks can be found in Deliverable 4.1.1.
  • Structured Machine Learning on streaming data: the structured machine learning benchmark is focused on analysis of the RDF streaming data generated by digital and analogue sensors embedded within manufacturing equipment. The benchmark distinguishes two scenarios related to the problem of automatic detection of anomalies for injection molding and assembly machines. The overall goal of both scenarios is to detect abnormal behavior of a manufacturing machine based on the observation of the stream of measurements provided by such a machine. The data produced by each sensor is clustered and the state transitions between the observed clusters are modeled as a Markov chain. Based on this classification, anomalies are detected as sequences of transitions that happen with a probability lower than a given threshold. The difference between the scenarios is in different amount of machines: in the first scenario the number of machines to observe is fixed, while in the second scenarios new machines dynamically join and leave the set of observable machines. The latency of anomaly detection is key performance indicator of the benchmark.
    The description of the first version of the benchmark is presented in Deliverable 4.2.1. The source codes of the benchmark are available at Github.


[1] Kyzirakos, Kostis, Manos Karpathiotakis, and Manolis Koubarakis. “Strabon: a semantic geospatial DBMS.” The Semantic Web–ISWC 2012 (2012): 295-311.
[2] Saveta, Tzanina, et al. “Pushing the limits of instance matching systems: A semantics-aware benchmark for linked data.” Proceedings of the 24th International Conference on World Wide Web. ACM, 2015.
[3] Strobl, Christian. “Dimensionally Extended Nine‐Intersection Model (DE-9IM).” Encyclopedia of GIS. Springer US, 2008. 240-245.