Since 2004 OAEI organises campaigns aiming at evaluating ontology matching tools and technologies. This year FORTH along with HOBBIT participated in the OAEI 2017 Instance Matching Track and introduced the HOBBIT Link Discovery Track. The Tasks come with two datasets the Sandbox and Mainbox characterized by the number of instances to match (i.e., scale). The Sandbox is the sample dataset given to the systems and the Mainbox is the largest dataset.
Instance Matching Track
The Instance Matching Track aims at evaluating the performance of matching tools in which the goal is to detect the degree of similarity between pairs of instances expressed in the form of OWL ABoxes. We participated in the Instance Matching Track with the SYNTHETIC Task.
The SYNTHETIC datasets are generated and transformed using SPIMBENCH . SPIMBENCH takes as input a dataset and produces a target dataset by employing a set of value-based, structure-based and semantics-aware transformations. During the generation of the target dataset, SPIMBENCH produces a weighted gold standard that records the matched pair of source, target instances in addition to the type of transformations that were employed to produce the target instance from its corresponding source one.
The participants to these tasks were the following systems: AgreementMakerLight (AML), I-Match, Legato and LogMap. The results of the systems during the Challenge are shown in the Tables 1 and 2 below.
Table 1: Results for the Sandbox dataset of the SYNTHETIC Task
Table 2: Results for the Mainbox dataset of the SYNTHETIC Task
HOBBIT Link Discovery Track
In this Track two benchmark generators are proposed to deal with Link Discovery for spatial data. We considered that spatial data are represented as trajectories, that is sequences of longitude, latitude pairs.
Task 1 (Linking) measures how well the systems can match traces that have been altered using string-based approaches in addition to addition and deletion of intermediate points. This task also contains changes on date format and changes on coordinate formats.
The participants to this task are the AgreementMakerLight (AML) and OntoIdea systems. The results of the systems for the Linking Task are shown in Tables 3 and 4 below.
|Sandbox Dataset||Precision||Recall||F-measure||Time Performance|
Table 3: Results for the Sandbox dataset of the Link Discovery Track (Linking Task)
|Maindbox Dataset||Precision||Recall||F-measure||Time Performance|
|OntoIdea||Platform Time Limit (75 mins)|
Table 4: Results for the Mainbox dataset of the Link Discovery Track (Linking Task)
Task 2 (Spatial) measures how well the systems can identify the DE-9IM (Dimensionally Extended nine-Intersection Model) topological relations. The supported spatial relations are the following: Equals, Disjoint, Touches, Contains/Within, Covers/CoveredBy, Intersects, Crosses, Overlaps and the traces are represented in Well-known text (WKT) format. For each relation, a different pair of source and target dataset was given to the participants.
The participants to the Spatial task were systems AgreementMakerLight (AML), OntoIdea, Rapid Discovery of Topological Relations (RADON) and Silk. The results are shown Figures 1 and 2 below.
Figure 1: Results for the Sandbox dataset of the Link Discovery Track (Spatial Task)
Figure 2: Results for the Mainbox dataset of the Link Discovery Track (Spatial Task)
 T. Saveta, E. Daskalaki, G. Flouris, I. Fundulaki, M. Herschel, and A.-C. Ngonga Ngomo. Pushing the limits of instance matching systems: A semantics-aware benchmark for linked data. In WWW, pages 105106. ACM, 2015. Poster.
 T. Saveta, E. Daskalaki, G. Flouris, I. Fundulaki, and A. Ngonga-Ngomo. LANCE: Piercing to the Heart of Instance Matching Tools. In ISWC, 2015.