Ontology Matching (OM 2017)

Since 2004 OAEI organises campaigns aiming at evaluating ontology matching tools and technologies. This year FORTH along with HOBBIT participated in the OAEI 2017 Instance Matching Track and introduced the HOBBIT Link Discovery Track. The Tasks come with two datasets the Sandbox and Mainbox characterized by the number of instances to match (i.e., scale). The Sandbox is the sample dataset given to the systems and the Mainbox is the largest dataset.

Instance Matching Track

The Instance Matching Track aims at evaluating the performance of matching tools in which the goal is to detect the degree of similarity between pairs of instances expressed in the form of OWL ABoxes. We participated in the Instance Matching Track with the SYNTHETIC Task.

The SYNTHETIC datasets are generated and transformed using SPIMBENCH [1]. SPIMBENCH takes as input a dataset and produces a target dataset by employing a set of value-based, structure-based and semantics-aware transformations. During the generation of the target dataset, SPIMBENCH produces a weighted gold standard that records the matched pair of source, target instances in addition to the type of transformations that were employed to produce the target instance from its corresponding source one.

The participants to these tasks were the following systems: AgreementMakerLight (AML), I-Match, Legato and LogMap. The results of the systems during the Challenge are shown in the Tables 1 and 2 below.

Sandbox Dataset Precision Recall F-measure
AML 0.849 1.000 0.918
I-Match 0.854 0.997 0.920
Legato 0.980 0.730 0.840
LogMap 0.938 0.763 0.841

Table 1: Results for the Sandbox dataset of the SYNTHETIC Task

Mainbox Dataset Precision Recall F-measure
AML 0.855 1.000 0.922
I-Match 0.856 0.997 0.921
Legato 0.970 0.700 0.810
LogMap 0.893 0.709 0.790

Table 2: Results for the Mainbox dataset of the SYNTHETIC Task

 

HOBBIT Link Discovery Track

In this Track two benchmark generators are proposed to deal with Link Discovery for spatial data. We considered that spatial data are represented as trajectories, that is sequences of longitude, latitude pairs.

Task 1 (Linking) measures how well the systems can match traces that have been altered using string-based approaches in addition to addition and deletion of intermediate points. This task also contains changes on date format and changes on coordinate formats.
The participants to this task are the AgreementMakerLight (AML) and OntoIdea systems. The results of the systems for the Linking Task are shown in Tables 3 and 4 below.

Sandbox Dataset Precision Recall F-measure Time Performance
AML 1.000 1.000 1.000 11722
OntoIdea 0.990 0.990 0.990 19806

Table 3: Results for the Sandbox dataset of the Link  Discovery Track (Linking Task)

Maindbox Dataset Precision Recall F-measure Time Performance
AML 1.000 1.000 1.000 134456
OntoIdea Platform Time Limit (75 mins)

Table 4: Results for the Mainbox dataset of the Link Discovery Track (Linking Task)

 

Task 2 (Spatial) measures how well the systems can identify the DE-9IM (Dimensionally Extended nine-Intersection Model) topological relations. The supported spatial relations are the following: Equals, Disjoint, Touches, Contains/Within, Covers/CoveredBy, Intersects, Crosses, Overlaps and the traces are represented in Well-known text (WKT) format. For each relation, a different pair of source and target dataset was given to the participants.

The participants to the Spatial task were systems AgreementMakerLight (AML), OntoIdea, Rapid Discovery of Topological Relations (RADON) and Silk. The results are shown Figures 1 and 2 below.

Figure 1: Results for the Sandbox dataset of the Link  Discovery Track (Spatial Task)

 

Figure 2: Results for the Mainbox dataset of the Link  Discovery Track (Spatial Task)

 

[1] T. Saveta, E. Daskalaki, G. Flouris, I. Fundulaki, M. Herschel, and A.-C. Ngonga Ngomo. Pushing the limits of instance matching systems: A semantics-aware benchmark for linked data. In WWW, pages 105106. ACM, 2015. Poster.
[2] T. Saveta, E. Daskalaki, G. Flouris, I. Fundulaki, and A. Ngonga-Ngomo. LANCE: Piercing to the Heart of Instance Matching Tools. In ISWC, 2015.

 

Spread the word. Share this post!

Leave A Reply

Your email address will not be published. Required fields are marked *