Hobbit @ ESWC 2017

The HOBBIT project organized three benchmarking challenges at the ESWC 2017 conference, which took place in Portoroz, Slovenia from May 28th to June 1st, as it aims to bring together and inspire people from academia and industry to come up with fresh and innovative ideas on solving real-life problems concerning Big Linked Data.

We are proud that under the umbrella of HOBBIT a new challenge was launched, called MOCHA (Mighty Storage Challenge), and two already established challenges reran; the 7th edition of the QALD (Question Answering over Linked Data) challenge and the 3rd edition of the OKE (Open Knowledge Extraction) challenge.

Each challenge consisted of four benchmarking tasks prepared by the HOBBIT partners and participants were invited to submit both a system that tackles one, or more, of the challenge’s tasks and a short paper describing their system. Specifically, the following benchmarking tasks were offered by the three HOBBIT challenges (the tasks are described in detail on the challenges’ websites):

MOCHA	Task 1: RDF Data Ingestion Task 2: Data Storage Benchmark Task 3: Versioning RDF Data Task 4: Faceted Browsing Benchmark
QALD	Task 1: Multilingual question answering over DBpedia Task 2: Hybrid question answering Task 3: Large-Scale Question answering over RDF Task 4: English question answering over Wikidata
OKE	Task 1: Focused NE Identification and Linking Task 2: Broader NE Identification and Linking Task 3: Focused Musical NE Recognition and Linking Task 4: Knowledge Extraction

The papers describing the systems were peer-reviewed by experts and the most promising systems were invited to participate in each challenge. In the MOCHA challenge 2 systems took part, in the QALD challenge 3 systems and in the OKE challenge 2 systems.

The HOBBIT platform successfully supported the running of the three challenges. The datasets pertaining to the benchmarking tasks were integrated into the platform and the evaluation of the competing systems was performed using the platform. Importantly the challenges provided to the HOBBIT consortium a golden opportunity to identify and fix bugs in the platform, improve its documentation, as well as to stress-test the platform.

A session dedicated to the three HOBBIT challenges took place on the 30th of May as part of the main ESWC 2017 conference. This ensured great exposure for HOBBIT and competing systems, as the session attracted several of the ESWC 2017 attendees. For each of the challenges an overview presentation was given by the challenge organizers and also the authors of the competing systems were given the opportunity to briefly present their work. Moreover, during the last day of the conference, on the 1st of June, authors were also given the opportunity to demonstrate their work in the form of a poster in a two-hour long session.

Some live reports from the three challenges is available on Twitter.

The icing on the cake came during the ESWC 2017 closing ceremony on the evening of the 1st of June, where the winners of the three challenges were announced and the winner certificates were handed to them. Also the winners received prize money for their achievement.

The winners of the three HOBBIT challenges were:

MOCHA – Mirko Spasić and Milos Jovanovik, MOCHA 2017 as a Challenge for Virtuoso

OKE – Julien Plu, Raphaël Troncy and Giuseppe Rizzo, ADEL@OKE2017: A Generic Method for Indexing Knowledge Bases for Entity Linking

QALD

Task 1 English – Dennis Diefenbach, Kamal Singh and Pierre Maret, WDAqua-core0: A Question Answering Component for the Research Community

Task 1 French – Nikolay Radoev, Mathieu Tremblay, Michel Gagnon and Amal Zouaq, Answering Natural Language Questions on RDF Knowledge Base in French