The HOBBIT project organized three benchmarking challenges at the ESWC 2018 conference, which took place in Heraklion, Crete in Greece from June 3rd to June 7th, as it aims to bring together and inspire people from academia and industry to come up with fresh and innovative ideas on solving real-life problems concerning Big Linked Data.
We are proud that under the umbrella of HOBBIT, we reran the challenges organized at ESWC 2017; the 2nd edition of MOCHA (Mighty Storage Challenge), the 4th edition of the OKE (Open Knowledge Extraction) challenge and the SQA (Scalable Question Answering) challenge, an offspring of the well-known QALD (Question Answering over Linked Data) challenge.
Challenges consisted of multiple benchmarking tasks prepared by the HOBBIT partners and participants were invited to submit both a system that tackles one, or more, of a challenge’s tasks and a short, 5-page, paper describing their system. Specifically, the following benchmarking tasks were offered by the three HOBBIT challenges (the tasks are described in detail on the challenges’ websites):
|MOCHA||Task 1: RDF Data Ingestion
Task 2: Data Storage Benchmark
Task 3: Versioning RDF Data
Task 4: Faceted Browsing Benchmark
|OKE||Task 1: Focused Named Entity Identification and Linking
Task 2: Broader Named Entity Identification and Linking
Task 3: Relation Extraction
Task 4: Knowledge Extraction
|SQA||Task 1: Large-Scale Question answering over RDF|
The papers describing the systems were peer-reviewed by experts and the most promising systems were invited to participate in each challenge. In the MOCHA challenge 2 systems took part, in the OKE challenge 1 system and in the SQA challenge 3 systems. The HOBBIT partners also offered additional baseline systems, against which those of the participants were evaluated.
The HOBBIT platform successfully supported the running of the three challenges. The datasets pertaining to the benchmarking tasks were integrated into the platform and the evaluation of the competing systems was performed using the platform. Importantly, the challenges provided to the HOBBIT consortium another opportunity to identify and fix bugs in the platform, improve its documentation, as well as to stress-test the platform.
A session dedicated to the three HOBBIT challenges took place on the 5th of June as part of the main ESWC 2018 conference. This ensured great exposure for HOBBIT and competing systems, as the session attracted around 22 attendees. For each of the challenges an overview presentation was given by the challenge organizers and also the authors of the competing systems were given the opportunity to briefly present their work.
The icing on the cake came during the ESWC 2018 closing ceremony, on the evening of the 7th of June, where the winners of the three challenges were announced and the winner certificates were handed to them. Also the winners received prize money for their achievement by the HOBBIT sponsors USU and Meltwater.
USU sponsored the MOCHA and OKE challenges with 250€ each, while Meltwater sponsored SQA challenge with 300€ and additionally organized a social dinner on the evening of the 5th of June for the question answering community.
The winners of the three HOBBIT challenges were:
MOCHA – Milos Jovanovik and Mirko Spasić, Benchmarking Virtuoso 8 at the Mighty Storage Challenge 2018: Training Results (prize: 250€ from USU + 100€ from Springer)
OKE – Hector Cerezo-Costas and Manuela Martın-Vicente, Relation Extraction for Knowledge Base Completion: A supervised approach (prize: 250€ from USU + 100€ from Springer)
SQA (winner) – Dennis Diefenbach, Kamal Singh and Pierre Maret, On the scalability of the QA system WDAqua-core1 (prize: 300€ from Meltwater)
SQA (second place) – Elizaveta Zimina, Jyrki Nummenmaa, Kalervo Järvelin, Jaakko Peltonen, Kostas Stefanidis and Heikki Hyyrö, GQA: Grammatical Question Answering for RDF Data (prize: 100€ from Springer)
Photos from the challenges workshop session (5th of June)