OKE2017 Challenge – ESWC 2017

Open Knowledge Extraction (OKE) Challenge – ESWC 2017

Challenge Motivation

The Open Knowledge Extraction Challenge invites researchers and practitioners from academia as well as industry to compete to the aim of pushing further the state of the art of knowledge extraction for the Semantic Web. The Open Knowledge Extraction Challenge is accepted at ESWC 2017.

Most of the Web content consists of natural language text, e.g., websites, news, blogs, micro-posts, etc., hence a main challenge is to extract as much relevant knowledge as possible from this content, and publish it in the form of Semantic Web triples. There is huge work on knowledge extraction (KE) and knowledge discovery contributing to address this problem. In fact, results of knowledge extraction systems are usually evaluated against tasks that do not focus on specific Semantic Web goals. For example, tasks such as named entity recognition, named entity disambiguation and relation extraction are certainly of importance for the SW, but in most cases such tasks are designed without considering the output design and formalization in the form of Linked Data and OWL ontologies. This makes results of existing methods often not directly reusable for populating the SW, until a translation from linguistic semantics to formal semantics is performed.

Challenge Overview

The goal of the Open Knowledge Extraction Challenge is to test the performance of Knowledge Extraction Systems with respect to the Semantic Web. The OKE challenge has the ambition to provide a reference framework for research on “Knowledge Extraction from text for the Semantic Web” by redefining a number of tasks (typically from information and knowledge extraction) by taking into account specific Semantic Web requirements.


For more information check out the forum google groups oke2017 or send an e-mail to: OKE-contact@googlegroups.com 

Important Dates

Paper submission deadline: Monday March 20th, 2017
Challenge paper reviews: Tuesday April 5th, 2017
Paper Notifications and invitation to task: Friday April 7th, 2017
Camera ready papers (5 pages document): Sunday April 23rd, 2017
Release of training data and instructions: Friday January 13th, 2017
Release of test dataset: Friday April 7th, 2017
Deadline for system submission: Sunday April 30th, 2017
Running of the systems: Monday May 15th, 2017
Results: Monday May 22nd, 2017
Presentation of challenge results: Thursday June 1st, 2017
Camera ready papers for the challenge proceedings (up to 15 pages): Friday June 30th, 2017 (tentative deadline)
Proclamation of winners: During ESWC2017 closing ceremony

Tasks and Training Data

  • Task 1: Focused NE Identification and Linking
  • Task 2: Broader NE Identification and Linking
  • Task 3: Focused Musical NE Recognition and Linking
  • Task 4: Knowledge Extraction

Each participant must provide a system that solves at least one of the tasks and a paper that describes this system. Both requirements are described in the following.

Note that if you have questions regarding the submission of the paper or the system feel free to write a mail to OKE-contact@googlegroups.com .

Registration and Submission

  • All challenge papers should be exactly five (5) pages in length in PDF file format and written in English.
  • In order to prepare their manuscripts, authors should follow Springer’s Lecture Notes in Computer Science (LNCS) style. For details and templates see Springer’s Author Instructions.
  • Paper submissions will be handled electronically via the EasyChair conference management system, available at the following address: https://easychair.org/conferences/?conf=oke2017.
  • Papers must be submitted no later than Monday March 20th, 2017, 23:59 Hawaii Time.
    NOTE: Eligible to submit papers are only authors participating in the challenge.
  • Each submission will be peer-reviewed by members of the challenge program committee.  Papers will be evaluated according to their significance, originality, technical content, style, clarity, and relevance to the challenge.
  • Each participant must provide his/her solution as a docker image that abides by the technical specifications.
  • Proceedings will be published by Springer in LNCS volume.
  • After the conference, challenge participants will be able to provide a detailed description of their system and evaluation results in a longer version of their paper (up to 15 pages). This longer paper will be included in the challenge proceedings.

Technical requirements for participation

Each participant must provide a system as Docker image. This image has to be uploaded to the HOBBIT Gitlab (it is possible to use a private repository, i.e., the system will not be visible for other people). In general, the uploaded Docker image can contain either a) the system itself or b) a web service client that forwards requests to the system that is hosted by you. Note that we highly recommend the first solution since a web service client won’t enable you to take part in the scenario B of the tasks.

Implementing the API

To be able to benchmark your system, it needs to implement our NIF-based (e.g., using a wrapper). There are several scenarios how this can be achieved.

1st possibility: GERBIL compatible APIs

If your system already implements a NIF-based API that is compatible with the GERBIL benchmarking framework, you do not have to implement anything additional to that. You only need to provide a Docker image of your system that implements the same API as your original web service and an adapted version of the following system meta data file.

@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .
@prefix hobbit: <http://w3id.org/hobbit/vocab#> .
@prefix gerbil2: <http://w3id.org/gerbil/hobbit/vocab#> .
@prefix nifsys: <http://w3id.org/hobbit/gerbil/nif-sys#&gt .

<http://www.maxpower.org/HyperNER> a  hobbit:SystemInstance; 
	rdfs:label            "HyperNER"@en;
	rdfs:comment	      "This is my own system defined in a simple way"@en;
	hobbit:imageName "git.project-hobbit.eu:4567/gerbil/gerbilnifsystemadapter";
	hobbit:implementsAPI  gerbil2:GerbilApiOKE2015T1;
	nifsys:instanceImageName "git.project-hobbit.eu:4567/maxpower/mysystem";
	hobbit:usesImage "git.project-hobbit.eu:4567/maxpower/mysystem";
	nifsys:numberOfInstances "3"^^xsd:unsignedInt;
	nifsys:webserviceUrl "http://HOST:8080/hyper-ner" .

Please replace

  • the URI <http://www.maxpower.org/HyperNER> with your own URI (it needs to be unique)
  • the label and comment with the name and a very short description of your annotation system
  • the URL git.project-hobbit.eu:4567/maxpower/mysystem with the name of your uploaded image in both lines, nifsys:instanceImageName and hobbit:usesImage
  • the URL http://HOST:8080/hyper-ner with the URL that should be used to call your web service (where HOST will be replaced with the host name at runtime)

2nd possibility: Java based system or system adapter

If your system is based on Java or you would like to implement a wrapper for your system in Java, we can provide you with a base implementation that is described in an article in the HOBBIT platform wiki. Additionally, you might want to use the code provided by the GERBIL project for implementing a NIF-based web service. Therefor, the gerbil.nif.transfer library needs to be included. The system adapters receiveGeneratedTask method could look as follows:

public void receiveGeneratedTask(String taskIdString, byte[] data) {
    TurtleNIFParser parser = new TurtleNIFParser();
    TurtleNIFWriter writer = new TurtleNIFWriter();
    List documents = parser.parseNIF(RabbitMQUtils.readString(data));
    try {
        Document document = documents.get(0);
        sendResultToEvalStorage(taskIdString, RabbitMQUtils.writeString(writer.writeNIF(documents)));
    } catch (Exception e) {
        // handle exception

Where annotator.annotate(document) adds the named entities to the document. If your system is not already compatible to GERBIL, we recommend this way.


3rd possibility: Direct implementation of the API

If you want to use a different language to implement our NIF-based API, you need to implement the API of a system that can be benchmarked in HOBBIT. Every message of the task queue will be a single NIF-document. The response of your system has to be send to the result queue. Your system won’t receive data through the data queue.

Uploading the Docker image

The uploading of the Docker image is described in the Hobbit project platform wiki.

The system meta data file

Your system needs a system meta data file (called system.ttl). For participating in Task 1 – 3, this file can have the following content.

@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
@prefix hobbit: <http://w3id.org/hobbit/vocab#> .
@prefix gerbil2: <http://w3id.org/gerbil/hobbit/vocab#> .

<http://www.maxpower.org/HyperNER> a  hobbit:SystemInstance; 
	rdfs:label            "HyperNER"@en;
	rdfs:comment	      "This is my own system defined in a simple way"@en;
	hobbit:imageName      "git.project-hobbit.eu:4567/maxpower/mysystem";
	hobbit:implementsAPI  gerbil2:GerbilApiOKE2015T1 .

The URI of the system is used as identifier – it does not have to be dereferencable. The system is defined as a system instance, it has a label and a description. The two last lines are very important since they define the image that is used to run the system and the API the system implements. Please note that the Tasks 1, 2 and 3 share the same API.

As described in the wiki page of the system meta data file it is possible to have several instances of a single system. Please, feel free to use this feature to adapt your system for the three different tasks.

Program & Accepted Papers

Challenge Session
Tuesday, May 30th, 2017
15:00 – 15:30
15:00 – 15:10 OKE Challenge Overview
Michael Röder
15:10 – 15:20 Julien Plu, Raphaël Troncy and Giuseppe Rizzo, ADEL@OKE2017: A Generic Method for Indexing Knowlege Bases for Entity Linking
15:20 – 15:30 Raabia Asif and Muhammad Abdul Qadir, “HmaraNER”: A Named Entity Recognizer and Linker
Posters and Demos Session
June 1st, 2017
9:00 – 11:00
The ADEL system will be presented as poster
Closing Ceremony
Thursday, June 1st, 2017
 Announcement of challenge winners during the ESWC closing ceremony

For possible, last minute changes to the program please also check the ESWC 2017 program (http://2017.eswc-conferences.org/program)

OKE 2017 Challenge Results

The results are described in Deliverable 7.3.1 and can be found in HOBBIT platform: here (login as guest).


The organization responsibility will be shared by the following four main organizers:

  • René Speck, University of Leipzig, Germany
    Expertise: Knowledge extraction, Named Entitiy Recognition, Relation Extraction
    Website: http://aksw.org/RenSpeck
    Email: speck@informatik.uni-leipzig.de
  • Michael Röder, University of Leipzig, Germany
    Expertise: Benchmarking and Topic Modeling
    Website: http://aksw.org/MichaelRoeder
    Email: roeder@informatik.uni-leipzig.de
  • Ricardo Usbeck, University of Leipzig, Germany
    Expertise: Knowledge extraction and Question Answering
    Website: http://aksw.org/RicardoUsbeck
    Email: usbeck@informatik.uni-leipzig.de
  • Axel-Cyrille Ngonga Ngomo, Institute for Applied Informatics, Germany
    Expertise: Knowledge extraction, Machine Learning, Question Answering, Information Retrieval
    Website:  http://aksw.org/AxelNgonga
    Email: ngonga@informatik.uni-leipzig.de
  • Horacio Saggion, Universitat Pompeu Fabra, Spain
    Expertise: Natural Language Processing and Human Language Technology
    Website: http://www.taln.upf.edu/users/hsaggion
    Email: horacio.saggion@upf.edu
  • Luis Espinosa-Anke, Universitat Pompeu Fabra, Spain
    Expertise: Natural Language Processing and Information Extraction
    Website: http://www.taln.upf.edu/users/lespinosa
    Email: uis.espinosa@upf.edu
  • Sergio Oramas, Universitat Pompeu Fabra, Spain
    Expertise: Information Extraction and Music Information Retrieval
    Email: sergio.oramas@upf.edu

The following set of people comprises the list of programme committee members. All the members of the list are experts from research and industry, who will review the paper submissions, independently of the organization team.

  • Andrea Nuzzolese, National Research Council Rome, IT
  • Anna-Lisa Gentile, University of Mannheim, DE
  • Davide Buscaldi, University Paris 13s, FR
  • Francesco Barbieri, Universitat Pompeu University, ES
  • Johannes Hoffar, Max Planck Institute, DE
  • José Camacho-Collados, Sapienza University of Rome, IT
  • Luciano Del Corro, Max Planck Institute, DE
  • Philipp Cimiano, Bielefeld University, DE
  • Raphaël Troncy, EURECOM, FR
  • Sergio Consoli, Data Science Department at Philips Research, DE
  • Xavier Serra, Universitat Pompeu Fabra, ES


OKE 2017 – Open Knowledge Extraction Challenge
URL: https://project-hobbit.eu/challenges/oke2017-challenge-eswc-2017/
Email: OKE-contact@googlegroups.com

May 28th to June 1st 2017, Portoroz, Slovenia

in conjunction with the 14th European Semantic Web Conference (ESWC 2017, http://2017.eswc-conferences.org)

The aim of this challenge is to test the performance of knowledge extraction systems in
aspects that are relevant for the Semantic Web. These include precision, recall and
runtime. The challenge will test the systems against data derived from real datasets.
We herewith invite system developers to participate in the aforementioned tasks.
The system developers are invited to write papers (5 page documents) presenting their results on the training data
(see important dates below). To ensure that the system results are comparable, we will
provide the HOBBIT benchmarking platform for the generation of the final results to be
included into the system publications. A specification of the hardware on which the
benchmarks will be ran will be released in due course.

This year, the challenge comprises the following tasks:
* Task 1: Focused Named Entity Identification and Linking
* Task 2: Broader Named Entity Identification and Linking
* Task 3: Focused Musical Named Entity Recognition and Linking
* Task 4: Knowledge Extraction

Important Dates
* Paper submission deadline (5 pages document): March 10th, 2017, 23:59 Hawaii Time**
* Notification of acceptance: April 7th, 2017
* Camera ready papers (5 pages document): April 23rd, 2017
* Deadline for submission of system answers/instructions for evaluation: TBA
* Release of evaluation results: TBA
* Proclamation of winners: During ESWC 2017 closing ceremony

**Eligible to submit papers are only authors participating in the challenge.

* René Speck, University of Leipzig, Germany
* Michael Röder, University of Leipzig, Germany
* Ricardo Usbeck, University of Leipzig, Germany
* Axel-Cyrille Ngonga Ngomo, Institute for Applied Informatics, Germany
* Horacio Saggion, Universitat Pompeu Fabra, Spain
* Luis Espinosa-Anke, Universitat Pompeu Fabra, Spain
* Sergio Oramas, Universitat Pompeu Fabra, Spain

For the complete list of organizers and program committee members,
visit the challenge website.

Further Information and Contact
For detailed information, including datasets and submission guidelines,
please visit the challenge website: https://project-hobbit.eu/challenges/oke2017-challenge-eswc-2017/
Contact Email: OKE-contact@googlegroups.com

The challenge outcome overview can be found here, while challenge proceedings are available by Springer here.