OKE2017 Challenge – ESWC 2017

Open Knowledge Extraction (OKE) Challenge – ESWC 2017

Challenge Motivation

The Open Knowledge Extraction Challenge invites researchers and practitioners from academia as well as industry to compete to the aim of pushing further the state of the art of knowledge extraction for the Semantic Web. The Open Knowledge Extraction Challenge is accepted at ESWC 2017.

Most of the Web content consists of natural language text, e.g., websites, news, blogs, micro-posts, etc., hence a main challenge is to extract as much relevant knowledge as possible from this content, and publish it in the form of Semantic Web triples. There is huge work on knowledge extraction (KE) and knowledge discovery contributing to address this problem. In fact, results of knowledge extraction systems are usually evaluated against tasks that do not focus on specific Semantic Web goals. For example, tasks such as named entity recognition, named entity disambiguation and relation extraction are certainly of importance for the SW, but in most cases such tasks are designed without considering the output design and formalization in the form of Linked Data and OWL ontologies. This makes results of existing methods often not directly reusable for populating the SW, until a translation from linguistic semantics to formal semantics is performed.

Challenge Overview

The goal of the Open Knowledge Extraction Challenge is to test the performance of Knowledge Extraction Systems with respect to the Semantic Web. The OKE challenge has the ambition to provide a reference framework for research on “Knowledge Extraction from text for the Semantic Web” by redefining a number of tasks (typically from information and knowledge extraction) by taking into account specific Semantic Web requirements.


For more information check out the forum google groups oke2017 or send an e-mail to: OKE-contact@googlegroups.com 

Important Dates

Paper submission deadline: Monday March 20th, 2017
Challenge paper reviews: Tuesday April 5th, 2017
Paper Notifications and invitation to task: Friday April 7th, 2017
Camera ready papers (5 pages document): Sunday April 23rd, 2017
Release of training data and instructions: Friday January 13th, 2017
Release of test dataset: Friday April 7th, 2017
Deadline for system submission: Sunday April 23rd, 2017
Running of the systems: Monday May 15th, 2017
Results: Monday May 22nd, 2017
Presentation of challenge results: Friday June 2nd, 2017
Camera ready papers for the challenge proceedings (up to 15 pages): Friday June 30th, 2017 (tentative deadline)
Proclamation of winners: During ESWC2017 closing ceremony

Tasks and Training Data

  • Task 1: Focused NE Identification and Linking
  • Task 2: Broader NE Identification and Linking
  • Task 3: Focused Musical NE Recognition and Linking
  • Task 4: Knowledge Extraction
Each participant must provide a system that solves at least one of the tasks and a paper that describes this system. Both requirements are described in the following.

Note that if you have questions regarding the submission of the paper or the system feel free to write a mail to OKE-contact@googlegroups.com .

Registration and Submission

  • All challenge papers should be exactly five (5) pages in length in PDF file format and written in English.
  • In order to prepare their manuscripts, authors should follow Springer’s Lecture Notes in Computer Science (LNCS) style. For details and templates see Springer’s Author Instructions.
  • Paper submissions will be handled electronically via the EasyChair conference management system, available at the following address: https://easychair.org/conferences/?conf=oke2017.
  • Papers must be submitted no later than Monday March 20th, 2017, 23:59 Hawaii Time.
    NOTE: Eligible to submit papers are only authors participating in the challenge.
  • Each submission will be peer-reviewed by members of the challenge program committee.  Papers will be evaluated according to their significance, originality, technical content, style, clarity, and relevance to the challenge.
  • Each participant must provide his/her solution as a docker image that abides by the technical specifications.
  • Proceedings will be published by Springer in LNCS volume.
  • After the conference, challenge participants will be able to provide a detailed description of their system and evaluation results in a longer version of their paper (up to 15 pages). This longer paper will be included in the challenge proceedings.

Technical requirements for participation

Each participant must provide a system as Docker image. This image has to be uploaded to the HOBBIT Gitlab (it is possible to use a private repository, i.e., the system will not be visible for other people). In general, the uploaded Docker image can contain either a) the system itself or b) a web service client that forwards requests to the system that is hosted by you. Note that we highly recommend the first solution since a web service client won’t enable you to take part in the scenario B of the tasks.

Implementing the API

To be able to benchmark your system, it needs to implement our NIF-based (e.g., using a wrapper). There are several scenarios how this can be achieved.

1st possibility: GERBIL compatible APIs

If your system already implements an API that is compatible with the GERBIL benchmarking framework, you do not have to implement anything additional to that. You only need to provide a Docker image of your system that implements the same API as your original web service and send us a description how it has to be started (e.g., which environmental variables have to be defined).

A system is compatible to GERBIL if it is either one of the systems that are already available in GERBIL or it is a NIF-based web service that can be benchmarked with GERBIL.

2nd possibility: Java based system or system adapter

If your system is based on Java or you would like to implement a wrapper for your system in Java, we can provide you with a base implementation that is described in an article in the HOBBIT platform wiki. Additionally, you might want to use the code provided by the GERBIL project for implementing a NIF-based web service. Therefor, the gerbil.nif.transfer library needs to be included. The system adapters receiveGeneratedTask method could look as follows:

public void receiveGeneratedTask(String taskIdString, byte[] data) {
    TurtleNIFParser parser = new TurtleNIFParser();
    TurtleNIFWriter writer = new TurtleNIFWriter();
    List documents = parser.parseNIF(RabbitMQUtils.readString(data));
    try {
        Document document = documents.get(0);
        sendResultToEvalStorage(taskIdString, RabbitMQUtils.writeString(writer.writeNIF(documents)));
    } catch (Exception e) {
        // handle exception

Where annotator.annotate(document) adds the named entities to the document. If your system is not already compatible to GERBIL, we recommend this way.


3rd possibility: Direct implementation of the API

If you want to use a different language to implement our NIF-based API, you need to implement the API of a system that can be benchmarked in HOBBIT. Every message of the task queue will be a single NIF-document. The response of your system has to be send to the result queue. Your system won’t receive data through the data queue.

Uploading the Docker image

The uploading of the Docker image is described in the Hobbit project platform wiki.

The system meta data file

Your system needs a system meta data file (called system.ttl)

A detailed description of the system.ttl file will be added soon.


The organization responsibility will be shared by the following four main organizers:

  • René Speck, University of Leipzig, Germany
    Expertise: Knowledge extraction, Named Entitiy Recognition, Relation Extraction
    Website: http://aksw.org/RenSpeck
    Email: speck@informatik.uni-leipzig.de
  • Michael Röder, University of Leipzig, Germany
    Expertise: Benchmarking and Topic Modeling
    Website: http://aksw.org/MichaelRoeder
    Email: roeder@informatik.uni-leipzig.de
  • Ricardo Usbeck, University of Leipzig, Germany
    Expertise: Knowledge extraction and Question Answering
    Website: http://aksw.org/RicardoUsbeck
    Email: usbeck@informatik.uni-leipzig.de
  • Axel-Cyrille Ngonga Ngomo, Institute for Applied Informatics, Germany
    Expertise: Knowledge extraction, Machine Learning, Question Answering, Information Retrieval
    Website:  http://aksw.org/AxelNgonga
    Email: ngonga@informatik.uni-leipzig.de
  • Horacio Saggion, Universitat Pompeu Fabra, Spain
    Expertise: Natural Language Processing and Human Language Technology
    Website: http://www.taln.upf.edu/users/hsaggion
    Email: horacio.saggion@upf.edu
  • Luis Espinosa-Anke, Universitat Pompeu Fabra, Spain
    Expertise: Natural Language Processing and Information Extraction
    Website: http://www.taln.upf.edu/users/lespinosa
    Email: uis.espinosa@upf.edu
  • Sergio Oramas, Universitat Pompeu Fabra, Spain
    Expertise: Information Extraction and Music Information Retrieval
    Email: sergio.oramas@upf.edu

The following set of people comprises the list of programme committee members. All the members of the list are experts from research and industry, who will review the paper submissions, independently of the organization team.

  • Andrea Nuzzolese, National Research Council Rome, IT
  • Anna-Lisa Gentile, University of Mannheim, DE
  • Davide Buscaldi, University Paris 13s, FR
  • Francesco Barbieri, Universitat Pompeu University, ES
  • Johannes Hoffar, Max Planck Institute, DE
  • José Camacho-Collados, Sapienza University of Rome, IT
  • Luciano Del Corro, Max Planck Institute, DE
  • Philipp Cimiano, Bielefeld University, DE
  • Raphaël Troncy, EURECOM, FR
  • Sergio Consoli, Data Science Department at Philips Research, DE
  • Xavier Serra, Universitat Pompeu Fabra, ES