OKE2018 Challenge – ESWC 2018

OKE2018 – Open Knowledge Extraction Challenge

The OKE challenge invites researchers and practitioners from academia as well as industry to compete to the aim of pushing further the state of the art in knowledge extraction from text for the Semantic Web. The challenge has the ambition to provide a reference framework for research in this field by redefining a number of tasks typically from information and knowledge extraction by taking into account Semantic Web requirements and has the goal to test the performance of knowledge extraction systems.

This year, the challenge goes in the fourth round and consists of four tasks which include named entity identification, disambiguation by linking to a knowledge base as well as relation and knowledge extraction. The challenge makes use of small gold standard datasets that consist of manually curated documents and large silver standard datasets that consist of automatically generated synthetic documents. The performance measure of a participating system is twofold based on (1) Precision, Recall, F1-measure and on (2) Precision, Recall, F1-measure with respect to the runtime of the system.

OKE2018 takes place in conjunction with the 15th European Semantic Web Conference (ESWC 2018), 3rd-7th June 2018, Heraklion, Crete, Greece

Stay tuned to get ready to participate!       Express your interest!

Sponsorship

The winner of the OKE challenge will receive a prize of 250 Euros.
Prize money is provided by USU.

Q&A

Contact Email: OKE-contact@googlegroups.com , speck@informatik.uni-leipzig.de

Important Dates

Paper submission deadline: Friday, April 4th, 2018 (extended)
Challenge paper reviews: Thursday, April 26th, 2018  (extended) 
Paper Notifications and invitation to task: Monday, April 30th, 2018 (extended) 
Camera ready papers (5 pages document): Monday, May 14th 2018 (extended) 
Camera ready papers for the challenge proceedings (up to 15 pages): Friday, July 6th, 2018 (tentative deadline)

Release of training data and instructions: Monday, February 12, 2018
Deadline for system submission: Sunday, May 13th, 2018
Running of the systems: Tuesday, May 15th, 2018
Presentation of challenge results: During ESWC2018
Proclamation of winners: During ESWC2018 closing ceremony

Tasks

This year, the challenge comprises the following tasks:

  • Task 1: Focused Named Entity Identification and Linking
  • Task 2: Broader Named Entity Identification and Linking
  • Task 3: Relation Extraction
  • Task 4: Knowledge Extraction

Participants will be expected to describe their solution and results on the training datasets over a 5 page paper. In particular, a short summary of the approach chosen, a link to the experimental results and an analysis of the strengths and weaknesses of the approach are expected.

Training Data

Scenario A

Scenario A offers 100 curated training data. The goal of this task is to achieve a high F1-score.

Scenario B

Scenario B offers a large number of synthetically generated data. The performance of a system is measured by β which is defined as β = F1-score / runtime.
The goal of this task is to achieve a high β.

Data

Scenario A
Task 1 training data (updated 23/03/2018)
Task 2 training data (updated 23/03/2018)
Task 3 training data (updated 05/06/2018)
Task 4 training data (updated 05/06/2018)

Prerequisites for participation

Each participant must provide a system that solves at least one of the tasks and a paper that describes this system. Both requirements are described in the following.

Note that if you have questions regarding the submission of the paper or the system feel free to write a mail to OKE-contact@googlegroups.com .

Registration and Submission

  • All challenge papers should be exactly five (5) pages in length in PDF file format and written in English.
  • In order to prepare their manuscripts, authors should follow Springer’s Lecture Notes in Computer Science (LNCS) style. For details and templates see Springer’s Author Instructions.
  • Paper submissions will be handled electronically via the EasyChair conference management system, available at the following address: https://easychair.org/conferences/?conf=oke2018 .
  • Papers must be submitted no later than Friday, March 30th, 2018  (extended), 23:59 Hawaii Time.
  • Each submission will be peer-reviewed by members of the challenge program committee.  Papers will be evaluated according to their significance, originality, technical content, style, clarity, and relevance to the challenge.
  • Proceedings will be published by Springer in LNCS volume.
  • After the conference, challenge participants will be able to provide a detailed description of their system and evaluation results in a longer version of their paper (up to 15 pages). This longer paper will be included in the challenge proceedings.

Technical requirements for participation

Each participant must provide a system as Docker image. This image has to be uploaded to the HOBBIT Gitlab (it is possible to use a private repository, i.e., the system will not be visible for other people). In general, the uploaded Docker image can contain either a) the system itself or b) a web service client that forwards requests to the system that is hosted by you. Note that we highly recommend the first solution since a web service client won’t enable you to take part in the scenario B of the tasks.

Implementing the API

To be able to benchmark your system, it needs to implement our NIF-based (e.g., using a wrapper). There are several scenarios how this can be achieved.

1st possibility: GERBIL compatible APIs

If your system already implements a NIF-based API that is compatible with the GERBIL benchmarking framework, you do not have to implement anything additional to that. You only need to provide a Docker image of your system that implements the same API as your original web service and an adapted version of the following system meta data file.

@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .
@prefix hobbit: <http://w3id.org/hobbit/vocab#> .
@prefix gerbil2: <http://w3id.org/gerbil/hobbit/vocab#> .
@prefix nifsys: <http://w3id.org/hobbit/gerbil/nif-sys#&gt .

<http://www.maxpower.org/HyperNER> a  hobbit:SystemInstance; 
	rdfs:label            "HyperNER"@en;
	rdfs:comment	      "This is my own system defined in a simple way"@en;
	hobbit:imageName "git.project-hobbit.eu:4567/gerbil/gerbilnifsystemadapter";
	hobbit:implementsAPI  gerbil2:GerbilApiOKE2015T1;
	nifsys:instanceImageName "git.project-hobbit.eu:4567/maxpower/mysystem";
	hobbit:usesImage "git.project-hobbit.eu:4567/maxpower/mysystem";
	nifsys:numberOfInstances "3"^^xsd:unsignedInt;
	nifsys:webserviceUrl "http://HOST:8080/hyper-ner" .

Please replace

  • the URI <http://www.maxpower.org/HyperNER> with your own URI (it needs to be unique)
  • the label and comment with the name and a very short description of your annotation system
  • the URL git.project-hobbit.eu:4567/maxpower/mysystem with the name of your uploaded image in both lines, nifsys:instanceImageName and hobbit:usesImage
  • the URL http://HOST:8080/hyper-ner with the URL that should be used to call your web service (where HOST will be replaced with the host name at runtime)

2nd possibility: Java based system or system adapter

If your system is based on Java or you would like to implement a wrapper for your system in Java, we can provide you with a base implementation that is described in an article in the HOBBIT platform wiki. Additionally, you might want to use the code provided by the GERBIL project for implementing a NIF-based web service. Therefor, the gerbil.nif.transfer library needs to be included. The system adapters receiveGeneratedTask method could look as follows:

public void receiveGeneratedTask(String taskIdString, byte[] data) {
    TurtleNIFParser parser = new TurtleNIFParser();
    TurtleNIFWriter writer = new TurtleNIFWriter();
    List documents = parser.parseNIF(RabbitMQUtils.readString(data));
    try {
        Document document = documents.get(0);
        annotator.annotate(document);
        sendResultToEvalStorage(taskIdString, RabbitMQUtils.writeString(writer.writeNIF(documents)));
    } catch (Exception e) {
        // handle exception
    }
}

Where annotator.annotate(document) adds the named entities to the document. If your system is not already compatible to GERBIL, we recommend this way.

3rd possibility: Direct implementation of the API

If you want to use a different language to implement our NIF-based API, you need to implement the API of a system that can be benchmarked in HOBBIT. Every message of the task queue will be a single NIF-document. The response of your system has to be send to the result queue. Your system won’t receive data through the data queue.

Uploading the Docker image

The uploading of the Docker image is described in the Hobbit project platform wiki.

The system meta data file

Your system needs a system meta data file (called system.ttl). For participating in Task 1 – 2, this file can have the following content.

@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
@prefix hobbit: <http://w3id.org/hobbit/vocab#> .
@prefix gerbil2: <http://w3id.org/gerbil/hobbit/vocab#> .

<http://www.maxpower.org/HyperNER> a  hobbit:SystemInstance; 
	rdfs:label            "HyperNER"@en;
	rdfs:comment	      "This is my own system defined in a simple way"@en;
	hobbit:imageName      "git.project-hobbit.eu:4567/maxpower/mysystem";
	hobbit:implementsAPI  gerbil2:GerbilApiOKE2015T1 .

The URI of the system is used as identifier – it does not have to be dereferencable. The system is defined as a system instance, it has a label and a description. The two last lines are very important since they define the image that is used to run the system and the API the system implements. Please note that the Tasks 1 and 2 share the same API.

As described in the wiki page of the system meta data file it is possible to have several instances of a single system. Please, feel free to use this feature to adapt your system for the three different tasks.

Program & Accepted Papers

Challenge Session
Tuesday, June 5th, 2018
14:00 – 14:30
14:00 – 14:15 OKE Challenge Overview
Kleanthi Georgala
14:15 – 14:30 Hector Cerezo-Costas and Manuela Martın-Vicente, Relation Extraction for Knowledge Base Completion: A supervised approach
Closing Ceremony
Thursday, June 7th, 2019
17:30
 Announcement of challenge winners during the ESWC closing ceremony

For possible, last minute changes to the program please also check the ESWC 2018 program (https://2018.eswc-conferences.org/program/)

OKE2018 Challenge Results

The results can be found in HOBBIT platform: here (login as guest).

Organization

The organization responsibility will be shared by the following organizers who will comprise the Organizing Committee.

  • Axel-Cyrille Ngonga Ngomo, Paderborn University, Paderborn, Germany
  • René Speck, Leipzig University, Leipzig, Germany
  • Michael Röder, Paderborn Leipzig, Paderborn, Germany
  • Ricardo Usbeck, Paderborn University, Paderborn, Germany

The following set of people comprises the list of programme committee members. All the members of the list are experts from research and industry, who will review the paper submissions, independently of the organization team.

  • Andrea Nuzzolese, National Research Council Rome, Italy
  • Anna-Lisa Gentile, IBM Research, Germany
  • Jose Camacho-Collados, Sapienza University of Rome, Italy
  • Philipp Cimiano, Bielefeld University, Germany
  • Raphael Troncy, EURECOM, France
  • Sergio Consoli, Data Science Department at Philips Research, Germany
  • Borja Navarrao, University of Alicante, Spain
  • Elena Cabrio, University of Nice Sophia Antipolis, France
  • Michel Buffa, University of Nice Sophia Antipolis, France
  • Paola Velardi, University of Roma La Sapienza, Italy
  • Thierry Declerck, Saarland University, Germany
  • Valerio Basile, University of Rome La Sapienza, Italy

———————————————————————————————
CALL FOR PAPERS
———————————————————————————————

OKE 2018 – Open Knowledge Extraction Challenge

in conjunction with the 15th European Semantic Web Conference (ESWC 2018)
3rd-7th June 2018, Heraklion, Crete, Greece

———————————————————————————————
URL: https://2018.eswc-conferences.org/
URL: https://project-hobbit.eu/challenges/oke2018-challenge-eswc-2018/
email: OKE-contact@googlegroups.com
email: speck@informatik.uni-leipzig.de
———————————————————————————————

The OKE challenge invites researchers and practitioners from academia as well as industry to compete to the aim of pushing further the state of the art in knowledge extraction from text for the Semantic Web. The challenge has the ambition to provide a reference framework for research in this field by redefining a number of tasks typically from information and knowledge extraction by taking into account Semantic Web requirements and has the goal to test the performance of knowledge extraction systems.

This year, the challenge goes in the fourth round and consists of four tasks which include named entity identification, disambiguation by linking to a knowledge base as well as relation and knowledge extraction. The challenge makes use of small gold standard datasets that consist of manually curated documents and large silver standard datasets that consist of automatically generated synthetic documents. The performance measure of a participating system is twofold based on (1) Precision, Recall, F1-measure and on (2) Precision, Recall, F1-measure with respect to the runtime of the system.

This year, the challenge comprises the following tasks:
-Task 1: Focused Named Entity Identification and Linking
-Task 2: Broader Named Entity Identification and Linking
-Task 3: Relation Extraction
-Task 4: Knowledge Extraction

Participants will be expected to describe their solution and results on the training datasets over a 5 page paper. In particular, a short summary of the approach chosen, a link to the experimental results and an analysis of the strengths and weaknesses of the approach are expected.

———————————————————————————————
Important Dates
———————————————————————————————
– Paper submission deadline (5 pages): Friday, March 30th, 2018
– Challenge paper reviews: Thursday, April 26th, 2018
– Notification of authors and invitation to challenge: Monday, April 30th, 2018
– Camera ready papers (5 pages): Monday, May 14th, 2018
– Camera ready papers for the challenge post-proceedings (up to 15 pages): Friday, July 6th, 2018 (tentative deadline)

– Release of training data and detailed instructions: Monday, February 12, 2018
– Release of test dataset: Monday, May 7th, 2018
– Deadline for system submission: TBA
– Running of the systems: TBA

– Presentation of challenge results: During ESWC 2018
– Proclamation of winners: During ESWC 2018 closing ceremony

———————————————————————————————
Organization
———————————————————————————————
* Axel-Cyrille Ngonga Ngomo, Paderborn University, Paderborn, Germany
* René Speck, Leipzig University, Leipzig, Germany
* Michael Röder, Paderborn Leipzig, Paderborn, Germany
* Ricardo Usbeck, Paderborn University, Paderborn, Germany

For the complete list of organizers and program committee members, visit the challenge website.

———————————————————————————————
Further Information and Contact
———————————————————————————————-
For detailed information, including datasets and submission guidelines, please visit the challenge website: https://project-hobbit.eu/challenges/oke2018-challenge-eswc-2018/

Contact Email: OKE-contact@googlegroups.com OR speck@informatik.uni-leipzig.de
————————————————————————————–

The challenge proceedings are available by Springer here.