Open Knowledge Extraction (OKE) Open Challenge 2017-2018

Challenge Motivation

After a successful organization of the OKE challenge by the HOBBIT project (https://project-hobbit.eu/) at the ESWC 2017, HOBBIT is proud to announce that OKE will be launched in September as an open challenge. The OKE Open Challenge will ensure continuous participation and systems evaluation.
A monetary prize of at least 250€ will be provided.
Stay tuned to get ready to participate!       Express your interest!

 Challenge Overview

The aim of this challenge is to test the performance of knowledge extraction systems in aspects that are relevant for the Semantic Web. These include precision, recall and runtime. The challenge will test the systems against data derived from several datasets and will comprise the following tasks:

  • Task 1 Focused Named Entity Identification and Linking
  • Task 2 Broader Named Entity Identification and Linking
  • Task 3 Focused Musical Named Entity Recognition and Linking

We herewith invite system developers to participate in the aforementioned tasks. To ensure that the system results are comparable, we will provide the HOBBIT benchmarking platform for the generation of the final results to be included into the system publications. A specification of the hardware on which the benchmarks will be ran will be released in due course.

Prizes

The winner of the challenge will get a prize of at least 250€. Further prizes are being organized and will be announced on the challenge website.

Q&A

Contact Email: OKE-contact@googlegroups.com OR speck@informatik.uni-leipzig.de

 Important Dates

  1. Training data set and small test data (to communicate with HOBBIT) release: middle of August
  2. Evaluation: release of evaluation results in every two weeks from the end of September onwards
  3. Final system: end of January 2018
  4. Proclamation of winners: February 2018

Tasks and Training Data

The challenge will test the systems against data derived from several datasets and will comprise the following tasks:

  • Task 1 Focused Named Entity Identification and Linking
  • Task 2 Broader Named Entity Identification and Linking
  • Task 3 Focused Musical Named Entity Recognition and Linking

Small test data and training datasets for all three OKE tasks:

Task 1: http://hobbitdata.informatik.uni-leipzig.de/OKE_OC/Task1/
Task 2: http://hobbitdata.informatik.uni-leipzig.de/OKE_OC/Task2/
Τask 3: http://hobbitdata.informatik.uni-leipzig.de/OKE_OC/Task3/

The documentation for the three first tasks is the same as in the OKE2017 challenge (https://project-hobbit.eu/challenges/oke2017-challenge-eswc-2017/#tasks-and-data).

Each participant must provide a system that solves at least one of the tasks and a paper that describes this system. Both requirements are described in the following.

Note that if you have questions regarding the submission of the paper or the system feel free to write a mail to OKE-contact@googlegroups.com .

Technical requirements for participation

Each participant must provide a system as Docker image. This image has to be uploaded to the HOBBIT Gitlab (it is possible to use a private repository, i.e., the system will not be visible for other people). In general, the uploaded Docker image can contain either a) the system itself or b) a web service client that forwards requests to the system that is hosted by you. Note that we highly recommend the first solution since a web service client won’t enable you to take part in the scenario B of the tasks.

Implementing the API

To be able to benchmark your system, it needs to implement our NIF-based (e.g., using a wrapper). There are several scenarios how this can be achieved.

1st possibility: GERBIL compatible APIs

If your system already implements a NIF-based API that is compatible with the GERBIL benchmarking framework, you do not have to implement anything additional to that. You only need to provide a Docker image of your system that implements the same API as your original web service and an adapted version of the following system meta data file.

@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .
@prefix hobbit: <http://w3id.org/hobbit/vocab#> .
@prefix gerbil2: <http://w3id.org/gerbil/hobbit/vocab#> .
@prefix nifsys: <http://w3id.org/hobbit/gerbil/nif-sys#&gt .

<http://www.maxpower.org/HyperNER> a  hobbit:SystemInstance; 
	rdfs:label            "HyperNER"@en;
	rdfs:comment	      "This is my own system defined in a simple way"@en;
	hobbit:imageName "git.project-hobbit.eu:4567/gerbil/gerbilnifsystemadapter";
	hobbit:implementsAPI  gerbil2:GerbilApiOKE2015T1;
	nifsys:instanceImageName "git.project-hobbit.eu:4567/maxpower/mysystem";
	hobbit:usesImage "git.project-hobbit.eu:4567/maxpower/mysystem";
	nifsys:numberOfInstances "3"^^xsd:unsignedInt;
	nifsys:webserviceUrl "http://HOST:8080/hyper-ner" .

Please replace

  • the URI <http://www.maxpower.org/HyperNER> with your own URI (it needs to be unique)
  • the label and comment with the name and a very short description of your annotation system
  • the URL git.project-hobbit.eu:4567/maxpower/mysystem with the name of your uploaded image in both lines, nifsys:instanceImageName and hobbit:usesImage
  • the URL http://HOST:8080/hyper-ner with the URL that should be used to call your web service (where HOST will be replaced with the host name at runtime)

2nd possibility: Java based system or system adapter

If your system is based on Java or you would like to implement a wrapper for your system in Java, we can provide you with a base implementation that is described in an article in the HOBBIT platform wiki. Additionally, you might want to use the code provided by the GERBIL project for implementing a NIF-based web service. Therefor, the gerbil.nif.transfer library needs to be included. The system adapters receiveGeneratedTask method could look as follows:

public void receiveGeneratedTask(String taskIdString, byte[] data) {
    TurtleNIFParser parser = new TurtleNIFParser();
    TurtleNIFWriter writer = new TurtleNIFWriter();
    List documents = parser.parseNIF(RabbitMQUtils.readString(data));
    try {
        Document document = documents.get(0);
        annotator.annotate(document);
        sendResultToEvalStorage(taskIdString, RabbitMQUtils.writeString(writer.writeNIF(documents)));
    } catch (Exception e) {
        // handle exception
    }
}

Where annotator.annotate(document) adds the named entities to the document. If your system is not already compatible to GERBIL, we recommend this way.

 

3rd possibility: Direct implementation of the API

If you want to use a different language to implement our NIF-based API, you need to implement the API of a system that can be benchmarked in HOBBIT. Every message of the task queue will be a single NIF-document. The response of your system has to be send to the result queue. Your system won’t receive data through the data queue.

Uploading the Docker image

The uploading of the Docker image is described in the Hobbit project platform wiki.

The system meta data file

Your system needs a system meta data file (called system.ttl). For participating in Task 1 – 3, this file can have the following content.

@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
@prefix hobbit: <http://w3id.org/hobbit/vocab#> .
@prefix gerbil2: <http://w3id.org/gerbil/hobbit/vocab#> .

<http://www.maxpower.org/HyperNER> a  hobbit:SystemInstance; 
	rdfs:label            "HyperNER"@en;
	rdfs:comment	      "This is my own system defined in a simple way"@en;
	hobbit:imageName      "git.project-hobbit.eu:4567/maxpower/mysystem";
	hobbit:implementsAPI  gerbil2:GerbilApiOKE2015T1 .

The URI of the system is used as identifier – it does not have to be dereferencable. The system is defined as a system instance, it has a label and a description. The two last lines are very important since they define the image that is used to run the system and the API the system implements. Please note that the Tasks 1, 2 and 3 share the same API.

As described in the wiki page of the system meta data file it is possible to have several instances of a single system. Please, feel free to use this feature to adapt your system for the three different tasks.

Organization

  • Axel-Cyrille Ngonga Ngomo, Institute for Applied Informatics/University of Paderborn, Germany
  • René Speck, Institute for Applied Informatics, Germany
  • Sergio Oramas, Universitat Pompeu Fabra, Spain
  • Vassiliki Rentoumi, NCSR “Demokritos”, Greece
  • Michael Röder, Institute for Applied Informatics/University of Paderborn, Germany

————————————————————————————–
Open Knowledge Extraction (OKE) Open Challenge 2017-2018
————————————————————————————–
After a successful organization of the OKE challenge by the HOBBIT project (https://project-hobbit.eu/) at the ESWC 2017, HOBBIT is proud to announce that OKE will be launched in September as an open challenge.
The OKE Open Challenge will ensure continuous participation and systems evaluation.
Stay tuned to get ready to participate!

  1. A monetary prize of at least 250€ will be provided.
  2. Please express your interest in the OKE Open Challenge by adding your email in the link: https://project-hobbit.eu/open-challenges/oke-open-challenge/express-interest. You will receive further information and notifications regarding the progress of the OKE Open Challenge.

————————————————————————————-
OKE Open Challenge at a glance:
————————————————————————————-
The aim of this challenge is to test the performance of knowledge extraction systems in aspects that are relevant for the Semantic Web. These include precision, recall and runtime. The challenge will test the systems against data derived from several datasets and will comprise the following tasks:

* Task 1 Focused Named Entity Identification and Linking
* Task 2 Broader Named Entity Identification and Linking
* Task 3 Focused Musical Named Entity Recognition and Linking

We herewith invite system developers to participate in the aforementioned tasks. To ensure that the system results are comparable, we will provide the HOBBIT benchmarking platform for the generation of the final results to be included into the system publications. A specification of the hardware on which the benchmarks will be ran will be released in due course.

Small test data and training datasets for all three OKE tasks:
Task 1: http://hobbitdata.informatik.uni-leipzig.de/OKE_OC/Task1/
Task 2: http://hobbitdata.informatik.uni-leipzig.de/OKE_OC/Task2/
Τask 3: http://hobbitdata.informatik.uni-leipzig.de/OKE_OC/Task3/
The documentation for the three first tasks is the same as in the OKE2017 challenge (https://project-hobbit.eu/challenges/oke2017-challenge-eswc-2017/#tasks-and-data).

————————————————————————————–
Prizes
————————————————————————————–
The winner of the challenge will get a prize of at least 250€. Further prizes are being organized and will be announced on the challenge website.

  1. Training data set and small test data (to communicate with HOBBIT) release: middle of August
  2. Evaluation: release of evaluation results in every two weeks from the end of September onwards
  3. Final system: end of January 2018
  4. Proclamation of winners: February 2018

————————————————————————————–
Organization
————————————————————————————–
* Axel-Cyrille Ngonga Ngomo, Institute for Applied Informatics/University of Paderborn, Germany
* René Speck, Institute for Applied Informatics, Germany
* Sergio Oramas, Universitat Pompeu Fabra, Spain
* Vassiliki Rentoumi, NCSR “Demokritos”, Greece
* Michael Röder, Institute for Applied Informatics/University of Paderborn, Germany

————————————————————————————–
Further Information and Contact
————————————————————————————–
For detailed information, including datasets and submission guidelines,
please visit the challenge website: https://project-hobbit.eu/open-challenges/oke-open-challenge/
Contact Email: OKE-contact@googlegroups.com OR speck@informatik.uni-leipzig.de