Important Dates

Paper submission deadline:	Monday March 20th, 2017
Challenge paper reviews:	Tuesday April 5th, 2017
Paper Notifications and invitation to task:	Friday April 7th, 2017
Camera ready papers (5 pages document):	Sunday April 23rd, 2017

Release of training data and instructions:	Friday January 13th, 2017
Release of test dataset:	Friday April 7th, 2017
Deadline for system submission:	Sunday April 30th, 2017
Running of the systems:	Monday May 15th, 2017
Results:	Monday May 22nd, 2017
Presentation of challenge results:	Thursday June 1st, 2017

Camera ready papers for the challenge proceedings (up to 15 pages):	Friday June 30th, 2017 (tentative deadline)
Proclamation of winners:	During ESWC2017 closing ceremony

Tasks and Training Data

• Task 1: Focused NE Identification and Linking
• Task 2: Broader NE Identification and Linking
• Task 3: Focused Musical NE Recognition and Linking
• Task 4: Knowledge Extraction

Each participant must provide a system that solves at least one of the tasks and a paper that describes this system. Both requirements are described in the following.

Note that if you have questions regarding the submission of the paper or the system feel free to write a mail to OKE-contact@googlegroups.com .

Registration and Submission

• All challenge papers should be exactly five (5) pages in length in PDF file format and written in English.
• In order to prepare their manuscripts, authors should follow Springer’s Lecture Notes in Computer Science (LNCS) style. For details and templates see Springer’s Author Instructions.
• Paper submissions will be handled electronically via the EasyChair conference management system, available at the following address: https://easychair.org/conferences/?conf=oke2017.
• Papers must be submitted no later than Monday March 20th, 2017, 23:59 Hawaii Time.
  NOTE: Eligible to submit papers are only authors participating in the challenge.
• Each submission will be peer-reviewed by members of the challenge program committee.  Papers will be evaluated according to their significance, originality, technical content, style, clarity, and relevance to the challenge.
• Each participant must provide his/her solution as a docker image that abides by the technical specifications.
• Proceedings will be published by Springer in LNCS volume.
• After the conference, challenge participants will be able to provide a detailed description of their system and evaluation results in a longer version of their paper (up to 15 pages). This longer paper will be included in the challenge proceedings.

Technical requirements for participation

Each participant must provide a system as Docker image. This image has to be uploaded to the HOBBIT Gitlab (it is possible to use a private repository, i.e., the system will not be visible for other people). In general, the uploaded Docker image can contain either a) the system itself or b) a web service client that forwards requests to the system that is hosted by you. Note that we highly recommend the first solution since a web service client won’t enable you to take part in the scenario B of the tasks.

Implementing the API

To be able to benchmark your system, it needs to implement our NIF-based (e.g., using a wrapper). There are several scenarios how this can be achieved.

1st possibility: GERBIL compatible APIs

If your system already implements a NIF-based API that is compatible with the GERBIL benchmarking framework, you do not have to implement anything additional to that. You only need to provide a Docker image of your system that implements the same API as your original web service and an adapted version of the following system meta data file.

@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .
@prefix hobbit: <http://w3id.org/hobbit/vocab#> .
@prefix gerbil2: <http://w3id.org/gerbil/hobbit/vocab#> .
@prefix nifsys: <http://w3id.org/hobbit/gerbil/nif-sys#&gt .

<http://www.maxpower.org/HyperNER> a  hobbit:SystemInstance; 
	rdfs:label            "HyperNER"@en;
	rdfs:comment	      "This is my own system defined in a simple way"@en;
	hobbit:imageName "git.project-hobbit.eu:4567/gerbil/gerbilnifsystemadapter";
	hobbit:implementsAPI  gerbil2:GerbilApiOKE2015T1;
	nifsys:instanceImageName "git.project-hobbit.eu:4567/maxpower/mysystem";
	hobbit:usesImage "git.project-hobbit.eu:4567/maxpower/mysystem";
	nifsys:numberOfInstances "3"^^xsd:unsignedInt;
	nifsys:webserviceUrl "http://HOST:8080/hyper-ner" .

Please replace

• the URI <http://www.maxpower.org/HyperNER> with your own URI (it needs to be unique)
• the label and comment with the name and a very short description of your annotation system
• the URL git.project-hobbit.eu:4567/maxpower/mysystem with the name of your uploaded image in both lines, nifsys:instanceImageName and hobbit:usesImage
• the URL http://HOST:8080/hyper-ner with the URL that should be used to call your web service (where HOST will be replaced with the host name at runtime)

2nd possibility: Java based system or system adapter

If your system is based on Java or you would like to implement a wrapper for your system in Java, we can provide you with a base implementation that is described in an article in the HOBBIT platform wiki. Additionally, you might want to use the code provided by the GERBIL project for implementing a NIF-based web service. Therefor, the gerbil.nif.transfer library needs to be included. The system adapters receiveGeneratedTask method could look as follows:

public void receiveGeneratedTask(String taskIdString, byte[] data) {
    TurtleNIFParser parser = new TurtleNIFParser();
    TurtleNIFWriter writer = new TurtleNIFWriter();
    List documents = parser.parseNIF(RabbitMQUtils.readString(data));
    try {
        Document document = documents.get(0);
        annotator.annotate(document);
        sendResultToEvalStorage(taskIdString, RabbitMQUtils.writeString(writer.writeNIF(documents)));
    } catch (Exception e) {
        // handle exception
    }
}

Where annotator.annotate(document) adds the named entities to the document. If your system is not already compatible to GERBIL, we recommend this way.

 

3rd possibility: Direct implementation of the API

If you want to use a different language to implement our NIF-based API, you need to implement the API of a system that can be benchmarked in HOBBIT. Every message of the task queue will be a single NIF-document. The response of your system has to be send to the result queue. Your system won’t receive data through the data queue.

Uploading the Docker image

The uploading of the Docker image is described in the Hobbit project platform wiki.

The system meta data file

Your system needs a system meta data file (called system.ttl). For participating in Task 1 – 3, this file can have the following content.

@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
@prefix hobbit: <http://w3id.org/hobbit/vocab#> .
@prefix gerbil2: <http://w3id.org/gerbil/hobbit/vocab#> .

<http://www.maxpower.org/HyperNER> a  hobbit:SystemInstance; 
	rdfs:label            "HyperNER"@en;
	rdfs:comment	      "This is my own system defined in a simple way"@en;
	hobbit:imageName      "git.project-hobbit.eu:4567/maxpower/mysystem";
	hobbit:implementsAPI  gerbil2:GerbilApiOKE2015T1 .

The URI of the system is used as identifier – it does not have to be dereferencable. The system is defined as a system instance, it has a label and a description. The two last lines are very important since they define the image that is used to run the system and the API the system implements. Please note that the Tasks 1, 2 and 3 share the same API.

As described in the wiki page of the system meta data file it is possible to have several instances of a single system. Please, feel free to use this feature to adapt your system for the three different tasks.

Program & Accepted Papers

Challenge Session Tuesday, May 30th, 2017 15:00 – 15:30
15:00 – 15:10	OKE Challenge Overview Michael Röder
15:10 – 15:20	Julien Plu, Raphaël Troncy and Giuseppe Rizzo, ADEL@OKE2017: A Generic Method for Indexing Knowlege Bases for Entity Linking
15:20 – 15:30	Raabia Asif and Muhammad Abdul Qadir, “HmaraNER”: A Named Entity Recognizer and Linker
Posters and Demos Session June 1st, 2017 9:00 – 11:00
The ADEL system will be presented as poster
Closing Ceremony Thursday, June 1st, 2017 17:30
Announcement of challenge winners during the ESWC closing ceremony

For possible, last minute changes to the program please also check the ESWC 2017 program (http://2017.eswc-conferences.org/program)

OKE 2017 Challenge Results

The results are described in Deliverable 7.3.1 and can be found in HOBBIT platform: here (login as guest).

Organization

The organization responsibility will be shared by the following four main organizers:

• René Speck, University of Leipzig, Germany
  Expertise: Knowledge extraction, Named Entitiy Recognition, Relation Extraction
  Website: http://aksw.org/RenSpeck
  Email: speck@informatik.uni-leipzig.de
• Michael Röder, University of Leipzig, Germany
  Expertise: Benchmarking and Topic Modeling
  Website: http://aksw.org/MichaelRoeder
  Email: roeder@informatik.uni-leipzig.de
• Ricardo Usbeck, University of Leipzig, Germany
  Expertise: Knowledge extraction and Question Answering
  Website: http://aksw.org/RicardoUsbeck
  Email: usbeck@informatik.uni-leipzig.de
• Axel-Cyrille Ngonga Ngomo, Institute for Applied Informatics, Germany
  Expertise: Knowledge extraction, Machine Learning, Question Answering, Information Retrieval
  Website:  http://aksw.org/AxelNgonga
  Email: ngonga@informatik.uni-leipzig.de
• Horacio Saggion, Universitat Pompeu Fabra, Spain
  Expertise: Natural Language Processing and Human Language Technology
  Website: http://www.taln.upf.edu/users/hsaggion
  Email: horacio.saggion@upf.edu
• Luis Espinosa-Anke, Universitat Pompeu Fabra, Spain
  Expertise: Natural Language Processing and Information Extraction
  Website: http://www.taln.upf.edu/users/lespinosa
  Email: uis.espinosa@upf.edu
• Sergio Oramas, Universitat Pompeu Fabra, Spain
  Expertise: Information Extraction and Music Information Retrieval
  Email: sergio.oramas@upf.edu

The following set of people comprises the list of programme committee members. All the members of the list are experts from research and industry, who will review the paper submissions, independently of the organization team.

• Andrea Nuzzolese, National Research Council Rome, IT
• Anna-Lisa Gentile, University of Mannheim, DE
• Davide Buscaldi, University Paris 13s, FR
• Francesco Barbieri, Universitat Pompeu University, ES
• Johannes Hoffar, Max Planck Institute, DE
• José Camacho-Collados, Sapienza University of Rome, IT
• Luciano Del Corro, Max Planck Institute, DE
• Philipp Cimiano, Bielefeld University, DE
• Raphaël Troncy, EURECOM, FR
• Sergio Consoli, Data Science Department at Philips Research, DE
• Xavier Serra, Universitat Pompeu Fabra, ES

CALL FOR PAPERS

————————————————————————
OKE 2017 – Open Knowledge Extraction Challenge
————————————————————————
URL: https://project-hobbit.eu/challenges/oke2017-challenge-eswc-2017/
Email: OKE-contact@googlegroups.com

May 28th to June 1st 2017, Portoroz, Slovenia

in conjunction with the 14th European Semantic Web Conference (ESWC 2017, http://2017.eswc-conferences.org)
————————————————————————————–

The aim of this challenge is to test the performance of knowledge extraction systems in
aspects that are relevant for the Semantic Web. These include precision, recall and
runtime. The challenge will test the systems against data derived from real datasets.
We herewith invite system developers to participate in the aforementioned tasks.
The system developers are invited to write papers (5 page documents) presenting their results on the training data
(see important dates below). To ensure that the system results are comparable, we will
provide the HOBBIT benchmarking platform for the generation of the final results to be
included into the system publications. A specification of the hardware on which the
benchmarks will be ran will be released in due course.

This year, the challenge comprises the following tasks:
* Task 1: Focused Named Entity Identification and Linking
* Task 2: Broader Named Entity Identification and Linking
* Task 3: Focused Musical Named Entity Recognition and Linking
* Task 4: Knowledge Extraction

————————————————————————————–
Important Dates
————————————————————————————–
* Paper submission deadline (5 pages document): March 10th, 2017, 23:59 Hawaii Time**
* Notification of acceptance: April 7th, 2017
* Camera ready papers (5 pages document): April 23rd, 2017
* Deadline for submission of system answers/instructions for evaluation: TBA
* Release of evaluation results: TBA
* Proclamation of winners: During ESWC 2017 closing ceremony

**Eligible to submit papers are only authors participating in the challenge.

————————————————————————————–
Organization
————————————————————————————–
* René Speck, University of Leipzig, Germany
* Michael Röder, University of Leipzig, Germany
* Ricardo Usbeck, University of Leipzig, Germany
* Axel-Cyrille Ngonga Ngomo, Institute for Applied Informatics, Germany
* Horacio Saggion, Universitat Pompeu Fabra, Spain
* Luis Espinosa-Anke, Universitat Pompeu Fabra, Spain
* Sergio Oramas, Universitat Pompeu Fabra, Spain

For the complete list of organizers and program committee members,
visit the challenge website.

————————————————————————————–
Further Information and Contact
————————————————————————————–
For detailed information, including datasets and submission guidelines,
please visit the challenge website: https://project-hobbit.eu/challenges/oke2017-challenge-eswc-2017/
Contact Email: OKE-contact@googlegroups.com
————————————————————————————–

The challenge outcome overview can be found here, while challenge proceedings are available by Springer here.