SQA2018 Challenge – ESWC 2018

Challenge Motivation & Overview

After the successful organization of the QALD-7 challenge at the ESWC 2017 and the QALD-8 challenge incorporated in the NLIWoD workshop at ISWC 2017, with more than 50 participants, the HOBBIT project is proud to announce that the Scalable Question Answering Challenge (SQA) over Linked Data will be hosted at ESWC 2018.

The key challenge for Scalable Question Answering over Linked Data is the need to translate a user’s information request into such a form that it can be efficiently evaluated using standard Semantic Web query processing and inferencing techniques.
Therefore, the main task of SQA Challenge is the following: given an RDF dataset and and a large volume of natural language questions or keywords, return the correct answers (or SPARQL queries that retrieves those answers).

In the Scalable Question Answering Challenge (SQA) users can display the capabilities of their systems using the HOBBIT platform (https://project-hobbit.eu/outcomes/hobbit-platform).

Stay tuned to get ready to participate!       Express your interest!

Q&A

For more information send an e-mail to: Giulio.Napolitano@iais.fraunhofer.de

Important Dates

Challenge papers submission deadline (5 pages document): Friday, April 4th, 2018 (extended)
Challenge paper reviews: Thursday, April 26th, 2018 (extended)
Notifications sent to participants and invitations to submit task results: Monday, April 30th, 2018 (extended)
Test data (and other participation tools) published: Monday, May 7th, 2018 (extended)
Camera ready papers for the conference (5 pages document): Monday, May 14th, 2018 (extended)
Instructions for evaluation: Monday, February 12th, 2018
Deadline for submission of system answers (cut-off): Friday June 1st, 2018
Camera ready papers for the challenge proceedings
(up to 15 pages):
Friday, July 6th, 2018
(tentative deadline)
Proclamation of winners: During ESWC2018 closing ceremony

Tasks & Training Data

The key challenge for Scalable Question Answering over Linked Data is the need to translate a user’s information request into such a form that it can be efficiently evaluated using standard Semantic Web query processing and inferencing techniques.

Therefore, the main task of SQA Challenge is the following: given an RDF dataset and and a large volume of natural language questions or keywords, return the correct answers (or SPARQL queries that retrieves those answers).

The ranking of the systems is based on the usual KPIs (precision, recall and F measure) plus a “response power” measure, which is also taking into account the ability of the systems to cope with high volume demand without failure. The response power is the harmonic mean of three measures: precision, recall and the ratio between processed questions (an empty answer is considered as processed, a missing answer is considered as unprocessed) and total number of questions sent to the system. The final ranking is on response power, precision, recall and F measure in this order.

Training dataset and documentation are available:

In the Scalable Question Answering Challenge (SQA) users can display the capabilities of their systems using the HOBBIT platform (https://project-hobbit.eu/outcomes/hobbit-platform).

Participants will be expected to describe their solution and results on the training datasets over a 5 page paper. In particular, a short summary of the approach chosen, a link to the experimental results and an analysis of the strengths and weaknesses of the approach are expected.

Prerequisites for participation

Each participant must provide a system that solves the Scalable Question Answering task and a paper that describes this system. Both requirements are described in the following.
Note that if you have questions regarding the submission of the paper or the system feel free to write an email to giulio.napolitano@iais.fraunhofer.de.

Registration and Submission

  • All challenge papers should be exactly five (5) pages in length in PDF file format and written in English.
  • In order to prepare their manuscripts, authors should follow Springer’s Lecture Notes in Computer Science (LNCS) style. For details and templates see Springer’s Author Instructions.
  • Paper submissions will be handled electronically via the EasyChair conference management system, available at the following address: https://easychair.org/conferences/?conf=sqa2018
  • Papers must be submitted no later than Friday, March 30th, 2018  (extended), 23:59 Hawaii Time.
  • Each submission will be peer-reviewed by members of the challenge program committee.  Papers will be evaluated according to their significance, originality, technical content, style, clarity, and relevance to the challenge.
  • Proceedings will be published by Springer in LNCS volume.
  • After the conference, challenge participants will be able to provide a detailed description of their system and evaluation results in a longer version of their paper (up to 15 pages). This longer paper will be included in the challenge proceedings.

Technical requirements for system submission

Each participant must provide a system as Docker image. This image has to be uploaded to the HOBBIT Gitlab (it is possible to use a private repository, i.e., the system will not be visible for other people). In general, the uploaded Docker image can contain either a) the system itself or b) a web service client that forwards requests to the system that is hosted by you. Note, general information can be found in our HOBBIT wiki. Do not hesitate to contact us if you have any questions.

Implementing the API

To be able to benchmark your system, it needs to implement our QALD-JSON-based format (e.g., using a wrapper). There are several scenarios how this can be achieved.

1st possibility: GERBIL-QA compatible APIs

If your system already implements an API that is compatible with the GERBIL QA benchmarking framework, you do not have to implement anything additional to that. You only need to provide a Docker image of your system that implements the same API as your original web service and send us a description how it has to be started (e.g., which environmental variables have to be defined).

A system is compatible to GERBIL QA if it is either one of the systems that are already available in GERBIL QA or it is a QALD-JSON-based web service that can be benchmarked with GERBIL QA.

2nd possibility: Java-based system or system adapter

Requirements
  • You can write Java code and you are familiar with maven.
  • You have a system written in Java (or at least a client for the system).
  • You have Docker installed
  • You can find an example implementation here:
    https://github.com/hobbit-project/hobbitqaexample
1. Write a System Adapter for your system
  • Create a new maven-Project in your favorite IDE.
  • Add the following to your pom.xml
<repositories>
     <repository>
             <id>maven.aksw.internal</id>
             <name>University Leipzig, AKSW Maven2 Repository</name>
             <url>http://maven.aksw.org/repository/internal</url>
     </repository>
     <repository>
             <id>maven.aksw.snapshots</id>
             <name>University Leipzig, AKSW Maven2 Repository</name>
             <url>http://maven.aksw.org/repository/snapshots</url>
     </repository>
        </repositories>
        <dependencies>
     <dependency>
             <groupId>org.hobbit</groupId>
             <artifactId>core</artifactId>
             <version>1.0.0</version>
     </dependency>
     <dependency>
         <groupId>org.aksw.qa</groupId>
         <artifactId>commons</artifactId>
         <version>0.4.6</version>
         <exclusions>
             <exclusion>
                 <groupId>org.aksw.qa</groupId>
                 <artifactId>datasets</artifactId>
             </exclusion>
         </exclusions>
     </dependency>
        <!-- Add your System here -->
        <!-- Add a slf4j log binding here -->
 </dependencies>
  <build>
     <plugins>
          <plugin>
                 <groupId>org.apache.maven.plugins</groupId>
                 <artifactId>maven-shade-plugin</artifactId>
                 <version>2.4.3</version>
                 <configuration>
                     <!-- filter all the META-INF files of other artifacts -->
                     <filters>
                       <filter>
                          <artifact>*:*</artifact>
                          <excludes>
                             <exclude>META-INF/*.SF</exclude>
                             <exclude>META-INF/*.DSA</exclude>
                            <exclude>META-INF/*.RSA</exclude>
                         </excludes>
                     </filter>
                  </filters>
                  <transformers>
                     <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                         <manifestEntries>
                            <X-Compile-Source-JDK>${maven.compile.source}</X-Compile-Source-JDK>
                            <X-Compile-Target-JDK>${maven.compile.target}</X-Compile-Target-JDK>
                         </manifestEntries>
                     </transformer>
                     <transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer" />
                  </transformers>
                </configuration>
                <executions>
                  <execution>
                    <phase>package</phase>
                    <goals>
                       <goal>shade</goal>
                    </goals>
                  </execution>
               </executions>
        </plugin>
     </plugins>
  </build>
  • Add your system as dependency.
2. Now, add a SystemAdapter:
  • Add following as Source file to your maven project:

package org.example;
public class ExampleSystemAdapter extends AbstractSystemAdapter {
        private MyAnnotator myAnnotator;

    public void init() throws Exception {
        super.init();
        myAnnotator= new MyAnnotator();
         // Your initialization code comes here...
        // You can access the RDF model this.systemParamModel to retrieve meta data about this system adapter
    }

    /**
     * You MIGHT need this, depends on the benchmark.
     * @see <a href="https://project-hobbit.eu/challenges">Challenges and their Benchmarks</a>
     */
    public void receiveGeneratedData(byte[] data) {
        // handle the incoming data as described in the benchmark description
        }

     /**
     *  Create results for the incoming data.
     *  The structure of the incoming data should be defied by the <a href="https://project-hobbit.eu/challenges">Challenges and their Benchmarks</a>
     *
     *  e.g If you want to benchmark against the QALD-Challenge,
     *  you can expect the incoming data to be questions. Accordingly,
     *  your result[] output should be answers.
     *  The data structure of the incoming and outgoing data should follow
     *  the QALD-Json format. You can find <a href="https://github.com/AKSW/NLIWOD/tree/master/qa.commons/src/main/java/org/aksw/qa/commons/load">here</a>
     *   a loder and parser and a complete class structure for QALD-Json.
     *   These are already included as dependency.
     * 
     *
     *  @see <a href="https://github.com/hobbit-project/platform/wiki/Develop-a-system-adapter#the-task-queue">The Task Queue and structure of data[]</a>
     *
     */
    public void receiveGeneratedTask(String taskId, byte[] data) {
        //Here is where your system has to do its job.
        byte[] result = myAnnotator.annotate(data);

                // Send the result to the evaluation storage
                try {
                        sendResultToEvalStorage(taskId, result);
                } catch (IOException e) {
                        //Log the error
                }
        }

    @Override
        public void close() throws IOException {
        // Free the resources you requested here
        // Always close the super class after yours!
        super.close();
    }
}
  • Where the Class MyAnnotator refers to your System.
  • Be sure to implement a proper init() and close() along with your program logic in recieveGeneratedTask()
  • Now you are ready to build your project: Use your IDE or run mvn package from command line in the root directory of your maven project
  • You should now have two jar’s in your (project_root)/target/ folder: The original and the shaded one. If you can’t differentiate the two: Look at their filesize. The shaded one is usually by far larger. Also, the not-shaded one has prefix original-.
3. Create an account for HOBBIT
4. Docker it
  • Create a new file at your project root with the name Dockerfile
  • The contents of this file should look like this:
FROM java
ADD target/<MyShadedJar.jar> /<MyAnnotator>/<MyShadedJar.jar>
WORKDIR /<MyAnnotator>
CMD java -cp <MyShadedJar.jar> org.hobbit.core.run.ComponentStarter org.example.ExampleSystemAdapter
5. Create a system.ttl file

Here is an example of a basic file (see also https://github.com/hobbit-project/platform/wiki/System-meta-data-file):

@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
@prefix hobbit: <http://w3id.org/hobbit/vocab#> .
 
<http://www.example.org/exampleSystem/MySystem> a  hobbit:SystemInstance;
rdfs:label        "MySystem"@en;
rdfs:comment        "A short description of MySystem"@en;
hobbit:imageName "git.project-hobbit.eu:4567/myhobbitusername/myprojectname";
hobbit:implementsAPI <http://benchmark.org/MyNewBenchmark/BenchmarkApi> 

Where http://benchmark.org/MyNewBenchmark/BenchmarkApi refers to the benchmark your system should be tested with. You can find those URIs at https://project-hobbit.eu/challenges.

Now, push the system.ttl to your gitlab project https://git.project-hobbit.eu/MyHobbitUsername/MyProject , or add it by hand to your gitlab project root.

6. Run a Benchmark

Go to http://master.project-hobbit.eu/ and run a benchmark on your system.

There is a detailed description in the platform wiki that you might want to reuse or refer to https://github.com/hobbit-project/platform/wiki/Experiments

Where to go now?

You can find a very detailed description of each step and in-depth information at:
https://github.com/hobbit-project/platform/wiki

Look at the example implementation at:
https://github.com/hobbit-project/hobbitqaexample

3rd possibility: Direct implementation of the API

If you want to use a different language to implement our QALD-JSON-based API, you need to implement the API of a system that can be benchmarked in HOBBIT. Every message of the task queue will be a single QALD-JSON-document. The response of your system has to be send to the result queue. Your system won’t receive data through the data queue.

Uploading the Docker image

The uploading of the Docker image is described in the Hobbit project platform wiki.

Program & Accepted Papers

TBA

Organization

The organization responsibility will be shared by the following organizers who will comprise the Organizing Committee.

  • Giulio Napolitano, IAIS Fraunhofer, Germany
  • Axel-Cyrille Ngonga Ngomo, University of Paderborn, Germany
  • Ricardo Usbeck, University of Paderborn, Germany

The following set of people comprises the list of programme committee members. All the members of the list are experts from research and industry, who will review the paper submissions, independently of the organization team.

  • Corina Forascu, Alexandru Ioan Cuza University, Iasi, Romania
  • Sebastian Walter,  Semalytix GmbH, Bielefeld, Germany
  • Bernd Müller, ZBMed, Germany
  • Christoph Lange, Fraunhofer Gesellschaft, Germany
  • Dennis Diefenbach, Université de Saint-Etienne, France
  • Edgard Marx, AKSW, University Leipzig, Germany
  • Hady Elsahar, Université de Saint-Etienne, France
  • Harsh Thakkar, University of Bonn, Germany
  • Ioanna Lytra, University of Bonn, Germany
  • John McCrae, INSIGHT – The Centre for Data Analytics, Ireland
  • Kuldeep Singh, University of Bonn, Germany
  • Saeedeh Shekarpour, Kno.e.sis Center, Ohio Center of Excellence in Knowledge-enabled Computing, USA
  • Sherzod Hakimov, CITEC, Bielefeld University, Germany
  • Elena Cabrio, University of Nice Sophia Antipolis, France
  • Philipp Cimiano, CITEC, Bielefeld University, Germany
  • Vanessa Lopez, IBM Research, Dublin, Ireland
  • André Freitas, University Passau, Germany
  • Elena Demidova, University of Southampton
  • Jin-Dong Kim, Database Center for Life Science (DBCLS), Japan
  • Key-Sun Choi, KAIST, Korea
  • Mohnish Dubey, Fraunhofer Gesellschaft, Germany

———————————————————————————————
CALL FOR PAPERS
———————————————————————————————

SQA 2018 – Scalable Question Answering

in conjunction with the 15th European Semantic Web Conference (ESWC 2018)
3rd-7th June 2018, Heraklion, Crete, Greece

———————————————————————————————
URL: https://2018.eswc-conferences.org/
URL: https://project-hobbit.eu/open-challenges/sqa-challenge-eswc-2018/
email: Giulio.Napolitano@iais.fraunhofer.de
———————————————————————————————

The key challenge for Scalable Question Answering over Linked Data is the need to translate a user’s information request into such a form that it can be efficiently evaluated using standard Semantic Web query processing and inferencing techniques.

Therefore, the main task of SQA Challenge is the following: given an RDF dataset and and a large volume of natural language questions or keywords, return the correct answers (or SPARQL queries that retrieves those answers).

Training dataset and documentation are available:
* Training dataset: https://figshare.com/articles/LC-QuAD_QALDformat/5818452
* Description: http://hobbitdata.informatik.uni-leipzig.de/SQAOC/Documentation.txt

In the Scalable Question Answering Challenge (SQA) users can display the capabilities of their systems using the HOBBIT platform (https://project-hobbit.eu/outcomes/hobbit-platform) provided by the homonymous H2020 project.

Participants will be expected to describe their solution and results on the training datasets over a 5 page paper. In particular, a short summary of the approach chosen, a link to the experimental results and an analysis of the strengths and weaknesses of the approach are expected.

———————————————————————————————
Important Dates
———————————————————————————————
– Challenge papers submission deadline (5 pages document): Friday March 30th, 2018
– Challenge paper reviews: Thursday, April 26th, 2018
– Notifications sent to participants and invitations to submit task results: Monday April 30th, 2018
– Test data (and other participation tools) published: Monday, May 7th, 2018
– Camera ready papers for the conference (5 pages document): Monday, May 14th, 2018
– Instructions for evaluation: Monday February 12th, 2018
– Deadline for submission of system answers (cut-off): Friday June 1st, 2018
– Camera ready papers for the challenge proceedings (up to 15 pages): Friday, July 6th, 2018 (tentative deadline)

– Proclamation of winners: During ESWC2018 closing ceremony

———————————————————————————————
Organization
————————————————————————————–
* Giulio Napolitano, IAIS, Fraunhofer, Germany
* Axel-Cyrille Ngonga Ngomo, University of Paderborn, Germany
* Ricardo Usbeck, University of Paderborn, Germany

————————————————————————————–
Further Information and Contact
————————————————————————————–
For detailed information including datasets and submission guidelines, please visit
the challenge website: https://project-hobbit.eu/open-challenges/sqa-challenge-eswc-2018/
Contact Email: Giulio.Napolitano@iais.fraunhofer.de

TBA