During the 3rd HOBBIT plenary meeting, which took place in Athens on the 1st and 2nd of February 2017, we summarized the current state of affairs. We have been working hard towards the project’s objectives. We are on schedule to:
- Organize five benchmarking challenges and obtain first results by Summer-Autumn 2017
- Establish the HOBBIT association by December 2017
- Release benchmarks and baseline systems in the HOBBIT platform by the start of April 2017
The aim of the plenary was to summarize the efforts carried out so far. Consortium members presented the current state of affairs for each work package and planned the next steps for HOBBIT’s immediate as well as long term goals. We took decisions on essential technical and practical matters, made further progress on the creation of the HOBBIT association and took further steps on more effectively disseminating the results of the project.
We are proud to announce that we will be organizing the first HOBBIT evaluation campaigns that will take place during the Summer-Autumn of 2017, and that the first version of HOBBIT’s platform is ready.
Regarding the challenges, we will be organizing challenges:
- the DEBS Grand Challenge
- the Instance Matching Benchmark Challenge for Spatial Data at the OAEI workshop at ISWC 2017
- the Mighty Storage Challenge at ESWC 2017
- the OKE challenge at ESWC 2017
- the QALD challenge at ESWC 2017
More details about the challenges can be found at the project’s website https://project-hobbit.eu/challenges/ .
All data generators are now ready and all the mimicking algorithms have been integrated in the HOBBIT platform. Apart from the ability to generate Twitter, molding machine and transportation data, we can now generate additional datasets such as publishing datasets as well as a transport dataset of linked connections and a dataset of DBpedia queries.
DataGen, the HOBBIT extension of the LDBC Social Network Benchmark generator,will be used to produce synthetic RDF datasets for the publishing domain. Those datasets will be used for HOBBIT’s storage benchmark. Additionally, the Semantic Publishing Benchmark (SPB) generator will be used to produce datasets of arbitrary size to be used in HOBBIT’s versioning benchmark. The structure of the latter dataset is in line with real-world RDF datasets, unlike the LDBC Social Network Benchmark dataset, that was designed to be more generic and very well structured. Further, the transport dataset of linked connections is provided by a data generator and consists of train connections modelled using the transport ontology following the GTFS (General Transit Feed Specification) standards. Finally, a multilingual dataset consisting of several thousands of automatically generated queries will be generated with the DBpedia benchmark generator.
HOBBIT plans to be present at various events, both industrial and academic across Europe, the US and Canada. We are also preparing a special issue on Benchmarking Linked Data for the Semantic Web Journal (SWJ). For more information you can visit the following link: https://project-hobbit.eu/special-issue-on-benchmarking-linked-data-at-semantic-web-interoperability-usability-applicability-an-ios-press-journal/ .
The goal of the HOBBIT association is to provide industry-grade benchmarks for Big Linked Data (BLD). The association plans to offer various membership types depending on companies’ specific needs. It will work hand in hand with other European benchmarking associations such as LDBC. The association will provide periodic open reports comparing system performances for each of the BLD steps. It will be financed by member fees, donations and the implementation of dedicated benchmarks for companies. These financial resources will be used to maintain the platform and organize further challenges for BLD even after the completion of the HOBBIT project.
We believe that the HOBBIT association will play a key role in disseminating and extending the HOBBIT benchmarking platform (including benchmarks, KPIs, use cases and datasets). Towards this direction, we intend to invite various professional bodies and associations to become members. With regards to the HOBBIT association the first version of the survey, through which we aim to collect the requirements and needs of the industry with respect to benchmarking technologies, is ready and is going to be sent out to high ranked people from the industry domain who are interested in HOBBIT related issues.
We also intend to include various engagement levels as well as free memberships to the association subscribers. We will initiate a new survey to gather information from the industry and other potential partners regarding frameworks used to evaluate their software.
Authors: Vassiliki Rentoumi, Grigorios Tzortzis, Irini Fundulaki, Axel Ngonga