The DEBS Grand Challenge 2018 was successfully executed on the HOBBIT online platform. Analytical and machine learning algorithms running in a distributed streaming manner are in the focus of Grand Challenge series, which is a joint event with the annual Distributed Event-Based Systems (DEBS) Conference. During the competition participating teams have to demonstrate their capabilities to implement their solutions as a distributed stream processing system, which differs the scope of Grand Challenge (e.g. from Kaggle, which technically is batch processing). AGT International co-organized the Grand Challenge for the second time, developed a benchmark as well as a number tools to assist participants with integration of their solutions into the HOBBIT platform, which helped to increase the number of total submitted systems.
The Grand Challenge the DEBS GC 2018 is rather oriented on machine learning community and focuses on maritime surveillance domain. The anonymized dataset was provided by the MarineTraffic portal and contained about 1 million of tracks (geo coordinates, course, speed and other relevant information) received from automatic identification systems (AIS) that are installed on real ships. Within the challenge analytical systems were expected to sequentially receive AIS track and to give the following predictions for each AIS track: (1) arrival port name and (2) expected arrival time. Using the given predictions, systems were evaluated and ranked as follows: (1) length (in % of the whole trip) is the last sequence of correctly predicted port names; (2) mean absolute error in arrival time prediction (against real arrival time). The overall ranking of the systems was calculated using described ranks (75% impact) as well as systems running time (25% impact).
From organizational point of view the online challenge was decided to be splitted into training and final phases. The training phase allowed the participants to continuously test and improve their systems using the benchmark accompanied with a small subset of the test data. Participants registered their systems for the periodical training challenge execution and the results were available at the online leaderboards (2 leaderboards – one per the challenge task). The training phase allowed the teams to thoroughly test their systems under realistic conditions and to solve all the technical problems before the final phase of the challenge. The final phase of the challenge included one execution of each task and was held in a non-public mode – leaderboards became publicly available online after the awarding procedure at the DEBS conference.
Technically the challenge was supported at GitHub, where two sample systems (in Java and Python) have been published. Both systems are built around the Hobbit Java SDK and allow participants to debug systems before making a docker container and uploading to the online platform. Sample systems as well as Java SDK have been created by AGT as a result of the experience received during the Grand Challenge 2017 – first DEBS challenge on the HOBBIT platform.
In total more than 15 teams expressed an interest in the challenge, 9 systems from 8 teams were submitted for the final challenge. The winning solution was created by the team from Alexandru Ioan Cuza University of Iaşi, Romania (the team also won the DEBS Grand Challenge 2017) which demonstrated the highest results among other systems (earliness rate is 70%, mean absolute error is about 800 min). Teams from Chungnam National University (Korea), Technische Universität Dresden (Germany), Technion (Israel), NUI Galway (Ireland), University of Illinois Urbana-Champaign (USA), Centre National de la Recherche Scientifique (France) and University of Carthage (Tunisia) presented their solutions at the DEBS Conference during the oral and poster sessions.
The DEBS steering committee as well as the teams that participated in the challenge provided a positive feedback to the HOBBIT platform and expressed their interest to continue using it for further Grand Challenges. The upcoming DEBS Conference 2019 will be in Darmstadt, Germany.