The second version of the Structured Machine Learning Benchmark is available online. The benchmark is created for the upcoming DEBS Grand Challenge 2018. The challenge is based on marine traffic data and focused on real-time prediction of destination ports and arrival times for vessels, which regularly send their geo coordinates, timestamps, course and other relevant information. Benchmarked systems shall provide predicted destinations and expected arrival times for each data tuple received from the benchmark. Each next data tuple for a trip will be sent to a benchmarked system only after the prediction for the current one have been returned.
The benchmark has a scalable architecture and provides a sufficient rate of input load even if a benchmarked system is distributed over all nodes available in the online platform. Tuples of a trip are serving sequentially, but trips performed by different ships are served in parallel, which makes it possible to increase the total input rate and prevent access to the tuples of next trips until the previous ones have been completed.
The components of the benchmark have been dockerized and uploaded into the online platform as well as some amount of test data (different from the released training dataset). Challenge participants may integrate their prediction algorithms directly into the proposed sample system and debug them by executing the benchmark locally (either as pure java or being containerized as docker image). Once docker image of a system is ready system’s owner may upload it to the platform and manually execute it under the online benchmark (named the DEBS GC 2018 Benchmark). Once the system executes correctly system’s owner may register it for the DEBS GC 2018 Training Phase as well as DEBS GC 2018 Final Challenge (deadline is 15 of April). Within the Training phase (till end of May 2018) the registered systems will be periodically benchmarked with some part of test data and results will be publicly available at leaderboards. Final executions will take place in the second half of May 2018 using unseen test data. Results of the final execution will be published after the awarding procedure at the DEBS Conference by 25-29 of June 2018.
Feel free to join the challenge and ask any questions about the challenge under the Issues tab.