Using HOBBIT platform at DEBS Grand Challenge considered as major improvement
At this year’s Distributed Events Processing Systems conference (DEBS 2017) the HOBBIT platform has demonstrated its ability to objectively quantify the performance of anomaly detection algorithms that process RDF data in real-time.
Dr. Roman Katerinenko and Dr. Martin Strohbach, both AGT International were representing the HOBBIT project by co-organizing this year’s Grand Challenge in which participating teams were asked to correctly detect anomalies in sensor data from injection molding machines provided by Weidmüller. By using the HOBBIT platform, we could for the first time objectively quantify and compare the performance of distributed stream processing analytical systems by measuring the correctness of found anomalies, latency of the system (how long does it take to send a response), and throughput (how much data can be processed in a second). Benchmarked systems had to process over 5GB of data that was sent to the systems under test at a rate of 250 data points second. In total, participants had to detect more than 100 anomalies and deal with an unknown number of machines that provided measurements at unknown periods. Data from more than 1000 machines had to be processed.
AGT used the HOBBIT platform and implemented a full benchmark including a data generator that creates realistic data of any size. The data generator has been carefully designed based on requirements from Weidmüller to hide business relevant aspects of the data and at the same time create measurements that allow to extrapolate the performance of the tested system in a realistic setting. The data generator is based on ML models (k-means and Markov Chains) learned from the actual Weidmüller data.
In total 14 teams registered to participate and 7 teams correctly identified the anomalies. The winning team from the Alexandru Ioan Cuza University, Romania had the best performing solution (average latency of 40ms) winning an award of $ 1000. A team of University Stuttgart, Germany won the audience award for the most elegant solution: they required only little optimisation to their system and offered data scientists an elegant abstraction from the details of real-time message processing. Correct solutions were also submitted by teams from WSO2, from Chungnam National University, South Korea, The Insight Centre for Data Analytics, Rice University, USA, and Technion, Israel. The top 4 performers presented their solution at the DEBS conference in the DEBS Grand Challenge session, the other 3 teams presented a poster about their solution.
The HOBBIT platform was highly appreciated by the DEBS community. The steering committee chair Prof. Hans-Arno Jacobson, from University of Toronto pointed out that using the HOBBIT benchmarking platform constitutes a big step forward in the way results of the DEBS Grand Challenge are evaluated. Participants praised the improvements that the HOBBIT platform brings to the world of event processing, highlighting for instance its capability to benchmark distributed systems. We got feedback from many people in the community who they were impressed with the achievement of the Grand Challenge team saying that using the HOBBIT platform was very successful and it was having the effort both for the organizers and participating teams. According to Prof. Holger Ziekow (University of Furtwangen), one of the HOBBIT external co-organizers of the Grand Challenge, the quality of this year’s submissions was notably higher, which may be due to the objective quantifications performed by the HOBBIT platform. As a result AGT has been invited to co-organize next year’s Grand Challenge in New Zealand and asked to use the HOBBIT platform for the next structure machine learning challenge.
DEBS is an ACM International Conference one of the world leading associations of computing professionals. The DEBS conferences series  is now in its 10th year and has regular attendance and contributions from world leading event processing experts from academia and industrial research. The DEBS Grand Challenge has always been considered as a highlight of the conference as participants need to demonstrate that their research solves actual real problems . Besides AGT, this year’s Grand Challenge organizers included Prof. Dr. Holger Ziekow, Dr. Zbigniew Jerzak, Development Manager in the Machine Learning Unit at SAP Germany and Dr. Vincenzo Gulisano, Assistant Professor at Chalmers University of Technology.
Some live reports from the DEBS Grand Challenge session is available on Twitter .