Stream Machine Learning (StreaML) Open Challenge 2017-2018

Challenge Motivation

After a successful organization of the DEBS Grand Challenge 2017 by the HOBBIT project at the DEBS 2017 Conference, HOBBIT is proud to announce the  Stream Machine Learning (StreaML) Open Challenge. The open challenge will ensure continuous participation and systems evaluation.

  • A monetary prizes will be provided to the best winning system at each cutoff date.
  • Stay tuned to get ready to participate!       Express your interest!

Challenge Overview

The goal of the StreaML Open Challenge competition is to evaluate event-based systems for real-time analytics over high velocity and high volume data streams generated by manufacturing equipment.

The predefined machine learning algorithm should be applied for analysis of the RDF streaming data generated by digital and analogue sensors embedded within multiple molding machines. The data produced by each sensor is clustered and the state transitions between the observed clusters is modeled as a Markov chain. Based on this machine learning-based classification, anomalies are detected by means of sequences of transitions that happen with a probability lower than a given threshold.

Please note, that StreaML Open Challenge reuses the dataset and the task description of DEBS GC 2017 and that in contrast to DEBS Challenge, systems will be continuously evaluated every week and results will be shown on a leaderboard (see detailed description for more details).

Participants can use the published benchmark as a reference implementation of anomaly detection algorithm to pass the correctness checks and focus on performance and stability of their systems, which are included into evaluation criteria.

Prizes

The best-winning system will get a prize of 500€

Q&A

For additional questions please refer to the StreaML Open Challenge google group

 Important Dates

Dry-run: December 2017 to end of January 2018
Training Phase*: February to  end of April 2018
Test Phase: May 2018
Cutoff and results proclamation: May 2018

* Submitted systems will be continuously evaluated by the HOBBIT platform with weekly updates of leaderboards.

** Actual dates are for first round of StreaML Open Challenge.

Tasks and Training Data

The StreaML Open Challenge focuses on the task related to the problem of automatic detection of anomalies for manufacturing equipment.

The overall goal of the task is to detect abnormal behavior of a manufacturing machine based on the observation of the stream of measurements provided by each of the machines from observable set, which is dynamic (i.e. machines dynamically join and leave the set of observable machines). The data produced by each sensor is clustered and the state transitions between the observed clusters are modelled as a Markov chain. Based on this classification, anomalies are detected as sequences of transitions that happen with a probability lower than a given threshold.

The detailed technical information regarding the task, data and evaluation criteria can be found here.

Registration and Submission

Participants should implement the predefined anomaly detection algorithm as a system according to HOBBIT specification.

Submitting your system to the challenge includes two steps:

  1. Upload your system to the HOBBIT platform
  2. Register your system for the StreaML Open Challenge

After submitting your system to the HOBBIT platform, you can use the StreaML-Benchmark to test the correctness of your implementation. Once the training phase has started, you can register your system for the challenge and it will be continuously evaluated.

For more information please refer to https://project-hobbit.eu/challenges/streaml-open-challenge_details.

Organization

  • Pavel Smirnov – AGT International
  • Martin Strohbach – AGT International

Support

We would like to explicitly thank Weidmüller (http://www.weidmueller.de/) for the provisioning of the original data set that AGT International used to generate a realistic data set for the purpose of this challenge.

The StreaML open challenge is sponsored by AGT International (http://www.agtinternational.com).

—————————————————————————————————
StreaML Open Challenge 2017-2018
—————————————————————————————————
After a successful co-organization of the DEBS Grand Challenge with the HOBBIT project (https://project-hobbit.eu/) at the DEBS 2017 Conference, HOBBIT is proud to announce the Stream Machine Learning (StreaML) Open Challenge, which starts in December 2017.

The StreaML Open Challenge will ensure continuous participation and systems evaluation with periodic cutoffs. The first cutoff is in May 2018.
* A monetary prize will be provided to the best-winning system!
* Stay tuned and get ready to participate!

————————————————————————————–
The StreaML Open Challenge at a glance
————————————————————————————–
The StreaML Open Challenge focuses on the task related to the problem of automatic detection of anomalies for manufacturing equipment.
The overall goal of the task is to detect abnormal behavior of a manufacturing machine based on the observation of the stream of measurements provided by each of the machines from observable set, which is dynamic (i.e. machines dynamically join and leave the set of observable machines). The data produced by each sensor is clustered and the state transitions between the observed clusters are modelled as a Markov chain. Based on this classification, anomalies are detected as sequences of transitions that happen with a probability lower than a given threshold.

We herewith invite system developers to participate in the aforementioned scenario. To ensure that the system results are comparable, we will provide the HOBBIT benchmarking platform for the generation of the final results to be included into the system publications. A specification of the hardware on which the benchmarks will be run will be released in due course.

Please note, that StreaML Open Challenge reuses the dataset (http://hobbitdata.informatik.uni-leipzig.de/StreamMachineLearning_1.0/) and the task description (https://project-hobbit.eu/open-challenges/streaml-open-challenge_details/) of DEBS GC 2017 but in contrast to DEBS Challenge, systems will be continuously evaluated every week and results will be shown at leaderboard.

Participants are allowed to use the published benchmark (https://github.com/hobbit-project/sml-benchmark) as a reference implementation of anomaly detection algorithm to pass the correctness checks and focus on performance and stability of their systems, which are included into evaluation criteria.

Read more details here: https://project-hobbit.eu/open-challenges/streaml-open-challenge/

————————————————————————————–
Prizes
————————————————————————————–
The winner of each round of the challenge will get a prize of 500€.

————————————————————————————–
Important Dates
————————————————————————————–
Dry run: December 2017 – end of January 2018
Training phase: February – end of April 2018
Cutoff and results proclamation: May 2018

————————————————————————————–
Registration and Submission
————————————————————————————–
Submission and registration procedure is documented here: https://project-hobbit.eu/open-challenges/streaml-open-challenge_details/

The evaluation platform can be reached under following address: http://master.project-hobbit.eu

————————————————————————————–
Organization
————————————————————————————–
* Pavel Smirnov – AGT Group GmbH
* Martin Strohbach – AGT Group GmbH

————————————————————————————–
Sponsorship
————————————————————————————–
We would like to explicitly thank AGT International (http://www.agtinternational.com) for being a sponsor for the StreaML Open Challenge.

————————————————————————————–
Further Information and Contact
————————————————————————————–
For detailed information, including datasets and submission guidelines,
please visit the challenge website:
https://project-hobbit.eu/open-challenges/streaml-open-challenge/
Feel free to join StreaML open challenge google group to ask any questions: https://groups.google.com/d/forum/streaml-open-challenge