Please enable JavaScript.

Coggle requires JavaScript to display documents.

SMACK (MILESTONES (

SMACK

MILESTONES

3 - Data Ingestion

Generator

HTTP API; TCP socket to Kafka

source

signal database

JavaApplet generator ECGSYN

n - datasources (st. deviation)

Docker Image - Deployment to DC/OS

Kafka

Brokers, Topic

Installation

1 - Define Data (PoC scope)

ECG / PPG analysis

SME: dawid.blonski@roche.com

4 - Data Processing

Flink

custom InfluxDBSink

aggregations

connecting streams (introduce signal distortion via nc)

2 - Provide working DC/OS Cluster

AWS

Requests

Infrastructure as code automation

Ansible example

Terraform example

5 - Data Visualization

InfluxDB

duration / retention policy

Grafana Dashboard

views

sensors raw

sensors mean

data points per second

aggr by Flink Job

aggr by InfluxDB

sensors total

data points total

6 - Pipeline fine-tuning

sheet

7 - System metrics

Graphite + Flink / Kafka Metrics

8 - Advanced usage

Anomaly detection

by Time Series paper

connecting streams

6 - Test automation

Pipeline Plumbing

Kafka

delete, create topic, set partitions, replication-factor, retention

InfluxDB

drop, create DB, duration

retention policy

Flink

stop, start `SensorDataInfluxJob`, set parallelism

Generator - reconfigure Marathon cofiguration (topic)text

LINKS

Slack

Bitbucket

Jira

Open QUESTIONS / enhancements

Kafka

Avro type messages

Message format with schema referable by KafkaProducer and KafkaConsumer

Processing Kafka Messages with KafkaStreams

Partitioning

Automation

CloudFormation AWC_EC2_Basic
coverted to Terraform script

choosing latest DC/OS builds

DC/OS gateway (Vagrant)

install dcenv script for switching between environments

DC/OS

auto-scaling DC/OS (Mesos) resources

auto-scaling DC/OS services

understand how DC/OS scale services (Redis can be scaled using Scale button, while Kafka doesn't have that option)

Redis is a Docker image while Kafka seems to be a Mesos Framework - maybe that's why

understand how Mesos Framework and DC/OS package relate to each other

FOCUS

TOP PRIORITY

Fast-Data

Scalability

AWS based auto-scaling

(more data sources, more mesos resources)

Full Automation / Infrastructure-as-code

SECONDARY

analytics and machine learning

storage

INFRASTRUCTURE

On-Premise (AE&O)

Current POC instance

AE&O pilot program

AWS

AWS Instance

README.md