![17
MQTT Architecture (large scale)
Load
Balancer
MQTT
Server 1
MQTT
Server 2
MQTT
Server 3
MQTT
Server 4
topic: [deviceid]/car
...
Processor
1
Processor
2
Processor
3
Processor
4](https://image.slidesharecdn.com/201901oopiotkafkaksqltensorflowkaiwaehner-190122185034/75/IoT-Sensor-Analytics-with-Python-Jupyter-TensorFlow-Keras-Apache-Kafka-KSQL-and-MQTT-11-2048.jpg)
![17
MQTT Architecture (large scale)
Load
Balancer
MQTT
Server 1
MQTT
Server 2
MQTT
Server 3
MQTT
Server 4
topic: [deviceid]/car
...
Processor
1
Processor
2
Processor
3
Processor
4](https://crownmelresort.com/image.slidesharecdn.com/201901oopiotkafkaksqltensorflowkaiwaehner-190122185034/75/IoT-Sensor-Analytics-with-Python-Jupyter-TensorFlow-Keras-Apache-Kafka-KSQL-and-MQTT-11-2048.jpg)
Uploaded byKai Wähner
IoT Sensor Analytics with Python, Jupyter, TensorFlow, Keras, Apache Kafka, KSQL and MQTT
The document outlines the integration of IoT data analytics using Apache Kafka, MQTT, and TensorFlow, describing various use cases, architecture, and processing options. It emphasizes Apache Kafka's role as a streaming platform and its advantages over MQTT for IoT applications, while also addressing integration methods and potential trade-offs. Key components include real-time analytics, model deployment for anomaly detection, and practical demonstrations of end-to-end sensor analytics.
More Related Content
Similar to IoT Sensor Analytics with Python, Jupyter, TensorFlow, Keras, Apache Kafka, KSQL and MQTT
![Support, Monitoring, Continuous Improvement & Scaling Agentic Automation [3/3]](https://cdn.slidesharecdn.com/ss_thumbnails/agenticcommunityseries-day3-cfd-251120170304-ddef8112-thumbnail.jpg?width=640&height=640&fit=bounds)
IoT Sensor Analytics with Python, Jupyter, TensorFlow, Keras, Apache Kafka, KSQL and MQTT
- 1. 1Confidential IoT Sensor Analyticswith Apache Kafka, KSQL, TensorFlow and MQTT Kai Waehner Technology Evangelist kontakt@kai-waehner.de LinkedIn @KaiWaehner www.confluent.io www.kai-waehner.de Kafka-Native End-to-End IoT Data Integration and Processing
- 2. 3 Agenda 1) IoT UseCases 2) MQTT Standard 3) Apache Kafka Ecosystem 4) TensorFlow for IoT Scenarios 5) End-to-End IoT Integration Architecture(s) 6) IoT Data Processing 7) Live Demo: End-to-End Sensor Analytics
- 3. 4 Agenda 1) IoT UseCases 2) MQTT Standard 3) Apache Kafka Ecosystem 4) TensorFlow for IoT Scenarios 5) End-to-End IoT Integration Architecture(s) 6) IoT Data Processing 7) Live Demo: End-to-End Sensor Analytics
- 4. 6 Connected Intelligence (Cars,Machines, Robots, …)
- 5.
- 6. 8 Smart Retail andCustomer 360
- 7. 9 Intelligent Applications (EarlyPart Scrapping, Predictive Maintenance, …)
- 8. 10 ? Architecture (High Level) KafkaBrokerKafka BrokerStreaming Platform Connect w/ MQTT connector GatewayDevicesDevicesDevicesDevice Device Tracking (Real Time) Predictive Maintenance (Near Real Time) Log Analytics (Batch) Edge Data Center / Cloud How to integrate?
- 9. 13 Agenda 1) IoT UseCases 2) MQTT Standard 3) Apache Kafka Ecosystem 4) TensorFlow for IoT Scenarios 5) End-to-End IoT Integration Architecture(s) 6) IoT Data Processing 7) Live Demo: End-to-End Sensor Analytics
- 10. 14 MQTT - Publish/ subscribe messaging protocol • Built on top of TCP/IP for constrained devices and unreliable networks • Many (open source) broker implementations • Many client libraries • IoT-specific features for bad network / connectivity • Widely used (mostly IoT, but also web and mobile apps via MQTT over WebSockets)
- 11. 17 MQTT Architecture (largescale) Load Balancer MQTT Server 1 MQTT Server 2 MQTT Server 3 MQTT Server 4 topic: [deviceid]/car ... Processor 1 Processor 2 Processor 3 Processor 4
- 12. 18 MQTT Trade-Offs Pros • Lightweight •Simple API • Built for poor connectivity / high latency scenario • Many client connections (tens of thousands per MQTT server) Cons • Queuing, not stream processing • Can’t handle usage surges (no buffering) • No high scalability (true for most MQTT brokers) • Very asynchronous processing (often offline for long time) • No good integration to the rest of the enterprise • No reprocessing of events
- 13. 19 Agenda 1) IoT UseCases 2) MQTT Standard 3) Apache Kafka Ecosystem 4) TensorFlow for IoT Scenarios 4) End-to-End IoT Integration Architecture(s) 5) IoT Data Processing 6) Live Demo: End-to-End Sensor Analytics
- 14. 20 Apache Kafka –The Rise of a Streaming Platform The Log ConnectorsConnectors Producer Consumer Streaming Engine
- 15.
- 16. 23 Apache Kafka ==Distributed Commit Log with Replication
- 17. 25 Apache Kafka atScale https://conferences.oreilly.com/strata/strata-ca/public/schedule/detail/63921 https://qconlondon.com/london2018/presentation/cloud-native-and-scalable-kafka-architecture (2018) (2018)
- 18. 26 Kafka Trade-Offs (fromIoT perspective) Pros • Stream processing, not just queuing • High throughput • Large scale • High availability • Long term storage and buffering • Reprocessing of events • Good integration to the rest of the enterprise Cons • Not built for tens of thousands connections • Requires stable network and good infrastructure • No IoT-specific features like keep alive, last will or testament
- 19. 27 (De facto) Standardsfor Processing IoT Data A Match Made In Heaven + =
- 20. 28 Agenda 1) IoT UseCases 2) MQTT Standard 3) Apache Kafka Ecosystem 4) TensorFlow for IoT Scenarios 5) End-to-End IoT Integration Architecture(s) 6) IoT Data Processing 7) Live Demo: End-to-End Sensor Analytics
- 21. 29 TensorFlow TensorFlow is anopen source software library for high performance numerical computation. Its flexible architecture allows easy deployment of computation across a variety of platforms (CPUs, GPUs, TPUs), and from desktops to clusters of servers to mobile and edge devices. Originally developed by researchers and engineers from the Google Brain team within Google’s AI organization, it comes with strong support for machine learning and deep learning and the flexible numerical computation core is used across many other scientific domains. https://www.tensorflow.org/
- 22. 30 The First AnalyticModels How to deploy the models in production? …real-time processing? …at scale? …24/7 zero downtime?
- 23. 31 Hidden Technical Debtin Machine Learning Systems https://papers.nips.cc/paper/5656-hidden-technical-debt-in-machine-learning-systems.pdf
- 24. 32 Apache Kafka’s OpenSource Ecosystem as Infrastructure for ML
- 25. 33 Apache Kafka’s OpenEcosystem as Infrastructure for ML Kafka Streams Kafka Connect Rest Proxy Schema Registry Go/.NET /Python Kafka Producer KSQL Kafka Streams
- 26. 37 Replayability — alog never forgets! Time Model B Model XModel A Producer Distributed Commit Log Different models with same data Different ML frameworks AutoML compatible A/B testing Google Cloud Storage HDFS
- 27. 38 Analytic Model (Autoencoderfor Anomaly Detection)
- 28. 39 Model Deployment #1:RPC Communication to do Model Inference Streams Input Event Prediction Request Response Model Serving TensorFlow Serving gRPC
- 29. 40 Model deployment #2:Model interference natively in the App Streams Input Event Prediction
- 30. 41 Agenda 1) IoT UseCases 2) MQTT Standard 3) Apache Kafka Ecosystem 4) TensorFlow for IoT Scenarios 5) End-to-End IoT Integration Architecture(s) 6) IoT Data Processing 7) Live Demo: End-to-End Sensor Analytics
- 31. 42 ? Architecture (High Level) KafkaBrokerKafka BrokerStreaming Platform Connect w/ MQTT connector GatewayDevicesDevicesDevicesDevice Device Tracking (Real Time) Predictive Maintenance (Near Real Time) Log Analytics (Batch) Edge Data Center / Cloud How to integrate?
- 32. 43 ? Architecture (High Level)– Machine Learning Perspective Kafka BrokerKafka BrokerStreaming Platform Connect w/ MQTT connector GatewayDevicesDevicesDevicesDevice Edge Analytics Real Time Model Serving Predictive Maintenance Near Real Time Model Serving Model Training (Batch) Edge Data Center / Cloud
- 33. 46 Kafka-Native Integration Optionsbetween MQTT and Apache Kafka Kafka Connect MQTT Proxy REST Proxy
- 34. 47 Kafka-Native Integration Optionsbetween MQTT and Apache Kafka Kafka Connect MQTT Proxy REST Proxy
- 35. 49 ? Integration with KafkaConnect (Source and Sink) Kafka BrokerKafka BrokerKafka Broker MQTT Broker Connect w/ MQTT connector Connect w/ MQTT connectorMQTT DevicesDevicesDevicesDevice Kafka Consumer MQTT Broker Persistent + offers MQTT-specific features Consumes push data from IoT devices Kafka Connect Kafka Consumer + Kafka Producer under the hood Pull-based (at own pace, without overwhelming the source or getting overwhelmed by the source) Out-of-the-box scalability and integration features (like connectors, converters, SMTs) ? Connect w/ MQTT connector Connect w/ MQTT connector
- 36. 56 Kafka-Native Integration Optionsbetween MQTT and Apache Kafka Kafka Connect MQTT Proxy REST Proxy
- 37. 57 MQTT Proxy Kafka BrokerKafkaBrokerKafka Broker MQTT ProxyMQTT DevicesDevicesDevicesDevices Kafka Consumer MQTT Proxy MQTT is push-based Horizontally scalable Consumes push data from IoT devices and forwards it to Kafka Broker at low-latency Kafka Producer under the hood No MQTT Broker needed Kafka Broker Source of truth Responsible for persistence, high availability, reliability
- 38. 59 Kafka-Native Integration Optionsbetween MQTT and Apache Kafka Kafka Connect MQTT Proxy REST Proxy
- 39. 60 Confluent REST Proxy RESTProxy IoT Applicatons Native Kafka Applications (Java, C, Go, …) REST / HTTP(S) TCP The „simple alternative“ for IoT • Simple and understood • HTTP(S) Proxy à Push-based • Security ”easier” • Scalable with standard load balancer (still synchronous HTTP) • Not for very high throughput • Implement Kafka Connect features in your client app
- 40. 62 Agenda 1) IoT UseCases 2) MQTT Standard 3) Apache Kafka Ecosystem 4) TensorFlow for IoT Scenarios 5) End-to-End IoT Integration Architecture(s) 6) IoT Data Processing 7) Live Demo: End-to-End Sensor Analytics
- 41. 6363 Processing Options forMQTT Data with Apache Kafka Streams Kafka native vs. additional big data cluster and technology (or others, you name it …)
- 42. 6464 IoT Data Processing KafkaClient Batch System AnalyticsKafka Cluster Kafka Connect Kafka Streams / KSQL MQTT Device Kafka Ecosystem Other Components Real Time System All Data Alerting Process Data Continuously Forward Processed Data On premise DC / CloudAt the edge
- 43. 6868 KSQL – ContinuousQueries for Streaming ETL / Anomaly Detection CREATE STREAM vip_actions AS SELECT userid, page, action FROM clickstream c LEFT JOIN users u ON c.userid = u.user_id WHERE u.level = 'Platinum'; CREATE TABLE possible_fraud AS SELECT card_number, count(*) FROM authorization_attempts WINDOW TUMBLING (SIZE 5 MINUTES) GROUP BY card_number HAVING count(*) > 3;
- 44. 6969 Agenda 1) IoT UseCases 2) MQTT Standard 3) Apache Kafka Ecosystem 4) TensorFlow for IoT Scenarios 5) End-to-End IoT Integration Architecture(s) 6) IoT Data Processing 7) Live Demo: End-to-End Sensor Analytics
- 45. 7070 KSQL and DeepLearning (Auto Encoder) for Anomaly Detection MQTT Proxy Elastic search Grafana Kafka Cluster Kafka Connect KSQL Car Sensors Kafka Ecosystem Other Components Real Time Emergency System All Data PotentialDefect Apply Analytic Model Filter Anomalies On premise DCAt the edge 5858 KSQL and Deep Learning (Auto Encoder) for Anomaly Detection MQTT Proxy Elastic search Grafana Kafka Cluster Kafka Connect KSQL Car Sensors Kafka Ecosystem Other Components Real Time Emergency System All Data PotentialDefect Apply Analytic Model Filter Anomalies On premise DCAt the edge
- 46. 7171 Model Training withPython, KSQL, TensorFlow, Keras and Jupyter https://github.com/kaiwaehner/python-jupyter-apache-kafka-ksql-tensorflow-keras
- 47. 7272 Model Deployment withApache Kafka, KSQL and TensorFlow “CREATE STREAM AnomalyDetection AS SELECT sensor_id, detectAnomaly(sensor_values) FROM car_engine;“ User Defined Function (UDF)
- 48. 73 Live Demo End-to-End SensorAnalytics… Python, Jupyter Notebook, TensorFlow, Keras, Apache Kafka, KSQL and MQTT
- 49. 74 Model Training withPython, KSQL, TensorFlow, Keras and Jupyter https://github.com/kaiwaehner/python-jupyter-apache-kafka-ksql-tensorflow-keras
- 50. 75 Deep Learning UDFfor KSQL for Streaming Anomaly Detection of MQTT IoT Sensor Data https://github.com/kaiwaehner/ksql-udf-deep-learning-mqtt-iot
- 51. 77 Kai Waehner Technology Evangelist kontakt@kai-waehner.de @KaiWaehner www.kai-waehner.de www.confluent.io LinkedIn Questions?Feedback? Please contact me! Come to our booth to find out more about Kafka and Confluent