Download as PDF, PPTX
More Related Content
![[@NaukriEngineering] Messaging Queues](https://cdn.slidesharecdn.com/ss_thumbnails/queueprocessing-161111064335-thumbnail.jpg?width=640&height=640&fit=bounds)
Similar to Scaling MQTT With Apache Kafka
![Support, Monitoring, Continuous Improvement & Scaling Agentic Automation [3/3]](https://cdn.slidesharecdn.com/ss_thumbnails/agenticcommunityseries-day3-cfd-251120170304-ddef8112-thumbnail.jpg?width=640&height=640&fit=bounds)
Scaling MQTT With Apache Kafka
- 1. SCALING MQTT WITH KAFKA TimKellogg April 7, 2014 @kellogh
- 2. • MQTT broker • Protocol onboarding • Cloud environment (we’re a startup)
- 4. • Standard • Lightweight o >= 2 byte overhead per message • Easy to parse o Length prefixed strings • Requires very little resources on client side o Broker keeps track of state
- 5. • Reliable o QoS1 & 2 o Last Will & Testament messages • Secure o Username + Password o Tunnel over TLS
- 6.
- 7. Topics • foo/bar/baz • com.example/device/17/thermo • Patterns • com.example/device/+/thermo • com.example/device/#
- 8. Scaling Goals • Morethan 2 Million connected publishers • More than 65,000 msg/s • Single subscriber
- 9. Scaling Goals • Amazon’sEC2 • Horizontal scaling o Reduce cost o Plan for the future o Less impact from downtime
- 10.
- 11. Load Balancing • Whichbroker to connect to? o DNS load balancing • HAProxy • QoS 1-2 messages stored in Cassandra o Consistent hash ring
- 12.
- 13.
- 14.
- 15.
- 16.
- 17. POST From TheBroker Pub Pub Pub Broker Topic/A Topic/B Topic/C Broker Broker HTTP POST LoadBalancing Server HTTP POST Server HTTP POST Server LoadBalancing
- 18. Benefits • Easy toload balance • Well known & well supported
- 19. Drawbacks • HTTP isheavy • Headers • Creating & destroying TCP connections • Subscriber servers must be available • Retry logic to guarantee delivery
- 20.
- 21. • • Distributed log aggregationframework • Server to server • “Smart” clients • Apache ZooKeeper
- 22. • Append-only filesper topic o Client keeps track of what messages it’s processed • No topic wildcards • Key is used for out of band data • device/42/thermo è topic: device-thermo key: 42
- 23.
- 24.
- 25.
- 26. • Linear scalingfor fire hose subscriber • At least 2 million clients • At least 65,000 msg/s
- 27. Wish List • Security • Configuration
- 28.
- 29. The Book: MasteringThe Internet of Things Questions? @kellogh