Download to read offline
Uploaded byTill Rohrmann
Elastic Streams at Scale @ Flink Forward 2018 Berlin
This document discusses Elastic Streams at scale using Apache Flink and Mesos. It describes how Flink jobs can be deployed on Mesos clusters by having the Flink master process request resources from the Mesos resource manager. The resource manager then allocates Mesos containers for the Flink master and task managers, allowing the Flink processes to be deployed and tasks to run on the Mesos cluster resources. A Mesos dispatcher can be used to start and monitor the Flink master process.
More Related Content
Similar to Elastic Streams at Scale @ Flink Forward 2018 Berlin
Elastic Streams at Scale @ Flink Forward 2018 Berlin
- 1. Elastic Streams atScale @stsffap @joerg_schad
- 3.
- 5.
- 6.
- 7.
- 8.
- 9.
- 10.
- 11. partitionId: 1, offset:42 partitionId: 3, offset: 10 partitionId: 6, offset: 27 partitionId: 1, offset: 42 partitionId: 3, offset: 10 partitionId: 6, offset: 27 • Store offset for each partition • Individual entries are repartitionable
- 12.
- 14.
- 15.
- 17.
- 19.
- 20. © 2018 Mesosphere,Inc. All Rights Reserved. Flink Mesos Integration Mesos Master Mesos Cluster Client (2) HTTP POST JobGraph/Jars Flink Master Process Flink Mesos ResourceManager JobManager (4) Start Process (and supervise) (8) Deploy Tasks (7) Register (5) Request slots Flink Mesos Dispatcher (3) Allocate container for Flink master (6) Allocate containers for TaskManagers Marathon (1) Start and monitor dispatcher Mesos Task TaskManager Mesos Task TaskManager
- 21. Mesos Master Mesos Cluster Cluster Client (1)Submit Mesos App. (JobGraph / JARs) Application Master Flink-Mesos ResourceManager JobManager TaskManager TaskManager TaskManager (2) Spawn Application Master (4) Start TaskManagers (6) Deploy Tasks (5) Register (3) Request slots
- 22.
- 23.
- 24. THANK YOU! @stsffap @dataArtisans @ApacheFlink WE AREHIRING data-artisans.com/careers