Download as PDF, PPTX
Uploaded byGetInData
Creating Real-Time Data Streaming powered by SQL on Kubernetes - Albert Lewandowski, GetInData
The document provides an overview of Apache Flink, focusing on its functionality in data streaming and batch processing, installation on Kubernetes, and its integration with the Ververica platform. It outlines key components, deployment strategies, monitoring with Prometheus, and logging analytics via tools like Loki and Elasticsearch. Additionally, it includes guidelines for observing job status and managing Flink jobs in various cloud environments such as AWS and GCP.
More Related Content
Similar to Creating Real-Time Data Streaming powered by SQL on Kubernetes - Albert Lewandowski, GetInData
Creating Real-Time Data Streaming powered by SQL on Kubernetes - Albert Lewandowski, GetInData
- 1. Flink SQL Workshop Author: AlbertLewandowski
- 2.
- 3. © Copyright. Allrights reserved. Not to be reproduced without prior written consent. Data Streaming vs. Batch Events 1 2 3 4 5 6 Batch Stream
- 4. © Copyright. Allrights reserved. Not to be reproduced without prior written consent. Use cases User activity Fraud detection Logistics Industrial IoT Location data Recommendations
- 5. © Copyright. Allrights reserved. Not to be reproduced without prior written consent. One tool, multiple languages Java 8 or 11 Python SQL Scala 2.11 or 2.12
- 6. © Copyright. Allrights reserved. Not to be reproduced without prior written consent. Where should I install? Standalone Kubernetes YARN cluster ● CICD process ● Service Discovery - monitoring with Prometheus ● Scalability ● Managing resources ● A/B Testing
- 7. © Copyright. Allrights reserved. Not to be reproduced without prior written consent. High Availability of Flink JobManager level Storage level ● ZooKeeper ● Kubernetes (beta) ● High Availability of storage to/from which Flink writes/reads savepoints and checkpoints ● Performance of storage Job Strategy ● Data reprocessing policy ● How to deploy new job?
- 8. © Copyright. Allrights reserved. Not to be reproduced without prior written consent. Flink K8S Operator Kubernetes Operator for Apache Flink Ververica Platform Native Kubernetes - Apache Flink CRDs Yes Yes No No CICD Kubernetes API Kubernetes API REST API or Web UI Kubernetes API Installation Helm chart or raw Kubernetes manifests Helm chart or raw Kubernetes manifests Helm chart or raw Kubernetes manifests No need to install any component SQL Editor No No Yes No Dependencies No No Persistence volume for database Object storage for artifactory No Status beta beta production beta
- 9. © Copyright. Allrights reserved. Not to be reproduced without prior written consent. Why Flink on Kubernetes? Simpler deployment process Flexible jobs management Simple Service Discovery - Prometheus Flexible testing
- 10. © Copyright. Allrights reserved. Not to be reproduced without prior written consent. Installation & Configuration Helm A package manager for Kubernetes CICD tool Example: Gitlab CI Kubernetes API Flink jobs
- 11. © Copyright. Allrights reserved. Not to be reproduced without prior written consent. Testing Incubating Mode A/B Testing Blue Green Deployment Production data Production job Job Incubating mode Separated output Standard output Dedicated TaskManagers Dedicated TaskManagers savepoi nt Proxy Flink Job #1 Flink Job #2 Result #1 Result #2
- 12. © Copyright. Allrights reserved. Not to be reproduced without prior written consent. Job Cluster & Session Cluster Job Cluster Session Cluster Full set of Flink cluster for each individual job Standalone Flink cluster on Kubernetes Short running tasks Ad-hoc queries Long running tasks Separate images for different jobs
- 13.
- 14. © Copyright. Allrights reserved. Not to be reproduced without prior written consent. Ververica Platform components
- 15. © Copyright. Allrights reserved. Not to be reproduced without prior written consent. Ververica Platform components ● AppManager It sits as an orchestrator between a user’s requests, Kubernetes, and Apache Flink ● Gateway Container hosting the SQL service and processing any SQL-related task. ● UI App responsible for the Web User Interface
- 16. © Copyright. Allrights reserved. Not to be reproduced without prior written consent. Ververica Platform external resources (optional) ● Blob storage (like AWS S3) It can be used to store artifacts if required ● IAM permissions We can define which Kubernetes ServiceAccount with associated IAM role we can use. ● Logging & Metrics We can set up links to external components in which we can read logs and metrics from Flink jobs.
- 17. © Copyright. Allrights reserved. Not to be reproduced without prior written consent.
- 18. © Copyright. Allrights reserved. Not to be reproduced without prior written consent. Ververica Platform dictionary ● Deployment Target DeploymentTargets correspond to different Kubernetes namespaces in the same Kubernetes cluster. ● Deployment A Deployment specifies the desired state of a Flink job and its configuration. Ververica Platform tracks and reports it. ● Namespaces (in Ververica, not Kubernetes!) Namespaces are the primary means to isolate resources between different groups of users and grant access to resources, allowing for multi-tenancy.
- 19. © Copyright. Allrights reserved. Not to be reproduced without prior written consent. Ververica Platform dictionary ● Session Cluster Your Deployment will be executed in a Flink Session Cluster that may be shared with other Deployments. The lifecycle of the Flink cluster is independent of the Deployment lifecycle. ● Application Cluster Your Deployment will be executed in a separate Flink cluster. The lifecycle of the Flink cluster is tied to the lifecycle of the Deployment.
- 20. © Copyright. Allrights reserved. Not to be reproduced without prior written consent. Lifecycle Management ● Upgrade or suspend job When a Deployment is suspended or a stateful upgrade is triggered, Ververica Platform will submit a Stop-with-Savepoint command to Apache Flink. This command will atomically trigger a Savepoint and stop the job. ● Drain the pipeline Draining emits the maximum watermark before stopping the job. When the watermark is emitted, all event time timers will fire, allowing you to process events that depend on this timer. This is useful when you want to fully shut down your job without leaving any unhandled events or state.
- 21.
- 22. © Copyright. Allrights reserved. Not to be reproduced without prior written consent. Repo with examples git clone https://github.com/getindata/ververica-platform-flink-workshop
- 23. © Copyright. Allrights reserved. Not to be reproduced without prior written consent. Prerequisites ● Container Runtime ● Installed Kubectl ● Installed Helm Instruction can be found on Confluence.
- 24.
- 25. © Copyright. Allrights reserved. Not to be reproduced without prior written consent. Observability Observability is about measuring how well internal states of the system can be inferred from knowledge of its external outputs (according to the control theory).
- 26. © Copyright. Allrights reserved. Not to be reproduced without prior written consent. Part One: Metrics Get metrics from environment and application - but how?
- 27. © Copyright. Allrights reserved. Not to be reproduced without prior written consent. Prometheus - Kubernetes-native solution open-source systems monitoring and alerting toolkit joined the Cloud Native Computing Foundation in 2016 as the second hosted project, after Kubernetes a lot of exporters you can write your own easily mature ecosystem PushGateway, Blackbox, AlertManager, etc.
- 28. © Copyright. Allrights reserved. Not to be reproduced without prior written consent. Pull vs. push-based monitoring Pull Push Collector takes metrics Agents push metrics Workload on central poller increases with the number of devices polled. Polling task fully distributed among agents, resulting in linear scalability. Polling protocol can potentially open up system to remote access and denial of service attacks. Push agents are inherently secure against remote attacks since they do not listen for network connections. Flexible: poller can ask for any metric at any time. Relatively inflexible: pre-determined, fixed set of measurements are periodically exported.
- 29. © Copyright. Allrights reserved. Not to be reproduced without prior written consent. Prometheus - Stories service discovery simple on k8s limited security archived data how old data is required? monitor monitoring
- 30. © Copyright. Allrights reserved. Not to be reproduced without prior written consent. Part Two: Logs analytics 1. Get logs from app or environment. 2. Save logs. 3. Query them. 4. Make your system self-healing and discover what’s happening inside your platform.
- 31. © Copyright. Allrights reserved. Not to be reproduced without prior written consent. Logs analytics - which tool should I choose? Logs Analytics for Developers Logs Analytics for Business Loki ElasticSearch
- 32. © Copyright. Allrights reserved. Not to be reproduced without prior written consent. ELK vs. Loki ELK Loki + Promtail/Fluentd Indexing Keys and content of each key Only labels Query language Query DSL or Lucene QL LogQL Tool for data visualisation Kibana Grafana Query performances Faster due to indexed all the data Slower due to indexing only labels Resource requirements Higher due to the need of indexing Lower due to index only labels
- 33. © Copyright. Allrights reserved. Not to be reproduced without prior written consent. What about alerts? Alerts signify that a human needs to take action immediately in response to something that is either happening or about to happen, in order to improve the situation.
- 34.
- 35. © Copyright. Allrights reserved. Not to be reproduced without prior written consent. How to deploy job? Web UI REST API
- 36. © Copyright. Allrights reserved. Not to be reproduced without prior written consent. Local Artifactory ● Ververica provides its own artifactory ● JARs can be also downloaded from the available source like Nexus
- 37. © Copyright. Allrights reserved. Not to be reproduced without prior written consent. Flink job & AWS ● We can connect Flink job with our IAM user or IAM role ● Flink supports AWS IAM credentials by default ● We can run Flink via EMR or Kinesis
- 38. © Copyright. Allrights reserved. Not to be reproduced without prior written consent. Flink job & GCP ● We can connect Flink job with our IAM user or IAM role ● Flink supports AWS IAM credentials by default ● We can run Flink via Dataproc or Dataflow (as Beam supports Flink jobs)
- 39.
- 40. © Copyright. Allrights reserved. Not to be reproduced without prior written consent. Check Flink jobs status ● Check status of TaskManagers and JobManager kubectl -n flow-ververica-jobs get po ● Check any issues with deployment TaskManagers kubectl -n flow-ververica-jobs get deployment kubectl -n flow-ververica-jobs describe deployment DEPLOYMENT_NAME - check Events tab JobManagers kubectl -n flow-ververica-jobs get job kubectl -n flow-ververica-jobs describe job DEPLOYMENT_NAME - check Events tab
- 41. © Copyright. Allrights reserved. Not to be reproduced without prior written consent. Check Flink jobs status ● Check volume claim used by Ververica kubectl -n flow-ververica-jobs get pvc ● Check volume used by Ververica kubectl get pv
- 42. © Copyright. Allrights reserved. Not to be reproduced without prior written consent. Get logs from application ● Ververica compontents kubectl -n flow-ververica logs $(kubectl -n flow-ververica get po | awk 'FNR == 2 {print $1}') CONTAINER_NAME Here you can pass: appmanager, gateway or ui ● Flink job kubectl -n flow-ververica-jobs get po kubectl -n flow-ververica-jobs logs POD_NAME_FROM_1st_COMMAND To follow all newly appeared logs, add flag -f just after logs, example: kubectl -n flow-ververica-jobs logs -f POD
- 43. © Copyright. Allrights reserved. Not to be reproduced without prior written consent. Get logs from application ● Access Exception from Flink Web User Interface
- 44. © Copyright. Allrights reserved. Not to be reproduced without prior written consent. Q&A https://linkedin.com/in/albert-lewandowski albert.lewandowski@getindata.com
- 45. Thank you foryour attention!