How to build a tool for operating Flink on Kubernetes

How to build a tool for
operating Flink on Kubernetes
Andrea Medeghini
Software Engineer / Contractor

Which products are available?
Ververica dA Platform:
● Automated deployments
● Easy management of jobs
● Monitoring
● Logging (ELK)
● Trial version available

Are there free alternatives?
There are few projects on GitHub, however…
● They mainly focus on deployment
● They do not provide a complete solution
● They might not work for all use cases

Can we use Helm?
It’s good, but it doesn’t help with…
● Jobs and Savepoints
● Monitoring / Alerting
● Automatic Scaling
# helm install --name my-flink-cluster charts/flink

Wait for next Flink release?
Better integration with Kubernetes it’s coming:
● Reactive container mode
https://issues.apache.org/jira/browse/FLINK-10407
● Active Kubernetes integration
https://issues.apache.org/jira/browse/FLINK-9953

Shall we build our own tool?
It’s going to be challenging! Because…
● Flink is a distributed engine
● Flink is a stateful engine
● Jobs need to be packaged and uploaded
● Jobs need to be monitored to detect failures
● Resources need to be adjusted according to workload

Do we need an open source tool?
Everybody likes open source tools…
… how do we build one?

Overview of a Flink Cluster
● One or more JobManagers
(typically one)
● One or more TaskManagers
(typically many)
● One or more jobs packaged as
JAR files
● Storage for savepoints

Exploiting Kubernetes API
It’s all REST!
There are clients libraries…
… for many languages not only Go!
See Kubernetes Documentation:
https://kubernetes.io/docs/reference/

Control Kubernetes Programmatically
val jobmanagerStatefulSet = V1StatefulSet()
.metadata(jobmanagerMetadata)
.spec(
V1StatefulSetSpec()
.replicas(1)
.template(
V1PodTemplateSpec().spec(jobmanagerPodSpec).metadata(jobmanagerMetadata)
)
.updateStrategy(updateStrategy)
.serviceName("jobmanager")
.selector(jobmanagerSelector)
.addVolumeClaimTemplatesItem(persistentVolumeClaim)
)
api.createNamespacedStatefulSet(namespace, jobmanagerStatefulSet, null, null, null)

What resources do we need?
● StatefulSet for JobManager (1 replica)
● StatefulSet for TaskManager (N replicas)
● Services for JobManager (headless, NodePort, ...)
● PersistentVolumeClaims
● …

What conﬁguration do we need?
● Set JOB_MANAGER_RPC_ADDRESS to JobManager service
● Set TASK_MANAGER_NUMBER_OF_TASK_SLOTS to 1
● Set memory limits of container higher than max heap
● Set CPU limits to sensible value
● Configure pod affinity to spread workload
● Expose relevant ports (usually only internally)
● Add sensible labels to identify resources

Run exec against the Job Manager
How does it work?
● Kubernetes Client for
managing clusters
● Exec for executing
commands in the
containers

Easy to implement but...
● It depends on commands installed in the container
● It seems too consuming in terms of resources (we need to
run a process inside the container for each operation)
● It doesn’t enforce any protocol (stdin/stdout)

Flink Monitoring API to the rescue!
Flink has a pretty useful REST API:
● Endpoints for managing jobs
● Endpoints for managing savepoints
● Endpoints for monitoring the cluster
Is there a client library? I am afraid not…

Create client using OpenAPI
I manually crafted an
OpenAPI specification file…
… It’s tedious but the
generated client works fine!
See Swagger Documentation:
https://swagger.io/docs/specification/about/

Swagger Editor and Code generator
/v1/jobs:
get:
operationId: getJobs
summary: Returns an overview over all jobs and their current state
responses:
'200':
description: |-
200 response
content:
application/json:
schema:
$ref: '#/components/schemas/JobIdsWithStatusOverview'
...
See full specification on GitHub:
https://github.com/nextbreakpoint/flink-client/blob/master/flink-openapi.yaml

Combine all in one application
We can combine the APIs:
● Kubernetes Client for
managing clusters
● Flink Client for
managing jobs

What are the limitations?
● Where does the client live?
● Still no monitoring or automatic scaling
● NodePort or Port Forward required for each
JobManager (for each Flink Cluster)
● Port Forward doesn’t work well with file
upload (there is a problem with timeout in
the Kubernetes Client for Java)

Run controller inside Kubernetes
What are the benefits?
● It can easily access
internal resources
● It runs with its own service
account
● It can monitor the clusters
● It can rescale the clusters

Better than before but...
● One port forward is still required
● Authorization is required for API
● It doesn’t follow best practises!

We need a Kubernetes Operator!
Everybody think we need Go, but…
… an Operator is like a pattern…
… and we can use any programming language!
Operator SDK for Go:
https://github.com/operator-framework/operator-sdk
Operator Pattern:
https://coreos.com/blog/introducing-operators.html

Custom Resource Deﬁnition
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: flinkclusters.beta.nextbreakpoint.com
spec:
group: beta.nextbreakpoint.com
versions:
- name: v1
served: true
storage: true
scope: Namespaced
names:
plural: flinkclusters
singular: flinkcluster
kind: FlinkCluster
shortNames:
- fc
/api/beta.nextbreakpoint.com/v1/namespaces/*/flinkclusters
# kubectl create -f flink-crd.yaml
# kubectl get crd
NAME AGE
flinkclusters.beta.nextbreakpoint.com 1d
…

Custom Objects
apiVersion: "beta.nextbreakpoint.com/v1"
kind: FlinkCluster
metadata:
name: test
spec:
clusterName: test
environment: test
pullSecrets: regcred
pullPolicy: Always
flinkImage: nextbreakpoint/flink:1.7.2-1
sidecarImage: flink-workshop-jobs:2
sidecarServiceAccount: flink-operator
sidecarClassName: com.nextbreakpoint.flink.jobs.TestJob
sidecarJarPath: /com.nextbreakpoint.flinkworkshop-1.0.0.jar
sidecarParallelism: 1
sidecarArguments:
- --BUCKET_BASE_PATH
- file:///var/tmp
# kubectl create -f cluster.yaml
# kubectl get flinkclusters
NAME AGE
test 4s

The Operator Loop
1. Receive updates of Custom Objects
2. Receive updates of StatefulSets,
Services, PVCs, …
3. Compare desired state to actual
state
4. Adjust current state to match
desired state
5. Repeat from 1

Run a Flink Operator
What are the benefits?
● It follows Kubernetes
best practises
● It runs with its own
service account
● We only need to create
cluster objects

Operator meets Controller
They can operate together:
● Use operator with CD pipeline
● Use controller for manual ops
● Use controller for monitoring
● Use controller for alerting
● Use controller for scaling

Time for a demo !
A preview of Flink K8S Toolbox:
● Easy installation
● Easy deployments
● Jobs management
● Cluster metrics
● Cluster scaling

Monitoring and Scaling
We can use Flink API for:
● Watching jobs status and
alerting when something is
broken
● Observing cluster metrics
and scaling cluster when
required

Checkpoints/Savepoints
We can use Flink API for:
● Monitoring checkpoints
● Managing savepoints
● Retrieving last savepoint

Continuous Delivery
We can use tools like Flux:
● Push changes into Git repo
● Changes are automatically
applied to resources
Flux (I haven’t actually tried it)
https://github.com/weaveworks/flux

Nice features to have...
● Pluggable alerting strategy
● Pluggable scaling strategy
● Web console
● Secure access
● Support for HA mode
● …

It’s all free!
Flink Kubernetes Toolbox:
https://github.com/nextbreakpoint/flink-k8s-toolbox
Related projects:
https://github.com/nextbreakpoint/flink-client
https://github.com/nextbreakpoint/flink-workshop
https://github.com/nextbreakpoint/kubernetes-playground

Fine.
Where to follow:
@AndreaMedeghini
nextbreakpoint.com

How to build a tool for operating Flink on Kubernetes

More Related Content

What's hot

Similar to How to build a tool for operating Flink on Kubernetes

Recently uploaded

How to build a tool for operating Flink on Kubernetes