From the course: Apache Kafka Essential Training: Building Scalable Applications
Join today to access over 24,900 courses taught by industry experts.
Managing partition counts - Kafka Tutorial
From the course: Apache Kafka Essential Training: Building Scalable Applications
Managing partition counts
- [Host] In this chapter, we will discuss some best practices for building scalable applications with Kafka. In this video, we will focus on managing partitions. One of the most important and difficult decisions to make while creating a topic is deciding on the number of partitions. Partition counts cannot be decreased after a topic creation so care should be taken in choosing the right size. If the number of partitions are too few then fewer brokers are involved in handling the topic and there is a lot of serialized processing happening. As a result, there could be producer lag and consumer lag. Some brokers and consumers may be overworked, handling big partitions, while others may be starved for work as they don't have any partitions assigned to them. On the other hand, if the number of partitions is too high, it results in more broker resources like file handles and memory. Note that there is an overhead with each…
Practice while you learn with exercise files
Download the files the instructor uses to teach the course. Follow along and learn by watching, listening and practicing.
Download courses and learn on the go
Watch courses on your mobile device without an internet connection. Download courses using your iOS or Android LinkedIn Learning app.