From the course: Apache Kafka Essential Training: Building Scalable Applications
Join today to access over 24,900 courses taught by industry experts.
Batching message consumption - Kafka Tutorial
From the course: Apache Kafka Essential Training: Building Scalable Applications
Batching message consumption
- [Instructor] In this video, let's explore how consumers can receive messages in batches and process them. How do consumers receive messages from Kafka? Consumers poll Kafka periodically for new messages. The polling interval is configured through the poll method. Each consumer is allocated a set of partitions to work with. Consumers poll corresponding partition leaders for new messages. When consumers poll, a batch of messages are returned from Kafka when new messages are available. The bigger the batch sizes and higher the poll interval, the lower are the round trips and request overheads. However, such large batch sizes and poll intervals directly impact the latency of processing also. It is required to choose the right parameters based on the use case to balance between round trips and latency. How do we configure this round-trip behavior? Here are some key parameters. The first one is the poll interval, which is…
Practice while you learn with exercise files
Download the files the instructor uses to teach the course. Follow along and learn by watching, listening and practicing.
Download courses and learn on the go
Watch courses on your mobile device without an internet connection. Download courses using your iOS or Android LinkedIn Learning app.