Complete guide for Apache Kafka stream processing including producers, consumers, Kafka Streams, connectors, schema registry, and production deployment

Kafka Stream Processing A comprehensive skill for building event driven applications with Apache Kafka. Master producers, consumers, Kafka Streams, connectors, schema registry, and production deployment patterns for real time data processing at scale. When to Use This Skill Use this skill when: Building event driven microservices architectures Processing real time data streams and event logs Implementing publish subscribe messaging systems Creating data pipelines for analytics and ETL Building streaming data applications with stateful processing Integrating heterogeneous systems with Kafka Connect Implementing change data capture (CDC) patterns Building real time dashboards and monitoring systems Processing IoT sensor data at scale Implementing event sourcing and CQRS patterns Building distributed systems requiring guaranteed message delivery Creating real time recommendation engines Processing financial transactions with exactly once semantics Building log aggregation and monitoring pipelines Core Concepts Apache Kafka Architecture Kafka is a distributed streaming platform designed for high throughput, fault tolerant, real time data processing. Key Components: 1. Topics : Named categories for organizing messages 2. Partitions : Ordered, immutable sequences of records within topics 3. Brokers : Kafka servers that store and serve data 4. Producers : Applications that publish records to topics 5. Consumers : Applications that read records from topics 6. Consumer Groups : Coordinated consumers sharing workload 7. ZooKeeper/KRaft : Cluster coordination (ZooKeeper legacy, KRaft modern) Design Principles: Topics and Partitions Topics are logical channels for data streams. Each topic is divided into partitions for parallelism and scalability. Partition Characteristics: Ordered : Messages within a partition are strictly ordered Immutable : Records cannot be modified after written Append only : New records appended to partition end Retention : Configurable retention by time or size Replication : Each partition replicated across brokers Adding Partitions: Note : Adding partitions doesn't redistribute existing data and may affect consumers using custom partitioning. Stream Partitions and Tasks Message Delivery Semantics At Least Once Delivery: Producer retries until acknowledgment received Consumer commits offset after processing Risk: Duplicate processing on failures Use case: When duplicates are acceptable or idempotent processing At Most Once Delivery: Consumer commits offset before processing No producer retries Risk: Message loss on failures Use case: When data loss acceptable (e.g., metrics) Exactly Once Semantics (EOS): Transactional writes with idempotent producers Consumer reads committed messages only Use case: Financial transactions, critical data processing Producer Load Balancing The producer directs data to the partition leader broker without a routing tier. Kafka nodes provide metadata to producers for directing requests to the correct partition leaders. Producers can implement custom partitioning logic or use random distribution. Producers Kafka producers publish records to topics with configurable reliability and performance characteristics. Producer API Producer Configuration Essential Settings: 1. bootstrap.servers : Kafka cluster connection string Format: host1:9092,host2:9092,host3:9092 Use multiple brokers for fault tolerance 2. key.serializer / value.serializer : Data serialization org.apache.kafka.common.serialization.StringSerializer org.apache.kafka.common.serialization.ByteArraySerializer Custom serializers for complex types 3. acks : Acknowledgment level 0 : No acknowledgment (fire and forget) 1 : Leader acknowledgment only all / 1 : All in sync replicas acknowledge (strongest durability) 4. retries : Retry count for failed sends Default: 2147483647 (Integer.MAX VALUE) Set to 0 to disable retries 5. enable.idempotence : Exactly once semantics true : Enables idempotent producer (prevents duplicates) false : Default behavior Performance Tuning: 1. linger.ms : Batching delay Default: 0 (send immediately) Higher values (5 100ms) increase throughput via batching Trade off: Latency vs throughput 2. batch.size : Batch size in bytes Default: 16384 (16KB) Larger batches improve throughput Maximum single batch size per partition 3. compression.type : Message compression Options: none , gzip , snappy , lz4 , zstd Reduces network bandwidth and storage CPU overhead for compression/decompression 4. buffer.memory : Total producer buffer memory Default: 33554432 (32MB) Memory for buffering unsent records Producer blocks when buffer full Producer Example (Java) Producer Best Practices 1. Use Batching : Configure linger.ms and batch.size for throughput 2. Enable Idempotence : Set enable.idempotence=true for reliability 3. Handle Errors : Implement proper callback error handling 4. Use Compression : Enable compression for large messages 5. Partition Keys : Use meaningful keys for ordered processing 6. Close Producers : Always close producers in finally blocks 7. Monitor Metrics : Track producer metrics (record send rate, compression rate) 8. Resource Pools : Reuse producer instances when possible Consumers Kafka consumers read records from topics, supporting both individual and group based consumption. Consumer Groups Consumer groups enable parallel processing with automatic load balancing and fault tolerance. Key Concepts: Group ID : Unique identifier for consumer group Partition Assignment : Each partition consumed by one consumer in group Rebalancing : Automatic reassignment when consumers join/leave Offset Management : Group tracks committed offsets per partition Consumer Group Monitoring: Consumer Configuration Essential Settings: 1. bootstrap.servers : Kafka cluster connection 2. group.id : Consumer group identifier 3. key.deserializer / value.deserializer : Data deserialization 4. enable.auto.commit : Automatic offset commits true : Auto commit offsets periodically false : Manual offset management 5. auto.offset.reset : Behavior when no offset found earliest : Start from beginning latest : Start from end none : Throw exception Consumer Specific Kafka Streams Defaults: Consumer Example (Java) Consumer Offset Management Offset Commit Strategies: 1. Auto commit (default): Simple but risky for at least once delivery May commit before processing completes 2. Manual Synchronous Commit: Blocks until commit succeeds Guarantees offset committed before continuing Lower throughput 3. Manual Asynchronous Commit: Non blocking commit Higher throughput Handle failures in callback 4. Hybrid Approach: Async commits during processing Sync commit before rebalance/shutdown Consumer Best Practices 1. Choose Right Auto commit : Disable for at least once semantics 2. Handle Rebalancing : Implement ConsumerRebalanceListener 3. Process Efficiently : Minimize poll() call duration 4. Graceful Shutdown : Close consumers properly 5. Monitor Lag : Track consumer lag metrics 6. Partition Assignment : Understand assignment strategies 7. Thread Safety : Kafka consumers are NOT thread safe 8. Error Handling : Retry logic for transient failures Kafka Streams Kafka Streams is a client library for building real time streaming applications with stateful processing. Kafka Streams Architecture Processor Topology: Sub topologies: Applications are decomposed into sub topologies connected by repartition topics. Each sub topology can scale independently. KStream vs KTable vs GlobalKTable KStream : Immutable stream of records KTable : Changelog stream (latest value per key) GlobalKTable : Fully replicated table available to all instances Writing Streams to Kafka Any streams and tables may be (continuously) written back to a Kafka topic. The output data might be re partitioned depending on the situation. Repartitioning Manual repartitioning with specified partition count: Kafka Streams manages the generated topic as an internal topic, ensuring data purging and allowing for scaling downstream sub topologies. This operation is useful when key changing operations are performed beforehand and auto repartitioning is not triggered. Joins and Co partitioning Stateful Operations Kafka Streams supports stateful operations like aggregations, windowing, and joins using state stores. State Store Types: Key Value Stores : For aggregations and joins Window Stores : For time based operations Session Stores : For session based aggregations State Store Configuration: Application Parallelism Kafka Streams Configuration Client Prefixes: Specific Consumer Types: Topic Configuration: Exactly Once Semantics in Streams EOS Configuration: Topology Naming and Stability Default Topology (Auto generated names): Explicit Naming for Stability: Enforce Explicit Naming: This prevents the application from starting with auto generated names, guaranteeing stability across topology updates. Topology Optimization Topology optimization allows reuse of source topics as changelog topics, crucial when migrating from KStreamBuilder to StreamsBuilder. WordCount Example with Topology This topology shows two disconnected sub topologies, their sources, processors, sinks, and the repartition topic (counts store repartition) used for shuffling data by aggregation key. Schema Registry Schema registries enforce data contracts between producers and consumers, ensuring data integrity and preventing malformed events. Schema Registry Benefits 1. Schema Evolution : Manage schema changes over time 2. Compatibility Checking : Enforce backward/forward compatibility 3. Centralized Management : Single source of truth for schemas 4. Type Safety : Compile time type checking 5. Documentation : Auto generated schema documentation 6. Versioning : Track schema versions per subject Schema Formats Supported Formats: Avro : Compact binary format with rich schema evolution JSON Schema : Human readable with schema validation Protobuf : Google's Protocol Buffers Schema Evolution Compatibility Modes 1. BACKWARD : New schema can read old data 2. FORWARD : Old schema can read new data 3. FULL : Both backward and forward compatible 4. NONE : No compatibility checking Avro Producer Example Kafka Connect Kafka Connect is a framework for streaming data between Kafka and external systems. Connector Types Source Connectors : Import data into Kafka Database CDC (Debezium) File systems Message queues Cloud services (S3, BigQuery) APIs and webhooks Sink Connectors : Export data from Kafka Databases (JDBC, Elasticsearch) Data warehouses Object storage Search engines Analytics platforms Connector Configuration Source Connector Example (JDBC): Sink Connector Example (Elasticsearch): Change Data Capture (CDC) CDC captures database changes and streams them to Kafka in real time. Benefits: Real time data synchronization Event sourcing from existing databases Microservices data integration Zero downtime migrations Popular CDC Connectors: Debezium (MySQL, PostgreSQL, MongoDB, SQL Server) Oracle GoldenGate Maxwell's Daemon Topic Management Topic Operations Produce Test Messages Monitoring and Metrics Common Metrics Key Monitoring Areas 1. Producer Metrics: record send rate record er

kafka-stream-processing