06-streaming

Module 6: Stream Processing

Code structure

• Java examples
• Python examples
• KSQLD examples

Confluent cloud setup

Confluent cloud provides a free 30 days trial for, you can signup here

Introduction to Stream Processing

• Slides

• 🎥 6.0.1 Introduction

• 🎥 6.0.2 What is stream processing

Introduction to Kafka

• 🎥 6.3 What is kafka?

• 🎥 6.4 Confluent cloud

• 🎥 6.5 Kafka producer consumer

Kafka Configuration

• 🎥 6.6 Kafka configuration

• Kafka Configuration Reference

Kafka Streams

• Slides

• Streams Concepts

• 🎥 6.7 Kafka streams basics

• 🎥 6.8 Kafka stream join

• 🎥 6.9 Kafka stream testing

• 🎥 6.10 Kafka stream windowing

• 🎥 6.11 Kafka ksqldb & Connect

• 🎥 6.12 Kafka Schema registry

Faust - Python Stream Processing

• Faust Documentation
• Faust vs Kafka Streams

Pyspark - Structured Streaming

Please follow the steps described under pyspark-streaming

• 🎥 6.13 Kafka Streaming with Python

• 🎥 6.14 Pyspark Structured Streaming

Kafka Streams with JVM library

• Confluent Kafka Streams
• Scala Example

KSQL and ksqlDB

• Introducing KSQL: Streaming SQL for Apache Kafka
• ksqlDB

Kafka Connect

• Making Sense of Stream Data

Docker

Starting cluster

Command line for Kafka

Create topic

./bin/kafka-topics.sh --create --topic demo_1 --bootstrap-server localhost:9092 --partitions 2

Homework

• 2024 Homework

Community notes

Did you take notes? You can share them here.

• Notes by Alvaro Navas
• Marcos Torregrosa's blog (spanish)
• Notes by Oscar Garcia
• 2024 videos transcript by Maria Fisher
• Notes by Shayan Shafiee Moghadam
• Add your notes here (above this line)