Apache Flink is widely growing in popularity for its ability to perform advanced stateful computations in a way that scales to meet the demands of both high throughput and high performance use cases. Not only is Apache Flink very scalable and performant it also integrates with a wide variety of source and sink data systems like flat files (CSV, TXT, TSV), Databases, and Message Queues (Kafka, AWSKinesis, GCPPub/Sub, RabbitMQ). In this course students will learn to harness the power of Apache Flink which is a modern distributed computing framework providing a unified approach to both batch and streaming data processing workloads. This course specifically focuses on the relational programming paradigm exposed through Apache Flink’s Table API and SQL interface (with examples in Python) offering intuitive yet powerful abstractions to process vast amounts of data in either bounded (batch) or unbounded (streaming) sources. Students learn batch processing with Flink through many examples of consuming, processing, and producing results from/to the filesystem in CSV format. Students also learn stream processing with Flink through several examples consuming, processing and producing results from/to Apache Kafka running in a local Dockerized Kafka cluster. Apache Flink offers support for developing Flink applications with the Table API and SQL interface in Java, Scala and Python. However, this course focuses on using the Python bindings for Apache Flink. The focus on Python for this course was chosen due to the popularity of the Python programming language, particularly in the big data engineering ecosystem, but also due to the underrepresentation of Python in existing Apache Flink courses which primarily cover the Java and Scala APIs of Flink.
