A talk entitled "Fish and Chips and Apache Kafka®"
This is a talk for presentation at PyCon UK 2022, 16th - 18th September 2022.
It was presented live, and a video recording is available.
It was also presented beforehand (virtually) at CamPUG on 6th September 2022.
Note to anyone visiting after CamPUG: The tag CamPUG should give you the slides as presented at CamPUG.
First, the links mentioned in the slides:
- Aiven, where I work, and our developer documentation at https://docs.aiven.io/ and https://github.com/aiven/devportal
- Get a free trial of Aiven services at https://console.aiven.io/signup/email
- Aiven is hiring (at least as of September 2022, and probably for a good while longer). See https://aiven.io/careers
The Kafka libraries used in the demos:
- kafka-python, which is a very good translation of the standard Java API to Python
- aiokafka, which provides asynchronous access to Kafka from Python. It's not as fully featured as kafka-python, but there's nothing stopping you from using both of them together.
For the terminal user interface I used in the demos, see Textual and Rich.
The following may also be of interest:
- Apache Kafka® simply explained on the Aiven blog, for a friendly explanation of the Apache Kafka fundamentals
- Teach yourself Apache Kafka® and Python with a Jupyter notebook on the Aiven blog. The Jupyter notebook referenced is at https://github.com/aiven/python-notebooks-for-apache-kafka
- Create a JDBC sink connector in the Aiven developer documentation shows how to setup Kafka Connect using the Aiven web console, and is thus useful for the "homework" on using Kafka Connect to output data to PostgreSQL.
Not used in the demos, but another interesting Kafka library for Python is faust, which provides Kafka Streams, allowing (for instance) merging of events from multiple Kafka topics.
Lastly, but definitely not least, The Log: What every software engineer should know about real-time data's unifying abstraction is the 2013 paper by Jay Kreps that explains the concepts behind Kafka. It is very worth a read.
The source code for the demonstration programs is in the src directory. See the readme.rst for how to run them.
The videos of the demos (which I use during the talk) are in a separate repository, at https://github.com/tibs/fish-and-chips-and-kafka-videos (it seems unfair to make you download them from here if you're not interested).
The slides are written using reStructuredText, and thus intended to be readable as plain text.
The sources for the slides are in slides.rst.
Note that github will present the .rst files in rendered form as HTML,
albeit using their own styling (which is occasionally a bit odd). If you want
to see the original reStructuredText source, you have to click on the "Raw"
link at the top of the file's page.
The PDF slides at 16x9 aspect ratio (slides-16x9.pdf) are stored here for convenience.
The PDF files may not always be as up-to-date as the source files, so check their timestamps.
The QR code on the final slide was generated using the command line program
for qrencode, which I installed with brew install qrencode on my Mac.
I use poetry to manage the dependencies needed to build the PDFs, and rst2pdf and its dependencies to do the actual work.
You will also need an appropriate make program if you want to use the
Makefile.
So, for instance, in this directory I would start a new poetry shell using:
$ poetry shell
and then install the dependencies using:
$ poetry install
After that, you should be able to use the Makefile to create the PDF files. For instance:
$ make pdf
to make them all.
For other things the Makefile can do, use:
$ make help
If you wish, you can exit the poetry shell using exit.
Apache, Apache Kafka, Kafka, and the Kafka logo are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries
Postgres and PostgreSQL are trademarks or registered trademarks of the PostgreSQL Community Association of Canada, and used with their permission
This talk and its related files are released under a Creative Commons Attribution-ShareAlike 4.0 International License. The source code for the demo programs is dual-licensed as CC Attribution Share Alike and MIT.
