Kafka Connect’s Single Message Transform functionality is really useful for building data pipelines that modify the data passing through. For more complex transformations (such as those requiring state, or joins, etc) check out Kafka Streams or ksqlDB.
Each of the tutorials below has the full code available for you to try using the provided Docker Compose, and a complete 🎥 recording to go with it.
-
Day 1 -
InsertField- add message timestamp as a field - a sink -
Day 2 -
ValueToKeyandExtractField- set the message key - a field from the value -
Day 3 -
Flatten- turn a nested structure into a flat one -
Day 4 -
RegexRouter- change the topic name based on a pattern match and replacement -
Day 5 -
MaskField- mask the value of fields with a fixed replacement string -
Day 6 -
InsertField- same SMT as Day 1, this time showing adding to the payload the topic name, Kafka message partition and offset, as well as hardcoded values -
Day 7 -
TimestampRouter- change the topic name based on the timestamp of the Kafka message -
Day 8 -
TimestampConverter- Convert timestamp fields between unix epoch, string, Timestamp, Date, and Time -
Day 9 -
Cast- Convert string, numeric, and boolean field types -
Day 10 -
ReplaceField- Drop or rename fields -
Day 11 -
Filterandpredicates- Drop messages based on conditions -
Day 12 - Community Transformations (
ChangeTopicCase,ExtractTimestamp,TimestampNowField)
🎥 Check out the recordings for all of these in this YouTube playlist