Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Part-2: Protocol Streams (intelligent decoding - on-the-fly tree manipulation) #32

Open
newfront opened this issue Jun 3, 2024 · 0 comments
Assignees
Labels
moving-fast-and-moving-slow Rate Limiting, Triggers, Scaling Patterns for Delta Lake

Comments

@newfront
Copy link
Owner

newfront commented Jun 3, 2024

This is more advanced, it might not make it into the presentation, but given that the Protobuf definitions include validations, and we can use the validations at the edge - prior to publishing the records onto Kafka in the first place — we present a world that acts-more-like-a-normal-api-service, and less like a dumping ground for things to be fixed later!

  • protobuf-AST -> transform -> specific Delta table
  • show how to steal-the-schema, and ensure a table is generated with the correct "schema" prior to the initial table write.
@newfront newfront added the moving-fast-and-moving-slow Rate Limiting, Triggers, Scaling Patterns for Delta Lake label Jun 3, 2024
@newfront newfront added this to the Data AI Summit 2024 milestone Jun 3, 2024
@newfront newfront self-assigned this Jun 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
moving-fast-and-moving-slow Rate Limiting, Triggers, Scaling Patterns for Delta Lake
Projects
Development

No branches or pull requests

1 participant