Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Spike: refactor CDP componentry out of monorepo #5525

Open
mollykarcher opened this issue Nov 12, 2024 · 1 comment
Open

Spike: refactor CDP componentry out of monorepo #5525

mollykarcher opened this issue Nov 12, 2024 · 1 comment

Comments

@mollykarcher
Copy link
Contributor

mollykarcher commented Nov 12, 2024

What problem does your feature solve?

We'd like to expand the scope of responsibility for our external SDK developers that receive infrastructure grants, to include the ability to interact with a galexie-exported data lake. As a precursor to this, we'd like to have a clear, concise example that they can model their implementations after in golang.

What would you like to see?

  • Design/spike to determine which libraries/components should be moved out into a separate repository, and which should stay

Defer to follow-on ticket(tbd) for implemenation:

  • Work to refactor said components out of the go monorepo and into their own repository
  • Documentation in the public developer docs advising developers as to how to use new repo

Considerations

  • Note that the ingest library includes a lot of code that handles the lifecycle of captive core and makes the interface to it look the same as that to the data lake. For external providers, we won't be asking them to build code that interfaces with captive-core directly
  • Consider that we eventually will want to facilitate open source contributions of other data stores (S3, R2, etc). Awareness of this may influence overall design and choice of where the interfaces live
@mollykarcher mollykarcher added this to the platform sprint 54 milestone Nov 12, 2024
@mollykarcher mollykarcher changed the title ingest: refactor CDP componentry out of monorepo refactor CDP componentry out of monorepo Nov 12, 2024
@mollykarcher
Copy link
Contributor Author

Per discussion at planning meeting on 11/12, we will explicitly scope this down to just the BufferedStorageBackend. So just wrapping of a gcs client, downloading/buffering ledgers, and outputting a LCM object.

@sreuland sreuland changed the title refactor CDP componentry out of monorepo Spike: refactor CDP componentry out of monorepo Dec 3, 2024
@urvisavla urvisavla moved this from To Do to In Progress in Platform Scrum Dec 3, 2024
@urvisavla urvisavla self-assigned this Dec 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: In Progress
Development

No branches or pull requests

4 participants