Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update ingest/change_compactor with extended Change struct #5540

Open
chowbao opened this issue Dec 2, 2024 · 3 comments
Open

Update ingest/change_compactor with extended Change struct #5540

chowbao opened this issue Dec 2, 2024 · 3 comments
Assignees
Labels

Comments

@chowbao
Copy link
Contributor

chowbao commented Dec 2, 2024

Related to #5535

The change_compactor should be updated to use the new fields in the Change struct

New fields:

  • Reason
  • OperationIndex
  • Transaction
  • Ledger
  • LedgerUpgrade

None of the existing Type, Pre, Post field logic needs to be changed.

The compacted changes from AddChange(change Change) should take the most recent fields from above if there is already an existing change for c.cache[ledgerKeyString]

For example in https://github.com/stellar/go/blob/master/ingest/change_compactor.go#L152-L166

	switch existingChange.LedgerEntryChangeType() {
	case xdr.LedgerEntryChangeTypeLedgerEntryCreated:
		// If existing type is created it means that this entry does not
		// exist in a DB so we update entry change.
		c.cache[ledgerKeyString] = Change{
			Type: key.Type,
			Pre:  existingChange.Pre, // = nil
			Post: change.Post,
			Reason: change.Reason,
			OperationIndex: change.OperationIndex,
			Transaction: change.Transaction,
			Ledger: change.Ledger,
			LedgerUpgrade: change.LedgerUpgrade,
		}

Alternatively instead of propagating the latest Change.Reason/OperationIndex/etc... to c.cache[ledgerKeyString] = Change{} it is possible to keep the existingChange.Reason/OperationIndex/etc... instead of change.Reason/OperationIndex/etc.... This would be the less preferred option from stellar-etl

@urvisavla
Copy link
Contributor

@chowbao It would be helpful for my understanding if you could explain how these new fields can be used (with an example?). Thanks!

@chowbao chowbao self-assigned this Dec 10, 2024
@chowbao chowbao moved this from Backlog to In Progress in Platform Scrum Dec 10, 2024
@chowbao
Copy link
Contributor Author

chowbao commented Dec 10, 2024

@chowbao It would be helpful for my understanding if you could explain how these new fields can be used (with an example?). Thanks!

Of course

So in stellar-etl we would use the changes to add new columns to some of our tables. For example, our contract_data state table that records the compacted ledger entry changes for contract_data. New columns that would be helpful for us to add would be like the transaction_hash or ledger_sequence that the compacted change occurred. This is nice because now we wouldn't have to manually pass the LedgerCloseMeta as we process changes (Ledger already in the change).

stellar-etl currently has this flow in the code if that helps clear things up as well

By also passing new fields like OperationIndex and Transaction we can also reasonable add new columns to the contract_data table like transaction_hash or operation_id that is tied to the change

@chowbao chowbao added data-scrum ingest New ingestion system labels Dec 10, 2024
@chowbao
Copy link
Contributor Author

chowbao commented Dec 17, 2024

We will copy and modify the changeCompactor for stellar-etl only within go/stellar-etl

The changeCompactor within the stellar/go/ingest will not be changed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
Status: In Progress
Development

No branches or pull requests

3 participants