Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New storage write API #2355

Closed
wants to merge 14 commits into from
Closed

New storage write API #2355

wants to merge 14 commits into from

Conversation

brentstone
Copy link
Collaborator

@brentstone brentstone commented Jan 3, 2024

Describe your changes

StorageWrite API changes.

Some notes:

  • Options to merklize data and add data to diffs are provided to various write and delete functions
  • Specific keys that are changed (all are no longer merklized or persisted in diffs):
    • masp/note_commitment_anchor/<hash>
    • masp/commitment_tree
    • masp/convert_anchor
    • masp/nullifiers/<hash>
    • masp/pin-<id>
    • ibc/clients/counter
    • ibc/connections/counter
    • ibc/channelEnds/counter
  • Data that is not written to diffs still maintain diffs for one block prior to the current height in order to maintain rollback functionality
  • Various uses of write_bytes are abstracted abstracted to a different write API call
  • Tests

Indicate on which release or other PRs this topic is based on

v0.30.1

Checklist before merging to draft

  • I have added a changelog
  • Git history is in acceptable state

@cwgoes cwgoes mentioned this pull request Jan 8, 2024
@brentstone brentstone force-pushed the brent/new-storage-write branch 4 times, most recently from b27bc60 to 5a6c6f8 Compare January 11, 2024 23:31
@brentstone brentstone marked this pull request as ready for review January 12, 2024 02:48
tx_prelude/src/lib.rs Outdated Show resolved Hide resolved
@brentstone brentstone mentioned this pull request Jan 13, 2024
@brentstone brentstone force-pushed the brent/new-storage-write branch 2 times, most recently from 03c5977 to fd7e817 Compare January 14, 2024 21:03
benches/txs.rs Outdated Show resolved Hide resolved
Copy link
Member

@yito88 yito88 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall, it looks good to me. I left some comments

apps/src/lib/node/ledger/storage/rocksdb.rs Outdated Show resolved Hide resolved
core/src/ledger/ibc/context/common.rs Outdated Show resolved Hide resolved
core/src/ledger/storage/mod.rs Outdated Show resolved Hide resolved
core/src/ledger/storage_api/mod.rs Outdated Show resolved Hide resolved
shared/src/ledger/ibc/mod.rs Outdated Show resolved Hide resolved
shared/src/ledger/native_vp/masp.rs Outdated Show resolved Hide resolved
yito88
yito88 previously approved these changes Jan 16, 2024
crates/state/src/lib.rs Outdated Show resolved Hide resolved
tzemanovic
tzemanovic previously approved these changes Jan 19, 2024

// If not persisting the diffs, remove the existing diffs from two block
// heights earlier than `height`
if !persist_diffs {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this works. This deletes a key with height height - 2. But say that we write a non-diff key at height 5 and than the next write happens at height 7, than we'll try to delete the keys 6/old/ and 6/new which don't exist. Instead we should be deleting the keys 5/old and 5/new. This can build up and we can end up with quite a few stale keys in storage


// If not persisting the diffs, remove the existing diffs from two block
// heights earlier than `height`
if !persist_diffs {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here

Copy link
Contributor

@cwgoes cwgoes left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new API is definitely an improvement. A few minor nits and some questions on negative gas.

batch.0.put_cf(cf, new_val_key, new_value);
}

// If not persisting the diffs, remove the last diffs.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems like the same code as above? Can we abstract this into a function?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

unfortunately they don't have the same interface - I think we'll want to get rid of the non-batch methods though

}

/// Write with diffs but no merklization
pub fn write_without_merkl(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit, but it's "merkle"

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed in 9ca40ed

StorageModification::Write {
ref value,
action: _,
} => len as i64 - value.len() as i64,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What happens here if value.len() > len? Do we charge negative gas? Is this intended?

Copy link
Member

@tzemanovic tzemanovic Jan 22, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is only used for size_diff and we don't do anything with it on the caller side(s). We don't subtract from gas cost

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah ah OK nvm then

StorageModification::Write {
ref value,
action: _,
} => len as i64 - value.len() as i64,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same question (what happens if len - value.len() < 0 ?)

tzemanovic added a commit that referenced this pull request Jan 22, 2024
* brent/new-storage-write:
  rename new fns
  prune non-persisted DB diffs
  No merkle/diffs for remaining masp keys
  changelog: add #2355
  ibc: storage keys refactor
  handle non-merklized values in `State` methods
  some TODOs and another comment
  remake wasms for tests
  ibc writes without merklization or diffs
  tests for writing without merklization or diffs
  use consts instead of string literals
  abstract instances of `write_bytes` to `write`
  masp writes without merklization or diffs
  storage write/delete options for inclusion in merkle tree and diffs
@brentstone brentstone mentioned this pull request Jan 24, 2024
2 tasks
@tzemanovic
Copy link
Member

closing in favor of #2438

@tzemanovic tzemanovic closed this Jan 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants