Skip to content

Releases: streamingfast/substreams

v0.0.12

03 Jun 18:32
@maoueh maoueh
v0.0.12
7b30088
Compare
Choose a tag to compare
v0.0.12

Summary

  • Fix null pointer exception at the end of CLI run in some cases.

  • Do log last error when the CLI exit with an error has the error is already printed to the user and it creates a weird behavior.

Changelog

  • b10966f Improved error reporting in CLI
  • 7b30088 Preparing for release 0.0.12
  • 13b2945 Updated releaase notes for v0.0.11
  • bcebc40 remove cancel() if prog is nil
  • 65b3740 returning ctx.Err() if ctx cancelled caught in the scheduler. caused issue that an error would be missed to be caught at top level
  • 721aeb1 worker pool: streamline contexts
Assets 7
Loading

v0.0.11

03 Jun 06:13
@maoueh maoueh
v0.0.11
d615bad
Compare
Choose a tag to compare
v0.0.11

Changelog

  • 27872dc Added mean on top level README (all linking to direct docs links) and reduce emphasis on Developer Preview
  • e631024 Added updates rate in CLI.
  • 105768c Better boundaries for job scheduling.
  • 20bd9e4 Better failure capture when a module fails.
  • c67dbe1 Exclude GitBook: commits from showing up in the changelog
  • d615bad Fixed all doctests, remove all ignores, move docs in substreams-macro directly to substreams::handlers
  • 936023c Merge remote-tracking branch 'origin/develop' into develop
  • 32d4183 Permit arguments to be passed to docker image on run directly
  • 916fc14 Preparing for 0.0.11 release
  • 43f0c59 Prioritize first ranges
  • 682c864 Send the writing of the output cache into a goroutine
  • b98ca77 better log
  • 363fa03 crate: substreams bumped version
  • 95dec45 fix squashing partial range
  • 1204cc1 fix squashing partial range again
  • dc65d53 pad block nums in tui
  • d16f0ae release terminal on ctrl+c
  • bd9d4d6 state: add retry for writing state to dstore, normalize derr retry to 3 everywhere
Assets 7
Loading

v0.0.10-beta

02 Jun 19:56
@abourget abourget
v0.0.10-beta
cf516ca
This commit was signed with the committer’s verified signature.
gitbook-bot GitBook Bot
GPG key ID: 07D2180C7B12D0FF
Learn about vigilant mode.
Compare
Choose a tag to compare
v0.0.10-beta

Changelog

  • 25a5b5d "state: keep delete of merge data for compatibility of files"
  • 20bd9e4 Better failure capture when a module fails.
  • 86e1db1 Fix log propagation between instances
  • 009a6b7 GitBook: [#62] No subject
  • c9dd9a2 GitBook: [#63] No subject
  • b4266f1 GitBook: [#64] No subject
  • d583d50 GitBook: [#65] No subject
  • 0b2036c GitBook: [#67] No subject
  • fd56dd5 GitBook: [#68] No subject
  • 8fab00b GitBook: [#69] more-additions
  • aa3827c GitBook: [#70] No subject
  • 7d84c92 GitBook: [#71] No subject
  • 5f13b4f GitBook: [#73] No subject
  • cf516ca GitBook: [#74] No subject
  • 5e293b1 Ok, now solved the issue. By simplifying!
  • 10fc034 Reset the instances between blocks.
  • a5c6797 small fix for subsequent executions of the same substreams which caused edge case issue and not detecting the stores where ready
  • 74c7410 state: output value types all normalized to lowercase
  • 8277aa4 state: remove merge data special key. no longer used
Assets 7
Loading

v0.0.9-beta

02 Jun 06:05
@abourget abourget
v0.0.9-beta
009a6b7
This commit was signed with the committer’s verified signature.
gitbook-bot GitBook Bot
GPG key ID: 07D2180C7B12D0FF
Learn about vigilant mode.
Compare
Choose a tag to compare
v0.0.9-beta

Changelog

substreams CLI

  • Added substreams run -o ui|json|jsonl|module-json|module-jsonl.

Commits

  • 5a4a193 Add a clause to write a temporary KV when not on boundaries.
  • eb53701 Added a bit of time between crate publishing
  • 5cd7e55 Added hex_literal macro re-export
  • 6fc452a Added more documentation
  • c282412 Align bounds properly.
  • 66c27c6 Always clean-up terminal
  • 3887379 An attempt to fix hanging, improved
  • 893b9b4 Bar graph! Hit 'm' to see :)
  • 1ba6af7 Better TUI
  • eb83089 Better clean-up and work on progress.
  • 322aa2a Better mermaid graph.
  • 6b38234 Better range calculations.
  • 79d46e7 Better ranges again.
  • ee0665e Better var name
  • 40ff1a0 COmment
  • a517c8d Cosmetics
  • 2a287f3 Default to BarMode. A few refactors.
  • 14e5358 Do error checking on incoming manifests, client and server.
  • 64616b3 First draft at bubbletea integration. Takes control of keyboard mon enfouaré.
  • 296f8e3 Fix change log
  • 60e2148 Fix documentation setup
  • 6c09bf2 Fix more range management issues.
  • fc1f855 Fixed small typo in docs
  • 79cf059 Fixup latest tests.
  • 8f32e9f Fixup progress.
  • c12c7ef Fixup some race between channels.
  • 2d8ba92 Fixup the squashing world
  • ff0b8a8 Fixup which ranges we're using
  • 6c99b8e Found a subtle bug that impeched the merging :)
  • d98928e Four new output modes
  • 92ef8fc Improve ranges again
  • a9cd3ff Incredible progress :)
  • 8569066 Merge commit 'ee0665ef71b94a1d10f72805fffd42f9b0c61956' into develop
  • 1f0962d Merge remote-tracking branch 'origin/develop' into develop
  • 37970e3 Properly replace the stores after a backprocessing shot.
  • 5d263fe Quick rename.
  • c95e01e Remove README by default
  • 1e228ce Remove the BlockRange on the Store
  • 731dc47 Removed extraneous import
  • 0b8f5e5 Removed metadata section in README.md
  • dff1fdd Removed progress package, now lives in the TUI. Repacked decode and print into a more formal and expansive TUI.
  • 55332d8 Removed unused code
  • 05d34b4 Rename
  • bce17d6 Revert "small fix on kv getting reinitialized on each new squasher for each store"
  • fd2c258 Simplify output
  • 8434b97 Small update to docs
  • eb6e21b SynchronizeStore to output the stores when they are built.
  • a631908 Updated to use tagged release of dependencies in docs
  • b8ba02c When we know there aren't requests from the STRATEGY we early exit.
  • ad68e06 Working version for SF Ethereum: * Need to compute the merge ranges better.
  • a1d7a32 Works!!! Oh well :)
  • 109259b adding tests for merge block ranges
  • d019739 doc: testing layout and file structure
  • 2a545dd fix back process not returning when no back process request need to be run
  • 8d8ef94 fixing waiter tests
  • a61c1ff getting out of the go func when we get and error from the worker of a scheduler
  • ff0041a orchestrator: add stringer methods for debugging, waiter does not wait for stores if its initial block is gt desired block num
  • d6f776e orchestrator: add tests
  • f1f7352 orchestrator: concurrnt squasher test
  • fcd2a88 orchestrator: fix squash test
  • e0df979 orchestrator: refactoring waiter condition, update strategy test
  • 4f5da66 pool: add log for notify
  • 3a1a8ca renaming stuff and adding logs
  • 96bf4e9 replacing some println statements for some zlog and fixing partial/kv boundary condition
  • 17c685d small fix for run cmd
  • 941dfe5 small fix on kv getting reinitialized on each new squasher for each store
Assets 7
Loading

v0.0.8-beta

30 May 19:59
@maoueh maoueh
v0.0.8-beta
409f4c1
Compare
Choose a tag to compare
v0.0.8-beta

Changelog

  • a9384e3 Adjusted release script to be more robust and correclty run publish commands in dry mode
  • 409f4c1 Prepare for 0.0.8-beta release
  • 6a986f2 Replaced rust/substreams/build.rs with explicit generation using protoc-gen-prost plugin
  • 4f68088 Updated to for next release, removed statik again
  • 8159f8d fix builder kv not initialize
  • dc9a3ca now failed when a store is reference as input with a wrong name
Assets 7
Loading

v0.0.7-beta

30 May 16:54
@maoueh maoueh
v0.0.7-beta
59215b0
Compare
Choose a tag to compare
v0.0.7-beta

Summary

  • Changed startBlock to initialBlock in substreams.yaml manifests.

  • code: is now defined in the binaries section of the manifest, instead of in each module. A module can select which binary with the binary: field on the Module definition.

  • Added substreams inspect ./substreams.yaml or inspect some.spkg to see what's inside. Requires protoc to be installed (which you should have anyway).

  • Added command substreams protogen that writes a temporary buf.gen.yaml and generates Rust structs based on the contents of the provided manifest or package.

  • Added substreams::handlers macros to reduce boilerplate when create substream modules.

    substreams::handlers::map is used for the handlers corresponding to modules of type map. Modules of type map should return a Result where the error is of type Error

    /// Map module example
#[substreams::handlers::map]
pub fn map_module_func(blk: eth::Block) -> Result<erc721::Transfers, Error> {
     ...
}

    substreams::handlers::store is used for the handlers corresponding to modules of type store. Modules of type store should have no return value.

    /// Map module example
#[substreams::handlers::store]
pub fn store_module(transfers: erc721::Transfers, s: store::StoreAddInt64, pairs: store::StoreGet, tokens: store::StoreGet) {
      ...
}

Changelog

  • eceec63 Added --exclude-paths flag on substream protocgen
  • 4945217 Added Substreams crate documentation
  • 82ae6b7 Added protogen
  • c02a5b4 Added some missing well known proto descriptors in the system.pb, removed statik replaced by
  • 621ba88 Added support for the sf.susbtreams.v1.Clock in inputs.
  • 1a779ca Added top-level doc.. really thin though :)
  • 42517ea Avoid panic in state builder when missing deltas is encountered
  • 81a642c Better print
  • c9742a0 Bunch of renames and refactors. BREAK BACKWARDS COMPATIBILITY. Needs to be redeployed everywhere. Packages need to be rebuilt and repackaged. New substreams needs to be used to interact with server.
  • db26ea7 Change all those store methods helpers to be u64 instead of i64, as that's the type we use in all ordinals all over the place (Eth, and Solana).
  • 153b2cf Cleaned up some unecessary deps
  • ce857aa Cleaned up substream lib and exported macros
  • c7debad Correctly handle error on call to grpcClientFactory
  • d380853 Doc fixes.
  • 29bdd3f Ensure that tests passes in rust code
  • f9a32d9 First draft at documentaiton.
  • 9dca46b Fix manifest info formatting
  • 280b9e1 Fix pack output message
  • 5545177 Fix proto import dependancy
  • 134a5f7 Fix typo
  • ec40b6c Fixed up docs. Added set_many for a few.
  • 59215b0 Fixed wrong version in Cargo on substreams crate
  • c2633a4 Fixup protogen
  • c260fca Fixup range.
  • 0286a7f Fixup some retries.
  • 0dc3c0a GitBook: [#10] No subject
  • 68efa88 GitBook: [#12] No subject
  • f4ab67a GitBook: [#13] Modules and packages
  • cd2ae90 GitBook: [#20] No subject
  • 43158ee GitBook: [#23] getting-started
  • ee01b07 GitBook: [#24] Better docs, merge strategies, improvements to modules.
  • bf2ee07 GitBook: [#25] getting-started
  • 709ee88 GitBook: [#26] No subject
  • 07382c8 GitBook: [#27] No subject
  • 26cddf4 GitBook: [#28] getting-started
  • 931c9de GitBook: [#29] No subject
  • dae3ffc GitBook: [#2] title modification
  • 97e2c49 GitBook: [#4] Substreams doc revamp
  • ef4cf64 GitBook: [#5] No subject
  • 408ed30 Improved hex encoder and added decoding from hex library
  • 9aec010 Merge branch 'develop' into wasm-refactor
  • 59f299e Merge branch 'develop' of github.com:streamingfast/substreams into develop
  • 4ddec88 Merge branch 'develop' of github.com:streamingfast/substreams into develop
  • c78ffc2 Merge branch 'develop' of github.com:streamingfast/substreams into feature/progress_wip
  • eac4b1b Merge branch 'develop' of github.com:streamingfast/substreams into feature/progress_wip
  • d9d39d9 Merge branch 'develop' of github.com:streamingfast/substreams into feature/progress_wip
  • e9da6a8 Merge branch 'develop' of github.com:streamingfast/substreams into feature/progress_wip
  • b5730c9 Merge branch 'feature/parallel_partial_sub_req' of github.com:streamingfast/substreams into feature/parallel_partial_sub_req
  • 5766c32 Merge branch 'feature/publish' into develop
  • deabc1f Merge branch 'manifest-reader' into develop
  • e7412d3 Merge branch 'rust-macros' into develop
  • 77fe753 Merge branch 'wasm-refactor' into develop
  • c174526 Merge pull request #13 from streamingfast/feature/parallel_partial_sub_req
  • 3682925 Merge pull request #14 from streamingfast/feature/broken_cache
  • af7288b One response func instead of many.
  • f023ca0 Preparing for release of CLI + crate
  • 654b58e Refactor substream macros
  • c885e3e Refactored instrensiv names
  • c3fcf05 Remove --partial-mode and --no-return-handler .. we can put --partial-mode back if necessary to ease debugging of backend services. We could use SUBSTREAMS_PARTIAL_MODE=true, as to not bloat the end user's experience.
  • 89c8a0f Remove ETH reference
  • cd4a4f6 Remove a Grrr
  • 67dafd8 Remove the Ethereum stuff
  • f3f52dc Removed blank lines in errors
  • dd2dd59 Renamed loadDetla to loadDeltas, replaced some panic with errors, updated some zlog statements
  • 450855f Review the Progress signaling.
  • c0e477d Some renames
  • 4ed5ae8 Take out Partial from the proto.
  • 45c5647 Take out non working install instructions.
  • 836ef82 Try again
  • 1b2672c Try to pad things. Fix the proto import priority.
  • ff67ce5 Trying to fix cargo publish command
  • 70d042e Tweaked docs.
  • cd1bd78 Updated CHANGELOG
  • 33abcc2 Updated CHANGELOG, with substream macros
  • 06f8f35 Updated README.md
  • 9309b53 Updated Summary
  • eb8171a Updated explicit traits
  • b74178a Updated latest rendered substreams version
  • ca3c4b0 Updated packages metadata
  • 3883630 WIP
  • beb843e WIP: cleaned up args
  • 8ffb631 Working
  • 3b3e008 Wrap up protogen. Fix tests.
  • 76b46f4 added an early exit when no catchup request are needed
  • 893e016 added couple of MarshalLogObject func
  • 5b8cf2d added lock to squasher
  • 1cecae2 added logs to debug context stuff
  • 381202c added missing return
  • 8edee92 added missing return worker to pool
  • 1da7057 added retry around dstore access
  • 6c92140 added retry on sub request
  • 3c4fd6a added some log
  • 2261fa9 added unlock to squasher ;-)
  • 022b2b7 adding config for scheduler's subrequests and block range size
  • 228b828 adding information on the progress bar
  • fe447ba adding log for request created and adding nil check on if the partial file doesn't exist
  • 7b47ce6 adding missing pointer for squasher in scheduler struct
  • 47e11a7 adding pipeline and scheduler wip
  • e3ff44c adding progress print for catch up of parallel requests
  • 54b8c54 adding return for scheduler workers
  • 70e1c39 better logging
  • 6345d61 better logging and grpc stream handling
  • a1e1517 better logging and remove sched config
  • 9c1e030 broken wip
  • 9629eab builder, etc: replace builder start block and end block with block range object
  • ed57b8a change block range split algo
  • 77001af clean up save stores snapshots
  • e654b3c cleaned up fix macros
  • 3eb410e client: fix package name overwrite
  • 339fed9 delete partial file after squash
  • 2767c43 do not reuse grpc client create a new for each sub request
  • 30b4096 fix "substreams-partial-mode" request header better logging fix early exit of workers
  • 1387ff2 fix NO_RETURN_HANDLER
  • 996c163 fix all merge tests
  • d5768da fix delta output
  • 1076c00 fix error message for max subrequest range size
  • cdbbb22 fix init builder block range
  • 1b5bf42 fix init builder block range
  • 09883e1 fix initialize store so only module outputs one can be initialized has partial fix builder initialization with deltas
  • 62790e9 fix load deltas when cache file is empty
  • 24869bb fix load state info
  • 420c037 fix logs
  • df92975 fix output caching
  • b76b0de fix partial file naming
  • 327d094 fix range split
  • 2d4c509 fix reader
  • c01e1bd fix request pop order from list
  • 78daa28 fix retry code scope
  • cd64eeb fix scheduler error handling
  • 4682ce6 fix some retry code
  • 83ecdc6 fix squasher not finding the right file
  • 3393f9b fix squashing
  • 41ee189 fix summary layout
  • 8270603 fix test
  • 54f2fa5 fix test
  • 89f8a2a fix test and remove Squash func from builder
  • e8898d5 fix worker threading issue
  • 63fa967 fixing mermaid
  • b28b287 flip builder merger func
  • a8778d5 handle grpc client getter error
  • a11432c merge develop
  • d8adcb3 merge values into struct
  • 587ce1b more logs
  • 783b45e move scheduler and squasher to orchestrator package
  • 082aa25 move some things around
  • 5f49840 moving stuff around and creating own concept for progress, instead of having 2 different files doing similarly the same thing
  • d6865d7 new global worker pool
  • 8252c5b orchestrator refactoring
  • fd9c3de orchestrator: add basic first test for squasher
  • 8fdc673 orchestrator: add priority queue to pool to re-order requests in the wait buffer
  • 057389c orchestrator: better context handling
  • 1915fa7 orchestrator: change lock to rw mutex
  • 6ebca76 orchestrator: first draft of ordered strategy
  • 6793cdc orchestrator: fix pool tests
  • bd2c674 orchestrator: fix race conditions in pool
  • 8c55852 orchestrator: fixed close of closed channel on waiter item, switch pipeline to ordered strategy
  • f237c49 orchestrator: more testing
  • 7abd562 orchestrator: more tests for block waiter
  • 28abc3a orchestrator: pass map of already processed blocks to Waiter for early exit condition
  • 791efcd orchestrator: priority queue now uses csp instead of polling, with some additional renamings and cleanup
  • 193ba09 orchestrator: renames
  • e65e8fa orchestrator: request stream handles context errors
  • d2b4d1a orchestrator: smarter locking fo...
Read more
Assets 7
Loading

v0.0.6-beta

17 May 05:34
@abourget abourget
v0.0.6-beta
1ef72a8
Compare
Choose a tag to compare
v0.0.6-beta

Changelog

  • b4df491 Add strings to beginning of signature.
  • 5112914 Added Semver check for package version.
  • d986c5d Added --stop-block count support
  • f3b0293 Added log::debug! and log::info! and improved log::println
  • 20a3646 Added substreams::Hex wrapper type to more easily deal with printing and encoding bytes to hexadecimal string
  • 8797355 Added reading of SF_API_TOKEN also if notjing else finding
  • 627ab02 Added validation for names. Merge for proto, packages, rename modules and implement package imports.
  • c4d2d41 Fix delta start block
  • 8cb54cf Fixed Docker image and improved its building speed
  • a799cfd Fixed Dockerfile
  • 2502a0b Fixed Slack notification
  • fb17a73 Fixup required/optional flags.
  • 86af019 Handle error, and better error reporting for packing.
  • 16812f9 Improved around logs handling and reporting
  • beb3616 Improved image around rust installation
  • 552ed05 Improved readability of substreams run
  • b6f3aaf Merge branch 'feature/new_store_sync' into develop
  • 1ef72a8 Merge branch 'feature/packaging' into develop
  • b8c3b6e Package vs Manifest vs Modules.
  • 9169b1b Remove RPC for Eth, not the concern of this repo. Make import paths absolute.
  • 3954a1b Remove left over debug log
  • b51ed28 Removed wasm-pack from substreams
  • ea114bd Removed wasm-pack mentions
  • 2b7b28c Support http and https for package dependencies.
  • b120ea8 The -c (or --compact-output) can be used to print JSON as a single compact line
  • 3148278 The whole thing builds now!
  • 87030db Trap context canceled error when WASMExtension are executed
  • 568e2d4 Try to dedupe some protobuf files based on Sum256 of its serialized payload. Ideally, we'd use the latest, so some reverse sort of some kind, so newer stuff takes precedence on older stuff.
  • c3273b7 Updated Dockerfile to have a better image
  • b848c7a Works! .spkg acts as a substreams.yaml :)
  • 4f3a55e add 'tools package' commands: get,list,info,create
  • 43e788d add protobuf descriptor.proto to statik fileDescriptorSet
  • 55ec114 added a grrr log
  • 621dac0 eee wut, fix typos
  • f9650b0 fix caching issues
  • 39008a5 fix init builder block range
  • ad8d17d fix output cache by encoding it into json
  • 3413675 fix start-block missing in command line to pick up from the manifest and streaming till live if no stop-block is set
  • 9e78226 fix temp removed code
  • cd29802 pipeline: blocksFunc split into chunks of 10k block ranges
  • 7d80854 remove 'cli', put all in cmd
  • 91b8549 remove mode name from all stored file remove baseCacheStore params remove all hardcode 10_000 value
  • bb43944 remove my packaging tools
  • 0b30f78 removing mermaid print on run of substreams
  • 4ce57ac replacing wasmer-go from wasmerio with streamingfast wasmer-go
  • e2f31d5 service: worker pool moved to struct-level
  • 0804fe6 start block validation
  • 6372493 sub requested for store synchronization load state at any block from kv and deltas data folder re-org
  • 2a83fe5 tools: fix store checker
  • 449cec2 wip for bundling proto files
Assets 7
Loading

v0.0.5-beta3

09 May 19:35
@maoueh maoueh
v0.0.5-beta3
61cc596
Compare
Choose a tag to compare
v0.0.5-beta3

Changelog

  • 0d517aa Added Docker build, GitHub Actions and improved Go released with brew
  • 61cc596 Fixed wrong detection of errors
  • 7862c07 Merge branch 'develop' of github.com:streamingfast/substreams into develop
  • 0f09be3 Merge branch 'develop' of github.com:streamingfast/substreams into develop
  • b2cbf5e Merge branch 'develop' of github.com:streamingfast/substreams into develop
  • 60d2d07 Merge pull request #11 from streamingfast/no-more-store
  • 0920aff Updated CHANGELOG
  • 190f31c added last todo for store deltas output caching
  • 95f1d59 added store delta to module output caching
  • 767b111 adding String method to Store and Mapper executor, adding optimize executors and adding tests for optimize executors
  • ff2f5b0 applying deltas on store deltas for caching
  • 5434ac1 bringing back native code
  • ecc7a84 bringing back oneof native code
  • 9d2239a cleaup tests
  • 5e53cc8 convert many if statement to switch
  • 2160ab7 extracted code to func
  • 95e3d4e fix all tests for @colin
  • f1cc0ad fix new cache handling
  • 5b7ff61 fix save builder block range handling
  • a1e517a move code around
  • a862cae moving stuff around
  • ef193e6 pipeline: added ctx to sync blocks function
  • 687dd98 pipeline: remove possibility of dupilicate requests in synchronize blocks
  • 2984cf8 pipeline: synchronize blocks
  • 79bb8c4 refactor executor
  • 67e3d8d remove executor optimization
  • a63c91e remove wait from pipeline
  • 7d0c4f9 small cache registration optimization
  • 00c425b split executor and pipeline
  • 4775db9 split store on build if it is a delta or not
  • 95cb811 state: refactor deleting of special keys
  • 6d3edf3 super wip
  • b7f5cff tools: fix bug in checker
  • 17b5448 tools: remove merge cmd and added check cmd
  • 1637eed wip
  • 3935a0f wip for output cache
  • 8a5e739 working output caches

Contributors

colin
Assets 7
Loading

v0.0.5-beta2

04 May 16:10
@abourget abourget
v0.0.5-beta2
0770804
Compare
Choose a tag to compare
v0.0.5-beta2

Changelog

Client

  • Improved defaults for --proto-path and --proto, using globs.
  • WASM file paths in substreams.yaml manifests now resolve relative to the location of the yaml file.
  • Added substreams manifest package to create .pb packages to simplify querying using other languages. See the python example.
  • Added substreams manifest graph to show the Mermaid graph alone.
  • Improved mermaid graph layout.
  • Removed native Go code support for now.

Server

  • Always writes store snapshots, each 10,000 blocks.
  • A few tools to manage partial snapshots under substreams tools

Commits

  • 05012e1 Added substreams manifest package command.
  • 0770804 Added good defaults for proto rendering experience.
  • 5adb936 Always add .pb and that's it.
  • e9677eb Improve mermaid graph. Added substreams manifest graph
  • cadb0e4 Merge branch 'develop' of github.com:streamingfast/substreams into develop
  • 26175e8 Merge branch 'develop' of github.com:streamingfast/substreams into develop
  • 5712290 Merge branch 'develop' of github.com:streamingfast/substreams into pipeline-rework
  • de77bd4 No need to install wasmer
  • d829192 Remove eth check, open up possibility for Solana.
  • b09403f Rework of pipeline, module executor, in prep for caching
  • 3222703 Update README.md
  • aa8436d Updated .gitignore
  • 35e1f4a Updated README.md
  • fb25f4d Updated README.md
  • 36eed17 added module name to state store folders fix some tests
  • dd1ff1e adding wip module ouput cache
  • 911b189 better error message when substreams terminate with error remove println
  • 215f243 cmd: added first version of squasher
  • 09b5bd4 database changes have been moved, adding validation for input sources
  • a606476 find proto files recursively and change manifest module code path to absolute
  • df77c1f fix panic handling
  • b5d922e lower level of some logs
  • 1652f15 make pipeline compile and test to runn
  • 03d1b6c more log to debug
  • 0463c3a only accepted value for input.source is sf.ethereum.type.v1.Block, in other cases, we return an error
  • 77d5015 pipeline: allow invalid state in partial mode
  • 99f3cda pipeline: fix snapshot interval
  • ca3f41d remove database changes stuff from substreams
  • 8ff8f25 remove database from proto build
  • 6f767ff remove database proto from substreams
  • 4041ba7 remove println again
  • 5b63a0a remove un-used code
  • 484cd9c removing a comment
  • 91e2048 removing test
  • c95415a renaming of parameter
  • a31eab5 setting remote cmd to root as we only have remote command available
  • 8923491 spec out module output cache
  • 49ee4fc squasher cmd, state.ParseFile refactor
  • c2625d0 squasher: fix args
  • 0a083d6 state: fix bug where BuilderFromFile would create an additional substore
  • 2bc44ae store sync todos
  • d715952 substreams run commend
  • a0ec093 todo management
  • 74bae47 tools: cleanup in parallel
  • 4c7931a uncommented panic recovery code in pineline
  • 6d4fe90 update readme
  • 067cb55 update readme
Assets 5
Loading

v0.0.5-beta

27 Apr 22:17
@abourget abourget
v0.0.5-beta
5964ae4
Compare
Choose a tag to compare
v0.0.5-beta

[v0.0.5-beta]

First chain-agnostic release. THIS IS BETA SOFTWARE. USE AT YOUR OWN RISK.
WE PROVIDE NO BACKWARDS COMPATIBILITY GUARANTEES FOR THIS RELEASE.

See https://github.com/streamingfast/substreams for usage docs.

  • Removed local command. See README.md for instructions on how to run locally now. Build sfeth from source for now.
  • Changed the remote command to run.
  • Changed run command's --substreams-api-key-envvar flag to ``--substreams-api-token-envvar, and its default value is changed from SUBSTREAMS_API_KEY` to `SUBSTREAMS_API_TOKEN`. See README.md to learn how to obtain such tokens.

Previous Changelog

  • 1a22a92 Added CHANGELOG.md
  • 378d759 Added LICENSE
  • 6f6e46a Added README.md
  • c6e537a Added notes.
  • b2e180d Added some docs. Renamed remote to run.
  • 84b6852 Adding Cargo lock to gitignore
  • 961d855 Adding decimals
  • 86b760d Adding proto wip
  • 408277e Adding sum float 64
  • 60b5f59 Adding tests and adding sum_float64
  • 6b9e5ab Apply table changes to an entity
  • 54b7a68 Attempt at integrating protobuf to json rendering upon output.
  • bb77f30 Better flags setting.
  • 2b128c7 Better manifest
  • b47ad60 Better print
  • 91e1cd4 Bump prost to 0.10.1
  • b246896 Bumped tidwall/gjson to fix sec issue.
  • 39ac0f2 Change hard code value of block
  • 856060c Changelog update. First draft of goreleaser.
  • c396a0f Changing variable names and adding logging
  • 2b2abce Closer closer to something functional. Missing the decoding and displaying the rest is pretty much there.
  • ae1b219 Compiles. No more deps on sparkle now.
  • 05e7960 Decode faster, support unsupported messages.
  • 79583a7 Do the decoding at the right place, and load the proto-path.
  • 1fbdfac Don't allow running local when not built through sf-ethereum. Only remote.
  • 3f31be3 Don't conditionally add the getters
  • f29022a Don't fail when proto not available. Show connecting/connected.
  • f19fde7 Draft of substreams-native gRPC endpoint.
  • bfc1c94 First draft at importing from PCS and splitting in a proper lib
  • e753428 First draft at moving proto defs around, to have a first-class Substreams endpoint.
  • a792613 First draft of proto for manifests
  • cb97e74 First workable draft of multi-module remote sync'ing.
  • 655782f Fix imports
  • ac1c28d Fix leak between blocks for input sources.
  • d47606c Fix tests. SetBuidlder -> SetOutputStore. We need the data to align for bigint
  • 853deea Fixed graph building, and renamed a few stream things into modules.
  • 1610439 Fixup RPC calls
  • 310e60a Fixup printing of those blocks.
  • 38b71c8 Fixup rust tests.
  • 4526a22 Fixup the registry, and finding the right func.
  • c3b725e Fixups
  • 2b242c9 Flush stores at stop block too. Fix compilation issues. Fix logs. Options here and there. Cosmetics in builder.go
  • 2f33f70 Go generated code
  • 4fd59bb Go mod tidy
  • 7c144c3 Gracefully terminate. Better print output. Optimize anypb.Any creation.
  • 8f73094 Implementing and adding tests for set max int64, bigint, flaot64 and bigfloat
  • 0a5fb9d Implementing sum_big_float and adding tests
  • c18fa29 Improved docs. Renamed remote to run.
  • fc6b067 Initialize the wasm modules within the Firehose thread.
  • 03cef88 LocalConfig and RemoteConfig now extends Config
  • 785889e Mega hack!
  • 19ac176 Merge
  • 64a2135 Merge branch 'develop' into feature/proto_manifest
  • 025a55f Merge branch 'develop' of github.com:streamingfast/substreams into develop
  • aa8db1e Merge branch 'develop' of github.com:streamingfast/substreams into develop
  • fd2ebbb Merge branch 'develop' of github.com:streamingfast/substreams into develop
  • 947015b Merge branch 'develop' of github.com:streamingfast/substreams into develop
  • 835a1cc Merge branch 'develop' of github.com:streamingfast/substreams into develop
  • 5cc72f2 Merge branch 'develop' of github.com:streamingfast/substreams into develop
  • fb55150 Merge branch 'develop' of github.com:streamingfast/substreams into develop
  • 0165628 Merge branch 'develop' of github.com:streamingfast/substreams into develop
  • 09b405c Merge branch 'develop' of github.com:streamingfast/substreams into feature/refactor-state-imports
  • 11efad5 Merge pull request #1 from streamingfast/master
  • 4925463 Merge pull request #6 from streamingfast/feature/waiter_refactoring
  • 67b9fab Merge pull request #7 from streamingfast/refactor-instance
  • 44eac00 Merge pull request #8 from streamingfast/graphnode_returnhandler
  • 31a8cb4 ModuleSignature WIP
  • 8fa5ae4 Move return handler out of there. Will be able to be re-used by the front-end to do the same sort of decoding there.
  • 5c747ea Much better on-the-fly decoding
  • a377ef7 Par dessus
  • 19d3238 Pass in deltas (serialized as JSON for now)
  • 48aecaa Proper fallback on ENV variables.
  • 81bd08b REVERT! now use block payload instead of unmarshal and re marshall it
  • 2951140 Recovered cache baby, somehow, baby tukka.
  • 3f00f60 Removing time handling log
  • 2a47bb9 Rename a few proto stuff
  • f8586cc Rename all those freaking fields. Fix mermaid output. Make that state deltas proto buf serialized instead of json.
  • a037fe2 Renames
  • 6a9136f Restore println's output. Cleaner output.
  • 43b99b4 Revert "Initialize the wasm modules within the Firehose thread."
  • db70091 Revert "remove all panic"
  • c6d7524 Revert "test commit"
  • ba27538 Safeguard against WASM data leaking between blocks
  • b36d168 Shared engine
  • 521a8e8 Sign native code too.
  • d85c2f5 Small refactor. Implement SetIfNotExists. Write as strings in storage for ints and floats. Runtime error when stores do not match imported function calls.
  • a96619f Start block int64. Better print.
  • 10cbc89 Try to pinpoint the odd behavior
  • 5287ec4 UPdate
  • 4340f80 Update README.md
  • 7674fe3 Update the rust package.
  • 5964ae4 Updated README.md
  • 0528605 Updated loggers for graph-node and rpc packages
  • ce9bbdc Updated proto
  • 3c37fbb Use protoreflect for dynamic proto message handling.
  • 0a7a75c Use startBlock from the manifest now.
  • 013ab56 Using require isEqual with testing suite
  • fe9c283 WIP
  • 312c326 WIP exchange dataloader
  • e40a039 WIP for bigfloat
  • 63a8106 WIP transform, print
  • 104e7ea WIP: have decode/print.go print with the new output format.
  • 4f77f84 Wire in the cursor, step still missing for the remote transforms-based version.
  • 54bff23 add a more verbose format and resetting time to now in if statement
  • c7fb4cd add context to progress tracker
  • db8b90b add default case to input mode and set keys to lowercase
  • 11e0c44 add eth_call to rpc to wasm machine, using proto
  • 54be505 add gen for substreams
  • 41d63da add log for rpc call time taken
  • fa6ba49 add logger module
  • 570c145 add memory leak test
  • 48076b9 add missing file
  • 0554f76 add prost-build in build dependencies
  • 2a21aff add proto generate files and build scripts
  • a4eb426 add protobuf
  • 2b4fb7e add remote runtime, update cli, cleanup runtime vars
  • a4a0405 add set_if_not_exist as extern function
  • e42dc23 add tests for graph methods from manifest file and fix some tests
  • 348cc22 add time blocks pps
  • b5f493e add time to process x number of blocks per second
  • df7c321 add timer for rpc
  • 9f00432 added a small tool to print some manifest info
  • 7077a9b added graph-node definition parsing
  • af02435 added initial block to module and start block is now a program argument
  • bd02d03 added no-return-handler param
  • ecf7264 added param rpc-cache-store-url
  • 50ac81c added secondary-rpc-endpoints params
  • d2a95ef added set_min support!
  • 9fe1dd1 added stack traces log when recovery from panic
  • d08b4a8 added stupid fmt.println
  • 7f411f8 added substreams Output "wrapper" around return value
  • d25dd6b added support to multi rpc endpoints
  • 6e5887b added test for graph
  • b53083a added unset to table_change operation
  • e5608b7 adding logging for types v1 package
  • 5ca3e6a adding more logs and adding statesSaveInterval
  • f1cb290 adding proto url for local command
  • f6f5e93 adding protobufBlockType missing in the LocalConfig constructor
  • 97fbd4b adding test orphan node in the test file
  • f295730 all broken wip
  • 7791c31 better logging
  • c6f764c better start block handling
  • af01a46 builder io: walk partials func
  • 6bf377d builder: compile-time check for interfaces
  • 05bbd05 builder: fix call to WriteState
  • 40555a6 builder: fix tests
  • 1154df0 builder: fix tests
  • 92d9dba builder: if partial file starts at module start block, save as full snapshot
  • 25521a5 builder: implement merge logic for all types (int64, bigInt, float64, big.Float)
  • 982de94 builder: max tests
  • ef1eb93 builder: min tests
  • 352a891 builder: partial prep work wip
  • b60b87b builder: remove bundler stuff
  • 96e64bd builder: remove patate log
  • df30de1 builder: sum tests
  • 2a44a23 builder: unify receiver names
  • 84f4eac bump bstream, add ground work for substreams as transform
  • a448af6 bump bstream, firehose with new transform patterns
  • c77f822 bump dstore
  • 515e830 bump dstore
  • ae83b68 bump eth-go
  • 2889dd3 bump eth-go, now uses context
  • c972832 bump substreams to pick up bugfix for gas uint64 deterministic error
  • 8db7cc9 change module initialStartBlock to startBlock
  • 9a3f3fe check requested start block before wait loop
  • dcfa9c9 cherry pick changes from substreams_transform branch and implement onReturn method
  • cbc2063 clear todo
  • 94c1c35 cli: add command to get ordered list of stores to run in parallel
  • a5f31be cli: add stop-block flag
  • af5e1f8 create a new store for each instance
  • 315403d database changes test update
  • 0d3fd72 database squash test cleanup
  • e1e8f43 debug/info -> println
  • de8c53c deps: use streamingfast fork of graph library
  • 5cde373 ensure substreams transform runner catches EOF as non-error
  • ce5cb6e ensure transform creates an (empty) cache to prevent nil pointer
  • 9013de9 entity package copied over from sparkle
  • 3132c4a extern state methods are no longer taking owner ship o...
Read more
Assets 5
Loading