v1.0.0

Highlights

• Added command substreams gui, providing a terminal-based GUI to inspect the streamed data. Also adds --replay support, to save a stream to replay.log and load it back in the UI later. You can use it as you would substreams run. Feedback welcome.
• Modified command substreams protogen, defaulting to generating the mod.rs file alongside the rust bindings. Also added --generate-mod-rs flag to toggle mod.rs generation.
• Added support for module parameterization. Defined in the manifest as:

module:
  name: my_module
  inputs:
    params: string
  ...
  
params:
  my_module: "0x123123"
  "imported:module": override value from imported module

and on the command-line as:

• substreams run -p module=value -p "module2=other value" ...

Servers need to be updated for packages to be able to be consumed this way.

This change keeps backwards compatibility. Old Substreams Packages will still work the same, with no changes to module hashes.

Added

• Added support for {version} template in --output-file flag value on substreams pack.
• Added fuel limit to wasm execution as a server-side option, preventing wasm process from running forever.
• Added 'Network' and 'Sink{Type, Module, Config}' fields in the manifest and protobuf definition for future bundling of substreams sink definitions within a substreams package.

v0.2.0

Highlights

• Improved execution speed and module loading speed by bumping to WASM Time to version 4.0.

• Improved developer experience on the CLI by making the <manifest> argument optional.

  The CLI when <manifest> argument is not provided will now look in the current directory for a substreams.yaml file and is going to use it if present. So if you are in your Substreams project and your file is named substreams.yaml, you can simply do substreams pack, substreams protogen, etc.

  Moreover, we added to possibility to pass a directory containing a substreams.yaml directly so substreams pack path/to/project would work as long as path/to/project contains a file named substreams.yaml.

• Fixed a bug that was preventing production mode to complete properly when using a bounded block range.

• Improved overall stability of the Substreams engine.

Operators Notes

• Breaking Config values substreams-stores-save-interval and substreams-output-cache-save-interval have been merged together into substreams-cache-save-interval in the firehose-<chain> repositories. Refer to chain specific firehose-<chain> repository for further details.

Added

• The <manifest> can point to a directory that contains a substreams.yaml file instead of having to point to the file directly.

• The <manifest> parameter is now optional in all commands requiring it.

Fixed

• Fixed production mode not completing when block range was specified

• Fixed tier1 crashing due to missing context canceled check.

• Fixed some code paths where locking could have happened due to incorrect checking of context cancellation.

• Request validation for blockchain's input type is now made only against the requested module it's transitive dependencies.

Updated

• Updated WASM Time library to 4.0.0 leading to improved execution speed.

Changed

• Remove distinction between output-save-interval and store-save-interval.

• substreams init has been moved under substreams alpha init as this is a feature included by mistake in latest release that should not have been displayed in the main list of commands.

• substreams codegen has been moved under substreams alpha codegen as this is a feature included by mistake in latest release that should not have been displayed in the main list of commands.

Commits

https://github.com/streamingfast/substreams/compare/v0.1.0..v0.2.0

v0.1.0

This release brings significant changes on how Substreams are developed, consumed and speed of execution. Note that there is no breaking changes related to your Substreams' Rust code, only breaking changes will be about how Substreams are run and available features/flags.

Here the highlights of elements of this release:

• Production vs Development Mode
• Single Output Module
• Output Module must be of type map
• InitialSnapshots is now a development mode feature only
• Enhanced Parallel Execution

Warning Operators, refer to Operators Notes section for specific instructions of deploying this new version.

Production vs development mode

We introduce an execution mode when running Substreams, either production mode or development mode. The execution mode impacts how the Substreams get executed, specifically:

• The time to first byte
• The module logs and outputs sent back to the client
• How parallel execution is applied through the requested range

The difference between the modes are:

• In development mode, the client will receive all the logs of the executed modules. In production mode, logs are not available at all.
• In development mode, module's are always re-executed from request's start block meaning now that logs will always be visible to the user. In production mode, if a module's output is found in cache, module execution is skipped completely and data is returned directly.
• In development mode, only backward parallel execution can be effective. In production mode, both backward parallel execution and forward parallel execution can be effective. See Enhanced parallel execution section for further details about parallel execution.
• In development mode, every module's output is returned back in the response but only root module is displayed by default in substreams CLI (configurable via a flag). In production mode, only root module's output is returned.
• In development mode, you may request specific store snapshot that are in the execution tree via the substreams CLI --debug-modules-initial-snapshots flag. In production mode, this feature is not available.

The execution mode is specified at that gRPC request level and is the default mode is development. The substreams CLI tool being a development tool foremost, we do not expect people to activate production mode (-p) when using it outside for maybe testing purposes.

If today's you have sink code making the gRPC request yourself and are using that for production consumption, ensure that field production_mode in your Substreams request is set to true. StreamingFast provided sink like substreams-sink-postgres, substreams-sink-files and others have already been updated to use production_mode by default.

Final note, we recommend to run the production mode against a compiled .spkg file that should ideally be released and versioned. This is to ensure stable modules' hashes and leverage cached output properly.

Single module output

We now only support 1 output module when running a Substreams, while prior this release, it was possible to have multiple ones.

• Only a single module can now be requested, previous version allowed to request N modules.
• Only map module can now be requested, previous version allowed map and store to be requested.
• InitialSnapshots is now forbidden in production mode and still allowed in development mode.
• In development mode, the server sends back output for all executed modules (by default the CLI displays only requested module's output).

Note We added output_module to the Substreams request and kept output_modules to remain backwards compatible for a while. If an output_module is specified we will honor that module. If not we will check output_modules to ensure there is only 1 output module. In a future release, we are going to remove output_modules altogether.

With the introduction of development vs production mode, we added a change in behavior to reduce frictions this changes has on debugging. Indeed, in development mode, all executed modules's output will be sent be to the user. This includes the requested output module as well as all its dependencies. The substreams CLI has been adjusted to show only the output of the requested output module by default. The new substreams CLI flag -debug-modules-output can be used to control which modules' output is actually displayed by the CLI.

Migration Path If you are currently requesting more than one module, refactor your Substreams code so that a single map module aggregates all the required information from your different dependencies in one output.

Output module must be of type map

It is now forbidden to request a store module as the output module of the Substreams request, the requested output module must now be of kind map. Different factors have motivated this change:

• Recently we have seen incorrect usage of store module. A store module was not intended to be used as a persistent long term storage, store modules were conceived as a place to aggregate data for later steps in computation. Using it as a persistent storage make the store unmanageable.
• We had always expected users to consume a map module which would return data formatted according to a final sink spec which will then permanently store the extracted data. We never envisioned store to act as long term storage.
• Forward parallel execution does not support a store as its last step.

Migration Path If you are currently using a store module as your output store. You will need to create a map module that will have as input the deltas of said store module, and return the deltas.

Examples

Let's assume a Substreams with these dependencies: [block] --> [map_pools] --> [store_pools] --> [map_transfers]

• Running substreams run substreams.yaml map_transfers will only print the outputs and logs from the map_transfers module.
• Running substreams run substreams.yaml map_transfers --debug-modules-output=map_pools,map_transfers,store_pools will print the outputs of those 3 modules.

InitialSnapshots is now a development mode feature only

Now that a store cannot be requested as the output module, the InitialSnapshots did not make sense anymore to be available. Moreover, we have seen people using it to retrieve the initial state and then continue syncing. While it's a fair use case, we always wanted people to perform the synchronization using the streaming primitive and not by using store as long term storage.

However, the InitialSnapshots is a useful tool for debugging what a store contains at a given block. So we decided to keep it in development mode only where you can request the snapshot of a store module when doing your request. In the Substreams' request/response, initial_store_snapshot_for_modules has been renamed to debug_initial_store_snapshot_for_modules, snapshot_data to debug_snapshot_data and snapshot_complete to debug_snapshot_complete.

Migration Path If you were relying on InitialSnapshots feature in production. You will need to create a map module that will have as input the deltas of said store module, and then synchronize the full state on the consuming side.

Examples

Let's assume a Substreams with these dependencies: [block] --> [map_pools] --> [store_pools] --> [map_transfers]

• Running substreams run substreams.yaml map_transfers -s 1000 -t +5 --debug-modules-initial-snapshot=store_pools will print all the entries in store_pools at block 999, then continue with outputs and logs from map_transfers in blocks 1000 to 1004.

Enhanced parallel execution

There are 2 ways parallel execution can happen either backward or forward.

Backward parallel execution consists of executing in parallel block ranges from the module's start block up to the start block of the request. If the start block of the request matches module's start block, there is no backward parallel execution to perform. Also, this is happening only for dependencies of type store which means that if you depends only on other map modules, no backward parallel execution happens.

Forward parallel execution consists of executing in parallel block ranges from the start block of the request up to last known final block (a.k.a the irreversible block) or the stop block of the request, depending on which is smaller. Forward parallel execution significantly improves the performance of the Substreams as we execute your module in advanced through the chain history in parallel. What we stream you back is the cached output of your module's execution which means essentially that we stream back to you data written in flat files. This gives a major performance boost because in almost all cases, the data will be already for you to consume.

Forward parallel execution happens only in production mode is always disabled when in development mode. Moreover, since we read back data from cache, it means that logs of your modules will never be accessible as we do not store them.

Backward parallel execution still occurs in development and production mode. The diagram below gives details about when parallel execution happen.

You can see that in production mode, parallel execution happens before the Substreams request range as well as within the requested range. While in development mode, we can see that parallel execution happens only before the Substreams request range, so between module's start block and start block of requested ra...

v0.0.21

Changelog

• Moved Rust modules to github.com/streamingfast/substreams-rs

Library

• Gained significant execution time improvement when saving and loading stores, during the squashing process by leveraging vtprotobuf

• Added XDS support for tier 2s

• Added intrinsic support for type bigdecimal, will deprecate bigfloat

• Significant improvements in code-coverage and full integration tests.

CLI

• Lowered GRPC client keep alive frequency, to prevent "Too Many Pings" disconnection issue.

• Added a fast failure when attempting to connect to an unreachable substreams endpoint.

• CLI is now able to read .spkg from gs://, s3:// and az:// URLs, the URL format must be supported by our dstore library).

• Command substreams pack is now restricted to local manifest file.

• Added command substreams tools module to introspect a store state in storage.

• Made changes to allow for substreams CLI to run on Windows OS (thanks @robinbernon).

• Added flag --output-file <template> to substreams pack command to control where the .skpg is written, {manifestDir} and {spkgDefaultName} can be used in the template value where {manifestDir} resolves to manifest's directory and {spkgDefaultName} is the pre-computed default name in the form <name>-<version> where <name> is the manifest's "package.name" value (_ values in the name are replaced by -) and <version> is package.version value.

• Fixed relative path not resolved correctly against manifest's location in protobuf.files list.

• Fixed relative path not resolved correctly against manifest's location in binaries list.

• substreams protogen <package> --output-path <path> flag is now relative to <package> if <package> is a local manifest file ending with .yaml.

• Endpoint's port is now validated otherwise when unspecified, it creates an infinite 'Connecting...' message that will never resolves.

Commits

• 918c8d3 💄
• 5558de4 💄
• 232454c Add Contributing and License
• 3ab5a92 Add directory for images and header image for docs
• 66fe9f4 Add header image
• d2f5b0c Add prelude to lib and fix to type convertion
• 3b8d3a5 Added Binary relative path support in manifest
• 52bdf04 Added benchmark to see how long it takes to serialize N keys to storage
• 9d2c891 Added benchmark to see how long it takes to serialize N keys to storage
• bfd9609 Added block stats trigger
• ea52c29 Added cargo lock
• dd0029b Added custom proto marshalling
• 6281caa Added debugging log
• 1b4992d Added logging
• e1bdd30 Added logging of requested_start_cursor
• ea0c7d8 Added more request stats
• de5c8cf Added possibility to specify --output-file <template> (short of -o) to substreams pack
• 5da0e0d Added request stats
• 9c4b007 Added request stats store squasher
• 0c176be Added squasher store tests
• 9280af8 Added tests
• cfe4b34 Added two more relevant changelog entries
• c976fb6 Added v0.0.21 release notes
• 488f1fa Added vtproto support
• 213a4ea Adding minimal changes required to get commands working properly on windows OS
• eac3f01 Adding optional flag to specify output directory for pack output.
• a027a1e Adding to changelog
• 6384403 Another grpc client fix
• 18c3c6a Bumped dstore to fix the latency in SubStore initializations.
• 049417a Bumped dgrpc again
• 316e592 CLI is now able to read .spkg from gs://, s3:// and az:// URLs, the URL format must be supported by our dstore library).
• b437533 Call promote waiting jobs after preparing the jobs!
• 4665c13 Clarify the nextStoreSaveBoundary condition, to be larger or equal than the current blockNum.
• 58da6ff Clean up boundary test
• 60b7333 Clean up logging
• fa92021 Cleaned up Store and Partial store
• 94eaade Cleaned up benchmarking marshaller test
• b23cb12 Cleaned up logging and req statas
• 4652618 Cleaned up proto
• bf1f6f8 Cleaned up service and test
• be92abc Cosmetics.. Reset change place, just a dance for it to be ready next.
• f952595 Decouple the Worker from the orchestrator (doesn't know about Job) anymore.
• 9a186d8 Edits to text
• 8dfa922 Enabled release draft mode and disabled linux & darwin amd64 build
• 86f8ccc Endpoint's port is now validated on substreams run
• 0131e43 Externs generation
• bef53f3 Few renames
• fb83fd5 Finish up the job scheduler. Tests now need to be written.
• 5fb9d8c First draft of BaseStore from Config.
• 55308e6 First draft of the scheduler testing. Better abstraction for the JobRunner.
• a8a3cf9 First pass at supporting cursor correctly
• 724e593 First stub of SessionInit message, to provide the trace_id.
• a6b5535 Fix assert_all test
• cbb2669 Fix error handling
• 8784513 Fix integration tests
• a8d9d71 Fix merge test
• ff63ef3 Fix miss refactor
• 1858418 Fix multiple store issues
• 491f99f Fix output saving on cache
• e7b6ba1 Fix partial merging and squasher
• 635e7a2 Fix pipeline test
• 30fad36 Fix rebase issue
• 0c4b797 Fix test
• c1ba0c6 Fix tests
• 479129c Fix tests
• 05a1cec Fix/sol squasher incomplete (#50)
• 55ec82c Fixed benchmark
• b13d0a7 Fixed cachev1 when running tests with -count N
• e36a3f8 Fixed relative path not resolved correctly against manifest's location in binaries list
• a322d74 Fixed relative path not resolved correctly against manifest's location in protobuf.files list
• e0329e0 Fixed tests in scheduler.
• 01760e8 Fixed usage of TargetBlock in tui view
• 099390d Fixup NextJob
• c1bac36 Fixup the scheduler's runOne.
• a4c9aa4 HTML test
• 6b32527 Have the backprocess job create the storeMap, and return it untouched. He's responsible for creating the whole graph.
• 852d904 Implemented grpc client factory
• b77de8b Implemented test for setupSubrequestStores Updated MockStore to support easier SetFile.
• 73f8dbd It a live!
• c77507d Last iteration of out-dir flag description
• 79bd235 Made more explicitely StartBlockNum vs EffectiveStartBlockNum and made a few comments
• 78f8c5f Made output path relative when generating proto for a local manifest file
• 0eccb52 Major Refactor
• 4a68ddf Making changes to make it super clear on how output-dir works.
• 16e9ec1 Merge branch 'develop' into feature/backprocessor-refactor
• 27309c7 Merge branch 'develop' into feature/backprocessor-refactor
• 45585b2 Merge branch 'develop' into feature/backprocessor-refactor
• badaaba Merge branch 'develop' into feature/backprocessor-refactor
• b83f8f3 Merge branch 'develop' into feature/codegen
• 9017ff1 Merge branch 'develop' into feature/integration-test
• 2587686 Merge branch 'develop' into feature/xds
• 8a9b81c Merge branch 'develop' of https://github.com/streamingfast/substreams into develop
• 5adfd83 Merge branch 'feature/backprocessor-refactor' into develop
• 3a2a98a Merge branch 'feature/binary-marshaller' into develop
• 5cc1b5b Merge branch 'feature/codegen' into develop
• 513eb12 Merge branch 'feature/some-rework' into develop
• 1435199 Merge branch 'feature/xds' into develop
• 6928079 Merge branch 'fix/merge-big-int' into develop
• 4118127 Merge branch 'refactor' into develop
• c061bb0 Merge pull request #48 from tilacog/patch-1
• 8f67cbf Merge pull request #54 from streamingfast/feature/bigdecimal
• 83c6bb6 Merge remote-tracking branch 'origin/develop' into develop
• fab246b Merge remote-tracking branch 'origin/develop' into develop
• 41ef384 Merge remote-tracking branch 'origin/develop' into feature/backprocessor-refactor
• b243da2 Minor formatting fixes
• 8bad9ed ModuleHashes abstraction, more speedy with cache. We can cut 9 seconds of initialization time!!!
• 192a7fa Mucho more tests better.
• 9e0251a Notes and renames and what not 💄
• 0729cde Now VTProto is the marshaller
• 7975970 OpenTelemetry
• a20b2af OpenTelemetry
• d8ded8e OutputCache fused with OutputCacheState.
• e96ad07 Pass in runtimeConfig here and there.
• 89f3e94 Pipeline needs a ForkHandler to not crash tests. Quick renames
• f70bdc8 Preparing for next unreleased version
• a31b124 Put back NewTestModules in testing because it's used in foreign packages
• b9631fe Put integration tests behind an env var to run, fixed path to be relative to substreams.yaml (now that it works for everything)
• 31bade9 Quite another refactor here. Fully broken!
• 8f46674 Registered Metrics & Updated Naming Conventions
• 0343fff Remove links to documentation pages
• 32adf53 Remove testing HTML
• 4d230ba Remove unused object.
• 0c86415 Removed pipeline.RequestContext stack recording of span(s)
• 5097429 Removed specialized errors.GRPCError and uses standard status.Error everywhere
• 9fe2c2e Removing unecessary check
• 7197df0 Replaced request context with context abstractions
• 6fc3287 Revert "removed ExecSpan"
• 0386e6b Scheduler rework, absorb the JobsPlanner, rework the WorkPlan. Make it the thing that prioritizes. Have an abstraction to prioritize it.
• 9f1c8ab Simplify
• aeb6b3a Small nitpick on CHANGELOG
• 2e1e9af Small refactor of SetStatus for traces.
• fbecf39 Temp HTML changes in docs
• 7a9f93a Test the init messages from the work.Plan. Fixup locking for the mutators of the Plan.
• 51eaf59 The error should be handled
• 2cf40c5 Tweak some display and logs.
• 7ef59ab Update header image for README
• 8ef14f0 Update header image for docs
• 365de5b Update setting-up-handlers.md
• 1541fe3 Use the right default values man.
• ae2e488 Using isAbs for deducing a relative path
• cb83e84 Using spkg in integration tests
• ff2c5be WIP
• 310cf7d WIP backprocessor refactor
• 36451f0 Wrote two more tests for runOne in scheduler. Fixup the jobrunnerpool broken test.
• ef64f75 add integration tests by default in bin/test.sh
• 98ccfaa add missing files
• a8a0ecf add the module hash to error ...

v0.0.20

Changelog

CLI

• Fixed error when importing http/https .spkg files in imports section.

Commits

• d2a5fef Added prost bumping to release notes
• 775c6c7 Fixed error when importing http/https .spkg files in imports section.
• 36796e6 Fixed typo in Cargo.toml files
• 5a8ea5f Merge branch 'develop' into feature/forkhandler
• 1cf4d98 Merge branch 'develop' into feature/forkhandler
• 367f6b3 Merge branch 'develop' into feature/forkhandler
• 2ef77b3 Merge branch 'feature/forkhandler' of github.com:streamingfast/substreams into feature/forkhandler
• a30e3f7 Merge pull request #47 from streamingfast/feature/forkhandler
• ae3f287 Preparing next unreleased version
• dae1548 Preparing release of 0.0.20
• c73d93a adding check for key len of 0 when setting a value
• ed49e0e adding constructor
• 2bfa35d adding delete method for cache and adding tests for delete function
• 9619961 adding step undo and irr and calling ForkHandler
• 2b228f3 adding test shell script file to run go tests
• 31ecb5a adding tests for forkHanlder
• 42de058 adding tests for reverse application of deltas for the forkHandler
• 89413be adding tests for the forkHandler when we are rollbacking multiple deltas and simplifying a switch case
• 1ead7ec adding wip forkHandler
• f432bb9 fix barmode
• d112ba4 fix state applyReverse, add tests
• 9953388 implementing apply delta reverse
• 6997fe4 merge develop
• 22e0804 merged develop
• 4874ca1 remove unwanted error log
• 8828993 removing the set to nil for the forkHandler.reversibleOutputs
• 5af98c0 removing types tests
• 4808918 revertoutputs, handleIrreversibility and tests for the latter
• 13c90f2 switch FinalBlocksOnly to false

v0.0.19

Changelog

New! A updatePolicy of type append has been added, it allows one to build a store that concatenates values and supports parallelism. This affects the server, the manifest format (additive only), the substreams crate and the generated code therein.

Rust API

• Store APIs methods now accept key of type AsRef<str> which means for example that both String an &str are accepted as inputs in:

  • StoreSet::set
  • StoreSet::set_many
  • StoreSet::set_if_not_exists
  • StoreSet::set_if_not_exists_many
  • StoreAddInt64::add
  • StoreAddInt64::add_many
  • StoreAddFloat64::add
  • StoreAddFloat64::add_many
  • StoreAddBigFloat::add
  • StoreAddBigFloat::add_many
  • StoreAddBigInt::add
  • StoreAddBigInt::add_many
  • StoreMaxInt64::max
  • StoreMaxFloat64::max
  • StoreMaxBigInt::max
  • StoreMaxBigFloat::max
  • StoreMinInt64::min
  • StoreMinFloat64::min
  • StoreMinBigInt::min
  • StoreMinBigFloat::min
  • StoreAppend::append
  • StoreAppend::append_bytes
  • StoreGet::get_at
  • StoreGet::get_last
  • StoreGet::get_first

• Low-level state methods now accept key of type AsRef<str> which means for example that both String an &str are accepted as inputs in:

  • state::get_at
  • state::get_last
  • state::get_first
  • state::set
  • state::set_if_not_exists
  • state::append
  • state::delete_prefix
  • state::add_bigint
  • state::add_int64
  • state::add_float64
  • state::add_bigfloat
  • state::set_min_int64
  • state::set_min_bigint
  • state::set_min_float64
  • state::set_min_bigfloat
  • state::set_max_int64
  • state::set_max_bigint
  • state::set_max_float64
  • state::set_max_bigfloat

• Bumped prost (and related dependencies) to ^0.11.0

CLI

• Environment variables are now accepted in manifest's imports list.

• Environment variables are now accepted in manifest's protobuf.importPaths list.

• Fixed relative path not resolved correctly against manifest's location in imports list.

• Changed the output modes: module-* modes are gone and become the
  format for jsonl and json. This means all printed outputs are
  wrapped to provide the module name, and other metadata.

• Added --initial-snapshots (or -i) to the run command, which
  will dump the stores specified as output modules.

• Added color for ui output mode under a tty.

• Added some request validation on both client and server (validate
  that output modules are present in the modules graph)

Service

• Added support to serve the initial snapshot

Commits

• 9360b95 Builder renamed to Store everywhere
• a21fe56 Builder renamed to Store everywhere
• 469ca52 Bumped prost to ^0.11.0
• 0431365 Fix error handling
• 5e2c29c Improvements to Manifest imports and protobuf.importPaths
• fd92720 Merge branch 'develop' into feature/wasmtime
• c05f9b2 Merge branch 'substreams-workable' into develop
• f68b9d4 Merge pull request #33 from streamingfast/feature/wasmtime
• 71d2318 Merge pull request #45 from streamingfast/feature/stack_trace
• b152c35 Merge remote-tracking branch 'origin/develop' into feature/wasmtime
• 8cda6d9 Prepare release v0.0.16
• df5a068 Preparing release
• 9952d2e Preparing release for version 0.0.17
• 69dbbd9 Remove noisy log
• 460cd0b Removed worker statistics as fmt.Println and turn it into a proper log line
• ac3d554 Replacing derr.RetryContext with for lopp to retry the job
• 88927e5 add cache entry for empty output
• 8d2051c added memory size log
• 58265e5 added memory size log (fixed)
• 3131b33 added todo
• 67c096a adding RetryableErr concept if the error is considered a RetryableErr and to retry on the other side of the scheduler.go and returning a normal error when it is considered an unRetryableErr
• 248f89a adding cacheEnabled attribute to pipeline and adding if statements to use and not use the cache in places where the output cache is loaded, get, save and update
• 5855ebe adding decode command for store and map
• a770d75 adding decoder for decoding bytes to a protobuf definition
• 8c609fb adding execution stack on executor and returning ErrorExecutor when getting an instance.Execute()
• 2bbda15 adding fix me for jobStat update method
• 2c09764 adding job planner test for dependency priority bug
• 44c8805 adding pipeline options to pipeline
• 603bb38 adding the execution stack when returning the logs if a substreams failed
• 90dce93 adding the logs to the execution stack when calling info,.bu debug! and println! in the substreams code
• d4b39ae all test passed
• a35b9b5 augmenting worker stat
• 2526acc better exec error handling
• 6ed2840 bring back job stats update
• bb33a01 bummped firehose@develop
• cba7427 bump bstream/firehose, don,t pre-decode block
• 08d1f84 bump crate version to 0.0.18
• feff865 bump crate version to 0.0.18
• 4559084 bumping substreams, substreams-macro and test cargo toml versions to 0.0.15 for release
• 013d83f change some log level
• a59364e create a workerPool by service instead of by request and add job stats
• d36f035 fix all tests
• e7132b7 fix all tests
• 7b33054 fix cache memory issue
• 494ba42 fix getLast
• 83626f1 fix index out of range
• 0fac649 fix log
• 3a73f56 fix memory leak
• e7a44cd fix panic when request does not contain any module
• 4d9b76c fix remote host
• c90238b fix some tests
• 2f8fe9b fix stats update
• 35e91f6 fix step to proto matching in response
• f3e318b fix step to proto matching in response
• 83b0a3e fix work progress handling
• 7f889d6 fix work progress handling again
• bd391c9 fix worker not returned to pool on error
• d21299f fixing cache no-output flag for backprocessing store
• c69345b fuel integration, wip
• 62b4d54 merge workable-substreams
• c099c2a merged develop
• 215313c more cleanup of builder keyword
• 84f788d more cleanup of builder keyword
• 8207556 moving functionnal options pattern for Service struct to options file and adding cacheEnabled attribute
• 040e0ff now backend by wasmtime. All tests pass
• 15378bb only send hostname in grpc header if SUBSTREAMS_SEND_HOSTNAME is set to 'true'
• 0e7f087 only send hostname in grpc header if SUBSTREAMS_SEND_HOSTNAME is set to 'true'
• ca19bdc remove way to verbose logs
• c71225f removing dependencies for wazero
• a890810 removing error returned value on cache get as there is no error that can occur and adding GetAtBlock method
• b8bb54a removing flip from stringMap and byteMap
• bc853af removing storeOptions as it is unused
• 2692aa0 resetting kv on store when unmarshalling
• 33a9600 resetting the kv once it was unmarshalled
• d335118 revert fuel integration. Merged develop
• 14c30da setting 8076 iterations for deterministic value where any other iteration wasmtime will fail deterministically
• fff9722 signaling completion in the outter loop after the write go routines have been launched
• 15577d8 skipping recursion test
• 6f56d1d skipping recursion test
• 43599e7 spacing, removing wazero sys dependency, no more check on the ExitError like previously done on wazero as the return value is handled differently and changing some Module struct value attribute names
• 05f20da state: remove unused file
• 62dfed6 state: remove unused file
• 5f35b85 switching back to github.com/tetratelabs/wazero
• d46e304 temporary commentating the update for jobStat
• 7017af4 update executionStack
• 67f0119 wasm module and instance refactoring
• 2974147 wip

v0.0.17

Changelog

• 670b369 Added Append to the intrinsics.
• 8b32c1e Added changelog docs.
• db6b195 Added some notes and comments on the patterns, things remaining, things to adjust.
• 970fadf Better next boundary handling.
• 41d643d Boundary check upon saving snapshots more legit, and aligned on intervals.
• 010eda2 Create book.json
• 3d59efb Don't run if the snapshots weren't found. Can't work with a nil pointer anyway.
• 0d13767 Drafted some tests for new snapshots-backed job splitting.
• 77f9dff First draft of the splitter rework. Next: * update the splitter_test and separate the two stages: partials expectations from request ranges expectations. * make sure the SplitWork feeds into the right things after: * squasher from partialsPresent * notification sent to end user with partialsPresent.Merged() * recompute reqChunk based on partialsMissing.MergedChunked() * finish computeRequests(subreqSplit)
• 0431365 Fix error handling
• 264299e Fixup risk of corruption for stores, by separating the generation of the content to be written from the upload, which is where the latency occurred.
• 5356f13 Github actions first attempt
• 0d86e6e Got rid of the boundary on the STORES!!! They belong to the Pipeline.
• f2eda18 If it's nil, don't access it for log purposes.
• a3af7d9 Implement append merge policy.
• 7c195b5 Implemented initial snapshot with -i
• b2cbced It's a live (with functioning partial state instead of load delta)
• 7d54831 Merge branch 'develop' of github.com:streamingfast/substreams into develop
• 5820d51 Merge branch 'develop' of github.com:streamingfast/substreams into develop
• db1a511 Merge branch 'feature/fix_nil_snapshot' into develop
• a9a233a Merge branch 'feature/fix_nil_snapshot' into develop
• 441ab4b Merge branch 'feature/github' into develop
• eaeaae1 Merge branch 'feature/splitter-rework' into develop
• f75b77c Merge pull request #16 from fubhy/chore/bsr
• 413c262 Merge pull request #17 from fubhy/chore/use-remote-builder
• 879430a Merge pull request #18 from schmidsi/patch-1
• f97d27f Merge pull request #29 from 0xbe1/patch-3
• f80d9ad More boundary work on the Store. Hopefully that nails it.
• c305c72 Move append to its file.
• d8dffec No logic change. Moved code around, and some renames.
• a30525a Now fetch the stores. Not yet using the new data to do anything.
• 2ffb44d Now send the proper progress messages, based on partialsPresent too. Reverted to storeSplit in the SplitWork job. Segregated the batchRequests() function with its own concern. Save partial stores with better conditions. Roll() now truncates.
• b359fe0 Only send request for initial store snapshots for store modules.
• 45b3f35 PIpeline notes
• 8cda6d9 Prepare release v0.0.16
• 9952d2e Preparing release for version 0.0.17
• a54f358 Ready test suite for modifications and using the snapshots
• 6a9cb83 Remove since reqChunk not exist any more // TODO(abourget): what is dispatched here would much better be some of those objects // in the WorkUnit instead, like the reqChunk directly // Ideally the Callback over there carries the reqChunk, and was seeded with that reqChunk
• 010f040 Remove some Grrr
• 3b2cc49 Remove unnecessary comment
• df2de33 Rename
• 431a344 Renamed a few things in the substreams crate docs.
• b7c4bbc Rework of the orchestrator.
• eaa58be Simplify
• 14a0a22 Slight rework of the barmode.
• cb7278d Snapshots: Sort once
• 6339b50 Some other renames.
• ee9f4ed Some workz
• 37f4455 Splitter -> workplan.go Fixed workplan now ALWAYS returning a WorkUnit for all stores, so you can safely iterate on the WorkPlan and be sure to cover all required stores. (backprocess.go and workplan_test.go where we never expect a nil anymore.)
• 70036bb Test and ensure bound checks for bump intervals in store saves.
• c8f1ff2 Test the ValidateStoresReady
• 243d48a Update book.json
• 478393a Update creating-protobuf-schemas.md
• ee1fab6 Update creating-your-manifest.md
• 95d04f8 Update latest finesses for append
• 965b714 Updated changelog.
• 3e6f55c Usage wording
• 0bcc95b WIP: now the partials that are computed are not controlled by the orchestrator, he only receives what the backends produced as partials, according to their own storeInterval configurations. They will therefore be able to take more or less snapshots depending on the size of stores, or other heuristics.
• 21b3dd8 Whoops
• f472b80 Wrap up comments
• faada5e add cache entry for empty output
• 5bb4129 add hostname in grpc_header in service, client and on pipeline
• bcca322 added host information in log
• 8d2051c added memory size log
• 58265e5 added memory size log (fixed)
• ba1262e added shouldReturnProgress, shouldReturnDataOutputs and should return
• 7c9a381 added squashing stats and log
• f6e9b90 adding append state method with extern calls and implementation
• 248f89a adding cacheEnabled attribute to pipeline and adding if statements to use and not use the cache in places where the output cache is loaded, get, save and update
• c06d3b6 adding check for the splitwork to make sure we have some work todo
• 2bbda15 adding fix me for jobStat update method
• 44c8805 adding pipeline options to pipeline
• d5de711 adding rust tests for append and appendBytes
• d68f955 adding todo to return the type when an error occurs in the parse_input method
• 05c8f25 always delete partial file when squashed
• a35b9b5 augmenting worker stat
• b25ac2d block waiter: close channel only once
• 4559084 bumping substreams, substreams-macro and test cargo toml versions to 0.0.15 for release
• d054a98 bypass pipeline execution when requested output is cached
• 02e3fca catching cancel context in strategy when client manually cancels substreams execution
• 59afbbb changing conversion of &String to &Vec for append
• 15e825a commented splitter and fix tests
• a59364e create a workerPool by service instead of by request and add job stats
• 078f4a4 creating work only when partials are missing, iterating on workPlan when initializing strategy and squasher instead of stores
• d88a8b3 declare out explicitly
• e4ae2eb do not send hostname as grpc header
• e2af2b4 docker: fix build path
• 3e13731 docker: fix tagging
• 3b4c46b fix GrpcClientFactory type
• cd74769 fix avg calculation in StoreSquasher.go
• caf7782 fix compilation issues
• 940ee4b fix error handling
• 9becdfb fix github repo url in cargo.toml
• 469e065 fix gorountine leak by making sure to close grpc connection after each sub request
• 36e7cc8 fix job scheduling order de-activated cached output stream
• 17f5293 fix last send block calculation
• d8270a0 fix lastBlock so it only return value from none partial file
• 0fac649 fix log
• 449c229 fix missing last partial file from sub request by fix the bumpStoreSaveBoundary func
• 4d9b76c fix remote host
• 7a093ee fix same hash for multuple modules
• 1d536e8 fix shouldReturnProgress, shouldReturnDataOutputs
• 0094514 fix some minor fixme
• 7ae1efa fix squasher shutdown sequence
• 2f8fe9b fix stats update
• 7172917 fix test
• 10d4b0a fix typo
• dfaee40 fixing typo and adding append enum for the manifest
• 6f775b3 flip sorting
• 2d0a2b3 github: fix docker build step
• 75cbdfb give a name to NewSubstreamsClient func outputs for clarity
• ddc4c6a handling panic when partialsChunks is empty
• 2088ca0 integrate partialsWritten from req trailer
• 8207556 moving functionnal options pattern for Service struct to options file and adding cacheEnabled attribute
• 01a951c multi-threaded squasher store access
• 33ba09b multi-threaded squasher store access, again
• 329599b nextExpectedBoundary refactor. We keep the explicit passing of block numbers for both LOADING and DELETING, as to not hack our way around setting nextExpectedBoundary only to get a proper filename (if it were to pluck the end boundary from its nextExpectedBoundary). Also, for WriteState(), there are 2 situations, one where the Squasher receives instructions as to what should be written, from the subrequest, and has its own tracker of nextExpectedStartBlock, so rather than being driven by the underlying nextExpectedBoundary, is the one driving the ranges. The only left is the real-time one in saveStoresSnapshots(). It is therefore going to instruct WriteState() with the right block number. It concentrates all of those boundary checking INSIDE that function. And that's it.
• 67bab9b only updating pipeline store map for backProcessedStores
• 249935f orchestrator: debugging job pool ordering. (queue channels are now unbuffered)
• 37ec9e6 orchestrator: first attempt at getting correct ordering
• 491892d orchestrator: fix tests
• b8fd7ea orchestrator: job pool test updated to ensure behaviour of ordering
• b9f2b54 orchestrator: more cleanups
• da00d75 orchestrator: patch test
• b913752 orchestrator: re-work of how to schedule jobs for stores with no dependencies
• 10a2978 orchestrator: refactorings
• b817955 orchestrator: renames and test fixes
• 577be97 orchestrator: requestpool -> jobpool
• 8b8bf98 orchestrator: some cleanups
• 0aff40b orchestrator: update tests
• 1b31537 pipeline: some cute renames
• dede515 pipeline: start refactoring functions
• f0f5514 prevent stream cached data when sub request
• d3f4148 prevent sub request creation for range for with we already have the partial file.
• 2996ca0 printing store deltas on a new line, and not right after the ending } of a mapper
• 77a4b12 publish protos to bsr
• 3d3ff10 remove //TODO(abourget): get the requests we want to submit here..
• 573ed94 remove host from progress block range
• 2baece6 remove blockRange from builder
• 9c40fc9 remove builder isLoaded fix tests
• a2ca4e1 remove chunks from reqChunk
• 4a05fb3 remove dead code
• 7893f2b remove graph-node package
• 2106227 remove multi-threaded squasher store access
• 252a47d remove reqChunk since it is the exact same them block.Range
• 5c5ef64 remove some completed todos
• a3bf0ec remove the EOF err when cache streaming is done
• 89cefe5 remove the block dep on ModuleExecutor run func
• a1920d4 rem...

v0.0.16

Changelog

• 8cda6d9 Prepare release v0.0.16

v0.0.15

Changelog

• 4559084 bumping substreams, substreams-macro and test cargo toml versions to 0.0.15 for release

v0.0.13

CLI

• Changed substreams manifest info -> substreams info
• Changed substreams manifest graph -> substreams graph
• Updated usage

Changelog

• 43e9bd7 Added a test with modInit starting at 0.
• b89c9c8 Added comment to --start-block. Fix ranges splitter tests.
• e5d3aaf Don't early kill the scheduler if it didn't fail.
• db8780d Don't send BlockScopedData when we're doing backprocessing.
• 13084a4 Fix tests.
• b0dfd11 Fixed issue with ranges.
• cc6c420 Major rework of the scheduling plan, called SplitWork, called WorkUnit, tobe renamed.
• cdce6e5 Merge remote-tracking branch 'origin/develop' into splitter
• fb9c955 Revise usage and CLI docs.
• 21e5772 Simplify the scheduler exit loop.
• e346177 SplitWork implementation, responsible for driving the show in Squasher and Strategy.
• 152de88 Squasher / Waiter / Queue / RequestPool / Strategy Simplified. Hanging bug fixed, which was caused by the Squasher never preparing the store when no work was required.
• 0a00c03 Unlock, but what does that test do anyway?
• b42926d Updated change-log.
• 8c4805f Updated some element of the docs
• e203658 Updated substreams info
• 8d82cb2 Whoops, don't print "" on 'enter'.. misfeature :)
• 2520a0b substreams info and substreams graph now.
• f752f3c block range: split occurs on chunk size boundary
• 069d028 orchestrator test: make adding to queue concurrent
• c75a8ba queue: add more testing
• 054021e splitter wip
• e761111 squasher: fix test