Skip to content

v0.0.21
Compare
Choose a tag to compare
@jubeless jubeless released this 31 Oct 19:44
· 2153 commits to develop since this release
v0.0.21
0fb6f55

Changelog

  • Moved Rust modules to github.com/streamingfast/substreams-rs

Library

  • Gained significant execution time improvement when saving and loading stores, during the squashing process by leveraging vtprotobuf

  • Added XDS support for tier 2s

  • Added intrinsic support for type bigdecimal, will deprecate bigfloat

  • Significant improvements in code-coverage and full integration tests.

CLI

  • Lowered GRPC client keep alive frequency, to prevent "Too Many Pings" disconnection issue.

  • Added a fast failure when attempting to connect to an unreachable substreams endpoint.

  • CLI is now able to read .spkg from gs://, s3:// and az:// URLs, the URL format must be supported by our dstore library).

  • Command substreams pack is now restricted to local manifest file.

  • Added command substreams tools module to introspect a store state in storage.

  • Made changes to allow for substreams CLI to run on Windows OS (thanks @robinbernon).

  • Added flag --output-file <template> to substreams pack command to control where the .skpg is written, {manifestDir} and {spkgDefaultName} can be used in the template value where {manifestDir} resolves to manifest's directory and {spkgDefaultName} is the pre-computed default name in the form <name>-<version> where <name> is the manifest's "package.name" value (_ values in the name are replaced by -) and <version> is package.version value.

  • Fixed relative path not resolved correctly against manifest's location in protobuf.files list.

  • Fixed relative path not resolved correctly against manifest's location in binaries list.

  • substreams protogen <package> --output-path <path> flag is now relative to <package> if <package> is a local manifest file ending with .yaml.

  • Endpoint's port is now validated otherwise when unspecified, it creates an infinite 'Connecting...' message that will never resolves.

Commits

  • 918c8d3 💄
  • 5558de4 💄
  • 232454c Add Contributing and License
  • 3ab5a92 Add directory for images and header image for docs
  • 66fe9f4 Add header image
  • d2f5b0c Add prelude to lib and fix to type convertion
  • 3b8d3a5 Added Binary relative path support in manifest
  • 52bdf04 Added benchmark to see how long it takes to serialize N keys to storage
  • 9d2c891 Added benchmark to see how long it takes to serialize N keys to storage
  • bfd9609 Added block stats trigger
  • ea52c29 Added cargo lock
  • dd0029b Added custom proto marshalling
  • 6281caa Added debugging log
  • 1b4992d Added logging
  • e1bdd30 Added logging of requested_start_cursor
  • ea0c7d8 Added more request stats
  • de5c8cf Added possibility to specify --output-file <template> (short of -o) to substreams pack
  • 5da0e0d Added request stats
  • 9c4b007 Added request stats store squasher
  • 0c176be Added squasher store tests
  • 9280af8 Added tests
  • cfe4b34 Added two more relevant changelog entries
  • c976fb6 Added v0.0.21 release notes
  • 488f1fa Added vtproto support
  • 213a4ea Adding minimal changes required to get commands working properly on windows OS
  • eac3f01 Adding optional flag to specify output directory for pack output.
  • a027a1e Adding to changelog
  • 6384403 Another grpc client fix
  • 18c3c6a Bumped dstore to fix the latency in SubStore initializations.
  • 049417a Bumped dgrpc again
  • 316e592 CLI is now able to read .spkg from gs://, s3:// and az:// URLs, the URL format must be supported by our dstore library).
  • b437533 Call promote waiting jobs after preparing the jobs!
  • 4665c13 Clarify the nextStoreSaveBoundary condition, to be larger or equal than the current blockNum.
  • 58da6ff Clean up boundary test
  • 60b7333 Clean up logging
  • fa92021 Cleaned up Store and Partial store
  • 94eaade Cleaned up benchmarking marshaller test
  • b23cb12 Cleaned up logging and req statas
  • 4652618 Cleaned up proto
  • bf1f6f8 Cleaned up service and test
  • be92abc Cosmetics.. Reset change place, just a dance for it to be ready next.
  • f952595 Decouple the Worker from the orchestrator (doesn't know about Job) anymore.
  • 9a186d8 Edits to text
  • 8dfa922 Enabled release draft mode and disabled linux & darwin amd64 build
  • 86f8ccc Endpoint's port is now validated on substreams run
  • 0131e43 Externs generation
  • bef53f3 Few renames
  • fb83fd5 Finish up the job scheduler. Tests now need to be written.
  • 5fb9d8c First draft of BaseStore from Config.
  • 55308e6 First draft of the scheduler testing. Better abstraction for the JobRunner.
  • a8a3cf9 First pass at supporting cursor correctly
  • 724e593 First stub of SessionInit message, to provide the trace_id.
  • a6b5535 Fix assert_all test
  • cbb2669 Fix error handling
  • 8784513 Fix integration tests
  • a8d9d71 Fix merge test
  • ff63ef3 Fix miss refactor
  • 1858418 Fix multiple store issues
  • 491f99f Fix output saving on cache
  • e7b6ba1 Fix partial merging and squasher
  • 635e7a2 Fix pipeline test
  • 30fad36 Fix rebase issue
  • 0c4b797 Fix test
  • c1ba0c6 Fix tests
  • 479129c Fix tests
  • 05a1cec Fix/sol squasher incomplete (#50)
  • 55ec82c Fixed benchmark
  • b13d0a7 Fixed cachev1 when running tests with -count N
  • e36a3f8 Fixed relative path not resolved correctly against manifest's location in binaries list
  • a322d74 Fixed relative path not resolved correctly against manifest's location in protobuf.files list
  • e0329e0 Fixed tests in scheduler.
  • 01760e8 Fixed usage of TargetBlock in tui view
  • 099390d Fixup NextJob
  • c1bac36 Fixup the scheduler's runOne.
  • a4c9aa4 HTML test
  • 6b32527 Have the backprocess job create the storeMap, and return it untouched. He's responsible for creating the whole graph.
  • 852d904 Implemented grpc client factory
  • b77de8b Implemented test for setupSubrequestStores Updated MockStore to support easier SetFile.
  • 73f8dbd It a live!
  • c77507d Last iteration of out-dir flag description
  • 79bd235 Made more explicitely StartBlockNum vs EffectiveStartBlockNum and made a few comments
  • 78f8c5f Made output path relative when generating proto for a local manifest file
  • 0eccb52 Major Refactor
  • 4a68ddf Making changes to make it super clear on how output-dir works.
  • 16e9ec1 Merge branch 'develop' into feature/backprocessor-refactor
  • 27309c7 Merge branch 'develop' into feature/backprocessor-refactor
  • 45585b2 Merge branch 'develop' into feature/backprocessor-refactor
  • badaaba Merge branch 'develop' into feature/backprocessor-refactor
  • b83f8f3 Merge branch 'develop' into feature/codegen
  • 9017ff1 Merge branch 'develop' into feature/integration-test
  • 2587686 Merge branch 'develop' into feature/xds
  • 8a9b81c Merge branch 'develop' of https://github.com/streamingfast/substreams into develop
  • 5adfd83 Merge branch 'feature/backprocessor-refactor' into develop
  • 3a2a98a Merge branch 'feature/binary-marshaller' into develop
  • 5cc1b5b Merge branch 'feature/codegen' into develop
  • 513eb12 Merge branch 'feature/some-rework' into develop
  • 1435199 Merge branch 'feature/xds' into develop
  • 6928079 Merge branch 'fix/merge-big-int' into develop
  • 4118127 Merge branch 'refactor' into develop
  • c061bb0 Merge pull request #48 from tilacog/patch-1
  • 8f67cbf Merge pull request #54 from streamingfast/feature/bigdecimal
  • 83c6bb6 Merge remote-tracking branch 'origin/develop' into develop
  • fab246b Merge remote-tracking branch 'origin/develop' into develop
  • 41ef384 Merge remote-tracking branch 'origin/develop' into feature/backprocessor-refactor
  • b243da2 Minor formatting fixes
  • 8bad9ed ModuleHashes abstraction, more speedy with cache. We can cut 9 seconds of initialization time!!!
  • 192a7fa Mucho more tests better.
  • 9e0251a Notes and renames and what not 💄
  • 0729cde Now VTProto is the marshaller
  • 7975970 OpenTelemetry
  • a20b2af OpenTelemetry
  • d8ded8e OutputCache fused with OutputCacheState.
  • e96ad07 Pass in runtimeConfig here and there.
  • 89f3e94 Pipeline needs a ForkHandler to not crash tests. Quick renames
  • f70bdc8 Preparing for next unreleased version
  • a31b124 Put back NewTestModules in testing because it's used in foreign packages
  • b9631fe Put integration tests behind an env var to run, fixed path to be relative to substreams.yaml (now that it works for everything)
  • 31bade9 Quite another refactor here. Fully broken!
  • 8f46674 Registered Metrics & Updated Naming Conventions
  • 0343fff Remove links to documentation pages
  • 32adf53 Remove testing HTML
  • 4d230ba Remove unused object.
  • 0c86415 Removed pipeline.RequestContext stack recording of span(s)
  • 5097429 Removed specialized errors.GRPCError and uses standard status.Error everywhere
  • 9fe2c2e Removing unecessary check
  • 7197df0 Replaced request context with context abstractions
  • 6fc3287 Revert "removed ExecSpan"
  • 0386e6b Scheduler rework, absorb the JobsPlanner, rework the WorkPlan. Make it the thing that prioritizes. Have an abstraction to prioritize it.
  • 9f1c8ab Simplify
  • aeb6b3a Small nitpick on CHANGELOG
  • 2e1e9af Small refactor of SetStatus for traces.
  • fbecf39 Temp HTML changes in docs
  • 7a9f93a Test the init messages from the work.Plan. Fixup locking for the mutators of the Plan.
  • 51eaf59 The error should be handled
  • 2cf40c5 Tweak some display and logs.
  • 7ef59ab Update header image for README
  • 8ef14f0 Update header image for docs
  • 365de5b Update setting-up-handlers.md
  • 1541fe3 Use the right default values man.
  • ae2e488 Using isAbs for deducing a relative path
  • cb83e84 Using spkg in integration tests
  • ff2c5be WIP
  • 310cf7d WIP backprocessor refactor
  • 36451f0 Wrote two more tests for runOne in scheduler. Fixup the jobrunnerpool broken test.
  • ef64f75 add integration tests by default in bin/test.sh
  • 98ccfaa add missing files
  • a8a0ecf add the module hash to error logs with module name
  • b8a7e16 added SquashTime/BlockProcessTime/Block metrics
  • 9720241 added couple tests
  • f250424 added debug logs
  • 19ed65d added missing files
  • 859e552 added more req stats
  • f7b0532 added partial metrics
  • d5b8315 added req stats fix
  • 2e0e621 added squash metrics
  • 376330c added xds creds
  • 82ea7e4 added xds support in client
  • 3b86c61 added xds support in client
  • fe26303 adding SubstreamsClientConfig struct
  • b389b76 adding bigint, bigfloat and int64 for stores cli validation
  • 8dccad7 adding different combinations for the manifest cli validation
  • f7c18cc adding forkable generator with tests
  • a5e0868 adding inputValue when hashing a module
  • b8464a3 adding protofiles to be read from url in substreams manifest file
  • 13e8e1f adding tests for StoreSet/StoreGet I64 and fixing some broken test
  • bb2549d adding tests for resolve start block
  • 4b383d7 adding the substreams client config to the worker and removing global config var
  • f41864f adding unit tests for generated rust files
  • 6c9299b adjust Register to match firehose new RegisterServiceExtension and use our own StreamFactory to specify the correct steps and decouple from firehose implementation
  • 5e2a714 all tests are now running. Remove SUBSTREAMS_INTEGRATION_TESTS
  • 03508e8 back to single grpc client
  • 32dee42 backprocess: err handling, and defer bubbling up.
  • 2f97a7b better SUBSTREAMS_SEND_HOSTNAME handling
  • 81b0456 better error message in client
  • 596dfaa better failure progress handling
  • 1d2f18c better logging
  • cd2ba33 better logging
  • 86405ff better store detection
  • d9086de bring back SUBSTREAMS_SEND_HOSTNAME
  • 7121f15 bring back test condition
  • 2e5f03f bump substreams
  • 3c5570b bump substreams to 0.3.0
  • 834fb48 bumped dgrpc
  • bcc9877 canceling ui when receiving a failed progress
  • 2c55225 change default marshaller to vtproto
  • 7bb9553 change logs
  • 172b895 cleaned up common functions
  • 97872c0 cleaned up more tracing and spanning
  • c0ce831 cleaned up store config
  • a664b00 cleaned up store map
  • 6a1df70 cleaning up logging and stats
  • c154bc8 commenting part where we call returnFailureProgress
  • e88706a delete test broken
  • d27bdff delete test broken
  • a654ab6 dgrpc refactoring
  • eaf4cd8 enabled darwin platform
  • 42a40a7 ensure we fail fast on grpc connection
  • 24e02ac error.go -> endofstream.go
  • e1385e4 execout: fix flush boundary
  • 164db8c execout: fix flush method parameters
  • 5a7dcd1 execout: flush at block range boundary, regardless of irreversibility condition
  • f559673 execout: flush at boundary, update test
  • 2cb1394 execout: interface and code cleanup
  • 6f63399 fix MAJOR typo
  • e5d5183 fix argument order
  • 93bf69d fix compilation issue
  • b5e9563 fix compilation issues
  • de93313 fix compilation issues
  • 9f44d2e fix compilation issues
  • e1e1d8e fix compilation issues after merging develop
  • c5e249f fix compilation issues cause be substreams-rs change
  • ba6d63f fix concurrent map write
  • 7199f30 fix cursor start at +1, add integration test
  • acfa4a3 fix delete prefix test
  • 9a6b3ff fix go 18
  • 6f54478 fix grpc client creation
  • 3b2ced7 fix handle irreversible block
  • 57200af fix integration test block source type
  • ea82695 fix merge conflict stuff
  • fd10945 fix minor bug and add protobuf generation
  • 6f63cfa fix missing clock support
  • d252305 fix output for a mapper when cached when not pushed downstream to the next substreams to use as input
  • b2a0562 fix setifnotexists bug when setting value on disk
  • 10f0834 fix signature test
  • 70a1ceb fix test still a and repack
  • f31145b fix test still a wip
  • 5f7a7f0 fix tests
  • 6dccbde fix tests
  • 71d6650 fix tests
  • 93b6221 fix typo
  • b4f765d fix wasm input source checker
  • 223fc19 fixed metrics
  • 9720b41 fixed squash metrics
  • 3166a11 fixing big int merge strategy
  • a9b9953 fixing tier2 client
  • a8edbf3 gitignore
  • ab8ff68 it is working but still polishing
  • 432a350 make GRPC_XDS_BOOTSTRAP log repeat less...
  • 126a2e0 module_executor: save to the cache only when module output is VALID. It is deemed invalid when a PartialKV produces its deltas. These are not pieces of data to be saved to the cache output. module_executor: sort functions by use storeexec: quick rename
  • d766dea more work and cleanup on orchestrator refactor
  • 17adee9 move rust libs to github.com/streamingfast/substreams-rs
  • d36c2d0 only save store state for output module when in sub request mode
  • 5aa8ee8 orchestrator: added tests, additional cleanup and refactoring
  • bd919e9 orchestrator: additional cleanup
  • e7fb497 orchestrator: cleanup and refactoring
  • 3ea2a2b orchestrator: pass ctx through waiting for completion, rename func
  • 83403ad pb/mod.rs generated
  • a67491a pipeline exec: add back logging and span
  • 2461745 pipeline execout: fix fmt string
  • fe46953 plan: add names to TestPlan_NextJob to make tests runnable
  • 0fb6f55 plan: make output of splitWorkIntoJobs deterministic
  • 71c7727 prevent annoying 'error getting stream header' log on disconnection
  • c176087 prevent overriding of pb/mod.rs
  • 350d625 properly retuing incoming message when getting an error
  • 60466e0 refactoring pipeline runExecutor. delegate responsibility to new ModuleExecutorRunner object. /colin
  • ad1dfe1 remove commented code
  • 16cf468 remove dead code
  • 32a30e9 remove go replace
  • 0b02059 remove skipAuth when in xsd mode
  • eb02bed remove useless test file
  • a749355 remove warning from integration test
  • c6dc82f remove xds support
  • adb75fb removed ExecSpan
  • 64ffed3 removing cancel defer
  • c4e83ea removing functionName and moduleName as we have the data in the Module.name
  • 7e33d7c removing grpcClientFactory pipeline attribute
  • d3f1293 removing grpcFactory attribute from the worker in the work pool and replacing it with newSubstreamsClientConfig
  • 5bf01d2 removing hard-coded hash values for map and store decode commands
  • 92e9378 removing pipeline cacheEnabled and partialModeEnabled boolean and service cacheEnabled which were used to run the server with or without cache for substreams
  • 4397d4e removing println from inputValue when hashing a manifest module input values
  • 9daa606 renaming BigFloat to BigDecimal, deprecating uses of BigFloat and it's externs to be backward compatible and cleaning up code in state_extern and the validations done
  • 0bda24c replacing json marshaller and json unmarshaller with binary marshaller, fixing tests and adding boundary tests for binary marshaller
  • 5078f67 revert "add squash metric"
  • de1441c revert proto files from url
  • b7f8689 scheduler: simplify runSingleJob
  • 49a88f6 shorter logger name
  • 9c35f1c skipping broken test
  • 99f3fe5 some refactoring
  • aac2974 stable test
  • a7ea2f0 store: cleanup functions in merge
  • 5df13d3 store: fix int64 merge functions
  • 113137a test are back on track
  • 65530b8 testing a fix
  • 77f565d tidy go modules
  • 2e88305 tool: added module debugging tool
  • 3b86ec2 update changelog
  • d9cb2d9 update generator and go template
  • 5e5fe18 use embedded file instead of constants for template
  • 4049e18 var and test cleanup
  • b0d9585 wip
  • e92293a wip
  • 9d6e2ab wip
  • 410a8e1 wip: refactoring block hooks
  • bb67b8a working on all tests
  • 24b63a9 xdscreds when GRPC_XDS_BOOTSTRAP is set

Contributors

robinbernon
Assets 7
Loading