v0.0.17

Changelog

• 670b369 Added Append to the intrinsics.
• 8b32c1e Added changelog docs.
• db6b195 Added some notes and comments on the patterns, things remaining, things to adjust.
• 970fadf Better next boundary handling.
• 41d643d Boundary check upon saving snapshots more legit, and aligned on intervals.
• 010eda2 Create book.json
• 3d59efb Don't run if the snapshots weren't found. Can't work with a nil pointer anyway.
• 0d13767 Drafted some tests for new snapshots-backed job splitting.
• 77f9dff First draft of the splitter rework. Next: * update the splitter_test and separate the two stages: partials expectations from request ranges expectations. * make sure the SplitWork feeds into the right things after: * squasher from partialsPresent * notification sent to end user with partialsPresent.Merged() * recompute reqChunk based on partialsMissing.MergedChunked() * finish computeRequests(subreqSplit)
• 0431365 Fix error handling
• 264299e Fixup risk of corruption for stores, by separating the generation of the content to be written from the upload, which is where the latency occurred.
• 5356f13 Github actions first attempt
• 0d86e6e Got rid of the boundary on the STORES!!! They belong to the Pipeline.
• f2eda18 If it's nil, don't access it for log purposes.
• a3af7d9 Implement append merge policy.
• 7c195b5 Implemented initial snapshot with -i
• b2cbced It's a live (with functioning partial state instead of load delta)
• 7d54831 Merge branch 'develop' of github.com:streamingfast/substreams into develop
• 5820d51 Merge branch 'develop' of github.com:streamingfast/substreams into develop
• db1a511 Merge branch 'feature/fix_nil_snapshot' into develop
• a9a233a Merge branch 'feature/fix_nil_snapshot' into develop
• 441ab4b Merge branch 'feature/github' into develop
• eaeaae1 Merge branch 'feature/splitter-rework' into develop
• f75b77c Merge pull request #16 from fubhy/chore/bsr
• 413c262 Merge pull request #17 from fubhy/chore/use-remote-builder
• 879430a Merge pull request #18 from schmidsi/patch-1
• f97d27f Merge pull request #29 from 0xbe1/patch-3
• f80d9ad More boundary work on the Store. Hopefully that nails it.
• c305c72 Move append to its file.
• d8dffec No logic change. Moved code around, and some renames.
• a30525a Now fetch the stores. Not yet using the new data to do anything.
• 2ffb44d Now send the proper progress messages, based on partialsPresent too. Reverted to storeSplit in the SplitWork job. Segregated the batchRequests() function with its own concern. Save partial stores with better conditions. Roll() now truncates.
• b359fe0 Only send request for initial store snapshots for store modules.
• 45b3f35 PIpeline notes
• 8cda6d9 Prepare release v0.0.16
• 9952d2e Preparing release for version 0.0.17
• a54f358 Ready test suite for modifications and using the snapshots
• 6a9cb83 Remove since reqChunk not exist any more // TODO(abourget): what is dispatched here would much better be some of those objects // in the WorkUnit instead, like the reqChunk directly // Ideally the Callback over there carries the reqChunk, and was seeded with that reqChunk
• 010f040 Remove some Grrr
• 3b2cc49 Remove unnecessary comment
• df2de33 Rename
• 431a344 Renamed a few things in the substreams crate docs.
• b7c4bbc Rework of the orchestrator.
• eaa58be Simplify
• 14a0a22 Slight rework of the barmode.
• cb7278d Snapshots: Sort once
• 6339b50 Some other renames.
• ee9f4ed Some workz
• 37f4455 Splitter -> workplan.go Fixed workplan now ALWAYS returning a WorkUnit for all stores, so you can safely iterate on the WorkPlan and be sure to cover all required stores. (backprocess.go and workplan_test.go where we never expect a nil anymore.)
• 70036bb Test and ensure bound checks for bump intervals in store saves.
• c8f1ff2 Test the ValidateStoresReady
• 243d48a Update book.json
• 478393a Update creating-protobuf-schemas.md
• ee1fab6 Update creating-your-manifest.md
• 95d04f8 Update latest finesses for append
• 965b714 Updated changelog.
• 3e6f55c Usage wording
• 0bcc95b WIP: now the partials that are computed are not controlled by the orchestrator, he only receives what the backends produced as partials, according to their own storeInterval configurations. They will therefore be able to take more or less snapshots depending on the size of stores, or other heuristics.
• 21b3dd8 Whoops
• f472b80 Wrap up comments
• faada5e add cache entry for empty output
• 5bb4129 add hostname in grpc_header in service, client and on pipeline
• bcca322 added host information in log
• 8d2051c added memory size log
• 58265e5 added memory size log (fixed)
• ba1262e added shouldReturnProgress, shouldReturnDataOutputs and should return
• 7c9a381 added squashing stats and log
• f6e9b90 adding append state method with extern calls and implementation
• 248f89a adding cacheEnabled attribute to pipeline and adding if statements to use and not use the cache in places where the output cache is loaded, get, save and update
• c06d3b6 adding check for the splitwork to make sure we have some work todo
• 2bbda15 adding fix me for jobStat update method
• 44c8805 adding pipeline options to pipeline
• d5de711 adding rust tests for append and appendBytes
• d68f955 adding todo to return the type when an error occurs in the parse_input method
• 05c8f25 always delete partial file when squashed
• a35b9b5 augmenting worker stat
• b25ac2d block waiter: close channel only once
• 4559084 bumping substreams, substreams-macro and test cargo toml versions to 0.0.15 for release
• d054a98 bypass pipeline execution when requested output is cached
• 02e3fca catching cancel context in strategy when client manually cancels substreams execution
• 59afbbb changing conversion of &String to &Vec for append
• 15e825a commented splitter and fix tests
• a59364e create a workerPool by service instead of by request and add job stats
• 078f4a4 creating work only when partials are missing, iterating on workPlan when initializing strategy and squasher instead of stores
• d88a8b3 declare out explicitly
• e4ae2eb do not send hostname as grpc header
• e2af2b4 docker: fix build path
• 3e13731 docker: fix tagging
• 3b4c46b fix GrpcClientFactory type
• cd74769 fix avg calculation in StoreSquasher.go
• caf7782 fix compilation issues
• 940ee4b fix error handling
• 9becdfb fix github repo url in cargo.toml
• 469e065 fix gorountine leak by making sure to close grpc connection after each sub request
• 36e7cc8 fix job scheduling order de-activated cached output stream
• 17f5293 fix last send block calculation
• d8270a0 fix lastBlock so it only return value from none partial file
• 0fac649 fix log
• 449c229 fix missing last partial file from sub request by fix the bumpStoreSaveBoundary func
• 4d9b76c fix remote host
• 7a093ee fix same hash for multuple modules
• 1d536e8 fix shouldReturnProgress, shouldReturnDataOutputs
• 0094514 fix some minor fixme
• 7ae1efa fix squasher shutdown sequence
• 2f8fe9b fix stats update
• 7172917 fix test
• 10d4b0a fix typo
• dfaee40 fixing typo and adding append enum for the manifest
• 6f775b3 flip sorting
• 2d0a2b3 github: fix docker build step
• 75cbdfb give a name to NewSubstreamsClient func outputs for clarity
• ddc4c6a handling panic when partialsChunks is empty
• 2088ca0 integrate partialsWritten from req trailer
• 8207556 moving functionnal options pattern for Service struct to options file and adding cacheEnabled attribute
• 01a951c multi-threaded squasher store access
• 33ba09b multi-threaded squasher store access, again
• 329599b nextExpectedBoundary refactor. We keep the explicit passing of block numbers for both LOADING and DELETING, as to not hack our way around setting nextExpectedBoundary only to get a proper filename (if it were to pluck the end boundary from its nextExpectedBoundary). Also, for WriteState(), there are 2 situations, one where the Squasher receives instructions as to what should be written, from the subrequest, and has its own tracker of nextExpectedStartBlock, so rather than being driven by the underlying nextExpectedBoundary, is the one driving the ranges. The only left is the real-time one in saveStoresSnapshots(). It is therefore going to instruct WriteState() with the right block number. It concentrates all of those boundary checking INSIDE that function. And that's it.
• 67bab9b only updating pipeline store map for backProcessedStores
• 249935f orchestrator: debugging job pool ordering. (queue channels are now unbuffered)
• 37ec9e6 orchestrator: first attempt at getting correct ordering
• 491892d orchestrator: fix tests
• b8fd7ea orchestrator: job pool test updated to ensure behaviour of ordering
• b9f2b54 orchestrator: more cleanups
• da00d75 orchestrator: patch test
• b913752 orchestrator: re-work of how to schedule jobs for stores with no dependencies
• 10a2978 orchestrator: refactorings
• b817955 orchestrator: renames and test fixes
• 577be97 orchestrator: requestpool -> jobpool
• 8b8bf98 orchestrator: some cleanups
• 0aff40b orchestrator: update tests
• 1b31537 pipeline: some cute renames
• dede515 pipeline: start refactoring functions
• f0f5514 prevent stream cached data when sub request
• d3f4148 prevent sub request creation for range for with we already have the partial file.
• 2996ca0 printing store deltas on a new line, and not right after the ending } of a mapper
• 77a4b12 publish protos to bsr
• 3d3ff10 remove //TODO(abourget): get the requests we want to submit here..
• 573ed94 remove host from progress block range
• 2baece6 remove blockRange from builder
• 9c40fc9 remove builder isLoaded fix tests
• a2ca4e1 remove chunks from reqChunk
• 4a05fb3 remove dead code
• 7893f2b remove graph-node package
• 2106227 remove multi-threaded squasher store access
• 252a47d remove reqChunk since it is the exact same them block.Range
• 5c5ef64 remove some completed todos
• a3bf0ec remove the EOF err when cache streaming is done
• 89cefe5 remove the block dep on ModuleExecutor run func
• a1920d4 remove this todo: because the only place StoresReady is call is after we received all sub request results and all callback and squash process are completed at that point // FIXME(abourget): Before checking the state of all those squashables, // we need to make sure all those Scheduler::Callback and Squash() calls // have finished, and that those merge() operations have completed.. // otherwise here we can't loop the squashables and expect to have // merged stores. // LET'S CHECK THAT LATER
• 0db5528 remove unused func params
• 3ca824d remove unused parameter
• ca19bdc remove way to verbose logs
• b8bb54a removing flip from stringMap and byteMap
• 218edfa removing if not found for snapshots, we can still have work todo even if we don't find any snapshots
• 7369ae6 removing the skip when setting a value when we have same values from one key set to another
• b91d8a9 rename all Store receiver to 's'
• 209f774 renaming variable for better clarity
• e324131 replaced chuck and chunks by block Range
• 8640f0d request scheduling and execution is working. Squashing is still broken
• 2692aa0 resetting kv on store when unmarshalling
• 2bec7e6 revert change on bumpStoreSaveBoundary
• b75ebaf revert some changes applied by mistake
• 5d91393 rust: some documentation proofreading
• 6835693 single threaded squasher
• c7fe61c skipping testLineBar
• 49f0b63 snapshots: add sorting to ensure LastBlock behaviour
• ae042f4 squash partial present into the workPlan
• e2af5c3 squashing fix
• ae27457 storagestate: fix a race between a loop and its closure value of storeName
• 8314cb8 substreams error connClose() returns error
• d46e304 temporary commentating the update for jobStat
• c50a137 use remote builder for protoc-gen-prost