v1.1.17

Fixed

• Missing decrement on metrics substreams_active_requests

v1.1.16

Added

• substreams_active_requests and substreams_counter metrics to substreams-tier1

Changed

• evt_block_time in ms to timestamp in lib.rs, proto definition and schema.sql

v1.1.15

Highlights

• This release brings the substreams init command out of alpha! You can quickly generate a Substreams from an Ethereum ABI:
  

• New Alpha feature: deploy your Substreams Sink as a deployable unit to a local docker environment!
  

• See those two new features in action in this tutorial

Added

• Sink configs can now use protobuf annotations (aka Field Options) to determine how the field will be interpreted in
  substreams.yaml:

  • load_from_file will put the content of the file directly in the field (string and bytes contents are supported).
  • zip_from_folder will create a zip archive and put its content in the field (field type must be bytes).

  Example protobuf definition:

  import "sf/substreams/v1/options.proto";
  
  message HostedPostgresDatabase {
    bytes schema = 1 [ (sf.substreams.v1.options).load_from_file = true ];
    bytes extra_config_files = 2 [ (sf.substreams.v1.options).zip_from_folder = true ];
  }
  

  Example manifest file:

  [...]
  network: mainnet
  
  sink:
    module: main:db_out
    type: sf.substreams.sink.sql.v1.Service
    config:
      schema: "./schema.sql"
      wire_protocol_access: true
      postgraphile_frontend:
        enabled: true
      pgweb_frontend:
        enabled: true

• substreams info command now properly displays the content of sink configs, optionally writing the fields that were bundled from files to disk with --output-sinkconfig-files-path=</some/path>

Changed

• substreams alpha init renamed to substreams init. It now includes db_out module and schema.sql to support the substreams-sql-sink directly.

• The override feature has been overhauled. Users may now override an existing substreams by pointing to an override file in run or gui command.
  This override manifest will have a deriveFrom field which points to the original substreams which is to be overriden.
  This is useful to port a substreams to one network to another.
  Example of an override manifest:

  deriveFrom: path/to/mainnet-substreams.spkg #this can also be a remote url
  
  package:
    name: "polygon-substreams"
    version: "100.0.0"
  
  network: polygon
  
  initialBlocks:
    module1: 17500000
  params:
    module1: "address=2a75ca72679cf1299936d6104d825c9654489058"
  

• The substreams run and substreams gui commands now determine the endpoint from the 'network' field in the manifest if no value is passed in the --substreams-endpoint flag.

• The endpoint for each network can be set by using an environment variable SUBSTREAMS_ENDPOINTS_CONFIG_<network_name>, ex: SUBSTREAMS_ENDPOINTS_CONFIG_MAINNET=my-endpoint:443

• The substreams alpha init has been moved to substreams init

Fixed

• fixed the substreams gui command to correctly compute the stop-block when given a relative value (ex: '-t +10')

v1.1.14

Bug fixes

• Fixed (bumped) substreams protobuf definitions that get embedded in spkg to match the new progress messages from v1.1.12.
• Regression fix: fixed a bug where negative start blocks would not be resolved correctly when using substreams run
  or substreams gui.
• In the request plan, the process previously panicked when errors related to block number validation occurred. Now the
  error will be returned to the client.

v1.1.13

Bug fixes

• If the initial block or start block is less than the first block in the chain, the substreams will now start from the
  first block in the chain. Previously, setting the initial block to a block before the first block in the chain would
  cause the substreams to hang.
• Fixed a bug where the substreams would fail if the start block was set to a future block. The substreams will now wait
  for the block to be produced before starting.

v1.1.12

Highlights

• Complete redesign of the progress messages:
  • Tier2 internal stats are aggregated on Tier1 and sent out every 500ms (no more bursts)
  • No need to collect events on client: a single message now represents the current state
  • Message now includes list of running jobs and information about execution stages
  • Performance metrics has been added to show which modules are executing slowly and where the time is spent (eth calls, store operations, etc.)

Upgrading client and server

Important

The client and servers will both need to be upgraded at the same time for the new progress messages to be parsed:

• The new Substreams servers will NOT send the old modules field as part of its progress message, only the new running_jobs, modules_stats, stages.
• The new Substreams clients will NOT be able to decode the old progress information when connecting to older servers.

However, the actual data (and cursor) will work correctly between versions. Only incompatible progress information will be ignored.

CLI

Changed

• Bumped substreams and substreams-ethereum to latest in substreams alpha init.

• Improved error message when <module_name> is not received, previously this would lead to weird error message, now, if the input is likely a manifest, the error message will be super clear.

Fixed

• Fixed compilation errors when tracking some contracts when using substreams alpha init.

Added

• substreams info now takes an optional second parameter <output-module> to show how the substreams modules can be divided into stages
• Pack command: added -c flag to allow overriding of certain substreams.yaml values by passing in the path of a yaml file.
  example yaml contents:

  package:
    name: my_custom_package_name 
  
  network: arbitrum-one
  initialBlocks:
    module_name_1: 123123123
  params:
    mod1: "custom_parameter"

Backend

Removed

• Removed Config.RequestStats, stats are now always enabled.

v1.1.11

Fixes

• Added metering of live blocks

v1.1.10

Backend changes

• Fixed/Removed: jobs would hang when config parameter StateBundleSize was different from SubrequestsSize. The latter has been removed completely: Subrequests size will now always be aligned with bundle size.
• Auth: added support for continuous authentication via the grpc auth plugin (allowing cutoff triggered by the auth system).

CLI changes

• Fixed params handling in gui mode

v1.1.9

Backend changes

• Massive refactoring of the scheduler: prevent excessive splitting of jobs, grouping them into stages when they have the same dependencies. This should reduce the required number of tier2 workers (2x to 3x, depending on the substreams).

• The tier1 and tier2 config have a new configuration StateStoreDefaultTag, will be appended to the StateStoreURL value to form the final state store URL, ex: StateStoreURL="/data/states" and StateStoreDefaultTag="v2" will make /data/states/v2 the default state store location, while allowing users to provide a X-Sf-Substreams-Cache-Tag header (gated by auth module) to point to /data/states/v1, and so on.

• Authentication plugin trust can now specify an exclusive list of allowed headers (all lowercase), ex: trust://?allowed=x-sf-user-id,x-sf-api-key-id,x-real-ip,x-sf-substreams-cache-tag

• The tier2 app no longer has customizable auth plugin (or any Modules), trust will always be used, so that tier can pass down its headers (e.g. X-Sf-Substreams-Cache-Tag). The tier2 instances should not be accessible publicly.

GUI changes

• Color theme is now adapted to the terminal background (fixes readability on 'light' background)

• Provided parameters are now shown in the 'Request' tab.

CLI changes

Added

• alpha init command: replace initialBlock for generated manifest based on contract creation block.

• alpha init prompt Ethereum chain. Added: Mainnet, BNB, Polygon, Goerli, Mumbai.

Fixed

• alpha init reports better progress specially when performing ABI & creation block retrieval.

• alpha init command without contracts fixed Protogen command invocation.

v1.1.8

Backend changes

Added

• Max-subrequests can now be overridden by auth header X-Sf-Substreams-Parallel-Jobs (note: if your auth plugin is 'trust', make sure that you filter out this header from public access
• Request Stats logging. When enable it will log metrics associated to a Tier1 and Tier2 request
• On request, save "substreams.partial.spkg" file to the state cache for debugging purposes.
• Manifest reader can now read 'partial' spkg files (without protobuf and metadata) with an option.

Fixed

• Fixed a bug which caused "live" blocks to be sent while the stream previously received block(s) were historic.

CLI changes

Fixed

• In GUI, module output now shows fields with default values, i.e. 0, "", false