Releases: graphprotocol/indexer
v0.19.2
What's Changed
- Upgrade @graphprotocol/cost-model, improve publish setup by @fordN in #412
- indexer-native: Install uses precompiled binaries hosted on GitHub releases by @fordN in #414
- indexer-native: Declare support for ARM in package.json by @fordN in #415
Full Changelog: v0.19.1...v0.19.2
NPM packages
https://www.npmjs.com/package/@graphprotocol/indexer-agent/v/0.19.2
https://www.npmjs.com/package/@graphprotocol/indexer-cli/v/0.19.2
https://www.npmjs.com/package/@graphprotocol/indexer-common/v/0.19.2
https://www.npmjs.com/package/@graphprotocol/indexer-native/v/0.19.2
https://www.npmjs.com/package/@graphprotocol/indexer-service/v/0.19.2
v0.19.0
What's Changed
- indexer-service: Rate limit indexer-service endpoints by @fordN in #336
- Upgrade dependencies by @fordN in #338
- Indexer agent logger improvement by @hopeyen in #333
- *: Upgrade yargs dependencies and bump docker node image versions to LTS by @fordN in #345
- Add option to allocate on network subgraph by @Theodus in #339
- Upgrade node runtime image versions by @fordN in #346
- indexer-service: Add /subgraphs/health/:deployment route by @Jannis in #348
- indexer-agent, indexer-common: Add live metric for operator ETH balance by @Jannis in #354
- indexer-common: Update cost-model dependency to fix the build by @Jannis in #355
- Fix indexing status query used for health endpoint by @Jannis in #357
- indexer-cli,indexer-common: Setup CLI test harness, create initial test suites by @fordN in #358
- Indexing rules by subgraph id by @fordN in #329
- indexer-agent: Support deployment id rules using either id format by @fordN in #361
- indexer-cli: offchain commands by @hopeyen in #350
- build(deps): bump follow-redirects from 1.14.6 to 1.14.7 by @dependabot in #353
- Indexer-agent, indexer-cli: indexing rules applied instantly by @hopeyen in #352
- indexer-agent: Batched voucher redemptions by @chriswessels in #349
- indexer agent & cli - max lifetime options by @hopeyen in #343
- indexer-agent reject unsupported subgraph unless bypassed by @hopeyen in #362
- Rule to set automatic renewal for allocations by @hopeyen in #366
- docs: Update graph-node to 0.25.1 by @otaviopace in #368
- Updates to reflect current parameters by @Aderks in #365
- Use signalledTokens instead of signalAmount for minSignal threshold by @Jannis in #367
- Use AllocationExchange contract from common-ts by @Theodus in #359
- indexer-agent: Fix subgraph identifier migration by @fordN in #369
New Contributors
- @hopeyen made their first contribution in #333
- @otaviopace made their first contribution in #368
- @Aderks made their first contribution in #365
Full Changelog: v0.18.6...v0.19.0
v0.18.6
NPM
npm install -g @graphprotocol/[email protected]
npm install -g @graphprotocol/[email protected]
npm install -g @graphprotocol/[email protected]
Note: Installing these packages requires Rust and it's package manager cargo
to be installed so that the cost model module can be built. If you already have Rust installed, make sure you're on the latest stable release. Installing the above packages may also require passing --unsafe-perm=true
to the above install commands.
Docker
docker pull ghcr.io/graphprotocol/indexer-agent:v0.18.6
docker pull ghcr.io/graphprotocol/indexer-service:v0.18.6
Changes (Overview)
The most notable change included in this release is the addition of support for type 02 transactions as specified in EIP-1559.
The agent will now send type 02 transactions if supported by the client. Parallel allocations are being deprecated and will
automatically be removed by the agent now if found, in future releases all vestiges of parallel allocation management
will be removed. Another valuable update to the indexer-agent bring significant gas efficiency gains by batching claim transactions.
The indexer-cli has received some subtle, but important, updates to the status
command outputs. 2 tables have been added
to the status
output showing the indexer's 1. active allocations and 2. subgraph deployments.
Configuration options updates with this release (all optional):
Removed
Environment Variable | CLI Option |
---|---|
INDEXER_AGENT_ALLOCATION_CLAIM_THRESHOLD |
--allocation-claim-threshold |
Added
Environment Variable | CLI Option | Recommended Value |
---|---|---|
INDEXER_AGENT_REBATE_CLAIM_THRESHOLD |
--rebate-claim-threshold |
250 |
INDEXER_AGENT_REBATE_CLAIM_BATCH_THRESHOLD |
--rebate-claim-batch-threshold |
1000 |
Changes (Per Component)
Indexer Agent
Added
- Support type 0x2 transactions (EIP-1559)
- Close all parallel allocations and only renew a single allocation per deployment
- Include an additional state for indexing dispute monitoring, references_unavailable, for the case where the indexer does not have a reference PoI available. Typically this case occurs when an indexer's deployment is not synced far enough
Changed
- Deprecate the parallel allocations feature
- Update Ethers dependencies
- Keep deployment indexing for 1 day after allocation close
- Reducing transaction logging redundancy to improve readability
- Bump priority gas fee on gas price too low retry
- Simplify transaction retry logic on nonce collisions; agent now introduces a delay and returns to reconciliation step to re-evaluate
Fixed
- Only add prefix to voucher address if not present
- Bump type 2 transaction gas fee configs on retry after timeout
- Update max-transaction-attempts default to unlimited to avoid orphaned transactions
- Treat deployments assigned to node = null or undefined as removed, so they are filtered out of the activeDeployments array
- Fix delete query fee vouchers logic, so they are actually removed after the expiration time
- Update tests to use the latest version of the Indexer class constructor
Indexer Service
Added
- Support unattestable query responses
Changed
- Broaden scope of allocations eligible for receiving traffic to include those recently closed
- Update Ethers dependencies
Fixed
- Use ENTRYPOINT instead of CMD in dockerfile so users can specify command line arguments in docker run command
Indexer CLI
Added
- Show the status of indexer's active allocations in
status
command output - Show the status of indexer's subgraph deployments in
status
command output
Changed
- Update Ethers dependencies
Indexer Common
Changed
- Update Ethers dependencies
Indexer Native
Fixed
- Fix build for Node v16 by upgrading neon-sys dependency.
v0.17.0
NPM
npm install -g @graphprotocol/[email protected]
npm install -g @graphprotocol/[email protected]
npm install -g @graphprotocol/[email protected]
Note: Installing these packages requires Rust and it's package manager cargo
to be installed so that the cost model module can be built. If you already have Rust installed, make sure you're on the latest stable release. Installing the above packages may also require passing --unsafe-perm=true
to the above install commands.
Docker
docker pull ghcr.io/graphprotocol/indexer-agent:v0.17.0
docker pull ghcr.io/graphprotocol/indexer-service:v0.17.0
Changes (Overview)
This release primarily contains updates to the indexer agent to improve efficiency. The process of reallocating towards a subgraph deployment has been updated to use closeAndAllocate(), so it requires one transaction instead of two (close then allocate). Handling of high gas price scenarios has also been improved; when estimated gas prices exceed the threshold the agent will now enter a holding pattern where it waits for gas prices to return to below the gas-price-max
before continuing with transaction execution. Along with the updated behavior, the gas-price-max
startup argument has been updated for ease of use to be denominated in gwei rather than wei.
There were a few additional minor improvements to indexer agent:
- improved handling of empty auth fields for the Ethereum endpoint,
- more clear log messages from subgraph query errors.
Some networking optimizations have been included for both the indexer service and indexer agent:
- both recommended k8s setups have both been updated to use a
ClusterIP
service instead ofLoadBalancer
since theLoadBalancer
services have been shown to introduce more latency and their extra features aren't required here, /network
execution has been optimized by switching to a simpler HTTP client (doesn't need to be GraphQL aware).
As promised in the last release, this release includes an update to make the /network
endpoint optional and introduces a dedicated (optional) auth token for /network
requests.
New configuration options with this release (all optional):
Environment Variable | CLI Option | Recommended Value |
---|---|---|
INDEXER_AGENT_NETWORK_SUBGRAPH_AUTH_TOKEN |
--network-subgraph-auth-token |
<Secret bearer token for network queries> |
INDEXER_AGENT_SERVE_NETWORK_SUBGRAPH |
--serve-network-subgraph |
true |
INDEXER_AGENT_VOUCHER_EXPIRATION |
--voucher-expiration |
2160 |
Changes (Per Component)
Indexer Agent
Added
- Reallocate to subgraph deployment in one transaction using closeAndAllocate.
- Wait for gas prices below the
gas-price-max
(gwei) before proceeding with transaction execution. - Add --voucher-expiration to configure the time (in seconds) after which a voucher with too few query fees is permanently deleted. Until then, keep retrying to redeem it to give the indexer a chance to adjust their --allocation-claim-threshold.
Changed
- Show subgraph query error reason in logs.
- Use
ClusterIP
in the k8s indexer agent service rather than aLoadBalalancer
sinceLoadBalancer
services have been
shown to introduce extra latency. - Use undefined for auth user and password if none specified.
- Update
gas-price-max
units to be gwei instead of wei for improved human readability. - Default
gas-price-max
changed from 20 gwei to 50 gwei.
Fixed
- Improve robustness of subgraph deployments query by querying batches of only 10 deployments at a time.
Indexer Service
Changed
- Make the
/network
endpoint optional by default and introduce a dedicated (optional) auth token for/network
requests. - Optimize
/network
execution by switching to a simpler HTTP client (doesn't need to be GraphQL aware). - Use
ClusterIP
in the k8s indexer service service rather than aLoadBalalancer
sinceLoadBalancer
services have been
shown to introduce extra latency.
Indexer CLI
Fixed
- Remove vestigial check for
channel-messages-inbox
endpoint.
Indexer Common
Changed
- Optimize
/network
execution by switching to a simpler HTTP client (doesn't need to be GraphQL aware).
Indexer Native
Fixed
- Fix build for Node v16 by upgrading neon-sys dependency.
v0.16.0
NPM
npm install -g @graphprotocol/[email protected]
npm install -g @graphprotocol/[email protected]
npm install -g @graphprotocol/[email protected]
Note: Installing these packages requires Rust and it's package manager cargo
to be installed so that the cost model module can be built. If you already have Rust installed, make sure you're on the latest stable release. Installing the above packages may also require passing --unsafe-perm=true
to the above install commands.
Docker
docker pull ghcr.io/graphprotocol/indexer-agent:v0.16.0
docker pull ghcr.io/graphprotocol/indexer-service:v0.16.0
Changes (Overview)
This is a minor release that introduces smaller changes, mostly to improve indexer agent transactions:
- The revert reason of transactions is now detected to avoid retrying transactions that have no chance of succeeding.
- The default value for
INDEXER_AGENT_TRANSACTION_ATTEMPTS
/--transaction-attempts
has been reduced from 5 to 2. - The
INDEXER_AGENT_ALLOCATION_CLAIM_THRESHOLD
/--allocation-claim-threshold
option has been extended to not only apply to claim transactions but also to redeeming query fee vouchers. This can help avoiding to collect query fees that are lower than the actual transaction cost.
Another change is that indexer service and indexer agent now both use the same network subgraph query logic, i.e. first trying a local deployment (if specified), otherwise falling back to querying the network subgraph via an (often external) endpoint. This means it is no longer necessary to point the indexer service at indexer agent. Instead, it is recommended to configure them both in the same way.
For mainnet, the recommended values are:
Environment Variable | CLI Option | Value |
---|---|---|
INDEXER_AGENT_NETWORK_SUBGRAPH_DEPLOYMENT |
--network-subgraph-deployment |
QmaUBw7sr8pBrwNgz6JkbmmGoUU7KJSaeRvCaY3NPDbQ7A |
INDEXER_AGENT_NETWORK_SUBGRAPH_ENDPOINT |
--network-subgraph-endpoint |
https://gateway.thegraph.com/network |
INDEXER_SERVICE_NETWORK_SUBGRAPH_DEPLOYMENT |
--network-subgraph-deployment |
QmaUBw7sr8pBrwNgz6JkbmmGoUU7KJSaeRvCaY3NPDbQ7A |
INDEXER_SERVICE_NETWORK_SUBGRAPH_ENDPOINT |
--network-subgraph-endpoint |
https://gateway.thegraph.com/network |
This release also introduces a /network
route in the indexer service. This, by default, is protected via the INDEXER_SERVICE_FREE_QUERY_AUTH_TOKEN
/ --free-query-auth-token
. It is safe, even advised, to keep this endpoint internal.
Addendum: It was planned to make the /network
endpoint disabled by default. This will happen in a follow-up bugfix release.
Changes (Per Component)
Indexer Agent
Changed
- Default to only one transaction retry attempt to reduce gas usage
- Apply allocation claim threshold to redeeming query fee vouchers as well
- Add the same network subgraph fallback logic as the agent uses
Fixed
- Improve handling of reverted transactions by detecting the revert reason
Indexer CLI
No changes
Indexer Common
Indexer Native
No changes
Indexer Service
Added
- Add auth-protected
/network
endpoint, disabled by default
v0.15.1
NPM
npm install -g @graphprotocol/[email protected]
npm install -g @graphprotocol/[email protected]
npm install -g @graphprotocol/[email protected]
Note: Installing these packages requires Rust and it's package manager cargo
to be installed so that the cost model module can be built. If you already have Rust installed, make sure you're on the latest stable release. Installing the above packages may also require passing --unsafe-perm=true
to the above install commands.
Docker
docker pull ghcr.io/graphprotocol/indexer-agent:v0.15.1
docker pull ghcr.io/graphprotocol/indexer-service:v0.15.1
Changes (Overview)
This is a bugfix release that fixes an issue where receipts for allocations that were created before updating to 0.15 could not be synced to the database, making it impossible to collect query fees for these allocations.
Changes (Per Component)
Indexer Agent
Fixed
- Ensure allocation summaries exist for allocations created before v0.15.0 or outside the indexer agent
Indexer CLI
No changes
Indexer Common
Added
- Add
ensureAllocationSummary
utility for agent and service to use to create allocation summaries for receipts
Indexer Native
No changes
Indexer Service
Changed
- Switch to
ensureAllocationSummary
utility from@graph-protocol/indexer-common
v0.15.0
NPM
npm install -g @graphprotocol/[email protected]
npm install -g @graphprotocol/[email protected]
npm install -g @graphprotocol/[email protected]
Note: Installing these packages requires Rust and it's package manager cargo
to be installed so that the cost model module can be built. Installing the above packages may require passing --unsafe-perm=true
to the above install commands.
Docker
docker pull ghcr.io/graphprotocol/indexer-agent:v0.15.0
docker pull ghcr.io/graphprotocol/indexer-service:v0.15.0
Changes (Overview)
Launch of Scalar on Mainnet
The main change in this release is the replacement of State Channels with Scalar (using allocation-base receipts for now). Scalar enables highly scalable micro-transactions for query fees.
At a high level this means:
- Along with every query, indexers now receive an updated Scalar receipt via the
Scalar-Receipt
header. - Receipts are collected via a
/collect-receipts
endpoint on the gateways. This will convert them into a signed (query fee) voucher that can then be redeemed via a newAllocationExchange
contract on chain.
With this release, the indexer infrastructure changes as follows:
- A new
INDEXER_AGENT_COLLECT_RECEIPTS_ENDPOINT
/--collect-receipts-endpoint
configuration option is added to Indexer Agent to configure the endpoint that converts bundles of receipts into query fee vouchers. - A new
INDEXER_SERVICE_CLIENT_SIGNER_ADDRESS
/--client-signer-address
configuration option is added to Indexer Service to configure the signer of the incoming receipts. For now, this limits indexers to being able to serve a single client, which is considered a temporary measure until the allocation-based receipts are replaced with receipts based on Vector transfers.
Network Subgraph Indexing
So far, indexers had to rely on https://gateway.network.thegraph.com/network
to provide network subgraph data like allocations, subgraph deployments and more. With this release, indexers can index this network subgraph themselves and still fall back to https://gateway.network.thegraph.com/network
if this subgraph hasn't fully synced yet or has failed locally.
In order to achieve this, the following infrastructure changes need to be applied:
- Indexer agent now exports an HTTP server at port 8002. This needs to be made available to the indexer service.
- Indexer agent needs to have the following configuration set:
INDEXER_AGENT_NETWORK_SUBGRAPH_ENDPOINT=https://gateway.network.thegraph.com/network
(alternative:--network-subgraph-endpoint
)INDEXER_AGENT_NETWORK_SUBGRAPH_DEPLOYMENT=QmaUBw7sr8pBrwNgz6JkbmmGoUU7KJSaeRvCaY3NPDbQ7A
(alternative:--network-subgraph-deployment
)
- Indexer service needs to have the following configuration set:
INDEXER_SERVICE_NETWORK_SUBGRAPH_ENDPOINT=http://indexer-agent:8002/network
(alternative:--network-subgraph-endpoint
)
This will make the agent index the network subgraph itself and serve it at :8002/network
by default (this can be changed with --syncing-port
), falling back to the gateway endpoint if there are any problems with the network subgraph. The service will then query the network subgraph through the agent.
Changes (Per Component)
Indexer Agent
Fixed
- Make receipts table renaming migration more robust
Added
- Allow network subgraph endpoint to be used as a fallback for the local network subgraph deployment
- Add allocation exchange contract address for mainnet
Changed
- Rename query fee related fields in database models
Indexer CLI
Changed
- Rename query fee related fields in database models
Indexer Common
Fixed
- Fix bug that could cause
AsyncCache
to fail perpetually
Indexer Native
No changes
Indexer Service
Changed
- Rename query fee related fields in database models
- Only sync database models in the agent to avoid race conditions
- Rename X-Graph-Payment header to Scalar-Receipt
v0.14.0
NPM
npm install -g @graphprotocol/[email protected]
npm install -g @graphprotocol/[email protected]
npm install -g @graphprotocol/[email protected]
Note: Installing these packages requires Rust and it's package manager cargo
to be installed so that the cost model module can be built. Installing the above packages may require passing --unsafe-perm=true
to the above install commands.
Docker
docker pull ghcr.io/graphprotocol/indexer-agent:v0.14.0
docker pull ghcr.io/graphprotocol/indexer-service:v0.14.0
Changes (Overview)
The main change in this release is the (temporary) replacement of Vector as the base layer of Scalar with allocation-base receipts. We will continue working on the Vector integration and plan to transition to Vector as soon as possible, but we first wanted to unblock our launch partners going live on mainnet.
At a high level this means:
- Scalar receipts will, for now, be created against allocations, not Vector transfers.
- Indexers will (for the moment) not need to run a Vector node.
- Receipts are collected via a
/collect-receipts
endpoint on the gateways. This will convert them into a signed (query fee) voucher that can then be redeemed via a newAllocationExchange
contract.
With this release, the indexer infrastructure changes as follows:
- The Vector node is removed.
- A new
INDEXER_AGENT_COLLECT_RECEIPTS_ENDPOINT
/--collect-receipts-endpoint
configuration option is added to Indexer Agent to configure the endpoint that converts bundles of receipts into query fee vouchers. - A new
INDEXER_SERVICE_CLIENT_SIGNER_ADDRESS
/--client-signer-address
configuration option is added to Indexer Service to configure the signer of the incoming receipts. For now, this limits indexers to being able to serve a single client. As a temporary change, this feels acceptable. Transitioning back to Vector will remove this limitation again.
Changes (Per Component)
Indexer Agent
Added
- Store allocations for which the POI has been checked
- Add subgraph deployment ID to POI disputes
- Add allocation-based receipts and query fee vouchers as an alternative to Vector. This is the default for now. Vector support can still be enabled with
--use-vector
.
Fixed
- Handle 'nonce has already been used' transaction failures
Indexer CLI
Added
- Add subgraph deployment ID to POI disputes
Indexer Common
Added
- Add subgraph deployment ID to POI disputes
- Add async cache from indexer-service
- Add database models for allocation-based receipts and query fee vouchers
Indexer Native
Changed
- Use verify instead of recover for signatures when possible
- Build native module in release mode for better performance
Fixed
- Fix logging native signers usefully
Indexer Service
Added
- Add allocation-based receipts and query fee vouchers as an alternative to Vector. This is the default for now. Vector support can still be enabled with
--use-vector
.
v0.13.0
NPM
npm install -g @graphprotocol/[email protected]
npm install -g @graphprotocol/[email protected]
npm install -g @graphprotocol/[email protected]
Note: Installing these packages requires Rust and it's package manager cargo
to be installed so that the cost model module can be built. Installing the above packages may require passing --unsafe-perm=true
to the above install commands.
Docker
docker pull ghcr.io/graphprotocol/indexer-agent:v0.13.0
docker pull ghcr.io/graphprotocol/indexer-service:v0.13.0
docker pull connextproject/vector_node:0.2.1
Changes (Overview)
The main change in this release is the addition of the internal @graphprotocol/indexer-native
package that speeds up the creation and signing of attestations as well verification of incoming receipts (for query payments) by roughly 3x.
Changes (Per Component)
@graphprotocol/indexer-agent
Changed
- Update to latest common-ts
Fixed
- Correctly determine the previous epoch when generating POIs
@graphprotocol/indexer-cli
Changed
- Update to latest common-ts
@graphprotocol/indexer-common
Changed
- Update to latest common-ts
@graphprotocol/indexer-native
Added
- Add package for native attestation signing and receipt signature verification
@graphprotocol/indexer-service
Added
- Add
--allocation-syncing-interval
option (default: 120s, previous 10s) - Use
indexer-native
package for native attestation signing and signature verification
Changed
- Update to latest common-ts
v0.12.0
NPM
npm install -g @graphprotocol/[email protected]
npm install -g @graphprotocol/[email protected]
npm install -g @graphprotocol/[email protected]
Note: Installing these packages requires Rust and it's package manager cargo
to be installed so that the cost model module can be built. Installing the above packages may require passing --unsafe-perm=true
to the above install commands.
Docker
docker pull ghcr.io/graphprotocol/indexer-agent:v0.12.0
docker pull ghcr.io/graphprotocol/indexer-service:v0.12.0
docker pull connextproject/vector_node:0.2.1
Changes (Overview)
The main change in this release is the update to Vector 0.2.1. This requires updating the Vector node itself but also involves a change in the Vector config, primarily the addition of authUrl
and natsUrl
:
{
"adminToken": "<secret token of your choice, keep to yourself>",
"chainProviders": {
"4": "<some rinkeby Ethereum node/provider"
},
"nodeUrl": "<internal URL of the Vector node in your infrastructure>",
"logLevel": "info",
"natsUrl": "nats://nats1.connext.provide.network:4222,nats://nats2.connext.provide.network:4222,nats://nats3.connext.provide.network:4222",
"authUrl": "https://messaging.connext.network",
"messagingUrl": "https://messaging.connext.network",
"production": true,
"baseGasSubsidyPercentage": 0,
"allowedSwaps": [],
"skipCheckIn": true,
"mnemonic": "<the same as INDEXER_AGENT_MNEMONIC>"
}
Changes (Per Component)
Indexer Agent
Changed
- Update common-ts and ethers
Indexer CLI
Changed
- Update common-ts and ethers
Indexer Common
Changed
- Update common-ts, vector and ethers
Indexer Service
Changed
- Update common-ts, vector and ethers