Add hydra as a source #823
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
paweljakubas
force-pushed
the
paweljakubas/test-framework
branch
from
a69fcb2
to
db0e8fa
Compare
- use tokio-tungstenite to communicate with hydra node
- deserialize some messages and pass data to oura
To run, first start the hydra demo:
```bash
cd hydra/demo
./prepare-devnet.sh
docker compose up -d cardano-node
./seed-devnet.sh
docker compose up -d hydra-node-{1,2,3}
```
Then, in this repositroy, run:
```bash
cargo run daemon --config examples/hydra/daemon.toml
```
Anviking
force-pushed
the
paweljakubas/test-framework
branch
from
c8bd0ec
to
bc48a28
Compare
|
this is looking really good! 🚀 |
This allows us to resume processing from a point specified in the config. Tested manually using the following steps: 1. Ensure hydra demo is running 2. `cargo run daemon --config examples/hydra/daemon.toml` 3. Read the `seq` and `head_id` of the last message printed 4. Set the following in `hydra/daemon.toml`: ``` [intersect] type = "Point" value = [ 7, "84e657e3dd5241caac75b749195f78684023583736cc08b2896290ab" ] ``` but with `7` and `84e657e3dd5241caac75b749195f78684023583736cc08b2896290ab` being the actually observed `seq` and `head_id` values. 5. Restart `oura` 6. Observe that `oura` ignores all previously processed messages
Add basic intersect logic to hydra source
With this we see output from oura like:
```json
{"event":"apply","point":{"hash":"0000000000000000000000000000000000000000000000000000000000000000","slot":1},"record":{"peer":"2","seq":1,"tag":"PeerConnected","timestamp":"2024-10-15T11:48:00.516366926Z"}}
```
Forward all `raw_json` of hydra messages
I believe the `skip_deserializing` turn messages of these types into the `Other` payload anyway, so having them here is confusing.
Drop unneeded `Optional` from `payload` field
- Use random TCP ports - Use random output files - Use goldenfile (has the practical UPDATE_GOLDENFILES=1 flag)
Tweak integration tests
Integration tests
Make intersection logic more robust
We can't know of the chain tip before we actually reach it when processing.
Drop unused metrics & correct dummy hash length
Anviking
reviewed
Nov 18, 2024
examples/.gitignore
Outdated
| scratchpad | ||
| tests/hydra/logs.txt |
There was a problem hiding this comment.
Suggested change
| tests/hydra/logs.txt |
Anviking
reviewed
Nov 18, 2024
src/sources/hydra.rs
Outdated
Comment on lines
115
to
119
| #[derive(Serialize, Deserialize, Debug, Clone)] | ||
| pub struct Snapshot { | ||
| number: u64, | ||
| confirmed_transaction_ids: Vec<String>, | ||
| } |
There was a problem hiding this comment.
Suggested change
| #[derive(Serialize, Deserialize, Debug, Clone)] | |
| pub struct Snapshot { | |
| number: u64, | |
| confirmed_transaction_ids: Vec<String>, | |
| } |
This is currently unused
Anviking
reviewed
Nov 18, 2024
src/sources/hydra.rs
Outdated
|
|
||
| #[derive(Deserialize)] | ||
| pub struct Config { | ||
| pub hydra_socket_url: String, |
There was a problem hiding this comment.
Probably this is more correct:
Suggested change
| pub hydra_socket_url: String, | |
| pub hydra_socket_path: String, |
|
hi @scarmuega ready for the review! cheers |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The PR adds hydra as a source talking through websocket opened as hydra side.
The core of the solution is pinned around:
As a result of Hydra websocket communication hydra source offers :
(a) sequence number which orders communication msgs
(b) optional hydra tail id (optional as it is established only when all parties commit)
(c) payload which determines if a given msg is either valid tx (here cbor is extracted) or of other sort (fanning out, commiting, ..etc)
(d) raw json in case consumer wants to get more than is extracted in (c)
Intersection support if added. If one wants to consume from a given slot onwards the older msg are filtered out for the consumer.
In order to test the solution the following tests were added.