sauron: Add mutinynet support #584

ca-ruz · 2024-09-03T00:55:09Z

Adds mutinynet support to sauron. Mutinynet.com uses the mempool.space REST API, which has a slightly different spec from blockstream's Esplora so there is new code to handle that.

We pinned pyln-client to <=24.5 because version 24.8 has some extra requirements that sauron does not meet, we should probably fix this but for now pinning 24.5 should work fine.

sauron/sauron.py

sip-21 · 2024-09-18T22:56:31Z

You need to pin the version of pyln-client in requirements.txt as
pyln-client==24.5 as otherwise a newer version gets installed which causes this error when starting the plugin:
lightningd: --sauron-api-endpoint=https://mutinynet.com/api/: unknown option

sip-21 · 2024-09-18T23:12:20Z

You might want to add some instructions in the README for how to start sauron with Mutinynet. Below command worked for me once I had pyln-client==24.5 installed (see comment above):
lightningd --signet --disable-plugin bcli --plugin $PWD/sauron.py --sauron-api-endpoint https://mutinynet.com/api/

sauron/sauron.py

chrisguida

See comments

sauron/sauron.py

sauron/requirements.txt

chrisguida · 2024-10-01T02:09:26Z

Btw @sip-21 noticed that this PR breaks mainnet on blockstream.info. Can you look into this @ca-ruz @iorch ?

ca-ruz · 2024-10-02T23:45:08Z

@sip-21 Can you please give me more details about the tests you wrote and what were you trying to do when Sauron failed? I am trying to replicate it to fix it. Thanks!

sip-21 · 2024-10-03T23:12:11Z

@ca-ruz The failing GitHub actions/tests are here: https://github.com/sip-21/plugins/actions/runs/11116007452/job/30885476517?pr=15

[gw0] [ 20%] PASSED test_sauron.py::test_rpc_getchaininfo 
[gw1] [ 40%] ERROR test_sauron.py::test_rpc_sendrawtransaction_invalid 
[gw2] [ 60%] ERROR test_sauron.py::test_rpc_getrawblockbyheight 
[gw3] [ 80%] ERROR test_sauron.py::test_rpc_getutxout 
[gw1] [ 80%] ERROR test_sauron.py::test_rpc_sendrawtransaction_invalid 
[gw2] [ 80%] ERROR test_sauron.py::test_rpc_getrawblockbyheight 
[gw4] [100%] ERROR test_sauron.py::test_rpc_estimatefees 
[gw3] [100%] ERROR test_sauron.py::test_rpc_getutxout 
[gw4] [100%] ERROR test_sauron.py::test_rpc_estimatefees

The tests run fine together locally, but in GitHub action lightningd doesn't seem to not shutdown properly after a test. This is why in the logs you then see

  lightningd-0 2024-10-01T00:02:01.508Z **BROKEN** connectd: Failed to listen on socket 127.0.0.1:9735: Address already in use
  Time-out: can't find [re.compile('Server started with public key')] in logs

lightningd seems to randomly be able to shutdown properly, though, and two subsequent tests pass as in this case: https://github.com/sip-21/plugins/actions/runs/11089120680/job/30809842202

ca-ruz · 2024-10-04T01:57:10Z

@sip-21 Thanks!! I recreated the bug and now it's fixed! Can you please run the tests again and let me know?

chrisguida · 2024-10-15T00:34:22Z

Waiting on @sip-21's PR here ca-ruz#2 (comment)

Co-authored-by: sip21 <[email protected]> Co-authored-by: iorch <[email protected]>

chrisguida · 2024-10-24T21:27:36Z

.ci/test.py

@@ -241,13 +241,14 @@ def run_one(p: Plugin, workflow: str) -> bool:

    logging.info(f"Virtualenv at {vpath}")

+    num_workers = 1 if p.name == "sauron" else 5


@sip-21 didn't you fix the concurrency? We can get rid of this I think?

If removing this breaks the tests, we should add the changes to this file back, but under the other commit (the one that adds the tests).

Removed these lines from the code. Tests are all failing now.

chrisguida · 2024-10-24T22:16:27Z

sauron/requirements.txt

@@ -1,2 +1,2 @@
-pyln-client>=23.2
+pyln-client>=23.2,<=24.5


We should create an issue to fix this

Created Issue: #618.

chrisguida

We're almost there! Just a couple of minor issues.

chrisguida · 2024-10-24T22:59:19Z

@sip-21

After adding back concurrency, the tests are failing because they're trying to listen on port 9735... that actually doesn't seem necessary. Can you fix the tests not to require listening on port 9735? (or maybe just listen on a random port)

cdecker · 2024-10-25T14:28:34Z

sauron/tests/test_sauron_esplora_bitcoin.py

+
+        self.daemon = LightningD(lightning_dir, None)  # noqa: F405
+        options = {
+            "disable-plugin": "bcli",


I don't think you need to override this class at all. Just set the options appropriately when using the NodeFactory.get_node(). That'll obviate the need for the subclass and the fixtures.

cdecker · 2024-10-25T14:32:53Z

Please don't override the fixtures and the node class unless you absolutely need to. options passed to NodeFactory.gey_node() is where you can disable one and enable the other. Furthermore, the tests differ only in the backend used, so you could use pytest's parametrized Tests to reuse the implementation but get the backend as a parameter (they'd run once for each backend then). Finally, you are doing format checking which is great, but we could go one further and actually use the JSON schemas from CLN to ensure all fields are present and types match up.

chrisguida · 2024-10-26T00:53:40Z

@cdecker do you have an example of any pytests that use the JSON schema from CLN to check the fields and types? That sounds amazing!

cdecker · 2024-10-31T09:43:16Z

It's done automatically by instrumenting the pyln-client library to load and check the schemas. Anything going through pyln-client in a test environment should be checked afaik. I don't think notifications and hooks are checked yet (there are no schemas yet), but that'd be an amazing extension covering the docs vs implementation too

chrisguida reviewed Sep 4, 2024

View reviewed changes