[FEATURE] Add synchronization from NODE REST API on local-node, and fix error at block 1610029 - 1610030

[username1565]

Аbstract

Is your feature request related to a problem? Please describe.

Yes. wmd v0.3.0 cann't synchronize database from NODE REST API on local node,
and need to raise gRPC server there. What if I use old version of the node???
Also, synchronization is stopped at blocks 1610029 - 1610030.
See the details in this issue: #285
So, need to fix it, in the next versions of wmd.exe

Motivation and Purposes

A clear and concise description of what the problem is. Ex. I'm always frustrated when [I cann't downloading candles and trading history for all trading pairs].

** Specification**

A clear and concise description of what you want to happen.
Describe alternatives you've considered

I want to load candles and orders history for all trading pairs, to display this data in waves-lite client.
I want to do this from my local node, not from remote node, which can go offline at any time.
I want to do this up to end block, without see any errors at blocks 1610029 - 1610030.

Backwards Compatibility

Can your proposition affect any existing features?

No. Just fix this WMD.exe.

Examples and Implementation

Examples of implementation in other projects?

https://marketdata.wavesplatform.com this is working fine, but this is remote server.

[username1565]

Also, can you add the ability to skip invalid blocks or just don't parse the details of unknown transaction types, which can be added in future?
For example, type 17, type 18, type 100518, etc...

[alexeykiselev]

The old version of WMD doesn't support new types of orders and transactions that required for WMD operation.
We will support both APIs in future versions of WMD.

Have you reported an exception you encountered then starting gRPC serve to Waves devs?

[username1565]

No. I don't want to create new issue, there, because maybe, this is my mistake.
But I can write about this here.
I just follow this guide https://docs.wavesplatform.com/en/waves-node/extensions/grpc-server/
didn't find .zip-file, but download .tgz-file, unpack it into thie folder,
and try to run with that extensions:

java -cp 'waves-all-1.0.0.jar:grpc-server-1.0.0/lib/*' com.wavesplatform.Application node.conf

(default example)

after adding this string in node.conf:

waves.extensions += com.wavesplatform.api.grpc.GRPCServerExtension

I see this throw exception:

2020-04-10 11:44:03,525 ERROR [main] c.w.actor.RootActorSystem$ - Error while in
itializing actor system wavesplatform
java.lang.NoSuchMethodError: com.wavesplatform.api.common.CommonTransactionsApi.
(Lcom/wavesplatform/state/Blockchain;Lcom/wavesplatform/utx/UtxPool;Lcom/w
avesplatform/wallet/Wallet;Lscala/Function2;)V
        at com.wavesplatform.api.grpc.TransactionsApiGrpcImpl.(Transaction
sApiGrpcImpl.scala:25)
        at com.wavesplatform.api.grpc.GRPCServerExtension.startServer(GRPCServer
Extension.scala:41)
        at com.wavesplatform.api.grpc.GRPCServerExtension.start(GRPCServerExtens
ion.scala:22)
        at com.wavesplatform.Application.$anonfun$run$20(Application.scala:296)
        at com.wavesplatform.Application.$anonfun$run$20$adapted(Application.sca
la:296)
        at scala.collection.Iterator.foreach(Iterator.scala:941)
        at scala.collection.Iterator.foreach$(Iterator.scala:941)
        at scala.collection.AbstractIterator.foreach(Iterator.scala:1429)
        at scala.collection.IterableLike.foreach(IterableLike.scala:74)
        at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
        at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
        at com.wavesplatform.Application.run(Application.scala:296)
        at com.wavesplatform.Application$.$anonfun$startNode$4(Application.scala
:453)
        at com.wavesplatform.Application$.$anonfun$startNode$4$adapted(Applicati
on.scala:433)
        at com.wavesplatform.actor.RootActorSystem$.start(RootActorSystem.scala:
25)
        at com.wavesplatform.Application$.startNode(Application.scala:433)
        at com.wavesplatform.Application$.main(Application.scala:416)
        at com.wavesplatform.Application.main(Application.scala)

node v1.1.2, and grpc v1.0.2

after comment that string:

#waves.extensions += com.wavesplatform.api.grpc.GRPCServerExtension

the node startin without that throw exception.

Also, then, I did write the following strings in node.conf:

waves.grpc {
  host = localhost
  port = 6870
}

node is starting without throw exception,
but http://127.0.0.1:6870/ not opening, and wmd.exe v1.0.3 not loading blocks from this port.
Also, I did try to 6871 port, and the same result.
Now, that strings is commented.

So, gRPC server not working for node v1.1.2, and I don't want to change the version of my node,
because synchronization of node with higher versions can damage the blockchain database,
and I'll not re-synch this blockchain again, from beginning, and validate all transactions again, and again...

[alexeykiselev]

This error indicates that versions of node and gRPC server plugin are incompatible.
In your command to start node:

java -cp 'waves-all-1.0.0.jar:grpc-server-1.0.0/lib/*' com.wavesplatform.Application node.conf

According to this article, on Windows you have to use ';' as classpath delimiter, not ':'.

I see that you are starting version 1.0.0 of node and 1.0.0 of gPRC server. Not 1.0.2. Please, check twice that you are using the same versions for both. And there is no version 1.0.3 of Waves node.

By the way, I am sure you can update Waves Node from version 1.0.2 to any version up to latest 1.1.10 without rebuilding state. Make backup of state if you want to downgrade later to your current version.

Also, I did try to 6871 port, and the same result.

You can't check the availability of gRPC with a browser, please use telnet to connect to gRPC port.

Do you run private blockchain or some fork of MainNet?

[username1565]

start_v1.1.2.bat:

D:
cd "D:\coins\WAVES_FULL_NODE\NEW_NODE"
java -cp "D:\coins\WAVES_FULL_NODE\NEW_NODE\waves-all-1.1.2.jar;D:\coins\WAVES_FULL_NODE\NEW_NODE\dex-1.0.2\lib\*;D:\coins\WAVES_FULL_NODE\NEW_NODE\dex-1.0.2\grpc-server-1.1.2\lib\*" com.wavesplatform.Application node_v1.1.2_extensions_gRPC.conf
pause

Run this...

Result:

2020-04-10 17:32:31,738 ERROR [main] c.w.actor.RootActorSystem$ - Error while in
itializing actor system wavesplatform
java.lang.NoSuchMethodError: com.wavesplatform.api.common.CommonTransactionsApi.
(Lcom/wavesplatform/state/Blockchain;Lcom/wavesplatform/utx/UtxPool;Lcom/w
avesplatform/wallet/Wallet;Lscala/Function2;)V
        at com.wavesplatform.api.grpc.TransactionsApiGrpcImpl.(Transaction
sApiGrpcImpl.scala:25)
        at com.wavesplatform.api.grpc.GRPCServerExtension.startServer(GRPCServer
Extension.scala:41)
        at com.wavesplatform.api.grpc.GRPCServerExtension.start(GRPCServerExtens
ion.scala:22)
        at com.wavesplatform.Application.$anonfun$run$20(Application.scala:296)
        at com.wavesplatform.Application.$anonfun$run$20$adapted(Application.sca
la:296)
        at scala.collection.Iterator.foreach(Iterator.scala:941)
        at scala.collection.Iterator.foreach$(Iterator.scala:941)
        at scala.collection.AbstractIterator.foreach(Iterator.scala:1429)
        at scala.collection.IterableLike.foreach(IterableLike.scala:74)
        at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
        at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
        at com.wavesplatform.Application.run(Application.scala:296)
        at com.wavesplatform.Application$.$anonfun$startNode$4(Application.scala
:453)
        at com.wavesplatform.Application$.$anonfun$startNode$4$adapted(Applicati
on.scala:433)
        at com.wavesplatform.actor.RootActorSystem$.start(RootActorSystem.scala:
25)
        at com.wavesplatform.Application$.startNode(Application.scala:433)
        at com.wavesplatform.Application$.main(Application.scala:416)
        at com.wavesplatform.Application.main(Application.scala)

Then, comment in node_v1.1.2_extensions_gRPC.conf:

#waves.extensions += com.wavesplatform.api.grpc.GRPCServerExtension

and node is running without this throw exception.

By the way, I am sure you can update Waves Node from version 1.0.2
to any version up to latest 1.1.10 without rebuilding state.
Make backup of state if you want to downgrade later to your current version.

I don't want to update my old working node, and process again this slow re-synchronization.
I don't want to update up to latest versions, which don't have full backward-compatibility.
I just want to have the complete full node with matcher, to trade my WUSD,
and have the warranty that I can provide this myself,
without any dependencies from any degenerative scammers, from centralized exchange-services, where trading WUSD was been blocked by coinomat.com - initiative.
I don't want to make any backups of this large blockchain,
with old transactions of another humans and trading bots,
and I don't want to download each .jar-file by 100 MB,
and spend my limited traffic after each small update.
For me, will be better to synchronize the blockchain once in one year,
after update the node up to latest version,
but I must to be sure, that source code of my node is complete,
and not damaged, by some degenerative actions, after project death,
or while this is had stagnation, or was been in agony-stage.

Do you run private blockchain or some fork of MainNet?

My node is synchronized, and I have the latest block.
No any forks of blockchain, like for some nodes have "forked blockchain" here.
So, I have the current blockchain on the node v1.1.2, and this working without any problem.

I just host this in TOR, on the onion-domain:
Here is hosted the old waves-lite-client: http://waves2bmihndo37h.onion/
node: http://waves2bmihndo37h.onion:6869/
matcher: http://waves2bmihndo37h.onion:6886/
and old waves-block-explorer at port 3000: http://waves2bmihndo37h.onion:3000/

This links working, and can be opened in TorBrowser.

Anyway, if this project will be closed, and WavesPlatform will scam people,
and source code will be deleted, I can decompile this binaries back, into human readable code,
replace first block, generate premine here, and run my own project,
with my own DEX, like that TurtleNode fork: https://github.com/BlackTurtle123/TurtleNetwork/

Anyway, I'm interested to have on my node the complete and full working source code.

Now, I just want to load candes, and trading history, to loading this in waves-lite-client,
because there is not working candles, and trading history.
This is loading from http://marketdata.wavesplatform.com/ but this can go offline at any time.
And... I'm not sure that this service can update candles and trading history, with orders on another matchers.

[alexeykiselev]

I've updated the latest release with WMD distributives.

Please, try to use this version. I forgot about a bug that was fixed in Waves Node gRPC API and the work around was introduced with the commit.

[username1565]

@alexeykiselev, fc not found any differences, between wmd.exe v0.3.0, which I downloaded earlier, and between version of wmd.exe v0.3.0, which exists in releases.
This means there is no any updates, and this error still exists there.
I don't understand anything in the source code of GoLang-files, and cann't compile this.
Just want to raise marketdata API, and download from remote gRPC-server, the candles and trading history. And will be better to be able to download this from my local node, by using connections and requests to NODE REST API at port 6869.

UPD:
I see some wmd.exe v0.5.0. I'll test this ASAP.

UPD2:
wmd v0.5.0 - try with node:

D:\coins\WAVES_FULL_NODE\NEW_NODE\WMD\wmd_v0.5.0_Windows-64bit>wmd -db D:\coins\
WAVES_FULL_NODE\NEW_NODE\WMD\wmd_v0.5.0_Windows-64bit\data -address 0.0.0.0:6990
 -node 127.0.0.1:6869 -symbols symbols.txt -sync-interval 1 -matcher 9cpfKN9suPNvfeUNphzxXMjcnn974eme8ZhWUjaktzU5
2020-04-12T19:47:50.713+0300    INFO    Waves Market Data (WMD) version v0.5.0
2020-04-12T19:47:50.771+0300    INFO    Imported 225 of symbol substitutions
2020-04-12T19:47:50.772+0300    INFO    Last stored height: 1610029
2020-04-12T19:47:50.788+0300    INFO    Synchronization interval set to 1s
2020-04-12T19:47:51.788+0300    ERROR   Failed to synchronize with node: rpc err
or: code = Unavailable desc = all SubConns are in TransientFailure, latest conne
ction error: connection closed
2020-04-12T19:47:52.788+0300    ERROR   Failed to synchronize with node: rpc err
or: code = Unavailable desc = all SubConns are in TransientFailure, latest conne
ction error: connection closed
... and repeat it...

node:

2020-04-12 19:49:28,791 WARN  [tor.default-dispatcher-17] akka.actor.ActorSystem
Impl - Illegal request, responding with status '501 Not Implemented': Unsupporte
d HTTP method: PRI
...
2020-04-12 19:50:34,377 WARN  [ctor.default-dispatcher-3] akka.actor.ActorSystem
Impl - Illegal request, responding with status '501 Not Implemented': Unsupporte
d HTTP method: PRI

wmd v0.5.0 - try with remote gRPC-server:

D:\coins\WAVES_FULL_NODE\NEW_NODE\WMD\wmd_v0.5.0_Windows-64bit>wmd -db D:\coins\
WAVES_FULL_NODE\NEW_NODE\WMD\wmd_v0.5.0_Windows-64bit\data -address 0.0.0.0:6990
 -node mainnet-aws-fr-1.wavesnodes.com:6871 -symbols symbols.txt -sync-interval
1 -matcher 9cpfKN9suPNvfeUNphzxXMjcnn974eme8ZhWUjaktzU5
2020-04-12T19:51:23.314+0300    INFO    Waves Market Data (WMD) version v0.5.0
2020-04-12T19:51:23.370+0300    INFO    Imported 225 of symbol substitutions
2020-04-12T19:51:23.370+0300    INFO    Last stored height: 1610029
2020-04-12T19:51:23.389+0300    INFO    Synchronization interval set to 1s
2020-04-12T19:51:24.424+0300    INFO    Local height 1610029, node height 201466
5
2020-04-12T19:51:25.185+0300    INFO    Synchronizing 404636 blocks starting fro
m height 1610030
2020-04-12T19:51:25.256+0300    ERROR   Failed to apply blocks: rpc error: code
= Internal desc = grpc: failed to unmarshal the received message unexpected EOF
2020-04-12T19:51:25.424+0300    INFO    Local height 1610029, node height 201466
5
2020-04-12T19:51:26.173+0300    INFO    Synchronizing 404636 blocks starting fro
m height 1610030
2020-04-12T19:51:26.211+0300    ERROR   Failed to apply blocks: rpc error: code
= Internal desc = grpc: failed to unmarshal the received message unexpected EOF
... and repeat this...

[alexeykiselev]

It's here https://github.com/wavesplatform/gowaves/releases/tag/v0.5.0

[username1565]

@alexeykiselev Yeap, I'd tested wmd v0.5.0 and you can see log in prevous message.
There is not fixed those errors.