Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move safety store to rust #982

Merged
merged 42 commits into from
Sep 13, 2024
Merged
Show file tree
Hide file tree
Changes from 19 commits
Commits
Show all changes
42 commits
Select commit Hold shift + click to select a range
a03e245
Intermediate commit: serialization, JNI, DB API.
siy Jul 11, 2024
7d7e992
Intermediate commit: migration store, safety store and address book a…
siy Jul 19, 2024
ef03a2b
Intermediate commit: all stores are working and tested
siy Jul 23, 2024
641f31d
Intermediate commit: address book implementation
siy Jul 25, 2024
e45fde8
Intermediate commit: dependency injection issues, 8 tests failing.
siy Jul 29, 2024
19524e9
Fix dependency injection in tests.
siy Jul 29, 2024
b830d17
Minor cleanup
siy Jul 29, 2024
9e55c3b
Mark each store as migrated.
siy Jul 29, 2024
34f667f
Remove unused class.
siy Jul 29, 2024
b961bf3
Update properties files and make all default paths consistent
siy Jul 29, 2024
5ddbfa2
Separation of DB-related code at Rust side (step #1)
siy Aug 7, 2024
7171b4d
Separation of DB-related code at Rust side (step #2)
siy Aug 7, 2024
1f9b18e
Separation of DB-related code at Rust side (step #3)
siy Aug 12, 2024
6fcc859
Separation of DB-related code at Rust side (step #4)
siy Aug 12, 2024
0a31180
Make separate address book, safety store and migration store.
siy Aug 13, 2024
1d988ad
Use rocks db for tests.
siy Aug 14, 2024
328039a
Move common rocks db code to node-common
siy Aug 14, 2024
5fe0fcd
Move state-manager store files back to store module.
siy Aug 14, 2024
8679a93
Move p2p files to dedicated crate.
siy Aug 14, 2024
a428d18
Address review comments.
siy Aug 15, 2024
66c44a6
Let address book and safety store hold their migration status.
siy Aug 15, 2024
fdf3a75
Remove base class from tests
siy Aug 16, 2024
a41b6c9
Fix conversion from DTO to preserve set ordering.
siy Aug 21, 2024
fe1fad4
Fix formatting
siy Aug 21, 2024
25c13ef
Reuse NodePersistenceModule in FunctionalRadixNodeModule
siy Aug 21, 2024
b99765b
Add migration tests
siy Aug 21, 2024
bcc2576
Minor cleanup
siy Aug 21, 2024
1ba5812
Address review comments
siy Aug 27, 2024
d3925ee
Fix docker-related CI part
siy Aug 27, 2024
fb4a814
Fix docker-related CI part
siy Aug 27, 2024
2eddb02
Address Sonar issues
siy Aug 28, 2024
464af48
Attempt to fix Docker builds
siy Aug 28, 2024
0ab42ea
Attempt to fix Docker builds
siy Aug 28, 2024
ffad413
Fix formatting
siy Aug 28, 2024
9930783
Add versioning to DB data format.
siy Aug 30, 2024
d9c609d
Fix serialization for the NextEpoch.
siy Sep 2, 2024
6668337
Bump curl version to available one
siy Sep 2, 2024
3056773
Bump libssl-dev version to available one
siy Sep 2, 2024
feac88f
Merge remote-tracking branch 'origin/develop' into feature/NODE-609-m…
siy Sep 3, 2024
d49e9b4
Merge branch 'develop' into feature/NODE-609-move-safety-store-to-rust
iamyulong Sep 6, 2024
a2829da
Address PR comments.
siy Sep 9, 2024
4f621f7
Bump libssl-dev version
siy Sep 9, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion cli-tools/src/main/java/com/radixdlt/shell/RadixShell.java
Original file line number Diff line number Diff line change
Expand Up @@ -178,7 +178,7 @@ public Node build() throws Exception {
dataDir = new File(Files.createTempDirectory("radix-shell-node-").toString());
}

customProperties.build().forEach((k, v) -> properties.set(k, v));
customProperties.build().forEach(properties::set);

properties.set("db.location", dataDir.toString());

Expand Down
10 changes: 10 additions & 0 deletions common/src/main/java/com/radixdlt/lang/Option.java
Original file line number Diff line number Diff line change
Expand Up @@ -218,6 +218,16 @@ default T or(T replacement) {
return fold(Functions::id, () -> replacement);
}

/**
* Return current value stored in current instance if current instance is present. If current
* instance is empty then return {@code null}.
*
* @return either value stored in current instance or {@code null} if current instance is empty
*/
default T toNullable() {
return fold(Functions::id, () -> null);
}

/**
* Return current value stored in current instance if current instance is present. If current
* instance is empty then return value returned by provided supplier. If current instance is not
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,120 @@
/* Copyright 2021 Radix Publishing Ltd incorporated in Jersey (Channel Islands).
*
* Licensed under the Radix License, Version 1.0 (the "License"); you may not use this
* file except in compliance with the License. You may obtain a copy of the License at:
*
* radixfoundation.org/licenses/LICENSE-v1
*
* The Licensor hereby grants permission for the Canonical version of the Work to be
* published, distributed and used under or by reference to the Licensor’s trademark
* Radix ® and use of any unregistered trade names, logos or get-up.
*
* The Licensor provides the Work (and each Contributor provides its Contributions) on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied,
* including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT,
* MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE.
*
* Whilst the Work is capable of being deployed, used and adopted (instantiated) to create
* a distributed ledger it is your responsibility to test and validate the code, together
* with all logic and performance of that code under all foreseeable scenarios.
*
* The Licensor does not make or purport to make and hereby excludes liability for all
* and any representation, warranty or undertaking in any form whatsoever, whether express
* or implied, to any entity or person, including any representation, warranty or
* undertaking, as to the functionality security use, value or other characteristics of
* any distributed ledger nor in respect the functioning or value of any tokens which may
* be created stored or transferred using the Work. The Licensor does not warrant that the
* Work or any use of the Work complies with any law or regulation in any territory where
* it may be implemented or used or that it will be appropriate for any specific purpose.
*
* Neither the licensor nor any current or former employees, officers, directors, partners,
* trustees, representatives, agents, advisors, contractors, or volunteers of the Licensor
* shall be liable for any direct or indirect, special, incidental, consequential or other
* losses of any kind, in tort, contract or otherwise (including but not limited to loss
* of revenue, income or profits, or loss of use or data, or loss of reputation, or loss
* of any economic or other opportunity of whatsoever nature or howsoever arising), arising
* out of or in connection with (without limitation of any use, misuse, of any ledger system
* or use made or its functionality or any performance or operation of any code or protocol
* caused by bugs or programming or logic errors or otherwise);
*
* A. any offer, purchase, holding, use, sale, exchange or transmission of any
* cryptographic keys, tokens or assets created, exchanged, stored or arising from any
* interaction with the Work;
*
* B. any failure in a transmission or loss of any token or assets keys or other digital
* artefacts due to errors in transmission;
*
* C. bugs, hacks, logic errors or faults in the Work or any communication;
*
* D. system software or apparatus including but not limited to losses caused by errors
* in holding or transmitting tokens by any third-party;
*
* E. breaches or failure of security including hacker attacks, loss or disclosure of
* password, loss of private key, unauthorised use or misuse of such passwords or keys;
*
* F. any losses including loss of anticipated savings or other benefits resulting from
* use of the Work or any changes to the Work (however implemented).
*
* You are solely responsible for; testing, validating and evaluation of all operation
* logic, functionality, security and appropriateness of using the Work for any commercial
* or non-commercial purpose and for any reproduction or redistribution by You of the
* Work. You assume all risks associated with Your use of the Work and the exercise of
* permissions under this License.
*/

package com.radixdlt.db;

import com.google.common.reflect.TypeToken;
import com.radixdlt.environment.NodeRustEnvironment;
import com.radixdlt.lang.Tuple.Tuple0;
import com.radixdlt.monitoring.Metrics;
import com.radixdlt.sbor.Natives;

public class RocksDbMigrationStore {
static {
System.loadLibrary("corerust");
}

private static native byte[] isMigrated(NodeRustEnvironment nodeRustEnvironment, byte[] payload);

private static native byte[] migrationDone(
NodeRustEnvironment nodeRustEnvironment, byte[] payload);

/**
* Stores a pointer to the rust core api server across JNI calls. In the JNI model, this is
* equivalent to the CoreApiServer "owning" the rust core api server memory. On each call into
* Rust, we map the rustCoreApiServerPointer onto a concrete implementation in Rust land, and it
* uses that to access all state and make calls.
*/
@SuppressWarnings("unused")
private final long rustCoreApiServerPointer = 0;

public static RocksDbMigrationStore create(
Metrics metrics, NodeRustEnvironment nodeRustEnvironment) {
return new RocksDbMigrationStore(metrics, nodeRustEnvironment);
}

private RocksDbMigrationStore(Metrics metrics, NodeRustEnvironment nodeRustEnvironment) {
final var timer = metrics.stateManager().nativeCall();
isMigratedFunc =
Natives.builder(nodeRustEnvironment, RocksDbMigrationStore::isMigrated)
.measure(timer.label(new Metrics.MethodId(RocksDbMigrationStore.class, "isMigrated")))
.build(new TypeToken<>() {});
migrationDoneFunc =
Natives.builder(nodeRustEnvironment, RocksDbMigrationStore::migrationDone)
.measure(
timer.label(new Metrics.MethodId(RocksDbMigrationStore.class, "migrationDone")))
.build(new TypeToken<>() {});
}

public boolean isMigrated(StoreId storeId) {
return this.isMigratedFunc.call(storeId);
}

public void migrationDone(StoreId storeId) {
this.migrationDoneFunc.call(storeId);
}

private final Natives.Call1<StoreId, Boolean> isMigratedFunc;
private final Natives.Call1<StoreId, Tuple0> migrationDoneFunc;
}
83 changes: 83 additions & 0 deletions core-rust-bridge/src/main/java/com/radixdlt/db/StoreId.java
Original file line number Diff line number Diff line change
@@ -0,0 +1,83 @@
/* Copyright 2021 Radix Publishing Ltd incorporated in Jersey (Channel Islands).
*
* Licensed under the Radix License, Version 1.0 (the "License"); you may not use this
* file except in compliance with the License. You may obtain a copy of the License at:
*
* radixfoundation.org/licenses/LICENSE-v1
*
* The Licensor hereby grants permission for the Canonical version of the Work to be
* published, distributed and used under or by reference to the Licensor’s trademark
* Radix ® and use of any unregistered trade names, logos or get-up.
*
* The Licensor provides the Work (and each Contributor provides its Contributions) on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied,
* including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT,
* MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE.
*
* Whilst the Work is capable of being deployed, used and adopted (instantiated) to create
* a distributed ledger it is your responsibility to test and validate the code, together
* with all logic and performance of that code under all foreseeable scenarios.
*
* The Licensor does not make or purport to make and hereby excludes liability for all
* and any representation, warranty or undertaking in any form whatsoever, whether express
* or implied, to any entity or person, including any representation, warranty or
* undertaking, as to the functionality security use, value or other characteristics of
* any distributed ledger nor in respect the functioning or value of any tokens which may
* be created stored or transferred using the Work. The Licensor does not warrant that the
* Work or any use of the Work complies with any law or regulation in any territory where
* it may be implemented or used or that it will be appropriate for any specific purpose.
*
* Neither the licensor nor any current or former employees, officers, directors, partners,
* trustees, representatives, agents, advisors, contractors, or volunteers of the Licensor
* shall be liable for any direct or indirect, special, incidental, consequential or other
* losses of any kind, in tort, contract or otherwise (including but not limited to loss
* of revenue, income or profits, or loss of use or data, or loss of reputation, or loss
* of any economic or other opportunity of whatsoever nature or howsoever arising), arising
* out of or in connection with (without limitation of any use, misuse, of any ledger system
* or use made or its functionality or any performance or operation of any code or protocol
* caused by bugs or programming or logic errors or otherwise);
*
* A. any offer, purchase, holding, use, sale, exchange or transmission of any
* cryptographic keys, tokens or assets created, exchanged, stored or arising from any
* interaction with the Work;
*
* B. any failure in a transmission or loss of any token or assets keys or other digital
* artefacts due to errors in transmission;
*
* C. bugs, hacks, logic errors or faults in the Work or any communication;
*
* D. system software or apparatus including but not limited to losses caused by errors
* in holding or transmitting tokens by any third-party;
*
* E. breaches or failure of security including hacker attacks, loss or disclosure of
* password, loss of private key, unauthorised use or misuse of such passwords or keys;
*
* F. any losses including loss of anticipated savings or other benefits resulting from
* use of the Work or any changes to the Work (however implemented).
*
* You are solely responsible for; testing, validating and evaluation of all operation
* logic, functionality, security and appropriateness of using the Work for any commercial
* or non-commercial purpose and for any reproduction or redistribution by You of the
* Work. You assume all risks associated with Your use of the Work and the exercise of
* permissions under this License.
*/

package com.radixdlt.db;

import com.radixdlt.sbor.codec.CodecMap;
import com.radixdlt.sbor.codec.EnumCodec;

/** Java side representation of the StoreId enum in Rust */
public sealed interface StoreId {
record AddressBookStoreId() implements StoreId {}

record SafetyStoreId() implements StoreId {}

AddressBookStoreId ADDRESS_BOOK = new AddressBookStoreId();
SafetyStoreId SAFETY_STORE = new SafetyStoreId();

static void registerCodec(CodecMap codecMap) {
codecMap.register(
StoreId.class, codecs -> EnumCodec.fromPermittedRecordSubclasses(StoreId.class, codecs));
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
/* Copyright 2021 Radix Publishing Ltd incorporated in Jersey (Channel Islands).
*
* Licensed under the Radix License, Version 1.0 (the "License"); you may not use this
* file except in compliance with the License. You may obtain a copy of the License at:
*
* radixfoundation.org/licenses/LICENSE-v1
*
* The Licensor hereby grants permission for the Canonical version of the Work to be
* published, distributed and used under or by reference to the Licensor’s trademark
* Radix ® and use of any unregistered trade names, logos or get-up.
*
* The Licensor provides the Work (and each Contributor provides its Contributions) on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied,
* including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT,
* MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE.
*
* Whilst the Work is capable of being deployed, used and adopted (instantiated) to create
* a distributed ledger it is your responsibility to test and validate the code, together
* with all logic and performance of that code under all foreseeable scenarios.
*
* The Licensor does not make or purport to make and hereby excludes liability for all
* and any representation, warranty or undertaking in any form whatsoever, whether express
* or implied, to any entity or person, including any representation, warranty or
* undertaking, as to the functionality security use, value or other characteristics of
* any distributed ledger nor in respect the functioning or value of any tokens which may
* be created stored or transferred using the Work. The Licensor does not warrant that the
* Work or any use of the Work complies with any law or regulation in any territory where
* it may be implemented or used or that it will be appropriate for any specific purpose.
*
* Neither the licensor nor any current or former employees, officers, directors, partners,
* trustees, representatives, agents, advisors, contractors, or volunteers of the Licensor
* shall be liable for any direct or indirect, special, incidental, consequential or other
* losses of any kind, in tort, contract or otherwise (including but not limited to loss
* of revenue, income or profits, or loss of use or data, or loss of reputation, or loss
* of any economic or other opportunity of whatsoever nature or howsoever arising), arising
* out of or in connection with (without limitation of any use, misuse, of any ledger system
* or use made or its functionality or any performance or operation of any code or protocol
* caused by bugs or programming or logic errors or otherwise);
*
* A. any offer, purchase, holding, use, sale, exchange or transmission of any
* cryptographic keys, tokens or assets created, exchanged, stored or arising from any
* interaction with the Work;
*
* B. any failure in a transmission or loss of any token or assets keys or other digital
* artefacts due to errors in transmission;
*
* C. bugs, hacks, logic errors or faults in the Work or any communication;
*
* D. system software or apparatus including but not limited to losses caused by errors
* in holding or transmitting tokens by any third-party;
*
* E. breaches or failure of security including hacker attacks, loss or disclosure of
* password, loss of private key, unauthorised use or misuse of such passwords or keys;
*
* F. any losses including loss of anticipated savings or other benefits resulting from
* use of the Work or any changes to the Work (however implemented).
*
* You are solely responsible for; testing, validating and evaluation of all operation
* logic, functionality, security and appropriateness of using the Work for any commercial
* or non-commercial purpose and for any reproduction or redistribution by You of the
* Work. You assume all risks associated with Your use of the Work and the exercise of
* permissions under this License.
*/

package com.radixdlt.p2p;

import com.radixdlt.lang.Option;
import com.radixdlt.sbor.codec.CodecMap;
import com.radixdlt.sbor.codec.StructCodec;
import java.util.Set;

public record AddressBookEntryDTO(
NodeIdDTO nodeId, Option<Long> bannedUntil, Set<PeerAddressEntryDTO> knownAddresses) {
public static void registerCodec(CodecMap codecMap) {
codecMap.register(
AddressBookEntryDTO.class,
codecs -> StructCodec.fromRecordComponents(AddressBookEntryDTO.class, codecs));
}
}
Loading
Loading