Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move safety store to rust #982

Merged
merged 42 commits into from
Sep 13, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
42 commits
Select commit Hold shift + click to select a range
a03e245
Intermediate commit: serialization, JNI, DB API.
siy Jul 11, 2024
7d7e992
Intermediate commit: migration store, safety store and address book a…
siy Jul 19, 2024
ef03a2b
Intermediate commit: all stores are working and tested
siy Jul 23, 2024
641f31d
Intermediate commit: address book implementation
siy Jul 25, 2024
e45fde8
Intermediate commit: dependency injection issues, 8 tests failing.
siy Jul 29, 2024
19524e9
Fix dependency injection in tests.
siy Jul 29, 2024
b830d17
Minor cleanup
siy Jul 29, 2024
9e55c3b
Mark each store as migrated.
siy Jul 29, 2024
34f667f
Remove unused class.
siy Jul 29, 2024
b961bf3
Update properties files and make all default paths consistent
siy Jul 29, 2024
5ddbfa2
Separation of DB-related code at Rust side (step #1)
siy Aug 7, 2024
7171b4d
Separation of DB-related code at Rust side (step #2)
siy Aug 7, 2024
1f9b18e
Separation of DB-related code at Rust side (step #3)
siy Aug 12, 2024
6fcc859
Separation of DB-related code at Rust side (step #4)
siy Aug 12, 2024
0a31180
Make separate address book, safety store and migration store.
siy Aug 13, 2024
1d988ad
Use rocks db for tests.
siy Aug 14, 2024
328039a
Move common rocks db code to node-common
siy Aug 14, 2024
5fe0fcd
Move state-manager store files back to store module.
siy Aug 14, 2024
8679a93
Move p2p files to dedicated crate.
siy Aug 14, 2024
a428d18
Address review comments.
siy Aug 15, 2024
66c44a6
Let address book and safety store hold their migration status.
siy Aug 15, 2024
fdf3a75
Remove base class from tests
siy Aug 16, 2024
a41b6c9
Fix conversion from DTO to preserve set ordering.
siy Aug 21, 2024
fe1fad4
Fix formatting
siy Aug 21, 2024
25c13ef
Reuse NodePersistenceModule in FunctionalRadixNodeModule
siy Aug 21, 2024
b99765b
Add migration tests
siy Aug 21, 2024
bcc2576
Minor cleanup
siy Aug 21, 2024
1ba5812
Address review comments
siy Aug 27, 2024
d3925ee
Fix docker-related CI part
siy Aug 27, 2024
fb4a814
Fix docker-related CI part
siy Aug 27, 2024
2eddb02
Address Sonar issues
siy Aug 28, 2024
464af48
Attempt to fix Docker builds
siy Aug 28, 2024
0ab42ea
Attempt to fix Docker builds
siy Aug 28, 2024
ffad413
Fix formatting
siy Aug 28, 2024
9930783
Add versioning to DB data format.
siy Aug 30, 2024
d9c609d
Fix serialization for the NextEpoch.
siy Sep 2, 2024
6668337
Bump curl version to available one
siy Sep 2, 2024
3056773
Bump libssl-dev version to available one
siy Sep 2, 2024
feac88f
Merge remote-tracking branch 'origin/develop' into feature/NODE-609-m…
siy Sep 3, 2024
d49e9b4
Merge branch 'develop' into feature/NODE-609-move-safety-store-to-rust
iamyulong Sep 6, 2024
a2829da
Address PR comments.
siy Sep 9, 2024
4f621f7
Bump libssl-dev version
siy Sep 9, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 5 additions & 3 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ concurrency:

on:
pull_request:
# Runs on all PRs
# Runs on all PRs
push:
branches:
- develop
Expand Down Expand Up @@ -208,7 +208,8 @@ jobs:
restore-keys: ${{ runner.os }}-gradle
- name: Run steady-state integration tests
env:
RADIXDLT_LOG_LEVEL: warn
# Might be set to warn for debugging purposes. Warning, log file will be huge.
RADIXDLT_LOG_LEVEL: error
run: ./gradlew clean runSteadyStateIntegrationTests --info --refresh-dependencies
targeted-integration:
name: Targeted integration tests
Expand All @@ -228,7 +229,8 @@ jobs:
restore-keys: ${{ runner.os }}-gradle
- name: Run targeted integration tests
env:
RADIXDLT_LOG_LEVEL: warn
# Might be set to warn for debugging purposes. Warning, log file will be huge.
RADIXDLT_LOG_LEVEL: error
run: ./gradlew clean runTargetedIntegrationTests --info --refresh-dependencies --parallel
cross-xwin:
name: Cross compile to Windows
Expand Down
17 changes: 10 additions & 7 deletions Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -50,13 +50,13 @@ ENV VERSION_LAST_TAG=$VERSION_LAST_TAG
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
docker.io=20.10.24+dfsg1-1+b3 \
libssl-dev=3.0.13-1~deb12u1 \
libssl-dev=3.0.14-1~deb12u2 \
pkg-config=1.8.1-1 \
unzip=6.0-28 \
wget=${WGET_VERSION} \
software-properties-common=0.99.30-4.1~deb12u1 \
&& apt-get install -y --no-install-recommends \
openjdk-17-jdk=17.0.11+9-1~deb12u1 \
openjdk-17-jdk=17.0.12+7-2~deb12u1 \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*

Expand Down Expand Up @@ -126,12 +126,12 @@ RUN apt-get update \
ca-certificates \
build-essential=12.9 \
# https://security-tracker.debian.org/tracker/CVE-2023-38545
curl=7.88.1-10+deb12u6 \
curl=7.88.1-10+deb12u7 \
g++-aarch64-linux-gnu \
g++-x86-64-linux-gnu \
libc6-dev-arm64-cross=2.36-8cross1 \
libclang-dev=1:14.0-55.7~deb12u1 \
libssl-dev=3.0.13-1~deb12u1 \
libssl-dev=3.0.14-1~deb12u2 \
pkg-config=1.8.1-1 \
&& rm -rf /var/lib/apt/lists/*

Expand Down Expand Up @@ -177,18 +177,21 @@ RUN USER=root "$HOME/.cargo/bin/cargo" init --lib --name dummy --vcs none . \
&& mkdir -p ./jni-export/src \
&& mkdir -p ./node-common/src \
&& mkdir -p ./state-manager/src \
&& mkdir -p ./p2p/src \
&& touch ./core-api-server/src/lib.rs \
&& touch ./engine-state-api-server/src/lib.rs \
&& touch ./jni-export/src/lib.rs \
&& touch ./node-common/src/lib.rs \
&& touch ./state-manager/src/lib.rs
&& touch ./state-manager/src/lib.rs \
&& touch ./p2p/src/lib.rs
COPY core-rust/Cargo.toml ./
COPY core-rust/Cargo.lock ./
COPY core-rust/core-api-server/Cargo.toml ./core-api-server
COPY core-rust/engine-state-api-server/Cargo.toml ./engine-state-api-server
COPY core-rust/jni-export/Cargo.toml ./jni-export
COPY core-rust/node-common/Cargo.toml ./node-common
COPY core-rust/state-manager/Cargo.toml ./state-manager
COPY core-rust/p2p/Cargo.toml ./p2p

COPY docker/build_scripts/cargo_build_by_platform.sh /opt/radixdlt/cargo_build_by_platform.sh

Expand Down Expand Up @@ -256,9 +259,9 @@ LABEL org.opencontainers.image.authors="[email protected]"
# - https://packages.debian.org/bookworm/libc6
RUN apt-get update -y \
&& apt-get -y --no-install-recommends install \
openjdk-17-jre-headless=17.0.11+9-1~deb12u1 \
openjdk-17-jre-headless=17.0.12+7-2~deb12u1 \
# https://security-tracker.debian.org/tracker/CVE-2023-38545
curl=7.88.1-10+deb12u6 \
curl=7.88.1-10+deb12u7 \
gettext-base=0.21-12 \
daemontools=1:0.76-8.1 \
# https://security-tracker.debian.org/tracker/CVE-2023-4911
Expand Down
2 changes: 1 addition & 1 deletion cli-tools/src/main/java/com/radixdlt/shell/RadixShell.java
Original file line number Diff line number Diff line change
Expand Up @@ -178,7 +178,7 @@ public Node build() throws Exception {
dataDir = new File(Files.createTempDirectory("radix-shell-node-").toString());
}

customProperties.build().forEach((k, v) -> properties.set(k, v));
customProperties.build().forEach(properties::set);

properties.set("db.location", dataDir.toString());

Expand Down
10 changes: 10 additions & 0 deletions common/src/main/java/com/radixdlt/lang/Option.java
Original file line number Diff line number Diff line change
Expand Up @@ -218,6 +218,16 @@ default T or(T replacement) {
return fold(Functions::id, () -> replacement);
}

/**
* Return current value stored in current instance if current instance is present. If current
* instance is empty then return {@code null}.
*
* @return either value stored in current instance or {@code null} if current instance is empty
*/
default T toNullable() {
return fold(Functions::id, () -> null);
}

/**
* Return current value stored in current instance if current instance is present. If current
* instance is empty then return value returned by provided supplier. If current instance is not
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
/* Copyright 2021 Radix Publishing Ltd incorporated in Jersey (Channel Islands).
*
* Licensed under the Radix License, Version 1.0 (the "License"); you may not use this
* file except in compliance with the License. You may obtain a copy of the License at:
*
* radixfoundation.org/licenses/LICENSE-v1
*
* The Licensor hereby grants permission for the Canonical version of the Work to be
* published, distributed and used under or by reference to the Licensor’s trademark
* Radix ® and use of any unregistered trade names, logos or get-up.
*
* The Licensor provides the Work (and each Contributor provides its Contributions) on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied,
* including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT,
* MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE.
*
* Whilst the Work is capable of being deployed, used and adopted (instantiated) to create
* a distributed ledger it is your responsibility to test and validate the code, together
* with all logic and performance of that code under all foreseeable scenarios.
*
* The Licensor does not make or purport to make and hereby excludes liability for all
* and any representation, warranty or undertaking in any form whatsoever, whether express
* or implied, to any entity or person, including any representation, warranty or
* undertaking, as to the functionality security use, value or other characteristics of
* any distributed ledger nor in respect the functioning or value of any tokens which may
* be created stored or transferred using the Work. The Licensor does not warrant that the
* Work or any use of the Work complies with any law or regulation in any territory where
* it may be implemented or used or that it will be appropriate for any specific purpose.
*
* Neither the licensor nor any current or former employees, officers, directors, partners,
* trustees, representatives, agents, advisors, contractors, or volunteers of the Licensor
* shall be liable for any direct or indirect, special, incidental, consequential or other
* losses of any kind, in tort, contract or otherwise (including but not limited to loss
* of revenue, income or profits, or loss of use or data, or loss of reputation, or loss
* of any economic or other opportunity of whatsoever nature or howsoever arising), arising
* out of or in connection with (without limitation of any use, misuse, of any ledger system
* or use made or its functionality or any performance or operation of any code or protocol
* caused by bugs or programming or logic errors or otherwise);
*
* A. any offer, purchase, holding, use, sale, exchange or transmission of any
* cryptographic keys, tokens or assets created, exchanged, stored or arising from any
* interaction with the Work;
*
* B. any failure in a transmission or loss of any token or assets keys or other digital
* artefacts due to errors in transmission;
*
* C. bugs, hacks, logic errors or faults in the Work or any communication;
*
* D. system software or apparatus including but not limited to losses caused by errors
* in holding or transmitting tokens by any third-party;
*
* E. breaches or failure of security including hacker attacks, loss or disclosure of
* password, loss of private key, unauthorised use or misuse of such passwords or keys;
*
* F. any losses including loss of anticipated savings or other benefits resulting from
* use of the Work or any changes to the Work (however implemented).
*
* You are solely responsible for; testing, validating and evaluation of all operation
* logic, functionality, security and appropriateness of using the Work for any commercial
* or non-commercial purpose and for any reproduction or redistribution by You of the
* Work. You assume all risks associated with Your use of the Work and the exercise of
* permissions under this License.
*/

package com.radixdlt.p2p;

import com.radixdlt.lang.Option;
import com.radixdlt.sbor.codec.CodecMap;
import com.radixdlt.sbor.codec.StructCodec;
import java.util.Set;

public record AddressBookEntryDTO(
NodeIdDTO nodeId, Option<Long> bannedUntil, Set<PeerAddressEntryDTO> knownAddresses) {
public static void registerCodec(CodecMap codecMap) {
codecMap.register(
AddressBookEntryDTO.class,
codecs -> StructCodec.fromRecordComponents(AddressBookEntryDTO.class, codecs));
}
}
76 changes: 76 additions & 0 deletions core-rust-bridge/src/main/java/com/radixdlt/p2p/NodeIdDTO.java
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
/* Copyright 2021 Radix Publishing Ltd incorporated in Jersey (Channel Islands).
*
* Licensed under the Radix License, Version 1.0 (the "License"); you may not use this
* file except in compliance with the License. You may obtain a copy of the License at:
*
* radixfoundation.org/licenses/LICENSE-v1
*
* The Licensor hereby grants permission for the Canonical version of the Work to be
* published, distributed and used under or by reference to the Licensor’s trademark
* Radix ® and use of any unregistered trade names, logos or get-up.
*
* The Licensor provides the Work (and each Contributor provides its Contributions) on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied,
* including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT,
* MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE.
*
* Whilst the Work is capable of being deployed, used and adopted (instantiated) to create
* a distributed ledger it is your responsibility to test and validate the code, together
* with all logic and performance of that code under all foreseeable scenarios.
*
* The Licensor does not make or purport to make and hereby excludes liability for all
* and any representation, warranty or undertaking in any form whatsoever, whether express
* or implied, to any entity or person, including any representation, warranty or
* undertaking, as to the functionality security use, value or other characteristics of
* any distributed ledger nor in respect the functioning or value of any tokens which may
* be created stored or transferred using the Work. The Licensor does not warrant that the
* Work or any use of the Work complies with any law or regulation in any territory where
* it may be implemented or used or that it will be appropriate for any specific purpose.
*
* Neither the licensor nor any current or former employees, officers, directors, partners,
* trustees, representatives, agents, advisors, contractors, or volunteers of the Licensor
* shall be liable for any direct or indirect, special, incidental, consequential or other
* losses of any kind, in tort, contract or otherwise (including but not limited to loss
* of revenue, income or profits, or loss of use or data, or loss of reputation, or loss
* of any economic or other opportunity of whatsoever nature or howsoever arising), arising
* out of or in connection with (without limitation of any use, misuse, of any ledger system
* or use made or its functionality or any performance or operation of any code or protocol
* caused by bugs or programming or logic errors or otherwise);
*
* A. any offer, purchase, holding, use, sale, exchange or transmission of any
* cryptographic keys, tokens or assets created, exchanged, stored or arising from any
* interaction with the Work;
*
* B. any failure in a transmission or loss of any token or assets keys or other digital
* artefacts due to errors in transmission;
*
* C. bugs, hacks, logic errors or faults in the Work or any communication;
*
* D. system software or apparatus including but not limited to losses caused by errors
* in holding or transmitting tokens by any third-party;
*
* E. breaches or failure of security including hacker attacks, loss or disclosure of
* password, loss of private key, unauthorised use or misuse of such passwords or keys;
*
* F. any losses including loss of anticipated savings or other benefits resulting from
* use of the Work or any changes to the Work (however implemented).
*
* You are solely responsible for; testing, validating and evaluation of all operation
* logic, functionality, security and appropriateness of using the Work for any commercial
* or non-commercial purpose and for any reproduction or redistribution by You of the
* Work. You assume all risks associated with Your use of the Work and the exercise of
* permissions under this License.
*/

package com.radixdlt.p2p;

import com.radixdlt.crypto.ECDSASecp256k1PublicKey;
import com.radixdlt.sbor.codec.CodecMap;
import com.radixdlt.sbor.codec.StructCodec;

public record NodeIdDTO(ECDSASecp256k1PublicKey publicKey) {
public static void registerCodec(CodecMap codecMap) {
codecMap.register(
NodeIdDTO.class, codecs -> StructCodec.fromRecordComponents(NodeIdDTO.class, codecs));
}
}
Loading
Loading