Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Build with Nix #29

Merged
merged 135 commits into from
Aug 17, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
135 commits
Select commit Hold shift + click to select a range
e256dd0
Add build with Nix flakes
Mar 26, 2024
84c71b3
Update Nix flake environment
quapka Jun 19, 2024
c051ef2
Merge branch 'master' into build-with-nix
quapka Jun 19, 2024
ada6198
Merge branch 'master' into build-with-nix
quapka Jun 21, 2024
456a6f9
Allow running different versions of OpenSSL
quapka Jun 21, 2024
5622632
Don't build LibreSSL from submodule
quapka Jul 18, 2024
134ff10
Build LibreSSLShim as a separate derivation
quapka Jul 18, 2024
556c45f
Build LibreSSL alongside OpenSSL
quapka Jul 18, 2024
4fbc454
Set default for phase
quapka Jul 22, 2024
156fcc3
Build shared libs for BoringSSL
quapka Jul 22, 2024
5312654
Build BoringSSL shim from packaged BoringSSL
quapka Jul 22, 2024
20aee84
Clear overlays
quapka Jul 22, 2024
8f8d797
Build MbedTLS shim with packaged MbedTLS
quapka Jul 22, 2024
a6f468f
Build shims in separate files
quapka Jul 22, 2024
af299ba
Remove unnecessary dependency since MbedTLS package is now used
quapka Jul 22, 2024
f27f67c
Fix build instruction for BoringSSL and MbedTLS
quapka Jul 22, 2024
694bf4f
Match the packaged BoringSSL to the submodule one
quapka Jul 22, 2024
272a2b4
Link BoringSSL statically
quapka Jul 22, 2024
c867651
Build IPP Crypto from package
quapka Jul 23, 2024
50c9d5b
Clear deprecated build targets
quapka Jul 23, 2024
4bb7652
Clean up old build instructions
quapka Jul 23, 2024
40dde8f
Fix IPPCP ECLib requirements
quapka Jul 23, 2024
bddeacb
Don't use newer version of WolfCrypt
quapka Jul 23, 2024
1d10818
Clean unnecessary WolfCrypt build instructions
quapka Jul 23, 2024
ccdd0a2
Unify patching OpenSSL for all Flake outputs
quapka Jul 24, 2024
a899f7d
Link GNU Crypto library statically
quapka Jul 24, 2024
eed2fbf
Build OpenSSL statically
quapka Jul 24, 2024
d798fff
Linked tomcrypt statically
quapka Jul 24, 2024
282704b
Link Botan statically
quapka Jul 24, 2024
98e95fb
Static linking does not require loading botan-2
quapka Jul 25, 2024
c7300fb
Link Crypto++ statically
quapka Jul 25, 2024
f3ce8a2
Link Nettle statically
quapka Jul 25, 2024
20633c4
Link Mbedtls statically
quapka Jul 25, 2024
36bb495
Link LibreSSL statically
quapka Jul 25, 2024
d4ffadb
Derive common libs with Nix
quapka Jul 25, 2024
2c48451
Build Botan shim through Nix
quapka Jul 25, 2024
b29f697
Link TomCrypt statically and build the shim with Nix
quapka Jul 25, 2024
f251b60
Add missing Nix derivation for Botan shim
quapka Jul 25, 2024
943c182
Add missing Nix derivation for TomCrypt shim
quapka Jul 25, 2024
c625e03
Build Crypto++ shim through Nix
quapka Jul 25, 2024
c363815
Build OpenSSL shim through Nix
quapka Jul 25, 2024
7600a40
Clean up Nix build of BoringSSL shim
quapka Jul 25, 2024
07bce96
Build Gcrypt shim through Nix
quapka Jul 25, 2024
aa36d35
Remove runtime lib deps unnecessary due to static linking
quapka Jul 25, 2024
dd5e280
Clean up Nix build of MbedTLS shim
quapka Jul 25, 2024
86597bf
Rename IPP Crypto shim
quapka Jul 25, 2024
d75a844
Remove redundant IPP Crypto shim
quapka Jul 26, 2024
888bf25
Keep consistent shims build ordering
quapka Jul 26, 2024
c44faa2
Clean up Nix Gradle build
quapka Jul 26, 2024
642d697
Merge branch 'master' into build-with-nix
quapka Jul 26, 2024
11530df
Build Nettle shim through Nix
quapka Jul 26, 2024
5660a09
Expose static libtommath and libtomcrypt through pkg-config
quapka Jul 29, 2024
ebacdc2
Don't use LTC_PTHREAD in TomCrypt
quapka Jul 29, 2024
52bd3d1
Explain libtommath and libtomcrypt patches
quapka Jul 29, 2024
095a210
Link Nettle statically and build the shim with Nix
quapka Jul 29, 2024
9c6de28
Clean up the Nix build a bit
quapka Jul 30, 2024
40f5279
Fix Crypto++ version calculation
quapka Jul 30, 2024
3fcaa7a
Add autogenerated releases for OpenSSL, Crypto++ and Botan
quapka Jul 30, 2024
c405d82
Build various version of libtomcrypt
quapka Aug 5, 2024
cf3bfc3
Keep consistent ordering of libs
quapka Aug 5, 2024
0cbdeba
Make default Tomcrypt build properly
quapka Aug 6, 2024
a5dd212
Build script for fetching releases with Nix
quapka Aug 6, 2024
66c239e
Start adding some libgcrypt version
quapka Aug 6, 2024
b5b4f69
Add libgcrypt version fetcher
quapka Aug 6, 2024
805df74
Allow testing against various versions of MbedTLS
quapka Aug 6, 2024
2ee39e7
Build various version of IPP Crypto
quapka Aug 6, 2024
9239380
Build various version of Nettle
quapka Aug 6, 2024
90d423d
Default to previous source not version
quapka Aug 6, 2024
d82e729
Build various version of LibreSSL
quapka Aug 6, 2024
358e66d
Remove old and release-candidate versions and hash unpacked source
quapka Aug 6, 2024
c47d49f
Build last 100 revisions of BoringSSL
quapka Aug 7, 2024
f352780
Compute BoringSSL source digests locally
quapka Aug 7, 2024
edbc1c2
Autoformat all Nix sources
quapka Aug 7, 2024
40f7282
Recalculate correct OpenSSL source hashes
quapka Aug 8, 2024
424d952
Remove accidental copy
quapka Aug 8, 2024
a103385
Update BoringSSL hashes
quapka Aug 8, 2024
6a8ecdd
Store package versions in JSON to simplify reusability
quapka Aug 8, 2024
cad198b
Allow fetching all libraries
quapka Aug 8, 2024
1c78a5c
Patch only LibreSSL 3.8.2
quapka Aug 8, 2024
e3373e5
Allow building the shims directly
quapka Aug 8, 2024
fee9a6d
Autoformat all JSON files
quapka Aug 8, 2024
fb5961f
Only Botan 2 is currently tested
quapka Aug 8, 2024
a1902a0
Add script for testing building all shims
quapka Aug 8, 2024
4e7c106
Call the correct script name
quapka Aug 9, 2024
91b4257
Show how long it takes to (re)build a shim
quapka Aug 9, 2024
a5a6ef2
Merge branch 'master' into build-with-nix
J08nY Aug 9, 2024
7f21a12
Log error in test shim build.
J08nY Aug 9, 2024
646607b
Build prng and preload as part of common libs
Aug 10, 2024
4b39681
Build library shims only when selected
Aug 11, 2024
efe198a
Show the correct version when building for all shims except TomCrypt …
Aug 11, 2024
e7eba2d
Allow building target libs separately
Aug 12, 2024
b83164b
Build shims and libs with a single script
Aug 12, 2024
23e27b1
Save results of building all libs and shim to simplify debugging
quapka Aug 12, 2024
0c40cf3
Fix Botan 2.0.0 and 2.0.1 build.
J08nY Aug 12, 2024
317c8f5
Fix typo
quapka Aug 12, 2024
8e9466c
Fix creating cryptopp dotted version
quapka Aug 12, 2024
042413d
Fix Botan builds.
J08nY Aug 12, 2024
7971f11
Add Nix build in CI.
J08nY Aug 12, 2024
86528d6
Add default targets for all lib and shim builds
quapka Aug 12, 2024
bac37f1
Cleanup passing of CFLAGS and LFLAGS to shims.
J08nY Aug 12, 2024
77f442d
Do not fail fast in Nix in CI.
J08nY Aug 12, 2024
71bb436
Fix Botan include directory.
J08nY Aug 12, 2024
7284b3e
Fix order of versions in lib version files.
J08nY Aug 12, 2024
2b8744a
Add version define to shim build.
J08nY Aug 12, 2024
b170ce5
Acutally use the last version in build.
J08nY Aug 12, 2024
e0bc60f
Fix mbedTLS and Botan default builds.
J08nY Aug 12, 2024
427bceb
Fix Crypto++ 7.
J08nY Aug 13, 2024
9a00cf0
Improve formatting
quapka Aug 13, 2024
3fa7861
Dumb shit, checkFlags don't work.
J08nY Aug 15, 2024
cf6816c
Fix building Mbed-TLS version 2.25.0
quapka Aug 15, 2024
eed91ff
Fix mbedtls shim build for many versions.
J08nY Aug 15, 2024
335782e
Plot tables with differences in results
quapka Aug 15, 2024
0dd90be
Create tables with all results
quapka Aug 15, 2024
d906ea9
Fixup libressl >= 3.8.2.
J08nY Aug 15, 2024
e4b75bd
Fixup many libressl versions.
J08nY Aug 15, 2024
60f4b2e
Fix mbedtls 3.4.0.
J08nY Aug 15, 2024
8d5c5fc
Fix mbedtls 3.2.0:
J08nY Aug 15, 2024
55c7586
Add more OpenSSL versions.
J08nY Aug 15, 2024
ba11d87
Fix broken nettle.
J08nY Aug 15, 2024
fc2ad83
Fix mbedtls v2.26.0.
J08nY Aug 15, 2024
6c5776d
Fix many more OpenSSLs.
J08nY Aug 15, 2024
19f304e
More OpenSSL.
J08nY Aug 15, 2024
cb8a0e0
OpenSSL 1.1.0.
J08nY Aug 15, 2024
280630a
Crypto++ 6.1.0.
J08nY Aug 15, 2024
e631d3f
Fix cryptopp 7.0.0.
J08nY Aug 15, 2024
7ca6294
Pull older BoringSSL versions, not 100 newest.
J08nY Aug 15, 2024
c1feb52
Reorder BoringSSL versions properly.
J08nY Aug 15, 2024
209a4f8
Attempt to fix submodule build.
J08nY Aug 15, 2024
87db8e0
Fix LibreSSl old build.
J08nY Aug 15, 2024
6e01272
Fallback to newest version assumption in shims.
J08nY Aug 17, 2024
a707a34
Fix default BoringSSL build.
J08nY Aug 17, 2024
3a8ccf8
More BoringSSL builds.
J08nY Aug 17, 2024
fe92e1e
Add Nix info to README.
J08nY Aug 17, 2024
94e821a
By default build ECTester with default, aka newest fetched, libraries
quapka Aug 17, 2024
8d54fae
Remove old devShell.
J08nY Aug 17, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
53 changes: 53 additions & 0 deletions .github/workflows/nix.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
name: Nix

on:
push:
branches: [ "master" ]
pull_request:
branches: [ "master" ]

jobs:
standalone:
runs-on: ubuntu-latest
permissions:
contents: read

strategy:
matrix:
library: [ "botan", "cryptopp", "openssl", "boringssl", "gcrypt", "mbedtls", "ippcp", "nettle", "libressl" ]
fail-fast: false
name: Build standalone ${{ matrix.library }}
steps:
- uses: actions/checkout@v4
with:
submodules: recursive
fetch-tags: true
fetch-depth: -1

- name: Setup libraries
run: |
sudo apt update
sudo apt install

- uses: DeterminateSystems/nix-installer-action@v13
with:
diagnostic-endpoint: ""

- uses: DeterminateSystems/magic-nix-cache-action@v7
with:
diagnostic-endpoint: ""

- name: Build library
run: |
nix build ".#lib.${{ matrix.library }}.default"

- name: Build shim
run: |
nix build ".#shim.${{ matrix.library }}.default"

- name: Build standalone
run: |
nix build ".?submodules=1#${{ matrix.library }}.default"

- name: List library
run: nix run ".?submodules=1#${{ matrix.library }}.default" -- list-libs
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -46,3 +46,4 @@

# Ignore Gradle build output directory
build
build_all
81 changes: 57 additions & 24 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ cryptographic libraries. It consists of four separate parts:

- The ECTester applet, a JavaCard applet that provides the testing interface
- The ECTesterReader app, a reader app that works with the applet
- The ECTesterStandalone app, which works with software libraries
- The ECTesterStandalone app, which tests software cryptographic libraries
- Jupyter notebooks for analysis and visualization of data from the apps

For more information on ECC support on JavaCards see the [GitHub page](https://crocs-muni.github.io/ECTester/), with results, tables and docs.
Expand Down Expand Up @@ -50,10 +50,9 @@ git submodule update --init --recursive # To initialize submodules (JavaCa
The applet comes in several flavors, targeting JavaCard `2.2.1`, `2.2.2` and `3.0.5`. The `2.2.2` and later flavors
support extended length APDUs which are necessary for some commands to work properly.

The `:standalone:libs` task invokes a Makefile in `standalone/src/main/resources/cz/crcs/ectester/standalone/libs/jni`, which tries to build the
C/C++ shim libraries required for ECTester to test the actual native cryptographic libraries from Java.
The Makefile uses pkg-config to locate the libraries installed, thus if non-standard location of the tested libraries is
used, the Makefile or your pkg-config needs some changes to work.
To build the standalone part, which involves numerous cryptographic libraries, one has two options.
- Install these cryptographic libraries system-wide and let the build use those.
- Use [Nix](https://nixos.org/) to build the tool part with given library versions.

See the section on [setup](#setup-1) of standalone library testing for more details.

Expand Down Expand Up @@ -309,6 +308,48 @@ For more information on ECC libraries see [LIBS](docs/LIBS.md).

### Setup

ECTester interfaces with native libraries by using custom shim libraries that expose the functionality via the
[Java Native Interface](https://en.wikipedia.org/wiki/Java_Native_Interface), these can be found in the
[standalone/src/main/java/cz/crcs/ectester/standalone/libs/jni](standalone/src/main/java/cz/crcs/ectester/standalone/libs/jni) directory along with a Makefile
(Makefile.bat for Windows). The shim library will depend on the native library, and have a name like
`boringssl_provider.so`, `botan_provider.so`, `cryptopp_provider.so` and `openssl_provider.so`.
The Makefile has a target for every library that it supports that builds its shim, see the `help`
target for more info.

However, building these shims and in general the target libraries can be tricky to do so reliably across
systems and setups. Thus ECTester offers two ways of doing so.

- The first way is to install the libraries system-wide and rely on pkg-config to find proper configuration
for the shims and link them dynamically.
- The second way is to use [Nix](https://nixos.org/) to build the given versions of the libraries and shims and
link them statically.

#### Nix

Install [Nix](https://nixos.org/download/), then to build:

```shell
# To build a library in a given version run (example OpenSSL 3.3.1):
nix build ".#lib.openssl.v331"
# To build a shim using a given version of a library (example mbedTLS 3.5):
nix build ".#shim.mbedtls.v35"
# To build ECTesterStandalone.jar with a given version of a library (example libgcrypt 1.9.4):
nix build ".#gcrypt.v194"
# The available versions of the libraries are in the nix/*_pkg_versions.json files.
# The "default" version always points to the most recent version.
```

Each of the build steps above puts (symlinks really) its results into `./result` directory.
However, subsequent builds then replace that with their own results. To run ECTesterStandalone
with a given library version and arguments do:

```shell
# This runs the default test-suite agains LibreSSL 3.9.2
nix run ".#libressl.v392" --- test default LibreSSL
```

#### Gradle

```shell
./gradlew :standalone:libs # To build the native library shims.
./gradlew :standalone:uberJar # To build the standalone tool (jar) -> "standalone/build/libs/ECTesterStandalone.jar"
Expand All @@ -324,16 +365,22 @@ successfully or that the actual native library couldn't be found and loaded on r
build errors during the ant run in the `libs-try` step, for the latter, if the library is in an non-standard location
specifying `LD_LIBRARY_PATH` will help load it. Consulting the next sections should help solve both.

#### Native

ECTester interfaces with native libraries by using custom shim libraries that expose the functionality via the [Java Native Interface](https://en.wikipedia.org/wiki/Java_Native_Interface), these can be found in the [standalone/src/main/java/cz/crcs/ectester/standalone/libs/jni](standalone/src/main/java/cz/crcs/ectester/standalone/libs/jni) directory along with a Makefile (Makefile.bat for Windows). The shim library will depend on the native library, and have a name like `boringssl_provider.so`, `botan_provider.so`, `cryptopp_provider.so` and `openssl_provider.so`. The Makefile has a target for every library that it supports that builds its shim, see the `help` target for more info. The Makefile is automatically ran when the `:standalone:libs` gradle task is triggered, so if all is setup correctly, you do not need to deal with the Makefile while building.
The `:standalone:libs` task invokes the Makefile, which tries to build the
C/C++ shim libraries required for ECTester to test the actual native cryptographic libraries from Java.
The Makefile uses pkg-config to locate the libraries installed, thus if non-standard location of the tested libraries is
used, the Makefile or your pkg-config needs some changes to work.

There are two important environmental variables that should be set in your environment. First, you should set `JAVA_HOME` which should point to your JDK. The tooling uses `JAVA_HOME` to locate native Java library headers, like `jni.h`. Second, ECTester uses pkg-config to locate the native libraries, if your pkg-config files are in an unusual place the pkg-config command would not find them by default, you should set `PKG_CONFIG_PATH` to the directory containing the `*.pc` files. If pkg-config files are unavailable for the library you are trying to test, you will need to change the Makefile manually to apply the correct options to the commands (CFLAGS, include options, linker options...).
There are two important environmental variables that should be set in your environment. First, you should set
`JAVA_HOME` which should point to your JDK. The tooling uses `JAVA_HOME` to locate native Java library headers,
like `jni.h`. Second, ECTester uses pkg-config to locate the native libraries, if your pkg-config files are in an
unusual place the pkg-config command would not find them by default, you should set `PKG_CONFIG_PATH` to the directory
containing the `*.pc` files. If pkg-config files are unavailable for the library you are trying to test, you will need
to change the Makefile manually to apply the correct options to the commands (CFLAGS, include options, linker options...).

Below you can see how a full build with all the libraries currently supported on Linux looks
```
> cd standalone/src/main/resources/cz/crcs/ectester/standalone/libs/jni
> make
> make -f Makefile.ext
cc -DLTM_DESC -fPIC -I"/usr/lib/jvm/java-21-openjdk/include" -I"/usr/lib/jvm/java-21-openjdk/include/linux" -I. -Wno-deprecated-declarations -O2 -c tomcrypt.c
cc -fPIC -I"/usr/lib/jvm/java-21-openjdk/include" -I"/usr/lib/jvm/java-21-openjdk/include/linux" -I. -Wno-deprecated-declarations -O2 -c c_utils.c
cc -o lib_timing.so -shared -fPIC -I"/usr/lib/jvm/java-21-openjdk/include" -I"/usr/lib/jvm/java-21-openjdk/include/linux" -I. -Wno-deprecated-declarations -O2 -Wl,-soname,lib_timing.so c_timing.c
Expand Down Expand Up @@ -421,20 +468,6 @@ on `LD_LIBRARY_PATH`.
Consult the GitHub CI [build script](.github/workflows/build.yml) for an example that runs on Ubuntu 22.04.


#### Java

OpenJDK JRE is required to test ECDH on Windows properly, as Oracle JRE requires the Java Cryptography Providers
for certain classes (such as a [KeyAgreement](https://docs.oracle.com/javase/8/docs/api/javax/crypto/KeyAgreement.html))
to be signed by keys that are signed by their JCA Code Signing Authority. ECTester internally uses Java Cryptography Provider
API to expose and test native libraries.

Installing the Java Cryptography Extension Unlimited Strength policy files is necessary to do testing
(for Java 8) with quite a lot of practical key sizes, they are available for download:

- [Java 8](http://www.oracle.com/technetwork/java/javase/downloads/jce8-download-2133166.html)

To install, place them in `${java.home}/jre/lib/security/`.

### Examples

Snippet below shows how the `list-libs` command for well, listing currently supported libraries, behaves if all supported libraries are present (on Linux).
Expand Down
Loading