Mastering two way TLS 🔐

Hey, hello there 👋 welcome to this tutorial! I hope you will like it ❤️

This tutorial will walk you through the process of protecting your application with TLS authentication, only allowing access for certain users based on their certificates. This means that you can choose which users are allowed to call your application.

Table of contents

1. Introduction
2. Tutorial
   • Starting the server
   • Saying hello to the server (without encryption)
   • Enabling HTTPS on the server (one-way TLS)
   • Require the client to identify itself (two way TLS)
   • Two way TLS based on trusting the Certificate Authority
3. Automated scripts
4. Tested Http Clients
5. Demo and walk-through video
6. Contributing

Introduction

This sample project demonstrates a basic setup of a server and a client. The communication between the server and client happens through HTTP, so there is no encryption at all yet. The goal is to ensure that all communication will be encrypted.

Definition

• Identity: A KeyStore which holds the key pair also known as private and public key
• TrustStore: A KeyStore containing one or more certificates also known as public key. This KeyStore contains a list of trusted certificates
• One way authentication (also known as one way tls, one way ssl): Https connection where the client validates the certificate of the counter party
• Two way authentication (also known as two way tls, two way ssl, mutual authentication): Https connection where the client as well as the counter party validates the certificate, also known as mutual authentication

Usefull links

• Keytool Cheatsheet
• Openssl Cheatsheet
• Curl with Java KeyStore
• Http Client Configuration Cheatsheet

Some History

I mostly worked with Apache Http Client and therefore I initially created this project with only a http client from Apache. After some time I discovered that there are a lot more Java clients and there also some clients available based on Kotlin and Scala. Configuring ssl/tls can be hard and every client requires a different setup. I want to share the knowledge with other developers as every developer will probably work with a different http client. I also want to provide a cheatsheet for all the developers and the community, see here for a list of 40+ http client) with example client configuration and example http request. The client module has grown over time and contains a-lot dependencies to test all these http clients for java, scala and kotlin. Therefore client module might look complicated. Beware that for this specific reason it will download a-lot dependencies during the initial build. Also GitHub - SSLContext Kickstart came into life during the lifecycle of this project to easily configure all those clients. Every http client can require a different ssl object to enable ssl and this library ensures it can deliver all the types to configure these clients while having a basic ssl configuration.

Tutorial

Starting the server

Minimum requirements:

1. Java 17
2. Maven 3.5.0
3. Eclipse, Intellij IDEA (or any other text editor like VIM)
4. A terminal

If you want to start instantly without installing any software, click the button below to open the project in an online development environment:

This project contains a maven wrapper, so you can run this project without installing maven. The documentation for this tutorial contains next to the default mvn command also the commands for the maven wrapper.

If you want to run this project with Java 8 or Java 11, you can get an older version with the git command below.

• git checkout tags/java-8-compatible
• git checkout tags/java-11-compatible

The server depends on the other components of the project, so run mvn install in the root directory first.

mvn install

Start the server by running the main method of the App Class in the server project or by running the following command from the terminal in the root directory:

cd server/ && mvn spring-boot:run

Or with the maven wrapper

cd server/ && ./../mvnw spring-boot:run

Saying hello to the server (without encryption)

Currently, the server is running on the default port of 8080 without encryption. You can call the hello endpoint with the following curl command in the terminal:

curl -i -XGET http://localhost:8080/api/hello

It should give you the following response:

HTTP/1.1 200
Content-Type: text/plain;charset=UTF-8
Content-Length: 5
Date: Sun, 11 Nov 2018 14:21:50 GMT

Hello

You can also call the server with the provided client in the client directory. The client is an integration test based on Cucumber, and you can start it by running the ClientRunnerIT class from your IDE or by running the following command from the terminal in the root directory cd client/ && mvn exec:java or with the maven wrapper cd client/ && ./../mvnw exec:java.

There is a Hello.feature file that describes the steps for the integration test. You can find it in the test resources of the client project. There is another way to run both the server and the client and that is with the following command in the root directory: mvn clean verify or with the maven wrapper ./mvnw clean verify. The client sends by default requests to localhost, because it expects the server on the same machine. If the server is running on a different machine you can still provide a custom url with the following VM argument while running the client: -Durl=http://[HOST]:[PORT]

Enabling HTTPS on the server (one-way TLS)

Now, you will learn how to secure your server by enabling TLS. You can do that by adding the required properties to the application properties file named: application.yml

Add the following property:

server:
  port: 8443
  ssl:
    enabled: true

You will probably ask yourself why the port is set to 8443. The port convention for a tomcat server with https is 8443, and for http, it is 8080. So, we could use port 8080 for https connections, but it is a bad practice. See Wikipedia for more information about port conventions.

Restart the server so that it can apply the changes you made. You will probably get the following exception: IllegalArgumentException: Resource location must not be null.

You are getting this message because the server requires a keystore with the certificate of the server to ensure that there is a secure connection with the outside world. The server can provide you more information if you provide the following VM argument: -Djavax.net.debug=SSL,keymanager,trustmanager,ssl:handshake

To solve this issue, you are going to create a keystore with a public and private key for the server. The public key will be shared with users so that they can encrypt the communication. The communication between the user and server can be decrypted with the private key of the server. Please never share the private key of the server, because others could intercept the communication and will be able to see the content of the encrypted communication.

To create a keystore with a public and private key, execute the following command in your terminal:

keytool -v -genkeypair -dname "CN=Hakan,OU=Amsterdam,O=Thunderberry,C=NL" -keystore server/src/main/resources/identity.jks -storepass secret -keypass secret -keyalg RSA -keysize 2048 -alias server -validity 3650 -deststoretype pkcs12 -ext KeyUsage=digitalSignature,dataEncipherment,keyEncipherment,keyAgreement -ext ExtendedKeyUsage=serverAuth,clientAuth -ext SubjectAlternativeName:c=DNS:localhost,DNS:raspberrypi.local,IP:127.0.0.1

Now, you need to tell your server where the location of the keystore is and provide the passwords. Paste the following in your application.yml file:

server:
  port: 8443
  ssl:
    enabled: true
    key-store: classpath:identity.jks
    key-password: secret
    key-store-password: secret

Congratulations! You enabled a TLS-encrypted connection between the server and the client! Now, you can try to call the server with the following curl command: curl -i --insecure -v -XGET https://localhost:8443/api/hello

Let's also run the client in the ClientRunnerIT class.

You will see the following error message: java.net.ConnectException: Connection refused (Connection refused). It looks like the client is trying to say hello to the server but the server is not there. The problem is that the client it trying to say hello to the server on port 8080 while it is active on the port 8443. Apply the following changes to the Constants class:

From:

private static final String DEFAULT_SERVER_URL = "http://localhost:8080";

To:

private static final String DEFAULT_SERVER_URL = "https://localhost:8443";

Let's try to run the client again, and you will see that the following message will appear: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target. This means that the client wants to communicate over HTTPS and during the handshake procedure it received the certificate of the server which it doesn't recognizes yet. Therefor you also need to create a truststore. A truststore is a suitcase containing trusted certificates. The client will compare the certificate, which it will receive during the SSL Handshake process with the content of its truststore. If there is a match, then the SSL Handshake process will continue. Before creating the truststores, you need to have the certificates of the server. You can get it with the following command:

Export certificate of the server

keytool -v -exportcert -file server/src/main/resources/server.cer -alias server -keystore server/src/main/resources/identity.jks -storepass secret -rfc

Now, you can create the truststore for the client and import the certificate of the server with the following command:

keytool -v -importcert -file server/src/main/resources/server.cer -alias server -keystore client/src/test/resources/truststore.jks -storepass secret -noprompt

You created the truststore for the client. Unfortunately, the client is not aware of this. Now, you need to tell that it needs to use the truststore with the correct location and password. You also need to tell the client that authentication is enabled. Provide the following property in the application.yml file of the client:

client:
  ssl:
    one-way-authentication-enabled: true
    two-way-authentication-enabled: false
    trust-store: truststore.jks
    trust-store-password: secret

Require the client to identify itself (two-way TLS)

The next step is to require the authentication of the client. This will force the client to identify itself, and in that way, the server can also validate the identity of the client and whether or not it is a trusted one. You can enable this by telling the server that you also want to validate the client with the property client-auth. Put the following properties in the application.yml of the server:

server:
  port: 8443
  ssl:
    enabled: true
    key-store: classpath:identity.jks
    key-password: secret
    key-store-password: secret
    client-auth: need

If you run the client, it will fail with the following error message: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate. This indicates that the certificate of the client is not valid because there is no certificate at all. So, let's create one with the following command

keytool -v -genkeypair -dname "CN=Suleyman,OU=Altindag,O=Altindag,C=NL" -keystore client/src/test/resources/identity.jks -storepass secret -keypass secret -keyalg RSA -keysize 2048 -alias client -validity 3650 -deststoretype pkcs12 -ext KeyUsage=digitalSignature,dataEncipherment,keyEncipherment,keyAgreement -ext ExtendedKeyUsage=serverAuth,clientAuth

You also need to create a truststore for the server. Before creating the truststore, you need to have the certificates of the client. You can get it with the following command:

Export certificate of the client

keytool -v -exportcert -file client/src/test/resources/client.cer -alias client -keystore client/src/test/resources/identity.jks -storepass secret -rfc

Create the server truststore with the certificate of the client

keytool -v -importcert -file client/src/test/resources/client.cer -alias client -keystore server/src/main/resources/truststore.jks -storepass secret -noprompt

You created the extra keystore for the client. Unfortunately, the client is not aware of this. Now, you need to tell that it also needs to use the keystore with the correct location and password. You also need to tell the client that two-way-authentication is enabled. Provide the following property in the application.yml file of the client:

client:
  ssl:
    one-way-authentication-enabled: false
    two-way-authentication-enabled: true
    key-store: identity.jks
    key-password: secret
    key-store-password: secret
    trust-store: truststore.jks
    trust-store-password: secret

The server is also not aware of the newly created truststore. Therefore, replace the current properties with the following properties:

server:
  port: 8443
  ssl:
    enabled: true
    key-store: classpath:identity.jks
    key-password: secret
    key-store-password: secret
    trust-store: classpath:truststore.jks
    trust-store-password: secret
    client-auth: need

If you run the client again, you will see that the test passed and that the client received the hello message from the server in a secured way. Congratulations! You finished installing two-way TLS!

Two way TLS based on trusting the Certificate Authority

There is another way to have mutual authentication and that is based on trusting the Certificate Authority. It has pros and cons.

Pros

• Clients do not need to add the certificate of the server
• Server does not need to add all the certificates of the clients
• Maintenance will be less because only the Certificate Authority's certificate validity can expire

Cons

• You don't have control anymore for which applications are allowed to call your application. You give permission to any application who has a signed certificate by the Certificate Authority.

These are the following steps:

1. Creating a Certificate Authority
2. Creating a Certificate Signing Request
3. Signing the certificate with the Certificate Signing Request
4. Replace unsigned certificate with a signed one
5. Trusting the Certificate Authority only

Creating a Certificate Authority

Normally there is already a Certificate Authority, and you need to provide your certificate to have it signed. Here you will create your own Certificate Authority and sign the Client and Server certificate with it. To create one you can execute the following command:

keytool -v -genkeypair -dname "CN=Root-CA,OU=Certificate Authority,O=Thunderberry,C=NL" -keystore root-ca/identity.jks -storepass secret -keypass secret -keyalg RSA -keysize 2048 -alias root-ca -validity 3650 -deststoretype pkcs12 -ext KeyUsage=digitalSignature,keyCertSign -ext BasicConstraints=ca:true,PathLen:3

Or you can use the one which is already provided in the repository, see identity.jks

Creating a Certificate Signing Request

To get your certificate signed you need to provide a Certificate Signing Request (.csr) file. This can be created with the following command:

Certificate Signing Request for the server

keytool -v -certreq -file server/src/main/resources/server.csr -keystore server/src/main/resources/identity.jks -alias server -keypass secret -storepass secret -keyalg rsa

Certificate Signing Request for the client

keytool -v -certreq -file client/src/test/resources/client.csr -keystore client/src/test/resources/identity.jks -alias client -keypass secret -storepass secret -keyalg rsa

The Certificate Authority need these csr files to be able to sign it. The next step will be signing the requests.

Signing the certificate with the Certificate Signing Request

Signing the client certificate

keytool -v -gencert -infile client/src/test/resources/client.csr -outfile client/src/test/resources/client-signed.cer -keystore root-ca/identity.jks -storepass secret -alias root-ca -validity 3650 -ext KeyUsage=digitalSignature,dataEncipherment,keyEncipherment,keyAgreement -ext ExtendedKeyUsage=serverAuth,clientAuth -rfc

Signing the server certificate

keytool -v -gencert -infile server/src/main/resources/server.csr -outfile server/src/main/resources/server-signed.cer -keystore root-ca/identity.jks -storepass secret -alias root-ca -validity 3650 -ext KeyUsage=digitalSignature,dataEncipherment,keyEncipherment,keyAgreement -ext ExtendedKeyUsage=serverAuth,clientAuth -ext SubjectAlternativeName:c=DNS:localhost,DNS:raspberrypi.local,IP:127.0.0.1 -rfc

Replace the unsigned certificate with a signed one

The identity keystore of the server and client still have the unsigned certificate. Now you can replace it with the signed one. The keytool has a strange limitation/design. It won't allow you to directly import the signed certificate, and it will give you an error if you try it. The certificate of the Certificate Authority must be present within the identity.jks.

Export CA Certificate

keytool -v -exportcert -file root-ca/root-ca.pem -alias root-ca -keystore root-ca/identity.jks -storepass secret -rfc

Client

keytool -v -importcert -file root-ca/root-ca.pem -alias root-ca -keystore client/src/test/resources/identity.jks -storepass secret -noprompt
keytool -v -importcert -file client/src/test/resources/client-signed.cer -alias client -keystore client/src/test/resources/identity.jks -storepass secret
keytool -v -delete -alias root-ca -keystore client/src/test/resources/identity.jks -storepass secret

Server

keytool -v -importcert -file root-ca/root-ca.pem -alias root-ca -keystore server/src/main/resources/identity.jks -storepass secret -noprompt
keytool -v -importcert -file server/src/main/resources/server-signed.cer -alias server -keystore server/src/main/resources/identity.jks -storepass secret
keytool -v -delete -alias root-ca -keystore server/src/main/resources/identity.jks -storepass secret

Trusting the Certificate Authority only

Now you need to configure your client and server to only trust the Certificate Authority. You can do that by importing the certificate of the Certificate Authority into the truststores of the client and server. You can do that with the following two commands:

Client

keytool -v -importcert -file root-ca/root-ca.pem -alias root-ca -keystore client/src/test/resources/truststore.jks -storepass secret -noprompt

Server

keytool -v -importcert -file root-ca/root-ca.pem -alias root-ca -keystore server/src/main/resources/truststore.jks -storepass secret -noprompt

The truststores still contains the client and server specific certificates and that needs to be removed. You can do that with the following command:

Client

keytool -v -delete -alias server -keystore client/src/test/resources/truststore.jks -storepass secret

Server

keytool -v -delete -alias client -keystore server/src/main/resources/truststore.jks -storepass secret

If you run the client again, you will see that the test passed and that the client received the hello message from the server while based on a certificate which is signed by the Certificate Authority.

Automated script for enabling authentication with TLS

You can also automate all the previous steps described above with the provided scripts at the script directory of this project. Run one of these commands to run the scripts:

• One way authentication: ./configure-one-way-authentication
• Two way authentication: ./configure-two-way-authentication-by-trusting-each-other my-company-name
• Two way authentication by trusting the Certificate Authority: ./configure-two-way-authentication-by-trusting-root-ca my-company-name

Tested clients

Below is a list of already tested clients, plain Java based Http Client configurations can be found at the ClientConfig class. Kotlin and Scala based http client configurations are included as nested class, see here for the full list: service directory. The service directory contains the individual Http Clients with an example requests. All client examples use the same base ssl configuration created within the SSLConfig class.

Java

• Apache HttpClient -> Client configuration | Example request
• Apache HttpAsyncClient -> Client configuration | Example request
• Apache 5 HttpClient -> Client configuration | Example request
• Apache 5 HttpAsyncClient -> Client configuration | Example request
• JDK HttpClient -> Client Configuration | Example request
• Old JDK HttpClient -> Client Configuration & Example request
• Netty Reactor -> Client Configuration | Example request
• Jetty Reactive HttpClient -> Client Configuration | Example request
• Spring RestTemplate -> Client Configuration | Example request
• Spring WebFlux WebClient Netty -> Client Configuration | Example request
• Spring WebFlux WebClient Jetty -> Client Configuration | Example request
• OkHttp -> Client Configuration | Example request
• Jersey Client -> Client Configuration | Example request
• Old Jersey Client -> Client Configuration | Example request
• Apache CXF JAX-RS -> Client Configuration | Example request
• Apache CXF using ConduitConfigurer -> Client Configuration | Example request
• Google HttpClient -> Client Configuration | Example request
• Unirest -> Client Configuration | Example request
• Retrofit -> Client Configuration | Example request
• Async Http Client -> Client Configuration | Example request
• Feign -> Client Configuration | Example request
• Methanol -> Client Configuration | Example request
• Vertx Webclient -> Client Configuration | Example request
• gRPC -> Client/Server Configuration & Example request
• ElasticSearch -> RestHighLevelClient Configuration & example request
• Jetty WebSocket -> Client configuration & example request

Kotlin

• Fuel -> Client Configuration & Example request
• Http4k with Apache 4 -> Client Configuration | Example request
• Http4k with Async Apache 4 -> Client Configuration | Example request
• Http4k with Apache 5 -> Client Configuration | Example request
• Http4k with Async Apache 5 -> Client Configuration | Example request
• Http4k with Java Net -> Client Configuration | Example request
• Http4k with Jetty -> Client Configuration | Example request
• Http4k with OkHttp -> Client Configuration | Example request
• Kohttp -> Client Configuration & Example request
• Ktor with Android engine -> Client Configuration | Example request
• Ktor with Apache engine -> Client Configuration | Example request
• Ktor with CIO (Coroutine-based I/O) engine -> Client Configuration | Example request
• Ktor with Java engine -> Client Configuration | Example request
• Ktor with Okhttp engine -> Client Configuration | Example request

Scala

• Twitter Finagle -> Client Configuration & Example request
• Twitter Finagle Featherbed -> Client Configuration & Example request
• Akka Http Client -> Client Configuration | Example request
• Dispatch Reboot -> Client Configuration & Example request
• ScalaJ / Simplified Http Client -> Client Configuration & Example request
• Sttp -> Client Configuration & Example request
• Requests-Scala -> Client Configuration & Example request
• Http4s Blaze Client -> Client Configuration | Example request
• Http4s Java Net Client -> Client Configuration | Example request

Other

This tutorial does not provide examples for the clients listed below, however it is proven to be working with the server provided in this tutorial. To guide you in the right direction you can convert the created java keystore files into pem files which in return can be used with one of the clients listed below. The following Gist page contains the commands to convert a Java keystore to a pem file: Gist - Curl with Java KeyStore

• curl -> Client Configuration & Example request
• Postman

Demo and walk-through video

Contributing

There are plenty of ways to contribute to this project:

• Give it a star
• Make a donation through GitHub or open collective
• Share it with a
• Join the Gitter room and leave a feedback or help with answering users questions
• Submit a PR