Do not send health probes to Semeru Cloud Compiler on the compilation request port (38400)

[mpirvu]

Bug Report

What did you do?

This issue is related to OpenLiberty/open-liberty#27665
Under the covers, OLO sets up health probes (liveness/readiness) for the Semeru Cloud Compiler pod. These probes target the port used by Semeru Cloud Compiler for compilation requests (default 38400). This works fine as long as this port does not use SSL encryption.
However, OLO sets up traffic encryption between Liberty clients and Semeru Cloud Compiler servers. Starting with OpenSSL 3.0, the OpenSSL library will log errors if the peer (in this case the kubelet) does not inform the other party about the connection being closed.

What did you expect to see?

No errors in the logs of Semeru Cloud Compiler pod

What did you see instead?

The log for the Semeru Cloud Compiler pod shows errors like:

A0F04FE4FE7F0000:error:0A000126:SSL routines:ssl3_read_n:unexpected eof while reading:ssl/record/rec_layer_s3.c:320:

Environment

Note: currently these errors are seen only with Java21 because Semeru containers with Java 21 include OpenSSL 3.0.
In contrast, Semeru containers for Java 8,11,17 include OpenSSL 1.1.1. This may change in the future.

Possible solution

Since currently there is no way to specify encryption for a TCP probe, the only solution is to send these probes to another Semeru Cloud Compiler which does not use encryption. Such a dedicated health port (default 38600) will be added to the Semeru Cloud Compiler in PR eclipse-openj9/openj9#19107
It may be helpful to allow the user to further configure the health probes (such as the targeted port or the frequency).

Additional context

OpenLiberty/open-liberty#27665