LIVY-979: A helm-based integration environment for debugging Livy on …

…Kubernetes

.rat-excludes

@@ -30,3 +30,6 @@ logs/*
 docs/**/*.html
 docs/**/JB/**
 venv/*
+dev/helmchart/templates/*.yaml
+dev/helmchart/templates/*.tpl
+dev/helmchart/requirements.lock

dev/helmchart/Chart.yaml

@@ -0,0 +1,31 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+apiVersion: v1
+name: livycluster
+version: 1.0.0
+appVersion: v0.9.0-incubating-SNAPSHOT
+home: https://github.com/askhatri/livycluster/
+description: Apache Livy server to run Spark on Kubernetes
+keywords:
+  - livy
+  - spark
+sources:
+  - https://github.com/askhatri/livycluster/
+maintainers:
+- name: apache
+  email: [email protected]

dev/helmchart/Docker.md

@@ -0,0 +1,81 @@
+## Steps to Create Docker Image for Spark with Python Binding
+
+### 1. Download Spark Binary
+
+Use the following command to download and extract the Spark binary:
+
+```sh
+wget https://archive.apache.org/dist/spark/spark-3.2.3/spark-3.2.3-bin-hadoop3.2.tgz
+tar -xzf spark-3.2.3-bin-hadoop3.2.tgz
+```
+
+### 2. Build and Push Docker Image
+
+Use the following commands to build and push the Docker image:
+
+```sh
+./spark-3.2.3-bin-hadoop3.2/bin/docker-image-tool.sh -r <your_repository> -t v3.2.3 -p kubernetes/dockerfiles/spark/bindings/python/Dockerfile build
+./spark-3.2.3-bin-hadoop3.2/bin/docker-image-tool.sh -r <your_repository> -t v3.2.3 -p kubernetes/dockerfiles/spark/bindings/python/Dockerfile push
+```
+
+Replace `<your_repository>` with your Docker repository name.
+
+## Steps to Create Docker Image for Livy
+
+### 1. Build Livy Code
+
+Build the Livy code using the following commands:
+
+```sh
+cd incubator-livy
+mvn -Pthriftserver -Pscala-2.12 -Pspark3 package
+```
+
+Copy the generated Livy binary into the `/tmp` directory:
+
+```sh
+cp assembly/target/apache-livy-0.9.0-incubating-SNAPSHOT_2.12-bin.zip /tmp
+```
+
+### 2. Create Dockerfile
+
+Create a `Dockerfile` in the `/tmp` directory with the following content:
+
+```Dockerfile
+FROM <your_repository>/spark-py:v3.2.3
+ENV LIVY_VERSION            0.9.0-incubating-SNAPSHOT
+ENV LIVY_PACKAGE            apache-livy-${LIVY_VERSION}_2.12-bin
+ENV LIVY_HOME               /opt/livy
+ENV LIVY_CONF_DIR           /conf
+ENV PATH                    $PATH:$LIVY_HOME/bin
+
+USER root
+
+COPY $LIVY_PACKAGE.zip /
+RUN apt-get update && apt-get install -y unzip && \
+    unzip /$LIVY_PACKAGE.zip -d / && \
+    mv /$LIVY_PACKAGE /opt/ && \
+    rm -rf $LIVY_HOME && \
+    ln -s /opt/$LIVY_PACKAGE $LIVY_HOME && \
+    rm -f /$LIVY_PACKAGE.zip
+
+RUN mkdir /var/log/livy  && \
+    ln -s /var/log/livy $LIVY_HOME/logs
+
+WORKDIR $LIVY_HOME
+
+ENTRYPOINT [ "livy-server" ]
+```
+
+Replace `<your_repository>` with your Docker repository name.
+
+### 3. Build and Push Docker Image
+
+Use the following commands to build and push the Docker image:
+
+```sh
+cd /tmp
+docker build -t <your_repository>/livy:spark3.2.3 .
+docker push <your_repository>/livy:spark3.2.3
+rm -f apache-livy-0.9.0-incubating-SNAPSHOT_2.12-bin.zip
+```

dev/helmchart/README.md

@@ -0,0 +1,76 @@
+# Apache Livy Deployed on Kubernetes using Docker Desktop
+
+This guide provides a Helm chart to deploy Livy on Kubernetes without relying on cloud services like AWS, GCP, or Azure. This setup can save development time and cost, and it allows debugging using an IDE. For debugging Livy on Kubernetes as a standalone setup, Apache Spark and Apache Livy must be deployed in Kubernetes.
+
+## Docker Desktop Installation
+1. Install [Docker Desktop](https://www.docker.com/products/docker-desktop/).
+2. Enable Kubernetes in Docker Desktop settings.
+
+## Helm Installation
+1. Install [Helm](https://helm.sh/docs/intro/install/).
+2. Add the required Helm chart repositories:
+    ```shell
+    helm repo add cert-manager https://charts.jetstack.io              
+    helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
+    helm repo update
+    ```
+
+3. Add an entry to the `/etc/hosts` file:
+    ```text
+    127.0.0.1 my-cluster.example.com
+    ```
+
+4. Install the cert-manager CustomResourceDefinition resources:
+    ```shell
+    kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.15.0/cert-manager.yaml
+    ```
+
+**References:**
+- [cert-manager ACME nginx-ingress tutorial](https://cert-manager.io/docs/tutorials/acme/nginx-ingress/)
+- [Artifact Hub cert-manager Helm chart](https://artifacthub.io/packages/helm/cert-manager/cert-manager)
+
+## Livy Cluster Deployment
+1. Build the Helm chart using the following command:
+    ```bash
+    helm dependency build
+    ```
+
+2. Create a Kubernetes namespace for the Livy deployment:
+    ```bash
+    kubectl create namespace <namespace-name>
+    ```
+
+3. Install the Livy cluster using the Helm chart:
+    ```bash
+    helm -n <namespace-name> install livycluster .
+    ```
+
+## Livy REST API Testing
+1. Create an interactive session:
+    ```shell
+    curl -k -X POST -H "Content-Type: application/json" --data '{"kind": "spark"}' https://my-cluster.example.com/livy/sessions | jq
+    ```
+   **Note:** You need `curl` and `jq` utilities installed on your local machine for testing.
+
+2. Create a statement:
+    ```shell
+    curl -k -X POST -d '{ "kind": "spark", "code": "sc.parallelize(1 to 10).count()" }' -H "Content-Type: application/json" \
+    https://my-cluster.example.com/livy/sessions/0/statements | jq
+    ```
+
+3. Create a batch job:
+    ```shell
+    curl -s -k -H "Content-Type: application/json" \
+        -X POST \
+        -d '{
+            "name": "testbatch1",
+            "className": "org.apache.spark.examples.SparkPi",
+            "numExecutors": 2,
+            "file": "local:///opt/spark/examples/jars/spark-examples_2.12-3.2.3.jar",
+            "args": ["10000"]
+        }' "https://my-cluster.example.com/livy/batches" | jq
+    ```
+
+## Steps to Create Docker Images
+
+Steps to create Docker images for Spark and Livy are documented at [Docker.md](Docker.md).

dev/helmchart/requirements.lock

@@ -0,0 +1,9 @@
+dependencies:
+- name: ingress-nginx
+  repository: https://kubernetes.github.io/ingress-nginx
+  version: 4.10.1
+- name: cert-manager
+  repository: https://charts.jetstack.io
+  version: v1.15.0
+digest: sha256:5c77152ce85e9dc2aadb09976df11286ab53147286cf2ccb93caec0e707bead6
+generated: "2024-06-19T11:16:02.896968+05:30"

dev/helmchart/requirements.yaml

@@ -0,0 +1,32 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+dependencies:
+
+- name: ingress-nginx
+  version: 4.10.1
+  repository: "https://kubernetes.github.io/ingress-nginx"
+  condition: nginx-ingress.enabled,global.nginx-ingress.enabled
+  tags:
+  - ingress
+
+- name: cert-manager
+  version: v1.15.0
+  repository: "https://charts.jetstack.io"
+  condition: cert-manager.enabled,global.cert-manager.enabled
+  tags:
+  - ingress

dev/helmchart/templates/NOTES.txt

@@ -0,0 +1,13 @@
+The Livy server has been installed.
+Check its status by running:
+  kubectl -n {{ .Release.Namespace }} get pods -w
+Connect to the Livy Web UI:
+{{- if .Values.ingress.enabled }}
+    # Open in browser:
+    {{- range .Values.ingress.hosts }}
+    #  https://{{ . }}/livy/ui
+    {{- end }}
+{{- else }}
+    kubectl -n {{ .Release.Namespace }} port-forward {{ include "livycluster.fullname" . }}-0 8998
+    # Open in browser: http://localhost:8998
+{{- end }}

dev/helmchart/templates/_helpers.tpl

@@ -0,0 +1,32 @@
+{{/* vim: set filetype=mustache: */}}
+{{/*
+Expand the name of the chart.
+*/}}
+{{- define "livycluster.name" -}}
+{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" -}}
+{{- end -}}
+
+{{/*
+Create a default fully qualified app name.
+We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec).
+If release name contains chart name it will be used as a full name.
+*/}}
+{{- define "livycluster.fullname" -}}
+{{- if .Values.fullnameOverride -}}
+{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" -}}
+{{- else -}}
+{{- $name := default .Chart.Name .Values.nameOverride -}}
+{{- if contains $name .Release.Name -}}
+{{- .Release.Name | trunc 63 | trimSuffix "-" -}}
+{{- else -}}
+{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" -}}
+{{- end -}}
+{{- end -}}
+{{- end -}}
+
+{{/*
+Create chart name and version as used by the chart label.
+*/}}
+{{- define "livycluster.chart" -}}
+{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" -}}
+{{- end -}}

dev/helmchart/templates/certificate-default.yaml

@@ -0,0 +1,28 @@
+{{- if .Values.defaultCertificate.enabled }}
+apiVersion: cert-manager.io/v1
+kind: Certificate
+metadata:
+  name: {{ include "livycluster.fullname" . }}-default-tls
+  namespace: {{ .Release.Namespace }}
+  labels:
+    app.kubernetes.io/name: {{ include "livycluster.name" . }}
+    helm.sh/chart: {{ include "livycluster.chart" . }}
+    app.kubernetes.io/instance: {{ .Release.Name }}
+    app.kubernetes.io/managed-by: {{ .Release.Service }}
+spec:
+  # The name of the Kubernetes secret resource to store the signed TLS keypair
+  secretName: {{ .Values.defaultCertificate.secretName }}
+  # The Issuer to use for this certificate
+  issuerRef:
+    name: {{ .Values.defaultCertificate.issuer }}
+    kind: {{ .Values.defaultCertificate.issuerKind }}
+  # The common name (CN) for the TLS certificate
+{{- if .Values.defaultCertificate.commonName }}
+  commonName: {{ squote .Values.defaultCertificate.commonName }}
+{{- end }}
+  # A list of domains to include on the TLS certificate
+  dnsNames:
+  {{- range $key, $val := .Values.defaultCertificate.domains }}
+  - {{ squote $val }}
+  {{- end }}
+{{- end }}

dev/helmchart/templates/clusterissuers.yaml

@@ -0,0 +1,18 @@
+{{- $name := include "livycluster.name" . -}}
+{{- $chart := include "livycluster.chart" . -}}
+{{- $releaseName := .Release.Name -}}
+{{- $releaseService := .Release.Service -}}
+{{- range .Values.clusterIssuers }}
+---
+apiVersion: cert-manager.io/v1
+kind: ClusterIssuer
+metadata:
+  name: {{ .name }}
+  labels:
+    app.kubernetes.io/name: {{ $name }}
+    helm.sh/chart: {{ $chart }}
+    app.kubernetes.io/instance: {{ $releaseName }}
+    app.kubernetes.io/managed-by: {{ $releaseService }}
+spec:
+{{ toYaml .spec | indent 2 }}
+{{- end }}

dev/helmchart/templates/config.yaml

@@ -0,0 +1,55 @@
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: cluster-config
+  namespace: {{ .Release.Namespace }}
+data:
+  livy.conf: |-
+    livy.file.local-dir-whitelist=/
+    livy.server.kubernetes.ingress.className=nginx
+    livy.server.kubernetes.ingress.create=true
+    livy.server.kubernetes.ingress.host=my-cluster.example.com
+    livy.server.kubernetes.ingress.protocol=https
+    livy.server.port=8998
+    livy.server.recovery.mode=recovery
+    livy.server.recovery.state-store.url=/tmp/livy/store
+    livy.server.recovery.state-store=filesystem
+    livy.server.session.state-retain.sec = 4h
+    livy.spark.deploy-mode=cluster
+    livy.spark.master=k8s://kubernetes.default.svc.cluster.local:443
+    livy.ui.basePath=/livy
+    livy.ui.history-server-url=https://my-cluster.example.com/historyserver
+  livy-client.conf: |-
+    livy.rsc.client.connect.timeout=600s
+    livy.rsc.rpc.server.address={{ include "livycluster.fullname" . }}-0.{{ include "livycluster.fullname" . }}-headless.{{ .Release.Namespace }}.svc.cluster.local
+    livy.rsc.server.connect.timeout=600s
+  log4j.properties: |-
+    log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n
+    log4j.appender.console.layout=org.apache.log4j.PatternLayout
+    log4j.appender.console.target=System.err
+    log4j.appender.console=org.apache.log4j.ConsoleAppender
+    log4j.logger.org.eclipse.jetty=WARN
+    log4j.rootCategory=DEBUG, console
+  spark-defaults.conf: |-
+    spark.driver.extraJavaOptions=--add-opens java.base/jdk.internal.misc=ALL-UNNAMED -Dio.netty.tryReflectionSetAccessible=true
+    spark.eventLog.dir=file:///tmp/livy/store
+    spark.eventLog.enabled=true
+    spark.executor.extraJavaOptions=--add-opens java.base/jdk.internal.misc=ALL-UNNAMED -Dio.netty.tryReflectionSetAccessible=true
+    spark.kryo.registrationRequired=false
+    spark.kryo.unsafe=false
+    spark.kubernetes.authenticate.driver.serviceAccountName={{ include "livycluster.fullname" . }}-spark
+    spark.kubernetes.container.image.pullPolicy={{ .Values.image.spark.pullPolicy }}
+    spark.kubernetes.container.image={{ .Values.image.spark.repository }}:{{ .Values.image.spark.tag }}
+    spark.kubernetes.driver.label.name=driver
+    spark.kubernetes.driver.volumes.persistentVolumeClaim.datafiles.mount.path=/tmp/livy/store
+    spark.kubernetes.driver.volumes.persistentVolumeClaim.datafiles.options.claimName={{ include "livycluster.fullname" . }}
+    spark.kubernetes.executor.label.name=executor
+    spark.kubernetes.executor.volumes.persistentVolumeClaim.datafiles.mount.path=/tmp/livy/store
+    spark.kubernetes.executor.volumes.persistentVolumeClaim.datafiles.options.claimName={{ include "livycluster.fullname" . }}
+    spark.kubernetes.file.upload.path=file:///tmp/livy/store
+    spark.kubernetes.namespace={{ .Release.Namespace }}
+    spark.network.timeout=800
+    spark.serializer=org.apache.spark.serializer.KryoSerializer
+    spark.speculation=false
+    spark.sql.broadcastTimeout=1200
+    spark.sql.warehouse.dir=/opt/spark/spark-warehouse