Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Permanent Session not creating #738

Open
jhaswati opened this issue Nov 9, 2023 · 1 comment
Open

Permanent Session not creating #738

jhaswati opened this issue Nov 9, 2023 · 1 comment

Comments

@jhaswati
Copy link

jhaswati commented Nov 9, 2023

      containers:
      - env:
        - name: LIGHTER_KUBERNETES_ENABLED
          value: "true"
        - name: LIGHTER_MAX_RUNNING_JOBS
          value: "15"
        - name: LIGHTER_KUBERNETES_SERVICE_ACCOUNT
          value: lighter
        - name: LIGHTER_URL
          value: http://lighter-lighter-chart-lighter.sf.svc.cluster.local:8080
        - name: KUBERNETES_CLUSTER_DOMAIN
          value: cluster.local
        - name: LIGHTER_SESSION_TIMEOUT_MINUTES
          value: "90000"
        - name: LIGHTER_SESSION_PERMANENT_SESSIONS
          valueFrom:
            configMapKeyRef:
              key: LIGHTER_SESSION_PERMANENT_SESSIONS
              name: lighter-session
        image: ghcr.io/exacaster/lighter:0.0.45-spark3.4.0
        imagePullPolicy: IfNotPresent

I have added the above env in the lighter deployment manifest. Below is the configmap:

Namespace:    sf
Labels:       <none>
Annotations:  <none>

Data
====
LIGHTER_SESSION_PERMANENT_SESSIONS:
----
[
  {
    "id": "permanent-id-used-on-api-calls",
    "submit-params": {
      "name": "Session Name",
      "numExecutors": 1,
      "executorCores": 1,
      "executorMemory": "1G",
      "driverCores": 1,
      "driverMemory": "1G"
    }
  }
]



BinaryData
====

Events:  <none>

However I dont see any lighter sessions getting created. How should I pass the value permanent session in the lighter deployment manifest.

@Minutis
Copy link
Member

Minutis commented Nov 13, 2023

Hello,

it looks like current documentation is incorrect and it's not possible to pass LIGHTER_SESSION_PERMANENT_SESSIONS at the moment.

As a workaround you could provide whole Lighter application config using following example:

                        -   name: MICRONAUT_APPLICATION_JSON
                            value: '{"lighter":{"session":{"permanent-sessions":[{"id":"permanentId2","submit-params":{"driverCores":"1","executorCores":"1","conf":{"spark.kubernetes.container":"<...>"}}}}}}'

We will look into the approached you used later and fix the documentation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants