Cannot write to /opt/jfrog/router/app/router/router.pid: Permission denied

[Momotoculteur]

Hello all,

i have a similar problem like in this thread, which is not resolved

Currently running on EKS v1.28.11 (BottlerocketOS), with Artifactory-ha helm chart on 107.59.12. I read some regression on your changelog of yours chart, i see that i can't update to latest directly. I would like to make that scenario :

• update 1, to - 7.77 or 7.84 appVersion
• update 2, to latest chart

When i apply the first update, i get problem with router micro service on the primary node :

┌──────────────────────────────────────────────────────────────────── Logs(jfrog/artifactory-ha-107-77-14-artifactory-ha-primary-0:router)[tail] ────────────────────────────────────────────────────────────────────┐
│                                                                         Autoscroll:On      FullScreen:Off     Timestamps:Off     Wrap:On                                                                           │
│ Testing directory /var/opt/jfrog/router has read/write permissions for user id 1030                                                                                                                                │
│ Permissions for /var/opt/jfrog/router are good                                                                                                                                                                     │
│ Setting JF_SHARED_NODE_ID to artifactory-ha-107-77-14-artifactory-ha-primary-0                                                                                                                                     │
│ Setting JF_SHARED_NODE_IP to 10.207.57.179                                                                                                                                                                         │
│ Setting JF_SHARED_NODE_NAME to artifactory-ha-107-77-14-artifactory-ha-primary-0                                                                                                                                   │
│ Using default router's certificate and private key                                                                                                                                                                 │
│ ======= Router PID 267                                                                                                                                                                                             │
│ /opt/jfrog/router/app/bin/entrypoint-router.sh: line 112: /opt/jfrog/router/app/router/router.pid: Permission denied                                                                                               │
│ 2024-08-07T17:31:26.235Z [jfrou] [WARN ] [                ] [config_holder.go:325          ] [main                ] [] - Could not resolve config value for key 'router.serviceRegistry.accessUrl', although it wa │
│ s just returned as part of all keys... Ignoring.                                                                                                                                                                   │
│ 2024-08-07T17:31:26.235Z [jfrou] [WARN ] [                ] [config_holder.go:325          ] [main                ] [] - Could not resolve config value for key 'router.serviceRegistry.topologyUrl', although it  │
│ was just returned as part of all keys... Ignoring.                                                                                                                                                                 │
│ 2024-08-07T17:31:26.236Z [jfrou] [FATAL] [648ea037a14a2f64] [config_holder.go:50           ] [main                ] [] - Could not initialize configuration: unable to resolve service registry URL (please config │
│ ure the JFrog Platform Deployment URL explicitly, e.g. using the environment variable: JF_SHARED_JFROGURL)                                                                                                         │
│ Stream closed EOF for jfrog/artifactory-ha-107-77-14-artifactory-ha-primary-0 (router)

The primary never start, other uService & members nodes.

By reading some actuals logs, i have the fealing we already have problem with this router component, but seems that doesn't block the start of our services.

Last question, do you have any migration guide ? Cause when i update the chart, i have new PVC/PV and i can't retrieve my current licence, see that logs on new members when apply updates. As we currently update our licenses through the Artifactory portal, I'm not sure how to add the license without UI, so that we can update without worrying about expired or already-used licenses.

│ artifactory-ha-107-68-22 2024-08-07T20:27:44.512Z [jfrt ] [ERROR] [66c230b9417daaf0] [tifactoryHomeConfigListener:55] [Catalina-utility-2  ] - Failed initializing Home. Caught exception:                         │
│ artifactory-ha-107-68-22 java.lang.IllegalStateException: Converter can't run since no matching license found, please add new license                                                                              │
│ artifactory-ha-107-68-22     at org.artifactory.storage.db.converter.markers.ConverterManagerImpl.handleException(ConverterManagerImpl.java:283)   

Thanks for your help :)

Is this a request for help?:
Yes

Is this a BUG REPORT or FEATURE REQUEST? (choose one):
BUG REPORT

Which chart:
artifactory-ha

Which product license (Enterprise/Pro/oss): Enterprise

JFrog support reference (if already raised with support team):

What happened:
Artifactory is not running on kubernetes

What you expected to happen:
Artifactory UP and running.

How to reproduce it (as minimally and precisely as possible):
install helm chart with mentioned version and smalls values.

Anything else we need to know:
[EDIT] ( not relevant )Our kubernetes clusters prevent apps from writing on filesystem

[shahiinn]

@Momotoculteur :

To have a better understanding of the issue can you please provide us the below details.

1. Can you share the complete values yaml file which you are using for this deployment?
2. Do you have any special security context configurations from the cluster or node?
3. Can you explain little more about "Our kubernetes clusters prevent apps from writing on filesystem".

Further, as you have mentioned you are an Enterprise customer, Can you please open a support ticket to expedite.

Also regarding your query about adding license from chart, you can have it added directly in values.yaml or as a secret.
Please have a look here in default values.yaml

[Momotoculteur]

Hello @shahiinn ,

1. We use theses values

helm {
        release_name = join("-", [local.artifactory_ha_chart_name, local.artifactory_ha_chart_formatted_version])
        value_files  = ["values-small.yaml"]
        values = yamlencode({
          global = {
            joinKey   = data.aws_kms_secrets.artifactory_keys_secret.plaintext["join-key-secret"]
            masterKey = data.aws_kms_secrets.artifactory_keys_secret.plaintext["master-key-secret"]
          }

          ingress = {
            enabled = true
            tls = [{
              hosts = [
                "X.X.net",
                "Y.Y.net",
                "Z.Z.net"
              ]
            }]
            hosts     = ["X.X.net", "Y.Y.net", "Z.Z.net"]
            className = "nginx"
            annotations = {
              "nginx.ingress.kubernetes.io/proxy-body-size" : "0"
            }
          }

          rbac = {
            create = true
          }

          serviceAccount = {
            create = true
            name   = local.sa_name
            annotations = {
              "eks.amazonaws.com/role-arn" : module.artifactory_role.iam_role_arn
            }
          }

          postgresql = {
            enabled = false
          }

          database = {
            type   = "postgresql"
            driver = "org.postgresql.Driver"
            secrets = {
              user = {
                name = kubernetes_secret.artifactory_ha_database_secret.metadata[0].name
                key  = "db-user"
              }
              password = {
                name = kubernetes_secret.artifactory_ha_database_secret.metadata[0].name
                key  = "db-password"
              }
              url = {
                name = kubernetes_secret.artifactory_ha_database_secret.metadata[0].name
                key  = "db-url"
              }
            }
          }

          artifactory = {
            openMetrics = {
              enabled = true
            }
            name = "artifactory-ha-${local.artifactory_ha_chart_formatted_version}"
            migration = {
              enabled = false
            }
            admin = {
              username = "betclic-admin"
              password = data.aws_kms_secrets.artifactory_ha_admin_secret.plaintext["artifactory-ha-admin-secret"]
            }
            license = {
              licenseKey = local.licenses
            }
            storageClassName = "gp3"

            persistence = {
              type = "aws-s3-v3"
              awsS3V3 = {
                testConnection = true
                region         = local.aws_region
                bucketName     = module.artifactory_bucket.bucket_name
                endpoint       = "s3.${local.aws_region}.amazonaws.com"
              }
            }

            topologySpreadConstraints = [{
              maxSkew           = 1
              topologyKey       = "topology.kubernetes.io/zone"
              whenUnsatisfiable = "ScheduleAnyway"
            }]

            service = {
              name = "artifactory-${local.artifactory_ha_chart_formatted_version}"
            }

            primary = {
              replicaCount = 1
              minAvailable = 1

              nodeSelector = {
                "karpenter.sh/provisioner-name" = local.karpenter_provisionner_name.infraOnDemand
              }

              tolerations = [{
                key      = local.node_taints.label
                value    = local.node_taints.values.infraOnDemand
                effect   = "NoSchedule"
                operator = "Equal"
              }]

            }
            node = {
              replicaCount = 2
              minAvailable = 2
              resources = {
                requests = {
                  cpu    = "3.5"
                  memory = "8Gi"
                }
                limits = {
                  cpu    = "5"
                  memory = "8Gi"
                }
              }
              nodeSelector = {
                "karpenter.sh/provisioner-name" = local.karpenter_provisionner_name.infraOnDemand
              }
              tolerations = [{
                key      = local.node_taints.label
                value    = local.node_taints.values.infraOnDemand
                effect   = "NoSchedule"
                operator = "Equal"
              }]
            }
          }

          nginx = {
            enabled = false
          }

          filebeat = {
            enabled = false
          }

          mc = {
            enabled = false
          }

          observability = {
            enabled = false
          }

          integration = {
            enabled = false
          }

          jfconnect = {
            enabled = false
          }

          event = {
            enabled = false
          }

          metadata = {
            enabled = false
          }

          access = {
            enabled = false
          }

        })
      }
    }

2. Not to my knowledge...

3. Mybad, i just copied template from another issue which was the same as us.

We are an enterprise customer, but Jfrog team aswer that we have not access to your support with our current licence.

[Momotoculteur]

hi,
my bad, i have a support. Do i need to see with them directly ?

[Momotoculteur]

Hey,

Some news on that issue, that i was able to resolve myself, the support was reactive and nice but do not have idea to fix that 🥲. Here a list of what i did to fix that, if someone else have this issue.

Problem of user permission which can't have permission to read/write on specific folder was fixed by set that env var : JFROG_SHARED_URL. Logs are not optimal cause i was able to fix that, but that has nothing to do with a specific env var..

We were asked to put that value JFROG_SHARED_URL to http://<your_jfrog_url_platform>. My previous version of Artifactory doesn't have that env var setted, and i didn't see any breaking change on that in documentation. Maybe i miss something about that ? By default you set that value to http://localhost:8046, is that normal ?
But that can't work because the service router is not up and running, so we had error with 503. By listening on artifactory container, which service was up & running and on which port they are listening, we set JFROG_SHARED_URL to http://localhost:8040, and that work. 🥳