Unable to upgrade from 5.1.0 to 5.4.3 using mongodb 4.4

[TLINDEN]

Howdy,

my deployment is unfortunately a little outdated, so I thaught it might be a good idea to upgrade rocketchat first, keep mongodb version and upgrade that later.

So I specified helm chart version 5.1.0 but mongodb image tag 4.4.11-debian-10-r12 and deployed.

The chart then tried to deploy an upgrade pod, which failed to start:

97s         Warning   FailedMount              pod/rocketchat-fitstest-rocketchat-pre-upgrade                \
 MountVolume.SetUp failed for volume "mongodb-script" : configmap "rocketchat-fitstest-rocketchat-scripts" not found

The problem was, that said configmap first appeared in 5.4.3, so it's not installed here, but the pre-upgrade pod just expects it to be there.

My next try was to just manually deploy that script (this would be required only once anyway). Now the pre-upgrade pod came up, but failed with:

sh: 1: mongosh: not found

because mongosh seems not to be installed in the 4.4 image.

My next try will be to directly upgrade to mongodb 5.0 and see what happens. But I think this behavior should be fixed anyway somehow.

[TLINDEN]

Ok, I tried the same process as above but using mongodb 5.0. That failed as well. First it had the same problem with the non-existing configmap. So I added it now to my deployment.

However, now the pre-update pod fails to execute:

% k logs rocketchat-fitstest-rocketchat-pre-upgrade
Current Mongosh Log ID: 64117dd3c10c36597a8d184c
Connecting to:          mongodb://<credentials>@rocketchat-fitstest-mongodb-headless:27017/rocketchat?replicaSet=rs0&appName=mongosh+1.8.0
Using MongoDB:          4.4.11
Using Mongosh:          1.8.0

For mongosh info see: https://docs.mongodb.com/mongodb-shell/


To help improve our products, anonymous usage data is collected and sent to MongoDB periodically (https://www.mongodb.com/legal/privacy-policy).
You can opt-out by running the disableTelemetry() command.

Loading file: /tmp/verifyMongodb.js
MongoServerError: not authorized on admin to execute command { getParameter: 1, featureCompatibilityVersion: 1, lsid: { id: UUID("0d4d5821-8149-4857-89e5-312ea8c2a2b0") }, $clusterTime: { clusterTime: Timestamp(1678867922, 1), signature: { hash: BinData(0, A18E69A731CE183BF26858E13F89D25DBA829ABD), keyId: 7210672631631052805 } }, $db: "admin" }

The reason is, that the user rocketchat is no admin user, there's only root:

rs0:PRIMARY> use admin
switched to db admin
rs0:PRIMARY> db.getUsers()
[
        {
                "_id" : "admin.root",
                "userId" : UUID("dcd90659-2364-4ed9-b446-15a511a96511"),
                "user" : "root",
                "db" : "admin",
                "roles" : [
                        {
                                "role" : "root",
                                "db" : "admin"
                        }
                ],
                "mechanisms" : [
                        "SCRAM-SHA-1",
                        "SCRAM-SHA-256"
                ]
        }
]

So, how is this supposed to work?

[TLINDEN]

Next iteration:

I've setup a test pod using the same image as the pre-update pod:

---
apiVersion: v1
kind: Pod
metadata:
  annotations:
  name: test
spec:
  containers:
  - command:
    - sleep
    - "1010101010101010"
    image: docker.io/bitnami/mongodb:5.0.15-debian-11-r6
    imagePullPolicy: IfNotPresent
    name: mongosh

logged into it, created the /tmp/verifyMongodb.js file, configured the proper connection string and tried to execute what the pre-update pod does:

I have no name!@test:/$ export MONGODB_HOST="mongodb://root:******@rocketchat-fitstest-mongodb-headless:27017/rocketchat?replicaSet=rs0"
I have no name!@test:/$ mongosh "$MONGODB_HOST" /tmp/verifyMongodb.js 5.0.15-debian-11-r6
Current Mongosh Log ID: 64119862238d764edda38ebd
Connecting to:          mongodb://<credentials>@rocketchat-fitstest-mongodb-headless:27017/rocketchat?replicaSet=rs0&appName=mongosh+1.8.0
MongoServerError: Authentication failed.

However, on the mongodb pod I actually CAN use the root user:

kubectl exec -it rocketchat-fitstest-mongodb-0 -- mongo -u root -p ****** --host localhost
Defaulted container "mongodb" out of: mongodb, volume-permissions (init)
MongoDB shell version v4.4.11
connecting to: mongodb://localhost:27017/?compressors=disabled&gssapiServiceName=mongodb
Implicit session: session { "id" : UUID("1ce823d0-226f-41ba-b442-7a38c9c5da0a") }
MongoDB server version: 4.4.11
---
The server generated these startup warnings when booting: 
        2023-03-15T07:48:47.794+00:00: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem
        2023-03-15T07:48:48.732+00:00: You are running on a NUMA machine. We suggest launching mongod like this to avoid performance problems: numactl --interleave=all mongod [other options]
        2023-03-15T07:48:48.732+00:00: /sys/kernel/mm/transparent_hugepage/enabled is 'always'. We suggest setting it to 'never'
---
---
        Enable MongoDB's free cloud-based monitoring service, which will then receive and display
        metrics about your deployment (disk utilization, CPU, operation statistics, etc).

        The monitoring data will be available on a MongoDB website with a unique URL accessible to you
        and anyone you share the URL with. MongoDB may use this information to make product
        improvements and to suggest MongoDB products and deployment options to you.

        To enable free monitoring, run the following command: db.enableFreeMonitoring()
        To permanently disable this reminder, run the following command: db.disableFreeMonitoring()
---
rs0:PRIMARY> use admin
switched to db admin
rs0:PRIMARY> db.getUsers()
[
        {
                "_id" : "admin.root",
                "userId" : UUID("dcd90659-2364-4ed9-b446-15a511a96511"),
                "user" : "root",
                "db" : "admin",
                "roles" : [
                        {
                                "role" : "root",
                                "db" : "admin"
                        }
                ],
                "mechanisms" : [
                        "SCRAM-SHA-1",
                        "SCRAM-SHA-256"
                ]
        }

]

But even IF that would work, the helm chart doesn't use the root user, but the rocketchat user, which doesn't even exist. Since I found no way to modify the chart to use root I tried just adding the rocketchat user as admin using the password defined in the deployment.

        {
                "_id" : "admin.rocketchat",
                "userId" : UUID("546c47df-9043-48de-bad0-3f3e8f08d7af"),
                "user" : "rocketchat",
                "db" : "admin",
                "roles" : [
                        {
                                "role" : "root",
                                "db" : "admin"
                        }
                ],
                "mechanisms" : [
                        "SCRAM-SHA-1",
                        "SCRAM-SHA-256"
                ]
        },

But even THEN it doesn't work:

I have no name!@test:/$ export MONGODB_HOST="mongodb://rocketchat:******@rocketchat-fitstest-mongodb-headless:27017/rocketchat?replicaSet=rs0"
I have no name!@test:/$ mongosh "$MONGODB_HOST" /tmp/verifyMongodb.js 5.0.15-debian-11-r6
Current Mongosh Log ID: 64119a54ef9d794cfe1dfb75
Connecting to:          mongodb://<credentials>@rocketchat-fitstest-mongodb-headless:27017/rocketchat?replicaSet=rs0&appName=mongosh+1.8.0
Using MongoDB:          4.4.11
Using Mongosh:          1.8.0

For mongosh info see: https://docs.mongodb.com/mongodb-shell/


To help improve our products, anonymous usage data is collected and sent to MongoDB periodically (https://www.mongodb.com/legal/privacy-policy).
You can opt-out by running the disableTelemetry() command.

Loading file: /tmp/verifyMongodb.js
MongoServerError: not authorized on admin to execute command { getParameter: 1, featureCompatibilityVersion: 1, lsid: { id: UUID("cc669895-7271-4f39-bf1f-ba2ff67aa1cf") }, $clusterTime: { clusterTime: Timestamp(1678875213, 1), signature: { hash: BinData(0, 3D14859A65F46FF974E9BBE6649C8FEA30315A7D), keyId: 7210672631631052805 } }, $db: "admin" }

So, please help me, the whole thing looks hopeless to me!

Thanks in advance,
Tom

[TLINDEN]

PS: I found a way to get the verify Script executed:

I have no name!@test:/$ export MONGODB_HOST="mongodb://root:******@rocketchat-fitstest-mongodb-headless:27017/admin?replicaSet=rs0"
I have no name!@test:/$ mongosh "$MONGODB_HOST" /tmp/verifyMongodb.js 5.0.15-debian-11-r6; echo $?
Current Mongosh Log ID: 64119c81411695734767a2d1
Connecting to:          mongodb://<credentials>@rocketchat-fitstest-mongodb-headless:27017/admin?replicaSet=rs0&appName=mongosh+1.8.0
Using MongoDB:          4.4.11
Using Mongosh:          1.8.0

For mongosh info see: https://docs.mongodb.com/mongodb-shell/

------
   The server generated these startup warnings when booting
   2023-03-15T07:48:47.794+00:00: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem
   2023-03-15T07:48:48.732+00:00: You are running on a NUMA machine. We suggest launching mongod like this to avoid performance problems: numactl --interleave=all mongod [other options]
   2023-03-15T07:48:48.732+00:00: /sys/kernel/mm/transparent_hugepage/enabled is 'always'. We suggest setting it to 'never'
------

------
   Enable MongoDB's free cloud-based monitoring service, which will then receive and display
   metrics about your deployment (disk utilization, CPU, operation statistics, etc).
   
   The monitoring data will be available on a MongoDB website with a unique URL accessible to you
   and anyone you share the URL with. MongoDB may use this information to make product
   improvements and to suggest MongoDB products and deployment options to you.
   
   To enable free monitoring, run the following command: db.enableFreeMonitoring()
   To permanently disable this reminder, run the following command: db.disableFreeMonitoring()
------

Loading file: /tmp/verifyMongodb.js
0

The important thing here is that I am not using the rocketchat database but admin. This even works with the rocketchat user (if you give it admin permissions, see above).

[TLINDEN]

It continues:

Now that I know that in principle upgrading mongodb from 4.4 to 5.0 seems to be supported and since I deployed the configmap myself anyway, I replaced the JS code with a bare quit(0). That way now the upgrade runs through, at least for mongodb.

However, now I'm getting this error from the rocketchat pod:

Some indexes for collection 'rocketchat_credential_tokens' could not be created:
     An equivalent index already exists with the same name but different options. Requested index: { v: 2, key: { expireAt: 1 }, name: "expireAt_1", sparse: true, expireAfterSeconds: 0 }, existing index: { v: 2, key: { expireAt: 1 }, name: "expireAt_1", sparse: 1, expireAfterSeconds: 0 }
LocalStore: store created at 
LocalStore: store created at 
LocalStore: store created at 
Error creating index: livechat_priority -> { name: 1 } {
  unique: true,
  partialFilterExpression: { '$and': [ [Object], [Object] ] }
} MongoServerError: An equivalent index already exists with the same name but different options. Requested index: { v: 2, unique: true, key: { name: 1 }, name: "name_1", partialFilterExpression: { $and: [ { name: { $exists: true } }, { name: { $gt: "" } } ] } }, existing index: { v: 2, unique: true, key: { name: 1 }, name: "name_1" }
    at Connection.onMessage (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/cmap/connection.js:230:30)
    at MessageStream.<anonymous> (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/cmap/connection.js:61:60)
    at MessageStream.emit (events.js:400:28)
    at MessageStream.emit (domain.js:475:12)
    at processIncomingData (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/cmap/message_stream.js:125:16)
    at MessageStream._write (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/cmap/message_stream.js:33:9)
    at writeOrBuffer (internal/streams/writable.js:358:12)
    at MessageStream.Writable.write (internal/streams/writable.js:303:10)
    at Socket.ondata (internal/streams/readable.js:731:22)
    at Socket.emit (events.js:400:28)
    at Socket.emit (domain.js:475:12)
    at addChunk (internal/streams/readable.js:293:12)
    at readableAddChunk (internal/streams/readable.js:267:9)
    at Socket.Readable.push (internal/streams/readable.js:206:10)
    at TCP.onStreamRead (internal/stream_base_commons.js:188:23)
    at TCP.callbackTrampoline (internal/async_hooks.js:130:17) {
  ok: 0,
  code: 85,
  codeName: 'IndexOptionsConflict',
  '$clusterTime': {
    clusterTime: new Timestamp({ t: 1678876192, i: 3 }),
    signature: {
      hash: new Binary(Buffer.from("8ef9a22c7f50037d8e65fdab1d0b947fbe11c3ce", "hex"), 0),
      keyId: new Long("7210672631631052805")
    }
  },
  operationTime: new Timestamp({ t: 1678876192, i: 3 }),
  [Symbol(errorLabels)]: Set(0) {}
}
strict mode: use allowUnionTypes to allow union type keyword at "#/properties/value" (strictTypes)

However - rocketchat comes up anyway without further errors and seems to work.