sfdx org create scratch command error "Cannot destructure property 'hubBaseUrl' of 'cache.get(...)' as it is undefined."

[HyruleConsulting]

Summary

I have a process that builds scratch orgs overnight for developers which has been running a while now however I would say roughly 10% of the time the Scratch Org Creation fails with the message
Cannot destructure property 'hubBaseUrl' of 'cache.get(...)' as it is undefined.

It's hasn't been too much of an issue as it only happens for 1 in every 10 scratch org creations however since the weekend (Winter24) I have seen a sharp uptick in this failure rate. It's more like 40-50% of the scratch orgs I create will now fail with this message.

It's worth noting that I use Org Shape in this process, and I have also recently had to rebuild the org shape following the Salesforce Major Release (Winter24).

Steps To Reproduce

sfdx org create scratch --target-dev-hub *** --definition-file <path to definition file> --duration-days 30 --async --json
sfdx org resume scratch --job-id <Scratch Org Job Id> --json
Receive error message Cannot destructure property 'hubBaseUrl' of 'cache.get(...)' as it is undefined about 40-50% of the time.

Expected result

Should show a status message either Scratch Org Created or Scratch Org Failed

Actual result

40-50% of the time it shows an unhandled exception
{ "code": 1, "context": "EnvResumeScratch", "commandName": "EnvResumeScratch", "message": "Cannot destructure property 'hubBaseUrl' of 'cache.get(...)' as it is undefined.", "name": "TypeError", "status": 1, "stack": "TypeError: Cannot destructure property 'hubBaseUrl' of 'cache.get(...)' as it is undefined.\n at EnvResumeScratch.run (/Users/**/.local/share/sf/client/2.12.9-007e29c/node_modules/@salesforce/plugin-org/lib/commands/org/resume/scratch.js:25:17)\n at async EnvResumeScratch._run (/Users/**/.local/share/sf/client/2.12.9-007e29c/node_modules/@oclif/core/lib/command.js:117:22)\n at async Config.runCommand (/Users/**/.local/share/sf/client/2.12.9-007e29c/node_modules/@oclif/core/lib/config/config.js:314:25)\n at async run (/Users/**/.local/share/sf/client/2.12.9-007e29c/node_modules/@oclif/core/lib/main.js:89:16)", "exitCode": 1, "warnings": [] }

System Information

Process runs on linux agents and authentication to DevHub is handled with sfdx org login jwt command.

{
    "cliVersion": "sfdx-cli/7.208.10",
    "architecture": "linux-x64",
    "nodeVersion": "node-v19.9.0",
    "osVersion": "Linux 5.15.0-1042-azure",
    "shell": "sh",
    "rootPath": "/usr/lib/node_modules/sfdx-cli",
    "pluginVersions": [
        "@oclif/plugin-autocomplete 2.3.1 (core)",
        "@oclif/plugin-commands 2.2.16 (core)",
        "@oclif/plugin-help 5.2.10 (core)",
        "@oclif/plugin-not-found 2.3.26 (core)",
        "@oclif/plugin-plugins 3.1.5 (core)",
        "@oclif/plugin-search 0.0.17 (core)",
        "@oclif/plugin-update 3.1.21 (core)",
        "@oclif/plugin-version 1.3.5 (core)",
        "@oclif/plugin-warn-if-update-available 2.0.40 (core)",
        "@oclif/plugin-which 2.2.22 (core)",
        "apex 2.3.3 (core)",
        "auth 2.8.4 (core)",
        "community 2.3.3 (core)",
        "custom-metadata 2.1.28 (core)",
        "data 2.4.2 (core)",
        "deploy-retrieve 1.13.3 (core)",
        "info 2.6.24 (core)",
        "limits 2.3.22 (core)",
        "org 2.9.17 (core)",
        "packaging 1.20.1 (core)",
        "schema 2.3.15 (core)",
        "settings 1.4.15 (core)",
        "signups 1.4.22 (core)",
        "source 2.10.19 (core)",
        "telemetry 2.2.1 (core)",
        "templates 55.5.0 (core)",
        "trust 2.4.24 (core)",
        "user 2.3.19 (core)",
        "sfdx-cli 7.208.10 (core)"
    ]
}

Let me know if there's anything I can do to reduce the failure rate

[github-actions]

Hello @HyruleConsulting 👋

It looks like you're using an outdated version of Salesforce CLI. sfdx (v7) is in "maintenance mode" as of July 12, 2023. We highly recommend you move from sfdx (v7) to sf (v2) ASAP.

Moving to sf (v2) is easy and takes just two commands. Find all the information here.

After you move to the latest version of sf (v2), run your command again and provide the output of sf version --verbose --json.

[github-actions]

Thank you for filing this issue. We appreciate your feedback and will review the issue as soon as possible. Remember, however, that GitHub isn't a mechanism for receiving support under any agreement or SLA. If you require immediate assistance, contact Salesforce Customer Support.

[cristiand391]

@HyruleConsulting are you running these commands in parallel?

[HyruleConsulting]

@cristiand391 Yes, Scratch orgs are built on parallel agents in this scenario

[cristiand391]

there's known issue when running commands that read/write the config file in parallel, see:
#2423 (comment)

from the stacktrace I see org resume scratch fails when it tries to get the hubBaseUrl from the cache created by org create scratch --async (could be b/c the file is in a bad state when it tries to read it).

[HyruleConsulting]

Ok thanks @cristiand391 this makes sense. I do try to space out agent invocation (I use 5 minute delays) when bulk building scratch orgs overnight which does help - What I might do is create a single CI service for Scratch Org building which guarantees an order of invocations to avoid the command being used in parallel. Alternatively, I could build a re-attempt loop when faced with the hubBaseUrl error.

[cristiand391]

Sounds good, added configFile to track these issues (my teammate is currently working on a fix for this, we'll ping you when it's on the RC if you want to give it a try).

[HyruleConsulting]

Sure sounds good, thanks @cristiand391

[mshanemc]

@HyruleConsulting the nightly release has a very large update to how configFile works to help with this.

Are you able to try your repro with the new version?

[HyruleConsulting]

@HyruleConsulting the nightly release has a very large update to how configFile works to help with this.

Are you able to try your repro with the new version?

Sorry @mshanemc I missed your comment before requesting to test this out. Looks like you’ve merged the feature now, thanks for delivering this! 👍