Finding the Metadata Limits for my project

[jhgihub0]

For my projects, I am trying to determine how close I am the Metadata Limits with REST API

I am running these commands locally on my Mac

export SF_MDAPI_TEMP_DIR = ./mydeploy
sf org login web --alias OrgAlias --instance-url https://test.salesforce.com
sf project deploy start --target-org OrgAlias --wait 120 --test-level "NoTestRun" --source-dir ./force-app --dry-run 
sf org logout --target-org OrgAlias --no-prompt

I am using the CLI

{
  "architecture": "darwin-arm64",
  "cliVersion": "@salesforce/cli/2.46.6",
  "nodeVersion": "node-v20.14.0",
  "osVersion": "Darwin 23.5.0",
  "rootPath": "/Users/work/.local/share/sf/client/2.46.6-191291b",
  "shell": "zsh",
  "pluginVersions": [
    "@oclif/plugin-autocomplete 3.1.2 (core)",
    "@oclif/plugin-commands 4.0.2 (core)",
    "@oclif/plugin-help 6.2.1 (core)",
    "@oclif/plugin-not-found 3.2.3 (core)",
    "@oclif/plugin-plugins 5.2.4 (core)",
    "@oclif/plugin-search 1.1.2 (core)",
    "@oclif/plugin-update 4.3.5 (core)",
    "@oclif/plugin-version 2.2.3 (core)",
    "@oclif/plugin-warn-if-update-available 3.1.5 (core)",
    "@oclif/plugin-which 3.2.2 (core)",
    "@salesforce/cli 2.46.6 (core)",
    "apex 3.1.18 (core)",
    "auth 3.6.21 (core)",
    "data 3.4.6 (core)",
    "deploy-retrieve 3.9.4 (core)",
    "info 3.3.6 (core)",
    "limits 3.3.12 (core)",
    "marketplace 1.2.12 (core)",
    "org 4.2.1 (core)",
    "packaging 2.4.6 (core)",
    "schema 3.3.11 (core)",
    "settings 2.3.1 (core)",
    "sobject 1.4.11 (core)",
    "source 3.4.2 (core)",
    "telemetry 3.4.0 (core)",
    "templates 56.2.11 (core)",
    "trust 3.7.5 (core)",
    "user 3.5.13 (core)"
  ]
}

I am trying to plan ahead and prevent from ever getting the "INVALID_OPERATION: Too many files in zip"

I want to be able to see the following:

1. How many files my projects attempt to deploy to Salesforce
2. The actual zip file being deployed to Salesforce.

I have already set export SF_MDAPI_TEMP_DIR = ./mydeploy, so I can view the number of files. Where can I go to see the zip file that sf project deploy start generates?

[jhgihub0]

Also see #2943

[WillieRuemmele]

Hey @jhgihub0 - that directory is what is zipped and sent to the MDAPI. The CLI doesn't expose any options to save that as a .zip because we never actually produce one, we use streams and buffers to send that data to the API to save local memory. The library we use uses the JSZip library to help us with that here... if you really want to get a zip, you could use the library, with our options, and save the result

[jhgihub0]

Thank you @WillieRuemmele

Is it then possible to just display the information in the log somewhere? Maybe have a --showzip option, for example:

sf project deploy start --showzip

... that can display the information in the console

• Zip File size
• Number of files

[jhgihub0]

@WillieRuemmele

ls -R md | wc -l seems to count files and folders. Is the 10,000 REST limit files only, or is it files and folders?

[WillieRuemmele]

hmmm, I'm not 100% sure, but since some metadata require folders, or are defined by their folders, I'd say both? but that's a hunch

[jhgihub0]

After some testing, it seems to be a files and folders. I was able to reproduce the error by having a large repo, and reducing the number of files and folders to be less than 10,000

Logged in
sf org login web --alias OrgAlias --instance-url https://test.salesforce.com

Generated my deployment folder
sf project convert source --source-dir ./sfdx-source --output-dir mydeploy

Counted the files
find mydeploy -type f | wc -l

Counted the folders
find mydeploy -type d | wc -l

Cleaned up and logged out
rm -rf mydeploy
sf org logout --target-org OrgAlias --no-prompt

[WillieRuemmele]

so I think we've been able to find a soltuion to some of your questions... I'm still wondering about your use case, and if a --save-zip flag on project convert source wouldn't be better? Maybe it'd output something like

converted 223/223 components with 250/10000 file limit
saved zip to : myFile.zip

[jhgihub0]

Yes, a --save-zip flag on sf project convert source would do the trick

[VivekMChawla]

@jhgihub0 - My name is Vivek Chawla. I'm the Product Manager for the Salesforce CLI.

Thank you for your detailed feature request and your brainstorming with @WillieRuemmele.

We're considering adding functionality that would deliver the following outcomes. Please help us by validating the utility of these outcomes and letting us know if they would make your job easier if we delivered them.

Proposed Outcomes

1. Without requiring a deployment, determine/predict the file count and compressed+encoded size of the MDAPI deployment ZIP for each of the following:
   • The entire SFDX project
   • Scoped SFDX project contents (e.g., all Apex classes)
   • The source deployed to a build org during package version create
2. If requested, save the MDAPI deployment ZIP to a location specified by the CLI user.
3. Report the MDAPI file count/ZIP size after successful deployments as human-readable or JSON output.
4. Warn the CLI user if the MDAPI file count/ZIP size approach current MDAPI limits.
5. Include the MDAPI file count/ZIP size in the CLI logs.
6. Update the CLI doctor to:
   • Provide a clear thumbs up/down diagnosis regarding MDAPI limits for the current SFDX project.
   • Identify components contributing most and least to hitting MDAPI limits.
   • Recommend strategies for managing large deployments within MDAPI limits.

Thank you!

[jhgihub0]

@VivekMChawla please let me know when this will be available. Thank you.

[shetzel]

@jhgihub0 - part of this is in the CLI already. Before sending the deploy request the CLI output displays the number of files in the zip and the size of the zip. More to come very soon, but that part is there now.

[jhgihub0]

That is great. Thank you!!!

[shetzel]

Forgot to mention that the zip size and zip file count are currently displayed for deploys when the --verbose flag is used. I have a change about ready to go in that adds a warning when a certain threshold is exceeded. Default will be 80% of the Metadata API limits but is overridable with an env var.

[jhgihub0]

Great. Please let me know the version once its live.

[jshackell-sfdc]

New warning and env var (SF_DEPLOY_SIZE_THRESHOLD) released in version 2.62.6 (October 16, 2024). Docs updated: https://developer.salesforce.com/docs/atlas.en-us.sfdx_setup.meta/sfdx_setup/sfdx_dev_cli_env_variables.htm. Enjoy!