Unexpected behavior in sf run apex test

[Cerelius]

Hello, I use a custom GitHub action to execute apex tests in my CI/CD pipeline that wraps the sf/cli and I've just made an update that uncovered some unexpected behavior for me and I wanted to ask if it's intended or not. And, potentially improve my understanding or put eyes on it.

In most asynchronous commands on the sf-cli I'm expecting to enqueue the operation through the command and then be able to reach out to check on the status using the job-id. For example, with a package install request the install is queued and then I am provided an id to let me check the status of the job. It will tell if it's in progress, completed, failed, etc. Providing a wait param will instead poll and wait for me until the job is complete or the wait times out.

I've had to update my action because we found that the --wait parameter on this command does not throw a failure when the waiting period times out. It provides an exit 0 and no indication that it timed out so it provides a false positive when using only the wait parameter.

Okay, to account for that we can just run without a wait and call out to get the current status of the test run. We've done that previously when creating a 2GP package version through a local polling system until the status is complete. However, I found that the sf apex get test command will sit and wait until the tests results are back to return a result. I've only been able to get it to return a complete pass or fail, never an in progress status. I've tried to catch an in-progress status with several different size packages and it's waited until the test completes every time. Curiously, this approach also looks to be significantly faster than running sf apex run test with a wait included.

Running the same set of tests using a test-suite, we received test results in 5-8 minutes using sf apex run test with no wait and immediately requesting sf apex get test. Compared to 18-35 minutes when running sf apex run tests with a sufficient wait time.

There's no blocker present here but the behavior is surprising and appears to contrast other behavior within the CLI in my experience.

Is this intentional?
Is this performance difference real or am I missing something here?

[github-actions]

Hello @Cerelius 👋 It looks like you didn't include the full Salesforce CLI version information in your issue.
Please provide the output of version --verbose --json for the CLI you're using (sf or sfdx).

A few more things to check:

• Make sure you've provided detailed steps to reproduce your issue.
  • A repository that clearly demonstrates the bug is ideal.
• Make sure you've installed the latest version of Salesforce CLI. (docs)
  • Better yet, try the rc or nightly versions. (docs)
• Try running the doctor command to diagnose common issues.
• Search GitHub for existing related issues.

Thank you!

[github-actions]

Thank you for filing this issue. We appreciate your feedback and will review the issue as soon as possible. Remember, however, that GitHub isn't a mechanism for receiving support under any agreement or SLA. If you require immediate assistance, contact Salesforce Customer Support.

[Cerelius]

I've personally observed the described behavior from @salesforce/cli/2.48.6 darwin-arm64 node-v20.15.0 up to latest.

[iowillhoit]

Hey @Cerelius 👋 Lot to unpack here. Going to do my best to summarize it.

• You are saying that the sf package install will exit with a 0 when the timeout is reached? If so, that is a bug and we will fix it.
• We are aware that some of the sf apex commands operate slightly different than our other commands (sync vs async)
  • We have plans to revamp this plugin soon to bring it more inline with our best practices
• I am not seeing any significant differences in execution times with and without the --wait flag with Suites. Here's what I tried:
  • I started with https://github.com/iowillhoit/dreamhouse-lots-o-tests
  • I created 150 (1..150) slow tests with source scripts/generate-slow-tests.sh
  • Created a scratch org
  • Deployed the generated tests
  • I went into Developer Console to create a new Suite
  • Ran sf apex run test --suite-names MySuite --wait 30 several times and noted the execution time
  • Then ran sf apex run test --suite-names MySuite followed by sf apex get test -i THD_ID
  • Compared the times (no really difference)
  • I also compared times without using a test suite (same results)

We will need more details on that one and ideally a repository that allows us to easily reproduce the issue. This might be best to create a new Issue if you are able to get us those details.

Thanks!

[github-actions]

This issue has not received a response in 7 days. It will auto-close in 7 days unless a response is posted.