[BUG] Violations are sometimes not written to the out-file

[painlesshippo]

Description:

We run scanner run dfa as part of our CI. We noticed that sometimes the command will not write the violations that it found to the out-file. We can see in the logs that the violations are found as they are many instances of "...adding X new entries...". However once the scanner has finished, it outputs "Executed sfge, found 0 violation(s) across 0 file(s)." and the scan results file is empty.

Running the same commands locally, on the same code, always writes the violations to the out-file correctly. This indicates a problem with our pipelines... however we have yet to find the issue and when we compare the logs produced locally, they are essentially identical to the ones produced by our CI...

It's as if, sometimes, the scanner doesn't write to the out-file, but no error is produced in the logs and the only difference we can find is that when it occurs, the violations were found, but not written to the out-file without any warning or error

Documentation:

Good, normal, run

good-sfge-log.txt
As you can see, the errors were written to the file & the scanner raised an error correctly

Analyzing with Salesforce Graph Engine. See /home/AzDevOps/.sfdx-scanner/sfge.log for details.... Please waitAnalyzing with Salesforce Graph Engine. See /home/AzDevOps/.sfdx-scanner/sfge.log for details.... Compiled 23 files.Analyzing with Salesforce Graph Engine. See /home/AzDevOps/.sfdx-scanner/sfge.log for details.... Building graph.Analyzing with Salesforce Graph Engine. See /home/AzDevOps/.sfdx-scanner/sfge.log for details.... Added all compilation units to graph.Analyzing with Salesforce Graph Engine. See /home/AzDevOps/.sfdx-scanner/sfge.log for details.... Identified 14 path entry point(s).Analyzing with Salesforce Graph Engine. See /home/AzDevOps/.sfdx-scanner/sfge.log for details.... Detected 2 violation(s) from 11 path(s) on 6/14 entry point(s).Analyzing with Salesforce Graph Engine. See /home/AzDevOps/.sfdx-scanner/sfge.log for details.... Detected 2 violation(s) from 24 path(s) on 11/14 entry point(s).Analyzing with Salesforce Graph Engine. See /home/AzDevOps/.sfdx-scanner/sfge.log for details.... Detected 4 violation(s) from 35 path(s) on 13/14 entry point(s).Analyzing with Salesforce Graph Engine. See /home/AzDevOps/.sfdx-scanner/sfge.log for details.... Overall, analyzed 39 path(s) from 14 entry point(s). Detected 4 violation(s).Analyzing with Salesforce Graph Engine. See /home/AzDevOps/.sfdx-scanner/sfge.log for details.... Done
Loaded Custom Settings: [ none found ]
Loaded Apex Controllers: [ none found ]
Error (1): Executed sfge, found 4 violation(s) across 2 file(s).
Rule violations of severity 1 or more severe were detected.
Rule violations were written to Tests/sf-scan-dfa-results-1.xml.

Problematic Run

problem-sfge-log.txt
For this log, the scanner outputted:

Analyzing with Salesforce Graph Engine. See /home/AzDevOps/.sfdx-scanner/sfge.log for details.... Please waitAnalyzing with Salesforce Graph Engine. See /home/AzDevOps/.sfdx-scanner/sfge.log for details.... Compiled 23 files.Analyzing with Salesforce Graph Engine. See /home/AzDevOps/.sfdx-scanner/sfge.log for details.... Building graph.Analyzing with Salesforce Graph Engine. See /home/AzDevOps/.sfdx-scanner/sfge.log for details.... Added all compilation units to graph.Analyzing with Salesforce Graph Engine. See /home/AzDevOps/.sfdx-scanner/sfge.log for details.... Identified 14 path entry point(s).Analyzing with Salesforce Graph Engine. See /home/AzDevOps/.sfdx-scanner/sfge.log for details.... Detected 2 violation(s) from 11 path(s) on 6/14 entry point(s).Analyzing with Salesforce Graph Engine. See /home/AzDevOps/.sfdx-scanner/sfge.log for details.... Detected 2 violation(s) from 24 path(s) on 11/14 entry point(s).Analyzing with Salesforce Graph Engine. See /home/AzDevOps/.sfdx-scanner/sfge.log for details.... Detected 4 violation(s) from 37 path(s) on 13/14 entry point(s).Analyzing with Salesforce Graph Engine. See /home/AzDevOps/.sfdx-scanner/sfge.log for details.... Overall, analyzed 39 path(s) from 14 entry point(s). Detected 4 violation(s).Analyzing with Salesforce Graph Engine. See /home/AzDevOps/.sfdx-scanner/sfge.log for details.... Done
Loaded Custom Settings: [ none found ]
Loaded Apex Controllers: [ none found ]
Executed sfge, found 0 violation(s) across 0 file(s).
Rule violations were written to Tests/sf-scan-dfa-results-1.xml.

This run was executed with the same code than the good run and within the same environment. Unfortunately, when we check Tests/sf-scan-dfa-results-1.xml, it's empty.

Has anyone had this issue before? What can we do?

Steps To Reproduce:
sf scanner run dfa --target './force-app/main/default/classes/*.cls' --projectdir './force-app/main/default' --category="Security" --normalize-severity --severity-threshold=1 --format=junit --outfile=Tests/sf-scan-dfa-results-1.xml

We also tried different combination of possible arguments such as the following, but it doesn't seem to impact the issue
sf scanner run dfa --target './force-app/main/default/classes/*.cls' --verbose --projectdir './force-app/main/default' --category="Security" --normalize-severity --severity-threshold=1 --format=junit --outfile=Tests/sf-scan-dfa-results-1.xml --rule-thread-count 6 --rule-thread-timeout 300000 --sfgejvmargs "-Xmx4g" --pathexplimit -1

Expected Behavior:
We expect that running the command would always write correctly to the out-file so that our pipeline can trigger the necessary operations.

Desktop:
cicd
OS: Ubuntu 22.04
Node: v18.19.1
NPM: v10.2.4
sf-cli: v2.29.5
sfdx-scanner: v3.23.0

local
Windows 11
Node: v18.20.2
NPM: 10.5.0
sf-cli: v2.29.5
sfdx-scanner: v3.23.0

Workaround:
The problem is intermittent and rerunning the pipeline will eventually work and block. But by that time, the violation may have been merge into the code.

Urgency:
Nothing is in production so we are not in a hurry. However, if we had been in production, then it could potentially have led to violating code in production.

Thank you for your time!

[jag-j]

Thanks for reporting this @painlesshippo. We will look into this. Thanks!

[git2gus]

This issue has been linked to a new work item: W-15488291