[Custom Policy Check] Plan steps success despite statut error 1

[tweiss-mdm]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request. Searching for pre-existing feature requests helps us consolidate datapoints for identical requirements into a single place, thank you!
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.

---

Overview of the Issue

Weird behavior when activating custom_policy_check on a custom workflow: all the failing plan/apply are in success status.

Before custom_policy_check: true

After custom_policy_check: true

Environment details

• Atlantis version: 0.27.3
• Deployment method: helm
• Gitlab 16.9

Atlantis server-side config file:

---
environment:
  ATLANTIS_ALLOW_COMMANDS: "plan,apply,unlock,approve_policies,import"
  ATLANTIS_FAIL_ON_PRE_WORKFLOW_HOOK_ERROR: true
  ATLANTIS_ENABLE_POLICY_CHECKS: true
  ATLANTIS_HIDE_UNCHANGED_PLAN_COMMENTS: true
  ATLANTIS_QUIET_POLICY_CHECKS: true
  repoConfig: |
  ---
  repos:
  - id: "/.*/"
    delete_source_branch_on_merge: true
    apply_requirements: [approved, mergeable]
    allowed_overrides: [workflow]
    repo_locking: false
    policy_check: true
    custom_policy_check: true
    workflow: terragrunt
    pre_workflow_hooks:
    - run: |
        echo "generating atlantis.yaml"
        terragrunt-atlantis-config generate --output atlantis.yaml --autoplan --automerge --parallel --create-workspace="true" --workflow="terragrunt" --ignore-dependency-blocks="true" --filter "live/staging" --filter "live/dev" --filter "live/rec" --filter "live/pp" --filter "live/prod";
  metrics:
    prometheus:
      endpoint: /metrics
  workflows:
    terragrunt:
      plan:
        steps:
        - env:
            # Reduce Terraform suggestion output
            name: TF_IN_AUTOMATION
            value: 'true'
        - run: terragrunt hclfmt --terragrunt-check
        - run:
            # Allow for targetted plans/applies as not supported for Terraform wrappers by default
            command: terragrunt plan -input=false $(printf '%s' $COMMENT_ARGS | sed 's/,/ /g' | tr -d '\\') -no-color -out $PLANFILE
            output: hide
        - run: terragrunt show $PLANFILE
      policy_check:
        steps:
          - run:
              command: terragrunt show -json $PLANFILE | awk '/^\{/ {print}' > "plan.json"
              output: hide
          - run: 
              command: |
                cat <<-EOF > ./variables.json
                {
                   "mode": "FULL"
                }
                EOF
              output: hide
          - run: |
              conftest test plan.json variables.json --combine --all-namespaces -p /atlantis-data/opa-rules/policy --quiet --no-color --strict
      apply:
        steps:
        - env:
            name: TF_IN_AUTOMATION
            value: 'true'
        - run: terragrunt apply -input=false $PLANFILE
      import:
        steps:
        - run: terragrunt import -input=false $(printf '%s' $COMMENT_ARGS | sed 's/,/ /' | tr -d '\\')
---

[Webgardener]

By refactoring our (old) rego code, I managed to use regular policy_check instead of the custom one. But Maybe this bug needs to be addressed nevertheless ?