CI Test test-secrets-no-filter-var and test-secrets-filter-var keeps failing on PR

[atalman]

I noticed that these two jobs are keeps failing on test-infra PR.

Failure:

  File "/home/ec2-user/actions-runner/_work/test-infra/test-infra/test-infra/.github/scripts/run_with_env_secrets.py", line 98, in main
    run_cmd_or_die(f"docker exec -t {container_name} /exec")
  File "/home/ec2-user/actions-runner/_work/test-infra/test-infra/test-infra/.github/scripts/run_with_env_secrets.py", line 39, in run_cmd_or_die
    raise RuntimeError(f"Command {cmd} failed with exit code {exit_code}")
RuntimeError: Command docker exec -t a555ccab70ac14d170f52ca70cb6a9b9a05a4e76acb803175825217561fe5d2d /exec failed with exit code 1
        
a555ccab70ac14d170f52ca70cb6a9b9a05a4e76acb803175825217561fe5d2d
Running command: docker exec -t a555ccab70ac14d170f52ca70cb6a9b9a05a4e76acb803175825217561fe5d2d /exec
+ [[ '' == \S\E\C\R\E\T\_\V\A\L\U\E ]]
+ exit 1
Error: Proces

Workflow:
https://github.com/pytorch/test-infra/actions/runs/11799135230/job/32866938776?pr=5898

[ZainRizvi]

Note that these tests seem to consistently fail when the base image is changed, but other PRs that include the new change were green afterwards.

https://github.com/pytorch/test-infra/actions/workflows/test_linux_job.yml

Failing example:
https://github.com/pytorch/test-infra/actions/runs/11713765214

A passing example that included the above PR:
#5894

[huydhn]

The secret is only available on test-infra. When you create a PR from your fork, there is no such secret and these tests will fail (expectedly). I think you could:

• Resubmit the PR to test-infra directly
• or create a secret called NOT_A_SECRET_USED_FOR_TESTING on your fork repo (I think the PR will inherit from that but haven't tried it out before)

[malfet]

Closing as this is expected behavior, secrets are not available to the forked PR according to doc:

With the exception of GITHUB_TOKEN, secrets are not passed to the runner when a workflow is triggered from a forked repository.

[ZainRizvi]

Reopening this: We shouldn't be leaving workflows that are expected to fail without giving the users any hints.

Instead we should skip these jobs when in a forked PR so that devs get a reliable signal.