Implement Spark pod template overrides

[andrewwdye]

Tracking issue

#4105

Describe your changes

Create driver, executor, and spark application spec from optional pod template. Uses ToK8sPodSpec as a shared solution for merging defaults with container or pod template target overrides. Driver and executor specs will share the same pod template values. See note below on this choice.

Why use the same pod template for driver and executor?
As part of #4179 we had a discussion about whether or not to use the task-level pod template, or introduce role-specific pod templates for driver and executor. For now this change adopts the task-level pod template pattern used by other k8s plugins. We may end up moving this and others to role-specific templates in the future.

Test plan

Unit tests
I refactored, extended, and renamed the existing container based Spark unit test (TestBuildResourceContainer). I added a new TestBuildResourcePodTemplate test to cover most of the interface.

Sanity check diff of driver and executor pods locally

1. Setup and build single binary
2. Add spark to enabled-plugins to the config following this guide
3. Add spark role, role binding, and service account template files from this guide to $HOME/.flyte/cluster-resource-templates/
4. Register and run a spark workflow following this guide
5. Compare driver and executor pods between master and this branch

Verify behavior of specifying @task(pod_template=...) locally
Modified the my_spark example to use a pod template

@task(
    ...
    pod_template=PodTemplate(
        primary_container_name="primary",
        labels={"lKeyA": "lValA", "lKeyB": "lValB"},
        annotations={"aKeyA": "aValA", "aKeyB": "aValB"},
        pod_spec=V1PodSpec(
            containers=[
                V1Container(
                    name="primary",
                    image=custom_image,
                    command="echo",
                    args=["wow"],
                ),
            ],
            tolerations=[
                V1Toleration(
                    key="x/custom",
                    operator="Equal",
                    value="foo",
                    effect="NoSchedule",
                ),
            ],
        ),
    ),
)

And verified that the resulting pod had tolerations set

❯ kubectl get pod -n flytesnacks-development a68zfbjq8r74mjb9q5tq-n0-0-driver -o yaml | grep "tolerations:" -A 12
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  - effect: NoSchedule
    key: x/custom
    operator: Equal
    value: foo

Check all the applicable boxes

• I updated the documentation accordingly.
• All new and existing tests passed.
• All commits are signed-off.

[codecov]

Codecov Report

Attention: 27 lines in your changes are missing coverage. Please review.

Comparison is base (26228bd) 59.46% compared to head (336fc8f) 59.26%.
Report is 1 commits behind head on master.

❗ Current head 336fc8f differs from pull request most recent head cb5994d. Consider uploading reports for the commit cb5994d to get more accurate results

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #4183      +/-   ##
==========================================
- Coverage   59.46%   59.26%   -0.20%     
==========================================
  Files         638      552      -86     
  Lines       54375    39769   -14606     
==========================================
- Hits        32336    23571    -8765     
+ Misses      19490    13867    -5623     
+ Partials     2549     2331     -218     

Flag	Coverage Δ
unittests	?

Flags with carried forward coverage won't be shown. Click here to find out more.

Files	Coverage Δ
flyteplugins/go/tasks/plugins/k8s/dask/dask.go	80.93% <100.00%> (+2.13%)	⬆️
...ns/go/tasks/pluginmachinery/flytek8s/pod_helper.go	70.00% <0.00%> (-4.11%)	⬇️
...asks/pluginmachinery/flytek8s/non_interruptible.go	0.00% <0.00%> (ø)
flyteplugins/go/tasks/plugins/k8s/spark/spark.go	76.87% <87.23%> (-0.49%)	⬇️

... and 572 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[hamersaw]

A few nit questions, otherwise looks good. Thanks!

[jeevb]

LGTM! Looks like this plugin might now be primed for GPU (and GPU affinity) support. A task for a later PR. Thanks!