Revisit gitops flux directory structure

[billimek]

For single-cluster home use, the official flux documentation showing an overly-complex mess of 'base', 'production', 'staging' per 'app', 'cluster' and 'infrastructure' seems like overkill to me. It's probably great for multiple different teams running against multiple clusters all sharing the same repo. This example is also dealing with the same complexities and difficulty organizing workloads (IMO).

Furthermore, needing to spew out over 100 nearly identical kustomize.yaml files everywhere doesn't seem very 'DRY'.

The purpose of this issue is to explore if it is possible to leverage flux in a way that we can maintain a sane (to me) repo directory structure and still leverage features like dependsOn such that we can ensure that CRDs and other items are applied before dependent workloads.

Will likely tackle this as part of #2865

[billimek]

Had some good insight from onedr0p in this thread, and did some experimentation trying an approach with and without kustomization.yamls everywhere:

inside ./kube-system,

external-secrets
├── 1password
│   ├── 1password.yaml
│   ├── clustersecretstore.yaml
│   └── kustomization.yaml
├── chart
│   ├── external-secrets.yaml
│   └── kustomization.yaml
├── ks.yaml
└── kustomization.yaml

... works as intended/expected where the ks.yaml is a flux kustomization with proper dependency settings.

---

When trying the same thing without the (IMO redundant kustomization.yamls),

external-secrets
├── 1password
│   ├── 1password.yaml
│   └── clustersecretstore.yaml
├── chart
│   └── external-secrets.yam
└──  ks.yaml

The flux apply ordering is not applied properly. This suggests that having kustomization.yaml files is a hard requirement.

[billimek]

Over in the flux slack, Kingdon B provided some additional insight (thread).

He noted that, in my case here, there is still the top-level flux kustomization referencing the repo root and as such there are two different flux kustomization's applying both on the external-secrets directory. This is why things break without the presence of the kustomize.yaml files, and this makes logical sense.

Given this information, there are likely three approaches to leveraging the flux ordering (dependsOn) features:

1. Ensure that there are kustomize.yaml files everywhere - or at least everywhere in trees where you need to control the ordering. This probably really means everywhere though. It appears that most of the k8s-at-home folks have either submitted that this is the way forward or simply copied from each-other this pattern
2. Better control/organization of top-level directories. If the 'root' of flux is going to point to the repo root (as is currently the case in this repo). Perhaps following the example from onedr0p and others where there is some sort of 'infrastructure' tree and 'applications' tree. Things that make CRDs would go in 'infrastructure' (e.g. cert manager, external secrets, prometheus CRDs, rook CRDs?, velero, etc) and everything else would go in 'applications'
3. Maybe some sort of top-level flux kustomization that specifically calls-out certain components that need be installed first, while still maintaining a flat directory structure - not sure if this is even possible

Something about the need to reflect the actual directory/file structure in separate (kustomization.yaml) really rubs me the wrong way and I would love to find a way around this problem without 'junking up' the repo with this cruft.

Will do deeper experiments with the new test cluster.

[billimek]

Another option that would avoid the necessity to let flux try to handle dependencies all-together, and resulting ugly hacks (IMO), would be to include required CRDs as part of the initial bootstrap process. This would be an extra one-time activity at cluster creation vs long-lasting poor structure.

Initially, CRDs that would be necessary to be in-place before flux installation would be:

• Prometheus: Lots of different servicemonitors and other prometheus-style things are invoked all over the cluster
• external-secrets: Anything using secrets needs this and are everywhere
• volsync: similar to secrets, there are a lot of references spread throughout the cluster
• cert-manager: This is relatively self-contained and I wonder if it's possible to handle bootstrapping within the cert-manager directory tree

Implementation would look like an additional set of steps to apply CRDs at cluster setup time, with extra reference (probably a single kustomization file) with reference to the version of externalized CRDs that could be revved automatically via mend.