feat(kuma-dp): rework on the virtual probes to support probing tcp and grpc ports

[jijiechen]

Checklist prior to review

Implements this MADR: gRPC and TCP probes

• Link to relevant issue as well as docs and UI issues
  • fixes Support TCP and grpc liveness probes #3875
• This will not break child repos: it doesn't hardcode values (.e.g "kumahq" as a image registry) and it will work on Windows, system specific functions like syscall.Mkfifo have equivalent implementation on the other OS --
• Tests (Unit test, E2E tests, manual test on universal and k8s)
  • tested with unit and manual tests
  • Don't forget ci/ labels to run additional/fewer tests
• Do you need to update UPGRADE.md? --
  • No, it's compatible
• Does it need to be backported according to the backporting policy? (this GH action will add "backport" label based on these file globs, if you want to prevent it from adding the "backport" label use no-backport-autolabel label)
  • No

Changelog: Supported TCP and gRPC probes for data planes running on Kubernetes

[lukidzi]

I don't think we can remove it like this. What about universal?

[jijiechen]

Checked the code, this field only serves Kubernetes scenarios. For universal, there is another field called serviceProbe. Docs at here

[lahabana]

I'm pretty sure that's not going to be fine from a backward compat pov

[jijiechen]

Hi @lahabana I don't quite understand why removing this field will break backward compatibility?
Without this field, we can still parse the existing dataplane objects stored in the DB when people upgrade their versions. So I think this is not a break change.

[lobkovilya]

There're might be users who rely on this functionality on Universal. Backward compatibility means not only being able to parse the format, but also preserve the functionality. Usually we mark the field as deprecated and remove it after the 2 minor releases

[jijiechen]

Thanks for the explanation of the "Backward compatibility", I agree with your view of "not only being able to parse the format, but also preserve the functionality". I meant that you guys may be confused this field with the other one serviceProbe which serves Universal scenarios. The field "probes" here was only serving Kubernetes scenarios. @lobkovilya

[jijiechen]

Hi reviewers, sorry that this is a large PR. To help you review the code easily, let me explain what are included in this PR.
This PR basicly refactors the existing HTTP-only "virtual probes" and implements the new model of "virtual probe" that support all three kinds of pod probes: TCP, gRPC and HTTP.

Here are the break down tasks:

1. introduce a new component into kuma-dp to start the virtual probe server, it's an HTTP server. This is placed at directory app/kuma-dp/pkg/dataplane/probes
2. configure kuma-dp from the injector.go to run the virtual probe server according to user's auto-injector configuration
3. in the function of OverridePodProbes that is called by injector.go, change the existing virtual probe overriding to support all the three kinds of probes
4. exclude the virtual probe server port on the ways of settings of transparent proxy:
   1. kuma-sidecar: PodRedirect in file pkg/transparentproxy/kubernetes/kubernetes.go
   2. kuma-cni: NewIntermediateConfig in file app/cni/pkg/cni/annotations_linux.go

[jijiechen]

The PR also includes some minor/revelant refactorings.

Here are some of them:

1. Removed Probes to be on dataplane objects sinec we don't generate an insecure listener for the virtual probes server any more. So I removed this field and its message definition.
2. Removed grpc health check proto files from the repo and use the package from GitHub since it was included in the testserver package previously and if I introduce the same proto into a package under kuma-dp, it will generate a conflict panic.
3. Added the capabilities of supporting scheme and httpHeaders for HTTP probes, supporting of timeout is added for all three kinds of probes.

[lobkovilya]

Please use http.NewServeMux() like we do for example here:

kuma/pkg/diagnostics/server.go

Lines 56 to 60 in 604610b

	mux.HandleFunc("/debug/pprof/", pprof.Index)
	mux.HandleFunc("/debug/pprof/cmdline", pprof.Cmdline)
	mux.HandleFunc("/debug/pprof/profile", pprof.Profile)
	mux.HandleFunc("/debug/pprof/symbol", pprof.Symbol)
	mux.HandleFunc("/debug/pprof/trace", pprof.Trace)

instead of parsing the Path.

[lobkovilya]

@jijiechen please also think about the case when old kuma-dp connects to the new kuma-cp, I believe it won't work because you removed probe_generator.go