reflector:_list_and_update check k8s api response error before read the items

[ealyn]

I got an KeyError: 'items' while using KubeIngressProxy, and I add some log to find out reason:

[I 2023-04-23 07:22:57.264 JupyterHub reflector:231] initial_resources keys: dict_keys(['kind', 'apiVersion', 'metadata', 'status', 'message', 'reason', 'details', 'code'])
[I 2023-04-23 07:22:57.265 JupyterHub reflector:232] initial_resources: {'kind': 'Status', 'apiVersion': 'v1', 'metadata': {}, 'status': 'Failure', 'message': 'ingresses.networking.k8s.io is forbidden: User "system:serviceaccount:jupyterhub:hub" cannot list resource "ingresses" in API group "networking.k8s.io" in the namespace "jupyterhub"', 'reason': 'Forbidden', 'details': {'group': 'networking.k8s.io', 'kind': 'ingresses'}, 'code': 403}
[E 2023-04-23 07:22:57.265 JupyterHub reflector:387] Initial list of ingresses failed
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/kubespawner/reflector.py", line 385, in start
await self._list_and_update()
File "/usr/local/lib/python3.9/site-packages/kubespawner/reflector.py", line 235, in _list_and_update
for p in initial_resources["items"]
KeyError: 'items'

Proposed change

1. function reflector:_list_and_update check k8s api response error before read the items
2. jupyterhub chart templates/hub/rbac.yaml add ingress api (for https://github.com/jupyterhub/helm-chart)

[consideRatio]

There may be another issue as well, but the key issue is permissions.

The jupyterhub chart isn't pre-configured to grant tge required permissions.

'Failure', 'message': 'ingresses.networking.k8s.io is forbidden: User "system:serviceaccount:jupyterhub:hub" cannot list resource "ingresses" in API group "networking.k8s.io" in the namespace "jupyterhub"'

I recall there are docstrings about required permissions to be found in the class defintion for KubeIngressProxy.

[consideRatio]

A fix for this project is to fail with a better error message if we receive a forbidden response.

[dolfinus]

reflector._list_and_update calls api.list_namespaced_ingress, but it does not raise exceptions if Kubernetes returned 403 error.

This is because _preload_content=False is used, and method parses response object manually:

kubespawner/kubespawner/reflector.py

Lines 218 to 223 in 8766d36

	kwargs = dict(
	label_selector=self.label_selector,
	field_selector=self.field_selector,
	_request_timeout=self.request_timeout,
	_preload_content=False,
	)

It was added in #424. I'm not sure that it should be used in reflector._list_and_update at all, watcher implementation already has its own options:

kubespawner/kubespawner/reflector.py

Lines 302 to 304 in 8766d36

	method = partial(
	getattr(self.api, self.list_method_name), _preload_content=False
	)

[ealyn]

@dolfinus as the log I printed begining, the exception is raise at for p in initial_resources["items"]:

kubespawner/kubespawner/reflector.py

Lines 228 to 234 in a1a4a6f

	initial_resources_raw = await list_method(**kwargs)
	# This is an atomic operation on the dictionary!
	initial_resources = json.loads(await initial_resources_raw.read())
	self.resources = {
	f'{p["metadata"]["namespace"]}/{p["metadata"]["name"]}': p
	for p in initial_resources["items"]
	}

[dolfinus]

If await list_method(**kwargs) is called without _preload_content=False, and Kubernetes API returned 403 error, this function raises exception before for p in initial_resources["items"].