[BUG] - pip installs overwrite and break conda environments

[krassowski]

Describe the bug

Many dependencies, including jhub-apps are installed with pip. This means that the dependencies previously installed by conda are not respected. For example:

jupyter-server-proxy >=4.2.0 depends on simpervisor >=1, but an older version gets installed because jhub-apps depends on jhsingle-native-proxy which depends on simpervisor=0.4 (jhsingle-native-proxy receives only minimal maintenance)

#27 168.8   Attempting uninstall: simpervisor
#27 168.8     Found existing installation: simpervisor 1.0.0
#27 168.8     Uninstalling simpervisor-1.0.0:
#27 168.8       Successfully uninstalled simpervisor-1.0.0
#27 168.8   Attempting uninstall: pydantic-core
#27 168.8     Found existing installation: pydantic_core 2.18.1
#27 168.8     Uninstalling pydantic_core-2.18.1:
#27 168.8       Successfully uninstalled pydantic_core-2.18.1
#27 168.8   Attempting uninstall: bokeh
#27 168.8     Found existing installation: bokeh 3.5.1
#27 168.8     Uninstalling bokeh-3.5.1:
#27 168.8       Successfully uninstalled bokeh-3.5.1

nebari-docker-images/jupyterlab/environment.yaml

Lines 59 to 64 in a64dd61

	- pip:
	# vscode jupyterlab launcher
	- git+https://github.com/betatim/vscode-binder
	- jupyterlab_nvdashboard==0.11.0
	- argo-jupyter-scheduler==2024.6.1
	- jhub-apps==2024.6.1

nebari-docker-images/jupyterlab/environment.yaml

Line 14 in a64dd61

- jupyter-server-proxy >=4.2.0

Expected behavior

• pip check is run on CI in the build process; build fails if pip constraints are not satisfied;
• as many dependencies are moved out of the pip group into the conda group as possible
  • this is especially important for dependencies which have multiple dependencies on their own

How to Reproduce the problem?

See
https://github.com/nebari-dev/nebari-docker-images/actions/runs/10113499250/job/27970005840#step:9:4386

Command output

No response

Versions and dependencies used.

No response

Anything else?

Context: for nebari-dev/nebari#2514 I needed to install a development version of jupyter-server-proxy but when I moved it from conda group to pip group the build failed. See

#37 256.0 INFO: pip is looking at multiple versions of jupyter-server-proxy to determine which version is compatible with other requirements. This could take a while.
#37 256.0 
#37 256.0 The conflict is caused by:
#37 256.0     jhsingle-native-proxy 0.8.2 depends on simpervisor~=0.4
#37 256.0     jupyter-server-proxy 4.3.1.post0.dev0 depends on simpervisor>=1.0.0
#37 256.0 
#37 256.0 To fix this you could try to:
#37 256.0 1. loosen the range of package versions you've specified
#37 256.0 2. remove package versions to allow pip attempt to solve the dependency conflict

https://github.com/nebari-dev/nebari-docker-images/actions/runs/10125975864/job/28002079311

[Adam-D-Lewis]

Thanks for reporting @krassowski. Adding pip check to the CICD seems like a good step.