Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

V1.8 cuda 12.5 ubuntu 22.04 #154

Merged
merged 5 commits into from
Nov 26, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
64 changes: 35 additions & 29 deletions .build/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,9 @@

# Use NVIDIA CUDA as base image and run the same installation as in the other packages.
# The version of cuda must match those of the packages installed in src/Dockerfile.gpulibs
FROM nvidia/cuda:12.2.2-cudnn8-runtime-ubuntu22.04
LABEL authors="Christoph Schranz <[email protected]>, Mathematical Michael <[email protected]>"
FROM nvidia/cuda:12.5.1-cudnn-runtime-ubuntu22.04
LABEL authors="Christoph Schranz <[email protected]>"

# This is a concatenated Dockerfile, the maintainers of subsequent sections may vary.
RUN chmod 1777 /tmp && chmod 1777 /var/tmp

Expand Down Expand Up @@ -41,7 +42,7 @@ USER root

# Install all OS dependencies for the Server that starts
# but lacks all features (e.g., download as all possible file formats)
ENV DEBIAN_FRONTEND noninteractive
ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update --yes && \
# - `apt-get upgrade` is run to patch known vulnerabilities in system packages
# as the Ubuntu base image is rebuilt too seldom sometimes (less than once a month)
Expand All @@ -51,6 +52,10 @@ RUN apt-get update --yes && \
bzip2 \
ca-certificates \
locales \
# - `netbase` provides /etc/{protocols,rpc,services}, part of POSIX
# and required by various C functions like getservbyname and getprotobyname
# https://github.com/jupyter/docker-stacks/pull/2129
netbase \
sudo \
# - `tini` is installed as a helpful container entrypoint,
# that reaps zombie processes and such of the actual executable we want to start
Expand Down Expand Up @@ -85,7 +90,7 @@ RUN sed -i 's/^#force_color_prompt=yes/force_color_prompt=yes/' /etc/skel/.bashr
# and docs: https://docs.conda.io/projects/conda/en/latest/dev-guide/deep-dives/activation.html
echo 'eval "$(conda shell.bash hook)"' >> /etc/skel/.bashrc

# Create NB_USER with name jovyan user with UID=1000 and in the 'users' group
# Create "${NB_USER}" user (`jovyan` by default) with UID="${NB_UID}" (`1000` by default) and in the 'users' group
# and make sure these dirs are writable by the `users` group.
RUN echo "auth requisite pam_deny.so" >> /etc/pam.d/su && \
sed -i.bak -e 's/^%admin/#%admin/' /etc/sudoers && \
Expand Down Expand Up @@ -134,9 +139,9 @@ RUN set -x && \
--root-prefix="${CONDA_DIR}" \
--prefix="${CONDA_DIR}" \
--yes \
"${PYTHON_SPECIFIER}" \
'jupyter_core' \
'mamba' \
'jupyter_core' && \
"${PYTHON_SPECIFIER}" && \
rm -rf /tmp/bin/ && \
# Pin major.minor version of python
# https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-pkgs.html#preventing-packages-from-updating-pinning
Expand Down Expand Up @@ -199,18 +204,18 @@ RUN apt-get update --yes && \

USER ${NB_UID}

# Install JupyterLab, Jupyter Notebook, JupyterHub and NBClassic
# Install JupyterHub, JupyterLab, NBClassic and Jupyter Notebook
# Generate a Jupyter Server config
# Cleanup temporary files
# Correct permissions
# Do all this in a single RUN command to avoid duplicating all of the
# files across image layers when the permissions change
WORKDIR /tmp
RUN mamba install --yes \
'jupyterlab' \
'notebook' \
'jupyterhub' \
'nbclassic' && \
'jupyterlab' \
'nbclassic' \
'notebook' && \
jupyter server --generate-config && \
mamba clean --all -f -y && \
npm cache clean --force && \
Expand Down Expand Up @@ -339,7 +344,7 @@ RUN mamba install --yes \
'dask' \
'dill' \
'h5py' \
'ipympl'\
'ipympl' \
'ipywidgets' \
'jupyterlab-git' \
'matplotlib-base' \
Expand All @@ -357,7 +362,7 @@ RUN mamba install --yes \
'sqlalchemy' \
'statsmodels' \
'sympy' \
'widgetsnbextension'\
'widgetsnbextension' \
'xlrd' && \
mamba clean --all -f -y && \
fix-permissions "${CONDA_DIR}" && \
Expand All @@ -383,7 +388,7 @@ WORKDIR "${HOME}"
########################## Dependency: gpulibs #############################
############################################################################

LABEL maintainer="Christoph Schranz <[email protected]>, Mathematical Michael <[email protected]>"
LABEL authors="Christoph Schranz <[email protected]>"

# Install dependencies for e.g. PyTorch
RUN mamba install --quiet --yes \
Expand All @@ -399,7 +404,7 @@ RUN mamba install --quiet --yes \
# using device_lib.list_local_devices() the cudNN version is shown, adapt version to tested compat
USER ${NB_UID}
RUN pip install --upgrade pip && \
pip install --no-cache-dir tensorflow==2.15.1 keras==2.15.0 && \
pip install --no-cache-dir tensorflow==2.18.0 keras==3.6.0 && \
fix-permissions "${CONDA_DIR}" && \
fix-permissions "/home/${NB_USER}"

Expand All @@ -410,11 +415,11 @@ RUN pip install --upgrade pip && \
# && torchviz==0.0.2 --extra-index-url https://download.pytorch.org/whl/cu121
RUN set -ex \
&& buildDeps=' \
torch==2.2.2 \
torchvision==0.17.2 \
torchaudio==2.2.2 \
torch==2.5.1 \
torchvision==0.20.1 \
torchaudio==2.5.1 \
' \
&& pip install --no-cache-dir $buildDeps --extra-index-url https://download.pytorch.org/whl/cu121 \
&& pip install --no-cache-dir $buildDeps --extra-index-url https://download.pytorch.org/whl/cu124 \
&& fix-permissions "${CONDA_DIR}" \
&& fix-permissions "/home/${NB_USER}"

Expand All @@ -435,8 +440,9 @@ RUN pip install --no-cache-dir nvidia-pyindex && \
fix-permissions "${CONDA_DIR}" && \
fix-permissions "/home/${NB_USER}"

# Install cuda-nvcc with sepecific version, see here: https://anaconda.org/nvidia/cuda-nvcc/labels
RUN mamba install -c nvidia cuda-nvcc=12.2.140 -y && \
# Install cuda-nvcc with sepecific version, see here:
# https://anaconda.org/nvidia/cuda-nvcc/labels
RUN mamba install -c nvidia cuda-nvcc=12.5.82 -y && \
mamba clean --all -f -y && \
fix-permissions $CONDA_DIR && \
fix-permissions /home/$NB_USER
Expand All @@ -450,7 +456,7 @@ USER $NB_UID
############################ Useful packages ###############################
############################################################################

LABEL authors="Christoph Schranz <[email protected]>, Mathematical Michael <[email protected]>"
LABEL authors="Christoph Schranz <[email protected]>"

USER root

Expand All @@ -463,7 +469,7 @@ USER $NB_UID
RUN set -ex \
&& buildDeps=' \
graphviz==0.20.3 \
pytest==8.1.1 \
pytest==8.3.3 \
' \
&& pip install --no-cache-dir $buildDeps \
&& fix-permissions "${CONDA_DIR}" \
Expand All @@ -472,8 +478,8 @@ RUN set -ex \
# upgrade jupyter-server for compatibility
RUN set -ex \
&& buildDeps=' \
distributed==2024.4.1 \
jupyter-server==2.13 \
distributed==2024.11.2 \
jupyter-server==2.14.2 \
' \
&& pip install --no-cache-dir $buildDeps \
&& fix-permissions "${CONDA_DIR}" \
Expand All @@ -483,15 +489,15 @@ RUN set -ex \
&& buildDeps=' \
# install extension manager
jupyter_contrib_nbextensions==0.7.0 \
jupyter_nbextensions_configurator==0.6.3 \
jupyter_nbextensions_configurator==0.6.4 \
# install git extension
jupyterlab-git==0.50.0 \
jupyterlab-git==0.50.2 \
# install plotly extension
plotly==5.20.0 \
plotly==5.24.1 \
# install drawio and graphical extensions, not compatible with Jupyterlab 4.X yet
# ipydrawio==1.3.0 \
ipyleaflet==0.18.2 \
ipywidgets==8.1.2 \
ipyleaflet==0.19.2 \
ipywidgets==8.1.5 \
# install spell checker
jupyterlab-spellchecker==0.8.4 \
' \
Expand Down
2 changes: 1 addition & 1 deletion .build/docker-stacks
Submodule docker-stacks updated 36 files
+1 −1 .github/workflows/docker-build-test-upload.yml
+1 −1 .github/workflows/docker-merge-tags.yml
+1 −1 .github/workflows/docker-tag-push.yml
+1 −1 .github/workflows/docker-wiki-update.yml
+10 −10 .pre-commit-config.yaml
+1 −1 Makefile
+8 −7 README.md
+4 −2 binder/Dockerfile
+1 −0 docs/conf.py
+1 −1 docs/contributing/lint.md
+1 −1 docs/contributing/stacks.md
+1 −1 docs/maintaining/aarch64-runner.md
+2 −2 docs/using/common.md
+2 −2 docs/using/faq.md
+1 −1 docs/using/recipe_code/microsoft_odbc.dockerfile
+14 −13 docs/using/recipe_code/oracledb.dockerfile
+4 −4 docs/using/recipes.md
+8 −8 docs/using/running.md
+11 −7 docs/using/selecting.md
+3 −3 docs/using/troubleshooting.md
+1 −1 images/all-spark-notebook/Dockerfile
+4 −4 images/base-notebook/Dockerfile
+2 −2 images/base-notebook/docker_healthcheck.py
+8 −4 images/docker-stacks-foundation/Dockerfile
+9 −5 images/docker-stacks-foundation/start.sh
+2 −2 images/pytorch-notebook/Dockerfile
+9 −4 images/pytorch-notebook/cuda11/Dockerfile
+9 −4 images/pytorch-notebook/cuda12/Dockerfile
+2 −2 images/scipy-notebook/Dockerfile
+5 −1 images/tensorflow-notebook/Dockerfile
+10 −0 images/tensorflow-notebook/cuda/20tensorboard-proxy-env.sh
+10 −1 images/tensorflow-notebook/cuda/Dockerfile
+2 −2 images/tensorflow-notebook/cuda/nvidia-lib-dirs.sh
+39 −0 tests/docker-stacks-foundation/test_user_options.py
+1 −0 tests/scipy-notebook/data/matplotlib/matplotlib_1.py
+1 −0 tests/scipy-notebook/units/unit_pandas.py
4 changes: 2 additions & 2 deletions .build/docker_healthcheck.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,9 +9,9 @@
import requests

# Several operations below deliberately don't check for possible errors
# As this is a healthcheck, it should succeed or raise an exception on error
# As this is a health check, it should succeed or raise an exception on error

# Docker runs healtchecks using an exec
# Docker runs health checks using an exec
# It uses the default user configured when running the image: root for the case of a custom NB_USER or jovyan for the case of the default image user.
# We manually change HOME to make `jupyter --runtime-dir` report a correct path
# More information: <https://github.com/jupyter/docker-stacks/pull/2074#issuecomment-1879778409>
Expand Down
14 changes: 9 additions & 5 deletions .build/start.sh
Original file line number Diff line number Diff line change
Expand Up @@ -155,11 +155,14 @@ if [ "$(id -u)" == 0 ]; then
unset_explicit_env_vars

_log "Running as ${NB_USER}:" "${cmd[@]}"
exec sudo --preserve-env --set-home --user "${NB_USER}" \
LD_LIBRARY_PATH="${LD_LIBRARY_PATH}" \
PATH="${PATH}" \
PYTHONPATH="${PYTHONPATH:-}" \
"${cmd[@]}"
if [ "${NB_USER}" = "root" ] && [ "${NB_UID}" = "$(id -u "${NB_USER}")" ] && [ "${NB_GID}" = "$(id -g "${NB_USER}")" ]; then
HOME="/home/root" exec "${cmd[@]}"
else
exec sudo --preserve-env --set-home --user "${NB_USER}" \
LD_LIBRARY_PATH="${LD_LIBRARY_PATH}" \
PATH="${PATH}" \
PYTHONPATH="${PYTHONPATH:-}" \
"${cmd[@]}"
# Notes on how we ensure that the environment that this container is started
# with is preserved (except vars listed in JUPYTER_ENV_VARS_TO_UNSET) when
# we transition from running as root to running as NB_USER.
Expand Down Expand Up @@ -187,6 +190,7 @@ if [ "$(id -u)" == 0 ]; then
# above in /etc/sudoers.d/path. Thus PATH is irrelevant to how the above
# sudo command resolves the path of `${cmd[@]}`. The PATH will be relevant
# for resolving paths of any subprocesses spawned by `${cmd[@]}`.
fi

# The container didn't start as the root user, so we will have to act as the
# user we started as.
Expand Down
Loading