Security updates in Docker images

[esheehan-gsl]

@ian-noaa do you remember if there was a reason we put the apt-get update lines so far down in the Dockerfile?

unified-graphics/services/api/Dockerfile

Lines 11 to 13 in 6f89b47

	RUN apt-get update \
	&& apt-get upgrade -y \
	&& rm -rf /var/lib/apt/lists/*

Is that to make sure that we always run the updates whenever anything has changed in the image? This feels like something we would do intentionally, because it seems obviously inefficient for the image, since any change to the code will result in us re-updating all of the packages in the image.

[ian-noaa]

A good question. You're correct that we did this to make sure we always have the latest security updates. It's a trade-off between build times/caching and ensuring we get up-to-date security fixes into the image. The current way the Dockerfile is set up guarantees we get the latest security updates in our build at the expense of build times.

If we wanted to speed up local image builds, I think we could move that command before the COPY command. If we do that, we'll need to be mindful of invalidating the cache in our CI builds so that the CI-built images get the latest security updates. However, there's a good chance this is already taking place since I believe each build uses a clean GitHub Actions runner.

More detail

Docker caches each command in the Dockefile in a layer. It will only update layers that it's detected changes for and Docker seems to only consider local file changes or changes to the Dockerfile itself as signals to invalidate a layer. (Their docs are a little vague on what invalidates a layer) Once a layer is invalidated, all subsequent layers are rebuilt.

As an example, if you built an image with just RUN apt-get update && apt-get -y upgrade in it last week and then rebuilt it this week - Docker will reuse the week-old cache since it can't detect any changes in that command and miss the security updates. Docker has a quick overview in their docs on how the build cache works.

I think we could try a Dockerfile something like the one below. If there's a pre-built image in the cache, Docker should skip OS updates and will reuse the cache for the Python dependencies as well as long as the poetry.lock and pyproject.toml haven't changed.

FROM python:3.11

WORKDIR /usr/src/app

RUN pip3 install --no-cache-dir gunicorn poetry \
    && poetry config virtualenvs.create false

# Pull in OS security updates - note that Docker will use any locally cached image layers it can 
# find here so if you need the latest security updates, you will need to disable your cache
# with docker build --no-cache
RUN apt-get update \
    && apt-get upgrade -y \
    && rm -rf /var/lib/apt/lists/*

# Copy just the dependencies - this changes infrequently and installing them is expensive so
# we'd like to cache this and the subsequent RUN command if possible.
COPY pyproject.toml poetry.lock .

RUN poetry install --no-interaction --no-ansi --no-dev

# Copy in the rest of the code - this changes frequently and will invalidate the cache
COPY src/ ./src

EXPOSE 80
CMD ["gunicorn", "unified_graphics.wsgi:app", "--bind=0.0.0.0:80", "--timeout=600"]

[esheehan-gsl]

I think it’s fine to leave it as-is. The build times aren’t bothering me, I just wanted to double-check that we did this on purpose. I’ll probably add a comment to the Docker files so that future-future-me knows this is intentional.

[ian-noaa]

Sounds good. Dockerfile layer caching gets involved.