Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ROX-19980 pre-built scanner-db image #1310

Open
wants to merge 10 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 11 additions & 2 deletions image/db/rhel/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ RUN curl -sSLf https://download.postgresql.org/pub/repos/yum/RPM-GPG-KEY-PGDG-${
groupadd -g 70 postgres && \
adduser postgres -u 70 -g 70 -d /var/lib/postgresql -s /bin/sh && \
microdnf install \
ca-certificates libicu systemd-sysv glibc-locale-source glibc-langpack-en \
ca-certificates tar libicu systemd-sysv glibc-locale-source glibc-langpack-en \
postgresql12-server && \
# The removal of /usr/share/zoneinfo from UBI minimal images is intentional.
# After building the image, the image is reduced in size as much as possible,
Expand All @@ -59,7 +59,16 @@ USER 70:70

COPY --from=extracted_bundle /bundle/docker-entrypoint-initdb.d/definitions.sql.gz /docker-entrypoint-initdb.d/

ENTRYPOINT ["docker-entrypoint.sh"]
COPY scripts/custom-entrypoint.sh /usr/local/bin/
COPY scripts/start-db.sh /usr/local/bin/

RUN /usr/local/bin/start-db.sh
USER root
RUN rm -rf /usr/local/bin/start-db.sh && \
rm -rf /docker-entrypoint-initdb.d/*
USER 70:70
ENV DATABASE_ALREADY_EXISTS=true
ENTRYPOINT ["custom-entrypoint.sh"]

EXPOSE 5432
CMD ["postgres", "-c", "config_file=/etc/postgresql.conf"]
13 changes: 11 additions & 2 deletions image/db/rhel/Dockerfile.slim
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ RUN curl -sSLf https://download.postgresql.org/pub/repos/yum/RPM-GPG-KEY-PGDG-${
groupadd -g 70 postgres && \
adduser postgres -u 70 -g 70 -d /var/lib/postgresql -s /bin/sh && \
microdnf install \
ca-certificates libicu systemd-sysv glibc-locale-source glibc-langpack-en \
ca-certificates tar libicu systemd-sysv glibc-locale-source glibc-langpack-en \
postgresql12-server && \
# The removal of /usr/share/zoneinfo from UBI minimal images is intentional.
# After building the image, the image is reduced in size as much as possible,
Expand All @@ -59,7 +59,16 @@ USER 70:70

ENV ROX_SLIM_MODE="true"

ENTRYPOINT ["docker-entrypoint.sh"]
COPY scripts/custom-entrypoint.sh /usr/local/bin/
COPY scripts/start-db.sh /usr/local/bin/

RUN /usr/local/bin/start-db.sh
USER root
RUN rm -rf /usr/local/bin/start-db.sh && \
rm -rf /docker-entrypoint-initdb.d/*
USER 70:70
ENV DATABASE_ALREADY_EXISTS=true
ENTRYPOINT ["custom-entrypoint.sh"]

EXPOSE 5432
CMD ["postgres", "-c", "config_file=/etc/postgresql.conf"]
66 changes: 66 additions & 0 deletions image/db/rhel/scripts/custom-entrypoint.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
#!/usr/bin/env bash

# The postgres server has been started once during the build process in the Dockerfile.
# Now we need to start it again, but this time with the correct password.
# So we need to issue a command to change the password.

set -e

if [ ! -d "/var/lib/postgresql/data/pgdata" ]; then

echo "Creating /var/lib/postgresql/data/pgdata..."
mkdir -p /var/lib/postgresql/data/pgdata

echo "Moving archive to target directory..."
mv /tmp/data.tar.gz /var/lib/postgresql/data/pgdata/data.tar.gz

echo "Uncompressing into /var/lib/pgsql/data/pgdata..."
tar -xzf /var/lib/postgresql/data/pgdata/data.tar.gz -C /var/lib/postgresql/data/pgdata

echo "Removing archive..."
rm /var/lib/postgresql/data/pgdata/data.tar.gz
Comment on lines +11 to +21
Copy link
Member

@SimonBaeumer SimonBaeumer Nov 2, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jvdm @RTann Could we move these steps to build time instead of start-up time?
@ludydoo proposes this to save IO in the cloud service which prevents us from cluster upgrades and increases ACS upgrade complexity.
Every restart of ACS instances causes to write ~1.8GB per instance.
Additionally the manifest must be updated that the scanner-db pod can read the Postgres Password.

What do you think about this approach? It seems simple and scaleable to us.
cc @ludydoo


echo "Starting database..."
POSTGRES_PASSWORD_FILE="" POSTGRES_PASSWORD=postgres /usr/local/bin/docker-entrypoint.sh postgres -c config_file=/etc/postgresql.conf &

echo "Waiting for database to be ready..."
while ! pg_isready -U postgres -h localhost -p 5432; do
sleep 1
done

echo "Changing password if POSTGRES_PASSWORD is set..."
if [ -n "$POSTGRES_PASSWORD" ]; then
PGPASSWORD=postgres psql -c "ALTER USER postgres WITH PASSWORD '$POSTGRES_PASSWORD';"
elif [ -n "$POSTGRES_PASSWORD_FILE" ]; then
PGPASSWORD=postgres psql -c "ALTER USER postgres WITH PASSWORD '$(cat "$POSTGRES_PASSWORD_FILE")';"
fi

echo "Renaming postgres user if necessary..."
if [ -n "$POSTGRES_USER" ]; then
if [ "$POSTGRES_USER" != "postgres" ]; then
if [ -n "$POSTGRES_PASSWORD" ]; then
PGPASSWORD="$POSTGRES_PASSWORD" psql -c "ALTER USER postgres RENAME TO $POSTGRES_USER;"
elif [ -n "$POSTGRES_PASSWORD_FILE" ]; then
PGPASSWORD="$(cat "$POSTGRES_PASSWORD_FILE")" psql -c "ALTER USER postgres RENAME TO $POSTGRES_USER;"
fi
fi
fi

echo "Stopping database..."
pg_ctl -w stop

else
echo "Database already initialized. Skipping initialization..."
fi

if [ "${ROX_SCANNER_DB_INIT}" == "true" ]; then
echo "ROX_SCANNER_DB_INIT is set to true. Exiting..."
exit 0
else
echo "Database initialized."
fi

# Now we can start the database for real. But we will
# forward any arguments to the actual entrypoint script
echo "Starting database for real..."
exec /usr/local/bin/docker-entrypoint.sh "$@"
22 changes: 0 additions & 22 deletions image/db/rhel/scripts/docker-entrypoint.sh
Original file line number Diff line number Diff line change
Expand Up @@ -306,18 +306,6 @@ _main() {
fi

if [ "$1" = 'postgres' ] && ! _pg_want_help "$@"; then
### STACKROX MODIFIED - If we are initializing, then ensure we start from scratch.
if [ -n "$ROX_SCANNER_DB_INIT" ]; then
echo
echo 'Initializing... Clearing any previous data from directories'
echo

rm -rf "$PGDATA"
if [ -n "${POSTGRES_INITDB_WALDIR:-}" ]; then
rm -rf "$POSTGRES_INITDB_WALDIR"
fi
fi

docker_setup_env
# setup data directories and permissions (when run as root)
docker_create_db_directories
Expand All @@ -329,16 +317,6 @@ _main() {
exec gosu postgres "$BASH_SOURCE" "$@"
fi

### STACKROX MODIFIED - Sanity check the database does not exist
### upon initialization.
if [ -n "$ROX_SCANNER_DB_INIT" ] && [ -n "$DATABASE_ALREADY_EXISTS" ]; then
echo
echo 'PostgreSQL Database appears to already exist upon initialization; Exiting with error...'
echo

exit 1
fi

# only run initialization on an empty data directory
if [ -z "$DATABASE_ALREADY_EXISTS" ]; then
docker_verify_minimum_env
Expand Down
28 changes: 28 additions & 0 deletions image/db/rhel/scripts/start-db.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
#!/usr/bin/env bash

set -eu

echo "Creating postgres.conf for initialization..."
cat <<EOF > /tmp/postgres.conf
listen_addresses = '*'
max_wal_size = 1GB
EOF

echo "Creating temporary PGDATA directory..."
mkdir -p /tmp/data

echo "Starting database..."
PGDATA=/tmp/data POSTGRES_PASSWORD=postgres /usr/local/bin/docker-entrypoint.sh postgres -c config_file=/tmp/postgres.conf

echo "Waiting for database to stop..."
while [ -f /tmp/data/pgdata/postmaster.pid ]; do
sleep 1
done

rm /tmp/postgres.conf

echo "Compressing database data folder..."
tar -czf /tmp/data.tar.gz -C /tmp/data .

echo "Removing temporary PGDATA directory..."
rm -rf /tmp/data