idea: customisation starting from img file?

[RafalSkolasinski]

In the past I used pi-gen to build a custom Raspbian image by adding stages with my customisations.
Though it was quite easy to do it this way I always find it a bit annoying that I had to go through the full build process.

Did you consider having option to run some custom stages with rootfs extracted from specified img file?

I did a very naive attempt of doing so by modifying Dockerfile used in the build process

ARG BASE_IMAGE=debian:buster
FROM ${BASE_IMAGE}

ENV DEBIAN_FRONTEND noninteractive

RUN apt-get -y update && \
    apt-get -y install --no-install-recommends \
        git vim parted \
        quilt coreutils qemu-user-static debootstrap zerofree zip dosfstools \
        bsdtar libcap2-bin rsync grep udev xz-utils curl xxd file kmod bc\
        binfmt-support ca-certificates \
    && rm -rf /var/lib/apt/lists/*

# Add requiered tools
RUN apt-get -y update && \
    apt-get -y install --no-install-recommends \
        wget p7zip-full \
    && rm -rf /var/lib/apt/lists/*

# Define customisation
ENV PREV_ROOTFS_DIR /base-rootfs/
ENV BASE_IMG_URL https://downloads.raspberrypi.org/raspios_lite_armhf/images/raspios_lite_armhf-2020-12-04/2020-12-02-raspios-buster-armhf-lite.zip

RUN mkdir -p ${PREV_ROOTFS_DIR} && \
    cd ${PREV_ROOTFS_DIR} && \
    wget ${BASE_IMG_URL}

RUN cd ${PREV_ROOTFS_DIR} && \
    7z x 2020-12-02-raspios-buster-armhf-lite.zip && \
    7z x 2020-12-02-raspios-buster-armhf-lite.img && \
    7z x 1.img && 7z x 0.fat -oboot/ && \
    rm 0.fat 1.img && rm *.zip *.img

# Define volumes and copy local files
COPY . /pi-gen/
VOLUME [ "/pi-gen/work", "/pi-gen/deploy"]

hoping that copy_previous will just do right thing if I have PREV_ROOTFS_DIR set, and it kind-of does but for some reason things fail on on_chroot stages with

execve /bin/bash failed!

Not sure if there is something in stage0 that I am missing or my procedure of extracting rootfs from img file was incorrect.

[RafalSkolasinski]

In case I wasn't clear what is my idea: I would like to come up with script that:

• takes as extra argument img file (local or url)
• extracts its content as it would be rootfs from previous stage
• runs specified extra stages

So for example if one would like to start with 2020-12-02-raspios-buster-armhf-lite.zip and have have one stage my-custom-stage the effect should be equivalent to checking out the right tag and running build with

STAGE_LIST='stage0 stage1 stage2 my-custom-stage

(more or less)

[XECDesign]

While thinking along the same of lines earlier, I was thinking maybe it's worth having a tool that runs pi-gen stages on a flexible target. Then pi-gen can wrap around that tool.

I'm having a bit of a hard time imagining how to fit your idea into the current workflow and source in a clean way. It's a good idea and I'm sure it's perfectly doable, I just can't spend time on it right now.

As is, pi-gen serves its primary purpose or building the official release images well enough and time is a bit limited. As much as I'd love to whip it into shape a bit more and make it more flexible for other use cases, I'm can't currently justify it. I'd strongly encourage a community fork or an alternative tool to fulfil those use cases.

If you send an appropriate PR, I can take a look. It would need to be a generic option, not something specific to the Docker script.

Personally, I think splitting the functionality of pi-gen into a stage runner, exporter and a wrapper to tie them together is the way to go to maximise flexibility, but it's too much of a change to look at right now.

[RafalSkolasinski]

Hi @XECDesign, thanks a lot for the reply. I'm glad you like the idea and I completely understand the time problems.
I think this is an issue for everyone :).

I am happy to try something out in a fork, and if it proves to be working we can think about bringing it back here.

Would you help me with some reference/example what is a most correct way to extract rootfs from img file?
My initial approach with 7z is probably not the most optimal one.

[XECDesign]

I don't know about it being correct or optimal, but my attempt is in the export-noobs stage, which pretty much does this already.

https://github.com/RPi-Distro/pi-gen/blob/master/export-noobs/prerun.sh

[wijjj]

I'm probably looking for something quite similar. Currently also looking at https://github.com/Nature40/pimod

[XECDesign]

pimod looks like the better choice because that seems to be exactly what it's for.

[RafalSkolasinski]

One other solutions I've also seen out there was https://www.pibakery.org/

[pablofrommars]

This would be of tremendous value and I would like to contribute in any way I can.

The use case I am after is pretty basic. Just the ability to have a script generating a batch of images with changed hostname, passwords and alter config files here and there from a base image would simplify my deployments so much.

I had a look at https://github.com/RPi-Distro/pi-gen/blob/master/export-noobs/prerun.sh as suggested and I believe it can be simplified a bit using the -P option of losetup.

pimod and pibakery look interesting but i think having this feature directly in pi-gen is useful.

[XECDesign]

I had a look at https://github.com/RPi-Distro/pi-gen/blob/master/export-noobs/prerun.sh as suggested and I believe it can be simplified a bit using the -P option of losetup.

Yeah, it was written before the -P parameter was widely available.

[aniongithub]

See my PR, it tries to address this exact issue but takes it a step further, enabling custom config and Github Actions based builds too.

#507