Building CircuitPython in a Docker container

CircuitPython on hardware including Adafruit's boards, and CircuitPython libraries using Blinka on host computers.

Moderators: adafruit_support_bill, adafruit

Please be positive and constructive with your questions and comments.
Locked
User avatar
dgnuff
 
Posts: 15
Joined: Wed Dec 30, 2015 6:16 pm

Building CircuitPython in a Docker container

Post by dgnuff »

Mostly as a technical exercise to see if it's possible, I'm trying to build CircuitPython from source inside of a Docker container running under Docker Desktop on a Windows 10 host. It needs a bit more test and check before I'm ready to sign off on it, but the image is created in a state when most everything is installed, and you're ready to build.

The nice thing is that you have the option of mounting a folder on the host Windows system inside the running container, check out the --mount option to docker run for all the gory details. I'm doing that and keeping the whole repo there. That means I can actually use git on the Windows 10 host to interact with github, which makes life a lot easier.

How much interest is there in the Dockerfile for this? It's less than 35 lines of text total, so it'd easily fit in a forum post, and that would let anyone build the image.

I've got as far as building mpy-cross, which is a good first step, currently just found the feather RP2040 build, so I'm about to start building that .... Watch this space for an edit with details.

User avatar
dgnuff
 
Posts: 15
Joined: Wed Dec 30, 2015 6:16 pm

Re: Building CircuitPython in a Docker container

Post by dgnuff »

... That took a while. but it did succeed ! :)

I'll check this tomorrow because it's real late here, but running the Docker container with more than one core accessible, and then using the "-j" option to make should help a lot.

Docker command will be something like:

docker run -it --rm --name python --mount src=C:\Etc\CircuitPython,target=/mnt/circuitpython,type=bind --cpus=6 python

with the --cpus= parameter telling how many cores to allocate. I've got an 8 core HT Intel CPU, so half the cores are rather less than useful. That's 8 "useful" cores, of which I'm giving 6 to the container. That leaves 2 plus the 8 "hyperthreads" for Windows, which should be enough to get something done.

And of course, I'll want the "-j 6" parameter to make inside the container.

User avatar
dgnuff
 
Posts: 15
Joined: Wed Dec 30, 2015 6:16 pm

Re: Building CircuitPython in a Docker container

Post by dgnuff »

Yeah. That went a WHOLE LOT faster, throwing six cores at it.

So I'm building the UF2, and to quote Lewis Carrol "Oh frubjous day!" my Feather RP2040 was waiting for me in the mailbox this morning. I've installed the firmware.uf2, and things are looking real promising, the CircuitPython blink program works, and I can see data on the serial line.

If there's any interest, give me a yell here, and I'll upload the Dockerfile. It grew a little during test, and clocks in now at just over 40 lines.

User avatar
danhalbert
 
Posts: 4649
Joined: Tue Aug 08, 2017 12:37 pm

Re: Building CircuitPython in a Docker container

Post by danhalbert »

I think this is quite interesting, because it's less or similar work to installing WSL. We would certainly be interested in your Docker file.

I'm guessing there is not USB support via Docker, so actually loading the UF2, connecting to the REPL, etc., would be done on the Windows side.

User avatar
dgnuff
 
Posts: 15
Joined: Wed Dec 30, 2015 6:16 pm

Re: Building CircuitPython in a Docker container

Post by dgnuff »

Dockerfile itself:

Code: Select all

FROM ubuntu:20.04

ENV DEBIAN_FRONTEND=noninteractive
ENV DEBCONF_NONINTERACTIVE_SEEN true

RUN \
    echo "tzdata tzdata/Areas select US" > /tmp/preseed.cfg ; \
    echo "tzdata tzdata/Zones/US select Pacific" >> /tmp/preseed.cfg ; \
    debconf-set-selections /tmp/preseed.cfg ; \
    rm -f /etc/timezone /etc/localtime ; \
    apt update ; \
    apt upgrade -y ; \
    apt install -y tzdata ; \
    apt install -y build-essential software-properties-common mtools ; \
    add-apt-repository ppa:pybricks/ppa ; \
    apt install -y git gettext uncrustify wget python3 python3-pip vim ; \
    wget 'https://developer.arm.com/-/media/Files/downloads/gnu-rm/10-2020q4/gcc-arm-none-eabi-10-2020-q4-major-x86_64-linux.tar.bz2?revision=ca0cbf9c-9de2-491c-ac48-898b5bbc0443&la=en&hash=68760A8AE66026BCF99F05AC017A6A50C6FD832A' ; \
    mv 'gcc-arm-none-eabi-10-2020-q4-major-x86_64-linux.tar.bz2?revision=ca0cbf9c-9de2-491c-ac48-898b5bbc0443&la=en&hash=68760A8AE66026BCF99F05AC017A6A50C6FD832A' gcc-arm-none-eabi-10-2020-q4-major-x86_64-linux.tar.bz2 ; \
    tar xf gcc-arm-none-eabi-10-2020-q4-major-x86_64-linux.tar.bz2 ; \
    wget https://github.com/dosfstools/dosfstools/releases/download/v4.2/dosfstools-4.2.tar.gz ; \
    tar xf dosfstools-4.2.tar.gz ; \
    cd dosfstools-4.2 ; \
    ./configure ; \
    make ; \
    cp src/mkfs.fat /usr/bin ; \
    cd / ; \
    rm -f gcc-arm-none-eabi-10-2020-q4-major-x86_64-linux.tar.bz2 dosfstools-4.2.tar.gz /tmp/preseed.cfg ; \
    rm -rf dosfstools-4.2 ; \
    apt purge -y wget ; \
    apt autoremove -y ; \
    apt clean -y ; \
    git clone --depth 1  https://github.com/adafruit/circuitpython ; \
    cd circuitpython ; \
    pip3 install --upgrade -r requirements-dev.txt ; \
    pip3 install --upgrade -r requirements-doc.txt ; \
    cd / ; \
    rm -rf circuitpython ; \
    echo 'export PATH=$PATH:/gcc-arm-none-eabi-10-2020-q4-major/bin' >> /root/.bashrc ; \
    true

ENTRYPOINT [ "/bin/bash" ]
One thing to note: the first two lines in the "RUN" block set up so installing tzdata proceeds without needing user input. I've got them set for US Pacific since I live on the left coast. You'll probably want to adjust as necessary for you local timezone.

The command I use to build this is:

Code: Select all

docker build -t python:latest .
That tags the new image as "python:latest", feel free to change that to whatever makes the most sense for you. To actually run the docker container:

Code: Select all

docker run -it --rm --name python --mount src=C:\Etc\CircuitPython,target=/mnt/circuitpython,type=bind --cpus=6 python
There's a bunch of stuff in there that can be changed based on how you want to do things. You'll need the "-it" if and only if you're using this interactively, more on that later. I'm a bit of a neat-nik, so I clean up containers after use, thus the "--rm". If you want to keep the container around to run again later, remove that. "--name python" just gives the container a meaningful name. The "--mount" option is how I get the repo on the host to show up in the container, that lets me use git on the host as I mentioned up-thread. There's nothing preventing you from making the container permanent, and actually cloning the whole repo in there. That'll depend on your use case: in a build farm type of setup, it might make sense to have the repo inside the image. "--cpus=6" works on my system, I have an 8 core HT Intel setup, so I figure keeping 2 full cores and all the HT "partial cores" for Windows
is a reasonable compromise, it allows the build to run without taking forever. And of course "python" on the end is how I tagged the image when building it.

Random thoughts on using this in a build farm. I've used this trick in the past, and it works surprisingly well.

The first thing is to adjust the Dockerfile to clone not just the top level of the repo, but instead clone the whole deal. And then of course, don't erase it. Then you'd want to add a small shell script to actually do the build, including a git update. Next you'd alter the "ENTRYPOINT" line to invoke that shell script. You'll need to rebuild the image at least once a week, because if you don't, the state of the repo inside the image will get further and further behind master to the point that the build will start bogging down because it's spending more time updating the internal repo than it is actually building anything.

This is where you also ditch the "-it" option when invoking the image. If this is being used in a nightly build environment, you'd run with a single core per container, or possibly even a fractional core per container. That way you could have ten or twenty of these running on one host at a time. Sure, each build might take a couple of hours, but if these are your nightly builds you just don't care.

And of course the best thing is that the image created by this Dockerfile will run on a Windows 10/11 host with Docker Desktop, but it should also work on a Linux host, so you have the freedom to chose there.

Agreed about the USB support, that's where the mount thing comes in handy. In my case, once the build was finished, I'd have the Feather RP 2040 attached to the host, drop it into bootloader mode, and use Windows explorer to copy the new UF2 over.

I have some on the thoughts on the topic, but since I'm on vacation at the moment, I don't have my Feather RP 2040 with me to experiment on. My ideas are pretty ugly, and without testing I'd say it's on the flip of a coin whether it would actually work. I'm thinking in terms of using a second "--mount" option to "docker run" to actually mount the "UF2" drive letter inside the container. That might let you directly install from inside the container, I just can't say whether Docker will get its nose out of joint when the UF2 drive vanishes and gets replaced by the CIRCUITPY drive. On my system they always have the same drive letter, but that doesn't prove anything.

Locked
Please be positive and constructive with your questions and comments.

Return to “Adafruit CircuitPython”