How to fluently forward Makefile’s targets to a Docker container ?

cli
makefile
bash
docker
container
Author

x0s

Published

March 16, 2023

What is it about ?

We will see how to painlessly redirect makefile’s targets to a docker container keeping the API intuitive.

What you will learn:

  • How to automatically read a Makefile’s targets from another Makefile
  • How to wrap Makefile targets to a docker container
  • How to extract and pass Makefile arguments to another Makefile

Introducing a use case

Let’s say we’ve already built an application using Makefile to run tasks (ie: run tests, build a docker image, launch scripts). As good practice we developed it within an virtual environment (ie: conda) and it worked perfectly… until one day… as the application grew, it had to deal with different versions of python, compiler dependencies… things that a conda environment cannot handle alone. So we decided to wrap your app in a container.

To sum up, from being able to do:

(my_env) $ make test
(my_env) $ make game WHEN=2022/01-2

we also want to be able to do:

(my_env) $ make_in_container test
(my_env) $ make_in_container game WHEN=2022/01-2

Why choosing make_in_container ?

  • We don’t want to duplicate Makefiles nor the targets (DRY)

  • We favour clarity, avoiding mingling arguments, prefering make_in_container test_this DAY=2022/01 to make test_this SKIP_CONTAINER=false DAY=2022/01. Also with this option we would have to manually prefix every command of the original Makefile

  • We also favour extensibility, what if later we want to switch from Docker to Podman ? make_in_container looks better than make_in_docker

To illustrate this, we will use the git repository I worked on to do the famous Advent Of Code ! So we have this tree structure:

Advent-Of-Code
├── Dockerfile
├── Makefile
├── Makefile_docker
├── advent_of_code/
├── tests/
| ...

There are two makefiles:

  • a Makefile having the project usual targets (install,build, test …)

  • a Makefile_docker aiming at forwarding the targets to a docker container.

The workflow is the following:

# Install the application following the guidelines
# - Create conda environment
# - make install
# - ...

# Build the docker image
(my_env) $ make build

# Run some tasks in container
(my_env) $ make_in_container test_this DAY=2022/01

Once the image is built, a user should be able to run any task within the container as soon as the conda environment is active. We’ll see in next sessions how to settle this.

Aliasing make_in_container

To avoid polluting system namespace (sourcing ~/.bashrc), we want to enable the make_in_container command only when working in this project. Enabling it when we activate the conda environment seems the right moment. Let’s configure it at building time:

# from Makefile
build:
    @docker build --tag aoc-image -f Dockerfile .
# make "make_in_container" command available when conda env is activated
ifdef CONDA_PREFIX
    @$(eval PATH_ALIAS := ${CONDA_PREFIX}/etc/conda/activate.d/aliases_.sh)
    @mkdir -p ${CONDA_PREFIX}/etc/conda/activate.d
    @echo "#!/bin/bash" >> $(PATH_ALIAS)
    @echo "alias make_in_container='make -f Makefile_docker'" >> $(PATH_ALIAS)
    @source $(PATH_ALIAS)
endif

First, we build and image named aoc-image using Dockerfile. Then if a conda environment is active (and it should!), we write a little script this conda env will execute every time it is activated. Therefore make_in_container will redirect to using Makefile_docker as a makefile:

#!/bin/bash
alias make_in_container='make -f Makefile_docker'

Of course, you can adapt this code to make it work with other environment manager (Virtualenvwrapper hooks for instance).

How to retrieve all targets ?

Here we want to automatically forward targets to the docker container. First we retrieve the targets from Makefile using a snippet heavily inspired from StackOverflow. We can see the result by invoking make_in_container list-targets.

# from Makefile_docker

# Tag of docker image
NAME := aoc-image

# Retrieve the AOC targets from main Makefile
# inspired from https://stackoverflow.com/a/26339924/3581903
AOC_TARGETS := $(shell LC_ALL=C make -pRrq -f Makefile : 2>/dev/null \
        | awk -v RS= -F: '/(^|\n)\# Files(\n|$$)/,/(^|\n)\# Finished Make data base/ {if ($$1 !~ "^[\#.]") {print $$1}}' \
        | sort | egrep -v -e '^[^[:alnum:]]' -e '^$@$$' \
        | xargs | tr -d :)

.PHONY : $(AOC_TARGETS) list_targets

# Forward any AOC_TARGET to the container
$(AOC_TARGETS):
    @docker run -it ${NAME} make $@

list-targets:
    @echo $(AOC_TARGETS)

With this setting, calls like make_in_container help or make_in_container test properly forward make help and make test in the container. But what about make_in_container test_this DAY=2022/01 ? For now it only forwards make_in_container test_this. Let’s see how to pass all arguments to the container.

How to pass all arguments ?

As per the manual, $@ only keeps the target name. Calling make_in_container test_this DAY=2022/01 will set $@ to test_this and DAY to 2022/01. To retrieve the argument, we have to check the environment variables. Here we will store the arguments’ names in AOC_ARGS (it is possible to retrieve it automatically with regexp on the Makefile help, but it would add unecessary complexity here).

For each argument, get_args will check if an environment variable is set (like DAY to 2022/01), and when it is, will return the string DAY=2022/01.

# Extract valid arguments and pass them with their value
# ie: calling "make_in_container game WHEN=2022/01-1" will result in passing "WHEN=2022/01-1"
get_args = $(foreach arg,$(AOC_ARGS),$(if $(value $(arg)),$(arg)=$($(arg))))

# Arguments to be passed to targets accordin to main Makefile
AOC_ARGS = EDIT TOKEN WHEN VERBOSE DAY

Therefore, we update the redirection so it can forward the arguments to the container. make_in_container test_this DAY=2022/01 will be forwarded as make test_this DAY=2022/01.

# Forward any AOC_TARGET to the container   
# ie: "make_in_container test VERBOSE=1" is run as "make test VERBOSE=1" in the container
$(AOC_TARGETS):
    @docker run -it ${NAME} make $@ $(call get_args)

You can reach the full source here.

Conclusion

We’ve seen how to wrap all makefile targets so they can be forwarded to a container without changing the API: Everything that was runnable with make ... is now also runnable in a container with make_in_container ....

Reuse

https://creativecommons.org/licenses/by/4.0/

Citation

BibTeX citation:
@online{x0s2023,
  author = {x0s},
  title = {How to Fluently Forward {Makefile’s} Targets to a {Docker}
    Container\,?},
  date = {2023-03-16},
  langid = {en}
}
For attribution, please cite this work as:
x0s. 2023. “How to Fluently Forward Makefile’s Targets to a Docker Container ?” March 16, 2023.