Speed up Docker build time with cache warming

Introduction

Every time you run a build, Docker will try to use the local cache if present. What if you have never built the image locally but the image is already on a remote docker registry?

In this post, we’ll go through how you can use a remote docker registry to warm the docker cache and speed up the local build. This can be also very useful in CI.

How to warm the cache in docker

From Docker 1.13 you can use a small trick. You can use the --cache-from flag to instruct docker to use as a cache the layer of another image. The only prerequisite is that you have the image locally.

Let’s say you have your Dockerfile locally that you build and push:

$ docker build -t my-docker .
$ docker push my-docker

Now, on another machine or in your CI, you want to build my-docker again. Here you can use the image you pushed on your docker registry to speed up the build.

$ docker pull my-docker || true
$ docker build --cache-from my-docker .

We use || true to avoid failures if the image has not been pushed and then we use the image as a cache for the build.

How to warm the cache with Buildkit

With the classical approach, you must pull the image before running the build. We can avoid it using Buildkit and BUILDKIT_INLINE_CACHE. You can check on a previous post on how to set up Buildkit on your machine.

What we really need, together with the cache, is the JSON with cache metadata. We can use Buildkit to run the Docker build and add the cache information to the Docker image.

$ export DOCKER_BUILDKIT=1
$ docker build -t my-docker --build-arg BUILDKIT_INLINE_CACHE=1 .
$ docker push my-docker

Here we enable Buildkit with export DOCKER_BUILDKIT=1 and then we use --build-arg BUILDKIT_INLINE_CACHE=1 during the build. At this point, the built image can be used as a cache for the other builds.

After that, we can run the build on another machine or in CI using the same syntax as before:

$ docker build --cache-from my-docker .

What will happen this time is that the builder will pull the JSON metadata from the registry to figure out if there’s a cache that can be used from the remote image. If there’s a cache hit, only the needed layers will be fetched locally and will be used by the builder.

This is it!

In this post we’ve seen how to warm up the Docker build cache using another image, this technique will help in speeding up the build time.

If you are looking for other ways to speed up the build time you can try to use a multistage parallel Docker build.

Reach me on Twitter @gasparevitta and let me know your thoughts!