Replacing Kaniko with BuildKit in shared CI

Kaniko was the default image build path in our shared GitLab includes for a long time: no Docker daemon, runs as a job container, pushes to the registry. It started to hurt when we needed reliable multi-arch builds and a build path aligned with what the wider platform was standardising on.

The replacement was BuildKit: a BuildKit daemon as a job service, buildctl from the job, same include file so consuming repos did not each invent a new package stage.

What changed in the include

The old .docker-build job used the Kaniko executor image and wrote Docker auth into /kaniko/.docker/config.json. The new path:

1. Runs a BuildKit-capable image as both the job image and a buildkitd service on a TCP address.
2. Waits until buildctl can see workers before building.
3. Builds with explicit Dockerfile path, destination, and optional TARGET_ARCH (amd64, arm64, or both).
4. Keeps the same variable names consumers already overrode (PATH_TO_DOCKERFILE, IMAGE_REPOSITORY, tags) so most repos only bumped the include version.

Kaniko stayed in the file as a commented deprecated block for a while — useful when debugging a single stubborn consumer without hunting git history.

The parts that bit

Variable compatibility. Some jobs overwrote IMAGE_REPOSITORY expecting Kaniko’s destination semantics. BuildKit’s destination composition was different enough that a few pipelines pushed to the wrong place until the override matched the new contract. Compatibility shims in the include beat asking every team to re-read the docs on day one.

Waiting for the daemon. Without an explicit wait loop, jobs failed sporadically when buildctl raced buildkitd. Sixty seconds of polling is ugly and necessary.

Auth surface. Kaniko’s config path is not BuildKit’s. Registry credentials had to land where the new tooling actually reads them, or you get a green script that fails on push with a misleading error.

What I’d tell someone doing this

Deprecate in-place in the shared include rather than forking a second include “for BuildKit.” Two parallel build abstractions guarantee half the estate never moves. Roll the include version through a representative set of repos first — especially ones with non-root Dockerfiles and custom image name overrides — then declare Kaniko dead.