Koray

One case worth flagging separately because none of the above applies: inside CI, or a container.

There is no agent, no home directory you expect, and often no known_hosts, so the connection can fail before authentication for a reason that reports as a key problem. The pattern that works is to write the key to a file at a known path with strict permissions, point at it explicitly, and pre-populate the host key rather than disabling checking.

Disabling host key verification is the shortcut everyone takes and it removes the only protection you had against connecting to the wrong server.

Worth knowing that most CI systems have built-in path filters, and for the common case they are better than any script.

A rule saying this job only runs when files under a given path changed is evaluated by the platform, handles the branch and merge cases itself, and cannot drift out of sync with your diffing logic.

Use the script only for what the filters cannot express, passing the actual file list to a tool, or fanning out over changed directories.

For a repository with several packages, the further step is a build tool that understands the dependency graph. Path filters know that a file changed; a build graph knows that changing it means three packages need rebuilding and the other twelve do not. That distinction is where the real time is saved, and no amount of diffing gets you there.

The fixes, best first:

1. Change the host directory to the container's UID. chown -R 1000:1000 /path/on/host, using whatever number id reported. Explicit, permanent, no surprises.

2. Run the container as the host directory's owner. Many images accept a user at run time, and many published images accept UID and GID as environment variables specifically to solve this. Check the image's documentation before doing anything else: a lot of the friction people hit here is reinventing a setting the image already has.

3. Use a named volume instead of a bind mount if you do not need the files at a specific host path. The volume driver initialises ownership from the image, so this problem largely disappears. Bind mounts exist for when you need the exact path; named volumes are the better default for data.

What not to do: world-writable. It works because it disables the check, on a directory holding your data, on the host. If the container is ever compromised, so is anything else on the box that can reach that path.

The thing to internalise: a bind mount does not translate ownership. The kernel checks numeric IDs, and the container and the host disagree about what those numbers mean.

Inside the container the process might run as a user called app with UID 1000. On the host, UID 1000 is your login account, or nobody at all. The directory on the host is owned by some other UID. The kernel compares the numbers, they do not match, and the write is refused.

So the name of the user is irrelevant on both sides. Only the number matters.

Find the two numbers and the problem becomes arithmetic:

docker exec <container> id          # what the process runs as
ls -ln /path/on/host                # numeric owner of the directory

Once those two agree, it works. Everything below is a way of making them agree.

To stop it recurring, which is the more useful half:

Cap how many kernels are kept. Most distributions have a setting for this and the default is often more generous than a small boot partition can hold. Two or three is plenty.

Make unattended upgrades clean up after themselves. If automatic updates are enabled and cleanup is not, the partition fills on a schedule. This is the usual cause of it happening at all.

Monitor it. A small boot partition is the classic disk that nobody watches because it never changes, until it does. Any alert on percentage used catches it weeks before it breaks anything.

And if this machine is one you build repeatedly, consider a larger boot partition next time. The sizes that were generous a few years ago are tight now, because boot images have grown.