Fixing 'no space left on device' in Docker images and containers

Why this happens

Docker consumes disk via images, layers, containers, volumes, build cache, and logs. When any of these or filesystem inodes are exhausted, operations fail with: no space left on device.

Quickstart (safe defaults)

Run these in order to quickly free space without deleting named volumes or running containers:

# 1) Inspect usage
sudo docker system df

# 2) Prune stopped containers, dangling images, and unused networks (keeps named volumes)
sudo docker system prune -f

# 3) Prune build cache (all builders)
sudo docker builder prune -a -f

# 4) Check biggest consumers under Docker data root (commonly /var/lib/docker)
sudo du -h -d1 /var/lib/docker | sort -h

# 5) If logs are huge, rotate/truncate safely (adjust path if different)
sudo find /var/lib/docker/containers -name "*.log" -type f -size +100M -exec truncate -s 0 {} \;

If space is still tight, consider moving Docker’s data-root to a larger disk (see Prevent recurrence).

Minimal working example: reclaim space script

This script diagnoses usage, prunes safely, and handles large logs. Review before running in production.

#!/usr/bin/env bash
set -euo pipefail

# docker-reclaim-space.sh
# Reclaims Docker disk space without touching named volumes or running containers.

require() { command -v "$1" >/dev/null || { echo "Missing: $1"; exit 1; }; }
require docker

echo "==> Docker disk usage summary"
docker system df || true

echo "==> Pruning unused containers/images/networks (keeps named volumes)"
docker system prune -f

echo "==> Pruning build cache (all builders)"
docker builder prune -a -f || true

DATA_ROOT=$(docker info --format '{{.DockerRootDir}}' 2>/dev/null || echo "/var/lib/docker")

echo "==> Top-level usage under ${DATA_ROOT}"
sudo du -h -d1 "$DATA_ROOT" | sort -h || true

LOG_DIR="$DATA_ROOT/containers"
if [ -d "$LOG_DIR" ]; then
  echo "==> Truncating container logs >100MB"
  sudo find "$LOG_DIR" -name "*.log" -type f -size +100M -print -exec truncate -s 0 {} \;
fi

echo "==> Final Docker disk usage"
docker system df || true

echo "Done. Consider moving data-root if space remains low."

Usage:

• Save as docker-reclaim-space.sh
• chmod +x docker-reclaim-space.sh
• sudo ./docker-reclaim-space.sh

Diagnose storage pressure

1. Check disk space and inodes on Docker’s data root:

# Find data root
sudo docker info --format '{{.DockerRootDir}}'

# Disk space and inode usage
df -h /var/lib/docker
df -i /var/lib/docker

1. Summarize Docker object sizes:

sudo docker system df -v

1. Identify large logs and volumes:

sudo du -h -d1 /var/lib/docker | sort -h
sudo du -h -d1 /var/lib/docker/volumes | sort -h
sudo find /var/lib/docker/containers -name "*.log" -type f -printf "%s %p\n" | sort -nr | head

Reclaim space safely

Common cleanup commands and what they remove:

Steps:

1. Stop and remove dead/unused containers explicitly if needed:

sudo docker ps -a
sudo docker rm -f <container_id>

1. Remove unused images and layers:

sudo docker image prune -a -f

1. Purge build cache thoroughly:

sudo docker builder prune -a -f

1. Remove orphaned volumes only if you know they aren’t needed:

sudo docker volume ls
sudo docker volume prune -f   # Caution: deletes unused volumes

1. Truncate oversized container logs (json-file driver):

sudo find /var/lib/docker/containers -name "*.log" -type f -size +100M -exec truncate -s 0 {} \;

If you use journald or another driver, rotate via its native tooling instead.

Prevent recurrence

1. Enable log rotation to avoid runaway logs:

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  }
}

• Save as /etc/docker/daemon.json (merge with existing config).
• Then: sudo systemctl restart docker

1. Move Docker data-root to a larger disk/partition:

{
  "data-root": "/mnt/docker-data",
  "features": { "buildkit": true }
}

• Stop Docker: sudo systemctl stop docker
• Move existing data: sudo rsync -aHAX --info=progress2 /var/lib/docker/ /mnt/docker-data/
• Update /etc/docker/daemon.json with the new data-root
• Start Docker: sudo systemctl start docker

1. Optimize image builds to use less space:

• Use .dockerignore to shrink build context.
• Prefer multi-stage builds and small base images.
• Clean caches in the same RUN layer.

Example Dockerfile (multi-stage):

# Build stage
FROM golang:1.22-alpine AS build
WORKDIR /src
COPY . .
RUN --mount=type=cache,target=/go/pkg/mod \
    --mount=type=cache,target=/root/.cache/go-build \
    CGO_ENABLED=0 go build -o /app

# Runtime stage (tiny)
FROM gcr.io/distroless/static:nonroot
COPY --from=build /app /app
USER nonroot
ENTRYPOINT ["/app"]

1. Use BuildKit and inline cache to avoid re-downloading layers:

export DOCKER_BUILDKIT=1
sudo docker build --build-arg BUILDKIT_INLINE_CACHE=1 -t myapp:latest .

Pitfalls

• Pruning can delete data you need. Named volumes survive docker system prune, but docker volume prune will remove unused ones.
• Truncating logs modifies files under Docker’s control; ensure containers don’t rely on historical logs.
• Low inodes can trigger the error even with free space. Check df -i and remove many small files.
• Some cleanup requires root; use sudo where appropriate.

Performance notes

• Over-pruning removes useful cache and slows subsequent builds. Prefer targeted prunes and maintain a weekly cadence.
• Cache mounts in BuildKit (--mount=type=cache) speed builds without inflating final images.
• Layer ordering matters: put frequently changing layers later to maximize cache hits.
• Smaller images pull/push faster and reduce registry and node storage pressure.

FAQ

Q: docker system prune didn’t free much. What next? A: Prune build cache (docker builder prune -a), truncate logs, and remove unused images (docker image prune -a). Check volumes and /var/lib/docker size.

Q: I still see errors during builds only. A: Build cache or tmpfs may be full. Run docker builder prune -a and ensure /tmp and the Docker data-root have free space/inodes.

Q: How do I fix inode exhaustion? A: Use df -i to confirm. Remove many small files (e.g., old containers, build cache) or move Docker data-root to a filesystem with more inodes.

Q: Where is Docker storing data? A: docker info shows DockerRootDir (commonly /var/lib/docker). Use du -h to see which subdirectories are largest.