containers-dockerfile-optimization

Writing Dockerfiles for production applications

Dockerfile Optimization

Scope: Layer caching, multi-stage builds, .dockerignore, image size reduction Lines: ~280 Last Updated: 2025-10-18

When to Use This Skill

Activate this skill when:

Core Concepts

Docker Image Layers

Images are built in layers: Each instruction creates a new layer.

Key properties:

Example:

FROM python:3.11-slim    # Layer 1: Base image
RUN apt-get update       # Layer 2: Package manager update
COPY requirements.txt    # Layer 3: Copy requirements
RUN pip install -r req   # Layer 4: Install dependencies
COPY . .                 # Layer 5: Copy application code

Layer Caching Strategies

Rule 1: Order Instructions by Change Frequency

❌ Wrong (bust cache on every code change):

FROM python:3.11-slim
COPY . .                        # Changes frequently
RUN pip install -r requirements.txt  # Reinstalls every time
CMD ["python", "app.py"]

✅ Correct (cache dependencies):

FROM python:3.11-slim
COPY requirements.txt .         # Changes rarely
RUN pip install -r requirements.txt  # Cached unless requirements change
COPY . .                        # Changes frequently
CMD ["python", "app.py"]

Principle: Place stable layers first, volatile layers last.

Rule 2: Combine Commands to Reduce Layers

❌ Wrong (many layers):

RUN apt-get update
RUN apt-get install -y curl
RUN apt-get install -y git
RUN apt-get install -y vim
RUN rm -rf /var/lib/apt/lists/*

✅ Correct (single layer):

RUN apt-get update && \
    apt-get install -y \
        curl \
        git \
        vim && \
    rm -rf /var/lib/apt/lists/*  # Safe: cleaning package manager cache

Benefits: Smaller image, fewer layers, better caching.

Rule 3: Use .dockerignore to Exclude Files

Create .dockerignore:

# Version control
.git
.gitignore

# Dependencies
node_modules/
vendor/
__pycache__/
*.pyc

# Build artifacts
dist/
build/
*.egg-info/

# IDE
.vscode/
.idea/
*.swp

# Logs and temp files
*.log
tmp/
.env.local

# Documentation
README.md
docs/

# CI/CD
.github/
.gitlab-ci.yml
Jenkinsfile

Impact: Faster builds, smaller context, no accidental secret leakage.

Rule 4: Cache Package Manager Dependencies

Python (pip):

# ✅ Cache pip dependencies
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .

Node.js (npm/pnpm):

# ✅ Cache npm dependencies
COPY package.json pnpm-lock.yaml .
RUN pnpm install --frozen-lockfile
COPY . .

Go:

# ✅ Cache Go modules
COPY go.mod go.sum .
RUN go mod download
COPY . .
RUN go build -o app

Rust (Cargo):

# ✅ Cache Cargo dependencies
COPY Cargo.toml Cargo.lock .
RUN mkdir src && echo "fn main() {}" > src/main.rs
RUN cargo build --release
COPY . .
RUN cargo build --release

Multi-Stage Builds

Purpose: Separate Build and Runtime Environments

Problem: Including build tools bloats final image.

Solution: Multi-stage builds.

Basic Multi-Stage Pattern

# Stage 1: Build
FROM golang:1.21 AS builder
WORKDIR /app
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 GOOS=linux go build -o app

# Stage 2: Runtime
FROM alpine:3.19
WORKDIR /app
COPY --from=builder /app/app .
CMD ["./app"]

Result: Final image contains only binary, not Go compiler.

Multi-Stage for Python

# Stage 1: Build dependencies
FROM python:3.11 AS builder
WORKDIR /app
COPY requirements.txt .
RUN pip install --user --no-cache-dir -r requirements.txt

# Stage 2: Runtime
FROM python:3.11-slim
WORKDIR /app
COPY --from=builder /root/.local /root/.local
COPY . .
ENV PATH=/root/.local/bin:$PATH
CMD ["python", "app.py"]

Multi-Stage for Node.js

# Stage 1: Build
FROM node:20 AS builder
WORKDIR /app
COPY package.json pnpm-lock.yaml ./
RUN npm install -g pnpm && pnpm install --frozen-lockfile
COPY . .
RUN pnpm build

# Stage 2: Runtime
FROM node:20-slim
WORKDIR /app
COPY --from=builder /app/dist ./dist
COPY --from=builder /app/node_modules ./node_modules
CMD ["node", "dist/index.js"]

Multi-Stage for Static Sites

# Stage 1: Build
FROM node:20 AS builder
WORKDIR /app
COPY package.json pnpm-lock.yaml ./
RUN npm install -g pnpm && pnpm install
COPY . .
RUN pnpm build

# Stage 2: Serve with nginx
FROM nginx:alpine
COPY --from=builder /app/dist /usr/share/nginx/html
EXPOSE 80
CMD ["nginx", "-g", "daemon off;"]

Result: Final image is nginx + static files (no Node.js).

Image Size Optimization

Strategy 1: Use Minimal Base Images

Base image comparison: | Image | Size | Use Case | |-------|------|----------| | ubuntu:22.04 | ~77MB | Full Ubuntu (avoid) | | python:3.11 | ~1GB | Full Python + build tools | | python:3.11-slim | ~180MB | Python without extras | | alpine:3.19 | ~7MB | Minimal Linux (musl libc) | | scratch | ~0MB | Empty (static binaries only) | | distroless | ~20MB | Runtime only (no shell) |

Recommendations:

Strategy 2: Use Distroless for Security

# Build stage
FROM golang:1.21 AS builder
WORKDIR /app
COPY . .
RUN CGO_ENABLED=0 go build -o app

# Runtime stage with distroless
FROM gcr.io/distroless/static-debian12
COPY --from=builder /app/app /app
ENTRYPOINT ["/app"]

Benefits:

Distroless variants:

Strategy 3: Remove Build Artifacts

RUN apt-get update && \
    apt-get install -y --no-install-recommends \
        build-essential \
        curl && \
    # Build app \
    make build && \
    # Clean up in same layer \
    apt-get purge -y build-essential && \
    apt-get autoremove -y && \
    rm -rf /var/lib/apt/lists/*  # Safe: cleaning package manager cache

Key: Cleanup in same RUN instruction (same layer).

Strategy 4: Use --no-cache-dir for Package Managers

# Python
RUN pip install --no-cache-dir -r requirements.txt

# Alpine apk
RUN apk add --no-cache curl git

# Debian/Ubuntu apt
RUN apt-get update && \
    apt-get install -y --no-install-recommends curl && \
    rm -rf /var/lib/apt/lists/*  # Safe: cleaning package manager cache

Strategy 5: Optimize COPY Instructions

❌ Wrong (copies everything):

COPY . .

✅ Correct (copy only needed files):

COPY requirements.txt .
RUN pip install -r requirements.txt
COPY src/ ./src/
COPY config/ ./config/

Use .dockerignore to exclude unnecessary files automatically.

Build Performance Optimization

Technique 1: BuildKit (Modern Docker)

Enable BuildKit (faster builds, better caching):

# Linux/Mac
export DOCKER_BUILDKIT=1
docker build .

# Or per-command
DOCKER_BUILDKIT=1 docker build .

BuildKit features:

Technique 2: Cache Mounts (BuildKit)

# Python with cache mount
FROM python:3.11-slim
COPY requirements.txt .
RUN --mount=type=cache,target=/root/.cache/pip \
    pip install -r requirements.txt

Benefits: Reuses pip cache across builds (faster).

Technique 3: SSH Mounts for Private Dependencies

# Clone private repo without embedding SSH key
FROM golang:1.21
RUN --mount=type=ssh \
    git clone git@github.com:private/repo.git

Build command:

docker build --ssh default .

Technique 4: Secrets Mounting

# Use secret without embedding in layer
FROM alpine
RUN --mount=type=secret,id=github_token \
    wget --header="Authorization: token $(cat /run/secrets/github_token)" \
    https://api.github.com/repos/private/repo/tarball

Build command:

docker build --secret id=github_token,src=$HOME/.github_token .

Benefit: Secret never stored in image layers.

Common Dockerfile Patterns

Pattern 1: Python Web App (FastAPI/Flask)

FROM python:3.11-slim

WORKDIR /app

# Install dependencies
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

# Copy application
COPY . .

# Non-root user
RUN useradd -m appuser && chown -R appuser:appuser /app
USER appuser

EXPOSE 8000
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]

Pattern 2: Node.js App

FROM node:20-slim

WORKDIR /app

# Install dependencies
COPY package.json pnpm-lock.yaml ./
RUN npm install -g pnpm && pnpm install --frozen-lockfile --prod

# Copy application
COPY . .

# Non-root user
RUN chown -R node:node /app
USER node

EXPOSE 3000
CMD ["node", "index.js"]

Pattern 3: Go Static Binary

# Build stage
FROM golang:1.21 AS builder
WORKDIR /app
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 GOOS=linux go build -ldflags="-w -s" -o app

# Runtime stage
FROM scratch
COPY --from=builder /app/app /app
ENTRYPOINT ["/app"]

Flags: -ldflags="-w -s" removes debug info (smaller binary).

Pattern 4: Rust Binary

# Build stage
FROM rust:1.75 AS builder
WORKDIR /app
COPY Cargo.toml Cargo.lock ./
RUN mkdir src && echo "fn main() {}" > src/main.rs
RUN cargo build --release
COPY . .
RUN cargo build --release

# Runtime stage
FROM debian:bookworm-slim
COPY --from=builder /app/target/release/app /usr/local/bin/app
CMD ["app"]

Dockerfile Linting and Analysis

Tool 1: hadolint (Dockerfile Linter)

# Install
brew install hadolint

# Lint Dockerfile
hadolint Dockerfile

Common warnings:

Tool 2: dive (Image Layer Explorer)

# Install
brew install dive

# Analyze image
dive myimage:latest

Features:

Tool 3: docker history (Built-in)

# Show layer sizes
docker history myimage:latest

# Show layer commands
docker history --no-trunc myimage:latest

Dockerfile Best Practices Checklist

Base Image:
[ ] Use official images (python:3.11-slim, node:20-slim)
[ ] Pin specific versions (not :latest)
[ ] Use minimal variants (-slim, -alpine, distroless)

Layer Optimization:
[ ] Order instructions by change frequency (stable → volatile)
[ ] Combine RUN commands with &&
[ ] Clean up in same RUN layer
[ ] Use .dockerignore to exclude files

Caching:
[ ] Copy dependency files before application code
[ ] Use cache mounts for package managers (BuildKit)
[ ] Leverage multi-stage builds
[ ] Order COPY instructions carefully

Security:
[ ] Use non-root USER
[ ] Don't store secrets in layers
[ ] Use --mount=type=secret for sensitive data
[ ] Scan images for vulnerabilities (Trivy)

Size:
[ ] Use multi-stage builds
[ ] Remove build dependencies
[ ] Use --no-cache-dir for package managers
[ ] Remove unnecessary files

Performance:
[ ] Enable BuildKit (DOCKER_BUILDKIT=1)
[ ] Use cache mounts for dependencies
[ ] Parallelize independent stages
[ ] Use SSH mounts for private repos

Common Dockerfile Anti-Patterns

❌ Using :latest tag: FROM python:latest ✅ Pin versions: FROM python:3.11-slim

❌ Running as root: No USER directive ✅ Create non-root user: USER appuser

❌ Busting cache: COPY . . before dependencies ✅ Copy dependencies first: COPY requirements.txt .

❌ Storing secrets: ENV API_KEY=secret123 ✅ Use runtime secrets or mount: --mount=type=secret

❌ Many small layers: Multiple RUN commands ✅ Combine commands: RUN cmd1 && cmd2 && cmd3

❌ Not using .dockerignore: Copying .git, node_modules ✅ Exclude with .dockerignore

❌ Installing unnecessary packages: apt-get install -y vim curl wget ✅ Install only required: --no-install-recommends

Quick Reference

Build Commands

# Basic build
docker build -t myapp:latest .

# BuildKit build
DOCKER_BUILDKIT=1 docker build -t myapp:latest .

# Build with secrets
docker build --secret id=token,src=./token.txt -t myapp .

# Build with SSH
docker build --ssh default -t myapp .

# Build specific stage
docker build --target builder -t myapp:builder .

# No cache
docker build --no-cache -t myapp .

Image Analysis

# Show layers
docker history myapp:latest

# Inspect image
docker inspect myapp:latest

# Analyze with dive
dive myapp:latest

# Lint Dockerfile
hadolint Dockerfile

Related Skills

Level 3: Resources

This skill has comprehensive Level 3 resources in skills/containers/dockerfile-optimization/resources/.

REFERENCE.md

Comprehensive 900+ line reference covering:

Scripts

Production-ready executable scripts with --help and --json support:

  1. analyze_dockerfile.py (500+ lines)
  1. optimize_image.sh (400+ lines)
  1. benchmark_builds.py (500+ lines)

Examples

Optimized Dockerfiles by Language

Before/After Optimization Examples

Usage

# Analyze Dockerfile for issues
cd skills/containers/dockerfile-optimization/resources/scripts
./analyze_dockerfile.py /path/to/Dockerfile

# Get JSON output for CI/CD integration
./analyze_dockerfile.py Dockerfile --json --min-score 80

# Analyze existing image
./optimize_image.sh myapp:latest

# Build optimized version and compare
./optimize_image.sh --build myapp:latest

# Benchmark different build strategies
./benchmark_builds.py --all myapp

# Compare specific strategies
./benchmark_builds.py -s cached -s buildkit -s parallel myapp --json

Last Updated: 2025-10-27 Format Version: 1.0 (Atomic)