Container images, volumes and networks

Before you read. A database runs in a container for three weeks. The image gets an update, somebody recreates the container, and every row is gone.

Nothing crashed. No disk failed. The command that removed the old container reported success.

If a container can write files, where do those files actually go?

Somewhere that was designed to be thrown away, and the throwing away is the point rather than a flaw. Understanding why requires one idea, layers, and everything else in this lesson follows from it.

Some words you will need

layer

One read-only step in an image. Images are stacks of them.

writable layer

A thin layer on top, belonging to one container. Deleted with it.

volume

Storage managed by the container engine, outside any container's lifetime.

bind mount

A host directory mapped into a container. From lesson 13.

Containerfile

The recipe for building an image. Also called a Dockerfile.

What breaks without this

You lose data, which is the subject of the opening question and happens to everybody once.

Rebuilds take minutes instead of seconds, because the caching rule was not understood.

Containers cannot reach each other, or reach each other far too easily.

Layers

An image is a stack of read-only layers, one per build step. A container adds a thin writable layer on top, and every file it creates or modifies goes there.

The layers below are shared. Ten containers from one image share one copy of the image and have ten small writable layers, which is why containers are cheap to start and cheap to have many of.

Modifying an existing file uses copy-on-write: the file is copied up into the writable layer and changed there. The original is untouched, which is what lets the lower layers stay shared.

And the writable layer is deleted with the container. Not corrupted, not lost, deliberately removed, because the whole model assumes a container is disposable and anything worth keeping lives elsewhere.

# Fedora CoreOS 44.20260707.3.1 on a virtual machine, aarch64
$ echo "--- write inside a container, then remove it ---"; sudo podman run --name tmp1 alpine sh -c "echo important > /data.txt; cat /data.txt"; sudo podman rm tmp1 >/dev/null; sudo podman run --rm alpine cat /data.txt 2>&1 | tail -1; echo "--- now with a volume ---"; sudo podman run --rm -v appdata:/data alpine sh -c "echo important > /data/file.txt"; sudo podman run --rm -v appdata:/data alpine cat /data/file.txt
--- write inside a container, then remove it ---
important
cat: can't open '/data.txt': No such file or directory
--- now with a volume ---
important

podman inspect thename --format '{{range .Mounts}}{{.Source}} -> {{.Destination}}
{{end}}'

Volumes and bind mounts

Two ways to attach storage, and the choice matters.

The syntax difference is one character. A source with a / is a path and therefore a bind mount; a source without one is a name and therefore a volume. -v data:/data and -v ./data:/data do different things.

podman volume create appdata
podman volume ls
podman volume inspect appdata      # where it actually lives
podman run -d -v appdata:/var/lib/postgresql/data postgres:16

Use a volume for state the container owns, database files, uploaded content. The engine manages the location, so nothing depends on the host’s directory layout.

Use a bind mount when the host path matters, a config file you edit, source code during development, or writing logs somewhere the host’s log shipper already watches.

:ro makes it read-only, and configuration mounted into a container should nearly always be read-only:

-v /srv/config/app.conf:/etc/app.conf:ro

Building an image

A Containerfile, the same format as a Dockerfile:

FROM alpine:3.20

RUN apk add --no-cache python3

WORKDIR /app
COPY requirements.txt .
RUN pip3 install --no-cache-dir -r requirements.txt

COPY . .

USER 1000
EXPOSE 8000
CMD ["python3", "app.py"]

podman build -t myapp:1.0 .

EXPOSE publishes nothing. It records the port the image expects to serve on, for humans and for tooling. Only -p at run time actually forwards anything, and believing otherwise is a common half-hour.

CMD versus ENTRYPOINT: CMD is a default that any argument replaces; ENTRYPOINT always runs and CMD supplies its default arguments. For a container that behaves like a command, use ENTRYPOINT. For one that behaves like a server you might want a shell in, use CMD.

USER matters and is skipped constantly. Without it the container runs as root, and root in a container is a smaller problem than root on a host but is not nothing. Set it to a non-root UID once the build steps that need privilege are done.

The caching rule

Each instruction is a layer, and layers are cached. On rebuild, the engine reuses cached layers until it reaches one whose inputs changed, and from that point everything below is rebuilt.

That single rule decides whether a rebuild takes two seconds or four minutes, and it dictates the order of the file:

COPY requirements.txt .
RUN pip3 install -r requirements.txt    # cached unless requirements.txt changed
COPY . .                                # invalidated by any source change

Dependencies before source code. Source changes on every commit; dependencies change rarely. Copying everything first and then installing means every one-line code change reinstalls every dependency.

Two more habits from that same file:

--no-cache on package managers, because a package index cached inside a layer is dead weight shipped to every user of the image.

Combine related RUN steps. RUN apt-get update && apt-get install -y foo && rm -rf /var/lib/apt/lists/* in one instruction, because deleting a file in a later layer does not reclaim the space, the earlier layer still contains it. A secret written and then removed in a later step is still in the image, which is worth remembering.

FROM golang:1.23 AS build
WORKDIR /src
COPY . .
RUN go build -o /app

FROM alpine:3.20
COPY --from=build /app /app
CMD ["/app"]

Networking

The container below was started with --name web. The command runs hostname inside it.

# Fedora CoreOS 44.20260707.3.1 on a virtual machine, aarch64
$ echo "--- a shell inside a running container ---"; sudo podman exec web sh -c "hostname; ls /etc/nginx/conf.d"; echo "--- and the container networks ---"; sudo podman network ls
--- a shell inside a running container ---
d75162c037d0
default.conf
--- and the container networks ---
NETWORK ID    NAME        DRIVER
2f259bab93aa  podman      bridge

The container ID, not the name. --name web is a label the runtime keeps so you can say podman exec web; the UTS namespace inside gets the short container ID as its hostname unless you pass --hostname explicitly. That trips up anything inside the container that identifies itself by hostname, application logs, metrics, and clustering software all report an ID that changes every time the container is recreated.

It also means the name is not resolvable by DNS on the default bridge, which is what the rest of this section is about.

A user-defined network is the one worth knowing, because it changes how containers find each other:

podman network create appnet
podman run -d --name db --network appnet postgres:16
podman run -d --name web --network appnet -p 8080:80 myapp:1.0

On a user-defined network, containers resolve each other by name. The application connects to db:5432 and the engine’s DNS resolves it, no IP addresses, no link flags, and it survives containers being recreated with different addresses.

On the default bridge there is no such DNS, which is why “connect by container name” advice sometimes works and sometimes does not.

--network host removes the isolation entirely: the container uses the host’s interfaces and ports, -p becomes meaningless, and a service binding port 80 binds the host’s port 80. Occasionally necessary for performance or for software that needs to see real interfaces; a significant loss of isolation otherwise.

The database in that example has no -p, and that is correct. It is reachable from web over appnet and from nowhere else. Publishing a database port to the host is a common and unnecessary exposure.

-v /srv/data:/data:Z     # private label, this container only
-v /srv/data:/data:z     # shared label, several containers

podman unshare chown -R 1000:1000 /srv/data

Across distributions

The image format is OCI and interchangeable everywhere. What differs is where things are stored and which extra restriction applies:

The :Z requirement is the practical difference. A bind mount that works on Ubuntu fails on RHEL with a permission error and identical file permissions, and that is SELinux.

Prove it

# What is attached to this container
podman inspect thename --format '{{range .Mounts}}{{.Type}} {{.Source}} -> {{.Destination}}
{{end}}'

# Where does this volume actually live
podman volume inspect appdata

# What is using disk
podman system df

# What layers does this image have, and how big
podman history myapp:1.0

# Can these two containers see each other
podman exec web getent hosts db

That first command is the one to run before removing any container. An empty mount list on something holding state means its data is in the writable layer and about to be deleted.

What trips people up

1. Data in the container

The writable layer is deleted with the container, and updating an image requires recreating the container.

A volume for anything durable. Check with podman inspect before removing anything.

2. EXPOSE does not publish

It is documentation in the image. Only -p at run time forwards a port.

3. Permission denied on a bind mount

Three candidates, in order: ordinary permissions, SELinux labelling on the RHEL family, and rootless UID mapping.

ausearch -m AVC -ts recent identifies the second. :Z fixes it, and relabels the host directory for real, so never point it at a system path.

4. Slow rebuilds

COPY . . early in the file invalidates the cache for everything after it on any source change.

Dependencies first, source last.

5. latest in production

A moving pointer. Two machines pulling it a month apart run different software.

Pin a version tag, or a digest.

Work it through

A team’s PostgreSQL container lost three weeks of data during a routine image update. You are asked to make sure it cannot happen again.

Reason it out before reading on.

What happened is now unambiguous. The container had no volume, so PostgreSQL wrote its data into the writable layer. Updating the image means podman rm and podman run with the new tag, and rm deleted the layer with everything in it.

Nothing failed. Every command reported success, and the data was destroyed by the documented behaviour of the model.

The fix, and each part earns its place:

podman volume create pgdata

podman run -d --name db \
  --network appnet \
  -v pgdata:/var/lib/postgresql/data \
  -e POSTGRES_PASSWORD_FILE=/run/secrets/pgpw \
  --secret pgpw \
  --restart=always \
  postgres:16.4

A volume at the data directory, so the state lives outside any container’s lifetime. No -p, because the database is reached over appnet by the application and should not be on the host’s network at all. A pinned version tag, not latest. And the password from a secret rather than an environment variable, because podman inspect shows environment variables to anyone who can run it.

Then verify it before trusting it, which is the step that would have caught the original problem:

podman exec db psql -U postgres -c 'create table t (i int); insert into t values (1);'
podman rm -f db
podman run -d --name db --network appnet -v pgdata:/var/lib/postgresql/data ... postgres:16.4
podman exec db psql -U postgres -c 'select * from t;'

Deliberately destroy the container and confirm the data comes back. That is five minutes and it is the only evidence that matters.

And the wider point for the team, because “add a volume” fixes one container:

A volume is not a backup. It lives on the same host and is removed by podman volume rm or by podman system prune --volumes. The backup lesson still applies: podman exec db pg_dump to something offsite, on a schedule, tested by restoring.

Now the point worth extracting. The container is the disposable part and the data is not, and the model requires you to say which is which. Nothing infers it. A container with no volume is a statement that it holds nothing worth keeping, and if that is not true, the failure arrives at the next update rather than at the next crash, which is why it catches people who have been running it successfully for weeks.

The habit: podman inspect --format '{{.Mounts}}' before removing any container, and treat an empty result on anything stateful as a stop.

Try it

Optional, on a machine with podman or docker.

1. podman run --name t1 alpine sh -c 'echo hi > /f.txt', then podman rm t1, then podman run --rm alpine cat /f.txt. Watch it fail.
2. podman volume create v1, then write and read across two containers with -v v1:/data.
3. podman volume inspect v1 and go and look at the directory on the host.
4. podman history nginx:alpine and find the largest layer.
5. podman system df, then podman system prune and compare.
6. podman network create n1, run two containers on it, and podman exec one getent hosts two.
7. Write a two-line Containerfile, build it twice, and watch the second build use the cache.

Verification step. You have it when you can look at a podman run command and say whether removing that container would lose anything.

Check yourself

COPY requirements.txt .
RUN pip3 install -r requirements.txt
COPY . .

podman network create appnet
podman run -d --name db --network appnet postgres:16
podman run -d --name web --network appnet myapp:1.0

References

• Containerfile / Dockerfile reference - Docker. Accessed 2026-08-07.
• podman-run(1) - Podman project. Accessed 2026-08-07.
• podman-volume(1) - Podman project. Accessed 2026-08-07.
• podman-network(1) - Podman project. Accessed 2026-08-07.
• OCI Image Format Specification - Open Container Initiative. Accessed 2026-08-07.
• overlayfs - The Linux Kernel documentation. Accessed 2026-08-07.

Every block above with a distribution and architecture header was captured by running the command on a Fedora CoreOS 44.20260707.3.1 virtual machine. Blocks without one are illustrative.