Orchestration

Before you read. You have a container. It runs, you can reach it on a port, and lesson 35 covered how it works. Now the application needs a database, a cache, a reverse proxy, and a background worker. They have to find each other. They have to start in a sensible order. If one dies at 3am something should restart it. And when you deploy a new version, the site should not go down while it happens.

You could write all of that in shell. People did, for about eighteen months, and then stopped.

Orchestration is the layer that answers those questions. This lesson is scoped to the vocabulary and concepts the exam objective asks for, enough to read a Compose file, hold a conversation about a Kubernetes cluster, and know what the words mean. It is not a Kubernetes course, and it says so at the end.

The good news is that almost everything here is one idea repeated: you declare what should be running, and something else continuously makes it so. That is the same desired-state loop from lessons 57 and 60, pointed at workloads instead of files.

Some words you will need

orchestration

Managing many containers as one system: placement, networking, scaling, recovery.

Compose

A file format and tool describing a multi-container application on one host.

service

A definition of a container to run, and how many copies. The unit of scaling.

replica

One running copy of a service.

pod

One or more containers that share a network identity and are scheduled together. The unit Kubernetes actually runs.

node

A machine in a cluster that can run workloads.

task

In Swarm, one container placed on one node as part of a service.

scheduler

The component that decides which node a workload runs on.

control plane

The cluster's brain: stores desired state, schedules, and reconciles.

manifest

A YAML file declaring a Kubernetes object.

overlay network

A virtual network spanning several hosts so containers on different machines share a subnet.

ingress

How outside traffic gets into the cluster.

What breaks without this

Containers cannot find each other reliably. IP addresses change every time a container is recreated, so anything that hardcodes one breaks on the next deploy.

Startup order is manual and wrong. The application starts before the database is accepting connections, crashes, and stays crashed because nothing retries it.

A crash at 3am is an outage until somebody wakes up. There is no reason a human should be involved in restarting a process that died.

Scaling is a manual copy-paste. Running four copies means four podman run commands with four different port numbers, and a load balancer config you now maintain by hand.

Deploys are downtime. Stop the old container, start the new one, and whatever happened in between was an error page.

Nothing is scheduled. With more than one machine, deciding which host runs what becomes a spreadsheet, and the spreadsheet is wrong within a week.

Compose: the first real step up

Compose describes a multi-container application in one file. It solves the single-host case, and it is where most people meet orchestration.

services:
  web:
    image: nginx:alpine
    ports:
      - "8080:80"
    depends_on:
      - api
  api:
    image: myapp:1.4.2
    environment:
      DATABASE_URL: postgres://db:5432/app
    depends_on:
      - db
  db:
    image: postgres:16
    volumes:
      - dbdata:/var/lib/postgresql/data
    environment:
      POSTGRES_PASSWORD_FILE: /run/secrets/db_password

volumes:
  dbdata:

The one line worth pausing on is postgres://db:5432/app. The hostname is db, the service name. Compose creates a network for the application and registers each service in its DNS, so services address each other by name and never need to know an IP. That single behaviour is most of what Compose is for.

The commands are few:

down versus down -v is worth memorising in the direction that keeps your data. The flag that looks like a tidy-up is the one that destroys the database.

  db:
    image: postgres:16
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U postgres"]
      interval: 5s
      timeout: 3s
      retries: 10
      start_period: 30s
  api:
    depends_on:
      db:
        condition: service_healthy

The pod: several containers, one network identity

Kubernetes does not schedule containers. It schedules pods, and a pod is one or more containers that share a network namespace and are always placed together. Podman implements the same object, which means the concept can be demonstrated on one machine.

Create a pod and put two containers in it:

# Fedora CoreOS 44.20260707.3.1 on a virtual machine, aarch64
$ podman pod create --name app --publish 8080:80 >/dev/null; podman run -d --pod app --name worker docker.io/library/almalinux:10 sleep 300 >/dev/null; podman run -d --pod app --name sidecar docker.io/library/almalinux:10 sleep 300 >/dev/null; podman pod ps --format "{{.Name}} {{.Status}} {{.NumberOfContainers}}"; echo "--- what is in it ---"; podman ps --pod --format "{{.Names}} {{.Pod}}"
app Running 3
--- what is in it ---
ddf75d92570b-infra ddf75d92570b
worker ddf75d92570b
sidecar ddf75d92570b

Two containers were created and the pod reports three. The extra one is the infra container: it does nothing except exist, holding the namespaces open so that the containers you care about can come and go without the pod losing its network identity or its published port. Kubernetes has the same thing, called the pause container.

Now the claim that the containers “share a network”. A namespace has an inode number, and two processes in the same namespace see the same one:

# Fedora CoreOS 44.20260707.3.1 on a virtual machine, aarch64
$ echo "--- same network namespace? compare the inode ---"; podman exec worker readlink /proc/self/ns/net; podman exec sidecar readlink /proc/self/ns/net; echo "--- but separate pid namespaces, by default ---"; podman exec worker readlink /proc/self/ns/pid; podman exec sidecar readlink /proc/self/ns/pid
--- same network namespace? compare the inode ---
net:[4026532529]
net:[4026532529]
--- but separate pid namespaces, by default ---
pid:[4026532723]
pid:[4026532726]

The network inodes are identical and the PID inodes are not. That is the pod, stated as precisely as it can be stated. The two containers are the same host as far as the network is concerned (same IP, same port space, they reach each other on localhost) while remaining separate as far as processes are concerned.

Which explains the constraint people find surprising: two containers in one pod cannot both listen on port 80, because there is only one port 80 between them.

Self-healing, and what it does not know

The restart policy is the simplest form of self-healing. Run a container that crashes two seconds in, and leave it alone:

# Fedora CoreOS 44.20260707.3.1 on a virtual machine, aarch64
$ podman rm -f flaky >/dev/null 2>&1; podman run -d --restart=on-failure --name flaky docker.io/library/almalinux:10 sh -c "sleep 2; exit 1" >/dev/null; sleep 12; podman inspect flaky --format "restarts: {{.RestartCount}}, exit code {{.State.ExitCode}}, state {{.State.Status}}"; echo "--- a stop you asked for is not a crash ---"; podman rm -f calm >/dev/null 2>&1; podman run -d --restart=always --name calm docker.io/library/almalinux:10 sleep 600 >/dev/null; podman stop -t 0 calm >/dev/null; sleep 3; podman inspect calm --format "restarts: {{.RestartCount}}, state {{.State.Status}}"
restarts: 5, exit code 0, state running
--- a stop you asked for is not a crash ---
restarts: 0, state exited

Five restarts in twelve seconds, and the state is running. Nobody intervened. That is the loop doing its job, and it is also the failure mode worth internalising, because a monitoring dashboard reading only state would show this container as healthy while it has in fact never once succeeded.

The second half is the part people get wrong. --restart=always did not restart the container that was deliberately stopped: zero restarts, state exited. An orchestrator distinguishes a crash from an instruction. You asked for it to stop, so it stayed stopped, otherwise you could never take anything down.

HEALTHCHECK --interval=30s --timeout=3s --start-period=40s --retries=3 \
  CMD curl -fsS http://localhost:8080/healthz || exit 1

Swarm: the same ideas across several machines

Docker Swarm is built into the Docker engine and is the gentlest introduction to a real cluster. The exam names its objects, so know them.

docker swarm init                                  # this node becomes a manager
docker swarm join --token <worker-token> <ip>:2377 # from each other machine
docker node ls                                     # nodes and their roles
docker service create --name web --replicas 3 -p 80:80 nginx:alpine
docker service ls                                  # services and replica counts
docker service ps web                              # the individual tasks, and where they run
docker service scale web=10
docker service update --image nginx:1.27 web       # rolling update
docker stack deploy -c compose.yaml myapp          # a Compose file, across the cluster

The relationship is the thing to hold: you declare a service with a replica count; the manager creates that many tasks; the scheduler places each task on a node. If a node dies, its tasks are gone and the manager notices the gap between three desired and two running, so it schedules a replacement elsewhere. Nobody restarts anything. The count is simply made true again.

Two networking behaviours are named in the objectives:

The overlay network spans hosts. Containers on different physical machines get addresses on one virtual subnet and reach each other directly, with service names resolving through the swarm’s internal DNS. Underneath it is VXLAN encapsulation, which is why an overlay needs UDP 4789 open between nodes.

The routing mesh means a published port is published on every node. Connect to port 80 on any machine in the swarm, including one running no replicas of that service, and the request is forwarded to a node that has one. So an external load balancer can point at all nodes without knowing or caring where anything is scheduled.

docker service update \
  --image myapp:1.5.0 \
  --update-parallelism 2 \
  --update-delay 30s \
  --update-failure-action rollback \
  --update-order start-first \
  myapp

The Kubernetes vocabulary

The objective asks for vocabulary, so here is the vocabulary, what each object is, and what problem it solves. Everything below is declared in YAML, submitted to the control plane, and continuously reconciled.

The three-layer chain is the one to be able to recite: you write a Deployment; it creates a ReplicaSet; the ReplicaSet creates Pods. A new image means a new ReplicaSet, pods shift from old to new gradually, and the old ReplicaSet is kept at zero replicas, which is precisely what makes kubectl rollout undo possible.

Pods are cattle, and the Service is why that works. Pods get new IPs every time they are recreated, so nothing addresses a pod. Everything addresses a Service, which has a stable virtual IP and DNS name and forwards to whichever pods currently exist and are ready.

Podman can render a running pod as a Kubernetes manifest, which is a good way to see the shape of one without a cluster:

# Fedora CoreOS 44.20260707.3.1 on a virtual machine, aarch64
$ cd /var/tmp && podman kube generate app 2>/dev/null | head -40
# Save the output of this file and use kubectl create -f to import
# it into Kubernetes.
#
# Created with podman-6.0.2
apiVersion: v1
kind: Pod
metadata:
  annotations:
    io.kubernetes.cri-o.SandboxID/sidecar: ce39b5a3d4aa243f0c03fb1f69a3ac770564baee328cd491e8bbc4147a22de97
    io.kubernetes.cri-o.SandboxID/worker: ce39b5a3d4aa243f0c03fb1f69a3ac770564baee328cd491e8bbc4147a22de97
  creationTimestamp: "2026-08-09T00:49:30Z"
  labels:
    app: app
  name: app
spec:
  containers:
  - command:
    - sleep
    - "300"
    image: docker.io/library/almalinux:10
    name: worker
    ports:
    - containerPort: 80
      hostPort: 8080
  - command:
    - sleep
    - "300"
    image: docker.io/library/almalinux:10
    name: sidecar

Every manifest has the same four top-level keys. apiVersion and kind say what sort of object this is; metadata names and labels it; spec describes the desired state. Once you can see that pattern, an unfamiliar manifest becomes readable even when the spec is not.

And because the manifest is the complete description, it round-trips:

# Fedora CoreOS 44.20260707.3.1 on a virtual machine, aarch64
$ podman pod rm -f app >/dev/null 2>&1; podman pod create --name app --publish 8080:80 >/dev/null; podman run -d --pod app --name worker docker.io/library/almalinux:10 sleep 600 >/dev/null; podman kube generate app > /var/tmp/app.yaml; echo "wrote $(wc -l < /var/tmp/app.yaml) lines of YAML"; podman pod rm -f app >/dev/null 2>&1; echo "--- pod destroyed, now replay the file ---"; podman kube play /var/tmp/app.yaml 2>&1 | tail -4; podman pod ps --format "{{.Name}} {{.Status}} {{.NumberOfContainers}}"
wrote 24 lines of YAML
--- pod destroyed, now replay the file ---
284990462e5609c17b1e0d169bacf7190b089f8917ebde364062465359c2c659
Container:
2d21ce93def04b235b09e74ca373f5ac4bfb155509ee02252cf732c20f06d2bf

app Running 2

Twenty-four lines of text recreated the whole thing. That is the entire promise of declarative orchestration in one command: the running system is disposable and the description is what you keep. It is the same claim lesson 57 made about infrastructure, applied to workloads.

ConfigMaps and Secrets

Both inject configuration into a pod. The difference is smaller than the names suggest, and the gap between what people assume and what is true matters.

apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
data:
  LOG_LEVEL: "warn"
  MAX_CONNECTIONS: "200"
---
apiVersion: v1
kind: Secret
metadata:
  name: app-secret
type: Opaque
data:
  DB_PASSWORD: c3VwZXJzZWNyZXQ=

Consumed either as environment variables or as files:

    envFrom:
      - configMapRef:
          name: app-config
    env:
      - name: DB_PASSWORD
        valueFrom:
          secretKeyRef:
            name: app-secret
            key: DB_PASSWORD

c3VwZXJzZWNyZXQ= is base64. Decode it and it says supersecret. Base64 is an encoding, not encryption, and anybody who can read the Secret object can read the value with one command. The data field is base64 solely so that binary values survive YAML.

What a Secret actually gives you over a ConfigMap is worth stating precisely, because it is not nothing:

• It is a distinct object kind, so RBAC can grant access to ConfigMaps and deny it for Secrets. This is the real control.
• It is not written to the node’s disk, Secrets are mounted in tmpfs, in memory.
• It is only sent to nodes running a pod that uses it.
• kubectl describe redacts the values, where get -o yaml does not.
• It can be encrypted at rest in etcd, but only if the cluster administrator configured an EncryptionConfiguration. Unencrypted is the historical default, and on an unmanaged cluster it is the likely state.

Where this stops

The objective is scoped to vocabulary, and this lesson honours that. Everything beyond it (writing real manifests, kubectl in anger, RBAC, network policy, Helm, operators, cluster upgrades, etcd backups, autoscaling, admission control) is a subject of its own and is not on this exam.

What is fair to expect of you here: read a Compose file and say what it creates; explain what a pod is; name the Kubernetes objects and what each is for; and describe desired-state reconciliation without hand-waving.

If you want to go further afterwards, the path most people find works is to run a single-node cluster locally (kind, k3s, or minikube) and deploy something you wrote yourself. The concepts in this lesson are the ones that make that possible; the rest is practice.

Across distributions

Kubernetes behaves the same wherever it runs, because it is a layer above the machine. The container engine underneath it, and the tooling you use to build and run things locally, is where the families diverge sharply.

Podman having pods is more than a naming coincidence. The pod is the Kubernetes unit, and podman implements it locally, so podman pod create gives you several containers sharing one network namespace exactly the way a Kubernetes pod does. podman kube play will even run a Kubernetes YAML file directly. That makes a RHEL-family workstation a closer rehearsal for a cluster than a Docker one, where the pod has no equivalent and the mental model has to be rebuilt when you move up.

The daemon difference matters for troubleshooting. Docker runs containers as children of a root daemon, so systemctl status docker is a real question and the daemon dying takes everything with it. Podman forks containers from your own shell with no daemon at all, so there is no service to check and containers survive independently.

Prove it

Most orchestration faults are a workload that will not start or a workload that started and is not serving. These separate the two:

# What state is it actually in, and how many times has it restarted
kubectl get pods -o wide
kubectl describe pod <name>          # Events at the bottom are the diagnosis

# Why did the last one die
kubectl logs <pod> --previous
kubectl get events --sort-by=.lastTimestamp | tail -20

# Is it running but not ready, and which probe is failing
kubectl get pod <name> -o jsonpath='{.status.conditions}' | tr ',' '\n'

# Does the service have anything behind it
kubectl get endpoints <service>      # empty means the selector matches nothing

# Locally, the same questions
podman ps -a
podman logs <container>
podman pod ps

kubectl describe puts the Events list at the bottom, and that is the part to read first. Image pull failures, scheduling failures, probe failures, and volume mount failures all appear there in plain language, while the top of the output is configuration you already know.

An empty endpoint list is the other high-value check. A service with no endpoints means its label selector matches no running pod, so the traffic goes nowhere and nothing in the pod’s own logs will ever mention it.

What trips people up

1. Confusing a liveness probe with a readiness probe

Readiness controls whether traffic is sent. Liveness controls whether the container is killed and restarted. Getting them the wrong way round gives you either a pod receiving traffic before it can serve it, or a healthy pod being restarted forever because its startup is slower than the probe’s patience.

2. Expecting self-healing to understand the problem

A restart fixes a process that crashed. It does nothing for a bad configuration, a missing secret, or a dependency that is down, and the restart loop hides the original error behind a wall of identical startup attempts. Read the first failure with --previous.

3. Treating Secrets as encrypted

A Kubernetes Secret is base64, which is an encoding rather than a cipher. Anyone who can read the object can read the value. Encryption at rest is a cluster setting somebody has to turn on, and it is a separate decision from using Secrets at all.

4. Deploying a mutable tag

:latest means the pod that restarts at 3am may not be running the image you tested, because the tag moved. Deploy a digest, or at least an immutable tag, or a rollback restores a version number rather than a version.

5. A service with no endpoints

The selector and the pod labels have to match exactly. They frequently drift when a deployment is edited and its template labels change, and the failure is silent: the service exists, it resolves, and it forwards to nothing.

6. Assuming the pod is the thing you manage

You manage a Deployment, which manages a ReplicaSet, which manages pods. Deleting a pod gets you an identical replacement in seconds, which looks like the deletion failed. Change the Deployment.

Work it through

A newly deployed service returns connection refused from inside the cluster. The pod shows Running with 1/1 ready and no restarts.

Reason it out before reading on.

Running and 1/1 are claims about lifecycle, not about reachability. Ready means the readiness probe passed, and a probe that checks the wrong thing, or that was never configured, passes for a container that serves nothing:

kubectl get pod <name> -o jsonpath='{.spec.containers[0].readinessProbe}'

If that is empty, readiness is defaulting to “the process started”, which is almost no information at all.

Check whether the service points at this pod. Refused from inside the cluster is at least as likely to be a routing problem as an application one:

kubectl get endpoints <service>
kubectl get pods --show-labels
kubectl get service <service> -o jsonpath='{.spec.selector}'

An empty endpoints list with a running pod means the selector and the labels disagree, and comparing those two outputs shows it immediately.

If the endpoints are populated, test the port rather than the service. A mismatch between containerPort, the service targetPort, and what the application actually binds to produces exactly this symptom:

kubectl exec <pod> -- ss -ltnp
kubectl get service <service> -o jsonpath='{.spec.ports}'

Watch for the application binding to 127.0.0.1 inside the container, which works when you exec in to test it and refuses everything from outside.

Only now look at the application. Three checks have eliminated readiness, service routing, and port mapping, so if all three are right the problem is genuinely in the workload and its own logs are worth reading.

The reasoning worth keeping: a pod reporting Running and Ready is making a claim about its own lifecycle, not about whether anything can reach it. Between the pod and a client sit a probe, a label selector, and a port mapping, and each fails quietly in a way that looks like the application refusing connections.

Try it

Optional, and podman or Docker on one machine covers most of it without a cluster.

1. Create a pod with two containers sharing a network namespace: podman pod create -p 8080:80, then run a web server and a second container in it. From the second, reach the first on localhost. That is the whole point of a pod in one exercise.
2. Write a Compose file with a web container and a database, bring it up, and stop the database. Watch what the web container does, and note that nothing restarts the dependency for you.
3. Generate a systemd unit for a container with podman generate systemd, or write one for docker run, and confirm it survives a reboot.
4. Deliberately mismatch a label: run a container with a label the service or selector does not match, and observe that everything reports healthy while nothing routes.

Verification step. Step 1 is right when the second container reaches the first on localhost rather than on an IP address, and you can say why that works inside a pod and fails between two separate containers.

For the exam

Compose is one host. Swarm and Kubernetes are many. If a question involves several machines, Compose is not the answer.

docker compose down -v deletes volumes. Without -v, named volumes survive.

Services find each other by service name, resolved by the orchestrator’s internal DNS. Never by IP.

A pod is the smallest deployable unit in Kubernetes, and may hold more than one container sharing a network namespace.

You deploy a Deployment, not a Pod. The chain is Deployment, then ReplicaSet, then Pods.

A Service gives a stable address in front of ephemeral pods. ClusterIP is internal, NodePort exposes a port on each node, LoadBalancer asks the cloud for one.

A Secret is base64-encoded, not encrypted. Expect to be asked.

ConfigMap for non-sensitive configuration, Secret for sensitive.

Swarm objects: node, service, task. A service is the declaration; a task is one container of it on one node.

Restart policies act on exit, health checks act on behaviour, and a deliberate stop is not a crash.

Where this sits

Lesson 35 gave you a container. This lesson is what happens when there are forty. Lesson 60’s reconciliation loop is the same loop the control plane runs, pointed at pods rather than files, and Argo CD and Flux exist precisely to connect the two.

The last lesson in this block is about a tool that will happily write all of this YAML for you, and what you owe your colleagues before you run it.

References

• Compose file specification - Docker. Accessed 2026-08-09.
• Swarm mode key concepts - Docker. Accessed 2026-08-09.
• Kubernetes concepts: Pods - Kubernetes. Accessed 2026-08-09.
• Kubernetes concepts: Secrets - Kubernetes. Accessed 2026-08-09.
• podman-pod(1) - Podman. Accessed 2026-08-09.
• podman-kube-play(1) - Podman. Accessed 2026-08-09.

The commands here were run on a real machine, not written from memory. The pod, namespace, restart, and manifest transcripts come from Fedora CoreOS 44.20260707.3.1 on aarch64, a virtual machine, with Podman 6.0.2 running the pods natively. Podman implements the same pod object as Kubernetes, which is why the namespace inodes can be compared directly. The Swarm and Kubernetes command listings are from the vendor documentation in the sources rather than from a running cluster, and are shown without captured output for that reason.