CI/CD and GitOps

Before you read. Your team has infrastructure as code, a Git repository, and Ansible. Everything is version controlled. And every release still involves somebody with production credentials on their laptop running ansible-playbook from a terminal at 6pm on a Thursday.

Nothing about that is version controlled. Not the moment it ran, not who ran it, not which branch they had checked out, and not whether the tests passed first.

The gap between “our configuration is in Git” and “our deployments come from Git” is where this lesson lives. Closing it is what CI/CD is for.

The vocabulary is heavier than the ideas. A pipeline is a script that runs somewhere else. A stage is one part of that script. GitOps is the observation that if the repository already describes the desired state, something ought to be continuously checking whether reality matches, and fixing it when it does not.

Some words you will need

continuous integration (CI)

Every change is merged into a shared branch often, and automatically built and tested when it is.

continuous delivery

Every change that passes CI is ready to release. A human still chooses when.

continuous deployment

Every change that passes CI is released, with no human gate. Same acronym, different promise.

pipeline

The ordered set of automated steps a change passes through on its way to production.

stage

One step in a pipeline. Lint, build, test, scan, deploy.

runner / agent

The machine that actually executes the pipeline. Usually a container, started fresh.

artefact

The output of a build, a tarball, a package, a container image. The thing that gets promoted.

promotion

Moving one already-built artefact through environments, rather than rebuilding per environment.

shift left

Moving checks earlier, toward the developer, where a failure is cheap.

GitOps

The repository is the source of truth, and an agent continuously reconciles reality to it.

reconciliation

Comparing actual state against desired state and correcting the difference.

drift

Reality having diverged from the declared desired state.

What breaks without this

The release depends on a person remembering. Steps get skipped under pressure, and the steps that get skipped are the boring ones, which is to say the tests.

“Works on my machine” survives to production. If the build only ever happens on a laptop, the laptop’s installed versions are an undeclared part of your build environment.

Nobody can say what is deployed. Without an artefact and a record of what was promoted, “which version is in production” becomes an act of archaeology.

Testing happens last, where it costs most. A bug found by a developer costs minutes. The same bug found in production costs an incident, a rollback, and a postmortem.

Credentials live on laptops. If deployment is manual, every person who might deploy needs production credentials permanently. If it is automated, the pipeline needs them and people do not.

Drift is permanent. Somebody fixes production by hand during an incident, the fix never reaches the repository, and the repository is now a lie that everybody still trusts.

The contract is the exit code

Before any of the tooling, understand the thing every CI system on earth is built on: a stage succeeds if its command exits zero and fails if it does not. That is the whole interface. GitHub Actions, GitLab CI, Jenkins, and a for loop in a shell script all agree on this and nothing else.

Which means a pipeline is only as honest as its exit codes. Here is a pipeline that looks completely reasonable:

#!/bin/bash
set -e
echo "stage: lint"
grep -q 'version' config.ini
echo "stage: test"
./run-tests.sh | tee test.log
echo "stage: deploy"
echo "shipped"

set -e says “stop at the first command that fails”, which is exactly what a pipeline should do. The test script genuinely fails, it prints a failure and exits 1. Watch what happens anyway:

# AlmaLinux 10.2, x86_64
$ ./pipeline.sh; echo "pipeline exit status: $?"
stage: lint
stage: test
3 tests run, 1 failed
stage: deploy
shipped
pipeline exit status: 0

It shipped. The tests failed in plain English on the screen, the deploy stage ran anyway, and the pipeline reported success to whatever was watching.

The cause is | tee test.log. The exit status of a pipeline in the shell is the exit status of the last command in it, and tee succeeded. run-tests.sh returning 1 was discarded the moment somebody added a pipe to capture the log.

# AlmaLinux 10.2, x86_64
$ sed -i "s/^set -e$/set -e -o pipefail/" pipeline.sh; echo "--- same pipeline, one word added to set ---"; ./pipeline.sh; echo "pipeline exit status: $?"
--- same pipeline, one word added to set ---
stage: lint
stage: test
3 tests run, 1 failed
pipeline exit status: 1

pipefail makes a shell pipeline return the status of the last command that failed, rather than only the last command. The deploy stage never runs, and the pipeline correctly reports failure.

The two versions of that script are one word apart. One ships broken code silently, forever, and looks green while doing it.

This is why nearly every well-written CI script begins:

set -euo pipefail

if ./run-tests.sh; then echo ok; fi     # -e does not trigger
./run-tests.sh && echo ok               # nor here
./run-tests.sh || echo "tests failed"   # nor here

deploy() {
  build_image        # if this fails...
  push_image         # ...this still runs
}
if deploy; then ...  # because deploy was called in a condition

export VERSION=$(get_version)   # get_version failing does not stop the script

VERSION=$(get_version)   # a plain assignment does propagate the failure
export VERSION

The stages, and what each is really for

Stage names vary by tool. The sequence is stable, because it is ordered by cost: cheap and fast checks first, so an obviously broken change fails in seconds rather than after a twenty-minute build.

Build once, deploy many. The single most important structural rule: the artefact built in the build stage is the artefact that goes to staging, and the same one goes to production. Rebuilding per environment means the thing you tested is not the thing you shipped, and every difference between them is unaudited.

Configuration is what varies between environments, not the artefact. Same container image, different environment variables, different secrets, different database endpoint.

Shift left, and DevSecOps

Shift left is one idea stated in a slightly annoying way: the earlier a problem is found, the cheaper it is to fix, so move the checks toward the start of the process.

Follow that arrow all the way and you arrive somewhere useful, the cheapest place to catch a problem is not the pipeline at all. It is the developer’s machine, before the commit exists.

DevSecOps applies the same arrow to security specifically. The traditional model was a security review at the end, as a gate, which meant it happened when changing anything was most expensive, and so it became a negotiation about what to accept rather than what to fix.

In practice DevSecOps is a set of pipeline stages, not a philosophy:

• Secret scanning, does this change contain a credential? (gitleaks, trufflehog)
• SAST, static application security testing, does the source contain a known-dangerous pattern? (semgrep, bandit, and for shell, shellcheck)
• Dependency scanning / SCA, do the libraries you pulled in have known CVEs? (grype, trivy, osv-scanner)
• Image scanning, does the built container have vulnerable OS packages? (trivy, clair)
• IaC scanning, does the Terraform or Kubernetes YAML declare something insecure, like a public bucket or a privileged container? (checkov, tfsec)
• SBOM generation, produce a software bill of materials so that when the next Log4Shell happens you can answer “are we affected?” in minutes rather than weeks. (syft)

Each is a command that exits non-zero when it finds something. That is the entire integration.

Catching it before it leaves your machine

Git will run scripts of yours at defined moments. They live in .git/hooks/, they are ordinary executables, and a hook that exits non-zero aborts the operation. Same contract as everything else in this lesson.

#!/bin/sh
if git diff --cached | grep -qE 'AKIA[0-9A-Z]{16}|BEGIN [A-Z ]*PRIVATE KEY'; then
  echo "pre-commit: refusing this commit, the staged diff contains a credential" >&2
  exit 1
fi

git diff --cached is the staged change, precisely what is about to be committed. Now somebody appends an access key to a config file and commits it:

# AlmaLinux 10.2, x86_64
$ echo "aws_key = AKIAIOSFODNN7EXAMPLE" >> config.ini; git add config.ini; git commit -m "add deploy credentials"; echo "git commit exit status: $?"
pre-commit: refusing this commit, the staged diff contains a credential
git commit exit status: 1

# AlmaLinux 10.2, x86_64
$ echo "aws_key = AKIAIOSFODNN7EXAMPLE" >> config.ini; git add config.ini; git commit -m "x" 2>/dev/null; echo "--- the change is still staged, and nothing was recorded ---"; git status --short; git log --oneline
--- the change is still staged, and nothing was recorded ---
M  config.ini
f550218 initial commit

Nothing was recorded and nothing was lost. The history still has one commit, and the change is still sitting in the index where the author can fix it. That is the ideal shape for a check: refuse, explain, and leave the work intact.

The hooks worth knowing:

GitOps: the repository is the source of truth

Everything so far is push-based. Something happens, a pipeline runs, and it pushes a change outward into your infrastructure.

GitOps inverts the direction. An agent runs inside the environment, holds the repository as the statement of what should be true, and continuously compares it against what actually is. When they differ, the agent corrects reality, not the other way round.

Four principles, from the OpenGitOps project:

1. Declarative. The desired state is described, not scripted.
2. Versioned and immutable. It lives in Git, so it has history and identity.
3. Pulled automatically, agents pull the desired state; nobody pushes to prod.
4. Continuously reconciled, agents keep converging reality toward it.

A reconciler is conceptually tiny. This one compares a directory against the committed state, and corrects it:

#!/bin/sh
cd /srv/desired
if git diff --quiet; then
  echo "reconcile: in sync, nothing to do"
else
  echo "reconcile: drift detected"
  git --no-pager diff --stat
  git checkout -- .
  echo "reconcile: actual state returned to desired state"
fi

Run it, then do the thing everybody does during an incident:

# AlmaLinux 10.2, x86_64
$ reconcile; echo "--- now somebody edits the server directly, at 2am, during an incident ---"; sed -i "s/max_connections = 200/max_connections = 5000/" app.conf; reconcile; echo "--- what does the file say now ---"; cat app.conf; reconcile
reconcile: in sync, nothing to do
--- now somebody edits the server directly, at 2am, during an incident ---
reconcile: drift detected
 app.conf | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
reconcile: actual state returned to desired state
--- what does the file say now ---
max_connections = 200
log_level = warn
reconcile: in sync, nothing to do

Read the last two lines carefully, because they are the whole argument. The 2am change is gone. Somebody raised max_connections to 5000 to get through an incident, and the reconciler put it back to 200 without asking.

That is the feature. It is also, the first time it happens to you, deeply annoying, and the annoyance is the point. There are exactly two honest responses to a reconciler reverting your fix:

• The fix was right, so commit it, and now the repository and reality agree and the next server built from it also gets the fix.
• The fix was wrong, and it has been removed automatically.

What is no longer possible is the third outcome, which is the one that used to happen every time: the fix stays on one server, nobody writes it down, and eighteen months later that machine is subtly different from its fifty siblings and nobody alive knows why.

Real GitOps tooling, Argo CD and Flux are the two you will hear named, does exactly this against a Kubernetes cluster instead of a directory, and adds a UI, health checks, and the ability to report drift rather than always correcting it.

Putting the pieces in order

For an ordinary web service, a complete path looks like this:

1. Developer commits. Pre-commit hook lints and scans for secrets locally.
2. Push to a branch. Pipeline runs lint, unit tests, SAST, dependency scan.
3. Pull request opened. Same checks run, plus a build, and the results become required status checks a human reviewer can see.
4. Review and approval. Branch protection means this cannot be skipped.
5. Merge to main. Build once, producing an artefact addressed by digest.
6. Scan the artefact, sign it, publish it to the registry.
7. Pipeline commits the new digest to the environment repository for staging.
8. The GitOps agent notices and reconciles staging to it. Smoke tests run.
9. Promotion to production is a commit to the production environment repository, often itself a pull request, requiring an approval.
10. The agent reconciles production. The deploy happened because you merged.

Note where the human decisions are: reviewing code, and approving a promotion. Everything else is a consequence.

Rollback in this model is a git revert, which is why keeping the artefact addressed by digest matters, reverting the commit restores the previous digest, and the previous digest is still exactly the bytes that were running before.

-  image: registry.example.com/app@sha256:9f86d081...
+  image: registry.example.com/app@sha256:2c26b46b...

Across distributions

A pipeline is mostly tool choices rather than distribution choices, and the hosted runners hide the question entirely. It comes back the moment you run your own, because the runner is a service on a machine somebody has to administer.

The bind-mount row is where a working pipeline stops working after a move. A runner that checks a repository out on the host and mounts it into a build container hits SELinux on the RHEL family, gets Permission denied on files whose mode bits are plainly fine, and the fix is :Z on the mount rather than anything about the pipeline. It is the single most common surprise in moving CI from an Ubuntu runner to a RHEL one.

Rootless is worth choosing deliberately rather than inheriting. A runner that executes arbitrary repository code as root on a machine inside your network is a large amount of trust to extend to whoever can open a pull request.

Prove it

The contract a pipeline rests on is the exit code, so most of verifying one is checking that failures actually fail:

# Does each step really report failure, or does it exit 0 regardless
<the step>; echo "exit: $?"

# A pipeline hides the failure of everything but the last command
set -o pipefail                 # then re-run the pipeline step

# Is the artefact the same one that was tested
sha256sum build/output.tar.gz   # compare against what the deploy pulled

# GitOps: does the cluster match the repository, and does it say so
kubectl diff -k overlays/prod   # or the reconciler's own status command

# Has anybody changed the running system by hand
kubectl get <resource> -o yaml | grep -A3 last-applied

echo "exit: $?" after a step is the cheapest useful habit here. A step that prints an error and exits 0 turns a pipeline into decoration: it goes green, it deploys, and it did not test anything. Every step is either trusted to gate a deployment or it should not be in the pipeline.

What trips people up

1. A step that fails and exits zero

The most common way a green pipeline means nothing. A shell script without set -e, a linter invoked in a way that only warns, or a test runner whose exit code was never checked. The pipeline is only as honest as its worst step.

2. A pipeline stage that passes and the pipeline still ships something else

Building once and promoting the same artefact is the rule for a reason. If the deploy stage rebuilds from source, it ships a binary that nothing tested, and the two can differ over a dependency that moved between the two builds.

3. Secrets in logs

An echoed variable, a set -x left in a script, or a tool that prints its own configuration on startup. CI logs are frequently readable by more people than the secret store is, and they are retained for months.

4. Manual changes to a GitOps-managed system

The reconciler will revert them, which is correct and surprising. Somebody fixes production by hand at 2am, it works, and twenty minutes later it breaks again with no human involved. The fix belongs in the repository, and the emergency route is to pause reconciliation deliberately rather than to fight it.

5. Mutable tags

Deploying :latest means the thing you tested and the thing that ran are related only by coincidence. Immutable tags, ideally a digest, make a deployment reproducible and a rollback meaningful.

6. Treating the pipeline as untested code

It is code that runs with credentials, and it usually gets no review, no tests, and no linting. A pipeline change that breaks the deploy is discovered at the worst possible moment, which is during a deploy.

Work it through

A pipeline reports success, the deployment completes, and the application is broken in production. The same commit passed every stage.

Reason it out before reading on.

Whether what ran is what was tested splits the problem in two. That is the fastest way to split the problem in two:

# the digest the pipeline built, against the digest running now
skopeo inspect docker://registry/app:abc123 | grep Digest
kubectl get pod -o jsonpath='{.items[0].status.containerStatuses[0].imageID}'

Two different digests means a rebuild happened somewhere between test and deploy, and the tests never saw this binary.

If they match, ask whether the stages tested anything at all. A green pipeline proves the steps exited zero, which is not the same claim:

# in the job log, find the test stage and read the summary line
# "0 tests ran" and "exit 0" are both green and both mean nothing

Look specifically for a test stage that collected no tests, a linter that ran against no files because a path changed, or a script without set -e whose middle command failed.

Look at what production has that the test environment does not. Configuration, secrets, and data are the usual three, and none of them are in the artefact:

kubectl diff -k overlays/prod

Close the gap rather than the incident. If the artefact differed, make the pipeline build once and promote by digest. If a stage was vacuous, make it fail when it collects nothing. Both are one-line changes that prevent a category.

The general point: “the pipeline passed” is a statement about exit codes, not about correctness. It is worth exactly as much as the weakest step in it, and finding out which step that is takes a few minutes on a quiet afternoon rather than during an incident.

Try it

Optional, and a local git repository is enough. No CI service required.

1. Write a three-line shell script where the second line fails and the third succeeds. Run it and check $?. Add set -e and run it again. This is the whole contract in two runs.
2. Do the same with a pipeline: false | tee out.txt; echo $?. Then set -o pipefail and repeat.
3. Add a pre-commit hook that runs a linter and rejects the commit on failure. Try to commit something broken and watch it stop you. That is shift-left in its smallest possible form.
4. Build a container image twice from the same source without pinning a dependency version, and compare the digests.

Verification step. Step 4 is right when the two digests differ, and you can say what changed between the builds even though the source did not. If they match, pin less and try again.

For the exam

Know the three C-words apart. Continuous integration is merge-and-test often. Continuous delivery is always ready to release, human decides. Continuous deployment is released automatically. Being asked to distinguish delivery from deployment is common.

Version control triggers the pipeline. If a question describes the trigger, it is a commit, a push, a tag, or a merge, not a schedule and not a person.

Shift left means testing earlier, and the security flavour of it is DevSecOps.

GitOps is defined by the repository being the source of truth plus continuous reconciliation. Automated deployment alone is not GitOps; the distinguishing feature is that an agent pulls and converges.

Build once, promote the artefact. Rebuilding per environment is the wrong answer whenever it is offered.

Exit codes are how stages report. Zero is success.

Where this sits

Lesson 56 gave you branches and merges; this lesson is what happens because of a merge. Lesson 57 gave you declarative desired state, and GitOps is that idea plus a loop that never stops running. Lessons 58 and 59 gave you the tools a pipeline usually calls.

The next lesson takes the containers from lesson 35 and asks what happens when there are forty of them that have to find each other.

References

• githooks(5) - Git. Accessed 2026-08-09.
• Git Book, chapter 8.3: Git Hooks - Git. Accessed 2026-08-09.
• Bash Reference Manual, the set builtin - GNU. Accessed 2026-08-09.
• OpenGitOps principles - CNCF OpenGitOps. Accessed 2026-08-09.
• SLSA supply chain framework - OpenSSF. Accessed 2026-08-09.

The commands here were run on a real machine, not written from memory. The pipeline, hook, and reconciler transcripts come from AlmaLinux 10.2 on x86_64. The pipeline that shipped failing tests really did exit 0; adding pipefail and changing nothing else really did stop it. If you want to see it for yourself, the scripts are short enough to retype in a scratch directory, and that is a better use of ten minutes than rereading this section.