Common network services

Before you read. Somebody reports that the website is down. You log into the web server and everything looks fine: the machine is up, load is low, disk is fine, and systemctl status nginx says active.

The website is still down.

Between somebody typing a URL and a file being read off this disk, how many separate things had to work?

DNS, routing, the firewall, the listening socket, the web server’s virtual host matching, the application behind it, the database behind that, and the permissions on the file. Eight, at least, and “the service is running” tests exactly one of them.

This lesson is a tour of the services you will meet and, more usefully, the order to check them in.

Some words you will need

daemon

A program that runs in the background providing a service. Conventionally named with a trailing d.

listening socket

An address and port a service has claimed, waiting for connections.

virtual host

One web server serving several sites, chosen by the hostname in the request.

reverse proxy

A web server that accepts a request and passes it to an application behind it.

stratum

How many hops a time source is from a reference clock. Lower is better.

What breaks without this

You cannot find anything on an unfamiliar machine. Config file locations differ by family and by package, and the difference between /etc/nginx/ and /etc/httpd/ is a real obstacle at 3am.

You check the wrong layer first, which is most of the time spent on a service outage.

Nothing that depends on time works, and the failures do not mention time.

The tour

PTP is the row most people have never configured, and the reason it exists is worth a paragraph. NTP over a normal network is accurate to somewhere between a few milliseconds and a few tens of milliseconds, which is fine for logs, certificates and Kerberos, and is nowhere near enough for a trading venue that has to timestamp orders, a telephone network synchronising radios, or a broadcast plant lining up video feeds. Precision Time Protocol gets to sub-microsecond by timestamping packets in the network card’s own hardware rather than in software, and by measuring the delay of the path rather than assuming it is symmetric.

On Linux that is the linuxptp package: ptp4l speaks the protocol and steers the card’s own clock, and phc2sys copies that clock into the system clock, because they are two separate clocks and synchronising one does nothing for the other. That split is the thing that catches people out the first time.

The two web servers are the ones you will meet most. nginx is event-driven, fast at serving files and proxying, and configured in one coherent syntax. httpd (Apache) is older, module-based, and has .htaccess for per-directory overrides. Both are correct choices; nginx is the more common default for new work and Apache is extremely common in place.

Where the configuration lives

# AlmaLinux 10.2, x86_64
$ dnf install -y -q nginx >/dev/null 2>&1; echo '--- the main config, and which package owns it ---'; rpm -qf /etc/nginx/nginx.conf; echo '--- where the drop-ins go ---'; ls -d /etc/nginx/conf.d /usr/share/nginx/html 2>/dev/null; echo '--- and the port it will listen on ---'; grep -m1 'listen' /etc/nginx/nginx.conf
--- the main config, and which package owns it ---
nginx-core-1.26.3-6.el10_2.5.x86_64
--- where the drop-ins go ---
/etc/nginx/conf.d
/usr/share/nginx/html
--- and the port it will listen on ---
        listen       80;

rpm -qf on a config file is the fastest orientation on an unfamiliar machine. Note the answer: nginx.conf is owned by nginx-core, not nginx. That is why searching for a package named after the command sometimes finds nothing useful, and it is lesson 31’s query tools doing real work.

The layout by family:

The sites-available and sites-enabled pattern is Debian’s, and the symlink between them is the enable switch: a2ensite and a2dissite just create and remove it, which is lesson 25 again. The RHEL family has no equivalent: everything in conf.d/ is active, and disabling means renaming the file so it no longer ends in .conf.

Every one of these has a config test, and using it before reloading is the single best habit in this lesson:

sudo nginx -t
sudo apachectl configtest
sudo named-checkconf
sudo postfix check
sudo sshd -t

A syntax error found by nginx -t costs five seconds. The same error found by systemctl restart nginx on a live server costs an outage, because the old process has already stopped.

Time, and why it is first

chronyc tracking reports how far the local clock is from the time source it has settled on. This machine has been running for hours with a working network.

# Fedora CoreOS 44.20260707.3.1 on a virtual machine, aarch64
$ echo "--- is the clock in sync, and with what ---"; chronyc tracking 2>/dev/null | head -5 || timedatectl show-timesync 2>/dev/null | head -5; echo "--- sources ---"; chronyc sources 2>/dev/null | head -4
--- is the clock in sync, and with what ---
Reference ID    : 178FC4CB (23.143.196.203)
Stratum         : 3
Ref time (UTC)  : Sat Aug 08 03:18:24 2026
System time     : 0.000838063 seconds slow of NTP time
Last offset     : -0.000397323 seconds
--- sources ---
MS Name/IP address         Stratum Poll Reach LastRx Last sample               
===============================================================================
^+ 172.104.209.204               5  10   377   560  +2341us[+1943us] +/-   50ms
^* 23.143.196.203                2  10   377   548  -3711us[-4108us] +/-   40ms

838 microseconds. Under a millisecond, held there continuously by small corrections rather than jumps. That precision is why the five-minute tolerances elsewhere feel generous. A machine that has drifted into breaking Kerberos has not drifted, it has lost its time source entirely, and chronyc tracking is the one command that distinguishes those two.

The ^* in the sources list marks the server currently selected; ^+ is a candidate being kept as a cross-check. Reach 377 is an octal bitmask of the last eight polls, so 377 means all eight arrived, anything less is packet loss to that source.

chronyc tracking is the health check and System time: 0.000838 seconds slow is what healthy looks like, under a millisecond.

In chronyc sources, the leading character is the state: * the source currently being used, + an acceptable alternative, ? unreachable, x believed to be wrong. Stratum is distance from a reference clock. Reach is an octal history of the last eight polls, so 377 means eight out of eight and anything less means packet loss.

Time is a prerequisite for things that do not mention time in their errors:

• TLS certificates have validity windows. A clock wrong by days produces certificate is not yet valid and looks like a certificate problem.
• Kerberos rejects tickets more than five minutes out, so a domain-joined machine with a drifting clock stops authenticating and reports a credentials error.
• Correlating logs across machines is impossible if their clocks disagree, which is discovered during an incident and not before.
• make and build systems compare timestamps, and a clock that jumped backwards makes them rebuild everything or nothing.

Use chrony. It is the default on both families now, converges faster than ntpd, and copes with laptops and VMs that suspend. timedatectl shows the summary; chronyc is the detail.

sudo ss -tlnp | grep :80

sudo firewall-cmd --list-all      # RHEL family
sudo ufw status                   # Ubuntu
sudo nft list ruleset             # either

The order to check

Work from the outside in, and stop at the first thing that fails.

# 1. Does the name resolve, and to the right place
getent hosts www.example.com
dig +short www.example.com

# 2. Can you reach the machine at all
ping -c2 <address>
nc -zv <address> 443

# 3. Is something listening, and on what address
sudo ss -tlnp | grep :443

# 4. Is the firewall allowing it
sudo firewall-cmd --list-all

# 5. Is the service healthy, and does its config parse
systemctl status nginx
sudo nginx -t

# 6. What does the service itself say
sudo journalctl -u nginx --since '10 minutes ago'
sudo tail -50 /var/log/nginx/error.log

# 7. Can the service read what it is serving
sudo -u nginx cat /var/www/html/index.html

Step 7 is the one people reach last and should reach sooner on the RHEL family, because SELinux denies access that the permission bits allow, and every visible piece of evidence looks correct. sudo ausearch -m AVC -ts recent is the check.

Logs

Web servers keep their own logs as well as the journal, and the two contain different things: the journal has startup and crash information, the access log has requests. A 403 appears in the error log and nowhere else.

Across distributions

The httpd versus apache2 split is the one that catches everybody, because it changes the package name, the service name, the config path, the log path, and the user the server runs as. A runbook written for one is useless on the other.

SELinux is the other big difference in practice. On the RHEL family, serving files from a non-standard directory requires the right context, semanage fcontext and restorecon, and without it every permission looks correct and access is denied.

Prove it

# Everything listening, and what owns it
sudo ss -tlnp

# Does the config parse, before you reload anything
sudo nginx -t

# Is it enabled as well as running
systemctl is-active nginx; systemctl is-enabled nginx

# What it said recently
sudo journalctl -u nginx --since '1 hour ago' --no-pager | tail -20

# End to end, from outside
curl -Iv https://example.com

ss -tlnp is the single most useful command in this lesson. It answers what is listening, on which address, and which process owns it, which covers “is it running”, “is it reachable”, and “is it the thing I think” at once.

What trips people up

1. Running but not listening where you think

0.0.0.0:80 is every interface; 127.0.0.1:80 is loopback only and unreachable from anywhere else regardless of the firewall.

ss -tlnp before anything else.

2. Reloading without testing the config

systemctl restart nginx on a broken config stops the running server and fails to start the new one, turning a typo into an outage.

nginx -t first, every time. And prefer reload to restart where the service supports it, because reload keeps serving while it re-reads.

3. Started but not enabled

The service runs now and does not come back after a reboot. Covered properly in the next lesson, and it belongs here too because it is discovered weeks later.

systemctl is-enabled alongside is-active.

4. The clock is wrong

Certificate errors, Kerberos failures, and logs that cannot be correlated. None of the messages mention time.

chronyc tracking and timedatectl. Check it early on any machine behaving strangely across a fleet.

5. SELinux, on the RHEL family

Files served from a non-standard path are denied with permissions that look perfect.

sudo ausearch -m AVC -ts recent, then semanage fcontext and restorecon. Never setenforce 0 as a fix.

Work it through

A site returns 502 Bad Gateway. nginx is running, nginx -t passes, and the server is otherwise healthy.

Reason it out before reading on.

502 is a specific message and it is the one that narrows this fastest. It means nginx accepted the request, tried to pass it to something behind it, and got nothing usable back. So nginx is working. It is talking to you, and the problem is between it and the application.

That immediately rules out DNS, routing, the firewall between client and server, the listening socket, and the virtual host. All of those would have produced a different failure.

What is it proxying to?

grep -rE 'proxy_pass|fastcgi_pass' /etc/nginx/

Suppose fastcgi_pass unix:/run/php-fpm/www.sock.

Is the thing at the other end running?

systemctl status php-fpm
ss -tlnp | grep php
ls -l /run/php-fpm/www.sock

Three common answers. The service is stopped, start it, then find out why it stopped, which the journal will say. The socket path is wrong, because a package update moved it; nginx’s error log names the path it tried. The socket exists and nginx cannot open it, which is a permissions or SELinux problem between two accounts.

nginx’s error log names the cause exactly:

sudo tail -20 /var/log/nginx/error.log

connect() to unix:/run/php-fpm/www.sock failed (2: No such file or directory) is a missing service. (13: Permission denied) is the socket’s permissions or an SELinux denial. Two different fixes, and the log distinguishes them without any guessing.

Now the point worth extracting. The HTTP status code told you which layer to look at, before you ran anything. 502 means the proxy is fine and the backend is not. 403 means the request reached a file and something refused it, permissions, or SELinux. 404 means the server looked and found nothing, so the virtual host matched and the path is wrong. 504 means the backend accepted and never answered, which is a slow application rather than a stopped one. And a timeout with no status at all means nothing reached nginx, which is the firewall or the address.

The habit: read the failure before choosing where to look. “The website is down” describes a symptom; the status code names a layer, and it is free.

Try it

Optional, on any machine.

1. sudo ss -tlnp and identify every listening service and the process behind it.
2. systemctl list-units --type=service --state=running and name what each is for.
3. chronyc tracking and chronyc sources, and read the state characters.
4. timedatectl and confirm NTP is active.
5. If a web server is installed: find its config with rpm -qf or dpkg -S, run its config test, and find its logs.
6. curl -Iv https://example.com and read the certificate and headers.
7. getent hosts example.com and dig +short example.com, and confirm they agree.

Verification step. You have it when you can be handed an HTTP status code and say which layer of the stack to investigate, before touching the machine.

Check yourself

References

• nginx documentation - nginx. Accessed 2026-08-07.
• Apache HTTP Server documentation - Apache Software Foundation. Accessed 2026-08-07.
• linuxptp - linuxptp project, for ptp4l and phc2sys and the two clocks. Free. Accessed 2026-08-21.
• chrony documentation - chrony project. Accessed 2026-08-07.
• BIND 9 ARM - Internet Systems Consortium. Accessed 2026-08-07.
• ss(8) - Linux man-pages project. Accessed 2026-08-07.
• CUPS documentation - OpenPrinting. Accessed 2026-08-07.

The service configurations here are from each project’s own documentation, since a container has no service manager to start them under. The rpm -qf and chrony captures are real. Blocks without a distribution and architecture header are illustrative.