Before you read. A monitoring system alerts on link utilisation above 70 per cent. It has fired at around two in the morning every night for four months, and every time somebody has looked, the answer has been the backup job.
Last Thursday afternoon a link ran at 65 per cent for six hours during an incident nobody spotted until users called.
The threshold is doing exactly what it was configured to do. What is wrong with it?
Monitoring is the part of this exam most easily reduced to a list of product categories. The categories are worth knowing and they are not the difficult part. The difficult part is that a metric is meaningless on its own, and every mistake below follows from forgetting that.
Some words you will need
- baseline
- What normal looked like, recorded before you needed to know.
- threshold
- The value at which something is meant to get a human's attention.
- anomaly
- A departure from the baseline rather than from a fixed number.
- alert fatigue
- What happens to people who receive alerts that are usually nothing.
- syslog
- The protocol devices use to send log messages somewhere else.
- SIEM
- Security information and event management. Log aggregation with correlation and alerting on top.
What breaks without this
An alert becomes noise. It fires nightly, it is never anything, and after four months nobody reads it. The mechanism that was supposed to protect you is now training people to ignore it.
A real problem sits under the threshold. Because the threshold was set for the peak rather than for the pattern.
A graph is wrong and looks fine. A counter wrapped, or the agent answered from a cache, and the number that came back is plausible and false.
A number is not evidence
Latency of 412 milliseconds is not good or bad. It is a number, and nothing about it says which.
A baseline is what normal looked like, recorded over long enough to include the variation that is ordinary. Time of day, day of week, and the month somebody runs a quarter end job.
The consequence is worth stating plainly, because it is the reason monitoring gets set up after the first bad week rather than before it. You cannot measure the past retrospectively. Whatever you are not collecting today is a question you will not be able to answer in three months.
If you already keep baselines: the baseline that recorded a fault as normal
A baseline records what happened rather than what should have happened, and nothing in the process distinguishes the two.
Take a link that has been quietly running at ninety percent every afternoon for a year. The baseline learns that, the thresholds are set around it, and the system is by its own measure healthy. Everybody using that path has been suffering all along, and the monitoring has been trained to consider it ordinary. The same happens with a service that has always taken four seconds to respond, and with error counters that have never been zero.
Which is why a baseline needs a sanity check against what the thing is supposed to do, taken once when it is established. Not a statistical check, since the data cannot contradict itself, but somebody looking at the numbers and asking whether that is acceptable. A link at ninety percent is a capacity decision somebody should be making rather than a normal to be preserved.
The related failure is baselining during an unusual period. A baseline built over a holiday, or during a migration, or in the fortnight after an office moved, learns a shape that never returns, and the alerts that follow are all noise. The cheap defence is to record when a baseline was taken and to re-take it after anything that changed the business rather than the network.
A threshold against a shape
Once you have a baseline, the obvious next step is to draw a line across it and alert above the line. This works for metrics that are flat and misfires on every metric that is not.
Three ways out, and they are worth knowing as a set because products present them as features rather than as answers to this specific problem.
Alert on the departure rather than the value. Compare against what this metric does at this hour on this day of the week, and alert when it differs. Products call this anomaly detection or dynamic thresholds. Day five fires; the backup job does not, because it is what Tuesday at two always looks like.
Alert on duration rather than on a sample. Sustained for six hours is a different event from a peak lasting twenty minutes, and most alerting can require a condition to persist. This alone fixes a great deal.
Suppress by schedule. Say that between two and four in the morning the threshold is higher. Crude, entirely effective, and it goes stale silently when the backup window moves, which is the argument for the first option.
And alert fatigue is a mechanism rather than a failing. An alert that is usually nothing teaches the person receiving it that it is usually nothing, and they are correct. Four months of nightly false alarms is training, and it works. By the time a real one arrives the response time is measured against the previous hundred, not against the severity of this one. That is worth saying to whoever insists the alert stays as it is, because the choice is not between an alert and no alert. It is between an alert somebody reads and one they do not.
If you already own an alerting system: the cost of a false positive, and why it is not symmetric
Tuning is usually discussed as balancing false positives against false negatives, and the two are not comparable costs.
A false negative is one missed event. It is expensive when it happens and it does not compound. A false positive is different: it costs the time of whoever investigated, and it also spends a little of the credibility of every future alert from that source. Do it often enough and the alert stops being read, at which point the system has produced a false negative for every real event as well, without anybody deciding to disable anything.
That asymmetry is why an alert nobody acts on is worse than no alert. It occupies the place a working one would have taken, and the rota still believes it is covered.
The practical consequence is that a new alert should have to justify itself: what will somebody do when it fires, and what happens if they do nothing. An alert with no answer to the first question is a dashboard entry rather than an alert, and moving it to a dashboard is not a demotion. It is putting it where somebody will look at it when they are asking rather than interrupting them when they are not.
Reviewing which alerts fired and what came of them, quarterly, is the other half. The ones that fired forty times and produced no action are the ones spending the credibility of everything else.
Two ways a counter lies
The metrics behind all of this arrive as counters, and counters are less straightforward than they look.
The first problem is that a counter is not a rate. ifInOctets counts
upwards forever, so the useful number is the difference between two readings
divided by the time between them, and that arithmetic assumes both readings are
accurate as of the moment they were taken.
An interface octet counter, polled twice a few seconds apart. Which of the two numbers tells you anything, and what has to be done to it first?
# Fedora CoreOS 44.20260707.3.1, kernel 7.1.3-200.fc44.aarch64
# linux network namespaces, topology managed-lan
# the octet counter on the agent, before anything happens
$ ip netns exec mon snmpget -v2c -c s3cr3t-ro -On 10.0.0.20 1.3.6.1.2.1.2.2.1.10.6
.1.3.6.1.2.1.2.2.1.10.6 = Counter32: 54
# twenty frames of 1400 bytes, which is about 28.8 kB on the wire
$ ip netns exec mon ping -c 20 -i 0.2 -s 1400 -q 10.0.0.20 | tail -1
rtt min/avg/max/mdev = 0.047/0.101/0.197/0.037 ms
# poll immediately. the agent answers, and the answer is already out of date
$ ip netns exec mon snmpget -v2c -c s3cr3t-ro -On 10.0.0.20 1.3.6.1.2.1.2.2.1.10.6
.1.3.6.1.2.1.2.2.1.10.6 = Counter32: 21816
$ sleep 6
# poll again. nothing has been sent in between
$ ip netns exec mon snmpget -v2c -c s3cr3t-ro -On 10.0.0.20 1.3.6.1.2.1.2.2.1.10.6
.1.3.6.1.2.1.2.2.1.10.6 = Counter32: 29158
# what the kernel on the agent had all along
$ ip netns exec agent grep agent0 /proc/net/dev
agent0: 29248 26 0 0 0 0 0 0 29199 25 0 0 0 0 0 0The traffic finished before the second command returned. The poll immediately afterwards reported 21 816 bytes. Six seconds later, with nothing else sent, the same poll reported 29 158, and the kernel’s own figure was 29 248 the whole time.
The agent was answering from a cache. That is a property of this particular agent rather than of the protocol, and agents that cache are common, because rebuilding an interface table for every request on a device with four hundred ports is expensive. The consequence is general: a poll returns a value that was true recently, and polling faster than the cache refreshes produces a graph that alternates between zero and double, from an interface with perfectly steady traffic.
The second problem is the size of the counter. The interfaces MIB defines
ifInOctets as a 32-bit counter, and 32 bits is not many at modern speeds.
Two counter definitions for what sounds like the same thing. What is different about them, and why does a fast link need the second?
# Debian 13 (trixie), x86_64
$ snmptranslate -Td IF-MIB::ifInOctets; echo; snmptranslate -Td IF-MIB::ifHCInOctets
IF-MIB::ifInOctets
ifInOctets OBJECT-TYPE
-- FROM IF-MIB
SYNTAX Counter32
MAX-ACCESS read-only
STATUS current
DESCRIPTION "The total number of octets received on the interface,
including framing characters.
Discontinuities in the value of this counter can occur at
re-initialization of the management system, and at other
times as indicated by the value of
ifCounterDiscontinuityTime."
::= { iso(1) org(3) dod(6) internet(1) mgmt(2) mib-2(1) interfaces(2) ifTable(2) ifEntry(1) 10 }
IF-MIB::ifHCInOctets
ifHCInOctets OBJECT-TYPE
-- FROM IF-MIB
SYNTAX Counter64
MAX-ACCESS read-only
STATUS current
DESCRIPTION "The total number of octets received on the interface,
including framing characters. This object is a 64-bit
version of ifInOctets.
Discontinuities in the value of this counter can occur at
re-initialization of the management system, and at other
times as indicated by the value of
ifCounterDiscontinuityTime."
::= { iso(1) org(3) dod(6) internet(1) mgmt(2) mib-2(1) ifMIB(31) ifMIBObjects(1) ifXTable(1) ifXEntry(1) 6 }Two objects, the same description, one word different. Counter32 holds
4 294 967 295 and then starts again at zero, and at a gigabit that takes about
thirty four seconds.
The practical rule is short. Poll the 64-bit objects on anything faster than a
few megabits, which is everything. They live under ifXTable, the extension
table, which is a separate part of the MIB, and a monitoring system configured
years ago on slower equipment may still be reading the 32-bit ones.
The rest of what a monitoring system does
Metrics are one of the two things being collected. The other is logs, and the exam separates three ideas that get used interchangeably.
Syslog is the protocol. A device sends a message to a collector over the network, traditionally on udp/514. Each message carries a facility and a severity, which is what makes filtering possible. RFC 3164 describes what the original implementations did and RFC 5424 is the standard that replaced it, which matters because a lot of equipment still sends the older format.
Log aggregation is putting them in one place. The value is not sophisticated: it is that a device which has crashed cannot be logged into, and its last words are already somewhere else. It also means searching one system instead of forty.
Security event management adds correlation. A SIEM takes logs from many sources, normalises them, and applies rules across them, so a failed login on one device and a successful login from the same address on another can be a single finding. That is the part that distinguishes it from a log server, and it is also the part that generates most of its cost and most of its false positives.
Two more things belong in the same list, and both come up as exam vocabulary.
API integration is how a monitoring system reads things that do not speak SNMP. Cloud platforms, virtualisation managers and modern equipment expose a REST interface, and reading it is frequently the only way to see anything at all.
Network discovery is the system finding devices rather than being told about them, by sweeping address ranges, reading neighbour tables and following what it finds. It can run on a schedule or on demand, and the interesting output is not the list. It is the difference between the list and the inventory topic 36 described, because everything on one and not the other is something nobody knew about.
If you already work on networks: what to alert on when you cannot alert on everything, and why availability monitoring lies about its own accuracy
The instinct with a new monitoring system is to alert on every metric it can collect, and the result is a system nobody trusts within a month. A more useful question is which small set of signals catches most real problems.
One widely used answer, from the practice of running large services, is to alert on four things per service: how much traffic it is handling, how many of those requests fail, how long they take, and how full the resource is. The names vary between organisations. The reason it works is that almost every user-visible problem shows up in at least one of them, and almost nothing else does without also showing up there.
Applied to a network the four become traffic on the link, errors and discards on the interface, latency across it, and utilisation against capacity. Note what is absent: CPU on the switch is not on the list. It is a fine thing to graph and a poor thing to page somebody about, because a busy CPU with no user impact is common and a switch dropping packets with a quiet CPU is entirely possible.
Availability monitoring deserves its own warning, because it produces the number that gets quoted in meetings. A system that pings a device every sixty seconds and reports 99.98 per cent availability is reporting the fraction of its own probes that succeeded, which is not the same statement as the fraction of time the device was up. An outage lasting forty seconds between two successful probes contributes nothing to the figure, and the device was still down.
The measurement is also as reliable as the path to it. A probe crossing the same failed link as the users cannot distinguish a device that is down from a device it cannot reach, and both are frequently reported as down. That is why availability figures from a single monitoring station are worth reading as approximate, and why anybody who needs a real number probes from more than one place.
The last one is discards against errors, which sit next to each other in the interface table and mean different things. An error is a frame that arrived damaged, which points at cabling, optics or a duplex mismatch. A discard is a frame that arrived intact and was thrown away anyway, usually because a queue was full, which points at congestion or at policy. Alerting on the sum of the two hides which of those you have, and they lead to entirely different work.
Across platforms
Every machine keeps the same cumulative counters an agent would expose, and reading them twice is the whole of what a poller does.
On Linux they are in /proc/net/dev, which is what the capture above compared
the agent’s answer against.
On macOS, netstat prints them.
# macOS 26.5.2, arm64
$ netstat -ib | awk 'NR==1 || $1 ~ /^en0$/' | head -3
Name Mtu Network Address Ipkts Ierrs Ibytes Opkts Oerrs Obytes Coll
en0 1500 <Link#7> c6:66:6c:d3:09:29 11259 0 14571601 4707 0 1052393 0
en0 1500 iad20-gt102 fe80:7::c9c:87f5: 11259 - 14571601 4707 - 1052393 -
# Two seconds later, and the difference between the two is the only useful number
$ sleep 2
$ netstat -ib | awk 'NR==1 || $1 ~ /^en0$/' | head -3
Name Mtu Network Address Ipkts Ierrs Ibytes Opkts Oerrs Obytes Coll
en0 1500 <Link#7> c6:66:6c:d3:09:29 11309 0 14583501 4781 0 1118512 0
en0 1500 iad20-gt102 fe80:7::c9c:87f5: 11309 - 14583501 4781 - 1118512 -
# Errors and drops, which are the counters worth alerting on rather than graphing
$ netstat -i | head -3
Name Mtu Network Address Ipkts Ierrs Opkts Oerrs Coll
lo0 16384 <Link#1> 327 0 327 0 0
lo0 16384 127 localhost 327 - 327 - -On Windows there are two routes, and the difference between them is the difference between reading a counter and sampling it.
# Microsoft Windows Server 2025 Datacenter, version 10.0.26100.0
> Get-NetAdapterStatistics | Format-Table Name, ReceivedBytes, SentBytes -AutoSize
Name ReceivedBytes SentBytes
---- ------------- ---------
Ethernet 3 18328437 2704363
vEthernet (nat) 0 3636
Ethernet 4 15983896 2753574
# Two seconds later. The difference is the only number that means anything
> Start-Sleep -Seconds 2
> Get-NetAdapterStatistics | Format-Table Name, ReceivedBytes, SentBytes -AutoSize
Name ReceivedBytes SentBytes
---- ------------- ---------
Ethernet 3 18331995 2716827
vEthernet (nat) 0 3636
Ethernet 4 15984976 2766056
# The same values as a rate, sampled rather than differenced by hand
> Get-Counter -Counter "\Network Interface(*)\Bytes Total/sec" -MaxSamples 1 | Select-Object -ExpandProperty CounterSamples | Select-Object -First 2 InstanceName, CookedValue | Format-Table -AutoSize
InstanceName CookedValue
------------ -----------
mellanox connectx-4 lx virtual ethernet adapter 0.00
microsoft hyper-v network adapter _3 0.00The last command is the one worth noticing. Performance counters do the differencing for you and hand back a rate, which is convenient and hides the subtraction that everything on this page has been about.
Prove it
Read a counter twice. On any machine, using any of the commands above. Work out the rate by hand, then check it against whatever graph you have for the same interface over the same period.
Find the oldest alert rule you have access to. Ask who set it, when, and what it has fired on in the last three months. If the answer is that it fires regularly and is never anything, you have found a live example of the figure on this page.
Check what your monitoring polls for interface traffic. If it is the 32-bit objects on a gigabit link, the graphs have been wrong at every busy moment and right the rest of the time, which is the worst way for a graph to be wrong.
What trips people up
1. Reading a metric without a baseline
A number on its own carries no verdict. The same latency figure is a normal Tuesday on one link and the worst reading of the fortnight on another.
2. Setting one fixed threshold on a cyclical metric
A backup window and a working afternoon are two different normals, and one line across both either fires nightly or misses the daytime problem.
3. Treating alert fatigue as a discipline problem
An alert that is usually nothing teaches people it is usually nothing. That is learning working correctly, and the fix is to the alert.
4. Polling 32-bit counters on fast links
ifInOctets wraps in about thirty four seconds at a gigabit. The 64-bit objects
in the extension table are what to poll, and a monitoring system set up years ago
may still be reading the old ones.
5. Believing a poll returns the value at the instant it is asked
Agents cache interface tables, and polling faster than the cache refreshes produces a graph that alternates between nothing and double.
6. Confusing a log server with a SIEM
Aggregation puts logs in one place, which is worth doing on its own. Correlating across sources and alerting on the combination is the part that makes it a SIEM.
Work it through
The 70 per cent threshold that fires nightly and missed a six hour incident.
The first thing to do is resist changing the number, because both directions are worse. Raise it and the daytime blind spot widens. Lower it and every ordinary afternoon starts alerting, which produces the same fatigue faster.
The number is not the problem. The shape is. This link has two normals: a quiet daytime pattern and a loud nightly one, and a single line cannot describe both. So the fix is to change what the alert is comparing against.
The lowest effort version is duration. Requiring the condition to hold for, say, thirty minutes removes most short peaks and keeps the six hour event, and most alerting systems support it with one setting. It does not fix the backup, which lasts longer than thirty minutes, but it halves the noise for very little work.
The version that actually fixes it is comparing against this hour on this day. Two in the morning on a weekday has a baseline of its own, the backup fits it, and day five’s afternoon does not fit the afternoon baseline at 65 per cent. That is the same fault the figure showed, now expressed as something a system can decide.
The schedule based suppression is worth mentioning because somebody will propose it and it is not wrong. Raising the threshold between two and four in the morning takes five minutes and works until the backup window moves, at which point it fails silently and nobody finds out for months. Take it as an interim measure and write down when to revisit it.
And then there is the question the ticket does not ask. Nobody noticed a six hour incident until users called, on a link that was being monitored the whole time. The alerting failed, and so did everything else: no dashboard was being watched, no anomaly was flagged, and the four months of nightly false alarms had already established that alerts from this system do not mean much. Fixing the threshold fixes the first of those. The other two are about what people do with the output, and no configuration change reaches them.
Try it
Plot one week of one metric. Anything, from any system you have. Look at the shape before deciding where a line would go, because the shape is usually the argument against a line.
Wrap a counter on paper. Take a link speed, divide 4 294 967 296 by the bytes per second, and find the wrap time. Doing it once for the speeds you actually run makes the 64-bit rule stop being arbitrary.
Audit one alert. Pick one that fires often and ask what action it is supposed to prompt. Alerts that prompt no action are the ones to delete, and deleting them is what makes the rest readable.
Check yourself
A link is at 412 milliseconds of latency. Is that a problem?
There is no answer without a baseline. On a satellite link or a long haul path that has always run near 400 milliseconds, it is an ordinary reading. On a link that has not exceeded 90 milliseconds in a fortnight, it is the worst reading in the record.
That is the whole argument for collecting a baseline before you need it. The figure only becomes evidence once there is something to compare it against, and the comparison cannot be collected retrospectively.
An alert at 70 per cent utilisation fires nightly on a backup job and missed a six hour incident that peaked at 65. What do you change?
Not the number. Raising it widens the daytime blind spot and lowering it alerts on every ordinary afternoon.
The metric has two normals, one for the backup window and one for the working day, so the alert has to compare against the pattern rather than against a constant. Alerting on a departure from this hour on this day catches the incident and ignores the backup. Requiring the condition to persist is a cheaper partial fix that removes short peaks without touching the backup.
Why is polling ifInOctets on a gigabit interface every five minutes unreliable?
Because it is a 32-bit counter. It holds 4 294 967 295 and starts again, which at a gigabit takes about thirty four seconds, so a five minute interval can contain eight wraps.
Nothing in the counter records how many times it went round. The monitoring system sees a smaller second sample, assumes one wrap, and reports a rate roughly a tenth of the real one, which looks entirely plausible. The 64-bit objects in the extension table exist for exactly this, and RFC 2863 recommends them above about 20 Mbit/s.
Two polls a few seconds apart return very different counter values from an interface with steady traffic. What is happening?
The agent is answering from a cache. Rebuilding the interface table for every request is expensive on a device with many ports, so many agents refresh it on an interval and answer from what they last built.
Polling faster than that interval gives one sample containing several seconds of traffic and the next containing none, so a steady link produces a graph alternating between zero and double. The fix is to poll no faster than the agent refreshes.
What does a SIEM do that a syslog server does not?
Correlation. A syslog server receives and stores messages from many devices, which is worth doing on its own, because a device that has crashed cannot be logged into and its last messages are already elsewhere.
A SIEM normalises messages from different sources into a common shape and applies rules across them, so events on separate devices can combine into one finding. A failed login here and a successful one there become a single item rather than two lines in two files.
References
- RFC 2863 - IETF, the interfaces group MIB, which defines both the 32-bit and 64-bit octet counters and states where each is appropriate. Free. Accessed 2026-08-12.
- RFC 5424 - IETF, the syslog protocol, including the facility and severity fields. Free. Accessed 2026-08-12.
- RFC 3164 - IETF, describing what the original BSD syslog implementations sent, which is still what a lot of equipment sends. Free. Accessed 2026-08-12.
- NIST SP 800-92 - NIST, on log management, including aggregation and retention. Free. Accessed 2026-08-12.
- Get-NetAdapterStatistics - Microsoft, for the Windows counters. Accessed 2026-08-12.
Pictures. A freely licensed file from Wikimedia Commons, downloaded and served from this site rather than linked across to somebody else’s server. Cropped to the status table and otherwise unaltered.
- Nagios Core, current network status by Dr.Lorenzeti, CC BY-SA 4.0.
Where the output came from. The counter polls ran on the managed-lan
namespace topology through blog/scripts/netlab.sh, with a real manager polling a
real agent, which is why the stale answer is a genuine one rather than a
constructed example. The two object definitions came from a Debian 13 container
through blog/scripts/capture.sh with the setup script
blog/scripts/setups/snmp-agent.sh, and they are the MIB module’s own text rather
than a paraphrase. The wrap figure is arithmetic rather than a capture: 4 294 967
296 bytes at 125 000 000 bytes per second is 34.36 seconds, and the two poll
values in it are computed from that rate, because reproducing the effect would
need a saturated gigabit interface for five minutes.
If you also work on Linux. Monitoring concepts on the Linux+ track covers the same ideas from the host’s point of view, where the metrics are processes and memory rather than interfaces, and the argument about baselines and thresholds is identical.