Job control and scheduling

Before you read. You start a job that will take four hours. Your terminal is now useless, it belongs to that job until it finishes.

You could open a second terminal. Then your laptop sleeps, the SSH session drops, and the job dies with it.

Two different problems hiding behind one inconvenience: how do you get your terminal back, and how do you make a job outlive the session that started it? They have different answers, and confusing them is why “I ran it in the background and it still died” is such a common sentence.

The third problem is the one that matters most in practice, and it is what most of this lesson is about: how to make something run every night when nobody is logged in at all.

Some words you will need

job

A command the shell is managing. Numbered per shell, unlike a PID.

foreground

Has the terminal. You cannot type until it finishes.

background

Running, but the prompt is yours again.

SIGHUP

The signal sent to a session's processes when the terminal goes away.

cron

A daemon that runs commands on a schedule, whether anyone is logged in or not.

What breaks without this

Long jobs die at the worst moment, usually a migration or a restore, three hours in.

Nothing happens overnight. Backups, log rotation, certificate renewal, and report generation are all scheduled work, and a job that never fires fails silently by definition.

Or it happens far too often. A misread crontab field turns “monthly” into “every minute”, and cron is happy to comply.

Getting your terminal back

The usual rescue, when you have started something long without thinking:

Ctrl+Z          # suspend it
bg              # carry on in the background
jobs            # confirm

Job numbers belong to one shell. %1 in this terminal is unrelated to %1 in another, and jobs in a new session shows nothing. PIDs are global; job numbers are not.

Ctrl+Z sends SIGSTOP and the process enters state T from the previous lesson. A job left suspended is not running at all, which is worth remembering before wondering why it never finished.

Surviving the disconnect

This is the second problem, and & does not solve it.

When a terminal closes, the kernel sends SIGHUP to the processes in that session, and the default action is to terminate. A background job is still in that session, so it dies with everything else.

nohup makes the process ignore SIGHUP and redirects its output, since there will be no terminal to write to:

nohup ./long-migration.sh > migration.log 2>&1 &

tmux is the better answer for interactive work. The session lives on the server, and you attach and detach from it:

tmux new -s migration     # start
# Ctrl+B then d           to detach
tmux attach -t migration  # reattach, from anywhere, later

The difference is that nohup gives you a process you cannot look at again, and tmux gives you a terminal you can come back to, including from a different machine after your laptop has been closed all night. For anything you might need to interact with, that is worth the small learning cost.

cron

crontab -u jordan - installs a crontab for jordan, read from standard input. The command then lists it back and looks at where it went.

# Debian 13 (trixie), x86_64
$ printf '# m h dom mon dow  command\n30 2 * * *  /usr/local/bin/backup.sh\n*/15 * * * *  /usr/local/bin/check.sh\n0 3 * * 0  /usr/local/bin/weekly.sh\n' | crontab -u jordan -; echo '--- jordan crontab ---'; crontab -u jordan -l; echo '--- where it was written ---'; ls -l /var/spool/cron/crontabs/
--- jordan crontab ---
# m h dom mon dow  command
30 2 * * *  /usr/local/bin/backup.sh
*/15 * * * *  /usr/local/bin/check.sh
0 3 * * 0  /usr/local/bin/weekly.sh
--- where it was written ---
total 4
-rw-------. 1 jordan crontab 313 Aug  8 03:23 jordan

/var/spool/cron/crontabs/, mode 600, owned by the user and the crontab group. Spool rather than home, because a home directory can be on NFS, be unmounted, or be deleted at offboarding, and the scheduler still has to be able to read it. Mode 600 because those lines run as that user, so being able to write the file is being able to run commands as them.

Never edit that file directly. crontab -e writes to a temporary copy, validates it, and installs it atomically, the same shape as visudo. Editing the spool file in place can leave cron reading a half-written crontab, and on some implementations the daemon only notices a change through crontab’s own signal.

Five time fields, then the command. In order:

So that first line is 02:30 every day. Reading right to left helps: any day of week, any month, any day of month, hour 2, minute 30.

The three captured lines read as: 02:30 daily, every fifteen minutes, and 03:00 every Sunday.

Note the file is mode 600 and owned by jordan. Never edit /var/spool/cron/ directly: crontab -e validates the syntax before installing, and a malformed file edited by hand can stop every job in it from running.

crontab -r sits next to -e on the keyboard and removes the whole crontab without asking. crontab -l > ~/crontab.bak before editing is a two-second habit worth having.

System crontabs, which have a sixth field

/etc/crontab and files in /etc/cron.d/ add a user field between the time and the command:

30 2 * * *  root  /usr/local/bin/backup.sh

That extra field is the single most common cron mistake. Put a user crontab line in /etc/cron.d/ and cron tries to run root as the command; put a system line in a user crontab and it tries to run /usr/local/bin/backup.sh as an argument to a command called root. Neither works and the error goes to mail nobody reads.

/etc/cron.daily/, /etc/cron.hourly/, and /etc/cron.weekly/ are simpler still: drop an executable script in and it runs on that schedule, no time fields at all. The script must be executable and must have no file extension on many systems: run-parts skips anything with a dot in the name, which is why backup.sh in /etc/cron.daily/ silently never runs.

*/15 * * * * /usr/bin/flock -n /var/lock/backup.lock /usr/local/bin/backup.sh

The environment problem

A cron job runs with almost nothing: a minimal PATH, no shell startup files, and frequently a different HOME. This is lesson 21 arriving with a schedule attached, and it is the reason most cron jobs fail the first time.

The three rules that avoid nearly all of it:

Use absolute paths for everything, the command, and every file it touches. /usr/local/bin/backup.sh rather than backup.sh, and /srv/data rather than data.

Set PATH at the top of the crontab. Cron accepts variable assignments as lines of their own, and they apply to every job in the file:

PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
MAILTO=ops@example.com
30 2 * * *  /usr/local/bin/backup.sh

Capture the output, because otherwise cron mails it to the local user and nobody reads local mail:

30 2 * * *  /usr/local/bin/backup.sh >> /var/log/backup.log 2>&1

MAILTO="" suppresses mail entirely, which is right only once you are logging somewhere you actually look.

And the percent sign is a trap. In a crontab, % means a newline and everything after the first one becomes standard input to the command. date +%Y-%m-%d in a cron line does not do what it does at a prompt. It has to be \%. This catches everybody once, usually on a log filename.

echo '/usr/local/bin/one-off.sh' | at 02:00 tomorrow
atq          # what is queued
atrm 3       # cancel job 3

# /etc/systemd/system/backup.timer
[Unit]
Description=Nightly backup
[Timer]
OnCalendar=*-*-* 02:30:00
RandomizedDelaySec=300
Persistent=true
[Install]
WantedBy=timers.target

sudo systemctl enable --now backup.timer
systemctl list-timers

Across distributions

cronie versus cron matters when a machine has neither. A minimal server image frequently ships without any cron daemon at all, and a crontab installed on it is stored correctly and never runs. systemctl status crond on RHEL, systemctl status cron on Debian, note the daemon names differ too.

Prove it

After adding a scheduled job:

# Is it installed and does it read the way you meant
crontab -l

# Is the daemon actually running
systemctl status crond    # RHEL family
systemctl status cron     # Debian family

# Did it run, and what happened
sudo journalctl -u crond --since today | grep backup
sudo grep CRON /var/log/syslog | tail

# For a timer, the definitive answer
systemctl list-timers
systemd-analyze calendar '*-*-* 02:30:00'

And the test that removes all doubt: schedule it for two minutes from now. Change the time, watch it fire, read the log, then set the real schedule. Waiting until 02:30 to find out that PATH was wrong is a day lost per attempt.

What trips people up

1. The job died with the session

& frees the prompt and does not detach the process. SIGHUP at logout kills it.

nohup ... & for fire-and-forget, tmux for anything you might want to look at again.

2. The cron job runs nothing

PATH is minimal and no startup files are read. command not found goes to mail nobody reads.

Absolute paths, a PATH= line at the top, and redirect output to a log file.

3. The fields are in the wrong order

Minute, hour, day-of-month, month, day-of-week. * 2 * * * is every minute of the 2am hour, not 2am.

Read them aloud, and check with crontab -l afterwards.

4. The sixth field

/etc/crontab and /etc/cron.d/ take a user field that user crontabs do not. Getting it wrong means the job silently never runs.

5. A dot in a /etc/cron.daily/ filename

run-parts skips files with extensions. backup.sh there never runs; backup does. It must also be executable.

Work it through

A nightly backup has not run for three weeks. crontab -l shows the entry and looks right. The script works when run by hand.

Reason through the order before reading on.

First: is the daemon running at all?

systemctl status crond    # or cron

A minimal image frequently ships without one. A crontab on a machine with no cron daemon is stored perfectly and never executes, and crontab -l looks exactly the same either way. This is the cheapest check and it is not the first thing most people try.

Second: did cron attempt it?

sudo journalctl -u crond --since '3 weeks ago' | grep backup
sudo grep CRON /var/log/syslog | grep backup | tail

That splits the problem cleanly. No log entries means cron never tried, a schedule problem, a daemon problem, or an access-control problem. Entries present means it tried and the script failed, which is a completely different investigation.

If cron tried and it failed, the answer is nearly always the environment:

env -i /bin/sh -c '/usr/local/bin/backup.sh'

Straight from lesson 21. This reproduces the minimal-environment case in one command rather than waiting for tomorrow night, and a command not found here names the problem immediately.

If cron never tried, check the schedule against what you meant, and check /etc/cron.allow, if that file exists and the user is not in it, they may install a crontab and cron will never run it.

And this is what would have caught it in week one: where was the output going? If the line has no redirection, cron mailed it to the local user, and /var/spool/mail/ on a server nobody reads is where three weeks of nightly error messages are sitting. mail or cat /var/spool/mail/root may hand you the answer directly.

Now the point worth extracting. A scheduled job that fails is silent by construction. There is nobody logged in to see it, and the default destination for its output is a mailbox from another era. Every one of these investigations is really about finding where the evidence went.

The habit, and it is one line: redirect to a log file, and make the success message carry a number.

30 2 * * *  /usr/local/bin/backup.sh >> /var/log/backup.log 2>&1

That, plus a monitoring check on the log’s modification time, turns “it has not run for three weeks” from something a person eventually notices into something that alerts on the second night. Which is the same conclusion as the backup lesson, arriving from the scheduling side: a check that can only report success is not a check.

Try it

Optional, on any machine.

1. sleep 300, then Ctrl+Z, then jobs, then bg, then jobs again.
2. fg to bring it back, then Ctrl+C.
3. nohup sleep 300 &, then exit the shell, reconnect, and pgrep -a sleep.
4. crontab -l > ~/cron.bak, the habit, then crontab -e and add * * * * * date >> /tmp/cron-test.log.
5. Wait two minutes, then cat /tmp/cron-test.log. Then remove the line.
6. systemctl list-timers and read what is already scheduled.
7. systemd-analyze calendar 'Mon *-*-* 09:00:00' and check it means what you think.

Verification step. You have it when you can write a crontab line for “03:15 on the first Sunday of the month”, explain why cron cannot express that directly, and say what you would do instead.

Check yourself

30 2 * * *  root  /usr/local/bin/backup.sh

References

• crontab(5) - Linux man-pages project. Accessed 2026-08-07.
• crontab(1) - Linux man-pages project. Accessed 2026-08-07.
• at(1) - Debian Project. Accessed 2026-08-07.
• systemd.timer(5) - Linux man-pages project. Accessed 2026-08-07.
• nohup(1) - Linux man-pages project. Accessed 2026-08-07.
• anacron(8) - Linux man-pages project. Accessed 2026-08-07.

Every block above with a distribution and architecture header was captured by running the command on a Debian 13 (trixie) container. Blocks without one are illustrative.