Our energy management product runs a demo that depends on three scheduled jobs: a market simulation that answers the requests a user makes, an hourly job that fetches prices and warms caches, and a nightly job that cleans up and checks data quality. Over three weeks in July and August 2026 we found four different ways those jobs had stopped working. Not one of them produced an error anyone saw. The worst had been triggered by cron 8,073 times without running its command once.
This post walks through the four, because each fails at a different layer, and a scheduler is only working if all of them hold.
Layer 1: one failing step kills the rest
On 19 July we noticed that the hourly job had never completed a single run on the demo server: 177 consecutive aborts, and not one "Hourly tasks completed" line. One step saved its results to a table that the model had always pointed at but that no migration had ever created. Every run threw "The table does not exist", the exception escaped the hourly action, and the schedule stopped on that line. Every task wired below it never ran, including a price fetch and a cache warm added earlier that same day. They looked scheduled and never fired.
Creating the table fixed the trigger. The structural fix was in the cron controller: every scheduled step now runs in isolation, and a failing step logs the error and lets the schedule continue. A cron entry is a list of independent jobs, and one broken job must not take the rest down with it.
Layer 2: isolated failures that nobody reads
Isolation has a cost, and eight days later we paid it. A sweep of the production log found three nightly steps that had failed every night since 20 July. The isolation code caught each failure, logged it under its own category and let the night continue, which is exactly why none of them surfaced: nothing read that category.
- A step that was never a command. The nightly schedule called a cleanup action that did not exist; the controller had a usage menu and one other action, but no
run. Expired password-reset tokens had never been purged in production. - A write that always failed. A data-quality model carried a timestamp behaviour that writes both a created and an updated time, but its table only had the created column. Every insert threw, so the sweep ran nightly and discarded all of its findings at the moment it tried to save them.
- A call to a route that did not exist. A reconciliation service fetched each account over HTTP from a route no tier served, with a token read from a column the model did not have, which a
??fallback quietly turned into an empty string. It returned 404 about 5,000 times a week. We changed it to read the database directly, since the API reads the same database and the round trip bought nothing.
We added a test that checks every route the nightly schedule calls resolves to a real action, and that the other two defects cannot return. Each assertion was validated by putting the original defect back and watching the test fail. If you isolate failures, something has to read them; a test that the schedule's routes exist is cheap insurance on top.
Layer 3: the jobs stayed behind on the old server
On 27 July we found that a server move had left the scheduled jobs behind. The application and its DNS had moved to a new server, but all four cron entries were still installed only on the old one. Nothing failed loudly. On the server actually serving traffic, the market simulation stopped advancing, the hourly feed stopped and the nightly demo reset stopped happening. Meanwhile the old server kept running all of them diligently against a database nobody was looking at.
The entries could not simply be copied across. The old server ran the console commands directly with PHP; the new one runs the application in containers and has no PHP on the host at all. Each job became a one-shot run of the application image, as the application's own user, so it cannot leave root-owned files the web server cannot write to. A crontab is host state, and host state does not move with a deploy unless someone moves it.
Layer 4: cron fired 8,073 times and never ran the command
On 7 August the demo still showed a market that never answered. Requests had sat eligible for processing since 3 August against a 30-second delay, the inbound message table was empty, and the message log held only outgoing messages. There was no inbound side, because the market simulation had never run on the new server.
The crontab ran as the deploy user. The application's runtime directories were owned by the container's web user, a different uid. All three cron entries appended their output to a log file inside those runtime directories:
* * * * * <command> >> shared/console/runtime/logs/<job>.log 2>&1The shell sets up a redirect before it runs the command. When it cannot open the target for appending, it reports an error and never executes the command at all. So cron fired the line 8,073 times, the shell aborted 8,073 times, and the application was never called. There was no error anywhere in the application, because the application never ran. The only symptom was a log file that did not exist.
It dated from the server move. On the old server the application ran natively and the path was writable by the cron user; on the new one the application ran in a container as its own uid, and the path was not. The redirect stayed the same.
The fix was to write cron logs to a directory owned by the cron user. We deliberately did not add the cron user to the application's group: that would have handed it write access to every runtime file of the application in order to fix one logging path. The first successful run then processed all eight waiting requests, and the message log went from one message type to four, with both directions present.
What to check on your own schedules
- Does cron run the line? Run the crontab line by hand, as the crontab's user, with its redirect. A missing log file is a symptom, not an absence of news.
- Does every step exist? Test that each scheduled route resolves to a real action.
- Does one failure stop the rest? Isolate steps, then make sure isolated failures are read by someone or something.
- Did the schedule move with the application? After any host move, list the crontab on the new host, and decide deliberately what the old host's entries should still do. We kept ours, as the documented rollback.
- Is the outcome visible? Check the effect a job is supposed to have, such as a row written or a status advanced, not the fact that cron fired.
Conclusion
A scheduled job fails silently by design: it runs when nobody is watching, and cron itself reports nothing about whether the command did its work. Every one of these four failures left the schedule looking correct. The only reliable test is the effect the job is supposed to have.
Our Linux Administration course covers cron, users, groups and permissions on a real server, and our PHP Application Developer course covers building console commands in Yii2.