Linux Unattended Upgrades: Morning Blips & Lost Cluster Quorums
Automatic package updates produce two classic operational mysteries in production:
- On a single host: a service “blinks” for 10–30 seconds around
06:xxin the morning. Logs show a clean stop and immediate start with no human operator logged in, sending engineers on a wild goose chase after memory leaks or GC pauses. - On a 3-to-5-node cluster: a properly redundant cluster survives single-node failures all year, then suddenly drops quorum one morning because two nodes applied a transitive library update within the same 30-second window.
Both stem from the exact same mechanism.
Where Automatic Updates Come From
Debian / Ubuntu — unattended-upgrades
On Ubuntu (24.04+), unattended-upgrades is both installed and active by default for the security pocket:
/etc/apt/apt.conf.d/20auto-upgrades— master switch (APT::Periodic::Unattended-Upgrade "1")./etc/apt/apt.conf.d/50unattended-upgrades— allowed origins,Package-Blacklist, andAutomatic-Reboot.apt-daily-upgrade.timer— installs upgrades onOnCalendar=*-*-* 6:00withRandomizedDelaySec=60m.
On Debian, the timer exists, but 20auto-upgrades is absent on minimal/cloud images unless enabled via dpkg-reconfigure -plow unattended-upgrades. Check effective settings with apt-config dump APT::Periodic.
Notice the trap right away: every identically provisioned node fires inside the same one-hour window after 06:00.
Fedora & RedOS — dnf-automatic
RPM distributions use dnf-automatic (/etc/dnf/automatic.conf, apply_updates = yes), triggered by dnf-automatic.timer — also defaulting to 06:00 with a 1h random delay.
What Actually Restarts Your Service (needrestart)
Even if your service package did not change, Ubuntu 22.04+ runs needrestart after apt upgrade. It scans /proc/<pid>/maps across every running process looking for deleted shared libraries replaced by a security patch (libssl3, libc6, libcurl4, libsystemd0, zlib1g) and automatically restarts every dynamically linked daemon holding them (nginx, postgresql, redis, custom C++/Python/.NET services).
Inspect pending restarts manually:
sudo needrestart -b
sudo lsof | grep '(deleted)'
Non-interactive behavior is controlled in /etc/needrestart/needrestart.conf via $nrconf{restart}:
'a'— restart services automatically (Ubuntu Server default);'i'— interactive prompt (skips restarts when no TTY is attached);'l'— list only, never restart automatically.
(Side benefit: statically linked binaries such as Go built with CGO_ENABLED=0 or musl Rust builds map no external .so files, so needrestart leaves them untouched during library upgrades.)
Confirming the Diagnosis in Two Minutes
sudo journalctl -u apt-daily-upgrade.service --since "7 days ago"
sudo journalctl -u dnf-automatic.service --since "7 days ago"
sudo tail -n 200 /var/log/apt/history.log
sudo journalctl -u <your-service>.service --since "7 days ago" | grep -E "Stopping|Started|Stopped"
Fixing Cluster Overlap
Option 1: Stagger OnCalendar Windows per Node
Override the systemd timer with a local drop-in (sudo systemctl edit apt-daily-upgrade.timer):
# Node 1
[Timer]
OnCalendar=
OnCalendar=*-*-* 02:00
RandomizedDelaySec=10m
# Node 2
[Timer]
OnCalendar=
OnCalendar=*-*-* 04:00
RandomizedDelaySec=10m
(The empty OnCalendar= line is mandatory to clear the vendor schedule before setting the new one.)
sudo systemctl daemon-reload
sudo systemctl restart apt-daily-upgrade.timer
Option 2: Disable Automatic Service Restarts
Set $nrconf{restart} = 'l'; in /etc/needrestart/needrestart.conf (and optionally exit 101 in /usr/sbin/policy-rc.d if blocking package postinst restarts) so security packages download on schedule while restarts happen during controlled maintenance windows.
Option 3: Orchestrate via Ansible / Kubernetes
For multi-node stateful or quorum clusters, disable node-local upgrade timers completely:
sudo systemctl disable --now apt-daily-upgrade.timer
sudo systemctl disable --now dnf-automatic.timer
And drive upgrades from an orchestrator aware of cluster health — such as Ansible with serial: 1 and health checks between nodes, or Kubernetes PodDisruptionBudget rolling updates.