Skip to content

Linux Unattended Upgrades: Morning Blips & Lost Cluster Quorums

Automatic package updates produce two classic operational mysteries in production:

  1. On a single host: a service “blinks” for 10–30 seconds around 06:xx in the morning. Logs show a clean stop and immediate start with no human operator logged in, sending engineers on a wild goose chase after memory leaks or GC pauses.
  2. On a 3-to-5-node cluster: a properly redundant cluster survives single-node failures all year, then suddenly drops quorum one morning because two nodes applied a transitive library update within the same 30-second window.

Both stem from the exact same mechanism.

Where Automatic Updates Come From

Debian / Ubuntu — unattended-upgrades

On Ubuntu (24.04+), unattended-upgrades is both installed and active by default for the security pocket:

  • /etc/apt/apt.conf.d/20auto-upgrades — master switch (APT::Periodic::Unattended-Upgrade "1").
  • /etc/apt/apt.conf.d/50unattended-upgrades — allowed origins, Package-Blacklist, and Automatic-Reboot.
  • apt-daily-upgrade.timer — installs upgrades on OnCalendar=*-*-* 6:00 with RandomizedDelaySec=60m.

On Debian, the timer exists, but 20auto-upgrades is absent on minimal/cloud images unless enabled via dpkg-reconfigure -plow unattended-upgrades. Check effective settings with apt-config dump APT::Periodic.

Notice the trap right away: every identically provisioned node fires inside the same one-hour window after 06:00.

Fedora & RedOS — dnf-automatic

RPM distributions use dnf-automatic (/etc/dnf/automatic.conf, apply_updates = yes), triggered by dnf-automatic.timer — also defaulting to 06:00 with a 1h random delay.

What Actually Restarts Your Service (needrestart)

Even if your service package did not change, Ubuntu 22.04+ runs needrestart after apt upgrade. It scans /proc/<pid>/maps across every running process looking for deleted shared libraries replaced by a security patch (libssl3, libc6, libcurl4, libsystemd0, zlib1g) and automatically restarts every dynamically linked daemon holding them (nginx, postgresql, redis, custom C++/Python/.NET services).

Inspect pending restarts manually:

sudo needrestart -b
sudo lsof | grep '(deleted)'

Non-interactive behavior is controlled in /etc/needrestart/needrestart.conf via $nrconf{restart}:

  • 'a' — restart services automatically (Ubuntu Server default);
  • 'i' — interactive prompt (skips restarts when no TTY is attached);
  • 'l' — list only, never restart automatically.

(Side benefit: statically linked binaries such as Go built with CGO_ENABLED=0 or musl Rust builds map no external .so files, so needrestart leaves them untouched during library upgrades.)

Confirming the Diagnosis in Two Minutes

sudo journalctl -u apt-daily-upgrade.service --since "7 days ago"
sudo journalctl -u dnf-automatic.service --since "7 days ago"

sudo tail -n 200 /var/log/apt/history.log
sudo journalctl -u <your-service>.service --since "7 days ago" | grep -E "Stopping|Started|Stopped"

Fixing Cluster Overlap

Option 1: Stagger OnCalendar Windows per Node

Override the systemd timer with a local drop-in (sudo systemctl edit apt-daily-upgrade.timer):

# Node 1
[Timer]
OnCalendar=
OnCalendar=*-*-* 02:00
RandomizedDelaySec=10m
# Node 2
[Timer]
OnCalendar=
OnCalendar=*-*-* 04:00
RandomizedDelaySec=10m

(The empty OnCalendar= line is mandatory to clear the vendor schedule before setting the new one.)

sudo systemctl daemon-reload
sudo systemctl restart apt-daily-upgrade.timer

Option 2: Disable Automatic Service Restarts

Set $nrconf{restart} = 'l'; in /etc/needrestart/needrestart.conf (and optionally exit 101 in /usr/sbin/policy-rc.d if blocking package postinst restarts) so security packages download on schedule while restarts happen during controlled maintenance windows.

Option 3: Orchestrate via Ansible / Kubernetes

For multi-node stateful or quorum clusters, disable node-local upgrade timers completely:

sudo systemctl disable --now apt-daily-upgrade.timer
sudo systemctl disable --now dnf-automatic.timer

And drive upgrades from an orchestrator aware of cluster health — such as Ansible with serial: 1 and health checks between nodes, or Kubernetes PodDisruptionBudget rolling updates.