Between June and August the lab went down three times. Two of those were blackouts. The third one was a laptop that decided to take a nap at 4 in the morning. None of the three was what I thought it was at first, and that is the part worth writing down.

Some context. My homelab is four Proxmox hosts in a 9-inch mini rack: three mini PCs (a Beelink U59, a Beelink MINI S, and a GT100 whitebox) plus a Lenovo ThinkPad P15v I call megatron, lid closed, acting as a server. On top of that runs Talos Kubernetes with seven nodes, Cilium, Flux, Kyverno, the whole thing. Storage and the UPS live on a Synology.

June 26. The power goes out. Nothing comes back.

The first blackout was a Thursday night. Power came back a few minutes later. None of my three mini PCs did.

My first guess was obvious: the UPS battery is dead. The lab crashed hard and the BIOS did not know what to do.

Wrong on both counts.

I checked last -x on the hosts. It said “shutdown”, not “crash”. The nodes had turned off cleanly. The Synology is the NUT master for the UPS, and all four Proxmox hosts are NUT netclients with upsmon configured to run shutdown -h +0. The UPS said “low battery”, the NAS believed it, and it told everyone to go to bed. Graceful, orderly, exactly as designed.

Except the battery was not low. The UPS is an APC Back-UPS RS 1500M2, battery from March 2025, sitting at 97% charge with about 40 minutes of runtime at 14% load. During a short mains sag it asserted the low-battery flag anyway, and the NUT FSD latch took the whole fleet down with 40 minutes of runway unused. Looking back at the logs, I had five mains flaps of about 5 seconds each in two months. The grid where I live is not stable, and the UPS was being dramatic about it.

So why did nothing come back when power returned? Because “Restore on AC Power Loss” in the BIOS was never set to “Power On”. And since NUT had shut them down cleanly, even “Last State” kept them off. The VMs all had onboot=1, which was correct and useless if the host underneath is dark.

What I did that weekend:

  • A PrometheusRule called UPSSpuriousLowBattery that fires when the low-battery flag is set while upsBatteryChargeValue > 50. That is the exact signature of this failure. It goes to Mattermost, ntfy and Telegram.
  • Changed the Synology UPS policy from “Until low battery” to “Customize time”, 600 seconds. Ride through the flaps, shut down only if it is a real outage.
  • Wake-on-LAN enabled and persisted on all four hosts.
  • BIOS “Restore on AC” set to “Power On” on the three mini PCs, one node at a time, with a proper drain and shutdown each. The ThinkPad has no such option, so it depends on WoL.

I wrote the postmortem and a line in my notes that said the cluster would now self-heal from a full-rack outage.

August 10. The power goes out. Nothing comes back. Again.

Seven in the morning, Monday. Cloudflare starts returning 530 for all twelve public services. Uptime Kuma, which runs on a small OCI cluster outside the lab, logs the exact minute: 07:40.

I got to it around 13:29. The Kubernetes API said network is unreachable. A sweep of the /24 showed only two things alive: the Sophos firewall and the Synology. All four Proxmox hosts were off. Almost six hours later. Same as June, I walked over and pressed the buttons.

So the June fix did nothing. My working theory, still not confirmed: NUT shut the hosts down to S5, but the UPS never actually cut its output, because mains came back before the battery ran out. The BIOS “Restore on AC” only fires if it sees AC go away and come back. From the motherboard’s point of view, power never left. It was told to turn off, and it stayed off.

WoL did not save me either. Magic packets from my Mac on VLAN20 did not cross the Sophos, and the two Beelinks have no WoL receive option in the BIOS at all.

Then it got worse. The hosts came up, the seven nodes went Ready, Cilium started rebuilding, and the cluster still would not recover on its own. Pods were dying and no replacements were being created anywhere. FailedCreate across many namespaces.

The culprit was Kyverno. The admission controller could not pull its image, which left the validate.kyverno.svc-fail webhook with no backend, and since it is configured fail-closed, the API server was rejecting every single pod creation in the cluster. I had a note about this from June, as a thing that could bite during a per-node drain. On a cold boot it is a different animal: it gates the entire recovery. Once Kyverno was running, pods went from 235 to 313 in a few minutes and everything else healed itself.

I also chased a false lead for a while. Image pulls to docker.io and registry.k8s.io were timing out on TLS handshakes, and I was sure the Sophos CDN block was eating them. I curled the same registries from my Mac and from a pod inside the cluster. Fast 200s both times. It was just congestion from everything pulling at once, and it cleared on backoff.

Recovery order that worked, and that I now have as a runbook: Cilium, then CoreDNS, then Kyverno admission, then about 50 workloads self-heal, then force a Flux reconcile for whatever is still stuck. By 14:30, 13 out of 13 services up.

August 12. Megatron falls asleep.

Two days later, 04:37:53. Megatron dies alone. Idle, no load, no power event. It stayed down 4 hours and 11 minutes until I powered it on by hand.

My first instinct was to look for logs. There were none. The journal stopped mid-sentence. SMART on the NVMe was clean, zero media errors.

It took me a while to accept it: the absence of logs was the evidence. The boot NVMe had APST enabled with a 100 ms idle timeout. It went into a deep power state and never woke up. Root lives on that disk, so there was nothing to write a panic to. SMART was clean because the controller was asleep and never saw an I/O error. A healthy SMART report does not clear the disk.

The other tell was what was on the screen when I got to it: the PXE boot menu. PXE is last in the boot order, after every disk. Landing there means the BIOS did not see the NVMe at POST. I should have asked “what was on the screen” in the first five minutes.

APST: Enabled — Idle Time Prior to Transition: 100 ms → Power State 3
PCIe LnkCtl: ASPM L1 Enabled

Fix: nvme_core.default_ps_max_latency_us=0 on the kernel cmdline.

And it was never a megatron problem. patroxmus2 and patroxmus3 both boot from NVMe and had the identical setting. Three of my four hosts were one quiet night away from the same failure. All three are fixed now. The fourth boots from SATA.

What the investigation dragged out into the light

Pulling on that thread exposed things that had nothing to do with power.

81% of the cluster’s CPU requests were fiction. 19,557m requested against a p95 actual of 3,688m. That is what turned a single node dying into a four-hour application outage: nothing could reschedule because the scheduler believed the other nodes were full. Right-sizing freed about 6.3 cores. One node went from 100% CPU requested to 52%.

Four nodeAffinity rules were silent no-ops. They referenced megatron and megatron.local, but Kubernetes nodes are named talos-*, so the NotIn matched all seven nodes and excluded nothing.

The node-restart automation ran every 5 minutes during the whole outage and could not help, because the hypervisor itself was down. It logged a generic “all restart paths failed” and never said what it should have said: a human needs to press a button. It now sends a distinct PHYSICAL INTERVENTION REQUIRED alert.

Adding host temperature telemetry paid for itself in 20 minutes. patroxmus2, an Intel N100, was sitting at 97 °C with 71 throttle events per second at a load average of 1.40 on 4 cores. Not a spike, a permanent state. The root cause was found by eye, not by sensor: the mini PCs were stacked on top of each other, each one breathing the exhaust of the one below. Compressed air, un-stacked them, a USB fan in the rack. 97 °C to 83-89 °C and zero throttling.

And one trap I almost fell into: patroxmus3, an AMD Ryzen 5 3550H, was at 94 °C and perfectly healthy, boosting to 2802 MHz over a 2.1 GHz base. AMD mobile chips are designed to run at their thermal ceiling. The diagnostic is not the temperature. It is whether the chip is still boosting above base clock. A flat alert above 90 °C would have cried wolf on that box forever. The alerts now measure the margin to each sensor’s own reported crit.

Last one. Late that night n8n was returning 503 “Database is not ready” while every health sweep I had run said the lab was fine. The pod was Running. It was also 0/1 Ready, for 78 minutes. Phase answers “is the kubelet running this container”. Readiness answers “can it serve”. Only the second one correlates with a user seeing an error. There is now a PodRunningButNotReady alert at 15 minutes, a threshold I measured by replaying the expression over the maintenance window, not one I guessed.

Still open: confirming on the Synology logs whether the UPS really keeps its output up during short outages, and getting a smart plug or switched PDU for the two Beelinks that cannot do WoL. Until then, the honest statement is that my lab survives a blackout only if I am home.