My Talos cluster runs on three Proxmox hosts: megatron, proxmox and patroxmus2. One control plane VM on each. For most of a year, one alert has followed me around: etcd WAL fsync latency. It fired in December, it fired in April, it fired in June, and it fired again in September. Every time I found something real and fixed it. Every time it came back.

This is the story of being right four times and still wrong.

December. Everything was on NFS

The first symptom was not even about etcd. It was KubeAPIErrorBudgetBurn. The API server was burning its error budget and I did not know why. Then I read the etcd logs and found “slow disk” and “heartbeat overloaded” everywhere. Operations that should finish under 100 ms were taking 400 to 750 ms.

The cause was embarrassing once I saw it. All three control plane VMs had their disks on the Synology, over NFS. Not one of them. All three. I had put etcd, the most fsync-sensitive thing in the cluster, on network storage.

I moved each VM to local storage, one at a time. Cordon, drain, stop the VM, qm move_disk, start, uncordon. Megatron got a thin pool on its 1 TB NVMe. Patroxmus2 already had one on its 512 GB drive. The host called proxmox only had directory storage on a 512 GB SATA SSD, so VM 145 went there.

Warnings gone. Error budget recovered. I also staggered the backup CronJobs across the night so they would stop fighting each other for I/O. Done, I thought.

April. One node kept wedging

By April one control plane, talos-cbw-xgr on the proxmox host, had turned into the problem child. It wedged at a low level and stopped responding. I had an automation that was supposed to power cycle it. It did not work, because it sent qm reset and never checked the result, and a reset does nothing to a guest that is stuck that deep.

I pulled the numbers. Over 30 days that node had 8 Ready-state transitions. The other two control planes had zero. The worker VM on the same host was the worst of all the workers. So it was the host, not Kubernetes.

Then I compared the three VMs’ disk configs. VMs 143 and 144 were raw disks on LVM-thin. VM 145 was a qcow2 file on directory storage. qcow2 keeps metadata tables that force extra fsyncs at the host level, and etcd is nothing but fsyncs.

qm disk move 145 scsi0 local --format raw --delete 1

Three minutes, 25 GB copied, in place. No migration to another host. fsync p99 went from wedge territory to 16.6 ms.

I also did the thing I should have done in December. Talos runs etcd as a system service, so kube-prometheus-stack does not scrape it by default. I added a Service with static Endpoints, a ServiceMonitor, a MachineConfig patch to expose listen-metrics-urls, and six alerts. One of them was the p99 fsync warning that, looking back at Loki, would have fired about 10 hours before the wedge.

And I fixed the power cycle script to do stop, wait, start, and verify.

June. The disk is the floor

Two months later the same node was spiking again. One spike reached 3,315 ms. Its sibling on NVMe never went above 5 ms in the same three days.

I checked the host tuning I had added: swappiness, zram, the bfq scheduler for sda, fixed memory with no ballooning for VM 145. All of it was still in place. It had even survived a host reboot. Baseline p99 was 14.8 ms. The software side was as good as it was going to get.

SMART said the drive was healthy. Zero reallocated sectors, zero CRC errors. It just is what it is: a generic, DRAM-less “512GB SSD” in the M.2 SATA slot of a Beelink U59. My first instinct was enterprise SSDs with power loss protection. Then I looked at the healthy control plane. It runs on a normal consumer NVMe. The gap was not “lacks PLP”. The gap was bottom-tier SATA versus anything with a DRAM cache.

I wrote the runbook. There is an empty 2.5-inch bay in that box. A drive with DRAM goes in, a thin pool goes on it, qm disk move 145 scsi0 fastssd. And then I did not do it, because nothing had actually broken. The node stayed Ready. etcd recovered on its own every time.

That same investigation found a 9.5 hour power outage on the host that I had not noticed. The journal cut off mid-line. The box does not power back on after AC loss. Different story.

September. A different node, the same drive

In September the alert moved. EtcdDiskWalFsyncCritical on talos-tji-a3a, the control plane on patroxmus2. Not the U59 this time.

I assumed swap. I had set swappiness to 10 on that host in an earlier PR. I went back to it. It was harmless and it fixed nothing.

So I correlated the host’s own disk metrics against every spike over 7 days:

rate(node_disk_write_time_seconds_total{instance="patroxmus2",device="nvme0n1"}[1m])
/ rate(node_disk_writes_completed_total{instance="patroxmus2",device="nvme0n1"}[1m])

Average host write latency sits around 1.5 ms. At every single etcd spike it jumped to 27 to 83 ms and stayed there for about 3 minutes. Host writes at those moments were about 0.3 MB/s. Zero swap-out. No backup running. Nothing was asking the disk to do anything. The drive stalls by itself.

It is another no-name “512GB SSD”. Firmware SN10660. 38 percent worn. It hosts VM 144, the control plane, and VM 114, a worker.

That node never went NotReady. It is not the etcd leader. So in the strict sense the alert was noise. In the useful sense it was telling me the truth the whole time: the storage under etcd is not good enough.

What I actually learned

The alert was correct in December, April, June and September. What changed was how deep I was willing to look. NFS was real. qcow2 was real. The host tuning was real. All of them were layers above the one that was actually failing.

I also learned to anchor on what works instead of what sounds correct. The control plane on megatron has been boring since December. It is on a consumer NVMe with a DRAM cache. That is the spec. Not enterprise, not PLP. Just not the cheapest drive in the drawer.

Both no-name SSDs are still in those boxes. The U59 has a free bay. Patroxmus2 needs its NVMe replaced with one that has DRAM, which means cordoning one control plane at a time and cloning or reinstalling from the vzdump backups. That swap is still pending.