TSYS Group » Chief Operations Officer » Known Element Enterprises - Technology & Facility Services

Symptom¶

Uptime Kuma reports intermittent up/down alerts for many different hosts
across the fleet (not just DNS). Alerts are transient — hosts come back within
seconds to minutes.

Root cause¶

pfv-netinfra-01 is the SOLE Tailscale subnet router advertising
192.168.0.0/22. There is no redundant subnet router. Every LAN host that
Uptime Kuma monitors (via the Cloudron VPS over Tailscale) routes through
this single node.

Three instability factors on netinfra-01:

1. DERP relay bouncing — tailscaled journal (Aug 5) shows the nearest
   DERP relay switching between #13 (Dallas), #9, and #12 every ~15 minutes.
   Each switch causes a brief WireGuard tunnel disruption.

2. Docker LinkChange events — creating/destroying Docker containers
   adds/removes bridge interfaces, triggering LinkChange: major, rebinding
   in tailscaled. This rebinds ALL interfaces and drops active connections.
   Observed during Pi-hole container recreation at 08:53 CDT Aug 6.

3. resolv.conf write failures — every link change produces:
   wgengine: error setting DNS config after major link change: writing to /etc/resolv.conf: operation not permitted

Fix plan¶

1. Add netinfra-02 as a second subnet router for 192.168.0.0/22.
   Tailscale supports ECMP failover between subnet routers — when netinfra-01
   hiccups, traffic fails over to netinfra-02 transparently.

2. Approve the new route in the Tailscale admin console (ACL/routes).

3. Investigate DERP bouncing — may need to pin a preferred DERP or
   investigate NAT traversal stability at the site.

4. Fix resolv.conf permissions on netinfra-01 (tailscaled needs write
   access or a systemd-resolved integration).

References¶

• Discovered during: #376 (DNS alerts investigation)
• netinfra-01 Tailscale IP: 100.70.181.72
• netinfra-02 Tailscale IP: 100.71.171.20
• Subnet: 192.168.0.0/22