Project

General

Profile

Actions
Copy link

Bug #377

closed

Bug: Tailscale subnet router SPOF — netinfra-01 sole router, DERP bouncing causes fleet-wide up/down alerts

Bug #377: Bug: Tailscale subnet router SPOF — netinfra-01 sole router, DERP bouncing causes fleet-wide up/down alerts

Added by Charles N 9 days ago. Updated 3 days ago.

Status:
Resolved
Priority:
High
Assignee:
-
Target version:
TSYS Group - Potential to Kinetic Ready
Start date:
08/06/2026
Due date:
% Done:

100%

Estimated time:

Description

Symptom

Uptime Kuma reports intermittent up/down alerts for many different hosts
across the fleet (not just DNS). Alerts are transient — hosts come back within
seconds to minutes.

Root cause

pfv-netinfra-01 is the SOLE Tailscale subnet router advertising
192.168.0.0/22. There is no redundant subnet router. Every LAN host that
Uptime Kuma monitors (via the Cloudron VPS over Tailscale) routes through
this single node.

Three instability factors on netinfra-01:

  1. DERP relay bouncing — tailscaled journal (Aug 5) shows the nearest
    DERP relay switching between #13 (Dallas), #9, and #12 every ~15 minutes.
    Each switch causes a brief WireGuard tunnel disruption.

  2. Docker LinkChange events — creating/destroying Docker containers
    adds/removes bridge interfaces, triggering LinkChange: major, rebinding
    in tailscaled. This rebinds ALL interfaces and drops active connections.
    Observed during Pi-hole container recreation at 08:53 CDT Aug 6.

  3. resolv.conf write failures — every link change produces:
    wgengine: error setting DNS config after major link change: writing to /etc/resolv.conf: operation not permitted

Fix plan

  1. Add netinfra-02 as a second subnet router for 192.168.0.0/22.
    Tailscale supports ECMP failover between subnet routers — when netinfra-01
    hiccups, traffic fails over to netinfra-02 transparently.

  2. Approve the new route in the Tailscale admin console (ACL/routes).

  3. Investigate DERP bouncing — may need to pin a preferred DERP or
    investigate NAT traversal stability at the site.

  4. Fix resolv.conf permissions on netinfra-01 (tailscaled needs write
    access or a systemd-resolved integration).

References

  • Discovered during: #376 (DNS alerts investigation)
  • netinfra-01 Tailscale IP: 100.70.181.72
  • netinfra-02 Tailscale IP: 100.71.171.20
  • Subnet: 192.168.0.0/22

Updated by Charles N 9 days ago Actions
Copy link
 #1

ACTIONS TAKEN (2026-08-06):

  1. netinfra-02 now advertises 192.168.0.0/22 as subnet route
    (tailscale set --advertise-routes=192.168.0.0/22)
    → Advertised locally, needs admin console approval to activate.

  2. Fixed resolv.conf immutable flag on netinfra-01 (chattr -i).
    Was preventing tailscaled from writing DNS config after every link change.
    netinfra-02 was clean (no immutable flag).

  3. tailscaled restarted on netinfra-01 to clear error state.

PENDING (requires user action):

Code/docs: docs/server-build/tailscale.md section 6 (commit 75e229c)

Updated by Charles N 9 days ago Actions
Copy link
 #2

UPDATE: netinfra-02 now also advertises exit node (0.0.0.0/0, ::/0),
matching netinfra-01 for full HA parity.

Verified:

  • Subnet route 192.168.0.0/22: APPROVED and active (PrimaryRoutes confirmed)
  • netinfra-02 visible as active; direct from netinfra-01

PENDING (requires user action):

Docs: docs/server-build/tailscale.md §6 (commit d4850a3)

Updated by Charles N 9 days ago Actions
Copy link
 #3

  • % Done changed from 0 to 100

Exit node approved on netinfra-02. Verified both nodes offer exit node + subnet route. Full HA achieved. Closing.

Updated by Charles N 9 days ago Actions
Copy link
 #4

  • Status changed from New to Resolved

Updated by Charles N 3 days ago Actions
Copy link
 #5

  • Target version set to Potential to Kinetic Ready
Actions
Copy link

Also available in: PDF Atom