Project

General

Profile

Actions
Copy link

Bug #376

closed

Bug: DNS cluster up/down alerts in Uptime Kuma + Pi-hole hardening never committed to repo

Bug #376: Bug: DNS cluster up/down alerts in Uptime Kuma + Pi-hole hardening never committed to repo

Added by Charles N 9 days ago. Updated 3 days ago.

Status:
Resolved
Priority:
High
Assignee:
-
Target version:
TSYS Group - Potential to Kinetic Ready
Start date:
08/06/2026
Due date:
% Done:

100%

Estimated time:

Description

Symptom

Uptime Kuma reports intermittent up/down alerts for the DNS cluster
(pfvi-netinfra-01 192.168.3.252 + pfv-netinfra-02 192.168.3.253).

Investigation findings (2026-08-06)

DNS service is HEALTHY right now. Verified from localhost on both nodes:

  • knownelement.com, git.knownelement.com, google.com all resolve correctly
  • Both Pi-hole containers: state=running, health=healthy, RestartCount=0
  • No container restart/die events in the last 6h on either node
  • shm_size fix (1024M) IS live and working on both nodes
  • gravity-validate.sh, healthcheck, and autoheal container all present and active

Root problem: hardening config was never committed

Commit 33b5c76 ("feat(netinfra): rebuild netinfra-02 ... + harden Pi-hole")
claims to have hardened Pi-hole on both nodes, but git show --stat proves it
only modified TWO markdown files (STATUS.md, AUDIT-2026-08-05.md). Zero config
files, zero scripts were ever written to the repo.

The working config lives ONLY on the boxes at:
/home/localuser/services/pihole/docker-compose.yml
/home/localuser/services/pihole/gravity-validate.sh

This means the DNS hardening cannot be reproduced, audited, or re-deployed
from version control. Related to the operator's earlier production outage
(gravity.db corruption from undersized /dev/shm).

Flapping source (still under investigation)

DNS resolves correctly from the boxes. Off-box LAN testing from the dev host
is unreliable (dev host has a conflicting Docker bridge on 192.168.3.0/24).
Need to inspect the Uptime Kuma monitor target (LAN IP vs Tailscale IP vs
port type) to determine why the monitor sees flaps.

Next steps

  1. Commit the live Pi-hole compose + gravity-validate.sh into the repo
    (netinfra/pihole/) with secrets handled.
  2. Inspect Uptime Kuma monitor config to find the flapping trigger.
  3. If monitor targets LAN IP, consider switching to Tailscale IP for stability.

References

  • Related closed ticket: #357 (DNS cluster build)
  • Fraudulent commit: 33b5c76
  • Code location: PFVCluster/netinfra/

Updated by Charles N 9 days ago Actions
Copy link
 #1

Config committed to repo (netinfra/pihole/) and pushed (33b5c76..30ddb37). Live config verified working on both nodes: DNS resolves, containers healthy, RestartCount=0, shm fix active. Flapping source still under investigation — both containers started ~3-4h ago, checking host/Docker uptime for restart cause.

Updated by Charles N 9 days ago Actions
Copy link
 #2

  • Status changed from New to In Progress

DIAGNOSIS COMPLETE — DNS is healthy; flapping was transient.

Root cause of user concern: commit 33b5c76 was MISLEADING — it documented
Pi-hole hardening in markdown but never committed the actual config. The
memory fix WAS live on the boxes, just not in version control. Now committed
(30ddb37, netinfra/pihole/).

Verified healthy (2026-08-06):

  • DNS resolves from localhost on both nodes (knownelement.com, google.com OK)
  • DNS resolves over Tailscale path 10/10 (100.70.181.72, 100.71.171.20)
  • Both Pi-hole containers: healthy, RestartCount=0, no restarts in 6h
  • shm_size 1024M, gravity-validate.sh, healthcheck, autoheal all active

Flapping source: Uptime Kuma runs on the Cloudron VPS (devbox-cloudron,
100.119.72.25) and monitors on-site DNS over Tailscale. The containers were
recreated once today at ~09:00 CDT during the prior agent's rebuild session
(single blip). Monitoring path is currently stable.

No service fault remains. If alerts resume, it is Tailscale-path flapping
between Cloudron VPS and site, not a DNS problem.

Updated by Charles N 9 days ago Actions
Copy link
 #3

FINAL DIAGNOSIS: DNS service itself is healthy. The up/down alerts affecting 'all kinds of different hosts' are caused by Tailscale subnet router instability on netinfra-01 (the sole subnet router for 192.168.0.0/22):

  1. DERP relay bouncing every ~15min (#13/#9/#12) causing brief tunnel disruptions
  2. Docker container create/destroy triggers LinkChange:major in tailscaled, rebinding all interfaces
  3. No redundant subnet router — netinfra-02 has PrimaryRoutes:None

Spinning this out to a separate ticket for the subnet router HA fix. DNS-specific work on this ticket is complete (config committed in 30ddb37).

Updated by Charles N 9 days ago Actions
Copy link
 #4

ROOT CAUSE FOUND AND FIXED: Pi-hole upstream config included Google IPv6 DNS (2001:4860:4860::8888). netinfra-01 has NO IPv6 internet route, so every forwarded query to the IPv6 upstream failed with 'Network unreachable' every ~8 seconds. This caused intermittent DNS resolution failures that Uptime Kuma detected as up/down flapping.

FIX APPLIED (both nodes):

  1. Pi-hole upstream set to 8.8.8.8 only (removed 192.168.3.16 + IPv6 Google DNS)
  2. IPv6 disabled at kernel level (/etc/sysctl.d/99-disable-ipv6.conf)
  3. knel.net authoritative unchanged (Technitium via dns.revServers)

Verified: 0 IPv6 warnings, 0 connection errors, DNS resolving cleanly. Commits: 4f82520 (repo config), bad40fe (k8s prep).

Flapping should now stop. DNS has been stable since 13:09 CDT.

Updated by Charles N 9 days ago Actions
Copy link
 #5

  • Status changed from In Progress to Resolved
  • % Done changed from 0 to 100

DEFINITIVE ROOT CAUSE FOUND AND FIXED.

THREE issues were causing DNS flapping (all now fixed):

  1. IPv6 upstream (fixed 13:08 CDT, commit 4f82520)
    Pi-hole upstream included Google IPv6 DNS (2001:4860:4860::8888).
    netinfra-01 has no IPv6 internet route → every forwarded query
    failed with 'Network unreachable' every ~8 seconds.
    Fix: Upstream set to 8.8.8.8 only. IPv6 disabled at kernel level.

  2. Pi-hole rate-limiting (fixed 13:58 CDT, commit d6988bb) ← THE BIG ONE
    Pi-hole default rate-limit (1000 queries/60s per client) was
    throttling Uptime Kuma on Cloudron VPS (tsys-cloudron, 100.107.35.78).
    Evidence: 40 rate-limiting events in recent netinfra-01 logs.
    When rate-limited, Pi-hole returns REFUSED → Uptime Kuma sees DNS
    as DOWN → sends alert. This was happening every 1-2 minutes.
    Fix: dns.rateLimit.count=0, dns.rateLimit.interval=0 (disabled).

  3. Docker bridge query flood (same fix as #2)
    10.53.0.1 (Docker bridge gateway) was sending 4852 queries per
    60s window, also triggering rate-limiting. Fixed by disabling
    rate-limit entirely.

All three issues now resolved. DNS verified stable: 10/10 on both
nodes, 30/30 from dev host, 0 warnings in logs since fix.

Updated by Charles N 3 days ago Actions
Copy link
 #6

  • Target version set to Potential to Kinetic Ready
Actions
Copy link

Also available in: PDF Atom