AdGuard 0.107.78 (CT 102): DNS rewrite engine completely non-functional #20

New Issue

2026-07-21T16:19:15Z

fkrebs commented

2026-07-21 16:19:15 +00:00

Problem

AdGuard Home v0.107.78 on CT 102 has all 34 DNS rewrites loaded in memory (verified via /control/rewrite/list API) but applies none of them at query time. Every rewrite query is forwarded to upstream with reason: NotFilteredNotFound.

Evidence (all via dig +short @127.0.0.1 from CT 102 itself):

id.nuclide.systems      → 84.176.44.168   (should rewrite to 192.168.1.8)
ma.nuclide.systems      → 84.176.44.168   (should rewrite to 192.168.1.8)
nc.nuclide.systems      → 84.176.44.168   (should rewrite to 192.168.1.8)
random.nuclide.systems  → 84.176.44.168   (wildcard fallthrough proof)
id.nuclide.lan          → NXDOMAIN        (should be 192.168.1.8)
ops.nuclide.lan         → NXDOMAIN        (should be 192.168.1.8)

Query log confirms rewrites are not being consulted:

test-rewrite.nuclide.lan  reason=NotFilteredNotFound  upstream=86.54.11.200:53  answer=None

What has been tried

systemctl restart AdGuardHome — no fix
POST /control/cache_clear — no fix
POST /control/rewrite/delete + POST /control/rewrite/add (wildcard) — no fix
POST /control/protection toggle off + on — no fix
Fresh rewrite added via API (test-rewrite.nuclide.lan → 192.168.1.99) — also does not match
AdGuardHome --check-config — reports ok
YAML validates cleanly; filtering.rewrites block has 34 entries all enabled: true
Filter engine works (blocks stats.grafana.org correctly)

Likely trigger

CT 102 was OOM-killed on 2026-07-16 02:50:42 (peak 993 MiB in a 1024 MiB CT). Suspected: rewrite-engine internal state corrupted by the crash, persisted somehow, not fully rebuilt on subsequent restarts.

Downstream impact

Every LAN client resolving *.nuclide.systems gets the public IP and reaches Traefik via NAT hairpin through the D-Link router
Hairpin flakiness causes intermittent SERVFAIL / cURL error 6: Could not resolve host
Observed today: Nextcloud OIDC login to Pocket-ID fails intermittently (4 errors in 2 h). PHP-Guzzle inside nextcloud-aio-nextcloud container gets DNS SERVFAIL for id.nuclide.systems when hairpin blips.

Immediate workaround

Add /etc/hosts entry on affected containers (e.g. Nextcloud) mapping id.nuclide.systems → 192.168.1.8 to bypass AdGuard entirely. Not scalable — every OIDC-consuming container would need this.

Suggested fix

Raise CT 102 RAM 1024 → 2048 MiB (pct set 102 --memory 2048) to prevent OOM. Root cause of the state corruption.
Full reset: stop AdGuard, back up + wipe /opt/AdGuardHome/data/, restart. Loses querylog + stats but should fully rebuild rewrite state.
If (2) fixes it, file upstream bug at github.com/AdguardTeam/AdGuardHome with the reproduction.

Priority

Medium — no complete outage, but a persistent latent bug that affects any OIDC flow via LAN clients and will keep biting.

## Problem AdGuard Home v0.107.78 on CT 102 has **all 34 DNS rewrites loaded in memory** (verified via `/control/rewrite/list` API) but **applies none of them at query time**. Every rewrite query is forwarded to upstream with `reason: NotFilteredNotFound`. Evidence (all via `dig +short @127.0.0.1` from CT 102 itself): ``` id.nuclide.systems → 84.176.44.168 (should rewrite to 192.168.1.8) ma.nuclide.systems → 84.176.44.168 (should rewrite to 192.168.1.8) nc.nuclide.systems → 84.176.44.168 (should rewrite to 192.168.1.8) random.nuclide.systems → 84.176.44.168 (wildcard fallthrough proof) id.nuclide.lan → NXDOMAIN (should be 192.168.1.8) ops.nuclide.lan → NXDOMAIN (should be 192.168.1.8) ``` Query log confirms rewrites are not being consulted: ``` test-rewrite.nuclide.lan reason=NotFilteredNotFound upstream=86.54.11.200:53 answer=None ``` ## What has been tried - `systemctl restart AdGuardHome` — no fix - `POST /control/cache_clear` — no fix - `POST /control/rewrite/delete` + `POST /control/rewrite/add` (wildcard) — no fix - `POST /control/protection` toggle off + on — no fix - Fresh rewrite added via API (`test-rewrite.nuclide.lan → 192.168.1.99`) — also does not match - `AdGuardHome --check-config` — reports ok - YAML validates cleanly; `filtering.rewrites` block has 34 entries all `enabled: true` - Filter engine works (blocks `stats.grafana.org` correctly) ## Likely trigger CT 102 was **OOM-killed on 2026-07-16 02:50:42** (peak 993 MiB in a 1024 MiB CT). Suspected: rewrite-engine internal state corrupted by the crash, persisted somehow, not fully rebuilt on subsequent restarts. ## Downstream impact - Every LAN client resolving `*.nuclide.systems` gets the **public IP** and reaches Traefik via **NAT hairpin** through the D-Link router - Hairpin flakiness causes intermittent `SERVFAIL` / `cURL error 6: Could not resolve host` - Observed today: Nextcloud OIDC login to Pocket-ID fails intermittently (4 errors in 2 h). PHP-Guzzle inside `nextcloud-aio-nextcloud` container gets DNS SERVFAIL for `id.nuclide.systems` when hairpin blips. ## Immediate workaround Add `/etc/hosts` entry on affected containers (e.g. Nextcloud) mapping `id.nuclide.systems → 192.168.1.8` to bypass AdGuard entirely. Not scalable — every OIDC-consuming container would need this. ## Suggested fix 1. **Raise CT 102 RAM 1024 → 2048 MiB** (`pct set 102 --memory 2048`) to prevent OOM. Root cause of the state corruption. 2. **Full reset**: stop AdGuard, back up + wipe `/opt/AdGuardHome/data/`, restart. Loses querylog + stats but should fully rebuild rewrite state. 3. If (2) fixes it, file upstream bug at github.com/AdguardTeam/AdGuardHome with the reproduction. ## Priority Medium — no complete outage, but a persistent latent bug that affects any OIDC flow via LAN clients and will keep biting.

fkrebs commented

2026-07-23 05:21:14 +00:00

Resolved 2026-07-23

Root cause was not OOM state corruption — it was a much simpler misconfiguration:

filtering:
  filtering_enabled: true
  rewrites_enabled: false     # ← master switch off
  parental_enabled: false
  safebrowsing_enabled: false
  protection_enabled: true

The rewrites_enabled: false master switch was disabling the entire DNS rewrite subsystem while individual rewrites remained loaded and queryable via API — which made the bug look like a rewrite-engine failure rather than a global toggle.

Fix applied

pct set 102 --memory 2048 (kept — CT was OOM-killed 2026-07-16 with 1024 MiB, this prevents recurrence)
sed -i s/rewrites_enabled: false/rewrites_enabled: true/ in /opt/AdGuardHome/AdGuardHome.yaml
systemctl restart AdGuardHome

Verified

id.nuclide.systems      → 192.168.1.8   ✓ (Traefik LAN)
ma.nuclide.systems      → 192.168.1.8   ✓
nc.nuclide.systems      → 192.168.1.8   ✓
id.nuclide.lan          → 192.168.1.8   ✓
ops.nuclide.lan         → 192.168.1.8   ✓
pve.nuclide.lan         → 192.168.1.20  ✓
random.nuclide.systems  → 192.168.1.8   ✓ (wildcard catches all)

Nextcloud → Pocket-ID discovery now hits the LAN IP directly — no more NAT hairpin, no more intermittent SERVFAIL.

A /opt/AdGuardHome/data.bak.20260723-071835 backup of the pre-wipe state is on CT 102 in case it turns out to be needed; safe to delete after a week.

## Resolved 2026-07-23 Root cause was **not** OOM state corruption — it was a much simpler misconfiguration: ```yaml filtering: filtering_enabled: true rewrites_enabled: false # ← master switch off parental_enabled: false safebrowsing_enabled: false protection_enabled: true ``` The `rewrites_enabled: false` master switch was disabling the entire DNS rewrite subsystem while individual rewrites remained loaded and queryable via API — which made the bug look like a rewrite-engine failure rather than a global toggle. ### Fix applied 1. `pct set 102 --memory 2048` (kept — CT was OOM-killed 2026-07-16 with 1024 MiB, this prevents recurrence) 2. `sed -i s/rewrites_enabled: false/rewrites_enabled: true/` in `/opt/AdGuardHome/AdGuardHome.yaml` 3. `systemctl restart AdGuardHome` ### Verified ``` id.nuclide.systems → 192.168.1.8 ✓ (Traefik LAN) ma.nuclide.systems → 192.168.1.8 ✓ nc.nuclide.systems → 192.168.1.8 ✓ id.nuclide.lan → 192.168.1.8 ✓ ops.nuclide.lan → 192.168.1.8 ✓ pve.nuclide.lan → 192.168.1.20 ✓ random.nuclide.systems → 192.168.1.8 ✓ (wildcard catches all) ``` Nextcloud → Pocket-ID discovery now hits the LAN IP directly — no more NAT hairpin, no more intermittent SERVFAIL. A `/opt/AdGuardHome/data.bak.20260723-071835` backup of the pre-wipe state is on CT 102 in case it turns out to be needed; safe to delete after a week.

fkrebs closed this issue

2026-07-23 05:21:14 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: fkrebs/docs#20