Skip to content

docs: Propose one writer per zone for DNS records - #201

Merged
vvoytovych merged 5 commits into
mainfrom
docs/per-zone-writer-enhancement
Oct 8, 2026
Merged

vvoytovych merged 5 commits into
mainfrom
docs/per-zone-writer-enhancement

Conversation

@vvoytovych

@vvoytovych vvoytovych commented Sep 29, 2026 •

Copy link
Copy Markdown
Contributor

What I noticed

Two record sets can claim the same name in a zone. The usual pair is a user's own record and the one Gateway DNS creates for the same hostname. Since v0.7.0 the agent writes one record set at a time, so it never sees both claims and cannot choose between them:

  • Both write, and both report Programmed=True. #83 found such pairs in production. In one, the two point at different targets, and the name answers with whichever wrote last.
  • When one lets the name go, the agent deletes the name, although the other still claims it. The name goes silent, and no alert fires (#188).
  • The losing record set no longer reports NotOwner, so DNSRecordRejected cannot fire (runbook).

The admission webhook refuses a second claim, but it fails open by design and cannot see older pairs. So the agent still has to decide.

What we plan

The agent writes one zone at a time. It reads every record set of the zone, picks one holder per name by the rule our record ownership document already states (the oldest claim wins, as in Gateway API), and writes only the difference.

We get there in small steps. Each step is its own release. Each is verified locally (unit tests, integration tests against PowerDNS, and the end-to-end suites on three clusters) and then on staging, before it reaches production. The release before it is the rollback.

Step Does it change what DNS serves?
The owner-name rule in its own package (#198) no
A separate fix, first because it is a live defect: stop lowering zones' SOA serials (#101) the serial only
Remove the duplicate pairs in production by hand, with their owners those names only
One election function for the webhook and the agent no
The per-zone writer, load-tested at production scale first only where a name is claimed twice, and leftovers
Remove what it makes unused, with the old reconciler (#86) no

What it gives us

  • A released name keeps answering. It goes only when nobody claims it.
  • A record set that loses a name says so again, and DNSRecordRejected works again.
  • One place decides who holds a name. No locks, no special release path, and a simpler backend interface (#88).
  • Leftover records of past failures are removed, within a deletion limit.
  • About 1,350 lines of unused code and its tests go.

How it fits where we are going

  • It is a step toward your point on #171: the control plane programs the data plane, zone by zone, from what is declared. The ownership notes stay for now, because they are the only mark of what the agent may delete. They can go once a zone is fully declared.
  • Each zone keeps one writer for its records. That fits the federation and zone transfer plans.
  • The agent stays additive outside the records it owns, as the datum.net DNS as code plan requires.
  • It is load-tested against the 60-second target of the convergence SLOs.
  • It follows common DNS practice: DNSControl and octoDNS build a zone from what is declared and apply the difference, and we take octoDNS's deletion limit.

Not in this plan: deleting records we did not write, and replacing the ownership notes.

Take a look, and let me know if you have any major concerns. Thanks.

Two record sets can claim one owner name. The agent writes one record
set at a time, so it cannot choose between them, and when either one
lets the name go, the agent deletes the name the other still claims
(#188).

The enhancement proposes one reconcile per zone, which builds the zone
from every record set that references it and writes the difference. It
is rolled out in small releases, each verified locally and on staging.

Related to #188
@vvoytovych
vvoytovych marked this pull request as ready for review September 29, 2026 13:05
@vvoytovych
vvoytovych requested a review from a team as a code owner September 29, 2026 13:05
A local load test of the prototype met the 60-second target at
production size. It also showed that a pass which writes one record
set's status after another waits on the API server once per record
set, so a pass now sends its writes in parallel, at most 16 at once.
@0xmc
0xmc self-requested a review September 29, 2026 14:53
The zone controller writes a zone's SOA and NS records only when it
creates the zone. After that they are record sets like any other, so
the per-zone pass writes them. The serial the SOA carries stays a
separate change (#101).
ecv
ecv previously approved these changes Sep 29, 2026
@ecv

ecv commented Sep 29, 2026

Copy link
Copy Markdown
Contributor

oh!

The replicator keeps one zone per domain with its accounting ConfigMap,
and nothing else writes that ConfigMap. A zone made another way has
none, so a project's zone for the same domain is not refused, and
EnsureZone treats the PowerDNS zone that already exists as its own. A
pass therefore deletes only records that a record set of its own project
wrote, and the dry run before rollout counts any domain that more than
one project holds.
PowerDNS stores some values in another form, so a text comparison would
rewrite those names on every pass and raise the zone's serial each time.
@vvoytovych
vvoytovych force-pushed the docs/per-zone-writer-enhancement branch from 1a4712b to f6c1c5a Compare October 5, 2026 18:25
@vvoytovych
vvoytovych merged commit 680891a into main Oct 8, 2026
12 checks passed
@vvoytovych
vvoytovych deleted the docs/per-zone-writer-enhancement branch October 8, 2026 17:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants