feat!: schema-wide invariants — datetime, padded ids, and Event-relation provenance (redesign 7/10)

[prashantasdeveloper]

Phase 7 of the indexer redesign. Started as two small invariants; the owner folded in the
entity-provenance rework (D13), making this the largest schema change in the programme. Base:
redesign/06-identity-keys. Review commit by commit — each is scoped to one concern and its
message says which.

10 commits (db79816, from oldest):

commit
3364e17	docs: datetime fields are UTC without a zone marker (A16 / D8 → documentation only)
9f27337	feat!: zero-pad chain-assigned numeric ids (A14 / D12)
20da7b1	docs: fold the entity-provenance rework into Phase 7 (plan 13, D13, A17/A18/A19)
15fd13c	feat: add Extrinsic.events and a synthetic seed Event (D13 / A17 genesis half)
a78d36a	fix: coerce venue_id to a string before padding it (7.2 regression)
91ccd2b	docs: revise the D13 plan after review (nullable-first dropped, no timestamptz, PolyxEntry.blockId)
2867ab3	feat!: record provenance as an Event relation, not a Block+index copy (D13, the atomic swap)
addc94d	feat!: remove the remaining Block-and-index copies (D13)
15f3511	feat: index tuning for the provenance rework (A18, compat.sql)
db79816	docs: mark the D13 provenance rework implemented

What changes

Datetime serialization (A16 / D8) — docs only

SubQuery types a GraphQL Date as Postgres timestamp without time zone; PostGraphile serialises
it with no zone marker, so a consumer parsing "2021-11-05T13:56:36" reads local time. D8
originally called for converting every column to timestamptz; that is an unconditional breaking
change for exact-string-equality consumers plus a generated artifact, while the stored instant is
already correct. So instead: a parse-as-UTC docstring on Block.datetime (covering every Date
field) and the entitlement-critical fields (tradeDate, valueDate, the expiry fields,
filedAt, Sto.start/end). Non-breaking.

Padded numeric ids (A14 / D12) — breaking

Instruction.id is the chain's settlement sequence stored as a String, so orderBy: [ID_DESC]
put "9999" above "14712". A shared padNumericId helper is applied at construction and every
lookup for Instruction, Venue, Proposal, Authorization ids and the FK columns that
reference them. Composite ids (Sto, Distribution, MultiSigProposal) are out of scope — their
leading segment is already a padded/fixed-width key.

Breaking: those four connections now have 10-digit zero-padded string ids. ID_* ordering is
correct; a consumer filtering by a literal unpadded id (id: { equalTo: "42" }) breaks.

Entity provenance (D13) — breaking, the bulk of the diff

A domain entity records provenance as a relation to the Event that caused it, and carries no
field derivable from that relation.

• createdEvent / updatedEvent replace createdBlock / updatedBlock on ~60 entities.
  Read the block as createdEvent { block }. createdBlock / updatedBlock survive on
  EvmAccountMapping alone (no event on any path).
• Standalone datetime comes off 14 entities (it was always createdEvent.block.datetime).
  Read it as createdEvent { block { datetime } }.
• Identity.eventId is dropped (it was constant — every write was DidCreated).
  Account.eventId is kept.
• EvmTransaction becomes extrinsic-only: block / datetime / createdBlock /
  updatedBlock removed; reach them through extrinsic.
• Nft.mintedBlock → createdEvent, Nft.burnedBlock → burnedEvent (nullable). The "in
  circulation" filter is now burnedEventId: { isNull: true }.
• Extrinsic.events added — the reverse of Event.extrinsic.
• Genesis- and storage-seeded rows point at one synthetic seed Event
  (seeding / Seeded, id 0000000000/0000000000), written by genesisHandler. This fixes
  A17: genesisHandler has always written createdEventId: '0000000000/0000000000' for an
  Event row that never existed — historical mode's virtual FKs never caught it.
• data_block_datetime_timestamp (A18) was an expression index no generated query could use.
  Replaced with a plain btree on blocks.datetime. One created_event_id btree added on
  multi_sig_proposals — the one portal-consumed connection whose id can't carry chronological
  order.

updateLegs (A19): the leg rewrite is now inside the if (address) guard and skips the bulk
write when nothing changed, so the scheduled/unsigned execution paths stop rewriting every leg.

Consumer impact (coordinate with the SDK and portal releases)

Removed selectable field	Read it as
createdBlock / updatedBlock on ~60 types	createdEvent { block } / updatedEvent { block }
datetime on 14 types	createdEvent { block { datetime } }
Identity.eventId	— (was constant)
EvmTransaction.block / datetime	extrinsic { block { datetime } }
Nft.burnedBlockId filter	burnedEventId: { isNull: true }

Confirmed compile break: the portal hardcodes orderBy: CREATED_BLOCK_ID_DESC on
portfolioMovements; PortfolioMovement.createdBlock is removed. Migration is ID_DESC —
strictly better, and it also fixes the intra-block pagination bug CREATED_BLOCK_ID_DESC has.
The other two hardcoded portal orderings (assetTransactions / distributionPayments
CREATED_EVENT_ID_DESC) should also move to ID_DESC.

Split out to follow-ups (from 7.5)

• The standalone eventIdx / extrinsicIdx removal (~22 entities) — several are written and
  some queried; each needs a per-entity consumer check. A redundant eventIdx: Int is harmless.
• Threading the real createdEventId through the asset-holder resolution chain —
  getOrCreateAccount / ledgerAccount currently fall back to the block's first event for an
  account discovered lazily (a chain read, or a side effect of an unrelated handler). The instant
  is right; only the precise event is approximate.

Verification

yarn codegen && yarn build && yarn typecheck && yarn lint && yarn test:unit && yarn check-handlers
— all green (496 unit tests). Not resynced. Like Phase 4, D13 needs a genesis resync to
validate the seed-event FK, that no Portfolio.createdEvent points at a system-module event,
and the removal of datetime from live queries. db/compat.sql's btree changes are exercised
only at a resync.

Merge

Base = redesign/06-identity-keys. Linear chain 04→05→06→07 — needs a rebase onto whatever lands
before it. Do not squash — semantic-release reads the individual commits and the three
BREAKING CHANGE: footers.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[F-OBrien]

I'd restore block: Block! @index(unique: false); here. This is similar to how Events and Extrinsics retain blocks. There is not other clear link to a block for a EVM transaction.

[F-OBrien]

The reasoning here rules out converting one column, but it's written as if it also rules out converting all of them — and for the all-columns case the objections listed are cost, not correctness.

The question that would actually settle it isn't mentioned: does SubQuery's schema sync revert the ALTER on every start and upgrade? If it does, the conversion isn't a maintenance cost, it's unworkable, and that's the reason worth recording so this doesn't get reopened later.

Same applies to §10.2 below — leaving the epoch integer "open" means the docstring approach wins by default rather than by decision. Worth recording it as declined, or scoping it to the entitlement-critical fields (tradeDate, valueDate, the expiries).

[F-OBrien]

eventId as unknown as InstructionEventEnum writes an EventIdEnum value into a 26-value enum with no check. The narrow enum's main benefit over EventIdEnum is that constraint, and the double cast discards it — if an event ever reaches here outside the subset, the row gets a value the GraphQL enum doesn't declare.

Please replace both casts (also line 619) with an explicit map from the handled events to InstructionEventEnum, recording an IndexerAnomaly on an unexpected value. That keeps the type guarantee the enum exists to provide, and surfaces drift instead of writing a bad row.

[F-OBrien]

InstructionEvent.event just above has no @index, but cheap filtering without joining events is the whole reason to keep a local enum column — and this entity uses only 2 of its 10 index slots. Please add @index(unique: false).

[F-OBrien]

This is one of eight eventId: EventIdEnum columns left after the rework, and the reasoning for keeping them isn't recorded anywhere except PolyxEntry ("denormalised — aggregation cannot traverse relations").

They aren't redundant — SubQuery only generates filters and aggregates over an entity's own columns, so createdEvent.eventId can't be filtered or grouped on. But that argument is strong for StakingEvent and this entity, where event type is the main query axis, and weak for DistributionPayment and Account — Account is now the only identity entity that kept one, after Identity, Portfolio, ChildIdentity and IdentityKey all dropped theirs in this PR.

Do we have a clear rule that we could stat once and apply it — keep eventId where consumers filter or group by it and give it the PolyxEntry docstring, drop it where it's only a copy? Eight undocumented columns is harder to review than one stated decision.

[F-OBrien]

reconcileBlock early-returns on every invocation, so the whole POLYX reconciliation — including the forced BalanceSet / DustLost checkpoints — never runs.

SubQuery indexes a block by running the Block handler first, before any of that block's event handlers (@subql/node/dist/indexer/indexer.manager.js, indexBlockData):

await this.indexContent(SubstrateHandlerKind.Block)(block, dataSources, getVM);
// ...only afterwards the init / extrinsic / finalize event handlers

So the real sequence is:

1. Block N's handleBlock runs → reconcileBlock(N). pendingBlock is still N-1 from the previous block's events, so blockNumber(block) !== pendingBlock → return.
2. Block N's event handlers run → reconcileAccount sets pendingBlock = N and fills pending.
3. Block N+1's handleBlock runs → reconcileBlock(N+1). pendingBlock is N → return.

pending is then cleared and re-keyed by the next queueing event, so nothing is ever flushed on any path. The new unit test doesn't catch this because it calls reconcileAccount and then reconcileBlock — the opposite of the production order.

On a fix — relaxing the guard alone won't work, because of what api is bound to. At block N's handler the sandbox api serves end-of-N storage while the database still reflects end-of-N-1, so flushing N-1's queue there would compare an end-of-N-1 derived balance against an end-of-N chain read. readOnChain calls api.query.system.account directly and its blockHeight argument is only a cache key, so it can't read a historical height; the sandbox api is an ApiAt with no .at(), and --unsafe isn't enabled.

Three directions, whichever suits:

• enable unsafeApi / api.at(parentHash) and flush the previous block's queue against the parent hash;
• drop the deferred queue and reconcile inline, restricted to events where the derived balance is already final (BalanceSet / DustLost);
• treat reconciliation as offline-only and rely on scripts/reconcile-polyx.ts, which already does this correctly.

Whichever way it goes, filter: { modulo: 1 } in project.ts should come back out with it, and the unit test should drive the handlers in production order so this can't regress silently.

[F-OBrien]

modulo: 1 makes this datasource request every height, which unions with the dictionary result and disables dictionary block-skipping — a full sequential scan of the chain.

It was added to support the POLYX reconcile flush, which never actually runs (see my comment on reconcilePolyx.ts:128). Please revert it alongside whatever fix that gets — the NftHolder buffer flush it also serves was previously fine on a coarse cadence.

[F-OBrien]

This parameter is blockEventId and is written straight into createdEventId / updatedEventId, but all three call sites pass a bare blockId — 3 call sites to fix - lines 209, 339 and 437.

Both are string, so tsc can't catch it. The result is every StatType row carrying a block id in an Event foreign key, which resolves to nothing.

[F-OBrien]

This parameter is blockEventId and goes straight into createdEventId / updatedEventId, but both call sites pass a bare blockId:

• mapPolyxLedger.ts:168 — the ledgerBalance miss path.
• seed/accountBalance.ts:52 — genesis-seeded balances get a bare block id instead of SEED_EVENT_ID.

src/seed/holding.ts was updated for exactly this in this PR; accountBalance.ts wasn't touched at all. Both types are string, so tsc can't catch either.

[F-OBrien]

The rotateIdentityKey call just above passes blockId (line 294), but the parameter is blockEventId and openIdentityKey writes it straight into createdEventId / updatedEventId.

So every SecondaryKeysPermissionsUpdated opens an IdentityKey row whose createdEvent points at a bare block id. handleSecondaryKeysPermissionsUpdated destructures blockId from extractArgs — it should take blockEventId. Both are string, so tsc can't catch it.