feat(datasource-pylon): messages / conversation thread (EXT-9)

[christophebrun-forest]

Closes EXT-9 — Story 5 of EXT-4.

Stacked on #354 (ext-8-collections-relations). Retarget to feat/datasource-pylon once that one is merged.

What

The conversation of an issue was invisible: opening a PylonIssue showed the initial body_html and nothing of the replies and internal notes that followed. This ports the Zendesk comments embedder — the thread is read through GET /issues/{id}/messages and embedded as a structured array column messages on PylonIssue.

File	Role
client.rb	fetch_issue_messages + the collect_pages / best_effort helpers
collections/issue/messages_embedder.rb	new module: projection check, fan-out cap, message and author serialization
collections/issue.rb	MESSAGE_THREAD_SCHEMA, MAX_MESSAGE_EMBEDS, wiring into list
collections/issue/schema_definition.rb	the messages column, neither filterable nor sortable

Three decisions worth reviewing

No limit is sent. Pylon then answers with the whole thread in a single request. Messages come back oldest first, so asking for a page would have handed back the oldest ones and cut the recent ones off — the half of a conversation nobody opens a ticket to read. The cursor is still followed defensively, bounded by MAX_COLLECTED_PAGES, in case Pylon starts paginating on its own.

No author lookup at all. Unlike Zendesk, which only gives an author_id, Pylon already nests author: {name, avatar_url, contact:{id,email}, user:{id,email}} in every message. The story's "bulk-resolve message authors" is therefore satisfied without a single extra request: the payload is flattened into author_name / author_email / author_avatar_url / author_contact_id / author_user_id. Both ids are kept, since Pylon puts the contact and user sides side by side with nothing telling them apart.

The fan-out is bounded, and truncation reads as unknown. One thread is one request against an endpoint allowing 20 per minute, so a page projecting the column is capped at MAX_MESSAGE_EMBEDS (10) like MAX_ID_LOOKUPS bounds the primary-key fan-out. Rows past the cap — and rows whose thread failed to be read — are left at nil, never at [], which would read as "this issue has no message". A degraded thread costs the operator a column, not the records they opened.

One deviation from the plan: want_messages? requires the projection to name the column. A nil projection — what a count and an export go through — embeds nothing, exactly like RelationEmbedder, rather than spending one request per row on a path that never asked for the conversation.

Field names follow the columns of the collection (body_html, created_at) rather than the payload (message_html, timestamp), so the operator does not have two conventions to reconcile inside one schema.

Verification

461 examples, 0 failures — coverage 934/934 lines (100%, threshold 90). RuboCop clean over the 49 files of the package.

Specs cover: the column schema shape; one request on a record detail with the thread in order; the message_html/timestamp renames; contact-authored vs agent-authored vs author-less messages; a projection without messages issuing zero requests; messages:body_html reaching inside the column; an empty thread as []; the cap leaving rows at nil with a warning; and a 500 leaving the page served with the column at nil.

🤖 Generated with Claude Code

Note

Add message thread embedding to the Pylon issue collection

• Adds fetch_issue_messages(issue_id) to Client, which walks cursor-paginated Pylon endpoints to collect full message threads, capping at MAX_COLLECTED_PAGES (10) with a warning.
• Adds a read-only messages field to the PylonIssue schema (array of structured message objects, not filterable or sortable) via SchemaDefinition.
• The MessagesEmbedder module populates messages per row when the projection requests it, flattening author contact/user fields into a flat author_* structure.
• Fan-out is capped at MAX_MESSAGE_EMBEDS (10) rows per page; rows past the cap receive nil with a truncation warning. Failed fetches also degrade to nil rather than raising.

^{Macroscope summarized f940b27.}

[linear-code]

EXT-9

[qltysh]

1 new issue

Tool	Category	Rule	Count
qlty	Structure	Function with high complexity (count = 10): collect_pages	1

[qltysh]

Function with high complexity (count = 10): collect_pages [qlty:function-complexity]

[macroscopeapp]

🟡 Medium issue/schema_definition.rb:80

A projection containing only messages:body_html returns every issue column and every message attribute instead of only the requested nested field. BaseCollection#project treats colon-qualified projections as having no scalar fields, and embed_foreign assigns the complete serialized record without applying the nested projection; the same leak affects relation projections such as account:name. Update projection handling at both levels so nested-only projections retain the requested fields.

Also found in 2 other location(s)

packages/forest_admin_datasource_pylon/lib/forest_admin_datasource_pylon/collections/issue/messages_embedder.rb:24

want_messages? accepts a nested-only projection such as ['messages:body_html'], but BaseCollection#project treats a projection containing only colon-qualified entries as having no scalar fields and returns the entire issue record. The resulting row therefore exposes every issue column in addition to messages, instead of honoring the requested projection. The existing spec masks this by always including id; nested-only structured-column projections trigger the leak.

packages/forest_admin_datasource_pylon/lib/forest_admin_datasource_pylon/collections/relation_embedder.rb:35

embed_foreign assigns the complete serialized foreign record to row[name] without applying the nested projection. For a projection such as account:name, the response therefore includes every account field (for example domains, CRM settings, and custom fields) instead of only name; when the projection contains only relation fields, BaseCollection#project also returns the complete source record because its scalar wanted set is empty. Relation/detail responses consequently over-return unrequested data.

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file @packages/forest_admin_datasource_pylon/lib/forest_admin_datasource_pylon/collections/issue/schema_definition.rb around line 80:

A projection containing only `messages:body_html` returns every issue column and every message attribute instead of only the requested nested field. `BaseCollection#project` treats colon-qualified projections as having no scalar fields, and `embed_foreign` assigns the complete serialized record without applying the nested projection; the same leak affects relation projections such as `account:name`. Update projection handling at both levels so nested-only projections retain the requested fields.

Also found in 2 other location(s):
- packages/forest_admin_datasource_pylon/lib/forest_admin_datasource_pylon/collections/issue/messages_embedder.rb:24 -- `want_messages?` accepts a nested-only projection such as `['messages:body_html']`, but `BaseCollection#project` treats a projection containing only colon-qualified entries as having no scalar fields and returns the entire issue record. The resulting row therefore exposes every issue column in addition to `messages`, instead of honoring the requested projection. The existing spec masks this by always including `id`; nested-only structured-column projections trigger the leak.
- packages/forest_admin_datasource_pylon/lib/forest_admin_datasource_pylon/collections/relation_embedder.rb:35 -- `embed_foreign` assigns the complete serialized foreign record to `row[name]` without applying the nested projection. For a projection such as `account:name`, the response therefore includes every account field (for example domains, CRM settings, and custom fields) instead of only `name`; when the projection contains only relation fields, `BaseCollection#project` also returns the complete source record because its scalar `wanted` set is empty. Relation/detail responses consequently over-return unrequested data.