feat(datasource-pylon): custom fields introspection (EXT-10)

[christophebrun-forest]

Closes EXT-10 — Story 6 of EXT-4.

What

The custom fields an organization defines in Pylon were invisible: the collections already knew how to hold them — add_custom_fields skips collisions and clamps operators, add_custom_field_values reads {slug, value} / {values}, api_filters merges the slugs — but nothing ever read the definitions, so every collection booted with an empty list. This adds the missing source.

File	Role
client.rb	fetch_custom_fields(object_type); collect_pages now carries query params
schema/custom_fields_introspector.rb	new: Pylon definitions → {column_name:, schema:} entries
datasource.rb	introspection at boot, one call per object type
spec/spec_helper.rb	stub_custom_fields, for the specs that build a datasource
collections/record_serialization.rb	coerces a read value to the form its filter compares it with
query/filter_value.rb	normalizes the number and the bare date a custom field filter sends
query/condition_tree_translator.rb	names the operators whose value is a moment in time

Six decisions worth reviewing, and a correction to the fourth

One call per collection, not one call in total. object_type is mandatory on GET /custom-fields, so boot spends three requests: issue, account, contact. Users and teams get none — Pylon carries no custom field on either, so an absent call rather than one answering empty. The parameter also rides along on every page of the cursor walk: dropped on the second request, it would answer for another object type.

A select advertises the slugs of its options, not their labels. Pylon reads a select back as the option slug and matches a filter against the slug ("the value must be the select option slug"), so exposing labels would need a mapping between what the operator sees and what travels — one more thing to keep in step, for a UI gain. Once every option is removed the column falls back to String, since Forest refuses an Enum carrying no value.

A Number gets no comparison operator. CUSTOM_FIELD_OPS spells the bare comparisons time_is_after / time_is_before, which is all Pylon documents on a custom field, so declaring greater_than on a numeric column would send a numeric range as a time filter. Numbers get equality and presence; dates get the comparisons; a multiselect gets nothing, its membership operators not being part of what a custom field accepts. Everything declared is drawn from CUSTOM_FIELD_OPS, so the per-collection clamp has nothing to drop and stays silent.

A date also gives up in / not_in, which the rest of CUSTOM_FIELD_OPS carries — Rules.get_allowed_operators_for_column_type grants a Date or a Dateonly column no array operator, the hazard operator_maps.rb already documents for MEMBERSHIP on a Json column. Native date columns declare the comparisons alone for the same reason.

Dropping them does not keep the filter out of the UI, though, and nothing in this datasource can. OperatorsEquivalenceCollectionDecorator#refine_schema republishes in from the equal the set declares — the IN fallback transform is depends_on: [EQUAL] with no for_types, so it fires on every column type — and a Dateonly also gets after_x_hours_ago / before_x_hours_ago republished from the comparisons, Times.compare deriving them for_types: %w[Date Dateonly] where Rules::DATEONLY refuses them. ConditionTreeValidator then rejects all three, so the operator is offered a date filter the agent answers with a 400.

Declaring equal is what triggers the first one, and giving it up to dodge a toolkit defect would cost an operator Pylon really accepts on a custom field. It would also protect nothing: QueryStringParser validates against the decorated collection before refine_filter rewrites, so the refusal does not depend on what this table declares. The same four operators are already published by the ActiveRecord datasource (parser/column.rb:108 declares equality + orderables on Date and Dateonly), so this is a live toolkit contradiction rather than anything introduced here — tracked as PRD-989, to be fixed in Rules outside this story. The set below is what Pylon accepts, which is the only question it can answer.

No custom field is groupable. ColumnSchema defaults is_groupable to true, and /_internal/capabilities turns a collection's supportGroups on as soon as a single field carries it, so one custom field left at the default is the whole collection offering a group-by that BaseCollection#aggregate raises on. Every native column declares it false already; a custom field now does too, and Pylon exposing no aggregate endpoint is the one reason behind both.

A type this datasource cannot map is skipped with a warning rather than guessed at: a column whose Forest type does not match what Pylon holds filters and displays wrong, which is worse for the operator than a column that is not there.

Two shapes no document settles

The custom fields are the first columns of this datasource typed Number, Boolean or Dateonly and filterable — no native column is both, API_FILTERS carrying none of them — so two value shapes reach the wire here for the first time. The API reference documents what a custom field is and never the form its value is read back in, and neither shape could be tried against a real organization: both are the canonical form rather than a verified one, and the specs pin them so a change is a decision rather than a drift.

A value is read as its column's type, not as a widened one. A number answering "42" is not ruled out by anything documented, so a value that is not already numeric is converted. One path turns that form into a result: an id filter is answered by GET /issues/{id} and the rest is applied in memory, so a number left as "42" was compared with the 42.0 ConditionTreeParser.cast_to_type casts the filter to, and the row was dropped without a word.

A value that is already numeric is handed back untouched. ConditionTreeLeaf#match compares with == and Array#include?, both of which hold across Integer and Float, so nothing needed widening — and widening it made an integer field display the 12.0 it does not hold, while format_float below narrows the very same value on the way out. A string is read to the tightest form for that reason: the two halves now agree on what an integer looks like. A Boolean becomes a real boolean, a date stays the ISO string it is — the filter carries one too, and comparing two of those is the ordering itself. A number that cannot be read reads as absent rather than as zero, and an empty boolean stays absent as well: false is an answer, and Pylon gave none.

A number and a bare date are normalized on the way out. The agent casts every Number column with to_f, so an integer custom field was filtered with 42.0, a form none of its values carry: a float with nothing after the point now travels as the integer it is, a decimal keeps its own. And time_is_after gets a timestamp from every native date column, where a Dateonly custom field sent it the bare date the frontend picks — it now gets the bound a Ruby Date already got, midnight in the timezone of the caller. That one is read off the emitted operator rather than off the shape of the string, so a text field holding what looks like a date keeps its value; a string no date can be read from is left to Pylon, which names what it refuses better than a guess here would.

What is left is the one failure these cannot rule out: a filter Pylon accepts without matching. It takes an organization carrying a field of each type to see, which EXT-11 has to probe before it trusts the write half.

Two deviations from the plan

No write half. The story scope mentions a payload builder, but no write endpoint exists yet — story 7 (EXT-11) owns create / update. Custom-field columns are declared is_read_only: true like every native column, and EXT-11 flips both at once, with a real endpoint to test against. Pylon's own is_read_only flag starts being honoured there too. The scope item is to be moved to EXT-11.

No pylon_slug key on the entries. The plan had one, mirroring Zendesk's zendesk_id / zendesk_key. It would have duplicated column_name: there is no rename here, since the slug is both the key a read payload indexes the value by and the field a filter sends. Entries are {column_name:, schema:}.

One behaviour change to existing specs

The datasource now performs HTTP while registering its collections, so does not call the API while registering collections no longer holds. It is replaced by the property that matters: definitions that cannot be read leave the agent booting on its native schema — fetch_custom_fields degrades to an empty list, like the message thread does, so a token missing the permission costs the operator the custom columns and not the datasource. The seven collection specs building a datasource declare what the introspection answers through stub_custom_fields; the helper takes the base url from the configuration default rather than from the datasource, since reading it there would boot the very introspection it stubs.

Verification

557 examples, 0 failures — coverage 1083/1083 lines (100%, threshold 90), branch 96.1%. RuboCop clean over the 51 files of the package.

Specs cover: the type table, an unmappable type and a slug-less definition being skipped, option slugs and the empty-options fallback, the operator set per type — including the exact set a date gets, asserted whole so the membership operators cannot creep back in — read-only / unsortable / ungroupable schemas; object_type sent and kept across pages, degradation on 403 and 500; three calls at boot and none for users or teams, each definition landing on the collection its object type registers, a slug colliding with a native column skipped end to end, two datasources not sharing columns; and the round trip that ties the three together — a select read back and filtered by its slug, a multiselect read out of values and left unfilterable.

And, for the two shapes above: every form a number and a boolean can come back in, including the ones that read as absent and the integer forms that must not widen; the integer-valued float travelling as an integer while a decimal keeps its own; the bare date read as midnight in the caller's timezone, the date-shaped text left alone on an operator that is not a time comparison, and the string no date can be read from left to Pylon. Each was checked to fail without its fix — the two that hold either way are named as the non-regression they are.

🤖 Generated with Claude Code

Note

Add custom fields introspection to Pylon datasource collections

• Adds CustomFieldsIntrospector to fetch custom-field definitions from Pylon's GET /custom-fields endpoint and expose them as typed columns on Issue, Account, and Contact collections.
• Custom-field values are now coerced to appropriate Ruby types (Number → Integer/Float, Boolean → true/false) during record serialization; unknown or unparseable values degrade gracefully to nil.
• Date-only strings used with time comparison operators (time_is_after, time_is_before) are converted to midnight UTC ISO8601 timestamps; integer-valued floats are sent as integers.
• Client#collect_pages is fixed to preserve query params (e.g. object_type) across all paginated requests.
• Behavioral Change: datasource initialization now calls GET /custom-fields per supported object type on boot; failures are logged and degrade to an empty custom-field set rather than raising.

^{Macroscope summarized 2e6a53f.}

[linear-code]

EXT-10

PRD-989

[qltysh]

1 new issue

Tool	Category	Rule	Count
qlty	Structure	Function with high complexity (count = 10): collect_pages	1

[qltysh]

Function with high complexity (count = 10): collect_pages [qlty:function-complexity]

[macroscopeapp]

🟡 Medium collections/record_serialization.rb:42

coerce_custom_field rounds valid large Pylon Number values by converting them to Float; for example, "9007199254740993" is exposed as 9007199254740992.0. This corrupts record reads and can make equality filters compare against the wrong number. Preserve integer and decimal precision instead of coercing every Number through Float.

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file @packages/forest_admin_datasource_pylon/lib/forest_admin_datasource_pylon/collections/record_serialization.rb around line 42:

`coerce_custom_field` rounds valid large Pylon Number values by converting them to `Float`; for example, `"9007199254740993"` is exposed as `9007199254740992.0`. This corrupts record reads and can make equality filters compare against the wrong number. Preserve integer and decimal precision instead of coercing every Number through `Float`.