Repository navigation
fix(bulk)!: decode all eighteen bulk routes as text/csv - #403
Merged
Merged
Conversation
Bulk FMP routes answer with `text/csv` rather than JSON (issue #376), so the response layer needs a way to accept and decode CSV bodies. Without it, those routes cannot be modelled correctly. **CSV response contract:** * Added a `TEXT_CSV` media type constant and a `Csv` variant to `ExpectedContentType`, validated case-insensitively against the body's media type. * Added a `ResponseContract::csv` constructor that decodes a headed CSV body into rows. An empty or header-only body yields no rows, and the JSON provider-message check is deliberately skipped since a short header-only body is data, not an error. * Added the `csv_rows` module with the row decoder and wired the `csv` dependency into the workspace and `libfmp` crate. **Error reporting:** * Extended `DecodeErrorKind::Syntax` to also cover malformed CSV records. No route uses the CSV contract yet.
Adds `EndpointSpec::get_csv`, which marks CSV bulk descriptors with a 256 MiB default response body limit. Whole-market bulk bodies exceed the 64 MiB general default (`ratios-ttm-bulk` measured 65.6 MiB on 2026-10-01), so the former shared default truncated them. The bulk default applies only while the client limit is unset. Making `ClientBuilder::max_response_body_bytes` an `Option` lets an explicitly configured limit, once set, apply to every endpoint, bulk and XLSX included, while each endpoint falls back to its own default otherwise. Adds decoder unit tests covering header-name mapping, quoted commas, verbatim numbers, null and malformed cells, missing columns, wrong cell counts, one-line headers, and content-type acceptance, plus a transport test asserting the per-endpoint and configured limits. Related to #376
This commit adds support for the bulk routes that answer with text/csv (issue #376). It introduces a hand-written getCSV decoder in sdk/go, though no generated method calls it yet. **CSV decoding:** * Added csv.go with a getCSV decoder and csv_test.go covering its behaviour, giving the SDK a path to parse the text/csv bodies that bulk endpoints return. **Bulk body limit:** * Added DefaultBulkMaxResponseBodyBytes (256 MiB) so CSV bulk bodies are buffered under a larger cap than the 64 MiB default while WithMaxResponseBodyBytes is unset; once it is set, bulk endpoints honour the configured value. * Tracked whether WithMaxResponseBodyBytes was supplied via maxResponseBodyBytesSet, and resolved bulkMaxResponseBodyBytes on the client accordingly. * Threaded an explicit maxBodyBytes argument through executeRedirects and roundTrip so each request enforces the limit that fits its route instead of the single client-wide cap.
Teach the registry wire resolver and gen_go about the CSV response contract so CSV endpoints are recognized and rendered correctly. * `EndpointSpec::get_csv` and `ResponseContract::csv()` now resolve to the new `Contract::CsvRows` variant in the wire resolver. * gen_go introduces a `ResponseKind::CsvRows` variant and renders such methods as `getCSV[Row]`, while reusing the row result type and zero value for the Go signature. * The wire surface validation and tests accept `Contract::CsvRows` alongside `Rows`, covering both the `get_csv` constructor and the explicit `ResponseContract::csv()` form. No descriptor uses the CSV contract yet, so generated output is unchanged. Refs #376
getCSV now verifies on the first record that every json-tagged model member, pointer or not, has a matching header column. A missing member is reported as a missing member at /0/<member> in declaration order, mirroring the Rust decoder's contract. A new missingColumn helper walks the model's fields in serde-reported declaration order and returns the first json-tagged name the header does not carry. Header-only bodies with missing columns still yield no rows without error, since there is no first record to check. Refs #376
Every member of the CSV row, `Option` or not, must now be present as a header column. The first record checks the serde struct field list, so a dropped or renamed provider column surfaces as a missing-member error instead of silently decoding to `None`. This catches upstream schema changes at the point of decode rather than leaving absent values to propagate unnoticed. Related to #376
Add a shared helper module that decodes a CSV bulk fixture through the real client and CSV contract. It verifies the round trip per cell, the member blank/drop contract, and handles empty and header-only bodies. The bulk suites adopt this helper in the following commits. Issue #376
The seven snapshot bulk routes (profile-bulk, rating-bulk, dcf-bulk, scores-bulk, price-target-summary-bulk, etf-holder-bulk, and upgrades-downgrades-consensus-bulk) now describe their endpoints with `EndpointSpec::get_csv`, because FMP answers these requests with text/csv rather than JSON. The generated Go methods regenerate to `getCSV`, real CSV fixtures replace the JSON bodies in the endpoint tests, and `.gitignore` un-ignores the CSV fixtures so they can be committed. The live etf-holder-bulk header now carries a clean `lastUpdated` column, so `BulkEtfHolding.last_updated_raw: String` (the wire key `lastUpdated` with a trailing quote, routed through the embedded fallback) becomes `last_updated: Date`. The Go `LastUpdatedRaw string` becomes `LastUpdated Date`, dropping the bespoke marshal/unmarshal shadow handling, and the Python `last_updated_raw: str` becomes `last_updated: date`. BREAKING CHANGE: the seven snapshot bulk methods now require a text/csv response; BulkEtfHolding.last_updated_raw is renamed to last_updated and typed as Date in Rust, Python and Go.
Reworks the bulk snapshot response suite to exercise the live CSV fixtures through the shared CSV helpers instead of the old JSON-model assertions, which no longer apply now that these routes do not answer JSON. **Testing changes:** * Replaces the JSON `include_bytes!` fixtures with their CSV counterparts and drives every assertion through the endpoint builders and the shared `bulk_csv` helpers (`assert_round_trip`, `assert_members`, `assert_empty_bodies`, `decode`). * Verifies a per-cell round trip with exact field counts for all seven bulk contracts. * Asserts the member contract: which members accept an empty cell, that every model column is required, and that unknown columns are tolerated. * Checks that empty and header-only bodies decode to no rows. * Confirms provider spellings and quoted text are preserved verbatim. * Drops the JSON-model assertions for required/nullable fields, numeric rejection, and bare-array roots. Relates to #376
The seven bulk snapshot JSON fixtures no longer described the wire, so the Go bulk parity tests now exercise the shared live CSV fixtures directly. **CSV parity helper:** * Added `assertCSVFixtureParity` in `sdk/go/csv_test.go`, mirroring `assert_round_trip` in `crates/libfmp/tests/support/bulk_csv.rs`: every row decodes through `decodeCSVRows`, the record count matches, and the first row re-encodes an exact number of members, each a header column whose re-encoded text (verbatim string, parsed number or boolean, or null/empty cell) equals the source cell. **Parity tests:** * Switched `TestBulkFixturesDecodeAndReencodeToTheSameMemberSet` and the renamed `TestBulkSnapshotsDecodeExactValues` to the CSV fixtures with per-type member counts, updating the expected values to the live rows and preserving the provider spellings (the "Stock Price" column, the quoted publishers list, the " -- " ETF symbol). * Folded the ETF holding date check into the exact-values test. **Removed fixtures:** * Deleted the seven `crates/libfmp/tests/fixtures/bulk_*.json` snapshot fixtures that no longer describe the wire format. Refs #376
Bulk snapshot routes answer with CSV bodies, so the seven snapshot tests now serve the shared live CSV fixtures as ``text/csv`` instead of parsed JSON, and assert the live values from those fixtures. * Add ``load_csv_fixture`` and ``CSV_CONTENT_TYPE`` to conftest so tests can read the raw CSV bodies from the shared libfmp fixtures. * Add a ``route_csv`` helper in the bulk test module that serves a shared CSV fixture on a path as ``text/csv``, matching how the bulk routes answer. * Repoint ``company_profiles``, ``stock_ratings``, ``dcf_valuations``, ``financial_scores``, ``price_target_summaries``, ``etf_holdings``, and ``upgrades_downgrades_consensus`` at the CSV fixtures and update the expected row counts and typed fields to the live values. * Assert ``etf_holdings`` decodes ``lastUpdated`` as a ``datetime.date`` rather than keeping the raw column string. Refs #376
The four metric bulk routes (key-metrics-ttm-bulk, ratios-ttm-bulk, peers-bulk, earnings-surprises-bulk) actually receive text/csv from FMP, so decoding them as JSON was wrong and failed against real responses. Switch these specs to EndpointSpec::get_csv and regenerate the Go methods to use getCSV instead of getJSON. Add live CSV fixtures for all four routes and update the Rust and Go endpoint suites to serve them as text/csv, adjusting assertions to the new fixture data. Refs: #376 BREAKING CHANGE: the four bulk methods KeyMetricsTTM, FinancialRatiosTTM, StockPeers, and EarningsSurprises now require a text/csv response and no longer decode JSON.
Rewrites the metric bulk response suite (key metrics TTM, ratios TTM, peers, earnings surprises) to run over the live CSV fixtures through the shared helpers instead of hand-rolled JSON assertions. Covers per-cell round trip with exact field counts, the member contract, empty and header-only bodies, numeric text kept verbatim beyond u64 and in exponent form, exact TTM wire spellings, the quoted peers list, and typed earnings dates. Adds a `with_cells` helper to the shared support for replacing named cells in a fixture's first record, and makes the member contract ignore blanks in columns the model does not read. Refs #376
The key metrics TTM, financial ratios TTM, stock peers, and earnings surprise parity tests now consume the shared live CSV fixtures instead of standalone JSON ones, exercising per-cell round trips with exact member counts. **Testing Improvements:** * Switched the four metric assertions in `sdk/go/bulk_parity_test.go` to `assertCSVFixtureParity`, pinning the expected member counts (43 key metrics, 60 ratios, 2 peers, 5 surprises). * Rebuilt the numeric-text preservation check to read and rewrite CSV cells directly, confirming beyond-u64, high-precision, and exponent values such as `6.9148336e-9` survive verbatim through `decodeCSVRows`. * Updated the expected symbols and values to match the CSV data for key metrics, ratios, peers, and earnings surprises. **Fixture Cleanup:** * Deleted the now-unused `bulk_key_metrics_ttm.json`, `bulk_financial_ratios_ttm.json`, `bulk_stock_peers.json`, and `bulk_earnings_surprises.json` fixtures. Refs #376
The four bulk metric route tests (key metrics TTM, ratios TTM, peers, earnings surprises) now serve the shared live CSV fixtures as text/csv and assert the live values instead of the former JSON fixtures. The decode-error test now sends a text/plain body to a CSV route to exercise the FmpDecodeError path for non-CSV responses. Relates to #376
FMP answers the income-statement-bulk, income-statement-growth-bulk, balance-sheet-statement-bulk, balance-sheet-statement-growth-bulk, cash-flow-statement-bulk, cash-flow-statement-growth-bulk, and eod-bulk routes with text/csv, so these descriptors now use EndpointSpec::get_csv. With this change all eighteen bulk routes are CSV. The generated Go methods switch from getJSON to getCSV, live CSV fixtures replace the former JSON ones, and the endpoint suites now serve those fixtures as text/csv while still checking that JSON bodies keep endpoint identities. The bulk descriptor unit test asserts the 256 MiB bulk default limit without overriding the client limit. Refs #376 BREAKING CHANGE: the income-statement-bulk, income-statement-growth-bulk, balance-sheet-statement-bulk, balance-sheet-statement-growth-bulk, cash-flow-statement-bulk, cash-flow-statement-growth-bulk, and eod-bulk methods now require a text/csv response.
Rewrites the bulk income statement and income growth response suite over the live CSV fixtures through the shared CSV helpers: per-cell round trip with exact field counts, every column required with no member accepting an empty cell, empty and header-only bodies, identity representations with invalid date and datetime cells, and the exact acronym wire names. Adds first_row and assert_cell_failure helpers to the shared support. The JSON-model assertions are dropped because these routes no longer answer JSON. Relates to #376
Rewrite the bulk balance sheet and balance growth response tests to run over the live CSV fixtures through the shared CSV helpers. Coverage now asserts per-cell round trip with exact field counts, requires every column with no member accepting an empty cell, checks empty and header-only bodies, verifies identity representations, and pins the exact provider typo and liabilities-equity wire names. Drop the JSON-model assertions because these routes no longer answer JSON. Relates to #376
Rewrites the bulk cash flow, cash flow growth and eod response suite over the live CSV fixtures through the shared CSV helpers: per-cell round trip with exact field counts, every column required with no member accepting an empty cell, empty and header-only bodies, identity and price representations, and the exact Activites typo wire names. The JSON-model assertions are dropped because these routes no longer answer JSON. Issue #376
Switch the Go statement and EOD bulk parity tests to decode the shared live CSV fixtures instead of JSON. Each assertion now exercises a per-cell round trip, exact member counts, identity representations (ten-zero CIK, naive accepted date, period), and provider typo wire names. Delete the seven obsolete statement and eod JSON fixtures that are no longer referenced. Extend TestEveryFixtureIsReferencedByAGoTest so it also requires every .csv fixture to be named by at least one Go test, alongside the existing .json check. Refs #376
Switch the six bulk statement routes and the eod route to serve the shared live CSV fixtures as text/csv via a new `route_csv` helper, and update the assertions to match the live values in those fixtures. Relates to #376
Under ADR 0035, an empty CSV cell is the only absence marker for bulk rows and sparse empties vary by period, so every numeric metric member becomes optional while identity members stay required. Every NumericString member of BulkStockRating, BulkDcfValuation, BulkFinancialScore, BulkPriceTargetSummary, BulkEtfHolding and BulkUpgradesDowngradesConsensus now decodes as absent when its cell is empty. Typed codes seen empty in the live census also become optional via the empty_or_null codec: BulkFinancialScore.reported_currency (313 of 62k rows), BulkEtfHolding.asset and BulkEtfHolding.isin. Fixtures gain the live rows with empty cells and tests assert None for them. Regenerated Python models/stubs and Go models are included. Refs #376 BREAKING CHANGE: discounted_cash_flow_score, return_on_equity_score, return_on_assets_score, debt_to_equity_score, price_to_earnings_score, price_to_book_score, dcf, stock_price, reported_currency, altman_z_score, piotroski_score, working_capital, total_assets, retained_earnings, ebit, market_cap, total_liabilities, revenue, last_month_count, last_month_avg_price_target, last_quarter_count, last_quarter_avg_price_target, last_year_count, last_year_avg_price_target, all_time_count, all_time_avg_price_target, shares_number, asset, weight_percentage, isin, market_value, strong_buy, buy, hold, sell and strong_sell are now Option<NumericString> (asset Option<Ticker>, isin Option<Isin>, reported_currency Option<CurrencyCode>) in Rust, Optional[str] in Python and *string in Go.
Under ADR 0035, an empty CSV cell is the only absence marker in bulk responses, and sparse empties vary by reporting period. Treating every numeric metric as required wrongly rejected valid live rows, so each numeric metric member now becomes optional while identity members stay required. Every NumericString member of BulkKeyMetricsTtm, BulkFinancialRatiosTtm and BulkEarningsSurprise is now Option<NumericString> (Optional[str] in Python, *string in Go). BulkStockPeer.peers stays a String, and an empty cell stays an empty string. Fixtures gain live rows exercising empty cells (ADAMO key metrics, ARS.AX ratios, GUD.TO epsEstimated, an empty peers list), and tests assert the absent members decode to None. Regenerated Python models and stubs and Go models are included. BREAKING CHANGE: every numeric member of BulkKeyMetricsTtm, BulkFinancialRatiosTtm and BulkEarningsSurprise is now optional in Rust, Python and Go.
Under ADR 0035, an empty CSV cell is the only absence marker in bulk rows, and sparse empties vary by period. Every numeric metric member of BulkIncomeStatement and BulkIncomeStatementGrowth now becomes Option<NumericString> (Python Optional[str], Go *string), while identity members (date, symbol, reportedCurrency, cik, filingDate, acceptedDate, fiscalYear, period) stay required. The live FY2024 census saw empty eps, epsDiluted, weightedAverageShsOut(Dil), netIncomeDeductions and others. The fixture gains the live OASMY row with empty cells, and tests assert None where the cells are absent. Regenerated Python models/stubs and Go models are included. BREAKING CHANGE: every numeric member of BulkIncomeStatement and BulkIncomeStatementGrowth is now optional in Rust, Python and Go.
Under ADR 0035, an empty cell is the only absence marker in bulk CSV and sparse empties vary by period, so every numeric metric member of a bulk row must be optional while identity members stay required. This covers the balance-sheet rows: every NumericString member of BulkBalanceSheetStatement and BulkBalanceSheetStatementGrowth becomes Option<NumericString>, with identity members left required. The member-contract test now blanks each column of the live row to check that only identity members reject an empty cell. Regenerated Python models/stubs (Optional[str]) and Go models (*string) are included. BREAKING CHANGE: every numeric member of BulkBalanceSheetStatement and BulkBalanceSheetStatementGrowth is now optional in Rust, Python and Go.
Under ADR 0035 an empty CSV cell is the only absence marker for bulk rows, and sparse empties vary by period, so every numeric metric member becomes optional while identity members stay required. This commit covers the cash-flow and end-of-day rows: every NumericString member of BulkCashFlowStatement, BulkCashFlowStatementGrowth and BulkEodBar becomes Option<NumericString> (Python Optional[str], Go *string). Identity members (symbol, date, and the statement identity columns such as reportedCurrency, cik, filingDate, acceptedDate, fiscalYear and period) stay required. The Go decoders drop the missing-member guards for the now-optional members, and the libfmp tests assert that only identity members reject an empty cell. Regenerated Python models/stubs and Go models are included. BREAKING CHANGE: every numeric member of BulkCashFlowStatement, BulkCashFlowStatementGrowth and BulkEodBar is now optional in Rust, Python and Go.
The live profile-bulk census over 46.7k rows surfaced empty cells that the previous non-optional CompanyProfile members could not represent: isin was empty in 33 rows, country in 7, and both change and changePercentage in 2. Since CompanyProfile is shared by company.profile (JSON) and profile-bulk (CSV), these gaps affect both endpoints. isin now decodes as Option<Isin> and country as Option<CountryCode> through the empty_or_null codec, so an empty string or null becomes None. change and change_percentage become Option<f64> through required_option, keeping the key required while allowing an absent value. Python exposes these as Optional and Go as pointers, with the Go shadow decoding raw jsontext and mapping empty strings to nil. The bulk fixture gains the live ACCV row (empty change) and CFHAX row (empty isin, country, ceo and website), and the company and bulk tests across the Rust, Python and Go SDKs assert the new optional types. Refs #376 BREAKING CHANGE: CompanyProfile.isin, country, change and change_percentage are now optional in Rust, Python and Go, affecting both company.profile and bulk company_profiles.
The mypy type-check contract now expects `str | None` for the bulk numeric members it reads, which became optional under the bulk CSV rule: * `BulkIncomeStatement.revenue` * `BulkEarningsSurprise.eps_actual` * `BulkEodBar.close` Related to issue #376.
Add a parametrized Python test verifying that an empty or header-only ``text/csv`` body on a bulk route decodes to an empty list rather than raising a provider error. Refs #376
Record that the eighteen bulk routes answer text/csv decoded by header name: the Rust csv crate splits records into a serde cell deserializer, Go uses encoding/csv with the model JSON rules, and the cell rules make every model member a required header column. Add the bulk-CSV-only rule that numeric metric members are optional while identity members stay required, with JSON endpoints keeping ADR 0033 evidence-only. Cover the CompanyProfile widening, the ETF lastUpdated date, and the 256 MiB bulk body-limit default, since ratios-ttm-bulk measured 65.6 MiB and exceeds the 64 MiB general default. Mark ADR 0028 superseded in part, update the bulk module doc, the ADR 0030 raw-key example, and the Python client limit docstring to match. close #376
The Rust csv reader strips a leading byte order mark before parsing the header, but Go's encoding/csv keeps it, so the first header name carried the BOM and the first column failed to decode. Trim the UTF-8 BOM from the body before reading. Add matching tests in both the Rust and Go decoders to confirm a leading byte order mark is not treated as part of the first header.
benbenbang
enabled auto-merge (rebase)
October 1, 2026 05:49
github-actions
Bot
added
documentation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
FMP answers every bulk route with text/csv rather than JSON (issue #376), so decoding these responses as JSON was wrong and failed against real whole-market bodies. This change teaches the response layer, the code generator and all three SDKs to accept CSV, and reconciles the bulk models with what the live wire actually carries.
CSV response contract:
Larger bulk body limit:
Code generation:
Bulk route migration:
Tests, fixtures and docs:
BREAKING CHANGE: all eighteen bulk methods now require a text/csv response; every numeric bulk member is now optional; BulkEtfHolding.last_updated_raw is renamed to last_updated and typed as Date; and CompanyProfile.isin, country, change and change_percentage are now optional, in Rust, Python and Go.
close #376