Skip to content

Step 3: source research for the columns that actually drift - ORE, wholesale, retail, construction, services #705

Open

Description

@WesIngwersen

Nowcast plan Step 3. A research task before it is a build task. Plan: intermediate_estimation_plan.md §Where the drift actually sits.

#564 probed the annual business surveys and answered "no" for manufacturing, services, wholesale and retail. That answer stands. But the probe was aimed at the sectors that have surveys, not at the columns that actually drift — and those are not the same set.

intermediate_structure_drift.py --where, summary, 2024, by dollars misplaced:

industry dissimilarity column $M misplaced $M has an annual source?
ORE Other real estate 0.141 1,216,705 171,923 ❌
GSLG State and local general government 0.127 1,223,742 155,977 rescoped, #578
42 Wholesale trade 0.096 1,233,214 118,761 ❌ absent from AIES exp02
5412OP Misc. professional, scientific 0.114 827,267 94,444 ⚠️ one row per sector
81 Other services, except government 0.193 467,465 90,196 ⚠️ one row per sector
23 Construction 0.065 1,206,105 78,028 ❌ quinquennial only
722 Food services and drinking places 0.118 658,866 77,787 ⚠️ two rows
622 Hospitals 0.129 569,310 73,394 ⚠️
4A0 Other retail 0.098 590,900 57,765 ❌ absent from exp02

Together these carry more misplaced dollars than everything on #564's survivor list combined, and the sources that survived #564 (ERS agriculture, government finances) point at columns that are either small or already rescoped.

At BEA detail the same picture is sharper, and 531ORE tops it in both eras and at both levels — 0.188 over 2012→2017, 140,555 $M reallocated.

What this issue is

Find out what, if anything, can source these columns. Not "build an extractor" — decide whether one is possible. Concretely, per column:

  1. ORE / 531ORE Other real estate. The largest drifter in the table, at every level and in every span. What does a real estate lessor's input column even consist of, and does anything publish it annually? Note this interacts with 531HSO owner-occupied housing, which is imputed rather than surveyed — worth establishing which of the two is moving.
  2. 42 Wholesale and 4A0 retail. Absent from AIES exp02 entirely. Their intermediate columns are large and they drift. The Economic Census does cover them — ecnbasic operating expenses — which would be a second quinquennial observation on the pattern that just worked for manufacturing (Step 3: turn the MATFUEL extract into a usable materials seed - suppression, group splits, vintage diff, interpolation #698).
  3. 23 Construction. No annual source at all; ecnvalcon and ecnloccons are quinquennial. Same quinquennial-pair question.
  4. 5412OP, 81, 722, 622. One to a few rows per sector in the annual data. Establish whether a sector-level movement applied to detail sub-columns beats a frozen mix — a much weaker claim than Step 3: Probe annual business-survey expense data as a source for annual intermediate requirements #564 tested, and possibly true where the strong claim failed.

The bar

The same one everywhere in Step 3: beat a frozen 2017 mix by more than the inflation carry gets, measured on the index of dissimilarity in intermediate_structure_drift.py. The carry gets about 4% in a quiet span and is negative in a volatile one, so the bar is low — but it is a real bar and #564's negative result came from applying it honestly.

Deliverable

A written verdict per column, in the plan, with a go/no-go. Not code.

Activity

  1. added this to the v0.5 milestone on Aug 24, 2026
  2. self-assigned this
    on Aug 24, 2026
  3. removed this from the v0.5 milestone on Sep 28, 2026
  4. WesIngwersen commented on Sep 28, 2026

    @WesIngwersen
    MemberAuthor

    Removed from the v0.5 milestone in the scope cut; stays on the Nowcast US IOT Backlog. No activity since August.

  5. WesIngwersen commented on Nov 13, 2026

    @WesIngwersen
    MemberAuthor

    Progress on ORE and commercial services electricity: #1305 (merged 85258d8, flag off until the combined re-pin) moves commercial buyers' kWh on survey electricity or other intermediate inputs instead of their bill. The column-structure research this issue asks for, covering ORE, wholesale, retail, construction and services, is still open.

    🤖 Generated with Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions