Skip to content

fix(disaggregation): derive match_io waste weights for calendar 2017 - #1312

Merged
bl-young merged 5 commits into
mainfrom
by_derive_2017_match_io_waste_weights
Nov 13, 2026
Merged

bl-young merged 5 commits into
mainfrom
by_derive_2017_match_io_waste_weights

Conversation

@bl-young

@bl-young bl-young commented Nov 13, 2026 •

Copy link
Copy Markdown
Member

cc:
Related: #1269

What changed? Why?

get_waste_disagg_weights routes after-redefinition waste_weights_year: match_io through derive_waste_weights for every IO year, including calendar 2017, so that year no longer loads bundled CSVs with the workbook's 2012 RCRA Use intersection. Explicit / default waste_weights_year: 2017 and before-redef USEEIOR paths stay on CSV loads.

Under match_io, interior years 2018-2021 also blend Economic Census who-buys (Business-firms row-sum and special customer-class rows) between EC 2017 and EC 2022 with the same interpolate_shares rule as Census_EC.pxi_weights_for_year (t = (year - 2017) / 5). Endpoints stay on their own census. Dead eeio_waste_disaggregation: keys dropped from three electricity test YAMLs.

Impact

Changes waste child Use/Make structure (and thus waste-child EFs) on every v1.0 / nowcast config with match_io for 2017 (RCRA path) and 2018-2021 (EC who-buys phase-in). 2022+ who-buys already used EC 2022.

Testing

Live weight loads plus EEIO Use/Y after waste on pinned MUT v1.0.0_fd271dc for 2017-2022; tables in Evidence. Hermetic routing + blend tests. Post-pin IV/triage for 562212 still open.

Heads-up

No Steps 5-7 MUT regen -- waste weights are EEIO-stage on the existing fd271dc pin. B-driver / IV refresh for 562212 waits on the next pin that includes this fix. #1253 Make=Use+Y reconcile grading stays on #1269 if the 2022 output step remains large after the phase-in.

Evidence -- Use intersection, EC blend, EEIO U/q on fd271dc

Weight path (2017 RCRA seam)

Config waste_weights_year source_name rcra_source_year use_intersection[562HAZ, 562212]
2026_usa_cornerstone_v1_0_2017 match_io WasteDisaggregationDetail2017_derived 2017 0.032376
2026_usa_cornerstone_v1_0_2018 match_io WasteDisaggregationDetail2018_derived 2017 0.032376
test_usa_config_waste_disagg 2017 (default) WasteDisaggregationDetail2017 2012 0.142961

EC who-buys phase-in (match_io, pinned MUT v1.0.0_fd271dc)

derive_U_after_waste / derive_y_after_waste. Landfill household share = Y[562212, F01000] / sum_waste_children Y[*, F01000].

year ec_blend_t q_target[562212] B$ Y[562212, F01000] B$ LAND HH share of waste PCE U[562212, 562HAZ] B$ dq LAND
2017 (none) 9.76 1.13 0.040 0.352 --
2018 0.2 10.66 1.33 0.045 0.370 +0.90
2019 0.4 11.37 1.54 0.051 0.448 +0.71
2020 0.6 11.50 1.79 0.057 0.458 +0.12
2021 0.8 13.12 2.13 0.063 0.275 +1.62
2022 (none) 14.75 2.51 0.070 0.308 +1.63

HH share phases 0.040 -> 0.070 across 2017-2022 instead of jumping in one year at the EC seam. 2021->2022 q step remains (~+1.6 B$); that residual is mostly non-share (parent PCE / Use / reconcile), to grade on #1269 after IV refresh.

Route after-redef waste_weights_year=match_io through derive_waste_weights
even when the IO year is 2017, so the Use intersection uses BR 2017 instead
of the bundled workbook's 2012 RCRA shares (#1269).
@bl-young
bl-young requested a review from jvendries November 13, 2026 18:41
@bl-young

bl-young commented Nov 13, 2026 •

Copy link
Copy Markdown
Member Author

Expected q / B / IV on the next pin that includes this fix

Counterfactual from the EEIO smoke on pinned MUT v1.0.0_fd271dc plus the #1269 q_target table -- not a re-run IV yet. Only 2017 waste structure moves; 2018-2024 already derived.

q (562212)

2017 drops ~ 10.90 -> 9.76 B$ (~ -1.1 B$, almost all the HAZ cell). Later years unchanged.

2017->2018 mid-span vs 2017 2022 step
Before dip (-0.54 B$) soft / below 2017 big
After rise (+0.60 B$) above 2017 and rising still big (same absolute dollars)

Indexed 2017 = 1, q stops looking like a hole then recovery; it looks like a gentle climb, then the same 2022 jump (now ~ +51% vs 2017 instead of ~ +35%, because the base year is lower).

B (E flat => intensity ~ 1/q)

Before: mid-span B up (~ +11-14% by 2020) because q dipped under a high 2017 base.
After: mid-span B down (~ -6 to -9% vs new 2017) because q is above that lower base. Endpoint still soft (output grows, E flat). The 2022 B drop stays -- if anything the 2017=1 index makes it look a bit larger.

IV

Expect some shrink, not a clean exit: the fake 2018 HAZ cliff / soft mid-span yo-yo goes away, but the 2022 output step (SAS-backed + EC/PCE) still drives a large move, so peak year likely stays 2022 and landfill likely stays on the tracker. Rank / IV number needs a pin rebuild + IV refresh.

Unchanged: E (flat GHGI), 2018+ Use structure, the 2022 EC/PCE story on #1269.

@jvendries jvendries left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review

Verdict: Approve with a small ask (regression test). Fix matches the #1269 root cause.

Context (why this exists)

#1253 made waste Make follow Use+FD. That is the right identity, but it made landfill q/B inherit every Use seam. Digging on #1269 showed the soft 2018–2021 path was not “reconcile inventing softness”: calendar 2017 under match_io still loaded bundled CSVs (2012 RCRA intersection), while 2018+ derived from BR 2017. The big 562HAZ→562212 cliff (~1.56 B$ → ~0.37 B$) was that seam. Reconcile then locked Make to that soft Use path, so flat landfill E produced a large B / IV move.

This PR removes the 2017 special case for match_io so 2017 derives like later years. That is the right mid-span fix. The 2022 EC who-buys / PCE share step is correctly out of scope.

What looks good

  • Routing change is minimal and clear: after-redef match_io → always derive_waste_weights (including calendar 2017); explicit/default waste_weights_year: 2017 and before-redef stay on CSV/USEEIOR.
  • Docstrings / USAConfig comment now state the policy so this does not quietly regress.
  • Evidence on pinned MUT v1.0.0_fd271dc is convincing: 2017 HAZ×landfill aligns with 2018 (~0.35 B$), and q_target drop matches the removed bogus HAZ mass.
  • Dead eeio_waste_disaggregation: YAML keys cleanup is fine (ignored by USAConfig).

Ask before merge

Add an automated guard so this cannot silently return. Something that asserts:

  1. match_io + IO year 2017 → derived weights (WasteDisaggregationDetail2017_derived / rcra_source_year == 2017), and Use intersection 562HAZ×562212 is in the BR family (~0.03), not bundled (~0.14).
  2. Explicit waste_weights_year: 2017 (e.g. test_usa_config_waste_disagg) still loads bundled CSVs.

Without that, the exact elif weights_year == 2017 footgun can come back.

Non-blockers / follow-ups

  • Re-run IV/triage for 562212 on the next pin that includes this (Heads-up already says so). Expect mid-span soft q / high B to improve; do not expect the 2022 peak to vanish (EC/PCE share flip remains).
  • All waste children change in 2017 under match_io, not only landfill — a quick EF/spot check beyond 562212 after pin is worthwhile.
  • Separate call still needed for 2022: freeze/phase EC who-buys vs accept the Census-supported dollar step (#1269 open items).

Nit

elif cfg.waste_weights_year == "match_io" or weights_year != 2017 is correct given resolved_waste_weights_year; worth keeping the docstring next to that branch so future edits don’t reintroduce weights_year == 2017 → bundled for match_io.

@bl-young

Copy link
Copy Markdown
Member Author

Thanks -- agree on the footgun. Added a hermetic routing guard on TestWeightProvider:

  1. match_io + IO year 2017 mocks into derive_waste_weights (bundled load_disagg_weights must not run) and checks provenance rcra_source_year == 2017.
  2. Explicit / default waste_weights_year: 2017 stays on the bundled load (rcra_source_year == 2012 + bundled note); derive must not run.

Skipped asserting the ~ 0.03 intersection cell in CI (needs live BR or a tautological mock); that stays as PR Evidence / derive coverage. IV + broader 2017 waste-child spot check on the next pin as in the Heads-up.

@bl-young
bl-young enabled auto-merge (squash) November 13, 2026 20:33
@WesIngwersen

Copy link
Copy Markdown
Member

I dont see the Volatility score changes...computing those for this fix

@bl-young
bl-young disabled auto-merge November 13, 2026 20:45

@WesIngwersen WesIngwersen left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The 2017 routing fix is right, and I'd keep it. But it fixes only 2017, so it removes the 2018 jump in landfill's B and leaves the 2022 drop. That drop is what makes landfill volatile. I'd like the 2022 seam in this PR before approving.

What's left after this fix

Landfill emissions are flat, so B moves on output. Below is real B for 562212, indexed to 2017 = 1. The "before" row is from #1269. The other two rows rescale it for the 2017 output change here ($10.90bn to $9.76bn) and, in the last row, for the household-share phase-in described below. These are estimates, not a rebuild.

B, 2017 = 1 2017 2018 2019 2020 2021 2022 2023 2024 volatility 2022's share of the movement
before 1.00 1.11 1.13 1.14 1.06 0.89 0.94 0.90 0.088 62%
with this PR 1.00 0.99 1.01 1.02 0.95 0.80 0.84 0.81 0.075 73%
plus the Economic Census phase-in 1.00 0.98 0.98 0.97 0.89 0.80 0.84 0.81 0.057 42%

Volatility uses IV_score.volatility at the default trend weight. With this PR alone, the 2021-to-2022 change stays at -17.4%, and 2022 becomes almost the whole score. With the phase-in, the change in 2022 is -11.1%.

As your #1269 note shows, about 93% of landfill's 2022 household-spending jump comes from the Economic Census buyer shares switching from 2017 to 2022 in one year. The landfill share of household waste spending goes from 0.040 to 0.070. Total household spending on waste rises 6%. The Census services survey supports the industry's 2022 revenue step, but nothing supports moving the buyer mix in a single year.

The fix: reuse pxi_weights_for_year

We already handle this seam for the same census in Census_EC.pxi_weights_for_year. The 2017 and 2022 product-line shares are blended with inputs_structure.interpolate_shares at the shipped form (geometric, fitted on the manufacturing materials holdout). The blend runs at t = (year - 2017) / 5 between the census years and holds 2022 after that. The waste customer shares are the same kind of quinquennial mix, so they should use the same rule rather than a new one.

In derive_waste_weights, when the target year is strictly between 2017 and 2022 (and ec_2022_wired):

  1. Load both FBAs with load_census_ec_fba(2017) and load_census_ec_fba(2022).
  2. Business firms row sum. Build build_business_firms_row_sum_shares_from_ec_fba for each year. These are Series over WASTE_CHILDREN. Blend them as one-column frames with interpolate_shares(base, end, t), then pass the result to _replace_row_sum.
  3. Special customer classes (F01000, S00102, the GSLG*/S00203 and 813* codes). Pivot each year's build_customer_class_rows_from_ec_fba output to children × codes on PercentUsed, blend the columns with interpolate_shares, stack the result back to long rows, and pass them to _replace_customer_class.
  4. Use the same hold rules as pxi_weights_for_year. A class or (child, class) cell that one census doesn't publish keeps the other census's share instead of blending toward 0. Check the equal-share fallback in build_customer_class_rows_from_ec_fba in particular: a class that falls back in one census should hold the other census's real shares, not blend with the uniform split. interpolate_shares already falls back to linear for zero cells.
  5. Years outside the blend. 2017 and earlier stay on EC 2017, and 2022 and later stay on EC 2022, so resolve_ec_year is unchanged at the ends. Name the vintages in one place, like PXI_VINTAGES, rather than writing 2017 and 2022 inline twice.
  6. Provenance. Record the blend in WeightDerivationProvenance (both census years and t), so a reader of a 2019 weight can see it's 40% of the way to EC 2022.
  7. Flag. Put it behind a USAConfig flag, off by default, like the other seam changes waiting for the re-pin. Or tell me if you'd rather it go straight on with match_io.

The blend moves every waste child's buyer mix, not only landfill, so 562HAZ and the collection sectors move too. The test should mirror the hermetic routing guard you added: mock two EC years and check that a 2019 share lies between them at t = 0.4 in the shipped form, and that 2017 and 2022 return their own census unchanged.

What stays open

This doesn't touch the #1253 reconcile holding Make to Use plus final demand. On your probe, 2018-2021 still sit about 10% below the Make-table split. The phase-in raises those years by 1.4%, 2.9%, 4.9% and 6.6% through household spending alone, and the business-firms blend adds to that. If the 2022 step is still large after a rebuild, that reconcile is the next place to look. We can grade it on #1269.

Done when

On a rebuild with both changes, check:

  • landfill's B: the 2022 step is spread over 2018-2022;
  • landfill's volatility and its share in 2022, from a rerun of IV_score;
  • q = L_dom y passes for 2017-2024.

Results go on the tracker row and in this PR's body.

🤖 Generated with Claude Code

Phase waste customer shares with the same interpolate_shares rule as
Census_EC.pxi_weights_for_year so the 2022 who-buys cliff is not a one-year
step under match_io.
@bl-young

Copy link
Copy Markdown
Member Author

Thanks -- agree the 2017 routing fix alone leaves the 2022 who-buys cliff as most of landfill's IV. Implemented the Economic Census phase-in under waste_weights_year: match_io only (no separate USAConfig flag): interior years 2018-2021 blend EC 2017 and 2022 who-buys with the same interpolate_shares rule as Census_EC.pxi_weights_for_year (t = (year - 2017) / 5, geometric shipped form). Endpoints stay on their own census; explicit waste_weights_year: 2017 / before-redef unchanged.

Covers Business-firms row-sum and special customer-class rows (hold to the real census when the other vintage is an equal-share fallback). Provenance records ec_blend_t / base / end years.

Local EEIO smoke on pinned MUT v1.0.0_fd271dc

Landfill HH share of waste PCE (Y[562212,F01000] / sum waste-child Y[*,F01000]):

year ec_blend_t LAND HH share q_target[562212] B$
2017 -- 0.040 9.76
2018 0.2 0.045 10.66
2019 0.4 0.051 11.37
2020 0.6 0.057 11.50
2021 0.8 0.063 13.12
2022 -- 0.070 14.75

Share phases 0.040 -> 0.070 instead of a one-year cliff. 2021->2022 q still steps ~+1.6 B$; residual for #1253 / #1269 after IV refresh on the next pin.

Still open as you noted: Make=Use+Y reconcile grading, and volatility / 2022 share from a rerun of IV_score once this is on a pin.

@WesIngwersen WesIngwersen left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed eb0a677 and b8e203a. The blend follows pxi_weights_for_year as asked: it uses interpolate_shares at the shipped form with t = (year - 2017) / 5, keeps each census at its own end year, and records the blend in provenance. It covers both the business-firms row sum and the special customer classes. Approving.

Volatility change

Here is what the comment above asked for. Landfill B, 2017 = 1, uses the #1269 real B series, rescaled each year by the output in your fd271dc smoke (the q_target path from the 2017 fix alone, then with the blend). The volatility uses IV_score.volatility at the default trend weight. It's an estimate until the pin and the IV_score rerun.

B, 2017 = 1 2017 2018 2019 2020 2021 2022 2023 2024 volatility largest deviation
before 1.00 1.11 1.13 1.14 1.06 0.89 0.94 0.90 0.088 2022, 0.183
2017 fix only 1.00 0.99 1.01 1.02 0.95 0.80 0.84 0.81 0.075 2022, 0.168
2017 fix + blend 1.00 0.97 0.96 0.93 0.84 0.80 0.84 0.81 0.056 2023, 0.089
  • The 2021-to-2022 change goes from -17.4% to -5.7%. B now falls steadily from 2017 to 2022, at a median of -3.5% a year. The largest remaining deviation is 2023's +5.5% rebound, about half the size of the old 2022 drop.
  • 2021 is now the largest single fall at -10.2%, because output rises 14% in 2021 under the blend. Some of that is the #1253 reconcile, so it should be graded on #1269 along with the 2022 residual you named.

Checks on the census inputs

I loaded EC 2017 and EC 2022 on the branch and ran the builders.

  • Landfill's share of household waste spending goes 0.040 -> 0.070, and the 2019 blend gives 0.051. The other classes move the same way: federal 0.006 -> 0.039, nonprofits 0.006 -> 0.062.
  • Only waste combustors (562213) are zero in one census and not the other, at shares of 0.000-0.002. So the class-level hold for the equal-share fallback is enough, and a cell-level hold like PxI's isn't needed here.
  • The business-firms row sum barely moves between the two censuses (landfill 0.098 -> 0.106). The blend's effect comes almost entirely from the special classes.

Minor, not blocking

  1. For blended years, prov.ec_source_year still says 2017 from resolve_ec_year, next to ec_blend_t. A reader could take 2019 as EC 2017 only. Consider setting it to None for blended years, or documenting that the blend fields override it.
  2. Add a short test that ec_clcust_blend_t returns None at 2017 and 2022 and 0.2/0.8 at 2018/2021. The end years are covered only through the routing.
  3. With no flag, this changes 2018-2021 waste structure on every match_io config as soon as it merges. It's the EEIO stage, so the pin doesn't move, but the snapshot cut at 0113212 won't reproduce it. Note that in the next snapshot bump.

🤖 Generated with Claude Code

@bl-young

Copy link
Copy Markdown
Member Author

Confirmed on the same proxy: pin fd271dc real B, rescale by smoke q_target (2017 fix alone, then + blend), IV_score.volatility at default trend weight. Matches your table -- blend V ~0.056, peak moves to 2023, 2021->2022 d shrinks to about -6%. With I held, that is IV ~0.17 (still on the tracker; not a clean exit). Official rank waits on the next pin + IV_score rerun.

Merging so this does not sit behind the minors. Follow-up after merge:

  1. Clear / null ec_source_year on blended years so it does not read as EC 2017-only next to ec_blend_t.
  2. Optional hermetic asserts on ec_clcust_blend_t endpoints.
  3. Note on the next snapshot bump that match_io 2018-2021 who-buys no longer match cut 0113212b.

@bl-young
bl-young enabled auto-merge (squash) November 13, 2026 21:32
@bl-young
bl-young merged commit f083121 into main Nov 13, 2026
5 checks passed
@bl-young
bl-young deleted the by_derive_2017_match_io_waste_weights branch November 13, 2026 21:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants