Repository navigation
fix(disaggregation): derive match_io waste weights for calendar 2017 - #1312
Conversation
Route after-redef waste_weights_year=match_io through derive_waste_weights even when the IO year is 2017, so the Use intersection uses BR 2017 instead of the bundled workbook's 2012 RCRA shares (#1269).
Expected q / B / IV on the next pin that includes this fixCounterfactual from the EEIO smoke on pinned MUT q (562212)2017 drops ~ 10.90 -> 9.76 B$ (~ -1.1 B$, almost all the HAZ cell). Later years unchanged.
Indexed 2017 = 1, B (E flat => intensity ~ 1/q)Before: mid-span B up (~ +11-14% by 2020) because IVExpect some shrink, not a clean exit: the fake 2018 HAZ cliff / soft mid-span yo-yo goes away, but the 2022 output step (SAS-backed + EC/PCE) still drives a large move, so peak year likely stays 2022 and landfill likely stays on the tracker. Rank / IV number needs a pin rebuild + IV refresh. Unchanged: |
jvendries
left a comment
There was a problem hiding this comment.
Review
Verdict: Approve with a small ask (regression test). Fix matches the #1269 root cause.
Context (why this exists)
#1253 made waste Make follow Use+FD. That is the right identity, but it made landfill q/B inherit every Use seam. Digging on #1269 showed the soft 2018–2021 path was not “reconcile inventing softness”: calendar 2017 under match_io still loaded bundled CSVs (2012 RCRA intersection), while 2018+ derived from BR 2017. The big 562HAZ→562212 cliff (~1.56 B$ → ~0.37 B$) was that seam. Reconcile then locked Make to that soft Use path, so flat landfill E produced a large B / IV move.
This PR removes the 2017 special case for match_io so 2017 derives like later years. That is the right mid-span fix. The 2022 EC who-buys / PCE share step is correctly out of scope.
What looks good
- Routing change is minimal and clear: after-redef
match_io→ alwaysderive_waste_weights(including calendar 2017); explicit/defaultwaste_weights_year: 2017and before-redef stay on CSV/USEEIOR. - Docstrings /
USAConfigcomment now state the policy so this does not quietly regress. - Evidence on pinned MUT
v1.0.0_fd271dcis convincing: 2017 HAZ×landfill aligns with 2018 (~0.35 B$), andq_targetdrop matches the removed bogus HAZ mass. - Dead
eeio_waste_disaggregation:YAML keys cleanup is fine (ignored byUSAConfig).
Ask before merge
Add an automated guard so this cannot silently return. Something that asserts:
match_io+ IO year 2017 → derived weights (WasteDisaggregationDetail2017_derived/rcra_source_year == 2017), and Use intersection562HAZ×562212is in the BR family (~0.03), not bundled (~0.14).- Explicit
waste_weights_year: 2017(e.g.test_usa_config_waste_disagg) still loads bundled CSVs.
Without that, the exact elif weights_year == 2017 footgun can come back.
Non-blockers / follow-ups
- Re-run IV/triage for
562212on the next pin that includes this (Heads-up already says so). Expect mid-span softq/ highBto improve; do not expect the 2022 peak to vanish (EC/PCE share flip remains). - All waste children change in 2017 under
match_io, not only landfill — a quick EF/spot check beyond562212after pin is worthwhile. - Separate call still needed for 2022: freeze/phase EC who-buys vs accept the Census-supported dollar step (#1269 open items).
Nit
elif cfg.waste_weights_year == "match_io" or weights_year != 2017 is correct given resolved_waste_weights_year; worth keeping the docstring next to that branch so future edits don’t reintroduce weights_year == 2017 → bundled for match_io.
|
Thanks -- agree on the footgun. Added a hermetic routing guard on
Skipped asserting the ~ 0.03 intersection cell in CI (needs live BR or a tautological mock); that stays as PR Evidence / derive coverage. IV + broader 2017 waste-child spot check on the next pin as in the Heads-up. |
|
I dont see the Volatility score changes...computing those for this fix |
WesIngwersen
left a comment
There was a problem hiding this comment.
The 2017 routing fix is right, and I'd keep it. But it fixes only 2017, so it removes the 2018 jump in landfill's B and leaves the 2022 drop. That drop is what makes landfill volatile. I'd like the 2022 seam in this PR before approving.
What's left after this fix
Landfill emissions are flat, so B moves on output. Below is real B for 562212, indexed to 2017 = 1. The "before" row is from #1269. The other two rows rescale it for the 2017 output change here ($10.90bn to $9.76bn) and, in the last row, for the household-share phase-in described below. These are estimates, not a rebuild.
B, 2017 = 1 |
2017 | 2018 | 2019 | 2020 | 2021 | 2022 | 2023 | 2024 | volatility | 2022's share of the movement |
|---|---|---|---|---|---|---|---|---|---|---|
| before | 1.00 | 1.11 | 1.13 | 1.14 | 1.06 | 0.89 | 0.94 | 0.90 | 0.088 | 62% |
| with this PR | 1.00 | 0.99 | 1.01 | 1.02 | 0.95 | 0.80 | 0.84 | 0.81 | 0.075 | 73% |
| plus the Economic Census phase-in | 1.00 | 0.98 | 0.98 | 0.97 | 0.89 | 0.80 | 0.84 | 0.81 | 0.057 | 42% |
Volatility uses IV_score.volatility at the default trend weight. With this PR alone, the 2021-to-2022 change stays at -17.4%, and 2022 becomes almost the whole score. With the phase-in, the change in 2022 is -11.1%.
As your #1269 note shows, about 93% of landfill's 2022 household-spending jump comes from the Economic Census buyer shares switching from 2017 to 2022 in one year. The landfill share of household waste spending goes from 0.040 to 0.070. Total household spending on waste rises 6%. The Census services survey supports the industry's 2022 revenue step, but nothing supports moving the buyer mix in a single year.
The fix: reuse pxi_weights_for_year
We already handle this seam for the same census in Census_EC.pxi_weights_for_year. The 2017 and 2022 product-line shares are blended with inputs_structure.interpolate_shares at the shipped form (geometric, fitted on the manufacturing materials holdout). The blend runs at t = (year - 2017) / 5 between the census years and holds 2022 after that. The waste customer shares are the same kind of quinquennial mix, so they should use the same rule rather than a new one.
In derive_waste_weights, when the target year is strictly between 2017 and 2022 (and ec_2022_wired):
- Load both FBAs with
load_census_ec_fba(2017)andload_census_ec_fba(2022). - Business firms row sum. Build
build_business_firms_row_sum_shares_from_ec_fbafor each year. These are Series overWASTE_CHILDREN. Blend them as one-column frames withinterpolate_shares(base, end, t), then pass the result to_replace_row_sum. - Special customer classes (
F01000,S00102, theGSLG*/S00203and813*codes). Pivot each year'sbuild_customer_class_rows_from_ec_fbaoutput to children × codes onPercentUsed, blend the columns withinterpolate_shares, stack the result back to long rows, and pass them to_replace_customer_class. - Use the same hold rules as
pxi_weights_for_year. A class or (child, class) cell that one census doesn't publish keeps the other census's share instead of blending toward 0. Check the equal-share fallback inbuild_customer_class_rows_from_ec_fbain particular: a class that falls back in one census should hold the other census's real shares, not blend with the uniform split.interpolate_sharesalready falls back to linear for zero cells. - Years outside the blend. 2017 and earlier stay on EC 2017, and 2022 and later stay on EC 2022, so
resolve_ec_yearis unchanged at the ends. Name the vintages in one place, likePXI_VINTAGES, rather than writing 2017 and 2022 inline twice. - Provenance. Record the blend in
WeightDerivationProvenance(both census years andt), so a reader of a 2019 weight can see it's 40% of the way to EC 2022. - Flag. Put it behind a
USAConfigflag, off by default, like the other seam changes waiting for the re-pin. Or tell me if you'd rather it go straight on withmatch_io.
The blend moves every waste child's buyer mix, not only landfill, so 562HAZ and the collection sectors move too. The test should mirror the hermetic routing guard you added: mock two EC years and check that a 2019 share lies between them at t = 0.4 in the shipped form, and that 2017 and 2022 return their own census unchanged.
What stays open
This doesn't touch the #1253 reconcile holding Make to Use plus final demand. On your probe, 2018-2021 still sit about 10% below the Make-table split. The phase-in raises those years by 1.4%, 2.9%, 4.9% and 6.6% through household spending alone, and the business-firms blend adds to that. If the 2022 step is still large after a rebuild, that reconcile is the next place to look. We can grade it on #1269.
Done when
On a rebuild with both changes, check:
- landfill's
B: the 2022 step is spread over 2018-2022; - landfill's volatility and its share in 2022, from a rerun of
IV_score; q = L_dom ypasses for 2017-2024.
Results go on the tracker row and in this PR's body.
🤖 Generated with Claude Code
Phase waste customer shares with the same interpolate_shares rule as Census_EC.pxi_weights_for_year so the 2022 who-buys cliff is not a one-year step under match_io.
|
Thanks -- agree the 2017 routing fix alone leaves the 2022 who-buys cliff as most of landfill's IV. Implemented the Economic Census phase-in under Covers Business-firms row-sum and special customer-class rows (hold to the real census when the other vintage is an equal-share fallback). Provenance records Local EEIO smoke on pinned MUT
|
| year | ec_blend_t |
LAND HH share | q_target[562212] B$ |
|---|---|---|---|
| 2017 | -- | 0.040 | 9.76 |
| 2018 | 0.2 | 0.045 | 10.66 |
| 2019 | 0.4 | 0.051 | 11.37 |
| 2020 | 0.6 | 0.057 | 11.50 |
| 2021 | 0.8 | 0.063 | 13.12 |
| 2022 | -- | 0.070 | 14.75 |
Share phases 0.040 -> 0.070 instead of a one-year cliff. 2021->2022 q still steps ~+1.6 B$; residual for #1253 / #1269 after IV refresh on the next pin.
Still open as you noted: Make=Use+Y reconcile grading, and volatility / 2022 share from a rerun of IV_score once this is on a pin.
WesIngwersen
left a comment
There was a problem hiding this comment.
Re-reviewed eb0a677 and b8e203a. The blend follows pxi_weights_for_year as asked: it uses interpolate_shares at the shipped form with t = (year - 2017) / 5, keeps each census at its own end year, and records the blend in provenance. It covers both the business-firms row sum and the special customer classes. Approving.
Volatility change
Here is what the comment above asked for. Landfill B, 2017 = 1, uses the #1269 real B series, rescaled each year by the output in your fd271dc smoke (the q_target path from the 2017 fix alone, then with the blend). The volatility uses IV_score.volatility at the default trend weight. It's an estimate until the pin and the IV_score rerun.
B, 2017 = 1 |
2017 | 2018 | 2019 | 2020 | 2021 | 2022 | 2023 | 2024 | volatility | largest deviation |
|---|---|---|---|---|---|---|---|---|---|---|
| before | 1.00 | 1.11 | 1.13 | 1.14 | 1.06 | 0.89 | 0.94 | 0.90 | 0.088 | 2022, 0.183 |
| 2017 fix only | 1.00 | 0.99 | 1.01 | 1.02 | 0.95 | 0.80 | 0.84 | 0.81 | 0.075 | 2022, 0.168 |
| 2017 fix + blend | 1.00 | 0.97 | 0.96 | 0.93 | 0.84 | 0.80 | 0.84 | 0.81 | 0.056 | 2023, 0.089 |
- The 2021-to-2022 change goes from -17.4% to -5.7%.
Bnow falls steadily from 2017 to 2022, at a median of -3.5% a year. The largest remaining deviation is 2023's +5.5% rebound, about half the size of the old 2022 drop. - 2021 is now the largest single fall at -10.2%, because output rises 14% in 2021 under the blend. Some of that is the #1253 reconcile, so it should be graded on #1269 along with the 2022 residual you named.
Checks on the census inputs
I loaded EC 2017 and EC 2022 on the branch and ran the builders.
- Landfill's share of household waste spending goes 0.040 -> 0.070, and the 2019 blend gives 0.051. The other classes move the same way: federal 0.006 -> 0.039, nonprofits 0.006 -> 0.062.
- Only waste combustors (
562213) are zero in one census and not the other, at shares of 0.000-0.002. So the class-level hold for the equal-share fallback is enough, and a cell-level hold like PxI's isn't needed here. - The business-firms row sum barely moves between the two censuses (landfill 0.098 -> 0.106). The blend's effect comes almost entirely from the special classes.
Minor, not blocking
- For blended years,
prov.ec_source_yearstill says 2017 fromresolve_ec_year, next toec_blend_t. A reader could take 2019 as EC 2017 only. Consider setting it toNonefor blended years, or documenting that the blend fields override it. - Add a short test that
ec_clcust_blend_treturnsNoneat 2017 and 2022 and 0.2/0.8 at 2018/2021. The end years are covered only through the routing. - With no flag, this changes 2018-2021 waste structure on every
match_ioconfig as soon as it merges. It's the EEIO stage, so the pin doesn't move, but the snapshot cut at 0113212 won't reproduce it. Note that in the next snapshot bump.
🤖 Generated with Claude Code
|
Confirmed on the same proxy: pin Merging so this does not sit behind the minors. Follow-up after merge:
|
cc:
Related: #1269
What changed? Why?
get_waste_disagg_weightsroutes after-redefinitionwaste_weights_year: match_iothroughderive_waste_weightsfor every IO year, including calendar 2017, so that year no longer loads bundled CSVs with the workbook's 2012 RCRA Use intersection. Explicit / defaultwaste_weights_year: 2017and before-redef USEEIOR paths stay on CSV loads.Under
match_io, interior years 2018-2021 also blend Economic Census who-buys (Business-firms row-sum and special customer-class rows) between EC 2017 and EC 2022 with the sameinterpolate_sharesrule asCensus_EC.pxi_weights_for_year(t = (year - 2017) / 5). Endpoints stay on their own census. Deadeeio_waste_disaggregation:keys dropped from three electricity test YAMLs.Impact
Changes waste child Use/Make structure (and thus waste-child EFs) on every v1.0 / nowcast config with
match_iofor 2017 (RCRA path) and 2018-2021 (EC who-buys phase-in). 2022+ who-buys already used EC 2022.Testing
Live weight loads plus EEIO Use/Y after waste on pinned MUT
v1.0.0_fd271dcfor 2017-2022; tables in Evidence. Hermetic routing + blend tests. Post-pin IV/triage for562212still open.Heads-up
No Steps 5-7 MUT regen -- waste weights are EEIO-stage on the existing
fd271dcpin. B-driver / IV refresh for562212waits on the next pin that includes this fix. #1253 Make=Use+Y reconcile grading stays on #1269 if the 2022 output step remains large after the phase-in.Evidence -- Use intersection, EC blend, EEIO U/q on fd271dc
Weight path (2017 RCRA seam)
waste_weights_yearsource_namercra_source_yearuse_intersection[562HAZ, 562212]2026_usa_cornerstone_v1_0_2017match_ioWasteDisaggregationDetail2017_derived2026_usa_cornerstone_v1_0_2018match_ioWasteDisaggregationDetail2018_derivedtest_usa_config_waste_disagg2017(default)WasteDisaggregationDetail2017EC who-buys phase-in (match_io, pinned MUT
v1.0.0_fd271dc)derive_U_after_waste/derive_y_after_waste. Landfill household share =Y[562212, F01000] / sum_waste_children Y[*, F01000].ec_blend_tq_target[562212]B$Y[562212, F01000]B$U[562212, 562HAZ]B$HH share phases 0.040 -> 0.070 across 2017-2022 instead of jumping in one year at the EC seam. 2021->2022
qstep remains (~+1.6 B$); that residual is mostly non-share (parent PCE / Use / reconcile), to grade on #1269 after IV refresh.