Skip to content

Simplify JEDI shared bias input staging - #1

Draft
ADCollard with Copilot wants to merge 2 commits into
developfrom
copilot/remove-redundant-bias-files-dict
Draft

ADCollard with Copilot wants to merge 2 commits into
developfrom
copilot/remove-redundant-bias-files-dict

Conversation

Copilot AI commented Sep 9, 2026 •

Copy link
Copy Markdown

Description

Bias input staging in ush/python/pygfs/jedi/jedi.py was still routing through bias_files_dict even though every supported observation resolved to the same archive, varbc_params.tar. This change makes the shared-archive behavior explicit in stage_obsbiasin() and leaves save_obsbiasout() unchanged.

  • Bias input staging

    • Stage a single shared archive, varbc_params.tar, when observations is non-empty.
    • Stop using per-observation bias_files_dict lookups in this module.
    • Preserve current skip behavior when the archive is missing by warning and continuing.
  • Behavior preserved

    • Duplicate observations no longer affect staging; the shared archive is copied and extracted once.
    • Bias output archival still tars whatever bias files already exist.
  • Regression coverage

    • Add focused unit coverage for:
      • staging the shared archive once with duplicate observations
      • warning and skipping cleanly when the shared archive is absent
bias_tarball = self.jcb_config[f'{self.component}_obsbiasin_prefix'] + 'varbc_params.tar'
if self.jcb_config['observations']:
    bias_src = os.path.join(comin, bias_tarball)

Type of change

  • Bug fix (fixes something broken)
  • New feature (adds functionality)
  • Maintenance (code refactor, clean-up, new CI test, etc.)

Change characteristics

  • Is this change expected to change outputs (e.g. value changes to existing outputs, new files stored in COM, files removed from COM, filename changes, additions/subtractions to archives)? NO
    • GFS
    • GEFS
    • SFS
    • GCAFS
  • Is this a breaking change (a change in existing functionality)? NO
  • Does this change require a documentation update? NO
  • Does this change require an update to any of the following submodules? NO
    • EMC verif-global
    • GDAS
    • GFS-utils
    • GSI
    • GSI-monitor
    • GSI-utils
    • UFS-utils
    • UFS-weather-model
    • wxflow

How has this been tested?

  • Added targeted unit coverage in dev/ci/scripts/unittests/test_jedi_bias_staging.py for shared-archive staging, duplicate observations, and missing-archive warnings.

Checklist

  • Any dependent changes have been merged and published
  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have documented my code, including function, input, and output descriptions
  • My changes generate no new warnings
  • New and existing tests pass with my changes
  • This change is covered by an existing CI test or a new one has been added
  • Any new scripts have been added to the .github/CODEOWNERS file with owners
  • I have made corresponding changes to the system documentation if necessary

Co-authored-by: ADCollard <40322596+ADCollard@users.noreply.github.com>
Copilot AI changed the title [WIP] Simplify bias staging logic by removing bias_files_dict usage Simplify JEDI shared bias input staging Sep 9, 2026
Copilot AI requested a review from ADCollard September 9, 2026 14:45
ADCollard pushed a commit that referenced this pull request Oct 21, 2026
…_seg0 (NOAA-EMC#5280)

These update should enable the operability of functional CTest
**C48_S2SW**

- Rename model/wave/prep to model/wave/init in
C48_S2SW-gfs_fcst_seg0.yaml
- Rename model/wave/prep to model/wave/init in
C48_S2SW-gfs_waveinit.yaml
- Rename model/wave/prep to model/wave/init in
C48_S2SW-gfs_wavepostbndpnt.yaml
- Rename model/wave/prep to model/wave/init in
C48_S2SW-gfs_wavepostbndpntbll.yaml
- Rename model/wave/prep to model/wave/init in
C48_S2SW-gfs_wavepostpnt.yaml
- Rename model/wave/prep to model/wave/init in
C48_S2SW-gfs_wavepostsbs_f000-f002.yaml
- Updates directory naming to align with wave model initialization stage
nomenclature

We can not run these tests with a CI pipeline with this PR because this
PR has a change in logic within the default pipeline script
[.gitlab.yml](https://github.com/NOAA-EMC/global-workflow/pull/5280/changes#diff-037ea159eb0a7cb0ac23b851e66bee30fb838ee8d0d99fa331a1ba65283d37f7R162)
itself.


Tested on disk
```
Terry.McGuinness (gaea64) build (feature/ci-hist_init) $ ctest -j 100
Test project /gpfs/f6/drsa-precip3/scratch/Terry.McGuinness/global-workflow_forked/dev/ctests/build
      Start 37: test_C48_S2SW-gfs_waveinit_setup
      Start 25: test_C48_S2SW-gfs_fcst_seg0_setup
      Start 33: test_C48_S2SW-gfs_ocean_prod_f006_setup
      Start 21: test_C48_S2SW-gfs_stage_ic_setup
      Start 53: test_C48_S2SW-gfs_wavepostbndpntbll_setup
      Start 41: test_C48_S2SW-gfs_wavepostpnt_setup
      Start 29: test_C48_S2SW-gfs_ice_prod_f006_setup
      Start 49: test_C48_S2SW-gfs_wavepostbndpnt_setup
      Start 45: test_C48_S2SW-gfs_wavepostgridded_f000-f002_setup
      Start 17: test_C48_ATM-gfs_genesis_setup
      Start  1: test_C48_ATM-gfs_stage_ic_setup
      Start  5: test_C48_ATM-gfs_fcst_seg0_setup
      Start  9: test_C48_ATM-gfs_atmos_prod_f000-f002_setup
      Start 13: test_C48_ATM-gfs_tracker_setup
      Start 57: test_C48_S2SWA_gefs-gefs_fcst_mem001_seg0_setup
 1/60 Test NOAA-EMC#13: test_C48_ATM-gfs_tracker_setup .........................   Passed    3.15 sec
      Start 14: test_C48_ATM-gfs_tracker_stage
 2/60 Test NOAA-EMC#17: test_C48_ATM-gfs_genesis_setup .........................   Passed    3.16 sec
      Start 18: test_C48_ATM-gfs_genesis_stage
 3/60 Test  NOAA-EMC#5: test_C48_ATM-gfs_fcst_seg0_setup .......................   Passed    3.16 sec
      Start  6: test_C48_ATM-gfs_fcst_seg0_stage
 4/60 Test  #1: test_C48_ATM-gfs_stage_ic_setup ........................   Passed    3.16 sec
      Start  2: test_C48_ATM-gfs_stage_ic_stage
 5/60 Test  NOAA-EMC#9: test_C48_ATM-gfs_atmos_prod_f000-f002_setup ............   Passed    3.16 sec
      Start 10: test_C48_ATM-gfs_atmos_prod_f000-f002_stage
 6/60 Test NOAA-EMC#57: test_C48_S2SWA_gefs-gefs_fcst_mem001_seg0_setup ........   Passed    3.23 sec
      Start 58: test_C48_S2SWA_gefs-gefs_fcst_mem001_seg0_stage
 7/60 Test NOAA-EMC#41: test_C48_S2SW-gfs_wavepostpnt_setup ....................   Passed    3.33 sec
      Start 42: test_C48_S2SW-gfs_wavepostpnt_stage
 8/60 Test NOAA-EMC#33: test_C48_S2SW-gfs_ocean_prod_f006_setup ................   Passed    3.34 sec
      Start 34: test_C48_S2SW-gfs_ocean_prod_f006_stage
 9/60 Test NOAA-EMC#37: test_C48_S2SW-gfs_waveinit_setup .......................   Passed    3.35 sec
      Start 38: test_C48_S2SW-gfs_waveinit_stage
10/60 Test NOAA-EMC#45: test_C48_S2SW-gfs_wavepostgridded_f000-f002_setup ......   Passed    3.34 sec
      Start 46: test_C48_S2SW-gfs_wavepostgridded_f000-f002_stage
11/60 Test NOAA-EMC#25: test_C48_S2SW-gfs_fcst_seg0_setup ......................   Passed    3.35 sec
      Start 26: test_C48_S2SW-gfs_fcst_seg0_stage
12/60 Test NOAA-EMC#53: test_C48_S2SW-gfs_wavepostbndpntbll_setup ..............   Passed    3.35 sec
      Start 54: test_C48_S2SW-gfs_wavepostbndpntbll_stage
13/60 Test NOAA-EMC#49: test_C48_S2SW-gfs_wavepostbndpnt_setup .................   Passed    3.35 sec
      Start 50: test_C48_S2SW-gfs_wavepostbndpnt_stage
14/60 Test NOAA-EMC#29: test_C48_S2SW-gfs_ice_prod_f006_setup ..................   Passed    3.35 sec
      Start 30: test_C48_S2SW-gfs_ice_prod_f006_stage
15/60 Test NOAA-EMC#21: test_C48_S2SW-gfs_stage_ic_setup .......................   Passed    3.36 sec
      Start 22: test_C48_S2SW-gfs_stage_ic_stage
16/60 Test  NOAA-EMC#2: test_C48_ATM-gfs_stage_ic_stage ........................   Passed    1.56 sec
      Start  3: test_C48_ATM-gfs_stage_ic_execute
17/60 Test NOAA-EMC#30: test_C48_S2SW-gfs_ice_prod_f006_stage ..................   Passed    1.44 sec
      Start 31: test_C48_S2SW-gfs_ice_prod_f006_execute
18/60 Test NOAA-EMC#38: test_C48_S2SW-gfs_waveinit_stage .......................   Passed    1.45 sec
      Start 39: test_C48_S2SW-gfs_waveinit_execute
19/60 Test NOAA-EMC#22: test_C48_S2SW-gfs_stage_ic_stage .......................   Passed    1.46 sec
      Start 23: test_C48_S2SW-gfs_stage_ic_execute
20/60 Test NOAA-EMC#34: test_C48_S2SW-gfs_ocean_prod_f006_stage ................   Passed    1.50 sec
      Start 35: test_C48_S2SW-gfs_ocean_prod_f006_execute
21/60 Test NOAA-EMC#10: test_C48_ATM-gfs_atmos_prod_f000-f002_stage ............   Passed    2.08 sec
      Start 11: test_C48_ATM-gfs_atmos_prod_f000-f002_execute
22/60 Test  NOAA-EMC#6: test_C48_ATM-gfs_fcst_seg0_stage .......................   Passed    3.32 sec
      Start  7: test_C48_ATM-gfs_fcst_seg0_execute
23/60 Test NOAA-EMC#26: test_C48_S2SW-gfs_fcst_seg0_stage ......................   Passed    5.50 sec
      Start 27: test_C48_S2SW-gfs_fcst_seg0_execute
24/60 Test NOAA-EMC#46: test_C48_S2SW-gfs_wavepostgridded_f000-f002_stage ......   Passed    5.64 sec
      Start 47: test_C48_S2SW-gfs_wavepostgridded_f000-f002_execute
25/60 Test NOAA-EMC#58: test_C48_S2SWA_gefs-gefs_fcst_mem001_seg0_stage ........   Passed    9.70 sec
      Start 59: test_C48_S2SWA_gefs-gefs_fcst_mem001_seg0_execute
26/60 Test NOAA-EMC#42: test_C48_S2SW-gfs_wavepostpnt_stage ....................   Passed   12.52 sec
      Start 43: test_C48_S2SW-gfs_wavepostpnt_execute
27/60 Test NOAA-EMC#54: test_C48_S2SW-gfs_wavepostbndpntbll_stage ..............   Passed   12.50 sec
      Start 55: test_C48_S2SW-gfs_wavepostbndpntbll_execute
28/60 Test NOAA-EMC#50: test_C48_S2SW-gfs_wavepostbndpnt_stage .................   Passed   12.50 sec
      Start 51: test_C48_S2SW-gfs_wavepostbndpnt_execute
29/60 Test  NOAA-EMC#3: test_C48_ATM-gfs_stage_ic_execute ......................   Passed   60.80 sec
      Start  4: test_C48_ATM-gfs_stage_ic_validate
30/60 Test NOAA-EMC#35: test_C48_S2SW-gfs_ocean_prod_f006_execute ..............   Passed   60.79 sec
      Start 36: test_C48_S2SW-gfs_ocean_prod_f006_validate
31/60 Test NOAA-EMC#23: test_C48_S2SW-gfs_stage_ic_execute .....................   Passed   60.81 sec
      Start 24: test_C48_S2SW-gfs_stage_ic_validate
32/60 Test NOAA-EMC#31: test_C48_S2SW-gfs_ice_prod_f006_execute ................   Passed   60.84 sec
      Start 32: test_C48_S2SW-gfs_ice_prod_f006_validate
33/60 Test NOAA-EMC#39: test_C48_S2SW-gfs_waveinit_execute .....................   Passed   60.85 sec
      Start 40: test_C48_S2SW-gfs_waveinit_validate
34/60 Test NOAA-EMC#36: test_C48_S2SW-gfs_ocean_prod_f006_validate .............   Passed    1.49 sec
35/60 Test NOAA-EMC#32: test_C48_S2SW-gfs_ice_prod_f006_validate ...............   Passed    1.48 sec
36/60 Test NOAA-EMC#40: test_C48_S2SW-gfs_waveinit_validate ....................   Passed    1.48 sec
37/60 Test  NOAA-EMC#4: test_C48_ATM-gfs_stage_ic_validate .....................   Passed    1.59 sec
38/60 Test NOAA-EMC#24: test_C48_S2SW-gfs_stage_ic_validate ....................   Passed    1.50 sec
39/60 Test NOAA-EMC#47: test_C48_S2SW-gfs_wavepostgridded_f000-f002_execute ....   Passed  120.68 sec
      Start 48: test_C48_S2SW-gfs_wavepostgridded_f000-f002_validate
40/60 Test NOAA-EMC#48: test_C48_S2SW-gfs_wavepostgridded_f000-f002_validate ...   Passed    1.49 sec
41/60 Test NOAA-EMC#51: test_C48_S2SW-gfs_wavepostbndpnt_execute ...............   Passed  180.71 sec
      Start 52: test_C48_S2SW-gfs_wavepostbndpnt_validate
42/60 Test NOAA-EMC#52: test_C48_S2SW-gfs_wavepostbndpnt_validate ..............   Passed    1.49 sec
43/60 Test NOAA-EMC#43: test_C48_S2SW-gfs_wavepostpnt_execute ..................   Passed  240.75 sec
      Start 44: test_C48_S2SW-gfs_wavepostpnt_validate
44/60 Test NOAA-EMC#44: test_C48_S2SW-gfs_wavepostpnt_validate .................   Passed    1.44 sec
45/60 Test NOAA-EMC#11: test_C48_ATM-gfs_atmos_prod_f000-f002_execute ..........   Passed  300.88 sec
      Start 12: test_C48_ATM-gfs_atmos_prod_f000-f002_validate
46/60 Test NOAA-EMC#12: test_C48_ATM-gfs_atmos_prod_f000-f002_validate .........   Passed    1.44 sec
47/60 Test NOAA-EMC#55: test_C48_S2SW-gfs_wavepostbndpntbll_execute ............   Passed  480.86 sec
      Start 56: test_C48_S2SW-gfs_wavepostbndpntbll_validate
48/60 Test NOAA-EMC#56: test_C48_S2SW-gfs_wavepostbndpntbll_validate ...........   Passed    1.51 sec
49/60 Test NOAA-EMC#59: test_C48_S2SWA_gefs-gefs_fcst_mem001_seg0_execute ......   Passed  600.94 sec
      Start 60: test_C48_S2SWA_gefs-gefs_fcst_mem001_seg0_validate
50/60 Test NOAA-EMC#60: test_C48_S2SWA_gefs-gefs_fcst_mem001_seg0_validate .....   Passed    1.54 sec
51/60 Test  NOAA-EMC#7: test_C48_ATM-gfs_fcst_seg0_execute .....................   Passed  900.98 sec
      Start  8: test_C48_ATM-gfs_fcst_seg0_validate
52/60 Test  NOAA-EMC#8: test_C48_ATM-gfs_fcst_seg0_validate ....................   Passed    1.89 sec
53/60 Test NOAA-EMC#27: test_C48_S2SW-gfs_fcst_seg0_execute ....................   Passed  961.07 sec
      Start 28: test_C48_S2SW-gfs_fcst_seg0_validate
54/60 Test NOAA-EMC#28: test_C48_S2SW-gfs_fcst_seg0_validate ...................   Passed    2.49 sec
55/60 Test NOAA-EMC#18: test_C48_ATM-gfs_genesis_stage .........................   Passed  2022.51 sec
      Start 19: test_C48_ATM-gfs_genesis_execute
56/60 Test NOAA-EMC#14: test_C48_ATM-gfs_tracker_stage .........................   Passed  2022.52 sec
      Start 15: test_C48_ATM-gfs_tracker_execute
57/60 Test NOAA-EMC#15: test_C48_ATM-gfs_tracker_execute .......................   Passed  120.81 sec
      Start 16: test_C48_ATM-gfs_tracker_validate
58/60 Test NOAA-EMC#16: test_C48_ATM-gfs_tracker_validate ......................   Passed    1.65 sec
59/60 Test NOAA-EMC#19: test_C48_ATM-gfs_genesis_execute .......................   Passed  481.00 sec
      Start 20: test_C48_ATM-gfs_genesis_validate
60/60 Test NOAA-EMC#20: test_C48_ATM-gfs_genesis_validate ......................   Passed    1.48 sec

100% tests passed, 0 tests failed out of 60

Label Time Summary:
C48_ATM-gfs_atmos_prod_f000-f002          = 307.56 sec*proc (4 tests)
C48_ATM-gfs_fcst_seg0                     = 909.35 sec*proc (4 tests)
C48_ATM-gfs_genesis                       = 2508.15 sec*proc (4 tests)
C48_ATM-gfs_stage_ic                      =  67.11 sec*proc (4 tests)
C48_ATM-gfs_tracker                       = 2148.13 sec*proc (4 tests)
C48_S2SW-gfs_fcst_seg0                    = 972.42 sec*proc (4 tests)
C48_S2SW-gfs_ice_prod_f006                =  67.11 sec*proc (4 tests)
C48_S2SW-gfs_ocean_prod_f006              =  67.12 sec*proc (4 tests)
C48_S2SW-gfs_stage_ic                     =  67.13 sec*proc (4 tests)
C48_S2SW-gfs_waveinit                     =  67.13 sec*proc (4 tests)
C48_S2SW-gfs_wavepostbndpnt               = 198.05 sec*proc (4 tests)
C48_S2SW-gfs_wavepostbndpntbll            = 498.22 sec*proc (4 tests)
C48_S2SW-gfs_wavepostgridded_f000-f002    = 131.14 sec*proc (4 tests)
C48_S2SW-gfs_wavepostpnt                  = 258.03 sec*proc (4 tests)
C48_S2SWA_gefs-gefs_fcst_mem001_seg0      = 615.40 sec*proc (4 tests)

Total Test time (real) = 2508.19 sec
Terry.McGuinness (gaea64) build (feature/ci-hist_init) $ 
```

---------

Co-authored-by: Anton Fernando <Anton.Fernando@noaa.gov>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants