Download, parse, and aggregate CW3E SurfaceMetObs hourly station files into a single time-indexed dataset.
This tool automates retrieval of surface meteorological data hosted by the Center for Western Weather and Water Extremes (CW3E) at UC San Diego/SIO. It supports auto‑discovering per‑station schemas by reading each station’s DataFormat.txt, robustly handles missing files, and exports combined results to CSV and Parquet.
- Auto‑schema: Reads
<SITE>/DataFormat.txtto build column names and missing‑value tokens per station (fallback schema provided if missing). - Flexible ranges: Fetches a continuous range from start year/Julian day to end year/Julian day (inclusive), across years with leap‑year handling.
- 404‑tolerant: Quietly skips missing hours or transient network errors.
- Memory‑only mode: Optionally does not save raw hourly files (parse directly from the HTTP response).
- Clean timeseries: Converts
Year + Julian_Day + HHMM(end of averaging) to a properDatetimeIndex. - Exports: Writes one combined dataset to CSV and Parquet.
- Python 3.9+
- Packages:
pip install pandas requests pyarrow
pyarrow(orfastparquet) is required for Parquet output.
- Remote path pattern:
Example:
https://cw3e-datashare.ucsd.edu/CW3E_SurfaceMetObs/{SITE_UPPER}/{YYYY}/{JJJ}/{site_lower}{YY}{JJJ}.{HH}mhttps://cw3e-datashare.ucsd.edu/CW3E_SurfaceMetObs/SIO/2026/001/sio26001.00m - Station metadata/schemas:
https://cw3e-datashare.ucsd.edu/CW3E_SurfaceMetObs/{SITE_UPPER}/DataFormat.txt
Assuming the script filename is cw3e_surfacemet_download.py:
python cw3e_surfacemet_download.py \
--site_name sio \
--start_year 2026 --start_jday 1 \
--end_year 2026 --end_jday 2 \
--out_folder downloadsOutputs:
downloads/sio_2026j001_to_2026j002.csvdownloads/sio_2026j001_to_2026j002.parquet
The script tries to fetch and parse DataFormat.txt from the site folder to:
- build column names
If a site lacks DataFormat.txt or it’s unparseable, the script will exit with an error
python cw3e_surfacemet_download.py --site_name sio --print_schema--site_name SITE Station code (e.g., "sio"). Case‑insensitive.
--start_year YYYY Start year (e.g., 2026).
--start_jday DDD Start Julian day (1..365/366).
--end_year YYYY End year (>= start_year).
--end_jday DDD End Julian day.
--out_folder PATH Folder for raw files and CSV/Parquet (default: ./downloads).
--timeout SECONDS HTTP timeout (default: 30).
--delete_unparsed If saving raw files, delete any hourly file that fails to parse.
--no_save_raw Do not save raw hourly files (parse directly from memory).
--dataformat_url URL Override the DataFormat.txt URL (advanced).
--print_schema Print discovered schema & NA tokens, then exit (no download).
-h, --help Show full help and examples.
python cw3e_surfacemet_download.py \
--site_name sio \
--start_year 2026 --start_jday 1 \
--end_year 2026 --end_jday 2 \
--out_folder downloadspython cw3e_surfacemet_download.py \
--site_name sio \
--start_year 2026 --start_jday 1 \
--end_year 2026 --end_jday 2 \
--no_save_rawpython cw3e_surfacemet_download.py \
--site_name sio \
--start_year 2025 --start_jday 365 \
--end_year 2026 --end_jday 2python cw3e_surfacemet_download.py --site_name sio --print_schema- CSV:
<site>_<startyear>j<startday>_to_<endyear>j<endday>.csv(includestimecolumn) - Parquet: same tag, DateTimeIndex preserved
The DataFrame columns include Datalogger_ID plus the discovered data variables for that site. The time index reflects the end time of averaging.
- The downloader silently skips missing hours (HTTP 404) and transient network errors.
- Only concatenates hourly tables if their column layouts are identical; otherwise, the combined output is omitted.
- Inspect downloaded data with the interactive plotter described below.
- For large ranges, consider parallelizing by day/hour (e.g.,
concurrent.futures). - If your environment lacks
pyarrow, comment out Parquet writing or installfastparquet.
Install the plotting dependency with pip install plotly or update the Conda
environment using conda env update -f environment.yaml.
Create an offline HTML plot and open it in your browser:
python cw3e_surfacemet_plot.py downloads/sio_2026j001_to_2026j002.parquet --showCSV inputs work too. By default, every numeric variable except Datalogger_ID
gets its own panel, except wind speed and maximum wind speed/gusts, which share
one panel with separate colored traces. Zooming or panning the time axis updates all panels. Scroll
to zoom, use the toolbar to switch between zoom and pan, and double-click to
reset the view. Hover displays timestamps and readings. The HTML embeds Plotly,
so it works offline and can be shared without a Python installation.
Discover the exact variable names for your station:
python cw3e_surfacemet_plot.py downloads/sio_2026j001_to_2026j002.csv --list_columnsSelect variables, show sample markers, and set sensor limits (replace these example names and limits with those appropriate for your station):
python cw3e_surfacemet_plot.py downloads/sio_2026j001_to_2026j002.csv \
--columns Air_Temperature Relative_Humidity --markers \
--expected_interval 1min --limit Air_Temperature:-20:50 \
--limit Relative_Humidity:0:100 --output plots/sio_inspection.html --showUse --start "2026-01-01 06:00" and --end "2026-01-01 12:00" to restrict
the plot and checks to an inclusive time range. A date without a time means
midnight. Timestamps retain their input timezone; the tool does not assume UTC
or convert to local time. Axis labels use the exported column names because the
downloader does not preserve unit metadata.
The console summary reports missing, nonnumeric, and infinite values per
variable, duplicate timestamp rows, sampling gaps, and readings outside limits.
Duplicate readings remain in the plot and are marked with orange circles;
out-of-limit readings are marked with red crosses and dashed limit lines.
Missing/nonnumeric/infinite values break lines. Sampling gaps longer than 1.5
times the expected interval also break lines. Without --expected_interval, the
median positive interval in the full input is used; specify the interval for
irregular or sparse data. This detects long intervals, not an exact count of
missing samples.
All original samples are retained, without smoothing, interpolation, or
downsampling. Large files can produce large HTML documents; use date filters
and column selection to reduce their size. Checks describe the displayed range
and do not diagnose sensor faults or automatically remove sentinel values.
No limits are applied unless supplied with --limit.
Wind-speed columns with descriptive names such as Wind_Speed,
Max_Wind_Speed, or Wind_Speed_Max are automatically overlaid on the same
y-axis. Their original names appear in the legend and hover details, and each
keeps its own gap handling and sensor-limit checks. For abbreviated names, use
--wind_speed WS WS_MAX (include both in --columns if selecting variables).
Columns in this panel should use the same units; the tool does not convert units.
Wind-direction columns with descriptive names such as Wind_Direction or
Wind_Dir automatically use arrow markers instead of connecting lines.
For another column name, specify --wind_direction COLUMN (include that column
in --columns if selecting variables). Values must be degrees clockwise from
north, indicating where the wind comes from. Arrows point toward the direction
of travel: a 0° north wind points down/south; a 90° east wind points left/west.
The vertical axis still shows the original direction in degrees, and hover shows
both the source direction and arrow destination. Wind-direction axes default to
0–360° with compass labels at 90° intervals. Invalid readings outside that range
are clipped from the plot but remain in the console checks when using
--limit COLUMN:0:360. Arrows have fixed size and
screen orientation when zooming; their length does not represent wind speed.
No lines connect direction samples, avoiding misleading jumps across 0°/360°.
Missing values produce no arrow; values outside 0–360° are shown as gray diamonds
without wrapping them into valid directions. Use --limit COLUMN:0:360 to also
count invalid directions in the sensor-limit summary.
Issues and pull requests are welcome. If you have stations with unusual DataFormat.txt styles, please share examples so we can improve the parser.
This software is Copyright © 2026 The Regents of the University of California. All Rights Reserved.
Data provided by CW3E (Scripps Institution of Oceanography, UC San Diego). Please cite CW3E appropriately in your work and follow their data policies.