NOAA services and research notes

These notes preserve endpoint research and dated probe results. They are not a list of implemented capabilities; use the NOAA guide and generated catalog for supported datasets.

Access notes

NOAA publishes through several unrelated systems. The ones usdata touches:

• NOAA Open Data Dissemination (NODD): bulk archives in public S3 buckets (also mirrored to Google Cloud and Azure). Anonymous; no account needed. Reached via usdata.protocols.s3 using plain HTTPS, no AWS SDK (ADR 0002). The NEXRAD Level II archive moved from noaa-nexrad-level2 to unidata-nexrad-level2 with the old bucket deprecated September 2025; the old name now returns Access Denied. Check the AWS Open Data registry page for a dataset before assuming a bucket name.
• NCEI Access Data Service (ncei.noaa.gov/access/services/data/v1): REST endpoint returning CSV or JSON for station datasets such as GHCN-Daily. Anonymous. It filters by explicit station ids and date range; it rejects spatial filters for daily-summaries, so a bbox query first goes to the companion search service (/access/services/search/v1/data) to find stations. In search responses count is the number of matches and totalCount is dataset-wide; paginate on count. Enable Python DEBUG logging for usdata.providers.noaa.ghcnd to inspect each search page's request URL, HTTP status, content type, match/result counts, new station count, and up to five station IDs. Pages yielding no new stations include at most 512 response characters. Live tests capture these diagnostics on failure and check geographic discovery separately from explicit-station downloads, so a search outage does not obscure data-service health.
• ERDDAP servers (CoastWatch, PolarWatch, and others): griddap and tabledap URLs with true server-side subsetting. Anonymous. CoastWatch SST uses griddap CSV for noaacwBLENDEDsstDNDaily; see the details below. Use the shared HTTP client: live probes found that the service can reject the default httpx User-Agent with HTTP 403.
• ArcGIS REST feature services: the NWS Damage Assessment Toolkit serves damage points, tracks, and areas from services.dat.noaa.gov/arcgis/rest/services/nws_damageassessmenttoolkit/DamageViewer/FeatureServer. Anonymous. Reached via usdata.protocols.arcgis: SQL where clauses, envelopes, GeoJSON output, pages cut by object-id range, historicMoment reads of the archive, and queryAttachments (ADR 0051). Failures come back as HTTP 200 with an error body; see the dated probes below.
• Other NCEI Access Data Service datasets (GSOM, GSOY, LCD, normals) share the GHCN-Daily client. All require startDate and endDate; the daily normals dataset expects them inside a placeholder year such as 2010. LCD stations use WBAN-based ids (for example 72530094846), not GHCN ids.
• CO-OPS Data API (api.tidesandcurrents.noaa.gov/api/prod/): water levels, predictions, and currents by station id, product, datum, and date range; station metadata from /mdapi/prod/webapi/stations.json. Anonymous.
• More NODD buckets verified anonymous: noaa-goes16/18/19 (ABI and GLM, keys PRODUCT/YYYY/DDD/HH/), noaa-mrms-pds (CONUS/PRODUCT/YYYYMMDD/), noaa-hrrr-bdp-pds, noaa-gfs-bdp-pds, noaa-nbm-grib2-pds (GRIB2 model output by cycle and forecast hour), noaa-cdr-*-pds (climate data records as daily NetCDF), and unidata-nexrad-level3 (flat keys SITE_PRODUCT_YYYY_MM_DD_HH_MM_SS, site id without the leading K).
• NCEI HTTPS directories: plain Apache-style listings for bulk archives (Storm Events, IBTrACS, ERSST, OISST mirror, OCADS). The nClimDiv path under /pub/data/cirs/climdiv/ redirects to /monitoring-content/data/us/climdiv/monthly/current/.
• THREDDS at ngdc.noaa.gov/thredds/ serves ETOPO 2022 with OPeNDAP; the direct mgg/global/relief/ file paths I tried return 404.
• Not yet located: the bulk access path for GHCN-Hourly. The product page exists; the data directory guesses under /oa/ and /data/ return 404.
• OneStop catalog search and NCEI archive services: not used yet; candidates for a future discover command.

Each registry entry names the product family or service behind it as its system, and the generated NOAA catalog groups the implemented datasets under that heading.

Radar site metadata comes from the NCEI HOMR station list, bundled as src/usdata/data/nexrad_sites.csv and regenerated by scripts/build_nexrad_sites.py. HOMR lists the Norman test and research radars (KCRI, KOUN) as NEXRAD; they are absent from the archive and are marked type TEST so default site selection skips them.

Licensing: NOAA data is a U.S. Government work in the public domain unless a dataset page says otherwise.

Data landscape

What NOAA publishes, by domain, and which registry entries cover it. Domains with no entry yet are recorded here so they are not forgotten; an entry is added once a concrete, anonymously accessible dataset has been verified.

Domain	What NOAA provides	Example products	Registry entries
Surface weather	Temperature, precipitation, wind, humidity, pressure, snowfall, station observations	GHCN-Daily, GHCN-Hourly, Local Climatological Data, GSOM/GSOY	ghcn-daily, gsom, gsoy, lcd, ghcn-hourly
Severe weather	Tornadoes, hail, damaging wind, storm events, damage reports	Storm Events Database, Storm Data, Damage Assessment Toolkit	storm-events, spc-tornado-reports, nws-vtec-events, nws-damage-surveys, nws-damage-photos
Weather radar	Reflectivity, radial velocity, dual-pol variables, derived products	NEXRAD Level II, NEXRAD Level III, MRMS	nexrad-level2, nexrad-level3, mrms
Weather satellites	Visible/IR imagery, clouds, lightning, fire, volcanic ash	GOES-R ABI, GOES GLM, POES, JPSS	goes-abi, goes-glm
Tropical cyclones	Best tracks, intensity, pressure, wind radii	HURDAT2, IBTrACS, HURSAT	hurdat2, ibtracs
Weather models	Forecasts, analyses, reanalyses	GFS, HRRR, RAP, NAM, GEFS, National Blend of Models	hrrr, gfs, rap, nbm
Climate	Normals, long-term records, divisional averages, indices	Climate Normals, nClimDiv, Climate Data Records	climate-normals, nclimdiv
Snow and ice	Snow cover and depth, sea ice concentration and extent	Sea Ice Index (NOAA@NSIDC), IMS snow cover	sea-ice-index
Ocean physics	SST, salinity, currents, waves, profiles	OISST, ERSST, World Ocean Database	oisst, ersst
Sea level and tides	Water levels, tides, currents, sea-level trends	CO-OPS NWLON, PORTS	coops-water-levels
Ocean chemistry	Carbon, oxygen, nutrients, pH	OCADS, SOCAT	ocads
Marine life and fisheries	Surveys, stock assessments, habitats, telemetry	NOAA Fisheries surveys, InPort	none yet: access is per-science-center, mostly ERDDAP and InPort
Satellite oceanography	SST, ocean color, winds, sea-surface height	CoastWatch, Coral Reef Watch, AVHRR Pathfinder	coastwatch-sst
Bathymetry and hydrography	Seafloor elevation, soundings, coastal relief	ETOPO 2022, Coastal Relief Model, hydrographic surveys	etopo
Coastal	Shorelines, coastal land cover, elevation, flooding	Digital Coast, C-CAP	none yet: Digital Coast is ArcGIS-served
Geophysics	Geomagnetism, marine geophysics	INTERMAGNET, World Magnetic Model	none yet
Natural hazards	Tsunamis, volcanic events, significant earthquakes	NCEI Global Historical Tsunami Database (HazEL)	tsunami-events
Paleoclimate	Tree rings, ice cores, sediments, corals	World Data Service for Paleoclimatology	paleo-search
Space weather	Solar activity, solar wind, geomagnetic indices	SWPC real-time products, DSCOVR, GOES SUVI	swpc-realtime
Land and environment	Vegetation, surface temperature, fire	Terrestrial Climate Data Records (NDVI, LAI)	cdr-ndvi

NWS Damage Assessment Toolkit

Probes on 2026-09-29 against the DamageViewer FeatureServer, recorded in full in the surveys and photos guides. The findings that shaped ADR 0051:

• Three layers, 240,169 points, 12,965 lines, 11,152 polygons; each serves at most 2000 features per request (maxRecordCount), flags a capped page with exceededTransferLimit in Esri JSON and GeoJSON, and lists every matching object id in one answer (1.8 MB for all points).
• Capabilities Query,ChangeTracking, and an archive from 25 November 2020: historicMoment before it answers a count of 0, after it the layer as it stood. Repeated GeoJSON requests returned identical bytes, with an ETag equal to the body's SHA-256.
• queryAttachments stops at 2000 attachments and does not flag it.
• TIMESTAMP 'YYYY-MM-DD HH:MM:SS.fff' literals compare to the millisecond, in UTC. The layers' advertised timeExtent (second century to 2029) is stale; statistics give 1855-05-22 to the present.
• The Experience Builder viewer at apps.dat.noaa.gov/StormDamage/DamageViewer/ states no terms; the NWS disclaimer applies.

B='https://services.dat.noaa.gov/arcgis/rest/services/nws_damageassessmenttoolkit/DamageViewer/FeatureServer'
curl "$B?f=json"                                                   # 3 layers, capabilities Query,ChangeTracking
curl -G "$B/0/query" --data-urlencode 'where=1=1' -d returnIdsOnly=true -d f=json | wc -c   # 1873070

MRMS gridded radar products

Bounded probes on 2026-09-14 listed every supported product's directory for 2024-05-06 in noaa-mrms-pds: 720 two-minute files for most products (719 for MergedBaseReflectivityQC_00.50 and PrecipRate_00.00, 48 hourly files for the 2-hour rotation tracks), with the per-file and per-day sizes recorded in the dataset guide. Rotation tracks stamp even minutes exactly (-200000); most other products carry the merge second (-200039). Listing CONUS/<PRODUCT>/ with a / delimiter found 20201014 as the first day for rotation tracks, composite reflectivity, and MESH. The bucket root holds ALASKA/, ANC/, CARIB/, CONUS/, CONUS_5KM/, ConvectProb/, GUAM/, HAWAII/, ProbSevere/, and unsupported/; only CONUS/ is served. A reproducible listing probe is:

curl --get 'https://noaa-mrms-pds.s3.amazonaws.com/' \
  --data-urlencode 'list-type=2' \
  --data-urlencode 'prefix=CONUS/RotationTrackML30min_00.50/20240506/' \
  --data-urlencode 'max-keys=1000'

The 20:00:00 rotation-track file that day was 109,540 bytes gzipped and 189,711 bytes decompressed, one PNG-packed GRIB2 message on a 7,000 × 14,000 regular grid from 55°N 230°E to 20°N 300°E; ecCodes reports its parameter as unknown (discipline 209). The live test downloads that file, checks the gzip and GRIB signatures, and restores it through a lockfile without relisting.

GOES ABI imagery and cloud-top products

Bounded probes on 2026-09-08 listed 192 CMIPC files in one hour for each of GOES-16/18 (2024 day 127), GOES-17 (2022 day 127), and GOES-19 (2025 day 127). A reproducible listing probe is:

curl --get 'https://noaa-goes18.s3.amazonaws.com/' \
  --data-urlencode 'list-type=2' \
  --data-urlencode 'prefix=ABI-L2-CMIPC/2024/127/12/' \
  --data-urlencode 'max-keys=1000'

The small channel-6 filename was OR_ABI-L2-CMIPC-M6C06_G18_s20241271201181_e20241271203560_c20241271204021.nc (255,384 bytes). The live test downloads it, checks the NetCDF4/HDF5 signature, then restores through a lockfile without relisting. Archive revisions still correctly fail restoration if the bytes no longer match the original checksum.

Mesoscale probes on 2026-09-22 followed both listing pages under ABI-L2-CMIPM/2024/127/22/ in noaa-goes16. This hour contains 120 channel-13 files, split between M1 and M2; the shared directory does not select a sector. A reproducible first-page probe is:

curl --get 'https://noaa-goes16.s3.amazonaws.com/' \
  --data-urlencode 'list-type=2' \
  --data-urlencode 'prefix=ABI-L2-CMIPM/2024/127/22/' \
  --data-urlencode 'max-keys=1000'

Follow NextContinuationToken with continuation-token for the remaining objects. The first M1 channel-13 scene is OR_ABI-L2-CMIPM1-M6C13_G16_s20241272200280_e20241272200349_c20241272200404.nc, 329,131 bytes, with a 500 × 500 grid. Its reported geographic bounds are 110.37°W–91.05°W, 29.72°N–43.40°N. The fifteen-minute example pins 15 scenes (4,925,644 bytes), with one start in every minute and intervals of 57.1–62.9 seconds. Coordinates, projection, and metadata extent agree across these scans; this is measured coverage for this window, not a permanent location for M1. The live mesoscale test fetches the first scene and restores it into an empty cache, comparing exact bytes.

Product probes on 2026-09-29 listed the top level of each bucket with a delimiter:

curl --get 'https://noaa-goes16.s3.amazonaws.com/' \
  --data-urlencode 'list-type=2' \
  --data-urlencode 'delimiter=/'

All four buckets hold the same ABI directories for these products: CMIPC/F/M, MCMIPC/F/M, ACHAC/F/M, ACHTF/M (no CONUS), CTPC/F (no mesoscale), and ACTPC/F/M, matching the sector column of NCEI's product type table. ACHA2KMC/F/M exists in noaa-goes16, 18, and 19 but not in noaa-goes17. Listing PRODUCT/2024/127/22/ in noaa-goes16 for each product found one filename shape throughout, with the sector code after the product and a channel only in CMIP names:

OR_ABI-L2-CMIPF-M6C01_G16_s20241272200205_e20241272209513_c20241272209574.nc
OR_ABI-L2-MCMIPC-M6_G16_s20241272201173_e20241272203551_c20241272204067.nc
OR_ABI-L2-MCMIPM1-M6_G16_s20241272200280_e20241272200350_c20241272200421.nc
OR_ABI-L2-ACHAM1-M6_G16_s20241272200280_e20241272200338_c20241272201218.nc
OR_ABI-L2-ACHTF-M6_G16_s20241272200205_e20241272209513_c20241272213066.nc
OR_ABI-L2-CTPC-M6_G16_s20241272201173_e20241272203546_c20241272206329.nc
OR_ABI-L2-ACTPM1-M6_G16_s20241272200280_e20241272200338_c20241272201009.nc

That hour held 12 files per CONUS selection, 6 per full-disk selection, and 60 per mesoscale sector, each directory in one listing page except CMIPM (1,920 files). Each mesoscale directory split evenly between M1 and M2. Sizes per file: CMIPC 2.2–63 MB, CMIPF 9.0–301 MB, CMIPM 0.20–5.0 MB, MCMIPC 55–58 MB, MCMIPF 297–312 MB, MCMIPM 4.3–4.5 MB, ACHAC 0.29 MB, ACHAF 1.5 MB, ACHAM 0.14 MB, ACHTF 28 MB, ACHTM 0.30–0.33 MB, CTPC 0.29 MB, CTPF 1.6 MB, ACTPC 0.54–0.59 MB, ACTPF 3.0–3.1 MB, ACTPM 0.09–0.10 MB. Night MCMIPM scenes were smaller, 3.0–3.8 MB in the 06:00 and 10:00 UTC hours. A reproducible probe is:

curl --get 'https://noaa-goes16.s3.amazonaws.com/' \
  --data-urlencode 'list-type=2' \
  --data-urlencode 'prefix=ABI-L2-MCMIPM/2024/127/22/' \
  --data-urlencode 'max-keys=1000'

The earliest scene per product and bucket came from listing PRODUCT/ with a delimiter for years, then the first non-2000 year for days, the first day for hours, and the first hour with max-keys=1. GOES-16 holds each product's earliest scene:

Directories	GOES-16	GOES-17	GOES-18	GOES-19
CMIP, MCMIP	2017 day 059 (Feb 28)	2018 day 240	2022 day 131	2024 days 284–292
ACHAC/F/M, ACTPM	2019 day 336 (Dec 2)	2019 day 336	2022 day 131	2024 day 289
ACHTF/M, CTPC/F	2019 day 339 (Dec 5)	2019 day 339	2022 day 131	2024 day 289
ACTPC/F	2017 day 136 (May 16)	2018 day 239	2022 day 131	2024 day 289
ACHA2KMC/F/M	2023 day 082	none	2023 day 086	2024 day 289

The first ACTPC scene, OR_ABI-L2-ACTPC-M4_G16_s20171360000221_e20171360000221_c20171360006337.nc, is Mode 4 with equal start and end stamps. Only noaa-goes16 has a 2000/ year of placeholder scenes, under the six CMIP and MCMIP directories and ACTPC and ACTPF; ACTPM and the height, temperature, and pressure directories have none. GOES-16's CMIPC and ACHAC listings end at 2025 day 097. Year folders per product in noaa-goes16: ACTPC has 212 days in 2017 and every day of 2018 and 2019; ACHAC has 30 days in 2019; MCMIPC has 293 days in 2017.

Opening one file of each product with the netcdf extra gave these main variables and spatial_resolution attributes: MCMIP CMI_C01–CMI_C16 and DQF_C01–DQF_C16 on a 2 km grid (500 × 500 mesoscale, 1500 × 2500 CONUS), units 1 for channels 1–6 and K for 7–16; ACHA HT (m) at 10 km CONUS and full disk, 4 km mesoscale (250 × 250); ACHT TEMP (K) at 2 km; CTP PRES (hPa) at 10 km; ACTP Phase at 2 km with flag_values 0–5 for clear_sky liquid_water super_cooled_liquid_water mixed_phase ice unknown. Every cloud-top file also has DQF. The M1 MCMIP scene above has CMI_C13 and DQF_C13 identical to CMI and DQF in OR_ABI-L2-CMIPM1-M6C13_G16_s20241272200280_e20241272200349_c20241272200404.nc, the single-channel file of the same scan. The live tests fetch that MCMIP scene (4,505,610 bytes) and the same scan's ACTPM file (86,847 bytes), restore both into an empty cache, and compare channel 13 between the two imagery products.

The GOES-R product pages (www.goes-r.gov/products/overview.html) state on 2026-09-29 that the site retires on 2026-09-30 and moves to nesdis.noaa.gov, so the dataset guide cites NCEI and the files instead.

GOES GLM lightning detections

Bounded probes on 2026-09-14 listed 180 GLM-L2-LCFA files in one hour for GOES-16 (2024 day 127, hour 20), between 270,140 and 394,471 bytes each, and confirmed GOES-19 files under the same layout (2025 day 127). A reproducible listing probe is:

curl --get 'https://noaa-goes16.s3.amazonaws.com/' \
  --data-urlencode 'list-type=2' \
  --data-urlencode 'prefix=GLM-L2-LCFA/2024/127/20/' \
  --data-urlencode 'max-keys=1000'

Filenames follow OR_GLM-L2-LCFA_G16_sYYYYDDDHHMMSSt_eYYYYDDDHHMMSSt_cYYYYDDDHHMMSSt.nc with tenth-of-second stamps; the first file of the hour was OR_GLM-L2-LCFA_G16_s20241272000000_e20241272000200_c20241272000213.nc (288,572 bytes, 194 flashes, 2,725 groups, 5,692 events). Day-by-day listing of GLM-L2-LCFA/2018/ found the earliest GOES-16 file on day 044 at 16:10 UTC and nothing earlier. The live test downloads that first 2024 file, checks the NetCDF4/HDF5 signature, restores through a lockfile without relisting, and opens the flash table with the netcdf extra.

HRRR model output

Bounded listings on 2026-09-14 of hrrr.20240506/conus/ in noaa-hrrr-bdp-pds found, for every cycle, the wrfsfcf, wrfprsf, wrfnatf, and wrfsubhf GRIB2 files with .grib2.idx sidecars, plus BUFR sounding archives. The 00, 06, 12, and 18 UTC runs carry forecast hours 00 through 48 (49 files per variant); every other cycle stops at 18. Subhourly files stop at 18 in all cycles. A reproducible listing probe is:

curl --get 'https://noaa-hrrr-bdp-pds.s3.amazonaws.com/' \
  --data-urlencode 'list-type=2' \
  --data-urlencode 'prefix=hrrr.20240506/conus/hrrr.t20z.wrfsfcf'

Sizes for the 20 UTC run on 2024-05-06: wrfsfcf00 150,114,757 bytes and wrfsfcf01 158,293,431; wrfprsf00 408,899,120; wrfnatf00 705,956,816; wrfsubhf01 228,194,453. The surface analysis holds 170 GRIB2 messages on a 1799 × 1059 Lambert conformal grid. The bucket's day prefixes begin at hrrr.20140730/, whose first CONUS run is 18 UTC; alaska is the only other domain. The live test resolves and downloads hrrr.t20z.wrfsfcf00.grib2 for 2024-05-06, restores it through a lockfile, and opens surface CAPE when the grib extra is installed.

GFS model output

Bounded listings on 2026-09-14 of gfs.20240506/ in noaa-gfs-bdp-pds found the four cycles 00, 06, 12, and 18, each with atmos/ and wave/ prefixes. Under atmos/, the gfs.tHHz.pgrb2.<resolution>.fNNN files exist for 0p25, 0p50, and 1p00, beside pgrb2b (the remaining fields), pgrb2full.0p50, .anl analysis files, .grib2.idx-style .idx sidecars, and BUFR archives. The 0.25 degree files run hourly from f000 to f120 and every three hours to f384 (209 files); the 0.5 and 1 degree files run every three hours from f000 to f384 (129 files), in every cycle probed. A reproducible listing probe is:

curl --get 'https://noaa-gfs-bdp-pds.s3.amazonaws.com/' \
  --data-urlencode 'list-type=2' \
  --data-urlencode 'prefix=gfs.20240506/00/atmos/gfs.t00z.pgrb2.1p00.f'

Sizes for the 00 UTC run on 2024-05-06: pgrb2.1p00.f000 42,362,644 bytes and f003 44,925,309; pgrb2.0p50.f000 150,321,638; pgrb2.0p25.f000 507,547,200 and f120 547,490,924. The 1 degree analysis holds 696 GRIB2 messages on a 360 × 181 regular grid, including surface CAPE and CIN, 0–3 km storm-relative helicity, and mixed-layer CAPE on pressureFromGroundLayer levels. Day prefixes begin at gfs.20210101/, but through the 06 UTC run of 2021-03-22 the cycles hold only WAFS files; the atmos/ layout with pgrb2 files begins with the 12 UTC run of 2021-03-22 (GFS v16). The live test resolves and downloads gfs.t00z.pgrb2.1p00.f000 for 2024-05-06, restores it through a lockfile, and opens surface CAPE when the grib extra is installed.

RAP model output

Bounded listings on 2026-09-16 of rap.20240506/ in noaa-rap-pds found, for every cycle, the awp130pgrb, awp130bgrb, wrfprs, wrfnat, wrfmsl, awp252pgrb, awp252bgrb, awp236pgrb, awp242, awp243, awp200, and awip32 GRIB2 families with .grib2.idx sidecars, plus BUFR sounding archives. The 03, 09, 15, and 21 UTC runs carry forecast hours 00 through 51 (52 files per family); every other cycle stops at 21. A reproducible listing probe is:

curl --get 'https://noaa-rap-pds.s3.amazonaws.com/' \
  --data-urlencode 'list-type=2' \
  --data-urlencode 'prefix=rap.20240506/rap.t20z.awp130pgrbf'

Sizes for the 20 UTC run on 2024-05-06: awp130pgrbf00 18,368,480 bytes; awp130bgrbf00 41,103,749; wrfprsf00 228,177,374; wrfnatf00 305,421,172. The 13 km analysis holds 355 GRIB2 messages on a 451 × 337 Lambert conformal grid, including surface CAPE and CIN, 0–1 km and 0–3 km storm-relative helicity, and mixed-layer and most-unstable CAPE on mb above ground layers. The bucket's day prefixes begin at narre.20201201/; the first rap. prefix is rap.20210222/, whose 00 UTC run holds awp130pgrbf00. The live test lists two hours with sizes, fetches surface CAPE and 0–3 km helicity as byte ranges, restores them through a lockfile, and opens them when the grib extra is installed.

NBM forecast guidance

Bounded listings on 2026-09-16 of blend.20240506/ in noaa-nbm-grib2-pds found every hourly cycle with core/, qmd/, and text/ prefixes. Under core/, blend.tHHz.core.fFFF.<region>.grib2 files exist for the co, ak, hi, pr, and gu regions with .grib2.idx sidecars, beginning at f001. The CONUS files of the 00 and 12 UTC runs are hourly to 36 and three-hourly to 192 (88 files); the 06 and 18 UTC runs add six-hourly files to 264 (100 files); the 01 and 13 UTC runs are hourly to 173 and the 07 and 19 UTC runs hourly to 152; the remaining cycles are hourly to 36 or 37, three-hourly to about 190, and six-hourly to about 262. A reproducible listing probe is:

curl --get 'https://noaa-nbm-grib2-pds.s3.amazonaws.com/' \
  --data-urlencode 'list-type=2' \
  --data-urlencode 'prefix=blend.20240506/20/core/blend.t20z.core.f001.'

Sizes for the 20 UTC run on 2024-05-06, forecast hour 1: co 171,059,531 bytes, ak 52,257,699, hi 6,110,448, gu 633,902. The CONUS file holds 300 GRIB2 messages on a 2345 × 1597 Lambert conformal grid at 2.54 km; index lines carry a fourth field for ensemble standard deviations and probability thresholds. Day prefixes begin at blend.20200518/ in a grib2/ layout of master files; the core/ layout begins with the 12 UTC run of 2020-09-29. The live test lists two hours with sizes, fetches the 2 m temperature message as a byte range, restores it through a lockfile, and opens it when the grib extra is installed.

Storm Events annual details

Bounded live probes verified on 2026-09-09 UTC (1950 archive: 10,508 bytes):

curl --fail 'https://www.ncei.noaa.gov/pub/data/swdi/stormevents/csvfiles/'
curl --fail --output /tmp/storm-1950.csv.gz \
  'https://www.ncei.noaa.gov/pub/data/swdi/stormevents/csvfiles/StormEvents_details-ftp_v1.0_d1950_c20260323.csv.gz'

The filename is a probe snapshot; the adapter discovers current filenames instead of hard-coding it. The design decision records the annual-file and compression contract. The notebook downloads one 2024 archive (~13 MB compressed), then filters locally; neither the test nor example downloads all archive years.

SPC tornado reports

Probes on 2026-09-14 UTC against https://www.spc.noaa.gov/wcm/. The data directory itself returns HTTP 403 and no index, so the adapter discovers files from the page's links instead of a directory listing. All files answered text/csv over anonymous HTTPS:

File	Size	Last-Modified
data/2025_torn.csv	217,624 bytes	2026-04-24
data/2024_torn.csv	230,094 bytes (1,873 rows)	2025-05-13
data/2023_torn.csv	166,684 bytes	2024-07-17
data/2008_torn.csv	193,676 bytes	epoch placeholder
data/2005-2007_torn.csv	3,525 rows	epoch placeholder
data/2000-2004_torn.csv	728,228 bytes	epoch placeholder
data/50-59_torn.csv … 90-99_torn.csv	4,905 to 12,276 rows	epoch placeholder
data/1950-2025_actual_tornadoes.csv	9,023,445 bytes	2026-04-24
data/1950-2025_all_tornadoes.csv	9,219,030 bytes	2026-04-24
data/1950-2025_torn.csv.zip	2,028,149 bytes	2026-04-24

Per-year files exist for 2008 onward; earlier years are only in decade and half-decade files. 2026_torn.csv and any single-year file before 2008 return HTTP 404. The all-years files embed the last year in their name, so 1950-2024_* disappeared when 1950-2025_* was published; per-year and decade names are stable and their bytes change in place (the 2024 file was modified after the 2025 file appeared, and the page prints an update date beside each link). Every file from 1950 through 2024 shares the 29-column header from the format specification; the 2025 file appends edat and etime. In 2024, tz was 3 on every row, sg was 1 on 1,791 rows and 2 on 81, one row carried -9, and mag ranged from -9 to 4. The specification is a scanned PDF, so its column definitions are restated in the dataset guide.

HURDAT2 best tracks

Probes on 2026-09-12 UTC listed the NHC data directory and fetched both current whole-basin files:

curl --fail 'https://www.nhc.noaa.gov/data/hurdat/'
curl --fail --output /tmp/hurdat2-atl.txt \
  'https://www.nhc.noaa.gov/data/hurdat/hurdat2-1851-2025-02272026.txt'
curl --fail --output /tmp/hurdat2-nepac.txt \
  'https://www.nhc.noaa.gov/data/hurdat/hurdat2-nepac-1949-2025-02272026.txt'

The data files are served as text/plain; charset=UTF-8 (the directory index itself is HTML) and the listing keeps past revisions: 41 files on 2026-09-12, the oldest hurdat2-nepac-1949-2016-041317.txt, a revision of 2017-04-13 covering data through the 2016 season. The current Atlantic file was 7,082,381 bytes (2,004 storms, 55,605 track points) and the northeast Pacific file 4,083,231 bytes (1,262 storms, 32,026 track points). Filenames are the probe snapshot; the adapter discovers current names instead of hard-coding them.

Naming is inconsistent in ways worth recording. The Atlantic basin token is usually omitted (hurdat2-1851-2025-02272026.txt) but appears as atl in a few names; the Pacific token is always nepac. Revision dates are MMDDYY in older names and MMDDYYYY in newer ones, never YYYYMMDD, so filenames do not sort chronologically as text; one Pacific name ends in a disambiguating a. Directory sizes are approximate (6.8M), unlike NCEI's exact byte counts, so asset size is left unknown. Every data line in both current files has 21 fields; revisions published before the 2021 season omit the trailing radius of maximum wind.

Archived revisions vary more than the current pair suggests, which is worth knowing because a lockfile pins one. Parsing all 41 listed files on 2026-09-12: 10 of them, all Atlantic, write at least one position east of Greenwich as an unwrapped 0-360 west longitude (358.0W for 2.0E, down to 299.0W for 61.0E), a form the current files no longer use: hurdat2-1851-2024-040425.txt rewrote those same points with E suffixes. Two files carry an upstream typo and cannot be parsed at all: hurdat2-1851-2024-040225.txt line 30687 drops the comma between latitude and longitude, and hurdat2-nepac-1949-2022-042723.txt line 23724 dates a track point C0091018. Each is fixed by the next revision of the same span, which is also the one the adapter selects. The -99 maximum-wind sentinel is documented as a 1967 convention but appears 57 times in the current Atlantic file, all on TD records from 1971 through 1987, and never in the Pacific file. The design decision records the whole-file and reader contract.

IBTrACS global best tracks

Probes on 2026-09-17 UTC listed the NCEI archive directory, its version directories, and the per-format listings, and fetched the smallest subset in each format:

BASE='https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs'
curl --fail "$BASE/"
curl --fail "$BASE/v04r01/access/csv/"
curl --fail --output /tmp/ibtracs-sa.csv "$BASE/v04r01/access/csv/ibtracs.SA.list.v04r01.csv"
curl --fail --output /tmp/ibtracs-sa.nc "$BASE/v04r01/access/netcdf/IBTrACS.SA.v04r01.nc"

The archive root lists three version directories, v03r09, v04r00, and v04r01. From v4 each holds access/csv/, access/netcdf/, and access/shapefile/, and each of those holds one file per subset: ALL, ACTIVE, last3years, since1980, and the basins NA, EP, WP, NI, SI, SP, and SA, named ibtracs.<subset>.list.v04r01.csv and IBTrACS.<subset>.v04r01.nc, beside a serial-number-to-name mapping. v03r09 instead publishes one NetCDF file per storm, so the adapter refuses versions before v4. Listings give exact byte sizes and a minute-precision modified stamp; the stamp matched the Last-Modified header (09:02 GMT) so it is read as UTC. On 2026-09-17 the CSVs ran from 56,442 bytes (SA) through 57,169,545 (NA) to 331,542,080 (ALL), and the NetCDF files from 688,651 (SA) to 23,379,578 (ALL); last3years was 10,426,557 bytes of CSV. The CSV is served as text/csv and the NetCDF as application/x-netcdf; the latter is HDF5-based NetCDF4 with storm by date_time arrays (360 time slots) and a quadrant axis for the wind radii.

Every file carried that morning's build stamp, and v04r01/archive/ held one tarball per format named for the build, ibtracs_v04r01_csv_s18421025_e20260917_c20260917.tar.gz, with no earlier builds beside it: files are rebuilt in place under unchanging names, and superseded bytes are not kept upstream. The change log records twenty-three data additions and corrections between 2024-06-21 and 2026-07-31, most adding one agency's season. The s18421025 in the tarball name and the files' time_coverage_start attribute both give 1842-10-25T03:00 as the archive's first record; the North Atlantic file begins in 1851 and the northeast Pacific in 1876.

The CSV has 174 columns and, like ERDDAP, a units row under the header (degrees_north, kts, mb, nmile; a single space for text columns). A missing value is written as a single space, not an empty field, and the North Atlantic basin and sub-basin code is the literal NA, which pandas reads as missing by default; the reader therefore treats only a space or an empty field as missing. Points are three-hourly (123,264 of the 124,946 consecutive steps in the North Atlantic file), with IFLAG marking each agency's value as original (O), interpolated (P), or otherwise filled. A basin file holds every storm that entered the basin, whole, so the North Atlantic file also carries EP and NI points from crossing storms. TRACK_TYPE separates main tracks from PROVISIONAL and US-PROVISIONAL operational ones: in the last3years file every 2023 and 2024 storm was main, 46 of 115 2025 storms were, and no 2026 storm was. In v04r00 an unnamed storm is NOT_NAMED; v04r01 writes UNNAMED.

Global Summary of the Month

Bounded live probes verified on 2026-09-08 (one airport, one month):

curl --get 'https://www.ncei.noaa.gov/access/services/data/v1' \
  --data-urlencode 'dataset=global-summary-of-the-month' \
  --data-urlencode 'stations=USW00013967' \
  --data-urlencode 'startDate=2024-05-06' --data-urlencode 'endDate=2024-05-07' \
  --data-urlencode 'dataTypes=PRCP,TAVG' --data-urlencode 'units=metric' \
  --data-urlencode 'format=csv' --data-urlencode 'includeStationLocation=1'

curl --get 'https://www.ncei.noaa.gov/access/services/search/v1/data' \
  --data-urlencode 'dataset=global-summary-of-the-month' \
  --data-urlencode 'bbox=35.40,-97.62,35.38,-97.58' \
  --data-urlencode 'startDate=2024-05-01' --data-urlencode 'endDate=2024-05-31' \
  --data-urlencode 'dataTypes=PRCP,TAVG' \
  --data-urlencode 'limit=1' --data-urlencode 'offset=0'

The data probe returned one 2024-05 row (91.8 mm, 21.7 °C); these are observed values, not permanent test expectations. Search returned USW00013967, count=1, and dataset-wide totalCount=127947; repeating with offset=1 returned no results. Integration tests independently check geographic discovery and a manifest's explicit-station download/locked restoration. Upstream revisions can still change CSV bytes and correctly fail a locked restore.

Global Summary of the Year

Bounded hosted probes verified one airport/year and geographic discovery on 2026-09-09. Both a full-year request and a May 6–7 request returned the 2024 annual record. A reproducible direct probe is:

curl --get 'https://www.ncei.noaa.gov/access/services/data/v1' \
  --data-urlencode 'dataset=global-summary-of-the-year' \
  --data-urlencode 'stations=USW00013967' \
  --data-urlencode 'startDate=2024-01-01' --data-urlencode 'endDate=2024-12-31' \
  --data-urlencode 'dataTypes=PRCP,TAVG' --data-urlencode 'units=metric' \
  --data-urlencode 'format=csv' --data-urlencode 'includeStationLocation=1'

The search probe used the same annual bounds, dataTypes=PRCP,TAVG, bbox=35.40,-97.62,35.38,-97.58, limit=1, and offset=0; it found USW00013967. The annual climate example provides a small manifest and local CSV analysis. Live checks independently exercise discovery and checksum-verified restoration.

U.S. Climate Normals

Hourly extension probe (2026-09-21)

The hourly dataset normals-hourly-1991-2020 supports the same anonymous station search and CSV endpoint. A box around Will Rogers World Airport (35.4,-97.62,35.38,-97.58 in NCEI's order) returned USW00013967 when searched over 1991–2020. A one-day query returned 24 MM-DDTHH:MM:SS labels, 00:00 through 23:00, in local standard time. Substituting 2024 for 2020 in the request returned identical bytes. February 28–March 1 returned 48 rows, omitting February 29 as the hourly documentation specifies.

curl --get 'https://www.ncei.noaa.gov/access/services/data/v1' \
  --data-urlencode 'dataset=normals-hourly-1991-2020' \
  --data-urlencode 'stations=USW00013967' \
  --data-urlencode 'startDate=2020-05-06' --data-urlencode 'endDate=2020-05-06' \
  --data-urlencode 'dataTypes=HLY-TEMP-NORMAL' --data-urlencode 'units=metric' \
  --data-urlencode 'format=csv' --data-urlencode 'includeStationLocation=1'

Repeating with units=standard confirmed all 24 Fahrenheit-to-Celsius conversions within rounding precision. At midnight the pair was 61.0 F and 16.1 C; the day's range was 14.2–23.4 C. This probe does not establish conversion correctness for any other hourly variable. The unchanged metric response is tests/fixtures/hourly-normals.csv.

LCD's 72353013967 query for May 6, 2024, restricted to HourlyDryBulbTemperature, returned 60 reports, including 24 routine FM-15 reports at :52. This is why the hourly comparison uses a stated ten-minute nearest-normal tolerance, requests the next midnight as well, and reports invalid values and unmatched observations.

Original period probes (2026-09-11)

Direct probes on 2026-09-11 against the Access Data Service: the search datasets endpoint lists monthly, daily, annual/seasonal, and hourly normals for 1991-2020, 2006-2020, and 1981-2010. Monthly and annual/seasonal requests succeed without dates; daily requests return HTTP 400 without startDate and endDate. The year in those dates is ignored: 2020-02-27 to 2020-03-01 returned 02-27 through 03-01 including 02-29, and a window from 2010-12-30 to 2011-01-02 returned only the header. The default is standard units with leading spaces. units=metric converts MLY-TMAX-NORMAL, MLY-TMIN-NORMAL, MLY-PRCP-NORMAL, DLY-TAVG-NORMAL, and ANN-TAVG-NORMAL, but a 2026-09-11 probe of USW00013967 and USW00014739 found two data types it handles wrongly, so the conversion cannot be assumed for an unprobed code. MLY-TAVG-NORMAL is not converted: it returned the same space-padded Fahrenheit value under both units=metric and units=standard (38.2 for USW00013967 January). Derived temperature quantities are converted as if they were absolute temperatures, applying the full (F - 32) x 5/9 offset to a difference: USW00013967 January returned MLY-DUTR-NORMAL -5.3, MLY-TAVG-STDDEV -16.2, and MLY-TMAX-STDDEV -15.8 under units=metric against 22.4, 2.8, and 3.6 under units=standard, and USW00014739 returned MLY-DUTR-NORMAL -10.2 and MLY-TAVG-STDDEV -15.5 against 13.7 and 4.1. The true January diurnal range at USW00013967 is 22.4 F = 12.4 C, matching the metric MLY-TMAX-NORMAL 9.6 minus MLY-TMIN-NORMAL -2.8. An unknown dataTypes code returned an empty column rather than an error. Station search with dataset=normals-monthly-1991-2020 found USW00013967 with or without dates.

curl --get 'https://www.ncei.noaa.gov/access/services/data/v1' \
  --data-urlencode 'dataset=normals-monthly-1991-2020' \
  --data-urlencode 'stations=USW00013967' \
  --data-urlencode 'startDate=2020-01-01' --data-urlencode 'endDate=2020-12-31' \
  --data-urlencode 'dataTypes=MLY-TMAX-NORMAL,MLY-TMIN-NORMAL,MLY-PRCP-NORMAL' \
  --data-urlencode 'units=metric' --data-urlencode 'format=csv' \
  --data-urlencode 'includeStationLocation=1'

The climate normals example provides a small manifest and local CSV analysis. Live checks exercise discovery, a daily window that spans February 29, and checksum-verified restoration.

CoastWatch SST

A minimal direct probe, also covered by the live integration test:

curl --fail --globoff -A 'usdata (+https://github.com/jakeryderv/usdata)' \
  'https://coastwatch.noaa.gov/erddap/griddap/noaacwBLENDEDsstDNDaily.csv?analysed_sst[(2024-05-06T12:00:00Z)][(30.025):(30.075)][(-80.075):(-80.025)]'

CO-OPS observed water levels

Hosted probes on 2026-09-09 verified three records for station 8518750 over the 2024-05-06T00:00Z–00:12Z interval, changed values for English units and MSL datum, and HTTP 400 for invalid stations/datums. Missing historical data or intervals between observations returned HTTP 200 with the normal CSV header followed by an error message. The adapter validates the CSV in a temporary file before replacing the destination. Malformed, empty, and semantic-error responses raise httpx.DecodingError (CLI exit 4); HTTP failures retain their status. allow_empty does not suppress these fetch errors. Source revisions may invalidate exact locked restoration.

CO-OPS tide predictions

Hosted probes on 2026-09-12 verified the predictions product for station 8727520 (Cedar Key) over 2024-09-25T00:00Z–2024-09-28T00:00Z: 721 six-minute rows with a Date Time, Prediction header, 26 hourly rows for interval=h, and a Date Time, Prediction, Type header with H/L rows for interval=hilo. A year of six-minute predictions returned 87,601 rows, matching NOAA's documented one-year limit for prediction intervals other than high/low. An invalid station returned HTTP 400 with a plain-text message and no CSV header. Observed and predicted six-minute rows for the same request bounds carried identical timestamps, so the two series subtract without alignment.

NEXRAD Level III products

Probes on 2026-09-14 listed the anonymous unidata-nexrad-level3 bucket with ListObjectsV2. Keys are flat, SITE_PRODUCT_YYYY_MM_DD_HH_MM_SS, with the three-character site prefix dropping the ICAO id's first letter. The earliest key in the whole bucket is ABC_DAA_2020_03_30_00_02_41, and every KTLX product probed begins on 2020-03-30 or the day after, so the archive start is a fixed date rather than a rolling window. KTLX carried 100 distinct product codes. Super-resolution reflectivity and velocity (N0B, N0G, and their 1, 2, 3, A, B tilt variants) begin 2022-02-18; the 256-level N0Q and N0U end 2022-09-08; the NTV tornado vortex signature ends 2022-05-24, while NMD mesocyclone detection and NST storm tracking continue to the probe date. One clear-air day of KTLX N0B listed 127 objects with a 243 kB median; NMD listed 323 objects at 150 bytes each (empty detections); EET listed 126 at 1.5 kB and NST 323 at 10 kB. On 2024-05-06 KTLX N0B listed 388 objects across the whole day, but NMD and NST listed 47 each, all after 20:42 UTC, so the feed capture has gaps that the continuous products do not show. A delimiter listing on _ enumerates products and years cheaply; the adapter lists one SITE_PRODUCT_YYYY_MM_DD_ prefix per day instead, which stays under one page.

Local Climatological Data

Hosted probes on 2026-09-12 verified the Access Data Service dataset local-climatological-data. The search service accepted a bbox and returned eleven-digit station ids such as 72353013967 with per-year file names. One day at that station returned 61 rows of 125 columns: 24 FM-15 hourly, 27 FM-16 special, 8 FM-12 synoptic, and 1 SOD daily-summary report, each DATE in local standard time without an offset. dataTypes narrowed the columns to the named fields plus station, date, report type, and source. units=metric returned Celsius temperatures. The ids 13967 and USW00013967 returned HTTP 200 with a header and no rows.

NGS emergency response imagery

Probes on 2026-09-29 UTC against the anonymous noaa-eri-pds bucket in us-east-1, named on the NODD registry page. The top level lists 52 event folders and three objects: an S3 browser page (index.html) and two Parquet files, noaa_eri_pds.parquet (15 MB, a listing) and noaa_eri_stac.parquet (16 MB, STAC items), both modified 2026-09-16. The adapter reads neither: they need a Parquet reader, and the listing is current without them.

curl --get 'https://noaa-eri-pds.s3.amazonaws.com/' \
  --data-urlencode 'list-type=2' --data-urlencode 'delimiter=/'
curl --get 'https://noaa-eri-pds.s3.amazonaws.com/' \
  --data-urlencode 'list-type=2' --data-urlencode 'delimiter=/' \
  --data-urlencode 'prefix=2020_Nashville_Tornado/20200307a_RGB/'
curl --range 0-65535 --output /tmp/eri-head.tif \
  'https://noaa-eri-pds.s3.amazonaws.com/2020_Nashville_Tornado/20200307a_RGB/20200307aC0852700w360900n.tif'

A full listing of every event prefix found 834,087 objects and about 39 TB: 268,231 GeoTIFFs (20.6 TB), 279,879 raw camera JPEGs with as many .geom sidecars (10.2 TB, under <flight>/raw/ since 2020), 511 tar archives (8.2 TB, under <event>/downloads/, the largest 126 GB), and per-flight VRT, tile-index shapefile, SQLite, MRF, FlatGeobuf, checksum, and README files. One Helene flight's README says its VRT and shapefile listed eight tiles the flight does not hold. Events range from 61 GeoTIFFs (Joplin, 2.1 GB) to 39,432 (2023 pre-event surveys); the largest is Helene at 2.7 TB of GeoTIFFs.

Of the 268,229 GeoTIFFs outside raw/ and downloads/, 244,642 are named by a degrees-minutes-seconds corner, 1,609 by a UTM corner (Joplin and Irene, whose folders name zones 15 and 18, and the 2009 Nor'easter, whose folders name none), 41 are flight mosaics named after their folder, and the rest name no position: numbered frames of Wilma, Ernesto, Humberto, Gustav, and Ike; obliques of Arthur, Matthew, and the Louisiana floods; Sally's near-infrared frames; Nicole's ortho-cogs; and one Midwest-flood flight named by row and column (mwflood-28-98.tif).

GeoTIFF headers, read with 64 KiB range requests, showed:

• The name is the tile's north-west corner. Degree tiles extend 0.0001° past it on every side: 20200307aC0852700w360900n.tif spans 85.4501°W to 85.4249°W and 36.1249°N to 36.1501°N, 18,681 pixels square in WGS 84. Tiles are 0.0252° (90") square through 2020 and 0.0127° (45") from 2021. Some names truncate a second: 20240927aC0853044w295830n.tif starts at 85.5126°W, which is 85°30'45" plus the buffer.
• UTM tiles are 2,504 to 2,600.15 m square with offsets from the name of 1.75 to 50 m, varying within a folder (Joplin's 26 May flight is offset 25.6 m in easting and 49.9 m in northing). They have no GeoKey directory, so the file states no coordinate system; may24JPEGtiles_UTMZone15 is the only record of the zone.
• Three tiles from each of the 319 folders with corner names, and a containment check of three more in each of the 315 that can be placed, found one tile size per folder, every header within 1" plus 0.0001° (or 50 m) of its name, and every tile inside the footprint the adapter computes.
• Compression is JPEG in every folder sampled through 2020 (three bands, four with a transparency band in 2017 to 2019), LZW in post-event folders from 2021, and Deflate in the pre-event surveys, all four bands. Helene tiles register their tiepoint to pixel centres (RasterPixelIsPoint).
• The 41 flight mosaics are 5 to 2,034 MB: 36 in WGS 84 at 0.00005° and five in Web Mercator (EPSG:3857) at 2.39 m; the two Web Mercator mosaics checked are BigTIFF.

Flight folder dates are local dates. The raw frames of the Texas flood flight 20250710a are all stamped 00:00 to 01:59 UTC on 11 July, and those of 20250711b and 20250712b run past midnight UTC into the next day; no frame was stamped earlier than its folder's date.

Metadata pages probed the same day: the NODD registry entry and its noaa-eri.yaml ("UpdateFrequency: Manually when needed"; "GeoTIFF & COG Imagery (COG Format available for recent events)"), the NGS emergency response page, storms.ngs.noaa.gov and its per-event pages (ground sample distance 35 cm for Katrina and Joplin, ~15 cm for Nashville, 15 - 30 cm for Helene), and the Nashville InPort record (acquired 2020-03-07 from 2,500 to 5,000 feet with a Trimble Digital Sensor System; horizontal accuracy not assessed, 3 to 5 m expected in flat terrain).