Fetch and analyze

New to usdata? Start with getting started. This guide covers the everyday fetch loop; pinning inputs for later is in pin inputs.

Fetch near a place, dry run first

usdata fetch noaa:ghcn-daily --lat 35.39 --lon -97.60 --radius-km 15 \
  --start 2024-05-06 --end 2024-05-07 --vars PRCP,TMAX --dry-run

--dry-run contacts the provider to list what the query resolves to, with sizes where the service reports them, and downloads nothing. Remove it to fetch. Pass shared options through their own flags, --start, --end, --location, --bbox, --vars, and provider-specific selectors through -p key=value, for example -p stations=USW00013967. For exact stations, use the dataset's id option rather than geographic discovery; a place is a rectangle, not a boundary.

Times are UTC

Timestamps without a timezone are treated as UTC, including queries built directly in Python; explicit offsets are converted to UTC. Both bounds are inclusive. A date alone as the end means the last instant of that UTC day, in any ISO 8601 spelling (2024-05-07, 20240507, or 2024-W19-2), so a window from 2024-05-07 to 2024-05-07 is that whole day; give the end a time to stop earlier. See time and place.

The same query from Python

from usdata import build_query, fetch, get

query = build_query(
    lat=35.39,
    lon=-97.60,
    radius_km=15,
    start="2024-05-06",
    end="2024-05-07",
    variables=["PRCP", "TMAX"],
)
for item in fetch(get("noaa:ghcn-daily"), query):
    print(item.path, item.provenance.checksum)
    frame = item.open()

fetch returns one item per asset with its cached path and provenance. item.provenance.source_url records the URL the bytes were actually fetched from and item.asset.href records where the listing pointed, which are equal today but need not stay equal. item.inspect(), or usdata inspect <path>, summarizes what a fetched file holds before you open it: CSV columns and row count with no extra installed, NetCDF4 variables or GRIB2 messages with the format's extra. open() is a separate local step that needs the reader extra for the format; without it you still have the path. What each reader returns is in readers, and the options in the reader reference.

Where files go

Files live under ~/.cache/usdata/ unless USDATA_CACHE_DIR or --cache-dir says otherwise, with a provenance sidecar beside each. A repeated query is a cache hit checked against that sidecar; --force re-downloads and usdata verify re-reads the bytes.

On a terminal, fetch reports resolved counts, known sizes, download progress, and verified cache hits on stderr. --no-progress disables it, and redirecting output disables it automatically.

Manage the cache

usdata cache shows and reclaims that space:

usdata cache path                       # where the cache is
usdata cache list --dataset noaa:ghcn-daily
usdata cache size
usdata cache prune --older-than 30d --dry-run

list prints one line per cached file with its dataset, asset id, size, and the retrieval time from its sidecar; a file with no readable sidecar is listed as unrecorded. --json emits the same records for scripts.

prune needs --older-than (30d, 12h, or an ISO 8601 duration such as P30D), --dataset, or both, and removes each data file together with its sidecar. Files pinned by a *.lock.json beside a manifest in the current directory are kept and counted in the summary, so pruning never breaks a usdata verify; --include-pinned removes them too. Pair either with --dry-run to see what would go first.

The SST example fetches a small gridded CSV and opens it with units; monthly climate does the same for station summaries.