Manifests and reproducible inputs

A manifest declares the datasets needed by a project. Commit it together with its lockfile. The cache holds the downloaded bytes and should be backed up separately when historical reproducibility matters.

Start with the NOAA and USGS example. Installation and development setup are in the README.

Manifest fields

name: weather-and-streamflow
version: "1.0"
sources:
  - dataset: noaa:ghcn-daily
    start: 2024-05-06
    end: 2024-05-07
    variables: [PRCP, TMAX]
    params:
      stations: USW00013967
      units: metric

Each source accepts:

Unknown manifest and source fields are rejected. params must not repeat location, bbox, start, end, or variables.

Place lookup uses 2025 Census bounding boxes for 56 states/DC/territories and 3,235 counties/equivalents. Use names/postal codes, "Cleveland County, OK", or quoted FIPS such as "40027". Rectangles can include points outside the actual boundary. Alaska and other antimeridian-spanning regions have very wide boxes; prefer explicit station IDs or a local bbox there. See the place reference for aliases, source vintage, and limitations.

Provider options

Plural stations/sites options accept strings or lists of strings; see each row for singular aliases. CO-OPS station requires one seven-digit string and does not accept a list. NOAA radar IDs are case-insensitive. USGS IDs may include the USGS- prefix. Empty explicit lists are invalid even with allow_empty: true; that flag permits an empty result from a valid query.

See NOAA and USGS for source behavior, units, limitations, and endpoint details.

Pull, refresh, and restore

In Python, pass pathlib.Path objects to pull() and verify(), for example pull(Path("dataset.yaml")) after from pathlib import Path. Their manifest-path arguments do not accept strings. CLI paths are ordinary command-line arguments.

usdata pull dataset.yaml --cache-dir .data
usdata verify dataset.yaml --cache-dir .data

The first pull resolves each source, downloads missing assets, and writes dataset.lock.json. Every required source must resolve to at least one asset; otherwise pull exits 1 and does not create or replace the lockfile. Files successfully fetched before the failure remain cached for the next attempt. This checks asset presence, not completeness of all scientific observations inside a returned file.

With a lockfile present, pull restores the pinned assets without re-querying listings. Cached bytes must match their pinned checksums; missing or altered files are downloaded again. If upstream now returns different bytes, restoration fails with a checksum mismatch and preserves any existing file at that path.

If the manifest changes, both pull and verify refuse it with exit code 2:

usdata pull dataset.yaml --cache-dir .data --force

For pull, --force re-resolves queries and replaces the lockfile after all required sources succeed. Valid cached responses may still be reused. For fetch, --force means re-download even a valid cached file. These flags have different purposes; re-locking is not a guarantee of upstream freshness.

The manifest checksum covers its exact bytes. Editing whitespace or comments also requires re-locking. An intentionally empty optional source is pinned as no assets; restore does not search for newly available results. Use pull --force to resolve it again.

What verification guarantees

verify is offline. It checks that the manifest matches the lockfile, then hashes the cached files. It exits 1 if a file is missing or has changed, 2 for an invalid/mismatched manifest or unreadable lockfile, and 0 when checks pass. It does not query upstream or validate the scientific meaning of the data. An older empty lockfile cannot prove that all original queries were satisfied; re-resolve it using the current empty-source policy.

Lockfiles contain the manifest checksum, generation time, usdata version, resolved asset URLs, file checksums, and provenance. Sidecars record source URL, provider, dataset, retrieval time, byte count, checksum, and license.

A lockfile detects changed data; it does not archive data. Query-based APIs may revise observations or page ordering and may not expose historical versions. If both the cached bytes and their upstream version are gone, usdata cannot recreate them. Preserve the cache alongside the manifest and lockfile for long-lived work. The cache stores one current file per asset ID; it is not a versioned archive.

Failures and retries

GET requests retry transient connection, timeout, and remote protocol failures, as well as HTTP 429, 500, 502, 503, and 504, for at most three attempts. Default backoff is 0.5 then 1 second. Retry-After seconds and HTTP dates are respected; a requested wait over 30 seconds is surfaced as an error instead of retried early. Interrupted downloads restart from the beginning in a temporary file. Permanent HTTP errors, local filesystem failures, and checksum mismatches are not retried.

CLI exit codes: 0 success; 1 no assets or verification drift; 2 invalid input; 3 unimplemented dataset; 4 upstream or checksum failure. Offline tests prohibit network connections; live tests exercise upstream services separately.

Resolve, restore, and verify

flowchart TD
    Manifest[Manifest] --> Pull[pull]
    Pull --> Locked{Lockfile exists and force not requested?}
    Locked -->|No| Resolve[Resolve sources through adapters]
    Resolve --> Fetch[Fetch assets and record provenance]
    Fetch --> Lock[Write checksummed lockfile]
    Locked -->|Yes| Match{Manifest checksum matches?}
    Match -->|No| Error[Error: use force to resolve changed inputs]
    Match -->|Yes| Restore[Restore the pinned assets]
    Restore --> Cache[Verify cache hits or download pinned URLs]
    Lock --> Verify[verify]
    Cache --> Verify
    Verify --> Report[Check manifest checksum and cached bytes]

Locked restoration avoids repeating discovery. Verification checks local integrity; neither operation can recreate upstream bytes that are no longer available. Preserve cached inputs for long-lived reproducibility.