Skip to content

Pin inputs for a paper

A manifest names the queries an analysis depends on. A lockfile pins the exact bytes they produced. With both committed and the cache backed up, anyone can repeat the download and prove the inputs match.

name: weather-and-streamflow
sources:
  - dataset: noaa:ghcn-daily
    start: 2024-05-06
    end: 2024-05-07
    variables: [PRCP, TMAX]
    params:
      stations: USW00013967
  - dataset: usgs:water-daily
    start: 2024-05-06
    end: 2024-05-07
    variables: ["00060"]
    params:
      sites: "07164500"
usdata pull dataset.yaml --cache-dir .data     # resolves, downloads, writes dataset.lock.json
usdata verify dataset.yaml --cache-dir .data   # offline: manifest checksum and every file

The first pull writes the lockfile. Later pulls restore from it without repeating discovery. Verify never touches the network, unless --listing asks it to (below). Pull prints one tab-separated line per asset and a summary line on stderr; --quiet keeps the summary and drops the per-asset lines, and --no-progress turns off the terminal progress display without changing either.

Restore on another machine

Copy the manifest and lockfile, then run the same pull into an empty directory. Missing files are downloaded from their pinned URLs and checked against their pins:

usdata pull dataset.yaml --cache-dir fresh && usdata verify dataset.yaml --cache-dir fresh

Would a re-resolve pick the same files?

A restore never lists again, so it cannot see a file added to an archive after the lockfile was written, or one the listing no longer shows. To ask, compare the lockfile with today's listings:

usdata verify dataset.yaml --listing

Each source is listed exactly as a fresh resolve would list it, select rules included, and nothing is downloaded. A line per difference names the source and says whether an asset was added, removed, or changed: a different href, or a different size where both the listing and the pin give one. Exit 1 means pull --force would pin something else. The bytes themselves are not compared: a restore checks those, and verify without --listing checks the cache. In Python, usdata.compare_listing(manifest) returns the same comparison, with matches true when nothing differs.

What to commit and what to back up

Commit dataset.yaml and dataset.lock.json. Back up the cache directory separately for anything that must be reproducible years later: a checksum proves bytes are unchanged, but cannot recover bytes an agency no longer serves. The archive-backed examples commit their lockfiles and are restored from them every week, so each example page has a lockfile to download alongside its manifest.

When pull exits 4

The agency revised a file behind a pinned URL. Pull restores everything that still matches, lists every asset that changed, and leaves the lockfile alone. Decide per entry:

usdata pull dataset.yaml --cache-dir .data --update noaa:ghcn-daily

That accepts the new bytes for that dataset only and rewrites only its pin. Use --force instead when the manifest itself changed or you want every source re-resolved. To keep the old bytes instead, a mirror that stores files by checksum can serve them: USDATA_MIRROR_URL names one, and the shipped examples restore from the project's at https://data.usdata.dev. The full contract is in provenance and drift.

A dataset that needs a key reads it from environment variables that usdata info names, so a manifest and lockfile never hold one and can be shared as they are. Someone without a key can still restore what the cache or a mirror holds.

Upstream revisions

Agencies revise files in place. A locked restore downloads each pinned URL and fails with a checksum mismatch when the bytes have changed; accept the new bytes for chosen entries with pull --update, or re-resolve the whole manifest with pull --force. A lockfile detects changed data; it does not archive it. Keep the cache with the manifest and lockfile. See provenance and drift.

The weather and streamflow example runs this whole loop, including restoring into an empty cache, and keeps its lockfile with the notebook.