Saved notebook output; this documentation build does not execute cells. Download the notebook.

Reproducible weather and streamflow inputs

Collect two days of NOAA station weather near Oklahoma City and USGS streamflow at the Tulsa site through one manifest. This demonstrates acquisition and provenance across agencies. The sites are different; this notebook does not infer a causal relationship between weather and streamflow.

Saved outputs are a recorded run. Execution and retrieval times below make the snapshot explicit. Requires usdata v0.5 or newer; opening CSVs requires v0.6 or newer. See examples setup, then restart the kernel and run all cells.

from datetime import UTC, datetime
from pathlib import Path

import pandas as pd
from IPython.display import Markdown, display

import usdata
from usdata.pull import pull, verify

manifest = Path("examples/weather-and-streamflow/dataset.yaml")
if not manifest.is_file():
    manifest = Path("dataset.yaml")

pd.set_option("display.max_rows", 8)
pd.set_option("display.max_columns", 8)
print(f"Executed (UTC): {datetime.now(UTC).isoformat(timespec='seconds')}")
print(f"usdata {usdata.__version__}; pandas {pd.__version__}")

Executed (UTC): 2026-09-09T00:11:46+00:00
usdata 0.7.0; pandas 3.0.5

1. Inspect the manifest

The NOAA source requests metric daily precipitation (PRCP) and maximum temperature (TMAX). The USGS source requests discharge (00060) with the daily-mean statistic (00003). Quoted identifiers and codes preserve leading zeros. The USGS response carries its own per-observation units and quality columns.

print(manifest.read_text())

name: weather-and-streamflow
version: "1.0"
sources:
  - dataset: noaa:ghcn-daily
    start: 2024-05-06
    end: 2024-05-07
    variables: [PRCP, TMAX]
    params:
      stations: USW00013967
      units: metric
  - dataset: usgs:water-daily
    start: 2024-05-06
    end: 2024-05-07
    variables: ["00060"]
    params:
      sites: "07164500"
      statistic_id: "00003"

2. Pull inputs and inspect a few rows

A first pull resolves both sources, downloads their files, records provenance, and writes a lockfile. Later pulls restore the pinned inputs. The displayed tables are small previews; the complete raw CSVs stay in the cache.

result = pull(manifest)
items = result.fetched
frames = {}
for item in items:
    frames.setdefault(item.asset.dataset_id, []).append(item.open())
weather = pd.concat(frames["noaa:ghcn-daily"], ignore_index=True)
streamflow = pd.concat(frames["usgs:water-daily"], ignore_index=True)
display(Markdown("**NOAA weather** — PRCP in mm; TMAX in °C (manifest requests metric)."))
display(weather[["STATION", "DATE", "PRCP", "TMAX"]].head())
display(Markdown("**USGS streamflow** — units and observation quality are retained in the CSV."))
display(
    streamflow[
        ["time", "monitoring_location_id", "parameter_code", "value", "unit_of_measure"]
    ].head()
)

NOAA weather — PRCP in mm; TMAX in °C (manifest requests metric).

	STATION	DATE	PRCP	TMAX
0	USW00013967	2024-05-06	10.9	27.2
1	USW00013967	2024-05-07	0.0	26.7

USGS streamflow — units and observation quality are retained in the CSV.

	time	monitoring_location_id	parameter_code	value	unit_of_measure
0	2024-05-06	USGS-07164500	00060	12300	ft^3/s
1	2024-05-07	USGS-07164500	00060	21500	ft^3/s

3. Inspect provenance

Each asset has its own source URL, retrieval time, size, and SHA-256 checksum. These describe the downloaded bytes. DataFrame edits or exported analyses need their own provenance; they do not change the raw input record.

for item in items:
    print(f"{item.asset.dataset_id}")
    print(f"Retrieved (UTC): {item.provenance.retrieved_at.isoformat()}")
    print(f"Bytes: {item.provenance.size}; cache hit: {item.from_cache}")
    print(f"Checksum: {item.provenance.checksum}")
    display(Markdown(f"[Source request](<{item.provenance.source_url}>)"))

noaa:ghcn-daily
Retrieved (UTC): 2026-09-09T00:11:46.962886+00:00
Bytes: 209; cache hit: False
Checksum: sha256:6327a4e6b081b1f19289a558663bc1c40f0114e2da8cbddb1cc2ad9de9af318a

Source request

usgs:water-daily
Retrieved (UTC): 2026-09-09T00:11:47.922130+00:00
Bytes: 473; cache hit: False
Checksum: sha256:7c936313099e77fb55fb63466f0ecc2266d17e68171192af753bdd736784d1e7

Source request

4. Verify and reuse the lockfile

verify checks both the manifest and cached bytes. A second pull should load the lockfile and reuse the verified cache without resolving the upstream catalog again.

drift = verify(manifest)
assert not drift, drift
again = pull(manifest)
print(f"Verification passed: {not drift}")
print(f"Restored from lockfile: {again.from_lockfile}")
display(
    pd.DataFrame(
        [
            {
                "dataset": item.asset.dataset_id,
                "cached": item.from_cache,
                "bytes": item.provenance.size,
            }
            for item in again.fetched
        ]
    )
)

Verification passed: True
Restored from lockfile: True

	dataset	cached	bytes
0	noaa:ghcn-daily	True	209
1	usgs:water-daily	True	473

5. Restore a missing input in a disposable cache

A lockfile can restore missing files while verifying their original checksums. Here we use a new temporary cache, leaving the files from the earlier cells intact. If upstream has revised the bytes, restoration fails with a checksum mismatch rather than silently changing the inputs.

from tempfile import TemporaryDirectory

with TemporaryDirectory(prefix="usdata-restore-") as directory:
    restored = pull(manifest, root=Path(directory))
    assert not verify(manifest, root=Path(directory))
    print(f"Restored from lockfile: {restored.from_lockfile}")
    print(f"Downloaded pinned assets: {sum(not item.from_cache for item in restored.fetched)}")
    print("All restored inputs match the locked checksums.")

Restored from lockfile: True
Downloaded pinned assets: 2
All restored inputs match the locked checksums.

Explore another input set

Edit the bundled manifest's station, dates, or variables, then use pull(manifest, force=True) when you intentionally want a new lockfile. Keep your analysis project's manifest and lockfile together, and preserve cached bytes when historical upstream files may disappear. See the manifest reference for the complete contract.