{ "cells": [ { "cell_type": "markdown", "id": "weather-intro", "metadata": {}, "source": [ "# Reproducible weather and streamflow inputs\n", "\n", "Collect two days of NOAA station weather near Oklahoma City and USGS streamflow\n", "at the Tulsa site through one manifest. This demonstrates acquisition and\n", "provenance across agencies. The sites are different; this notebook does not\n", "infer a causal relationship between weather and streamflow.\n", "\n", "**Saved outputs are a recorded run.** Execution and retrieval times below make\n", "the snapshot explicit. Requires `usdata` v0.5 or newer; opening CSVs requires\n", "v0.6 or newer. See [examples setup](../README.md), then restart the kernel and\n", "run all cells." ] }, { "cell_type": "code", "execution_count": 1, "id": "weather-setup", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Executed (UTC): 2026-09-09T00:11:46+00:00\n", "usdata 0.7.0; pandas 3.0.5\n" ] } ], "source": [ "from datetime import UTC, datetime\n", "from pathlib import Path\n", "\n", "import pandas as pd\n", "from IPython.display import Markdown, display\n", "\n", "import usdata\n", "from usdata.pull import pull, verify\n", "\n", "manifest = Path(\"examples/weather-and-streamflow/dataset.yaml\")\n", "if not manifest.is_file():\n", " manifest = Path(\"dataset.yaml\")\n", "\n", "pd.set_option(\"display.max_rows\", 8)\n", "pd.set_option(\"display.max_columns\", 8)\n", "print(f\"Executed (UTC): {datetime.now(UTC).isoformat(timespec='seconds')}\")\n", "print(f\"usdata {usdata.__version__}; pandas {pd.__version__}\")" ] }, { "cell_type": "markdown", "id": "weather-manifest-notes", "metadata": {}, "source": [ "## 1. Inspect the manifest\n", "\n", "The NOAA source requests metric daily precipitation (`PRCP`) and maximum\n", "temperature (`TMAX`). The USGS source requests discharge (`00060`) with the\n", "daily-mean statistic (`00003`). Quoted identifiers and codes preserve leading\n", "zeros. The USGS response carries its own per-observation units and quality columns." ] }, { "cell_type": "code", "execution_count": 2, "id": "weather-manifest", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "name: weather-and-streamflow\n", "version: \"1.0\"\n", "sources:\n", " - dataset: noaa:ghcn-daily\n", " start: 2024-05-06\n", " end: 2024-05-07\n", " variables: [PRCP, TMAX]\n", " params:\n", " stations: USW00013967\n", " units: metric\n", " - dataset: usgs:water-daily\n", " start: 2024-05-06\n", " end: 2024-05-07\n", " variables: [\"00060\"]\n", " params:\n", " sites: \"07164500\"\n", " statistic_id: \"00003\"\n", "\n" ] } ], "source": [ "print(manifest.read_text())" ] }, { "cell_type": "markdown", "id": "weather-pull-notes", "metadata": {}, "source": [ "## 2. Pull inputs and inspect a few rows\n", "\n", "A first pull resolves both sources, downloads their files, records provenance,\n", "and writes a lockfile. Later pulls restore the pinned inputs. The displayed\n", "tables are small previews; the complete raw CSVs stay in the cache." ] }, { "cell_type": "code", "execution_count": 3, "id": "weather-pull", "metadata": {}, "outputs": [ { "data": { "text/markdown": [ "**NOAA weather** — PRCP in mm; TMAX in °C (manifest requests metric)." ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
STATIONDATEPRCPTMAX
0USW000139672024-05-0610.927.2
1USW000139672024-05-070.026.7
\n", "
" ], "text/plain": [ " STATION DATE PRCP TMAX\n", "0 USW00013967 2024-05-06 10.9 27.2\n", "1 USW00013967 2024-05-07 0.0 26.7" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/markdown": [ "**USGS streamflow** — units and observation quality are retained in the CSV." ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
timemonitoring_location_idparameter_codevalueunit_of_measure
02024-05-06USGS-071645000006012300ft^3/s
12024-05-07USGS-071645000006021500ft^3/s
\n", "
" ], "text/plain": [ " time monitoring_location_id parameter_code value unit_of_measure\n", "0 2024-05-06 USGS-07164500 00060 12300 ft^3/s\n", "1 2024-05-07 USGS-07164500 00060 21500 ft^3/s" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "result = pull(manifest)\n", "items = result.fetched\n", "frames = {}\n", "for item in items:\n", " frames.setdefault(item.asset.dataset_id, []).append(item.open())\n", "weather = pd.concat(frames[\"noaa:ghcn-daily\"], ignore_index=True)\n", "streamflow = pd.concat(frames[\"usgs:water-daily\"], ignore_index=True)\n", "display(Markdown(\"**NOAA weather** — PRCP in mm; TMAX in °C (manifest requests metric).\"))\n", "display(weather[[\"STATION\", \"DATE\", \"PRCP\", \"TMAX\"]].head())\n", "display(Markdown(\"**USGS streamflow** — units and observation quality are retained in the CSV.\"))\n", "display(\n", " streamflow[\n", " [\"time\", \"monitoring_location_id\", \"parameter_code\", \"value\", \"unit_of_measure\"]\n", " ].head()\n", ")" ] }, { "cell_type": "markdown", "id": "weather-provenance-notes", "metadata": {}, "source": [ "## 3. Inspect provenance\n", "\n", "Each asset has its own source URL, retrieval time, size, and SHA-256 checksum.\n", "These describe the downloaded bytes. DataFrame edits or exported analyses need\n", "their own provenance; they do not change the raw input record." ] }, { "cell_type": "code", "execution_count": 4, "id": "weather-provenance", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "noaa:ghcn-daily\n", "Retrieved (UTC): 2026-09-09T00:11:46.962886+00:00\n", "Bytes: 209; cache hit: False\n", "Checksum: sha256:6327a4e6b081b1f19289a558663bc1c40f0114e2da8cbddb1cc2ad9de9af318a\n" ] }, { "data": { "text/markdown": [ "[Source request]()" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "usgs:water-daily\n", "Retrieved (UTC): 2026-09-09T00:11:47.922130+00:00\n", "Bytes: 473; cache hit: False\n", "Checksum: sha256:7c936313099e77fb55fb63466f0ecc2266d17e68171192af753bdd736784d1e7\n" ] }, { "data": { "text/markdown": [ "[Source request]()" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "for item in items:\n", " print(f\"{item.asset.dataset_id}\")\n", " print(f\"Retrieved (UTC): {item.provenance.retrieved_at.isoformat()}\")\n", " print(f\"Bytes: {item.provenance.size}; cache hit: {item.from_cache}\")\n", " print(f\"Checksum: {item.provenance.checksum}\")\n", " display(Markdown(f\"[Source request](<{item.provenance.source_url}>)\"))" ] }, { "cell_type": "markdown", "id": "weather-cache-notes", "metadata": {}, "source": [ "## 4. Verify and reuse the lockfile\n", "\n", "`verify` checks both the manifest and cached bytes. A second pull should load the\n", "lockfile and reuse the verified cache without resolving the upstream catalog again." ] }, { "cell_type": "code", "execution_count": 5, "id": "weather-cache", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Verification passed: True\n", "Restored from lockfile: True\n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
datasetcachedbytes
0noaa:ghcn-dailyTrue209
1usgs:water-dailyTrue473
\n", "
" ], "text/plain": [ " dataset cached bytes\n", "0 noaa:ghcn-daily True 209\n", "1 usgs:water-daily True 473" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "drift = verify(manifest)\n", "assert not drift, drift\n", "again = pull(manifest)\n", "print(f\"Verification passed: {not drift}\")\n", "print(f\"Restored from lockfile: {again.from_lockfile}\")\n", "display(\n", " pd.DataFrame(\n", " [\n", " {\n", " \"dataset\": item.asset.dataset_id,\n", " \"cached\": item.from_cache,\n", " \"bytes\": item.provenance.size,\n", " }\n", " for item in again.fetched\n", " ]\n", " )\n", ")" ] }, { "cell_type": "markdown", "id": "weather-restore-notes", "metadata": {}, "source": [ "## 5. Restore a missing input in a disposable cache\n", "\n", "A lockfile can restore missing files while verifying their original checksums.\n", "Here we use a new temporary cache, leaving the files from the earlier cells\n", "intact. If upstream has revised the bytes, restoration fails with a checksum\n", "mismatch rather than silently changing the inputs." ] }, { "cell_type": "code", "execution_count": 6, "id": "weather-restore", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Restored from lockfile: True\n", "Downloaded pinned assets: 2\n", "All restored inputs match the locked checksums.\n" ] } ], "source": [ "from tempfile import TemporaryDirectory\n", "\n", "with TemporaryDirectory(prefix=\"usdata-restore-\") as directory:\n", " restored = pull(manifest, root=Path(directory))\n", " assert not verify(manifest, root=Path(directory))\n", " print(f\"Restored from lockfile: {restored.from_lockfile}\")\n", " print(f\"Downloaded pinned assets: {sum(not item.from_cache for item in restored.fetched)}\")\n", " print(\"All restored inputs match the locked checksums.\")" ] }, { "cell_type": "markdown", "id": "weather-next", "metadata": {}, "source": [ "## Explore another input set\n", "\n", "Edit the bundled manifest's station, dates, or variables, then use\n", "`pull(manifest, force=True)` when you intentionally want a new lockfile.\n", "Keep your analysis project's manifest and lockfile together, and preserve\n", "cached bytes when historical upstream files may disappear. See the\n", "[manifest reference](../../docs/reference/manifests.md) for the complete contract." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.14.3" } }, "nbformat": 4, "nbformat_minor": 5 }