Ingestion

Before anything can be matched or checked, the two source datasets have to be loaded into a fast, columnar backend. This stage does that, converting both into Parquet datasets that the rest of the pipeline queries with DuckDB.

Two notebooks:

The two data sources

The pipeline joins together two independent datasets:

  • Rainfall Rescue monthly data — the volunteer-digitised monthly rainfall totals from Ed Hawkins’ Rainfall Rescue project. Crucially, these records already carry station names, coordinates, and other metadata. They are the anchor we use to locate the daily transcriptions.

  • Daily ensemble transcriptions — the output of the parent Auto Daily Rainfall project: one JSON file per station-year image, holding keys Day 1 … Day 31 plus a Totals block, with five ensemble-member values for every day and month slot.

Rainfall Rescue monthly data

The RR_data_ingest notebook clones the Rainfall Rescue data from GitHub and ingests the combined station CSV files into a Parquet dataset for fast DuckDB processing. It then draws an interactive map of every station’s location for a chosen year and month, coloured by that month’s rainfall, so you can confirm the ingest looks right.

This dataset is small enough to ingest in a single process; no cluster job is needed.

Daily ensemble transcriptions

The Daily_transcriptions_ingest notebook rebuilds the ensemble Parquet dataset from the JSON transcription files. The in-process rebuild shown first is fine for smoke tests, but the full dataset is far too large for one Python process.

Because each JSON file is self-contained, the work is embarrassingly parallel: the sorted file list is split into shards (default 100) that each parse their own slice, followed by a single merge step that renumbers the file_id values and writes one consolidated dataset.

Stage

Script

SLURM

Purpose

ingest

scripts/slurm/ingest_ensemble_array.sbatch

array 0–99

parse each shard’s JSON slice in parallel

merge

scripts/slurm/merge_ensemble_shards.sbatch

1 job

combine shards into one Parquet dataset

Launch the whole pipeline from a login node, pointing ENSEMBLE_ROOT at the JSON tree to ingest:

ENSEMBLE_ROOT=/path/to/ensemble_transcriptions \
    scripts/slurm/submit_ensemble_ingest.sh

The two jobs are chained with --dependency=afterok, so the merge only runs once every shard has succeeded. Shard count, an optional file cap for testing, and the per-job resources are all set in scripts/slurm/config.sh. See the SLURM reference for how to monitor jobs.

Next: Matching.