Scripts and SLURM jobs

The scripts/ directory holds the command-line entry points that the notebooks call and the SLURM jobs run. This page is an inventory; for how they fit together see the workflow overview and the SLURM reference.

Processing scripts (scripts/)

These are plain Python entry points, each a thin wrapper around a module in src/rainfall_rescue_sqlite/. A single SLURM array task typically runs one of these for its shard.

Script	Stage	Purpose
build_rainfall_rescue_parquet.py	Ingestion	Rainfall Rescue CSV → Parquet
build_ensemble_transcriptions_parquet.py	Ingestion	Ensemble JSON → Parquet
run_ensemble_ingest_shard.py / merge_ensemble_shards.py	Ingestion	Sharded ensemble ingest and merge
build_similarity_vectors.py	Matching	Build the normalised comparison vectors
run_similarity_shard.py / merge_similarity_shards.py	Matching	Sharded matching and merge
run_monthly_similarity_baseline.py	Matching	Baseline similarity scoring
assign_ensemble_metadata.py	Matching	Copy RR metadata onto matched records
run_qc_check_exact_monthly.py / run_qc_shard.py / merge_qc_shards.py	QC1	Monthly-total check, sharded, and merge
run_daily_consensus_shard.py / merge_daily_consensus_shards.py	QC2	Build the daily-consensus table
run_regional_stats_shard.py / merge_regional_stats_shards.py	QC2	Regional neighbour statistics
train_secondary_qc_models.py / score_secondary_qc.py	QC2	Train and score the secondary ML check
export_sef.py	Export	Merge duplicates → SEF .tsv
build_sef_analysis_parquet.py	Analysis	Parse SEF → analysis Parquet
render_animation_manifest.py / render_animation_shard.py / render_animation_validate.py / render_animation_encode.py	Analysis	Consensus-rainfall animation stages
render_sef_animation_manifest.py / render_sef_animation_shard.py	Analysis	SEF-rainfall animation stages

SLURM jobs (scripts/slurm/)

Each pipeline has a submit_*.sh driver that sources config.sh and submits the .sbatch files below with the right dependencies.

Driver	.sbatch files	Shape
submit_ensemble_ingest.sh	ingest_ensemble_array, merge_ensemble_shards	array + merge
submit_all.sh	build_vectors, match_array, merge_shards	build + array + merge
submit_qc.sh	qc_array, qc_merge	array + merge
submit_daily_consensus.sh	daily_consensus_array, daily_consensus_merge	array + merge
submit_regional_stats.sh	regional_stats_array, regional_stats_merge	array + merge
submit_secondary_qc.sh	secondary_qc_train, secondary_qc_score	train + score
submit_sef_export.sh	sef_export_array	array
submit_sef_analysis.sh	sef_analysis_array	array
submit_animation.sh	render_precompute, render_array, render_validate, render_encode	4-stage
submit_sef_animation.sh	sef_render_precompute, sef_render_array, sef_render_validate, sef_render_encode	4-stage

config.sh is the single source of truth for paths, shard counts, parameters and resources; see the SLURM reference.

Diagnostic scripts (scripts/diagnostics/)

These build the maps and figures the notebooks embed. Most have a build_figure function the notebooks import, and a __main__ block so they can be run from the command line to write a standalone image or self-contained interactive HTML.

Script	Produces
plot_image_consensus_metadata.py	One-page per-transcription diagnostic (image, consensus table, monthly comparison, map)
plot_daily_rainfall_map.py	Static UK map of consensus rainfall for a date
plot_daily_rainfall_interactive.py	Interactive (Plotly) version of the daily map
plot_daily_qc_interactive.py	Interactive map coloured by QC1 flag
plot_regional_stat_interactive.py	Interactive map of any regional statistic
plot_secondary_qc_interactive.py	Interactive map coloured by the secondary-QC flag
plot_rr_match_counts_interactive.py	Matching-health map: exact matches per RR station
plot_rank1_exact_agreement_distribution.py	Distribution of rank-1 exact-agreement counts

For example:

python scripts/diagnostics/plot_daily_rainfall_map.py 1931-10-15 \
    --output daily_map.webp