Export

The located, quality-controlled observations are only useful to others if they are shared in a standard form. This stage writes them out in the Station Exchange Format (SEF), the community standard for rescued climate data.

One notebook:

• export_sef.ipynb

What SEF is

SEF is a simple tab-separated text format where one file holds one variable from one station. Here each file is a single station-year of daily rainfall: twelve header lines of metadata, a column header, then one line per observed day.

Merging the ensemble’s duplicates

The ensemble frequently contains duplicate transcriptions of the same station-year (the same physical page transcribed more than once). The export merges these — exact matches sharing a location and year — into one SEF .tsv per real station-year, written to <output_root>/tsv/<year>/<ID>.tsv.

For every day the value is taken from the duplicate with the best QC verdict, using this precedence so no day is dropped:

qc1=pass  >  qc1=fail & qc2=pass  >  qc1=review
          >  qc1=fail & qc2=indeterminate  >  the rest

Each observation is the consensus daily total (the member median), converted from inches to millimetres. The QC verdicts and the contributing source= travel in each observation’s Meta column (qc1=… from the exact-monthly check, qc2=… from the secondary check), and the file-level Meta lists every merged source.

Running the export

The notebook runs a small local export into /var/tmp, inspects a generated file, and validates its structure — confirming the twelve header lines are in the required order, the data-table header is correct, every day round-trips back as a row, and the QC verdicts are present.

The full station-year export runs as a single SLURM array — there is no merge stage, because sharding by year keeps every duplicate of a station-year in the same task and each shard writes disjoint year-partitioned files:

scripts/slurm/submit_sef_export.sh

Each array task exports a contiguous matched-year slice. Shard count, the year range, and the SEF metadata fields (SEF_SOURCE, SEF_LINK, SEF_OBS_HOUR, …) are configured in the SEF_* block of scripts/slurm/config.sh. The prerequisites are the daily-consensus table and the assigned ensemble metadata; the QC tables are joined when present. Files land in $PDIR/sef_export/tsv/<year>/<ID>.tsv with per-shard manifests alongside.

Next: Analysis.