Workflow overview

From a user’s point of view, the whole project is controlled through a series of Jupyter notebooks. Each notebook documents and drives one stage of the pipeline: it submits the heavy work to Azure ML, waits for the jobs, downloads the results, and produces validation summaries. To reproduce or extend the project, you open the notebooks and run them in order.

Important

Run every notebook in the weather-doc-extractor Conda environment (conda activate weather-doc-extractor). See Installation.

The stages

The notebooks are grouped into stages, run top to bottom:

preparation/          Download the source images, build synthetic training
      │               data, and import the real test set
      ▼
zeroth_order/         Measure the raw, un-fine-tuned models  (baseline)
      │
      ▼
first_order/          Fine-tune on synthetic data, then re-measure
      │
      ▼
second_order/         Build a consensus set from real images, fine-tune again,
      │               then re-measure
      ▼
operations/           Run the finished ensemble over the full dataset

Two ideas run through the whole workflow:

• Fake data for training, real data for testing. We can cheaply generate synthetic images whose values we know, and use them to teach the models the task. We keep a small, precious set of hand-transcribed real images (from Ciara Ryan) purely for validation, so every accuracy number is measured against genuine records.

• Consensus instead of ground truth. We have no transcriptions for the real images, so we can’t fine-tune on them directly. Instead we run several models, keep the cells where they agree, and treat that agreement as training truth for a second round of fine-tuning.

The notebooks, in order

#	Stage	Notebook	What it does
1	Preparation	download_document_images	Download the archive PDFs and split them into per-page images
2	Preparation	make_fake_daily_rainfall_training_data	Generate ~1000 synthetic images with known values
3	Preparation	add_test_data_from_Ciara	Import the 64-image hand-transcribed test set
4	Zeroth order	validate_0th_order_fake	Baseline of raw models on fake data
5	Zeroth order	validate_0th_order_real	Baseline of raw models on real data
6	First order	finetune_original_models	Fine-tune all models on the synthetic data
7	First order	validate_1st_order_fake	First-order models on fake data
8	First order	validate_1st_order_real	First-order models on real data
9	Second order	make_1st_training_consensus	Build the consensus training set from real images
10	Second order	finetune_1st_order_models	Fine-tune again on the consensus set
11	Second order	validate_2nd_order_real	Second-order models on real data
12	Operations	run_operational_extractions_sample	Ensemble extraction on a large sample
13	Operations	run_operational_extractions_all	Ensemble extraction on all ~660,000 images

Work through the stage pages next, starting with Preparation.

Reading the validation figures

The validation notebooks produce a comparison figure for each test image: the scanned document on the left, and the model’s transcription laid over the table grid. Numbers in blue match the ground truth; numbers in red are wrong. These figures appear throughout the stage pages that follow, and make the effect of each round of fine-tuning immediately visible.