Weather observations rescued from RN WW2 logs

Positions of the 1.5 million records rescued (Video page)

Summary

ICOADS has many fewer observations during the period of the second world war than either before or after. To try and fill this observational gap, we set out to rescue observations recorded in the Royal Navy ship’s logbook collection at the National Archives (ADM53). (Image samples are in the paper).

Around 250,000 page images were made from the logbooks, from the years 1938-9, and 1941-46 (we ran out of funds before getting to 1940). This is not all the logbooks in the collection - voyages exclusively in UK waters or the North Atlantic were left out as those areas were the best observed. There would have been value in the logs left out, we just couldn’t afford to rescue them.

This project was driven and funded by NOAA’s Climate Data Modernisation Program, who funded the document discovery, imaging, and transcription (the data conversion and analysis was done by the Met Office). This means the transcription was done by CDMP’s professional contractors.

The project had two objectives: to generate additional observations for ICOADS in the sparsely-observed WW2 period, and to start-up a program of marine data rescue in the UK. It was very successful on both fronts.

Costs and efficiency


Date run 2005 - 2008
observations rescued 8,086,529
Elapsed time (per ob.) 3.5 years (0.23 minutes)
Financial cost (per ob.) £1 million (£0.12)
Effort required (per ob.) 16 person years (0.19 person minutes)

These figures are very uncertain: (How are these (very uncertain) numbers estimated? (& Data source).

Lessons learned

  • The Observations in the ADM53 series in the National Archives are well worth rescuing. They are made by mariners, rather than meteorologists, and are less accurate than most observations in meteorological archives; but they are not bad, and there are a lot of them, with geographically widespread coverage.
  • If you can afford it, CDMP’s transcription technique of using a group of experienced, professional data keyers is a good way to go. They produced accurate transcriptions, in a well thought out, consistent, and easy to use format.
  • A major practical challenge for marine data is keeping track of the ship position. There is a position given on most log pages, but often it is the name of a port or place, and geolocating these can be very time-consuming.