Global observations count

The total number of observations, in ISPD3.2.9, over the period jan 1st 1851 to Dec 31st 2000, is 1,133,185,756.

Code to count them

Download all the observations for the period:

# Retrieve ISPD3.2.9 observations from NERSC

import datetime
import IRData.twcr as twcr

for year in range(1851,2001):
    dte=datetime.datetime(year,1,1)
    twcr.fetch_observations(dte,version='2c')

The data files have 1 observation per line, so it’s fastest to count the lines

#!/usr/bin/env python

# Count the total number of observations, 1851-2000
#  Fastest way is to count the lines in the text files

import os
import sys

# How many lines in a file
def file_len(fname):
    with open(fname) as f:
        for i, l in enumerate(f):
            pass
    return i + 1

total=0
for year in range(1851,2001):
    ydir=("%s/20CR/version_2c/observations/%04d" %
                            (os.getenv('SCRATCH'),year))
    files=os.listdir(ydir)
    for file in files:
        total=total+file_len("%s/%s" % (ydir,file))
    print "%04d %11d" % (year,total)
    sys.stdout.flush()
print total

    

Note, the unix utility ‘wc’ would do this, except that ‘wc -l $SCRATCH/20CR/version_2c/observations//.txt’ wll fail, as there are too many files.

Counts up to the end of each year

Output of the script above.

1851       70494
1852      152050
1853      253569
1854      395476
1855      563051
1856      754510
1857      968433
1858     1169088
1859     1346510
1860     1491532
1861     1619875
1862     1732267
1863     1836152
1864     1952599
1865     2057934
1866     2171387
1867     2292133
1868     2424389
1869     2569416
1870     2732551
1871     2887946
1872     3075807
1873     3288540
1874     3528650
1875     3775583
1876     4026394
1877     4272946
1878     4532741
1879     4868953
1880     5217015
1881     5622646
1882     6113218
1883     6724446
1884     7413229
1885     8251782
1886     9067525
1887     9691980
1888    10256081
1889    10736425
1890    11218765
1891    11647182
1892    12061366
1893    12442888
1894    12839430
1895    13256232
1896    13694499
1897    14103088
1898    14529973
1899    14981156
1900    15433026
1901    15936340
1902    16488108
1903    17047930
1904    17581666
1905    18122040
1906    18670011
1907    19230802
1908    19854172
1909    20551337
1910    21274018
1911    22023892
1912    22785888
1913    23687586
1914    24520399
1915    25358694
1916    26196906
1917    26958855
1918    27637678
1919    28316554
1920    29057869
1921    29924282
1922    30902362
1923    31975098
1924    33019359
1925    34051766
1926    35125468
1927    36238154
1928    37339351
1929    38523973
1930    39884118
1931    41510488
1932    43293059
1933    45191423
1934    47270775
1935    49573942
1936    52663425
1937    55991177
1938    59581669
1939    62957140
1940    66324799
1941    70057700
1942    74076908
1943    80065295
1944    87465829
1945    95141950
1946   100207947
1947   105453571
1948   112031658
1949   120290085
1950   129126345
1951   138440948
1952   148301562
1953   159581009
1954   171139092
1955   181775765
1956   192238222
1957   203967052
1958   215460578
1959   226696858
1960   238504007
1961   250686811
1962   262876098
1963   275553700
1964   288387555
1965   298380603
1966   311489921
1967   328450298
1968   346554120
1969   364204917
1970   382087922
1971   399356183
1972   416923975
1973   438383344
1974   460129055
1975   482062871
1976   503778970
1977   525645617
1978   547942523
1979   570873731
1980   594384426
1981   618377545
1982   641710950
1983   665863342
1984   690592626
1985   715697574
1986   741365973
1987   767351950
1988   794217071
1989   821210504
1990   848954176
1991   876481353
1992   903617492
1993   930639877
1994   958239373
1995   986483206
1996  1014945416
1997  1043580350
1998  1073310152
1999  1102652862
2000  1133185756
1133185756