Setting up the software environment

To run the scripts in this repository, you will need to set up an environment with the necessary compute resources.

First, you will need to assign some disc space for the output files. The scripts in this repository will write a lot of data to disc, and you will need to have a few hundred gigabytes of free space available. They rely on an environment variable SCRATCH.

  • Set the SCRATCH environment variable to a directory with plenty of free disc space.

Then, you will need to install some software. The software is all open-source, and should be available for most operating systems (but it’s only been tested on Linux-x86). By far the easiest way to do this is to use conda:

When you have conda installed, you can create an environment with all the necessary software by using the YML file in this repository:

  • Edit the YML file DCVAE-Climate.yml to set the PYTHONPATH environment variable to the directory you have installed the code in.

  • Create and activate the DCVAE-Climate environment specified in the YML file DCVAE-Climate.yml.

name: DCVAE-Climate
channels:
  - conda-forge
dependencies:
  # Basics
  - python=3.9 # Later versions don't work with Tensorflow 2.9.1
  - libwebp>=1.3.2 # Earlier versions have a security vulnerability
  - iris=3.8
  - pandas=2.2
  - cmocean=4.0
  - parallel
  # Get data from ERA5
  # You'll need to register, see https://cds.climate.copernicus.eu/api-how-to
  - cdsapi=0.6
  # ML model building and diagnostics
  # If you don't have a GPU, tensorflow-eigen might be faster
  - tensorflow=2.9.1=gpu_py39hd4ae112_0
  # Optional, code formatter
  - black
  # Optional - documentation generator
  - sphinx=7.*
  # Optional - needed only for monitoring jobs
  - tmux

  # Some packages are only available via pip
  - pip
  - pip:
      # For bilinear interpolation
      - tensorflow-addons==0.18
      # For input space search
      - tensorflow-probability==0.17

# Tell python to look for modules in the root directory of the project
# (A hack, needs to be edited for every installation, but makes code
#  management much easier.)
# Replace with the path to your project directory root.

variables:
  PYTHONPATH: /home/h03/hadpb/Projects/DCVAE_Climate:/home/mo-hadpb/Projects/DCVAE_Climate