ISIMIP data access using the DKRZ server & rsync



Creating an account to access ISIMIP data

Modellers actively participating in ISIMIP need a user account at DKRZ to access the ISIMIP input & output data. For uploading model results such an account is mandatory.

An individual username and password can be obtained by:
  • filling out this form here.
  • click on the link in the email you receive and click on 'log in' to join the ISIMIP project.
  • Log in using your email address and the password you created.
  • Select 'Projects' > 'join existing project'
  • Enter project number 820, shortly describe your institutional background, your context for using the ISIMIP data and send.
  • The project managers will approve your request as soon as possible (normally within 2-3 working days).
  • You will receive a confirmation email including your username (something like b383300 - not your email address).
  • You now have an account for the DKRZ server (and other services) where the ISIMIP input data are stored.

Logging in to the DKRZ server

ISIMIP data at the DKRZ is accessible via two different servers:

  • vre2.dkrz.de - this is a virtual machine
  • mistralpp.dkrz.de - this is a physical cluster with more computing power

Data folders look the same on the two servers, except home directories.

Connecting and transferring data to and from these servers is only possible using rsync, ssh, sftp and scp - not http. For example:

via a terminal:

ssh [YOUR-USERNAME]@vre2.dkrz.de


using WinSCP:

File Protocol: SFTP
Host name: mistralpp.dkrz.de 
Port number: 22
User name: [YOUR-USERNAME]

 

How to find and transfer ISIMIP data

The ISIMIP data are located in the folder to /work/bb0820/ISIMIP/.

Data are then organised according to the simulation round. For each simulation round there are folders for:

  • InputData. All data necessary to run your ISIMIP simulations organised into folders according to the type of data.
  • UploadArea. This is where you should place your simulation data organised into sectors, models and time frames covered. Take care that only data belonging to a certain period is uploaded in the periods folders accordingly. Data will be moved from this folder on a daily basis for the purposes of the quality check. You cannot and should not download simulation data from other models from this location. Please be sure to read these important instructions about preparing your simulation files.
  • OutputData. This is where you will find simulation data that has passed the quality check. If, for whatever reason, data in OutputData is replaced or removed, these changes will be documented in the Output Data Change Log. It is your responsibility to check this log before making official use of the output data (e.g. in a paper or conference presentation), to ensure you have the latest data.
  • DerivedOutputData. Here you will find data products derived from ISIMIP simulation data, available for you to use.

Examples

You want to download the land-only, bias-corrected, atmospheric data for ISIMIP2b for the GCM IPSL-CM5A-LR for the RCP2.6 scenario?

scp -r [YOUR-USERNAME]@mistralpp.dkrz.de:/work/bb0820/ISIMIP/ISIMIP2b/InputData/GCM_atmosphere/biascorrected/landonly/rcp26/IPSL-CM5A-LR [LOCAL-DESTINATION-PATH]

or

rsync -auv [YOUR-USERNAME]@mistralpp.dkrz.de:/work/bb0820/ISIMIP/ISIMIP2b/InputData/GCM_atmosphere/biascorrected/landonly/rcp26/IPSL-CM5A-LR [LOCAL-DESTINATION-PATH]

You want to upload your ISIMIP2b results?


You will need to prepare your files according to these instructions, and place the files in the appropriate temporal folder (pre-industrial, historical, future or future_extended). 

scp [LOCAL-PATH-TO-DATA]  [YOUR-USERNAME]@mistralpp.dkrz.de:/work/bb0820/ISIMIP/ISIMIP2b/UploadArea/[YOUR-SECTOR]/[YOUR-MODEL]/_tmp/historical/

 


Access to ISIMIP input data for external users

A specialized rsync service is available via PIK which grants read-only access to some of the input data. rsync is an open source file transfer program for Unix systems providing a very fast method for bringing remote files into sync. Windows ports of rsync are available from several internet sites.

rsync hostname: rsync.pik-potsdam.de
rsync password: Request current password by writing to info@isimip.org. This password should not be shared and will be updated on a regular basis.

To receive a list of remotely available files and sub folders use the following command:

RSYNC_PASSWORD='PASSWORD' rsync -vL isimip_pik@rsync.pik-potsdam.de::isimip/rsync_external/

Example
The following example will download all 2m temperature (tas) files from the ISIMIP2b bias-corrected land-only IPSL-CM5A-LR input data set to the current local folder. Replace the single dot at the end of the line with a different destination folder if wanted. The -u option will check for already present files on your end and will only transfer newer or updated files.

RSYNC_PASSWORD='PASSWORD' rsync -auvL isimip_pik@rsync.pik-potsdam.de::isimip/rsync_external/input_data/ISIMIP2b/gcms_bced_landonly/IPSL-CM5A-LR/EWEMBI/tas_*historical*.nc4 .

A few notes
  • You will not gain any improvements in download speeds or amount of data being transmitted by using the -z option since all NetCDF data are already internally compressed.
  • Make sure your machine and institution's network allow outgoing connections on port 873.

Notes on the ISIMIP2b Data

Here you can see how the input data for ISIMIP2b are organised on the DKRZ server. Note that if you don't require data over oceans, you can make use of the smaller, 'landonly' climate data files.

Here you can find an overview of the input data available. More detailed information can be found in the ISIMIP2b protocol (tables 1-7) and the input-data table.


Notes on the ISIMIP2a Data

Before starting to download the ISIMIP2a input data, please read these important instructions about versioning!

Notes on the folder Structure
  • For ISIMIP2a there is an additional InputData folder EvalData, which contains data sets for model evaluation.
  • The climate input data is divided into HistObs (observational climate input data sets for historic validation runs) and FastTrackGCMs (Fast-Track climate input data for catch-up runs (ISIMIP2a) for sectors not in the ISIMIP Fast Track).

Notes on the Fast-Track Data

All input (and output) Fast Track data is available via the ESGF node esg.pik-potsdam.de. Please find all information on how to obtain the ISIMIP Fast Track data in the Data Archive section.

There are two versions of the climate input data from the ISIMIP Fast Track:

  • The first version was used to force impact models in the ISIMIP Fast Track phase and should only be used for the ISIMIP2 catch-up experiments for sectors that were already part of the Fast Track (global water, global biomes, agriculture, vector-borne diseases & coastal infrastructure). It was published under the ISIMIP Fast Track ESGF project.
  • For all other purposes, i.e. future runs for new sectors and modeling exercises with no relation to ISIMIP, version 2 published under the ISIMIP2a ESGF project should be used. This corrected and extended version (denoted as ISIe in Hempel et al. 2013) overcomes several limitations in adjusting the daily variability.