feedback requested: new ISIMIP data repository


Posted by Martin Park on Oct. 27, 2020

Dear ISIMIP modelers,

We are currently developing the new ISIMIP data repository as a user friendly portal for accessing both, input and output data. Meanwhile, the beta version went online and you can access it at data.isimip.org. The repository currently contains the input data for ISIMIP3.

We are in the process of re-publishing the ISIMIP2 output data using the new software, as well. As an intermediate step, we created a development version of the new repository located at dev.isimip.org (username: “isimip”, password: “isimip2020”). We invite you to have a pre-release look at the presentation of the output data, especially of your model. Please do not hesitate to give us feedback and report bugs and pitfalls that need to be fixed. Later this year, we will transfer the data from the development version to data.isimip.org, which we will also use for all ISIMIP3 output data.

The new portal offers similar search and filtering capabilities as our old ESGF portal, but does not require you to log in to access and download files. This should make access and download using bulk download scripts and tools like wget much easier. In fact, the files are now located on a separate file server files.isimip.org in a structure similar to our cluster environment at DKRZ.

We also added a masking service to the repository, which enables you to download data for a specific country, latitude-longitude box, or land only. While this considerably reduces the volume of data to be downloaded, it takes some time on the server to process the files. The interface for this API is labeled “Configure download” in the search interface.

The new portal also comes with a powerful REST API, which can be used for scripted access. For Python, we created a small client library with some example Jupyter notebooks [1].

The code for the portal itself [2], the masking service [3] and our publication pipeline [4] are also available as Open Source Software on GitHub.

While more features will be added to the data portal in the next month, we encourage you to use both versions (data and dev) of the portal for searching and downloading data. We are looking forward to your thoughts and suggestions for improving these exciting new services.

Cheers, the ISIMIP team at PIK

[1] https://github.com/ISI-MIP/isimip-client

[2] https://github.com/ISI-MIP/isimip-data

[3] https://github.com/ISI-MIP/isimip-files-api

[4] https://github.com/ISI-MIP/isimip-publisher