Problems downloading ERA5?

inputs-outputs
1

Hi,

Does anyone is having problems downloading ERA5 data? My scripts used to work but now they get stuck after downloading LSM and SST fields.

2

Hi,
andres
Please check following announcement. ( UPDATE 20-Mar-2024)

Can you try this following code.
parallel download for ERA5 variables using cdsapi
ERA5_request.py ::
START>
#!/usr/bin/env python
import cdsapi
import calendar
import datetime
import json
import os
from multiprocessing import Pool

Importing addmonths4date function from ERA5_utilities

from ERA5_utilities import addmonths4date

Function to download data for a single variable

def download_data(variable, options, product, output):
c = cdsapi.Client()
c.retrieve(product, options, output)

Main function to process tasks in parallel

def process_parallel(tasks):
with Pool() as pool:
pool.starmap(download_data, tasks)

Read parameters from the configuration file

from era5_crocotools_param import *

Adjusting coordinates based on dl value

dl = 2
if ownArea == 0:
lines = [line.rstrip(‘\n’) for line in open(paramFile)]
for line in lines:
if “lonmin” in line:
iStart=line.find(‘=’)+1
iEnd=line.find(‘;’)
lonmin = line[iStart:iEnd]
elif “lonmax” in line:
iStart=line.find(‘=’)+1
iEnd=line.find(‘;’)
lonmax = line[iStart:iEnd]
elif “latmin” in line:
iStart=line.find(‘=’)+1
iEnd=line.find(‘;’)
latmin = line[iStart:iEnd]
elif “latmax” in line:
iStart=line.find(‘=’)+1
iEnd=line.find(‘;’)
latmax = line[iStart:iEnd]

lonmin = str(float(lonmin)-dl)
lonmax = str(float(lonmax)+dl)
latmin = str(float(latmin)-dl)
latmax = str(float(latmax)+dl)
print ('lonmin-dl = ', lonmin)
print (‘lonmax+dl =’, lonmax)
print (‘latmin-dl =’, latmin)
print (‘latmax+dl =’, latmax)

Define the area of interest

area = [latmax, lonmin, latmin, lonmax]

Create the directory to store downloaded data if it doesn’t exist

os.makedirs(era5_dir_raw, exist_ok=True)

Load ERA5 variables’ information from JSON file

with open(‘ERA5_variables.json’, ‘r’) as jf:
era5 = json.load(jf)

Set up monthly dates limits

monthly_date_start = datetime.datetime(year_start, month_start, 1)
monthly_date_end = datetime.datetime(year_end, month_end, 1)
len_monthly_dates = (monthly_date_end.year - monthly_date_start.year) * 12 +
(monthly_date_end.month - monthly_date_start.month) + 1
monthly_date = monthly_date_start

Construct tasks for parallel processing

tasks =
for j in range(len_monthly_dates):
year = monthly_date.year
month = monthly_date.month
days_in_month = calendar.monthrange(year, month)[1]
date_start = datetime.datetime(year, month, 1)
date_end = datetime.datetime(year, month, days_in_month)
n_start = datetime.date.toordinal(date_start)
n_end = datetime.date.toordinal(date_end)
datestr_start_overlap = datetime.date.fromordinal(n_start - n_overlap).strftime(‘%Y-%m-%d’)
datestr_end_overlap = datetime.date.fromordinal(n_end + n_overlap).strftime(‘%Y-%m-%d’)
vdate = datestr_start_overlap + ‘/’ + datestr_end_overlap

for k in range(len(variables)):
    vname = variables[k]
    vlong = era5[vname][0]
    vlevt = era5[vname][3]

    options = {
        'product_type': 'reanalysis',
        'type': 'an',
        'date': vdate,
        'variable': vlong,
        'levtype': vlevt,
        'area': area,
        'format': 'netcdf',
    }

    if vlong == 'sea_surface_temperature':
        options['time'] = '00'
    elif vlong == 'land_sea_mask':
        options['time'] = '00:00'
    else:
        options['time'] = time

    if vlong == 'specific_humidity' or vlong == 'relative_humidity':
        options['pressure_level'] = '1000'
        product = 'reanalysis-era5-pressure-levels'
    else:
        product = 'reanalysis-era5-single-levels'

    fname = 'ERA5_ecmwf_' + vname.upper() + '_Y' + str(year) + 'M' + str(month).zfill(2) + '.nc'
    output = os.path.join(era5_dir_raw, fname)

    tasks.append((vname, options, product, output))

monthly_date = addmonths4date(monthly_date, 1)

Process tasks in parallel

process_parallel(tasks)

Print success message

print(‘ERA5 data request has been done successfully!’)

<END
I hope its take less time. but when CDS server fixed. :grinning:
Have a good day.
Cheers!
Subhadeep

1 Like
3

Thank you Subhadeep!

Could you upload the file? I have problem copying the code from your post, some caracters are changed and the format, important in python is affected.

4

Hello,
andres
find the attachment.
ERA5_request.py (4.0 KB)

its useful when CDS problem resolved.
Have a good day.
Cheers!
Subhadeep

1 Like
5

image
image1042×400 70.5 KB

I was trying to download ERA5 data. This problem is showing up while running both ERA5_parallel_request.py and ERA5_request.py. Can anyone help me with this?

6

@whoerfanul
check with this file. At the beginning of the code there are some instructions. please follow that.
ERA5_parallel_download_request_croco.py (8.3 KB)

Best,

7

Hi,
Complementary to this issue, I downloaded ERA5 data after they have changed their CDSapi a few months ago. The new netcdf files are now in Conventions CF 1.8 and our good old matlab netcdf interface routines are not opening these files (conflicting declaration of variables). Does any one know how to solve this?
thanks!