Problems downloading ERA5?

inputs-outputs
1

Hi,

Does anyone is having problems downloading ERA5 data? My scripts used to work but now they get stuck after downloading LSM and SST fields.

2

Hi,
andres
Please check following announcement. ( UPDATE 20-Mar-2024)

Can you try this following code.
parallel download for ERA5 variables using cdsapi
ERA5_request.py ::
START>
#!/usr/bin/env python
import cdsapi
import calendar
import datetime
import json
import os
from multiprocessing import Pool

Importing addmonths4date function from ERA5_utilities

from ERA5_utilities import addmonths4date

Function to download data for a single variable

def download_data(variable, options, product, output):
c = cdsapi.Client()
c.retrieve(product, options, output)

Main function to process tasks in parallel

def process_parallel(tasks):
with Pool() as pool:
pool.starmap(download_data, tasks)

Read parameters from the configuration file

from era5_crocotools_param import *

Adjusting coordinates based on dl value

dl = 2
if ownArea == 0:
lines = [line.rstrip(‘\n’) for line in open(paramFile)]
for line in lines:
if “lonmin” in line:
iStart=line.find(‘=’)+1
iEnd=line.find(‘;’)
lonmin = line[iStart:iEnd]
elif “lonmax” in line:
iStart=line.find(‘=’)+1
iEnd=line.find(‘;’)
lonmax = line[iStart:iEnd]
elif “latmin” in line:
iStart=line.find(‘=’)+1
iEnd=line.find(‘;’)
latmin = line[iStart:iEnd]
elif “latmax” in line:
iStart=line.find(‘=’)+1
iEnd=line.find(‘;’)
latmax = line[iStart:iEnd]

lonmin = str(float(lonmin)-dl)
lonmax = str(float(lonmax)+dl)
latmin = str(float(latmin)-dl)
latmax = str(float(latmax)+dl)
print ('lonmin-dl = ', lonmin)
print (‘lonmax+dl =’, lonmax)
print (‘latmin-dl =’, latmin)
print (‘latmax+dl =’, latmax)

Define the area of interest

area = [latmax, lonmin, latmin, lonmax]

Create the directory to store downloaded data if it doesn’t exist

os.makedirs(era5_dir_raw, exist_ok=True)

Load ERA5 variables’ information from JSON file

with open(‘ERA5_variables.json’, ‘r’) as jf:
era5 = json.load(jf)

Set up monthly dates limits

monthly_date_start = datetime.datetime(year_start, month_start, 1)
monthly_date_end = datetime.datetime(year_end, month_end, 1)
len_monthly_dates = (monthly_date_end.year - monthly_date_start.year) * 12 +
(monthly_date_end.month - monthly_date_start.month) + 1
monthly_date = monthly_date_start

Construct tasks for parallel processing

tasks =
for j in range(len_monthly_dates):
year = monthly_date.year
month = monthly_date.month
days_in_month = calendar.monthrange(year, month)[1]
date_start = datetime.datetime(year, month, 1)
date_end = datetime.datetime(year, month, days_in_month)
n_start = datetime.date.toordinal(date_start)
n_end = datetime.date.toordinal(date_end)
datestr_start_overlap = datetime.date.fromordinal(n_start - n_overlap).strftime(‘%Y-%m-%d’)
datestr_end_overlap = datetime.date.fromordinal(n_end + n_overlap).strftime(‘%Y-%m-%d’)
vdate = datestr_start_overlap + ‘/’ + datestr_end_overlap

for k in range(len(variables)):
    vname = variables[k]
    vlong = era5[vname][0]
    vlevt = era5[vname][3]

    options = {
        'product_type': 'reanalysis',
        'type': 'an',
        'date': vdate,
        'variable': vlong,
        'levtype': vlevt,
        'area': area,
        'format': 'netcdf',
    }

    if vlong == 'sea_surface_temperature':
        options['time'] = '00'
    elif vlong == 'land_sea_mask':
        options['time'] = '00:00'
    else:
        options['time'] = time

    if vlong == 'specific_humidity' or vlong == 'relative_humidity':
        options['pressure_level'] = '1000'
        product = 'reanalysis-era5-pressure-levels'
    else:
        product = 'reanalysis-era5-single-levels'

    fname = 'ERA5_ecmwf_' + vname.upper() + '_Y' + str(year) + 'M' + str(month).zfill(2) + '.nc'
    output = os.path.join(era5_dir_raw, fname)

    tasks.append((vname, options, product, output))

monthly_date = addmonths4date(monthly_date, 1)

Process tasks in parallel

process_parallel(tasks)

Print success message

print(‘ERA5 data request has been done successfully!’)

<END
I hope its take less time. but when CDS server fixed. :grinning:
Have a good day.
Cheers!
Subhadeep

3

Thank you Subhadeep!

Could you upload the file? I have problem copying the code from your post, some caracters are changed and the format, important in python is affected.

4

Hello,
andres
find the attachment.
ERA5_request.py (4.0 KB)

its useful when CDS problem resolved.
Have a good day.
Cheers!
Subhadeep

1 Like