ERROR: Abnormal termination: netCDF INPUT: GET_GRID ERROR: Cannot open input NetCDF file: croco_grd.nc

eduardolima255 · May 21, 2024, 10:15pm

Hi dear croco users,

I’m encountering an error while running my model in a regional and climatological configuration. I’m using Open MPI and running it for 1 month as a test. I will provide detailed information below and share my croco.in, run_croco.bash, cppdefs.h, and param.h files.

I am running on a remote cluster, where I have two directories: $HOME and $SCRATCH. In this cluster, the compilation needs to be done in $HOME, while the execution must be done in $SCRATCH. Therefore, after compilation, I copy my croco directory to $SCRATCH.

The message with the error from croco_Y1M1.out after running

WARNING: Unrecognized keyword: diagnostics → DISREGARDED.

WARNING: Unrecognized keyword: diag_avg → DISREGARDED.

WARNING: Unrecognized keyword: diag3D_history_fields → DISREGARDED.

WARNING: Unrecognized keyword: diag2D_history_fields → DISREGARDED.

WARNING: Unrecognized keyword: diag3D_average_fields → DISREGARDED.

WARNING: Unrecognized keyword: diag2D_average_fields → DISREGARDED.

WARNING: Unrecognized keyword: diagnosticsM → DISREGARDED.

WARNING: Unrecognized keyword: diagM_avg → DISREGARDED.

WARNING: Unrecognized keyword: diagM_history_fields → DISREGARDED.

WARNING: Unrecognized keyword: diagM_average_fields → DISREGARDED.

WARNING: Unrecognized keyword: diags_vrt → DISREGARDED.

WARNING: Unrecognized keyword: diags_vrt_avg → DISREGARDED.

WARNING: Unrecognized keyword: diags_vrt_history_fields → DISREGARDED.

WARNING: Unrecognized keyword: diags_vrt_average_fields → DISREGARDED.

WARNING: Unrecognized keyword: diags_ek → DISREGARDED.

WARNING: Unrecognized keyword: diags_ek_avg → DISREGARDED.

WARNING: Unrecognized keyword: diags_ek_history_fields → DISREGARDED.

WARNING: Unrecognized keyword: diags_ek_average_fields → DISREGARDED.

WARNING: Unrecognized keyword: surf → DISREGARDED.

WARNING: Unrecognized keyword: surf_avg → DISREGARDED.

WARNING: Unrecognized keyword: surf_history_fields → DISREGARDED.

WARNING: Unrecognized keyword: surf_average_fields → DISREGARDED.

WARNING: Unrecognized keyword: diags_pv → DISREGARDED.

WARNING: Unrecognized keyword: diags_pv_avg → DISREGARDED.

WARNING: Unrecognized keyword: diags_pv_history_fields → DISREGARDED.

WARNING: Unrecognized keyword: diags_pv_average_fields → DISREGARDED.

WARNING: Unrecognized keyword: diags_eddy → DISREGARDED.

WARNING: Unrecognized keyword: diags_eddy_avg → DISREGARDED.

WARNING: Unrecognized keyword: diags_eddy_history_fields → DISREGARDED.

WARNING: Unrecognized keyword: diags_eddy_average_fields → DISREGARDED.

WARNING: Unrecognized keyword: diagnostics_bio → DISREGARDED.

WARNING: Unrecognized keyword: diagbio_avg → DISREGARDED.

WARNING: Unrecognized keyword: diagbioFlux_history_fields → DISREGARDED.

WARNING: Unrecognized keyword: diagbioVSink_history_fields → DISREGARDED.

WARNING: Unrecognized keyword: diagbioGasExc_history_fields → DISREGARDED.

WARNING: Unrecognized keyword: diagbioFlux_average_fields → DISREGARDED.

WARNING: Unrecognized keyword: diagbioVSink_average_fields → DISREGARDED.

WARNING: Unrecognized keyword: diagbioGasExc_average_fields → DISREGARDED.

WARNING: Unrecognized keyword: biology → DISREGARDED.

WARNING: Unrecognized keyword: wkb_boundary → DISREGARDED.

WARNING: Unrecognized keyword: wkb_wwave → DISREGARDED.

WARNING: Unrecognized keyword: wkb_roller → DISREGARDED.

WARNING: Unrecognized keyword: wave_history_fields → DISREGARDED.

WARNING: Unrecognized keyword: wave_average_fields → DISREGARDED.

WARNING: Unrecognized keyword: wci_history_fields → DISREGARDED.

WARNING: Unrecognized keyword: wci_average_fields → DISREGARDED.

WARNING: Unrecognized keyword: sediments → DISREGARDED.

WARNING: Unrecognized keyword: sediment_history_fields → DISREGARDED.

WARNING: Unrecognized keyword: bbl_history_fields → DISREGARDED.

WARNING: Unrecognized keyword: floats → DISREGARDED.

WARNING: Unrecognized keyword: float_fields → DISREGARDED.

WARNING: Unrecognized keyword: stations → DISREGARDED.

NUMBER OF THREADS: 1 BLOCKING: 1 x 48.

WARNING: Unrecognized keyword: station_fields → DISREGARDED.

WARNING: Unrecognized keyword: psource → DISREGARDED.

WARNING: Unrecognized keyword: psource_ncfile → DISREGARDED.

WARNING: Unrecognized keyword: online → DISREGARDED.

NUMBER OF THREADS: 1 BLOCKING: 1 x 48.

Activated C-preprocessing Options:

      REGIONAL
      BACIACAMPOS
      OPENMP
      OBC_EAST
      OBC_NORTH
      OBC_SOUTH
      CURVGRID
      SPHERICAL
      MASKING
      NEW_S_COORD
      SOLVE3D
      UV_COR
      UV_ADV
      SALINITY
      NONLIN_EOS
      ANA_DIURNAL_SW
      QCORRECTION
      SFLX_CORR
      CLIMATOLOGY
      ZCLIMATOLOGY
      M2CLIMATOLOGY
      M3CLIMATOLOGY
      TCLIMATOLOGY
      ZNUDGING
      M2NUDGING
      M3NUDGING
      TNUDGING
      UV_HADV_UP3
      UV_VADV_SPLINES
      TS_HADV_RSUP3
      TS_DIF4
      TS_VADV_SPLINES
      SPONGE
      LIMIT_BSTRESS
      LMD_MIXING
      LMD_SKPP
      LMD_BKPP
      LMD_RIMIX
      LMD_CONVEC
      LMD_NONLOCAL
      ANA_BSFLUX
      ANA_BTFLUX
      OBC_M2CHARACT
      OBC_M3ORLANSKI
      OBC_TORLANSKI
      AVERAGES
      AVERAGES_K
      TS_HADV_C4
      MPI_COMM_WORLD
      M2FILTER_POWER
      TRACERS
      TEMPERATURE
      HZR
      VAR_RHO_2D
      SPLIT_EOS
      UV_MIX_S
      DIF_COEF_3D
      TS_MIX_ISO
      TS_MIX_IMP
      TS_MIX_ISO_FILT
      NTRA_T3DMIX
      SPONGE_GRID
      SPONGE_DIF2
      SPONGE_VIS2
      LMD_SKPP2005
      NF_CLOBBER
      ENDDOLOOP2D
      DOLOOP2D_R
      DOLOOP2D

NUMBER OF THREADS: 1 BLOCKING: 1 x 48.
NUMBER OF THREADS: 1 BLOCKING: 1 x 48.

GET_GRID ERROR: Cannot open input NetCDF file: croco_grd.nc

MAIN - number of records written into history file(s): 0
number of records written into restart file(s): 0
number of records written into averages file(s): 0

ERROR: Abnormal termination: netCDF INPUT

but my croco files (including croco_grd.nc) are in my CROCO_FILES folder.

Scripts:

cppdefs.h (44.7 KB)
param.h (33.6 KB)

croco.in: croco.in - Google Drive
run_croco.bash: run_croco.bash - Google Drive

Thank you all for your attention.

Matt29 · May 22, 2024, 3:59pm

Hello,
I think you are using the wrong parallel mode.
You should use MPI instead of OpenMP.
In cppdefs.h you should have

define MPI
undef OPENMP

In your param.h only change NP_XI and NP_ETA to match the number of desired nodes.

#ifdef MPI
integer NP_XI, NP_ETA, NNODES
parameter (NP_XI=6, NP_ETA=8, NNODES=NP_XI*NP_ETA)
parameter (NPP=1)
parameter (NSUB_X=1, NSUB_E=1)

eduardolima255 · May 27, 2024, 10:05pm

Hi, thank you for your response.

Why can’t I use OpenMP?
I am conducting a one-month test with OpenMP and MPI to determine which option is faster on this cluster. I followed your recommendation and switched the settings to MPI with NPP=1, compiled, and submitted the job. However, I am still receiving the same error. I tried specifying the full path to my CROCO_FILE in croco.in, but it still cannot open the file croco_grd.nc.

Matt29 · May 28, 2024, 5:41am

Did you try in sequential ?
What is the command you launch to run your simulation ?
Are you sure the nodes on which you launch your job “see” the disk on which is your grid file. Every files are on scratch ?

eduardolima255 · May 28, 2024, 4:48pm

Yes, I used the command ‘sbatch run_croco.bash’ to submit the job. Following the cluster’s recommendation, I specified the number of nodes inside the run_croco.bash file, with the commander:

#SBATCH --nodes=2
#SBATCH --ntasks-per-node=24
#SBATCH --ntasks=48
#SBATCH -p cpu_small
#SBATCH -J mpi
#SBATCH --exclusive

I copied all the files from the croco and my modeling folder to the scratch directory; only inside the scratch directory can the nodes ‘see’ my files.

However, I suspect that the problem lies in this path. For some reason, the nodes cannot interact with my files. I’m new to this cluster. I’m going to test it on another remote cluster at the university I am part of to see if it works and if the problem is indeed with the current cluster I’m using.

Thank you once again.

eduardolima255 · June 3, 2024, 12:43am

Hello again,

I conducted some tests on a workstation using a new configuration with configure name ‘TesteBacia’, but unfortunately, the same error persists:

BENGUELA TEST MODEL
720 ntimes Total number of timesteps for 3D equations.
3600.00 dt Timestep [sec] for 3D equations
60 ndtfast Number of 2D timesteps within each 3D step.
1 ninfo Number of timesteps between runtime diagnostics.

WARNING: Unrecognized keyword: time_stepping_nbq → DISREGARDED.

7.000E+00 theta_s S-coordinate surface control parameter.
2.000E+00 theta_b S-coordinate bottom control parameter.
2.000E+02 Tcline S-coordinate surface/bottom layer width used in
vertical coordinate stretching, meters.

WARNING: Unrecognized keyword: start_date → DISREGARDED.

WARNING: Unrecognized keyword: end_date → DISREGARDED.

WARNING: Unrecognized keyword: xios_origin_date → DISREGARDED.

WARNING: Unrecognized keyword: output_time_steps → DISREGARDED.

      Grid File:  CROCO_FILES/croco_grd.nc

Forcing Data File: CROCO_FILES/croco_frc.nc

WARNING: Unrecognized keyword: bulk_forcing → DISREGARDED.

Climatology File: CROCO_FILES/croco_clm.nc

WARNING: Unrecognized keyword: boundary → DISREGARDED.

Initial State File: CROCO_FILES/croco_ini.nc Record: 1
Restart File: CROCO_FILES/croco_rst.nc nrst = 720 rec/file: -1
History File: CROCO_FILES/croco_his.nc Create new: T nwrt = 72 rec/file = 0
1 ntsavg Starting timestep for the accumulation of output
time-averaged data.
72 navg Number of timesteps between writing of time-averaged
data into averages file.
Averages File: CROCO_FILES/croco_avg.nc rec/file = 0

Fields to be saved in history file: (T/F)
T write zeta free-surface.
T write UBAR 2D U-momentum component.
T write VBAR 2D V-momentum component.
T write U 3D U-momentum component.
T write V 3D V-momentum component.
T write T( 1) Tracer of index 1.
T write T( 2) Tracer of index 2.

  F  write RHO   Density anomaly.
  F  write Omega Omega vertical velocity.
  T  write W     True vertical velocity.
  F  write Akv   Vertical viscosity.
  T  write Akt   Vertical diffusivity for temperature.
  F  write Aks   Vertical diffusivity for salinity.
  F  write bvf   Brunt Vaisala Frequency.
  F  write Visc3d Horizontal diffusivity.

  T  write Hbl   Depth of model boundary layer.
  T  write Hbbl   Depth of bottom planetary boundary layer.
  T  write Bostr Bottom Stress.
  F  write Bustr U-Bottom Stress.
  F  write Bvstr V-Bottom Stress.
  T  write Wstress Wind Stress.
  T  write U-Wstress comp. U-Wind Stress.
  T  write V-Wstress comp. V-Wind Stress.

  T  write Shflx [W/m2] Surface net heat flux
  T  write Swflx [cm/day] Surface freshwater flux (E-P)
  T  write Shflx_rsw [W/m2] Short-wave surface radiation

WARNING: Unrecognized keyword: gls_history_fields → DISREGARDED.

Fields to be saved in averages file: (T/F)
T write zeta free-surface.
T write UBAR 2D U-momentum component.
T write VBAR 2D V-momentum component.
T write U 3D U-momentum component.
T write V 3D V-momentum component.
T write T( 1) Tracer of index 1.
T write T( 2) Tracer of index 2.

  F  write RHO   Density anomaly
  T  write Omega Omega vertical velocity.
  T  write W     True vertical velocity.
  F  write Akv   Vertical viscosity
  T  write Akt   Vertical diffusivity for temperature.
  F  write Aks   Vertical diffusivity for salinity.
  F  write bvf   Brunt Vaisala Frequency.
  F  write diff3d Horizontal diffusivity

  T  write Hbl   Depth of model boundary layer
  T  write Hbbl   Depth of the bottom planetary boundary layer
  T  write Bostr Bottom Stress.
  F  write Bustr U-Bottom Stress.
  F  write Bvstr V-Bottom Stress.
  T  write Wstr Wind Stress.
  T  write U-Wstress comp. U-Wind Stress.
  T  write V-Wstress comp. V-Wind Stress.

  T  write Shflx [W/m2] Surface net heat flux.
  T  write Swflx [cm/day] Surface freshwater flux (E-P)
  T  write Shflx_rsw [W/m2] Short-wave surface radiation.

WARNING: Unrecognized keyword: gls_averages → DISREGARDED.

1025.0000 rho0 Boussinesq approximation mean density, kg/m3.

WARNING: Unrecognized keyword: lateral_visc → DISREGARDED.

WARNING: Unrecognized keyword: tracer_diff2 → DISREGARDED.

0.000E+00 tnu4( 1) Horizontal biharmonic mixing coefficient [m4/s]
for tracer 1.
0.000E+00 tnu4( 2) Horizontal biharmonic mixing coefficient [m4/s]
for tracer 2.

WARNING: Unrecognized keyword: vertical_mixing → DISREGARDED.

0.000E+00 rdrg Linear bottom drag coefficient (m/si).
0.000E+00 rdrg2 Quadratic bottom drag coefficient.
1.000E-02 Zobt Bottom roughness for logarithmic law (m).
1.000E-04 Cdb_min Minimum bottom drag coefficient.
1.000E-01 Cdb_max Maximum bottom drag coefficient.

  1.00  gamma2   Slipperiness parameter: free-slip +1, or no-slip -1.

WARNING: Unrecognized keyword: sponge → DISREGARDED.

1.157E-05 tauT_in Nudging coefficients [sec^-1]
3.215E-08 tauT_out Nudging coefficients [sec^-1]
3.858E-06 tauM_in Nudging coefficients [sec^-1]
3.215E-08 tauM_out Nudging coefficients [sec^-1]

WARNING: Unrecognized keyword: diagnostics → DISREGARDED.

WARNING: Unrecognized keyword: diag_avg → DISREGARDED.

WARNING: Unrecognized keyword: diag3D_history_fields → DISREGARDED.

WARNING: Unrecognized keyword: diag2D_history_fields → DISREGARDED.

WARNING: Unrecognized keyword: diag3D_average_fields → DISREGARDED.

WARNING: Unrecognized keyword: diag2D_average_fields → DISREGARDED.

WARNING: Unrecognized keyword: diagnosticsM → DISREGARDED.

WARNING: Unrecognized keyword: diagM_avg → DISREGARDED.

WARNING: Unrecognized keyword: diagM_history_fields → DISREGARDED.

WARNING: Unrecognized keyword: diagM_average_fields → DISREGARDED.

WARNING: Unrecognized keyword: diags_vrt → DISREGARDED.

WARNING: Unrecognized keyword: diags_vrt_avg → DISREGARDED.

WARNING: Unrecognized keyword: diags_vrt_history_fields → DISREGARDED.

WARNING: Unrecognized keyword: diags_vrt_average_fields → DISREGARDED.

WARNING: Unrecognized keyword: diags_ek → DISREGARDED.

WARNING: Unrecognized keyword: diags_ek_avg → DISREGARDED.

WARNING: Unrecognized keyword: diags_ek_history_fields → DISREGARDED.

WARNING: Unrecognized keyword: diags_ek_average_fields → DISREGARDED.

WARNING: Unrecognized keyword: surf → DISREGARDED.

WARNING: Unrecognized keyword: surf_avg → DISREGARDED.

WARNING: Unrecognized keyword: surf_history_fields → DISREGARDED.

WARNING: Unrecognized keyword: surf_average_fields → DISREGARDED.

WARNING: Unrecognized keyword: diags_pv → DISREGARDED.

WARNING: Unrecognized keyword: diags_pv_avg → DISREGARDED.

WARNING: Unrecognized keyword: diags_pv_history_fields → DISREGARDED.

WARNING: Unrecognized keyword: diags_pv_average_fields → DISREGARDED.

WARNING: Unrecognized keyword: diags_eddy → DISREGARDED.

WARNING: Unrecognized keyword: diags_eddy_avg → DISREGARDED.

WARNING: Unrecognized keyword: diags_eddy_history_fields → DISREGARDED.

WARNING: Unrecognized keyword: diags_eddy_average_fields → DISREGARDED.

WARNING: Unrecognized keyword: diagnostics_bio → DISREGARDED.

WARNING: Unrecognized keyword: diagbio_avg → DISREGARDED.

WARNING: Unrecognized keyword: diagbioFlux_history_fields → DISREGARDED.

WARNING: Unrecognized keyword: diagbioVSink_history_fields → DISREGARDED.

WARNING: Unrecognized keyword: diagbioGasExc_history_fields → DISREGARDED.

WARNING: Unrecognized keyword: diagbioFlux_average_fields → DISREGARDED.

WARNING: Unrecognized keyword: diagbioVSink_average_fields → DISREGARDED.

WARNING: Unrecognized keyword: diagbioGasExc_average_fields → DISREGARDED.

WARNING: Unrecognized keyword: biology → DISREGARDED.

WARNING: Unrecognized keyword: wkb_boundary → DISREGARDED.

WARNING: Unrecognized keyword: wkb_wwave → DISREGARDED.

WARNING: Unrecognized keyword: wkb_roller → DISREGARDED.

WARNING: Unrecognized keyword: wave_history_fields → DISREGARDED.

WARNING: Unrecognized keyword: wave_average_fields → DISREGARDED.

WARNING: Unrecognized keyword: wci_history_fields → DISREGARDED.

WARNING: Unrecognized keyword: wci_average_fields → DISREGARDED.

WARNING: Unrecognized keyword: sediments → DISREGARDED.

WARNING: Unrecognized keyword: sediment_history_fields → DISREGARDED.

WARNING: Unrecognized keyword: bbl_history_fields → DISREGARDED.

WARNING: Unrecognized keyword: floats → DISREGARDED.

WARNING: Unrecognized keyword: float_fields → DISREGARDED.

WARNING: Unrecognized keyword: stations → DISREGARDED.

WARNING: Unrecognized keyword: station_fields → DISREGARDED.

WARNING: Unrecognized keyword: psource → DISREGARDED.

WARNING: Unrecognized keyword: psource_ncfile → DISREGARDED.

WARNING: Unrecognized keyword: online → DISREGARDED.

Activated C-preprocessing Options:

      REGIONAL
      TESTEBACIA
      OBC_EAST
      OBC_WEST
      OBC_NORTH
      OBC_SOUTH
      CURVGRID
      SPHERICAL
      MASKING
      NEW_S_COORD
      SOLVE3D
      UV_COR
      UV_ADV
      SALINITY
      NONLIN_EOS
      ANA_DIURNAL_SW
      QCORRECTION
      SFLX_CORR
      CLIMATOLOGY
      ZCLIMATOLOGY
      M2CLIMATOLOGY
      M3CLIMATOLOGY
      TCLIMATOLOGY
      ZNUDGING
      M2NUDGING
      M3NUDGING
      TNUDGING
      UV_HADV_UP3
      UV_VADV_SPLINES
      TS_HADV_RSUP3
      TS_DIF4
      TS_VADV_SPLINES
      LIMIT_BSTRESS
      LMD_MIXING
      LMD_SKPP
      LMD_BKPP
      LMD_RIMIX
      LMD_CONVEC
      LMD_NONLOCAL
      ANA_BSFLUX
      ANA_BTFLUX
      OBC_M2CHARACT
      OBC_M3ORLANSKI
      OBC_TORLANSKI
      AVERAGES
      AVERAGES_K
      TS_HADV_C4
      MPI_COMM_WORLD
      M2FILTER_POWER
      TRACERS
      TEMPERATURE
      HZR
      VAR_RHO_2D
      SPLIT_EOS
      UV_MIX_S
      DIF_COEF_3D
      TS_MIX_ISO
      TS_MIX_IMP
      TS_MIX_ISO_FILT
      NTRA_T3DMIX
      LMD_SKPP2005
      NF_CLOBBER
      ENDDOLOOP2D
      DOLOOP2D_R
      DOLOOP2D

NUMBER OF THREADS: 1 BLOCKING: 1 x 1.

GET_GRID ERROR: Cannot open input NetCDF file: CROCO_FILES/croco_grd.nc

MAIN - number of records written into history file(s): 0
number of records written into restart file(s): 0
number of records written into averages file(s): 0

ERROR: Abnormal termination: netCDF INPUT

This time, I am running without OPENMP and MPI, and using /croco croco.in.

Here my files crocotools_param.m, start.m, cppdefs.h, and param.h:

cppdefs.h (44.7 KB)
param.h (33.6 KB)
param.h (33.6 KB)
start.m (4.2 KB)

I couldn’t upload the croco.in file here, so I am sharing it via Google Drive:

I have uploaded the files from my CROCO_FILES to a folder on Google Drive:

https://drive.google.com/drive/folders/1re_r6PtgisMHs__wSCil2JTkfIQcqUw_?usp=sharing

eduardolima255 · June 3, 2024, 12:16pm

Hi,
I also created a new configuration to run the BENGUELA_LR test case, and I got the same error: “GET_GRID ERROR: Cannot open input NetCDF file: CROCO_FILES/croco_grd.nc”. The croco.in file (in the …/Run_BENGUELA_LR/croco.in directory) specifies the directory for CROCO_FILES (CROCO_FILES/croco_grd.nc, etc.). I considered the possibility of it being a permission error on my system, but the files have owner permissions.

NetCDF is also working; I am using the path …/croco_tools/UTILITIES/netcdf_x86_64, and the compilation was successful. I tested opening the croco_grd.nc file using ncdump -h, and it worked as well. In param.h, I used xi_rho - 2 and eta_rho - 2 for LLm0 and MMm0, respectively, as specified by make_grid.

eduardolima255 · June 3, 2024, 2:58pm

Hello, I managed to run the model. The problem was with the NetCDF provided in croco_tools. I installed NetCDF and specified the path in jobcomp, and it worked.