Netcdf error in cluster

ze miscellaneous
1

NF_FREAD ERROR: nf_get_vara netCDF error code = -57 mynode = 2

GET_GRID - error while reading variable: h
in grid netCDF file: CROCO_FILES/croco_grd.nc

Spherical grid detected.

NF_FREAD ERROR: nf_get_vara netCDF error code = -57 mynode = 0

GET_GRID - error while reading variable: h
in grid netCDF file: CROCO_FILES/croco_grd.nc

MAIN - number of records written into history file(s): 0
number of records written into restart file(s): 0
number of records written into averages file(s): 0

ERROR: Abnormal termination: netCDF INPUT

any idea what this error is?

Error code -57 without parallelisation
2

Hi @febin,

It sounds like you have no bathymetry defined within your croco_grd.nc file. You probably missed a step when building it. The error code -57 can mean that there is an issue with variable dimension too.

  • check dimensions
ncdump -h CROCO_FILES/croco_grd.nc
  • check values
 ncdump -v "h" CROCO_FILES/croco_grd.nc
  • Also mynode=2 may suggest that you are using CROCO in parallel. If you defined PARALLEL_FILES cpp key you have to split your files using partit

./partit np_xi np_eta croco_grd.nc

same for any other file required at start like croco_ini.nc

I hope this helps,
Jeremy

3

Hello,

I found this thread while searching for the error message.
I’m getting the same error, sometimes the error_code is -40, sometimes -57

 ...
 NUMBER OF THREADS:  1 BLOCKING:  1 x  1.

 NF_FREAD ERROR: nf_get_vara netCDF error code =  -40  mynode = 127

 GET_GRID - error while reading variable: h
            in grid netCDF file: CROCO_FILES/run_nea/croco_grd_nea.nc

 NF_FREAD ERROR: nf_get_vara netCDF error code =  -40  mynode = 126

 GET_GRID - error while reading variable: h
            in grid netCDF file: CROCO_FILES/run_nea/croco_grd_nea.nc

 Spherical grid detected.

 NF_FREAD ERROR: nf_get_vara netCDF error code =  -57  mynode =   0

 GET_GRID - error while reading variable: h
            in grid netCDF file: CROCO_FILES/run_nea/croco_grd_nea.nc
 ...

I confirm that

  1. the variable his present in the grid file croco_grd_nea.nc.
  2. we don’t use parallel_files:
# ifdef MPI
#  undef  PARALLEL_FILES
#  undef  NC4PAR
#  undef  MPI_NOLAND
#  undef  MPI_TIME
# endif

The code was run with MPI and the error only appears for certain combinations of the parameters NP_XI and NP_ETA.

For instance:

  • NP_XI=1, NP_ETA=64 fails (with the error messages above)
  • NP_XI=2, NP_ETA=32 works.
4

Hi,

I rarely these kinds of combinations for NP_XI and NP_ETA. What are your grid dimensions?

5

Hello,

I agree, those combinations are not usual, yet my point here is to show that the error mentioned by febin can have its origin in the tiling.

My grid in this test was: 884 X 800.

Thanks

6

Hello,

The splitting 1x64 with your grid give really small mpi domain <15 points… I suggest to not use this splitting. The 2x32 is better.

But beside that, it seems your grid file does not match the compiled grid dimensions.

Can you precise which version of CROCO you are using ?
And can you attached your param.h, cppdefs.h and the results of ncdump -h on your grid file please ?

1 Like
7

Yes you’re right, 1 X 64 is not the best, I was testing different combinations to see which one was the fastest.

I’m using croco-v2.1.1.

This is because the error log I copied was from another experiment, but the same messages appear with the test 1 x 64 (except that mynode takes value from 0 to 63).

8

I think the issue comes from the MPI domain decomposition. With 800 grid points split over 64 MPI subdomains, you end up with about 12–13 points per subdomain. In that case, the last MPI domain can go out of bounds: for example, if 13 points are assigned per domain, then 13×63=819, which already exceeds 800 before reaching the last domain.

As far as I know, CROCO does not explicitly check for this situation in its MPI decomposition, so this kind of configuration can lead to indexing errors.

In this case, the safest solution is to use a decomposition that results in larger subdomains — typically at least 20–30 grid points per MPI domain — which avoids these boundary issues and is generally more robust.

1 Like
9

Thanks for the explanation! I was thinking the same

CROCO does not explicitly check for this situation in its MPI decomposition

Usually I try to have a more regular decomposition (8 X 8) but as I was testing how the code reacted with this type if tiling.

With regards

1 Like
10

you are undef PARALLEL_FILES It’s very good (they have a very critical bug) . but define NC4PAR . its good for mpi job. how many vertical levels you have in. in your system how many cores do you want to use at a time?
Thank you!

11

Thanks for the suggestion, I will check now with NC4PAR.

Now 32 levels but that might change soon, I’m just testing.

64 cores; I’ve tried with more but that did not make the simulation faster (I guess because of the MPI tiling).

12

Indeed. MPI tiling. (for better mpi you need your region of interest x/y ratio maintain square then decomposition is easy).
check there is an easy grid there you can choose LLm MMm etc (maybe).
and your OMP_NUM_THREADS is 1 or 2?
First try :
export HDF5_USE_FILE_LOCKING=FALSE
(Disabling locking does NOT make parallel I/O safe but try it first)
during compile the lib use --enable-parallel --enable-shared.
then test it all ($ make check all)
then check:
h5pcc -showconfig | grep -i parallel

which compiler did you use for the compiling model?
Thank you.

13

Thanks for the suggestions, I take note, unfortunately I cannot work anymore on this topic for the moment due to other committments, but I’ll surely try what you indicate and will keep you informed.

1 Like