Okay, I set the netcdf path to the following in jobcomp and it worked:
NETCDFLIB=“-L/usr/lib/x86_64-linux-gnu -lnetcdff -lnetcdf -lnetcdf -lm”
I have some interesting results.
- Compilation: It completes successfully with both the #define DBLEPREC and -r8 (for FP64) and without them (for FP32). I include the compilation logs from both cases.
- Run: I tested the Kelvin-Helmoltz test case in 3D mode, with both OPENACC and MPI, and with only OPENACC and nvhpc-nompi
# define
KH_INST
CPP options:
# undef KH_INSTY
# define KH_INST3D
# define OPENACC
# define MPI (and second time with # undef MPI)
# define NBQ
Results for FP64:
on 13900H CPU only with MPI (14 cores) run time: 72 min
on Laptop RTX 2000 Ada (similar to RTX 4060 mobile but lower frequency) run time with MPI & OPENACC: 709 min (extrapolated)
on RTX 2000 Ada with only OPENACC and nvhpc-nompi, run time: 65 min
Results for FP32:
When compiled with /* #define DBLEPREC */ and no -r8 flag, compiles but run fails (attached logs)
When compiled with /* #define DBLEPREC */ but including the -r8 flag, compiles and runs in identical time and debug log as in FP64: 65 min, but netCDF history file is corrupted.
In essence, I only managed to run GPU in FP64 mode. Taking into account that the small mobile GPU is faster than the 14-core CPU in nvhpc-nompi mode, and the potential to be at least 10x times faster in FP32 based on the TFLOP difference between the FP64 and FP32, there is great potential for up to 50x faster on a high-end workstaion GPU e.g. RTX 5000 Ada in FP32 compared to a workstation CPU!
I hope we can make the FP32 mode work to test this.
Regards
Konstantinos
GPU compile FP64 log.txt (238.4 KB)
GPU compile FP32 log.txt (239.5 KB)
Failed FP32 run log.txt (23.2 KB)