Hello everyone,
I am using CROCO Pytools v2.0.5 (2026-06-22) to generate hourly boundary conditions from Copernicus IBI data for a high-resolution, interannual CROCO simulation around the Canary Islands.
During preprocessing, I identified several potential issues and developed a Python wrapper to address them without modifying the original toolkit. I would appreciate feedback.
1. Major performance improvement through time-first loading
I noticed that the original preprocessing workflow loads the spatial subset of an external dataset into memory before selecting the requested timestamp and variable. With large hourly datasets, this results in considerable unnecessary I/O.
I modified the loading order to select the required timestamp and variable before loading the data.
The following results were obtained for a monthly boundary-generation job:
| Metric | Original toolkit | Time-first wrapper |
|---|---|---|
| Horizontal grid resolution | 500 m | 350 m |
| Execution time | 142 h 20 min | 20 min 51 s |
| Maximum memory usage | 24.3 GiB | 0.98 GiB |
This corresponds to an observed speedup of approximately 410 times, alongside a substantial reduction in memory usage.
The comparison is not perfectly controlled because the grid resolution changed. However, the geographical domain, vertical levels and temporal coverage remained the same. These measurements refer to the time-first loading modification alone, before the subsequent interpolation corrections.
2. Potential errors in horizontal extrapolator caching
I identified two issues involving NearestNDInterpolator:
-
Cached extrapolators retain source field values, potentially causing missing cells at later timestamps to be filled with outdated values.
-
In
get_uv(), depth-averageduandvcan share an extrapolator, potentially causing missingvvalues to be filled usinguvalues.
My current wrapper forces fresh extrapolators for every field. A possible future optimization would be to cache only the source geometry or nearest-neighbour indices, while always retrieving values from the current field.
3. Potential depth-coordinate mismatch
In interp_horiz_var3D(), source depth levels with insufficient valid points are discarded, but the remaining fields are assigned the first N source depth coordinates.
If an intermediate level is discarded while a deeper one is retained, this can result in incorrect depth assignments during subsequent vertical interpolation.
My wrapper explicitly tracks the retained depth indices. This issue was not triggered by my particular IBI dataset, but was reproduced and corrected in a synthetic test.
4. Monthly boundary-file time coverage
I noticed that when using the MONTHLY output option, CROCO Pytools restricts the time records written to each boundary file according to predefined monthly segments, regardless of how far the requested interpolation period extends.
For my interannual simulation, I need the January boundary file to contain data through February 1 at 01:00, providing an additional hourly record beyond the monthly endpoint. However, the original script does not allow this: the February 1 at 01:00 record can only be written to the February boundary file.
To address this, I extended the available input data period by two days and modified the monthly time-segmentation logic so that each monthly boundary file includes the actual forcing record at 01:00 on the first day of the following month.
For example, the January 2025 boundary file now includes data through February 1, 2025 at 01:00, even though that timestamp also belongs to the February file.
This modification currently applies only to hourly forcing data (ibc_freq = 1h).
Validation and feedback
The wrapper preserves the original horizontal bicubic interpolation, vertical interpolation, velocity correction and NetCDF-writing procedures. The changes are applied temporarily, leaving the original toolkit files unchanged.
The interpolation corrections passed seven regression tests, and short boundary-generation runs were successfully completed across all four boundaries.
I would particularly appreciate the developers’ feedback on:
-
Whether these identified issues and proposed corrections are valid.
-
Whether my wrapper could introduce unintended numerical consequences.
-
Whether some improvements, particularly time-first loading and extrapolator caching, could be integrated into the official preprocessing tools.
I would be happy to share the wrapper.
Thank you for maintaining CROCO Pytools and for any feedback!