How to convert netcdf time series data into individual text data which is classified based on latitude?

Question

I am preparing new input data for a fortran-based DGVM, but I am having trouble adjusting the data according to the DGVM format that will be used. I'm doing this task on a linux server.

Here is the climate data that I will use: download climate data MIROC6 historical monthly air temperature data from 1851 to 1860 (0.5 x 0.5).

I need to these steps in order

1. The text output data must be separated into 360 folders (each folder represent latitudes from 1-360) sample folders 1-360
2. In each folder there will be 720 text files (which represent longitude 1-720 on that latitude/folder) sample txt files inside each folder: 1-720
3. Inside the txt file, the order of the data from top to bottom represents monthly data (January to December, and repeated).

I tried converting the netcdf data by several method: CDO, NCO and ncdump comamnd and I'm able to get in one .txt format for whole grid value. But it will take huge time to extract/ classify the individual grid value according to the needed format manually. I hope there is other efficient way to convert those data, and I hope there is someone who willing to help me with this matter.

Thank you.

Answer 1

I found a solution by using the CDO outputtab operator in a loop, written in a simple Linux bash script. The conversion process is done every 20 points to avoid putting a heavy load on the server's CPU (this can be adjusted).

The code I created is as follows:

#!/bin/bash

# Define input file and log file
input_file="/path/canesm5_r1i1p1f1_w5e5_ssp126_tas_global_daily_2015_2100_05gc.nc"
log_file="log.txt"

# Remove existing log file if it exists
rm -f $log_file

# Function to extract data for a specific latitude
extract_latitude() {
    lat=$1
    lon_index=1
    max_jobs=20
    job_count=0

    # Loop through longitudes from -180 to 180 in 0.5 degree increments
    for lon in $(seq -180 0.5 179.5); do
        # Define the output file name based on the current latitude and longitude index
        output_file="${lat}_${lon_index}.txt"
        
        # Extract data for the grid point at the current latitude and longitude, save only values, and remove the header
        (cdo -outputtab,value -remapnn,lon=${lon}_lat=${lat} $input_file | sed '1d' > $output_file) &>> $log_file &
        
        # Increment the longitude index and job count
        lon_index=$((lon_index + 1))
        job_count=$((job_count + 1))

        # Check if the max number of jobs has been reached
        if [ $job_count -ge $max_jobs ]; then
            wait  # Wait for all background jobs to complete
            job_count=0  # Reset job count
        fi
    done

    # Wait for any remaining background processes to finish
    wait
}

# Loop through latitudes from 90N to -90S in 0.5 degree increments
for lat in $(seq 90 -0.5 -90); do
    echo "Extracting data for latitude $lat"
    extract_latitude $lat
done

echo "Data extraction completed. Check $log_file for details."

Thank you