Search code examples
pythonpython-3.xshutilastropyfits

PermissionError: [WinError 32] with fits files

I'm trying to read and move FITS files from one folder to an other using this code and the shutil package :

    MATISSE_DIR_N     = MATISSE_DIR+'N'  
    MATISSE_DIR_LM    = MATISSE_DIR+'LM'
    MATISSE_DIR_TRASH = MATISSE_DIR+'TRASH' 
    
        
    for filenames in glob.glob(MATISSE_DIR+'*.fits'):

        print(filenames)
        FOLDER_FLAG_LM    = False 
        FOLDER_FLAG_N     = False
        FOLDER_FLAG_TRASH = False
        
        if 'IR-N' in filenames:
            FOLDER_FLAG_N = True
        elif 'IR-LM' in filenames:
            FOLDER_FLAG_LM = True

        
        fichier = fits.open(filenames)    
        
        
        # VISIBILITY
        
#        Visibility_2_fichier = fichier["OI_VIS2"].data["VIS2DATA"]


        fichier.close()

        if np.logical_and(FOLDER_FLAG_TRASH==False,FOLDER_FLAG_N==True):
            shutil.move(filenames,MATISSE_DIR_N+'/')
        
        elif np.logical_and(FOLDER_FLAG_TRASH==False,FOLDER_FLAG_LM==True):
            shutil.move(filenames,MATISSE_DIR_LM+'/')
            
        elif FOLDER_FLAG_TRASH == True :
            shutil.move(filenames,MATISSE_DIR_TRASH+'/')

This works perfectly but when I uncomment the commented line :

        Visibility_2_fichier = fichier["OI_VIS2"].data["VIS2DATA"]

This does no longer works and rise the following error :

runfile('C:/Users/jdrevon/Desktop/THESE/Modeling/DATA_SORTING/untitled0.py', wdir='C:/Users/jdrevon/Desktop/THESE/Modeling/DATA_SORTING')
C:/Users/jdrevon/Desktop/THESE/DATA/DATA_RSCL_test/NOMEANBCD\NAMEOFMYFILE
Traceback (most recent call last):

  File "C:\Users\jdrevon\anaconda3\lib\shutil.py", line 788, in move
    os.rename(src, real_dst)

PermissionError: [WinError 32] Le processus ne peut pas accéder au fichier car ce fichier est utilisé par un autre processus: 'C:/Users/jdrevon/Desktop/THESE/DATA/DATA_RSCL_test/NOMEANBCD\\NAMEOFMYFILE' -> 'C:/Users/jdrevon/Desktop/THESE/DATA/DATA_RSCL_test/NOMEANBCD/LM/NAMEOFMYFILE'


During handling of the above exception, another exception occurred:

Traceback (most recent call last):

  File "C:\Users\jdrevon\Desktop\THESE\Modeling\DATA_SORTING\untitled0.py", line 76, in <module>
    V2_MATISSE,UV, UV_TP,TP_MATISSE,FLUX_MATISSE = OIFITS_READING(MATISSE_DIR)

  File "C:\Users\jdrevon\Desktop\THESE\Modeling\DATA_SORTING\untitled0.py", line 64, in OIFITS_READING
    shutil.move(filenames,MATISSE_DIR_LM+'/')

  File "C:\Users\jdrevon\anaconda3\lib\shutil.py", line 803, in move
    os.unlink(src)

PermissionError: [WinError 32] Le processus ne peut pas accéder au fichier car ce fichier est utilisé par un autre processus: 'C:/Users/jdrevon/Desktop/THESE/DATA/DATA_RSCL_test/NOMEANBCD\\NAMEOFMYFILE'

I don't understand why the fichier.close() command is not longer sufficient to close the file when you start to stock data coming from the data file. I've already tried also the with version of my code to open the file but this didn't change anything.

Solution

  • This warning in the documentation should answer your question (perhaps the warning should be moved elsewhere since it's not specific just to "large files"):

    When opening a file with memmap=True, because of how mmap works this means that when the HDU data is accessed (i.e., hdul[0].data) another handle to the FITS file is opened by mmap. This means that even after calling hdul.close() the mmap still holds an open handle to the data so that it can still be accessed by unwary programs that were built with the assumption that the .data attribute has all of the data in-memory.

    In order to force the mmap to close, either wait for the containing HDUList object to go out of scope, or manually call del hdul[0].data. (This works so long as there are no other references held to the data array.)