Search code examples
pythonpandasastropyastronomy

pandas DataFrame: How to avoid 'ValueError: Big-endian buffer not supported on little-endian compiler' when deleting certain values

I have various .fits files that contain data for spectra. I am converting the data to pandas DataFrames. Some of the .fits files contain two pages of data, which is why I include the try function. Once I have all of the data, I try to eliminate the rows that do not contain data:

try:
    df = pd.DataFrame(grism1D[1].data)
    dfb = pd.DataFrame(grism1D[2].data)
    df = df.append(dfb)
    df = df[df.flux != 0]
except:
    print("Missing Data")
    df = pd.DataFrame(grism1D[1].data)
    df = df.append(df)
    df = df[df.flux != 0]

This works, but if I do not include the df = df.append(df) line, I get an error 

ValueError: Big-endian buffer not supported on little-endian compiler

I have gotten around this in an exceptionally dumb manner by appending the data to itself, which essentially double plots the points of the spectrum on top of each other. This looks fine in the final image, but I am hoping there is a way to do this where I do not get the error messages without appending the data to itself.

I have also tried 

df=pd.DataFrame(np.array(grism1D).byteswap().newbyteorder())

which I found from another post about this error message, but this does not work for me.

Solution

  • Can you try something like this:

    from astropy.table import Table, vstack
tbl = Table.read('your_fits_file.fits', hdu=1)
try:
    tbl2 = Table.read('your_fits_file.fits', hdu=2)
except:
    pass
else:
    tbl = vstack([tbl, tbl2])

df = tbl.to_pandas()
df = df[df.flux != 0]