I have the file '3dhst.v4.1.5.master.fits.gz', which I want to open and make into a DataFrame. I want to pull out a selection of the columns from this, which I will then merge with another DataFrame. I have attempted the following:
import astropy
from astropy.io import fits
master = fits.open('3dhst.v4.1.5.master.fits.gz')
masterdf = pd.DataFrame(master[1].data)
masterdf = masterdf[['grism_id' , 'field' , 'ra' , 'dec' , 'z_best', 'z_best_s' , 'z_spec' ,
'z_peak_phot' , 'z_max_grism']]
I get:
ValueError: Big-endian buffer not supported on little-endian compiler
Which is an error message I've seen before with this file, and still do not understand why. The astropy documentation says 'The open() function will seamlessly open FITS files that have been compressed with gzip, bzip2 or pkzip. Note that in this context we are talking about a FITS file that has been compressed with one of these utilities (e.g., a .fits.gz file).' If I run just the opening of the file I get no errors. I also tried to open the file using:
with gzip.open('3dhst.v4.1.5.master.fits.gz') as f:
master = fits.open(f)
This returns an empty response to master[0].header, so this approach fails.
Using the Table class from Astropy is probably a better option, it has a method to convert to a pandas DataFrame:
from astropy.table import Table
t = Table.read('3dhst.v4.1.5.master.fits.gz')
t = t[['grism_id' , 'field' , 'ra' , 'dec' , 'z_best', 'z_best_s' , 'z_spec' , 'z_peak_phot' , 'z_max_grism']]
df = t.to_pandas()