IndexError: too many indices for array Numpy Plotting CCDF with CSV

Question

I am trying to plot a CCDF using numpy and input is csv with #keywords as col[0] and frequency as col[1].

Input

#Car,45
#photo,4
#movie,6
#life,1

Input has more than 10K rows and two column out of which col[0] is not used at all and only the frequency from col[1] is used to plot the CCDF. The data has no empty rows in-between nor eof has any blank row.

Code:

import numpy as np
import matplotlib.pyplot as plt
import numpy as np
import matplotlib.pyplot as plt
from pylab import*
import math
from matplotlib.ticker import LogLocator

data = np.genfromtxt('input.csv', delimiter=",")

d0=data[:,1]
X0 = np.sort(d0)
cdf0 = np.arange(len(X0))/float(len(X0))
#cumulative = np.cumsum(data)
ccdf0 = 1 - cdf0
plt.plot(X0,ccdf0, color='b', marker='.', label='Frequency')

plt.legend(loc='upper right')
plt.xlabel('Freq (x)')
plt.ylabel('ccdf(x)')
plt.gca().set_xscale("log")
#plt.gca().set_yscale("log")
plt.show()

Error

Traceback (most recent call last):
  File "00_plot_ccdf.py", line 17, in <module>
    d0=data[:,1]
IndexError: too many indices for array

Thanks in Advance

Answer 1

genfromtxt by default treats lines starting with # as comments, so actually your data is empty:

In [1]: genfromtxt('test.csv', delimiter=',')         
/usr/lib/python3/dist-packages/numpy/lib/npyio.py:1385: UserWarning: genfromtxt: Empty input file: "test.csv"
  warnings.warn('genfromtxt: Empty input file: "%s"' % fname)
Out[1]: array([], dtype=float64)

data is a 1-dimensional empty array and so [:,1] is too many indices.

To disable this pass comments=None to genfromtxt:

In [20]: genfromtxt('test.csv', delimiter=',', comments=None)
Out[20]: 
array([[ nan,  45.],
       [ nan,   4.],
       [ nan,   6.],
       [ nan,   1.]])

Since you need only the 2. column, you can also limit the results to that directly:

In [21]: genfromtxt('test.csv', delimiter=',', comments=None, usecols=(1,))
Out[21]: array([ 45.,   4.,   6.,   1.])