Search code examples
pythonsqlsqliteblaze

Using odo to migrate data to SQL

I have a large 3 GB CSV file, and I'd like to use Blaze to investigate the data, select down to the data I'm interesting in analyzing, with the eventual goal to migrate that data into a suitable computational backend such as SQlite, PostgresSQL etc. I can get that data into Blaze and work on it fine, but this is the part I'm having trouble with:

db = odo(bdata, 'sqlite:///report.db::report')`

I'm not sure how to properly create a db file to open with sqlite.

Solution

  • You can go directly from CSV to sqlite using the directions listed here. http://odo.pydata.org/en/latest/perf.html?highlight=sqlite#csv-sqlite3-57m-31s I think you are missing the column names as warned about here: http://odo.pydata.org/en/latest/sql.html?highlight=sqlite

    dshape = discover(resource('report_2015.csv'))
t = odo('report_2015.csv', 'sqlite:///report.db::report', dshape=dshape)