I need to grab rows in a database which contain an item that matches any of 175,000 items and convert the results into a csv file (which I will later parse and analyze with python script). Some issues that come to mind are: [can u actually input that large a list of items into a workbench sql query (there is not enough memory to copy it)? would the network support such a large data transfer? other things I don't know?] What would be a smart way to query and fetch this large amount of data? I am using MySql Workbench on windows to a windows server, but am open to trying a better interface option.
Simple (but not practical in this case) query format:
select * from database where date>='2017-06-01 00:00:00' and date<='2017-07-01 00:00:00' and instr in ('ab123', 'azx0456', 'rtpz888')
*there should be about 10,000,000 records (or rows) between those two specified dates. *the "instr in (...)" part actually would require a list of about 175,000 unique items
select * from database where date>='2017-06-01 00:00:00' and date<='2017-07-01 00:00:00' and instr in (select instr from xx)
I have not really using sql for a while. This should be fine.
the exporting part select * from database into outfile "aa.txt" where ....