mySQLdb connection Returns a Truncated Output

Question

I'm trying to connect to a sql server remotely that runs a store procedure and returns a huge file as an output.

When I run the file locally on the SQLbox its fine and returns ~800,000 rows as expected, but when I try to run it using the mySQLdb library from python, it receives a truncated output of only ~6000 rows.

It runs fine for smaller data, so I'm guessing there's some result limit that's coming into play.

I'm sure there's some property that needs to be changed somewhere but there doesn't seem to be any documentation on the pypi library regarding the same.

For explanatory purposes, I've included my code below:

import MySQLdb
import pandas as pd

connection = MySQLdb.connect(sql_server,sql_admin,sql_pw,sql_db)
sql_command = """call function(4)"""
return pd.read_sql(sql_command, connection)

Answer 1

I was able to solve this using cursors. The approach I took is shown below and hopefully should help anyone else.

connection = MySQLdb.connect (host = sql_server, user = sql_admin, passwd = sql_pw, db = sql_db)
cursor = connection.cursor ()
cursor.execute("""call function(4)""")
data = cursor.fetchall()

frame = []
for row in data:
    frame.append(row)
return pd.DataFrame(frame)

cursor.close ()
# close the connection
connection.close ()