Search code examples
pythonfdb

python fdb save huge data from database to file

I have this script

SELECT = """
            select 
              coalesce (p.ID,'') as id, 
              coalesce (p.name,'') as name, 
            from TABLE as p  
         """
self.cur.execute(SELECT)
for row in self.cur.itermap():         
    id = '%(id)s' % row
    name = '%(name)s' % row

    xml +="  <item>\n"      
    xml +="    <id>"         + id + "</id>\n"
    xml +="    <name>"    + name + "</name>\n"    
    xml +="  </item>\n\n"
    
#save xml to file here
f = open...

and I need to save data from huge database to file. There are 10 000s (up to 40000) of items in my database and it takes very long time when script runs (1 hour and more) until finish.

How can I take data I need from database and save it to file "at once"? (as quick as possible? I don't need xml output because I can process data from output on my server later. I just need to do it as quickly as possible. Any idea?)

Many thanks!

P.S. I found out this interesting thing: When I use this code to "erase" xml variable every 2000 records and save it to another variable it works pretty fast! So there must be something "wrong" with filling in xml variable according to my former code.

result = float(id)/2000
if result == int(result):
  xml_whole += xml
  xml = ""
Solution

  • wow, after testing with code 

    result = float(id)/2000
if result == int(result):
  xml_whole += xml
  xml = ""

    is my script up to 50x faster! i would like to know why is python so slow with xml +=... ?