Search code examples
phpdatabasedbfdbase

Iterating through a large db


I already tried using PHP 5.3 with the dbase extension, but it doesn't work reliably for large databases over 2 GB. I need a way to iterate through a subsection of a large DBF and have the ability to read/edit fields. Can this be done (I use Windows)?

First attempt:

table = dbf.Table('myhugeDBF.dbf') 
#is this the only way to access the dbf data? 
#I only need the last 10k records as opposed to the whole 4.5 GB beast


table.open()

for i in xrange(len(table)-10000, len(table)):
    table[i].desc = (table[i].desc).replace("\n","")
    print "*" + str(table[i].desc) + "*" #for debug purposes

Solution

  • table = dbf.Table('myhugeDBF.dbf') 
    # is this the only way to access the dbf data?
    # yes.  the above only reads the header, though, so you can get basic
    # info about the dbf (size, field names, etc.)
    
    
    table.open()
    # this creates a data structure with one (small) element per record
    
    for record in table[-10000:]:
        with record:
            record.desc = record.desc.replace('\n','')