Search code examples
pythonfiletextmmap

How to read lines from a mmapped file?

Is seems that the mmap interface only supports readline(). If I try to iterate over the object I get character instead of complete lines.

What would be the "pythonic" method of reading a mmap'ed file line by line?

import sys
import mmap
import os


if (len(sys.argv) > 1):
  STAT_FILE=sys.argv[1]
  print STAT_FILE
else:
  print "Need to know <statistics file name path>"
  sys.exit(1)


with open(STAT_FILE, "r") as f:
  map = mmap.mmap(f.fileno(), 0, prot=mmap.PROT_READ)
  for line in map:
    print line # RETURNS single characters instead of whole line
Solution

  • The most concise way to iterate over the lines of an mmap is

    with open(STAT_FILE, "r+b") as f:
    map_file = mmap.mmap(f.fileno(), 0, prot=mmap.PROT_READ)
    for line in iter(map_file.readline, b""):
        # whatever

    Note that in Python 3 the sentinel parameter of iter() must be of type bytes, while in Python 2 it needs to be a str (i.e. "" instead of b"").