can collections.deque be extended to build a "file buffer"?

Question

I want to build a circular file buffer, in python, to hold file names (strings). The buffer should have the following properties.

• The size of the buffer is the sum of the sizes of the files whose names are stored in the buffer. The buffer will have a maximum permitted size.
• When a new file is added, if the buffer size is less than the maximum permitted size, that file name string is added. Else, the oldest modified file is pushed out and the new one is added. If the newly added file is older than all the files already present in the buffer, nothing takes place.

Is it possible to extend deque for such a purpose?

Or should I write it from scratch? Is there any design ideas that I can use for this purpose?

thanks

suresh

Answer 1

OK, I believe that Raymond Hettinger's interpretation of your question is correct, and your comment has clarified that you're not concerned with the length of the queue, but rather with the sum of all the filesizes. That makes a lot more sense, and I'm glad I finally understand what you mean. With that in mind, here's a simple implementation based on heapq that I believe satisfies all your stated requirements. Use it by putting (timestamp, filename, filesize) tuples on the queue, and note that when you get an item from the queue, it will be the oldest file (i.e. the file with the smallest timestamp.)

import heapq

class FilenameQueue(object):
    def __init__(self, times_sizes_names, maxsize):
        self.maxsize = maxsize
        self.size = sum(s for t, s, n in times_sizes_names)
        self.files = list(times_sizes_names)
        heapq.heapify(self.files)
        while self.size > self.maxsize:
            self.get()
    def __len__(self):
        return len(self.files)
    def put(self, time_size_name):
        self.size += time_size_name[1]
        if self.size < self.maxsize:
            heapq.heappush(self.files, time_size_name)
        else:
            time_size_name = heapq.heappushpop(self.files, time_size_name)
            self.size -= time_size_name[1]
    def get(self):
        time_size_name = heapq.heappop(self.files)
        self.size -= time_size_name[1]
        return time_size_name

I added a __len__ method so that you can test the queue before getting from it. Here's a usage example:

>>> f = FilenameQueue(((22, 33, 'f1'), (44, 55, 'f2'), (33, 22, 'f3')), 150)
>>> while f:
...     f.get()
... 
(22, 33, 'f1')
(33, 22, 'f3')
(44, 55, 'f2')
>>> f = FilenameQueue(((22, 33, 'f1'), (44, 55, 'f2'), (33, 22, 'f3')), 150)
>>> f.put((55, 66, 'f4'))
>>> while f:
...     f.get()
... 
(33, 22, 'f3')
(44, 55, 'f2')
(55, 66, 'f4')

See my edit history for a completely different solution involving Queue.PriorityQueue that is suboptimal. I forgot that maxsize enforces the limit by blocking, not by discarding elements. That's not so useful!