I have created a DQN with a max memory size of 100000. I have a function that removes the oldest element in the memory if its size is greater than the max size. When I ran it doing 200 episodes, I noticed that the memory was already full at the 125th episode. Is it okay that my DQN will delete its oldest experience for the remaining episodes?
Yes. Naturally with dqn's it's fine to remove older episodes from the buffer. However the agent may not train as well if there was important events that lead to reward in those earlier episodes. What is the size of your observation space, is it an image or something? You should be able to calculate the size of the total buffer in gigabytes. As long as this fits on your ram I would suggest increasing your buffer size. If you increase it beyond ram capacity the program will crash. I'd also recommend using the python collection "deque" for your buffer. It acts as an array with a max size. So as you append to it when it is at its limit it will remove the first elements automatically so the buffer is always the same size.