Difference between reading from memory and reading from a file on desk in performance

Question

I am working on a c# project where I have some text files (they are about 30mb) The program extract those files from a game client. then stores the content in memory (Lists) to work with. so the process I am working with is:

1. Parse game client.
2. Save content as txt files in a directory.
3. Use stream reader to read content line by line and store each line in lists.

But I end up with my program using about 100MB ram which is kinda a lot, cuz the program is meant to be a bot that runs multiple accounts for that game (20 accounts expected so that's about 2GB ram I believe).

Would it be better if I parse game client then save txt files and then read from the txt files when needed without storing content in memory? or would it be worse in performance?

Answer 1

The answer is...it depends on your use case.

If you're often referring to values that are stored within those files while the program is running, and those files are not too big, it generally makes sense to store them in memory. You get MUCH faster access than reading from disk.

One commonly used solution to this kind of issue is to use a cache (e.g. if you had more values than you can feasibly store in memory).

You can limit the size of your cache and store the most commonly used variables there. Have your program try to read from the cache first - if the value it wants is there (a hit) then job done. If the value isn't there (a miss), it will be fetched from disk and placed into the cache for future reference.

Old or less commonly used values can be evicted from the cache if it starts getting too full.

If you're running multiple instances of the same program, with common data, you you use programs like Redis as a cache that they all refer to. (i.e. one instance of 100mb, instead of 100mb x number of bots)