Facing "MemoryError" while doing multithread txt file I/Os, looking for better solution

I'm working with only one txt file which is about 4 MB, and the file needs frequently I/O such as append new lines/search for certain lines which includes specific phrases/replace certain line with another line etc.

In order to process the file "at the same time", threading.RLock() is used to lock the resource when its under operation. As it's not a big file, I simply use readlines() to read them all into a list and do the search job, and also use read() to read the whole file into a string FileContent, and use FileContent.replace("demo", "test") to replace certain phrases with anything I want.

But the problem is, I'm开发者_如何学Python occasionally facing "MemoryError", I mean sometimes every 3 or 4 days, sometimes longer like a week or so. I've checked my code carefully and there's no unclosed file object when each thread ends. As to file operation, I simply use:

CurrentFile = open("TestFile.txt", "r")
FileContent = CurrentFile.read()
CurrentFile.close()

I think maybe python is not deleting useless variables as fast as I expected which finally result into out of memory, so I'm considering to use with statement which might be quick in garbage collecting. I'm not experienced with such statement, anybody knows if this would help? Or is there a better solution for my problem?

Thanks a lot.

Added: My script would do lots of replacement in a short period of time, so my guess is maybe hundreds of threads using FileContent = CurrentFile.read() would cause out of memory if FileContent not deleted quickly? How do I debug such problem?

Without seeing more of your code, it's impossible to know why you are running out of memory. The with statement is the preferred way to open files and close them when done though:

with open("TestFile.txt", "r") as current_file:
    file_content = current_file.read()

(sorry, UpperCamelCase for variables just doesn't look right to me...)

Frankly, I doubt this will solve your problem if you are really closing files as you show in the question, but it's still good practice.

Sounds like you are leaking memory. Python will use all available system memory before giving MemoryError and 4 MB does not sound much. Where you leak memory depends on your code which you didn't give in your question.

Have you watched the memory usage in the task manage of the OS?

Here is a tool to debug Python memory usage (needs Python debug compiliation):

http://guppy-pe.sourceforge.net/#Heapy

Use it to analyze your code memory usage and see what objects you are creating which don't get freed.