Efficient way to delete lines from a file in python (and keep the same file name)?

My program is keeping a log for the user. If the log ever gets bigger than a set amount, I want to delete the first 20% of lines.

From similar questions, I've seen su开发者_如何转开发ggestions to do read in the old file, and write out all the lines I want to keep into a new file. However, my files might be too large to be constantly reading them in, and using that method wouldn't let me keep the same file name.

Can I delete lines from a file without reading in my old file?

The general method to achieve this for logfiles is 'rotation' - when the logfiles gets older or hits a certain size, you rename it and start writing a new one. If you are using logging module, there is even a preconfigured one - RotatingFileHandler that does this automatically.

As for your question: you can only truncate from the back, not from the beginning. An approximate solution would be to seek() to 20% of the file, find first '\n' and copy it out - but it will be slow and prone to race conditions. Go with logging and RotatingFileHandler.

As others have said, the traditional way to solve this problem is to keep 5 different files instead of 1 large one. When you need to delete 20%, just delete the oldest file and rename the others.

As handy as text files are, you might also consider a database. It is designed to be able to delete any part of the data at any time.

if size > MAX_SIZE:
    f = open(your_file, 'r')
    lines = f.readlines()
    f.close()

    f = open(your_file, 'w')
    f.write('\n'.join(lines[len(lines)/5:]))
    f.close()

That might do it. Although, as the first two said, it's much better to use multiple files or even a DB if you can. This code is not tested.