3

I have been working with some fairly large text files containing about two million lines of text. I don't know the length of the content or the lines in advance, just the number of lines. I have been considering two problems that probably have a similar solution:

  • How do I change or add to the nth line, without my computer having to shift everything after the nth line in the computer memory.
  • Say I want to delete all the lines in the text file except for the last 100k, every time the number of lines grows beyond two million. How do I keep these last 100k lines, without having to read the entire text file, deleting everything, and rewriting the last 100k?

I suppose it could be solved by putting everything in some database system. However, as that is no option for me, I want to know how to work with text files efficiently.

user4752157
  • 33
  • 2

1 Answers1

4

How do I change or add to the nth line

You can't.

You have lines of different lengths so you cannot overwrite a chunk of the file, "knowing" it to contain an entire "line".

If your records were all the same length, then you would probably be able tom using direct file access.

How do I keep these last 100k lines, without having to read the entire text file, deleting everything, and rewriting the last 100k?

You can't.

You don't know where each line begins and ends, so you have to read them line-by-line, starting at the beginning. In this case, the best you can do is to create a new file containing only the last 100k lines and then overwrite the original file with it.

Looking beyond your immediate problem(s), you're now well into the sort of sizes where a database would massively simplify your code and massively improve your application's performance.

Phill W.
  • 11,891
  • 4
  • 21
  • 36