ksh: is it safe to concatenate a really long string

I have a very large data file (2GB-3GB). I need to parse some data out of it and check if there is a duplication. So I have a empty string to start with, so data that I parse out from input file, will be check against this string. If it is 开发者_JAVA百科not already there, append it. This string can potential be very very long. Is it dangerous?

It is not dangerous. You just might have not enough memory to store a very very long string. So will encounter out of memory error.