Scalable stl set like container for C++

I need to store large number of integers. There can be duplicates in the input stream of integers, I just need to store distinct amongst them. I was using stl set initially but It went OutOfMem when input number of integers went too high. I am looking for some C++ container library which would allow me to store numbers with开发者_Python百科 the said requirement possibly backed by file i.e container should not try to keep all numbers in-mem. I don't need to store this data persistently, I just need to find unique values amongst it.

Take a look at the STXXL; might be what you're looking for.

Edit: I haven't used it myself, but from the docs - you could use stream::runs_creator to create sorted runs of your data (however much fits in memory), then stream::runs_merger to merge the sorted streams, and finally use stream::unique to filter uniques.

Since you need larger than RAM allows you might look at memcached

Have you considered using DB (maybe SQLite)? Or it would be too slow?

You should seriously at least try a database before concluding it is too slow. All you need is one of the lightweight key-value store ones. In the past I have used Berkeley DB, but here is a list of other ones.