dictionary interface for large data sets

I have a set of key/values (all text) that is开发者_运维技巧 too large to load in memory at once. I would like to interact with this data via a Python dictionary-like interface.

Does such a module already exist?

Reading key values should be efficient and values compressed on disk to save space.

Edit:

• Ideally cross platform, but only using Linux for now
• Needs to be thread safe

Take a look at the shelve module. It provides dictionaries with various backing database systems include BerkleyDB and dbm.

Try bsddb(Python bindings for Berkley db)

ended up creating my own solution based on this existing module.

Assuming you are on a *nix platform, you are just BEGGING for tokyo-cabinet... It has a native set of ruby bindings... You can more information at 1978th.net...