Why does running SQLite (through python) cause memory to "unofficially" fill up?

I'm dealing with some big (tens of millions of records, around 10gb) database files using SQLite. I'm doint this python's standard interface.

When I try to insert millions of records into the database, or create indices on some of the columns, my computer slowly runs out of memory. If I look at the normal system monitor, it looks like the majority of the system memory is free. However, when I use top, it looks like I have almost no system memory free. If I sort the processes by their m开发者_如何学Cemory consuption, then non of them uses more than a couple percent of my memory (including the python process that is running sqlite).

Where is all the memory going? Why do top and Ubuntu's system monitor disagree about how much system memory I have? Why does top tell me that I have very little memory free, and at the same time not show which process(es) is (are) using all the memory?

I'm running Ubuntu 11.04, sqlite3, python 2.7.

10 to 1 says you are confused by linux's filesystem buffer/cache

see

• ofstream leaking memory
• https://superuser.com/questions/295900/linux-sort-all-data-in-memory/295902#295902

Test it by doing (as root)

echo 3 > /proc/sys/vm/drop_caches

The memory may be not assigned to a process, but it can be e.g. a file on tmpfs filesystem (/dev/shm, /tmp sometimes). You should show us the output of top or free (please note those tools do not show a single 'memory usage' value) to let us tell something more about the memory usage.

In case of inserting records to a database it may be a temporary image created for the current transaction, before it is committed to the real database. Splitting the insertion into many separate transactions (if applicable) may help.

I am just guessing, not enough data.

P.S. It seems I mis-read the original question (I assumed the computer slows down) and there is no problem. sehe's answer is probably better.