Fuzzy queries to database

I'm curious about how works feature on many social sites today.

For exa开发者_StackOverflowmple, you enter list of movies you like and system suggests other movies you may like (based on movies that like other people who likes the same movies that you). I think doing it straight-sql way (join list of my movies with movies-users join with user-movies group by movie title and apply count to it ) on large datasets would be just impossible to implement due to "heaviness" of such query.

At the same time we don't need exact solution, approximate would be enough. I wonder is there way to implement something like fuzzy query to traditional RDBMS that would be fast to execute but has some infelicity. Or how such features implemented on real systems.

that's collaborative filtering, or recommendation

unless you need something really complex the slope one predictor is one of the more simple ones it's like 50 lines of python, Bryan O’Sullivan’s Collaborative filtering made easy, the paper by Daniel Lemire et al. introducing "Slope One Predictors for Online Rating-Based Collaborative Filtering"

this one has a way of updating just one user at a time when they change without in some cases for others that need to reprocess the whole database just to update

i used that python code to do predict the word count of words not occurring in documents but i ran into memory issues and such and i think i might write an out of memory version maybe using sqlite

also the matrix used in that one is triangular the sides along the diagonal are mirrored so only one half of the matrix needs to be stored

The term you are looking for is "collaborative filtering"

Read Programming Collective Intelligence, by O'Reilly Press

The simplest methods use Bayesian networks. There are libraries that can take care of most of the math for you.