How to design a database schema for a search engine?

I'm writing a small search engine in C with curl, libxml2, and mysql. The basic plan is to grab pages with curl, parse them w开发者_运维问答ith libxml2, then iterate over the DOM and find all the links. Then traverse each of those, and repeat, all while updating a SQL database that maintains the relationship between URLs.

My question is: how can I best represent the relationship between URLs?.

Why not have a table of base urls (ie www.google.com/) and a table of connections, with these example columns:

• starting page id (from url table)
• ending page id (from url table)
• the trailing directory of the urls as strings in two more columns

This will allow you to join on certain urls and pick out information you want.

Your solution seems like it would be better suited to a non relational datastore, such as a column store.

Most search engine indices aren't stored in relational databases, but stored in memory as to minimize retrieval time.

Add two fields to table - 'id' and 'parent_id'.

id - unique identifier for URL parent_id - link between URL's

If you want to have a single entry for each URL then you should create another table that maps the relationships.

You then lookup the URL table to see if it exists. If not create it.

The relationship table would have

SourceUrlId,
UrlId

Where the SourceUrlId is the page and the UrlId is the url it points to. That way you can have multiple relationships for the same URL and you won't need to have a new entry in the Url table for every link to that url. Will also mean only 1 copy of any other info you are storing.

Why are you interested in representing pages graph?

If you want to compute the ranking, then it's better to have a more succinct and efficient representation (e.g., matricial form if you want to compute something similar to PageRank).