How do search engine bots work? [closed]

Closed. This question needs to be more focused. It is not currently accepting answers.

---

Want to improve this question? Update the question so it focuses on one problem only by editing this post.

Closed 5 years ago.

开发者_高级运维 Improve this question

How are bots (that index for search engines) programmed and how do they operate?

---

10 Go to web page and get the html.
20 Index all the content in the page
30 Find all the links in that page
40 For each link in the page goto 10