Just curious (so I know how it works): how do search engines find web-sites (if no one knows it) and folders in it? [closed]

Closed. This question is off-topic. It is not currently accepting answers. 开发者_如何转开发

---

Want to improve this question? Update the question so it's on-topic for Stack Overflow.

Closed 11 years ago.

Improve this question

1. The answers for the first question would be link to the web-site from crawling page(from the page search engine knows already). But, if you type very_long_name_without_any_sense_123kni.com, I guess it will find it anyway.

2. The second question is about folders.... If you have robots.txt in your root directory, then it's a bit clear. But, if you have no robots.txt on your web-site, how will search engine find all the folders that are allowed to be accessed?

3. If a search engine knows your web-site but your web-site has no robots.txt, how long will it take to appear at most popular search engine? In 10 minutes? 1 hour? 1 day? 1 week? never? How dangerous is it to leave pages (that should be protected) unprotected even for 1 minute, if your web-site is not crawled yed (because it's protected)?

P.S. These questions are not about steps how to make your web-site popular and to appear on the first pages among others... I'm just curious about principles how it works...

---

They can't, and don't.

that said, they can make some guesses based on knowing domain names (That information is accessible) and typical default website locations at those domain names.