Allow search engine to crawl usernames [closed]

Closed. This question needs to be more focused. It is not currently accepting answers.

---

Want to improve this question? Update the question so it focuses on one problem only by editing this post.

Closed last year.

Improve this question

I have a site where users can enter their profile and password-protect certain details. I would like search engines to crawl the 'unprotected' parts of the profile (which varies from user to user). Similar to how if you enter a user's name in facebook, their Facebook profile comes up in the search results. Do I have to do anything special to ensure that the bot doesn't crawl the password-protected sections, but still crawls the (always-public) username? I'm not sure if this is even an issue, but I'd like to update my robots.txt to allow for this.

Also,开发者_如何学Python how do I ensure that the usernames are available to the bots (in a safe manner)? Do I have to create a separate directory with a list of names, or is there a better way?

Thanks for any advice

---

The search engines will only index what an anonymous user sees. If you don't already, I would create a listing page to browse the user profiles in which you only show the data you want to. This ensures that a link exists for every userProfile.aspx?uid=XXXXXX you have. The search engine spiders will not be able to see any data that is behind a password protected.

I would also add a site map to ensure the search engine spiders get to the listing page. Don't assume that Google will magically find ALL of your pages though typically they do based on links to your content. Submit a site map to Google.

Edit regarding Site Maps and Search Results
In order for spiders to crawl search results, I would specify an entry in the site map that points spiders to the search results page that displays all (e.g. search.aspx?param=all).

---

You don't have to do anything. Search bots won't be able to access to your protected pages while they'll access without problems to the public content as long as you don't explicitly disallow it on robots.txt