Why can't my SharePoint 2010 Enterprise Search index a static HTML site?

I want to index this site: http://as开发者_StackOverflowia-pacific.undp.org, so I put it in as a content source, and immediately get:

Access is denied. Verify that either the Default Content Access Account has access to this repository, or add a crawl rule to crawl this repository. If the repository being crawled is a SharePoint repository, verify that the account you are using has "Full Read" permissions on the SharePoint Web Application being crawled.

But the thing is that I can open a command window as the content access user and fire off a HTTP request to the site and it works fine - from the desktop of the server.

So what else stops sites getting indexed?

P.S: It seems to be able to index my wordpress site, on a server overseas but not our local public machine.

Is the site you indexing actually the SharePoint server that is indexing the content? If so, you might want to disable the loopback check. It's fairly easy to do. You can run the command below in PowerShell:

New-ItemProperty HKLM:\System\CurrentControlSet\Control\Lsa -Name "DisableLoopbackCheck" -Value "1" -PropertyType dword

You might need to reboot the server for the change to take place.