PHP file_get_contents now gets forbidden error

I'm working on a PHP application that will cycle through a list of URLs checking to see if the link is valid or not. The way I accomplish this is by opening the URL using the php function file_get_contents. I then search for a certain string value within the page source in order to determine if the link is good or bad. So in testing the application, towards the end of the day, whenever I would try to check a URL on this website I w开发者_如何学Goould get this message:

failed to open stream: HTTP request failed! HTTP/1.1 403 Forbidden in...

The message is a bit longer containing information about the location of my code but this part is the part that stood out to me. I'm thinking that maybe the companies router/firewall thinks I'm trying to spam/attack them based off what I have been Googling. I'm wondering if I might be on some permanent "blacklist" or something like that and how would I find out? I wasn't trying to do anything bad. Actually, what I'm doing will help out this company as I'm doing something that will help to generate sales. Total accident :-) I'm going to call the company later and ask them about it.

Many sites block access from user agents that fail to identify themselves. Introduce yourself properly and you're likely to get better service.

ini_set('user_agent', "CharlesUserAgent1.0"); // Anything usually should do as long as it's not blank

EDIT: You may also want to check out cURL, it does a much better job at making HTTP requests than PHP's builtin URL fopen wrappers.

1. it could be the websites check the user_agent header and then blocks your request.
2. Some URLs could be having a query string and the file_get_contents might not be able to perform your request as a normal browser would, thus the page you will be requesting could be something which is actually forbidden :/

Browse the URLs manually and see if you get the same error