cURL gets less cookies than FireFox! How to fix it?

How can I make cURL to get all cookies?

I thought maybe firefox gets different cookies as the page loads or it has some built-in javascript that sets some cookies after the page is loaded, or maybe it redirects to other pages and other pages set other cookies, but I don't know how to make curl do the same thing. I set curl to follow redirects but still no success. Curl does sets some cookies but not all of them.

following is the code I use in php:

$url = 'https://www.example.com';
$handle = curl_init($url);
curl_setopt($handle, CURLOPT_COOKIESESSION, true);
curl_setopt($handle, CURLOPT_RETURNTRANSFER, true);
curl_setopt($handle, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($handle, CURLOPT_COOKIEJAR, "cookies.txt");
curl_setopt($handle, CURLOPT_COOKIEFILE, "cookies.txt");
curl_setopt($handle, CURLOPT_AUTOREFERER, true);
curl_setopt($handle, CURLOPT_USERAGENT, 'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.0.3705; .NET CLR 1.1.4322)');
$htmlContent = curl_exec($handle);

Following is from Live HTTP header in Firefox

https://www.example.com

GET /index.ext HTTP/1.1

Host: www.example.com User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2.10) Gecko/20100914 Firefox/3.6.10

Accept: text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8

Accept-Language: en-us,en;q=0.5

Accept-Encoding: gzip,deflate

Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7

Keep-Alive: 115

Connection: keep-alive

Cookie: JSESSIONID=3E85C5D0436D160D0623C085F68DC50E.catalog2; __utma=137925942.1883663033.1299196810.1299196810.1299198374.2; __utmz=137925942.1299196810.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); citrix_ns_id=0pQdumY48kxToPcBPS/QQC+w2vAA1; __utmc=137925942

HTTP/1.1 200 OK

开发者_StackOverflow社区

Date: Fri, 04 Mar 2011 01:20:30 GMT

Server: Apache/2.2.15

Keep-Alive: timeout=5, max=100

Connection: Keep-Alive

Transfer-Encoding: chunked

Content-Type: text/html;charset=UTF-8

I only get JSESSIONID with curl

Please help!

possibly page you are loading has some other content that actually sets cookies and since ou are only rading one page you don't get them, or some cookies are set through javascript.

Try using a Firefox user agent on CURL and see if you get the same amount of cookies. You should.

Use a network sniffer or a proxy to compare requests and responses, you have differences for sure. Post the requests and responses here if you still can't find.

If faking the user agent on curl side does not work, try to do the opposite by installing a firefox extension which fakes the user agent, and set it to the one used by curl. If it works, it may be some passive browser fingerprinting (such as p0f by lcamtuf) relying on network timing, and you may have hard time to workaround it. Would be extremely surprising though!

I figured it out. It was actually JavaScript that set cookies after the page was loaded:) Thanks everybody