how to use scrapy to simulate login done through POST request (encoded)

Is it possible to开发者_JS百科 simulate login on site like www.eoddata.com with scrapy? The website use a post request to submit login information.

http://readthedocs.org/docs/scrapy/en/latest/topics/request-response.html?highlight=login#using-formrequest-from-response-to-simulate-a-user-login

Yes, but due to the dynamic nature of the form you're may to need to read the login page first to get the exact name of the fields and any CSRF type fields. You'll want to follow a sequence like this

1. read original form page with your scraper
2. parse this HTML to get the name of username field, password field, and the name/value of any CSRF type authentication fields
3. send a POST to the login URL with the username field set to the username, the password field set to the password, and any other CSRF type fields captured in the previous step
4. get the authentication cookies in the response

Have a look at a login with Firebug to get the login URL, and to see if the field names and values are dynamic or not.

It seems scrapy itself is not enough, a sniffer tool like wireshark is a must the solve the problem :)