Java - Login to a website that is using form-based authentication

I'm trying to login to a site that is using form-based authentication so that my application can go in, download the protected pages, and then exit (yes, I have a valid username/password combination).

I know:

1. the url to the login page

2. the url to the login authenticator

3. the method (post)

4. my information (obviously)

5. the username and password fields (which change based on...something. I already wrote a method to get the names).

Currently I'm using the code at this dream.in.code page as a base for my efforts.

Every time I run the application, it gets the login page sent back with a "bad username/password" message.

Code:

import java.net.*;
import java.util.LinkedList;
import java.io.*;

import javax.swing.JOptionPane;

public class ConnectToURL
{

  // Variables to hold the URL object and its connection to that URL.
  private static URL URLObj;
  private static URLConnection connect;
  private static String loginField;
  private static String passwordField;

  private static void getFields()
  {
    try
    {
      URLObj = new URL("http://url.goes.here/login.jsp");
      connect = URLObj.openConnection();
      // Now establish a buffered reader to read the URLConnection's input
      // stream.
      BufferedReader reader = new BufferedReader(new InputStreamReader(
          connect.getInputStream()));

      String lineRead = "";
      LinkedList<String> lines = new LinkedList<String>();

      // Read all available lines of data from the URL and print them to
      // screen.
      while ((lineRead = reader.readLine()) != null)
      {
        lines.add(lineRead);
      }
      reader.close();

      while(lines.peekFirst().indexOf("<th>Username or E-mail:</th>") == -1)
      {
        lines.removeFirst();
      }
      String usernameCell = "";
      while (usernameCell.indexOf("</td>") == -1)
      {
        usernameCell = usernameCell + lines.removeFirst().trim();
      }
      usernameCell = usernameCell.substring(usernameCell.indexOf("name=\"") + 6);
      usernameCell = usernameCell.substring(0, usernameCell.indexOf("\""));
      loginField = usernameCell;

      while(lines.peekFirst().indexOf("<th>Password:</th>") == -1)
      {
        lines.removeFirst();
      }

      String passwordCell = "";
      while (passwordCell.indexOf("</td>") == -1)
      {
        passwordCell = passwordCell + lines.removeFirst().trim();
      }
      passwordCell = passwordCell.substring(passwordCell.indexOf("name=\"") + 6);
      passwordCell = passwordCell.substring(0, passwordCell.indexOf("\""));
      passwordField = passwordCell;
    }
    catch (MalformedURLException e)
    {
      // TODO Auto-generated catch block
      e.printStackTrace();
    }
    catch (IOException e)
    {
      // TODO Auto-generated catch block
      开发者_Go百科e.printStackTrace();
    }
  }

  public static void main(String[] args)
  {
    try
    {
      // getFields() grabs the names of the username and password fields and stores them into variables above
      getFields();
      // Establish a URL and open a connection to it. Set it to output
      // mode.
      URLObj = new URL("http://url.goes.here/login_submit.jsp");
      connect = URLObj.openConnection();
      HttpURLConnection.setFollowRedirects(true);
      connect.setDoOutput(true);
    }
    catch (MalformedURLException ex)
    {
      System.out
          .println("The URL specified was unable to be parsed or uses an invalid protocol. Please try again.");
      System.exit(1);
    }
    catch (Exception ex)
    {
      System.out.println("An exception occurred. " + ex.getMessage());
      System.exit(1);
    }

    try
    {
      // Create a buffered writer to the URLConnection's output stream and
      // write our forms parameters.
      BufferedWriter writer = new BufferedWriter(new OutputStreamWriter(
          connect.getOutputStream()));
      // For obvious reasons, login info is editted.
      // The line begins with username=& because there's a username field that send no data and is set to display:none.
      // When I observed the request in Chrome, username was sent, but left blank. Without it, my request doesn't go through.
      writer.write("username=&" + loginField + "=" + URLEncoder.encode("Username", "UTF-8") + "&" + passwordField + "=" + URLEncoder.encode("myPassword", "UTF-8"));
      writer.close();

      // Now establish a buffered reader to read the URLConnection's input
      // stream.
      BufferedReader reader = new BufferedReader(new InputStreamReader(
          connect.getInputStream()));

      String lineRead = "";

      // Read all available lines of data from the URL and print them to
      // screen.
      while ((lineRead = reader.readLine()) != null)
      {
        System.out.println(lineRead);
      }

      reader.close();
    }
    catch (Exception ex)
    {
      System.out.println("There was an error reading or writing to the URL: "
          + ex.getMessage());
    }
  }
}

I would try to use something like HttpFox or Fiddler to see what exactly is being sent during the login and try to emulate that. Sometimes login pages massage what is being sent with Javascript.

Use LiveHTTPHeaders to check out EVERYTHING that gets posted. There is probably cookie/session data that you aren't passing though to the POST command.

Also, referrer is sometimes monitored and should be faked as well by passing the header "Referrer: http://homepage.com.../login.html"

have you tried using Appache httpClient (http://hc.apache.org/httpcomponents-client-ga/) rather writing your own code where you are parsing html and inserting values?

I believe, you don't have to parse html. Your steps should be

• Look at the html and see to which "url" your authentication request is going to
• open your connection to the url you found rather sending to the "login page" and parsing it to find.
• httpclient class can help you manage your session to keep your session alive. you can do it urself but it would be a lot of work