Process for telling when a new ec2 host can be connected to

I've been using fabric and boto to start up new ec2 hosts for some temporary processing but I've always had trouble knowing when I can connect to the host. The problem is that I can ask ec2 when something is ready but it's never really ready.

This is the process that I've noticed works best (though it still sucks):

1. Poll ec2 until it says that the host it "active"
2. Poll ec2 until it has a public_dns_name
3. Try to connect to the n开发者_JAVA技巧ew host in a loop until it accepts the connection

But sometimes it accepts the connection seemingly before it knows about the ssh key pair that I've associated it with and then asks for a password.

Is there a better way to decide when I can start connecting to my ec2 hosts after they've started up? Has anyone written a library that does this nicely and efficiently?

I do the same for #1 and #2, but for #3 I have a code loop that attempts to make a simple TCP connection to the ssh port (22) with short timeouts and retry. When it finally succeeds, it waits five more seconds an then run the ssh command.

The timing and order in which sshd is started and the public ssh key is added to .ssh/authorized_keys may vary depending on the AMI you are running.

Note: I mildly recommend using the public IP address directly instead of the DNS name. The IP address is encoded in the DNS name, so there's no benefit to adding DNS lookups into the process.

EC2 itself doesn't have any way of knowing when your instance is ready to accept SSH connections; it operates on a much lower level than that.

The best way to do this is to update your AMI to have some sort of health servlet. It can be very simple -- just a few lines of web.py script -- that runs at the later stages of startup, and which just returns status code 200 to any HTTP request. By the time that servlet is responding to requests, everything else should be up too, so you can check your instance with exponential backoff on that URL.

If you ever put your instances behind a load balancer (which has its own benefits), this health servlet is required anyway, and has the added benefit of telling the load balancer when an instance has gone down, for any reason. It's just a general best-practice on EC2.