What is the correct way to use the Crypto Api for encryption and decryption between client and server?

After many headaches and people advising to stop, I finally managed to get my Server/Client App to work with this API and create the required keys, i.e. Session and Exchange.

When I send the public key to the client, it successfully imports the key and will also encrypt a message using that key, but when I pass it back to the server; it decrypts the message using the session key but the message is returned as garbage (hmm.. private key is needed!). Now this could be due to way I am passing the encrypted message back via rpc, but something tells me it is something else. Ideally what I need is a clear and plain explanation of what it is I should be doing with all these keys, because the information I am currently getting is quite confused.

Do I pass the exchange public key to the client so it can encrypt a message and return for decryption.

Or:

Should I actually be encrypting the clients session key with the servers public key and then return that? (This doesn't so开发者_运维技巧und right to me, but I am all ears!!!)

Please leave out comments to move to another API, or copy pasties from MSDN (I have already read all that). I am working with the Crypto API and just need a clear explanation of what keys the server should pass to the client, and then what the client should do and pass back so I can finally move on...

Sounds like you are on the right track if you really are determined to stick with that API :)

There are two distinct families of encryption algorithms in cryptography. 1) Ones that use symmetric keys and 2) those that use asymmetric keys. Symmetric key algorithms (e.g. AES, DES...) are very fast and should be used as long as there's a safe way to make sure both client and server have the same key (i.e. session key) and no one else can gain access to that key. On the other hand, asymmetric key algorithms (e.g. RSA...), which are also known private/public key algorithms, are much more computationally expensive. They have one key which can only be used to encrypt data and a second key which can only be used to decrypt data. These algorithms, as you found out, are perfect for the initial handshake and session key exchange. The server creates public/private key pair and sends the client the public key. Anyone can intercept it, but when the client encodes the session key and sends that back, pbulic key is useless if an eavesdropper wants to find out the session key. Only the server can decode the message as it is the only entity that is holding the private key. So your initial problem was that when the message came back, instead of using the private key from the pair, you were using synchronous session key and thus were getting garbage.

Essentially you've just implemented the basic handshake that SSL does (and you could easily do with very few lines of code if using OpenSSL library).

Once the handshake is performed you now have a secure channel between the client and the server. The only problem you might have is, what if someone piggy backs on your server's IP address and starts pretending like they are the real server? Your client will think he is talking to the real server, it'll do the key exchange and will start sending secure information, but that information might all end up in malicious hands if an attacker's PC happens to be on the other end.

This is where SSL's use of certificates comes in. Certificates are another example of where public/private keys are used. A trusted authority uses private key to sign certificates hash code and anyone can verify that certificate is valid by using it's attach public key against certificates identity data. This way even if attacker takes over your server's IP address, it won't be able to spoof your server's certificate.