java & phpseclib, RSA and OAEP?

I am encrypting in Java using Cipher.getInstance("RSA/ECB/OAEPWITHSHA-512ANDMGF1PADDING") and setEncryptionMode(CRYPT_RSA_ENCRYPTION_OAEP) in phpseclib, but the phpseclib is not decrypting the data correctly.

It worked perfectly when I used RSA/ECB/PKCS1Padding in Java, and setEncryptionMode(CRYPT_RSA_ENCRYPTION_PKCS1) in phpseclib.

Here are the supported ciphers in Java: http://download.oracle.com/javase/6/docs/technotes/guides/security/S开发者_运维百科unProviders.html#SunJCEProvider

Are none of those ciphers compatible with phpseclib's OAEP implementation?

The problem lies in the size of the keys used, had me puzzled for a while as well.

To use OAEP safely, you have to use >=2048 bit RSA keys.

Also, make sure you run

$rsa->setHash('sha512');
$rsa->setMGFHash('sha512');

before setEncryptionMode() on the PHP side.

edit: it seems 1024 keys won't work correctly even with sha256, so I've modified my answer to only include the safe 2048+ bits route.

You'd probably have to do $rsa->setHash('sha512'); By default sha1 is used.