What's wrong with IBM's JCE provider?

I have a JCE test that works fine with all Sun JDKs I have tried, but fails with various IBM J9 JDKs (e.g. 1.6.0 build pwi3260sr8-20100409_01(SR8)). The exception below happens when the cipher is initialized in encrypt mode. Why can the IBM JCE not use its own private key? Am I missing something in my code?

  public void testBasicKeyGeneration() throws NoSuchAlgorithmException, 
      NoSuchPaddingException, InvalidKeyException, IllegalBlockSizeException, 
      BadPaddingException, NoSuchProviderException, SignatureException {
      KeyPairGenerator generator = KeyPairGenerator.getInstance( "RSA" );
      generator.initialize( 2048 );
      KeyPair pair = generator.generateKeyPair();

      String data1 = "123456789012345678901234567890123456789012345678901234567890";
      Cipher cipher = Cipher.getInstance( "RSA" );
      cipher.init( Cipher.ENCRYPT_MODE, pair.getPrivate() );
      byte[] encrypted = cipher.doFinal( data1.getBytes() );

      cipher.init( Cipher.DECRYPT_MODE, pair.getPublic() );
      byte[] decrypted = cipher.doFinal( encrypted );
      String data2 = new String( decrypted );
      assertEquals( "en/decryption failed", data1, data2 );
  }

Here is the stack trace:

java.security.InvalidKeyException: Private key cannot be used to encrypt.
at com.ibm.crypto开发者_开发知识库.provider.RSA.engineInit(Unknown Source)
at javax.crypto.Cipher.a(Unknown Source)
at javax.crypto.Cipher.a(Unknown Source)
at javax.crypto.Cipher.init(Unknown Source)
at javax.crypto.Cipher.init(Unknown Source)
at test.Test.testBasicKeyGeneration(LicenseHelperTest.java:56)

There is a solution, see http://www-01.ibm.com/support/docview.wss?uid=swg1IV18625

with the property

-Dcom.ibm.crypto.provider.DoRSATypeChecking=false

you can use private keys to encrypt data.

I don't know this for sure but I believe that the JCE has an embedded policy limiting encryption to the public key and decryption to the private key.

In the example code the encryption was done with the private key. This would require the public key to decrypt, meaning that anyone with the public key could access the encoded data. Although this has it's uses it is not the accepted pattern and the IBM implementation may be "protecting" you from accidentally creating encrypted data that was publicly readable.

The fact that it tested properly when these were reversed tends to confirm my suspicions but I haven't yet found an official document stating as much.

IBM insists private keys cannot be used for encryption and public keys cannot be used for decryption, so they either see this artificial restriction as a feature, or someone is seriously confused here.

Here is how I worked around this problem:

RSAPrivateCrtKey privateKey = (RSAPrivateCrtKey) ks.getKey(keyAlias, ksPassword.trim().toCharArray());
RSAPublicKeySpec spec = new RSAPublicKeySpec(privateKey.getModulus(),privateKey.getPrivateExponent());
Key fakePublicKey = KeyFactory.getInstance("RSA").generatePublic(spec);
encryptCipher.init(Cipher.ENCRYPT_MODE, fakePublicKey);

Essentially, I created a public key object with private key's crypto material. You will need to do the reverse, create a private key object with public key's crypto material, to decrypt with public key if you want to avoid the "Public key cannot be used to decrypt" exception.

I recently ran in to the same problem. This was eventually solved by using the bouncy castle implementation and adding this line to the java.security file

security.provider.1=org.bouncycastle.jce.provider.BouncyCastleProvider

@T.Rob commented that you may have made a mistake in encrypting with the private key. If "everyone" knows the public key, then anyone can decrypt your file. IBM's JCE behaviour is thus protecting people against this mistake.

I can see the logic of that.

However, there may be cases where you really do need to encrypt with the private key; e.g. as part of a protocol that needs to prove that you know the private key corresponding to a published public key.

If this is really what you want to do, you probably need to use a recent Sun JCE implementation (older Sun JCEs didn't implement RSA), or Bouncy Castle.

@Stephen C / @FelixM: IBM seems to be completely clueless about how RSA cryptography works and how it is intended to be used. Basically both operations (encrypt / decrypt) must be available for the public AND private key.

Encrypt with public key is needed to transmit the client-side part of the pre master secret in SSL/TLS handshakes. The server needs to decrypt with its private key. But if they negotiate something like ECDHE_RSA the server needs to SIGN parts of the handshake with the private key - thats encrypt with PrivateKey. Vice versa the client needs to decrypt with the public key from the certificate of the server to verify the hash value of the signature. (proving authenticity of the message)

So if I try to run ECDHE_RSA (server-side) on latest IBM JDK 7 the following happens:

java.security.InvalidKeyException: Private key cannot be used to encrypt.
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614)
   at java.lang.Thread.run(Thread.java:777)
   at com.ibm.crypto.provider.RSASSL.engineInit(Unknown Source)
   at javax.crypto.Cipher.init(Unknown Source)
   at javax.crypto.Cipher.init(Unknown Source)
   at java.security.Signature$CipherAdapter.engineInitSign(Signature.java:1239)
   at java.security.Signature$Delegate.init(Signature.java:1116)
   at java.security.Signature$Delegate.chooseProvider(Signature.java:1076)
   at java.security.Signature$Delegate.engineInitSign(Signature.java:1140)
   at java.security.Signature.initSign(Signature.java:522)
   at net.vx4.lib.tls.core.TLSSignature.createSignature(TLSSignature.java:120)

As you can see we're using "Signature" and call "initSign", which requires indeed a PrivateKey. This proves IBM being clueless about this fact and obviously they don't even have valid regression tests!

Use another crypto provider and don't believe IBM until they change their mind.

Best regards, Christian