OpenSSL: can input and output buffers be the same for encrypt/decrypt routines?

For example, in:

int EVP_EncryptUpdate(EVP_CIPHER_CTX *ctx, unsigned char *out,
                int *out开发者_StackOverflowl, unsigned char *in, int inl);

… can out == in?

I just stumbled upon this question because I was curious myself. Since no one answered, I tried it out and it does indeed work (at least with AES CTR 128 decryption) so I would venture to guess that it works for other types as well. Here's my code sample in case you are interested.

/* Test Vector from http://www.inconteam.com/software-development/41-encryption/55-aes-test-vectors */

const unsigned char key[16] = { 0x2b, 0x7e, 0x15, 0x16,
                                0x28, 0xae, 0xd2, 0xa6,
                                0xab, 0xf7, 0x15, 0x88,
                                0x09, 0xcf, 0x4f, 0x3c };

const unsigned char IV[16] = { 0xf0, 0xf1, 0xf2, 0xf3,
                               0xf4, 0xf5, 0xf6, 0xf7,
                               0xf8, 0xf9, 0xfa, 0xfb,
                               0xfc, 0xfd, 0xfe, 0xff };

unsigned char test[16] = { 0x6b, 0xc1, 0xbe, 0xe2,
                                 0x2e, 0x40, 0x9f, 0x96,
                                 0xe9, 0x3d, 0x7e, 0x11,
                                 0x73, 0x93, 0x17, 0x2a };

EVP_CIPHER_CTX mCtx;
EVP_DecryptInit(&mCtx, EVP_aes_128_ctr(), key, IV);
int out_size;
EVP_DecryptUpdate(&mCtx, test, &out_size, test, 16);

Inbuf and outbuf can be written the same in some cases, but there are many pitfalls.

Pitfall 1: If inbuf and outbuf write the same one, in the case of padding, you will find that the outgoing outlen is 16 bytes less than inlen, if it is parsed in blocks, each block will be 16 bytes less, and the parsed result is totally wrong! It is correct that inbuf and outbuf are not the same.

Pitfall 2: The openssl documentation clearly specifies that the length requirement of the outgoing outbuf of EVP_DecryptUpdate is (inlen + cipher_block_size). Generally, the cipher_block_size of AES is 16, so it means that the length of the binding buffer area must be prepared + the number of bytes prepared is 6. Otherwise, it will cause memory write overflow and produce unpredictable results. If inbuf and outbuf use the same one, the details of buffer area length must be handled well.

To sum up, don't pass inbuf and outbuf into the same one, digging pits and harming yourself.

---translated by google.